You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 11:53
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack8slot4.osadl.org (updated Mon Nov 24, 2025 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2161932227169,19sleep50-21swapper/519:08:105
2161972225193,20sleep00-21swapper/019:08:130
2161062220166,18sleep30-21swapper/319:06:553
2160692219167,41sleep20-21swapper/219:06:222
2162462216186,19sleep40-21swapper/419:08:574
2161002210177,21sleep60-21swapper/619:06:506
2140762210179,19sleep70-21swapper/719:05:397
2161152209181,18sleep10-21swapper/119:07:011
21668399330,28cyclictest298930-21apt-get21:40:554
21668099320,31cyclictest371435-21turbostat.cron23:55:003
216693993128,2cyclictest376859-21dump-pmu-power00:05:007
216693993128,2cyclictest228127-21cron19:30:007
216693993127,3cyclictest306519-21apt21:55:007
21668099310,30cyclictest331258-21apt-get22:40:553
21667599310,29cyclictest376856-21sh00:05:002
21669399300,29cyclictest0-21swapper/723:20:017
21669399300,1cyclictest266332-21apt-get20:40:577
21668999300,2cyclictest273913-21cron20:55:006
21668999300,29cyclictest0-21swapper/623:40:006
21668399301,29cyclictest298429-21sh21:40:004
21668399301,29cyclictest298429-21sh21:40:004
216680993028,1cyclictest349616-21sh23:15:013
21667599300,2cyclictest244278-21sh20:00:002
21667599300,29cyclictest0-21swapper/222:05:552
21667199300,2cyclictest228130-21idleruntime-cro19:30:001
216667993026,3cyclictest366090-21awk23:45:000
21666799300,29cyclictest246986-21sh20:05:000
216693992928,1cyclictest0-21swapper/700:16:007
21669399290,28cyclictest388118-21apt-get00:25:587
21669399290,28cyclictest0-21swapper/720:45:567
21669399290,1cyclictest0-21swapper/719:35:567
216689992927,1cyclictest0-21swapper/621:00:586
216689992926,3cyclictest0-21swapper/619:45:576
21668999290,28cyclictest301621-21apt-get21:45:566
21668999290,28cyclictest0-21swapper/600:00:566
216687992927,1cyclictest344724-21apt-get23:05:565
21668799290,29cyclictest0-21swapper/521:00:005
21668799290,28cyclictest0-21swapper/523:00:555
21668799290,28cyclictest0-21swapper/500:30:575
21668799290,28cyclictest0-21swapper/500:20:565
216683992927,1cyclictest284705-21turbostat.cron21:15:004
216683992927,0cyclictest0-21swapper/423:55:574
21668099290,28cyclictest306996-21apt-get21:55:573
21668099290,28cyclictest0-21swapper/320:40:563
21668099290,28cyclictest0-21swapper/320:35:583
216675992927,1cyclictest69150irq/124-eno123:20:012
216675992927,1cyclictest269018-21apt-get20:45:562
216675992926,2cyclictest265830-21sh20:40:002
21667599290,28cyclictest0-21swapper/221:30:552
21667599290,28cyclictest0-21swapper/221:30:552
21667599290,28cyclictest0-21swapper/221:25:582
216671992929,0cyclictest0-21swapper/119:30:571
21667199290,28cyclictest279795-21apt-get21:05:571
21667199290,28cyclictest0-21swapper/122:20:571
216667992927,1cyclictest0-21swapper/000:30:570
21666799290,28cyclictest0-21swapper/023:30:570
21666799290,28cyclictest0-21swapper/000:16:000
216693992826,1cyclictest336636-21apt-get22:50:567
216693992826,1cyclictest0-21swapper/700:06:007
21669399280,27cyclictest0-21swapper/723:25:577
21669399280,27cyclictest0-21swapper/700:40:007
216689992826,1cyclictest0-21swapper/621:35:006
216689992826,1cyclictest0-21swapper/621:35:006
21668999280,27cyclictest0-21swapper/623:40:576
216687992826,1cyclictest0-21swapper/521:30:005
21668799280,27cyclictest0-21swapper/520:05:575
21668399280,27cyclictest0-21swapper/422:30:564
21668399280,0cyclictest0-21swapper/420:00:574
216680992828,0cyclictest0-21swapper/300:20:553
216680992827,1cyclictest0-21swapper/319:15:003
216680992827,1cyclictest0-21swapper/300:00:003
216675992826,1cyclictest0-21swapper/219:30:002
216675992825,2cyclictest306516-21latency_hist21:55:002
21667599280,0cyclictest0-21swapper/223:40:572
21667199280,0cyclictest0-21swapper/120:40:001
21667199280,0cyclictest0-21swapper/120:10:581
216667992825,2cyclictest374197-21munin-run00:00:000
216693992727,0cyclictest0-21swapper/723:05:567
216693992727,0cyclictest0-21swapper/722:55:577
216693992727,0cyclictest0-21swapper/720:50:577
216689992727,0cyclictest0-21swapper/623:00:016
216689992727,0cyclictest0-21swapper/619:50:576
216689992726,1cyclictest0-21swapper/620:20:576
216689992724,2cyclictest234009-21apt-get19:40:596
216687992727,0cyclictest0-21swapper/522:25:595
216687992727,0cyclictest0-21swapper/521:10:565
216687992726,1cyclictest361140-21apt-get23:35:555
216687992725,1cyclictest230817-21sh19:35:005
216683992727,0cyclictest0-21swapper/400:25:584
216683992725,1cyclictest390804-21apt-get00:30:574
216680992727,0cyclictest0-21swapper/322:45:553
216680992725,1cyclictest0-21swapper/323:50:003
216675992727,0cyclictest0-21swapper/220:40:572
216675992727,0cyclictest0-21swapper/200:25:582
216671992726,1cyclictest0-21swapper/122:10:571
216671992726,1cyclictest0-21swapper/120:55:551
216671992726,1cyclictest0-21swapper/119:35:551
216671992725,1cyclictest249687-21munin-run20:10:001
216667992727,0cyclictest0-21swapper/023:15:550
216667992726,1cyclictest0-21swapper/022:55:570
216667992726,1cyclictest0-21swapper/020:50:560
216693992626,0cyclictest0-21swapper/721:35:577
216693992626,0cyclictest0-21swapper/721:35:577
216693992626,0cyclictest0-21swapper/721:30:567
216693992626,0cyclictest0-21swapper/721:30:567
216693992622,3cyclictest330752-21sh22:40:007
216689992626,0cyclictest0-21swapper/600:06:006
216689992624,1cyclictest266332-21apt-get20:40:576
216687992626,0cyclictest0-21swapper/520:50:575
216683992626,0cyclictest0-21swapper/423:50:574
216683992625,1cyclictest247467-21apt-get20:05:574
216683992625,1cyclictest0-21swapper/420:35:574
216680992626,0cyclictest0-21swapper/322:35:573
216680992626,0cyclictest0-21swapper/321:30:563
216680992626,0cyclictest0-21swapper/321:30:563
216680992626,0cyclictest0-21swapper/321:15:563
216680992626,0cyclictest0-21swapper/320:10:583
216680992624,1cyclictest309709-21apt-get22:00:573
216680992624,1cyclictest288155-21apt-get21:20:573
216675992626,0cyclictest0-21swapper/223:25:572
216675992626,0cyclictest0-21swapper/219:20:572
216675992626,0cyclictest0-21swapper/200:40:002
216675992625,1cyclictest0-21swapper/220:50:572
216671992626,0cyclictest0-21swapper/100:30:571
216667992625,0cyclictest0-21swapper/023:10:010
216667992623,2cyclictest279312-21sh21:05:000
216693992524,1cyclictest0-21swapper/722:45:557
216693992522,2cyclictest319983-21turbostat.cron22:20:007
216693992522,2cyclictest311909-21sh22:05:007
216689992525,0cyclictest0-21swapper/600:25:586
216687992525,0cyclictest0-21swapper/500:11:005
216687992524,1cyclictest252843-21apt-get20:15:585
216683992525,0cyclictest0-21swapper/423:45:554
216683992525,0cyclictest0-21swapper/422:35:584
216683992525,0cyclictest0-21swapper/421:15:564
216683992525,0cyclictest0-21swapper/421:05:574
216680992525,0cyclictest0-21swapper/319:15:553
216680992524,1cyclictest0-21swapper/320:00:563
216675992525,0cyclictest0-21swapper/200:16:002
216675992524,1cyclictest0-21swapper/221:35:572
216675992524,1cyclictest0-21swapper/221:35:572
216671992525,0cyclictest0-21swapper/123:00:551
216671992525,0cyclictest0-21swapper/121:50:561
216671992525,0cyclictest0-21swapper/121:45:561
216671992525,0cyclictest0-21swapper/119:10:591
216671992525,0cyclictest0-21swapper/100:20:551
216671992524,1cyclictest0-21swapper/123:45:551
216667992524,1cyclictest0-21swapper/023:50:570
216693992424,0cyclictest0-21swapper/720:05:567
216693992423,1cyclictest0-21swapper/723:20:597
216693992422,1cyclictest0-21swapper/700:00:007
216689992424,0cyclictest0-21swapper/623:20:596
216689992424,0cyclictest0-21swapper/600:30:566
216687992424,0cyclictest0-21swapper/523:30:575
216683992424,0cyclictest0-21swapper/420:30:554
216683992424,0cyclictest0-21swapper/419:40:584
216683992424,0cyclictest0-21swapper/400:15:004
216680992424,0cyclictest0-21swapper/320:05:563
216680992424,0cyclictest0-21swapper/300:30:003
216680992422,1cyclictest303827-21sh21:50:003
216675992424,0cyclictest0-21swapper/223:05:562
216675992424,0cyclictest0-21swapper/222:55:572
216675992424,0cyclictest0-21swapper/200:06:002
216667992421,2cyclictest3820478-21systemd-journal19:55:000
216693992322,1cyclictest0-21swapper/719:15:007
216689992323,0cyclictest0-21swapper/620:20:006
216689992321,2cyclictest0-21swapper/621:05:576
216689992321,1cyclictest0-21swapper/623:15:556
216689992321,1cyclictest0-21swapper/620:25:576
216689992313,9cyclictest233508-21apt19:40:006
216687992323,0cyclictest0-21swapper/522:40:015
216687992323,0cyclictest0-21swapper/522:20:575
216687992323,0cyclictest0-21swapper/522:10:575
216687992323,0cyclictest0-21swapper/521:45:565
216687992322,1cyclictest0-21swapper/519:30:005
216687992321,1cyclictest366537-21apt-get23:45:565
216683992323,0cyclictest0-21swapper/421:20:574
216680992323,0cyclictest0-21swapper/321:35:573
216680992323,0cyclictest0-21swapper/321:35:573
216680992323,0cyclictest0-21swapper/321:15:013
216680992323,0cyclictest0-21swapper/320:35:003
216675992323,0cyclictest0-21swapper/222:50:562
216675992322,1cyclictest0-21swapper/222:45:552
216671992322,1cyclictest0-21swapper/100:25:581
216667992323,0cyclictest0-21swapper/023:35:550
216667992323,0cyclictest0-21swapper/022:05:550
216667992323,0cyclictest0-21swapper/021:55:570
216667992323,0cyclictest0-21swapper/021:15:560
216667992323,0cyclictest0-21swapper/020:30:560
216693992222,0cyclictest0-21swapper/719:30:577
216693992220,1cyclictest290845-21apt-get21:25:587
216689992222,0cyclictest0-21swapper/621:40:556
216689992220,1cyclictest325859-21apt-get22:30:566
21668999220,17cyclictest313414-21cat22:05:596
216687992221,1cyclictest279795-21apt-get21:05:575
216683992222,0cyclictest0-21swapper/423:25:574
216683992222,0cyclictest0-21swapper/419:50:574
216683992221,1cyclictest217154-21apt-get19:10:584
216683992220,1cyclictest252373-21munin-run20:15:004
216675992220,1cyclictest3820478-21systemd-journal22:05:002
216675992220,1cyclictest319985-21dump-pmu-power22:20:002
216675992220,1cyclictest1-21systemd22:40:002
216675992220,1cyclictest0-21swapper/219:30:572
216675992220,1cyclictest0-21swapper/219:15:002
216671992221,1cyclictest0-21swapper/123:30:571
216667992222,0cyclictest0-21swapper/022:50:560
216693992121,0cyclictest0-21swapper/722:05:557
216693992119,1cyclictest0-21swapper/721:25:007
216689992117,3cyclictest360693-21idleruntime-cro23:35:006
216687992121,0cyclictest0-21swapper/500:35:575
216683992121,0cyclictest0-21swapper/420:25:564
216683992121,0cyclictest0-21swapper/419:25:584
216683992119,1cyclictest0-21swapper/422:25:594
216680992119,1cyclictest325376-21sh22:30:003
216675992121,0cyclictest0-21swapper/223:20:592
216675992121,0cyclictest0-21swapper/220:05:562
216671992121,0cyclictest0-21swapper/121:40:561
216671992120,1cyclictest0-21swapper/122:10:001
216667992121,0cyclictest0-21swapper/022:15:570
216667992121,0cyclictest0-21swapper/022:00:570
216667992121,0cyclictest0-21swapper/000:00:560
216667992115,5cyclictest276618-21awk21:00:000
216693992018,1cyclictest349631-21munin-run23:15:017
21669399200,18cyclictest244292-21munin-run20:00:007
216689992020,0cyclictest0-21swapper/623:10:576
216689992017,3cyclictest0-21swapper/619:10:586
216689992017,2cyclictest387647-21munin-run00:25:006
216683992019,1cyclictest0-21swapper/419:45:584
216675992020,0cyclictest0-21swapper/200:00:002
216671992020,0cyclictest0-21swapper/121:00:581
216693991919,0cyclictest0-21swapper/722:35:007
216693991919,0cyclictest0-21swapper/722:20:577
216693991918,1cyclictest0-21swapper/722:40:557
216693991918,1cyclictest0-21swapper/721:05:577
216693991918,0cyclictest0-21swapper/719:20:577
216693991917,1cyclictest284713-21munin-run21:15:007
216689991919,0cyclictest0-21swapper/619:55:576
216687991919,0cyclictest0-21swapper/520:10:585
21668799190,18cyclictest0-21swapper/520:05:005
216683991919,0cyclictest0-21swapper/400:35:564
216680991917,2cyclictest306528-21munin-run21:55:003
216680991916,2cyclictest360656-21cron23:35:003
216671991917,1cyclictest384927-21apt00:20:001
216667991918,1cyclictest0-21swapper/021:10:560
216667991917,1cyclictest380031-21apt-get00:11:000
216667991915,3cyclictest236210-21sh19:45:000
216689991817,1cyclictest342033-21apt-get23:00:556
216689991816,2cyclictest0-21swapper/623:25:576
216689991815,2cyclictest298432-21cron21:40:006
216689991815,2cyclictest298432-21cron21:40:006
21668999180,18cyclictest0-21swapper/619:16:036
216687991818,0cyclictest0-21swapper/520:40:005
216683991818,0cyclictest0-21swapper/423:05:564
216683991818,0cyclictest0-21swapper/421:25:574
216683991817,1cyclictest279349-21awk21:05:004
216683991816,2cyclictest255051-21sh20:20:004
216680991818,0cyclictest0-21swapper/322:05:553
216680991816,1cyclictest374653-21apt-get00:00:563
216675991817,1cyclictest290397-21awk21:25:002
216671991818,0cyclictest0-21swapper/122:50:571
216671991817,1cyclictest0-21swapper/122:15:571
216671991817,1cyclictest0-21swapper/120:20:571
216667991818,0cyclictest0-21swapper/021:50:560
216667991818,0cyclictest0-21swapper/019:15:550
216667991816,1cyclictest379547-21turbostat.cron00:10:000
216689991717,0cyclictest0-21swapper/623:45:556
216689991717,0cyclictest0-21swapper/622:25:596
216689991712,0cyclictest0-21swapper/623:05:566
21668999170,16cyclictest0-21swapper/621:11:016
216687991717,0cyclictest0-21swapper/522:50:575
216687991717,0cyclictest0-21swapper/520:30:555
216683991717,0cyclictest0-21swapper/423:15:554
216683991717,0cyclictest0-21swapper/422:45:554
216683991717,0cyclictest0-21swapper/422:05:554
216683991717,0cyclictest0-21swapper/421:45:564
216683991717,0cyclictest0-21swapper/419:15:554
216680991714,2cyclictest346198-21python323:06:043
216675991716,1cyclictest0-21swapper/222:20:572
216675991716,0cyclictest0-21swapper/222:35:002
216671991716,1cyclictest0-21swapper/123:25:011
216667991717,0cyclictest0-21swapper/000:35:560
216667991714,2cyclictest357974-21apt23:30:010
216693991614,1cyclictest0-21swapper/722:26:017
21669399160,15cyclictest0-21swapper/723:40:217
21669399160,15cyclictest0-21swapper/723:30:447
21669399160,15cyclictest0-21swapper/721:56:027
21669399160,15cyclictest0-21swapper/721:01:507
21669399160,15cyclictest0-21swapper/700:32:597
21669399160,15cyclictest0-21swapper/700:20:567
216689991614,2cyclictest316396-21ps22:11:036
216689991614,1cyclictest320482-21apt-get22:20:576
216689991614,1cyclictest304308-21apt-get21:50:566
21668999160,16cyclictest0-21swapper/620:11:356
21668999160,15cyclictest334979-21memory22:45:596
21668999160,14cyclictest0-21swapper/622:35:416
216687991616,0cyclictest0-21swapper/519:35:555
216687991615,1cyclictest0-21swapper/519:10:595
21668799160,15cyclictest0-21swapper/523:15:155
216683991614,1cyclictest0-21swapper/421:55:584
21668399160,15cyclictest340169-21if_eno122:56:004
21668399160,15cyclictest0-21swapper/420:45:004
216680991616,0cyclictest0-21swapper/323:00:553
216680991615,1cyclictest282000-21sh21:10:003
216680991615,1cyclictest238902-21latency_hist19:50:003
21668099160,16cyclictest0-21swapper/319:31:053
21668099160,15cyclictest291062-21grep21:25:583
21668099160,15cyclictest0-21swapper/319:40:213
216675991616,0cyclictest0-21swapper/222:40:552
216675991615,1cyclictest284736-21awk21:15:002
21667599160,16cyclictest392696-21unixbench_singl00:31:062
21667599160,15cyclictest323798-21diskstats22:26:002
21667599160,15cyclictest0-21swapper/223:30:592
216671991616,0cyclictest0-21swapper/122:55:571
216671991615,1cyclictest0-21swapper/122:45:011
216671991614,1cyclictest0-21swapper/119:45:521
21667199160,15cyclictest0-21swapper/123:11:061
21667199160,15cyclictest0-21swapper/122:48:531
21667199160,15cyclictest0-21swapper/121:38:541
21667199160,15cyclictest0-21swapper/121:38:541
21667199160,15cyclictest0-21swapper/120:31:011
21667199160,15cyclictest0-21swapper/120:02:071
216667991614,1cyclictest0-21swapper/020:41:010
216667991613,2cyclictest316944-21unixbench_singl22:11:070
21666799160,15cyclictest227397-21python319:26:060
21666799160,15cyclictest0-21swapper/022:46:040
216693991515,0cyclictest0-21swapper/720:15:247
216693991514,1cyclictest0-21swapper/721:46:057
21669399150,15cyclictest0-21swapper/720:21:067
21669399150,15cyclictest0-21swapper/700:14:297
21669399150,14cyclictest0-21swapper/723:45:267
21669399150,14cyclictest0-21swapper/723:35:007
21669399150,14cyclictest0-21swapper/723:00:077
21669399150,14cyclictest0-21swapper/720:36:447
21669399150,14cyclictest0-21swapper/720:25:337
21669399150,14cyclictest0-21swapper/719:45:327
21669399150,14cyclictest0-21swapper/719:16:327
216689991514,1cyclictest0-21swapper/620:00:576
216689991512,0cyclictest0-21swapper/620:35:586
21668999150,1cyclictest262839-21users20:31:056
21668999150,15cyclictest0-21swapper/619:32:126
21668999150,14cyclictest0-21swapper/623:55:126
21668999150,14cyclictest0-21swapper/622:01:336
21668999150,14cyclictest0-21swapper/621:56:006
21668999150,14cyclictest0-21swapper/621:20:546
21668999150,14cyclictest0-21swapper/621:16:056
21668999150,14cyclictest0-21swapper/620:58:076
21668999150,14cyclictest0-21swapper/600:35:316
216687991514,1cyclictest0-21swapper/521:50:565
216687991513,2cyclictest353944-21if_err_eno123:21:035
21668799150,1cyclictest243970-21grep19:56:065
21668799150,1cyclictest0-21swapper/500:25:585
21668799150,15cyclictest223199-21/usr/sbin/munin19:21:065
21668799150,15cyclictest0-21swapper/519:15:105
21668799150,14cyclictest0-21swapper/500:15:055
216683991513,1cyclictest0-21swapper/400:06:034
21668399150,1cyclictest0-21swapper/423:21:024
21668399150,15cyclictest275290-21irqstats20:55:594
21668399150,15cyclictest224010-21sed19:21:004
21668399150,14cyclictest0-21swapper/423:30:384
21668399150,14cyclictest0-21swapper/422:40:134
21668399150,14cyclictest0-21swapper/421:30:104
21668399150,14cyclictest0-21swapper/421:30:104
216680991515,0cyclictest0-21swapper/300:32:213
216680991514,1cyclictest0-21swapper/320:30:003
21668099150,15cyclictest0-21swapper/323:18:263
21668099150,15cyclictest0-21swapper/320:16:453
21668099150,14cyclictest0-21swapper/322:15:023
21668099150,14cyclictest0-21swapper/319:35:313
21668099150,14cyclictest0-21swapper/300:10:123
21668099150,14cyclictest0-21swapper/300:05:043
216675991515,0cyclictest0-21swapper/223:15:012
216675991515,0cyclictest0-21swapper/221:05:572
216675991514,1cyclictest385641-21tr00:20:562
21667599150,15cyclictest0-21swapper/220:56:362
21667599150,15cyclictest0-21swapper/220:31:032
21667599150,15cyclictest0-21swapper/220:21:032
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional