You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-06 - 21:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot5s.osadl.org (updated Thu Nov 06, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1022999408,0cyclictest0-21swapper/307:35:483
1022799250,24cyclictest23110-21turbostat12:00:002
10229992422,1cyclictest1-21systemd08:55:003
1022799241,22cyclictest28182-21apt-get07:45:032
1021899240,23cyclictest18528-21idleruntime-cro09:40:000
1021899240,23cyclictest17223-21turbostat.cron08:30:000
1022999230,22cyclictest313-21systemd-journal11:50:003
1022999230,22cyclictest11051-21basename11:35:003
10227992323,0cyclictest0-21swapper/207:55:442
10227992321,1cyclictest10899-21cron10:30:002
10222992321,1cyclictest31031-21grep11:10:001
10222992320,2cyclictest13426-21turbostat.cron10:34:591
1021899230,22cyclictest8866-21basename09:20:010
10229992221,0cyclictest4349-21apt-get09:10:463
10229992220,1cyclictest3764-21apt-get11:20:483
10229992220,1cyclictest10932-21munin-run10:30:003
10229992219,2cyclictest17450-21perf07:25:003
1022999220,21cyclictest26118-21perf09:54:593
1022999220,21cyclictest25539-21munin-run12:05:003
1022999220,21cyclictest24144-21apt-get10:55:473
1022999220,21cyclictest0-21swapper/312:30:463
1022999220,21cyclictest0-21swapper/311:25:483
1022999220,21cyclictest0-21swapper/308:10:003
1022999220,21cyclictest0-21swapper/307:30:473
1022999220,1cyclictest593-21dbus-daemon11:05:013
1022999220,1cyclictest4033-21munin-run09:09:593
10227992220,1cyclictest27912-21cron12:10:002
1022799220,2cyclictest5816-21cron11:25:002
1022799220,21cyclictest6440-21apt-get10:20:462
1022799220,21cyclictest613-21apt-get12:20:472
1022799220,21cyclictest25483-21apt-get07:40:482
1022799220,21cyclictest25157-21perf07:39:592
1022799220,21cyclictest23325-21munin-run09:49:592
1022799220,21cyclictest17427-21idleruntime-cro07:25:002
1022799220,21cyclictest0-21swapper/212:15:472
1022799220,21cyclictest0-21swapper/207:15:452
1022799220,1cyclictest31626-21sh08:59:592
10222992220,1cyclictest26233-21sendmail-msp11:00:011
10222992220,1cyclictest0-21swapper/111:25:471
1022299220,21cyclictest6744-21apt-get09:15:471
1022299220,21cyclictest31960-21apt-get09:00:471
1022299220,21cyclictest0-21swapper/108:20:441
10218992222,0cyclictest0-21swapper/007:30:450
10218992221,1cyclictest1-21systemd12:00:000
10218992220,1cyclictest15832-21apt-get10:40:000
10218992219,2cyclictest593-21dbus-daemon10:45:000
1021899220,21cyclictest20725-21perf11:55:010
1021899220,21cyclictest19934-21apt-get08:35:480
1021899220,21cyclictest19633-21idleruntime-cro08:35:010
1021899220,21cyclictest13526-21munin-run11:40:000
1021899220,21cyclictest0-21swapper/010:55:130
1021899220,1cyclictest5134-21munin-run12:30:010
10229992119,1cyclictest24431-21perf08:45:003
10229992119,1cyclictest20910-21idleruntime-cro09:45:003
10229992119,1cyclictest0-21swapper/310:05:003
10229992119,1cyclictest0-21swapper/310:04:593
10229992119,1cyclictest0-21swapper/308:30:003
1022999210,21cyclictest0-21swapper/310:35:503
1022999210,20cyclictest0-21swapper/307:55:433
1022999210,20cyclictest0-21swapper/307:50:003
1022999210,1cyclictest0-21swapper/309:35:493
10227992121,0cyclictest16421-21apt-get09:35:492
10227992121,0cyclictest0-21swapper/211:05:482
10227992119,1cyclictest18650-21awk10:44:592
10227992119,1cyclictest13431-21ls10:34:592
10227992119,1cyclictest11018-21cron11:34:592
10227992119,1cyclictest0-21swapper/208:45:442
10227992119,1cyclictest0-21swapper/208:05:002
1022799210,21cyclictest0-21swapper/210:15:022
1022799210,20cyclictest26423-21apt-get09:55:482
1022799210,20cyclictest0-21swapper/211:45:482
1022799210,20cyclictest0-21swapper/210:55:482
1022799210,20cyclictest0-21swapper/209:25:462
1022799210,1cyclictest3311-21perf10:15:012
10222992121,0cyclictest25844-21apt-get12:05:501
10222992121,0cyclictest0-21swapper/111:45:481
10222992120,1cyclictest19628-21grep08:35:001
10222992119,1cyclictest28498-21cron10:00:001
10222992119,1cyclictest23317-21perf09:50:001
1022299210,21cyclictest0-21swapper/111:30:441
1022299210,20cyclictest27143-21apt-get08:50:461
1022299210,20cyclictest0-21swapper/112:40:011
1022299210,20cyclictest0-21swapper/111:55:481
1022299210,20cyclictest0-21swapper/108:00:001
1022299210,0cyclictest0-21swapper/110:15:471
10218992121,0cyclictest0-21swapper/007:35:480
10218992120,1cyclictest0-21swapper/011:00:450
10218992119,1cyclictest906-21perf10:10:000
10218992119,1cyclictest0-21swapper/009:50:000
10218992119,1cyclictest0-21swapper/009:35:000
1021899210,21cyclictest0-21swapper/011:30:440
1021899210,0cyclictest5473-21apt-get08:05:460
10229992020,0cyclictest0-21swapper/312:20:483
10229992020,0cyclictest0-21swapper/312:10:473
10229992020,0cyclictest0-21swapper/309:30:483
10229992020,0cyclictest0-21swapper/308:10:463
10229992019,1cyclictest0-21swapper/310:10:483
10229992019,0cyclictest1329-21apt-get11:15:473
10229992018,1cyclictest5124-21perf12:30:013
10229992018,1cyclictest26840-21munin-run08:50:003
1022999200,0cyclictest0-21swapper/312:05:503
1022999200,0cyclictest0-21swapper/308:35:483
10227992020,0cyclictest0-21swapper/212:25:462
10227992020,0cyclictest0-21swapper/211:10:492
10227992020,0cyclictest0-21swapper/208:10:452
10227992019,0cyclictest0-21swapper/211:35:472
10227992017,2cyclictest22226-21sh07:35:002
1022799200,20cyclictest0-21swapper/211:15:472
1022799200,0cyclictest7835-21apt-get12:35:472
1022799200,0cyclictest0-21swapper/212:10:472
1022799200,0cyclictest0-21swapper/209:10:472
10222992020,0cyclictest0-21swapper/112:30:471
10222992020,0cyclictest0-21swapper/112:24:591
10222992020,0cyclictest0-21swapper/109:40:461
10222992018,1cyclictest23036-21apt-get07:35:471
10222992017,2cyclictest30336-21cron07:50:011
10222992017,2cyclictest1009-21apt11:15:001
1022299200,1cyclictest13747-21apt-get10:35:501
1022299200,19cyclictest0-21swapper/111:35:481
1022299200,0cyclictest0-21swapper/108:25:431
10218992020,0cyclictest0-21swapper/008:40:480
10218992020,0cyclictest0-21swapper/008:10:450
10218992020,0cyclictest0-21swapper/008:00:460
10218992019,1cyclictest0-21swapper/009:10:470
10218992019,0cyclictest0-21swapper/007:50:480
1021899200,0cyclictest0-21swapper/012:05:490
1021899200,0cyclictest0-21swapper/011:25:480
1021899200,0cyclictest0-21swapper/007:45:470
10229991919,0cyclictest31359-21apt-get11:10:493
10229991919,0cyclictest31238-21apt-get10:05:473
10229991919,0cyclictest0-21swapper/311:35:483
10229991919,0cyclictest0-21swapper/309:45:473
10229991919,0cyclictest0-21swapper/307:50:473
10229991918,1cyclictest12687-21apt-get08:20:453
10229991917,1cyclictest19632-21munin-plugin-st08:35:013
1022999190,18cyclictest0-21swapper/307:40:493
10227991919,0cyclictest0-21swapper/210:35:492
10227991919,0cyclictest0-21swapper/209:30:482
10227991919,0cyclictest0-21swapper/208:35:482
10227991919,0cyclictest0-21swapper/208:30:502
10227991918,1cyclictest6161-21apt-get11:25:482
10227991918,1cyclictest13836-21apt-get11:40:472
10227991918,1cyclictest0-21swapper/209:50:102
1022799190,18cyclictest0-21swapper/210:00:472
1022799190,18cyclictest0-21swapper/210:00:472
10222991919,0cyclictest14022-21apt-get09:30:471
10222991919,0cyclictest0-21swapper/111:50:501
10222991919,0cyclictest0-21swapper/109:10:471
10222991919,0cyclictest0-21swapper/109:05:481
10222991919,0cyclictest0-21swapper/107:40:491
10222991919,0cyclictest0-21swapper/107:30:451
10222991918,1cyclictest0-21swapper/111:20:471
1022299190,18cyclictest3075-21apt-get08:00:461
10218991919,0cyclictest24742-21apt-get08:45:450
10218991919,0cyclictest0-21swapper/009:40:470
10218991919,0cyclictest0-21swapper/009:25:450
10218991919,0cyclictest0-21swapper/007:40:490
10218991919,0cyclictest0-21swapper/007:10:470
10218991918,1cyclictest9163-21apt-get09:20:440
10218991918,1cyclictest21352-21apt-get10:50:470
10218991917,1cyclictest16235-21apt-get11:45:480
1021899190,0cyclictest23444-21apt-get12:00:460
10229991818,0cyclictest11568-21apt-get09:25:463
10229991818,0cyclictest0-21swapper/310:20:463
10229991818,0cyclictest0-21swapper/308:55:463
10229991818,0cyclictest0-21swapper/308:20:003
10229991816,1cyclictest8860-21munin-run09:20:003
10229991816,1cyclictest6113-21turbostat10:20:013
1022999180,14cyclictest0-21swapper/311:09:053
1022999180,0cyclictest0-21swapper/311:40:483
10227991818,0cyclictest0-21swapper/212:00:472
10227991818,0cyclictest0-21swapper/209:15:472
10227991818,0cyclictest0-21swapper/209:05:482
10227991818,0cyclictest0-21swapper/208:20:452
10227991817,0cyclictest5438-21apt-get12:30:462
10227991817,0cyclictest26561-21apt-get11:00:452
10227991817,0cyclictest10500-21apt-get07:10:472
10227991816,1cyclictest0-21swapper/207:30:002
10222991818,0cyclictest10268-21apt-get08:15:451
10222991818,0cyclictest0-21swapper/112:30:011
10222991818,0cyclictest0-21swapper/112:15:461
10222991818,0cyclictest0-21swapper/110:45:471
10222991816,1cyclictest24406-21cron08:44:591
10222991816,1cyclictest19845-21perf07:29:591
10218991818,0cyclictest0-21swapper/011:20:470
10218991818,0cyclictest0-21swapper/008:55:470
10218991817,0cyclictest28841-21apt10:00:470
10218991817,0cyclictest28841-21apt10:00:470
10218991816,1cyclictest313-21systemd-journal10:15:000
1021899180,1cyclictest31012-21cron11:09:590
10229991716,0cyclictest0-21swapper/310:40:483
10229991715,1cyclictest593-21dbus-daemon11:59:593
10229991715,1cyclictest28527-21wc09:59:593
10229991715,1cyclictest21029-21apt10:49:593
1022999170,16cyclictest32767-21latency_hist12:20:003
10227991717,0cyclictest0-21swapper/210:45:472
10227991717,0cyclictest0-21swapper/208:40:472
10227991716,0cyclictest9163-21apt-get09:20:452
10222991717,0cyclictest12891-21apt-get07:15:451
10222991717,0cyclictest0-21swapper/109:25:461
10222991717,0cyclictest0-21swapper/108:45:441
10222991715,1cyclictest22-21ksoftirqd/111:05:011
10222991714,2cyclictest30303-21latency_hist12:15:001
1022299170,0cyclictest0-21swapper/109:20:451
10218991717,0cyclictest6440-21apt-get10:20:460
10218991717,0cyclictest0-21swapper/011:10:480
10218991717,0cyclictest0-21swapper/009:00:470
10218991717,0cyclictest0-21swapper/008:50:460
10218991717,0cyclictest0-21swapper/007:15:440
1021899170,15cyclictest0-21swapper/012:30:570
10229991616,0cyclictest0-21swapper/311:55:003
1022999160,1cyclictest0-21swapper/309:05:003
1022999160,15cyclictest0-21swapper/310:30:473
10227991616,0cyclictest0-21swapper/208:25:442
10227991615,0cyclictest0-21swapper/207:50:482
1022799160,16cyclictest18632-21apt-get11:50:502
10222991616,0cyclictest0-21swapper/110:50:471
10222991616,0cyclictest0-21swapper/108:55:461
10222991614,1cyclictest7864-21apt-get08:10:461
1022299160,15cyclictest24025-21dpkg09:50:471
10218991616,0cyclictest0-21swapper/012:35:470
1021899160,15cyclictest1550-21memory07:55:480
10229991515,0cyclictest0-21swapper/312:35:463
10229991515,0cyclictest0-21swapper/310:50:463
10229991512,2cyclictest583-21avahi-daemon07:20:003
1022999150,0cyclictest0-21swapper/309:20:453
10227991513,1cyclictest23444-21apt-get10:55:012
10222991515,0cyclictest28842-21apt-get10:00:471
10222991515,0cyclictest28842-21apt-get10:00:471
10222991515,0cyclictest0-21swapper/110:40:481
10222991515,0cyclictest0-21swapper/107:20:451
10222991513,1cyclictest10985-21munin-run10:29:591
10222991513,1cyclictest0-21swapper/111:41:201
10222991513,1cyclictest0-21swapper/111:15:471
1022299150,1cyclictest0-21swapper/112:00:481
1022299150,1cyclictest0-21swapper/110:10:571
1022299150,1cyclictest0-21swapper/110:05:501
10218991515,0cyclictest0-21swapper/009:55:470
10218991515,0cyclictest0-21swapper/009:50:470
10218991514,1cyclictest8840-21apt-get10:25:500
10218991513,1cyclictest15341-21acpi07:20:430
1021899150,3cyclictest2716-21sh12:25:000
1021899150,1cyclictest18808-21ntp_states07:25:510
1021899150,15cyclictest0-21swapper/010:16:350
10229991414,0cyclictest0-21swapper/308:00:463
10229991414,0cyclictest0-21swapper/307:10:463
10229991412,1cyclictest18186-21diskstats07:25:473
10227991414,0cyclictest0-21swapper/210:08:152
10227991414,0cyclictest0-21swapper/209:44:592
10227991414,0cyclictest0-21swapper/209:00:302
10227991414,0cyclictest0-21swapper/208:52:552
10227991414,0cyclictest0-21swapper/208:05:472
1022799140,14cyclictest0-21swapper/208:17:212
10222991414,0cyclictest0-21swapper/110:20:461
10222991414,0cyclictest0-21swapper/108:35:061
10218991414,0cyclictest0-21swapper/011:40:130
10218991414,0cyclictest0-21swapper/009:06:080
10218991413,1cyclictest32574-21seq12:18:340
1021899140,7cyclictest0-21swapper/010:45:470
1021899140,14cyclictest0-21swapper/012:14:060
1021899140,14cyclictest0-21swapper/011:15:350
1021899140,14cyclictest0-21swapper/008:20:140
1021899140,0cyclictest0-21swapper/010:31:380
10222991312,0cyclictest30672-21apt-get07:50:481
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional