You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 00:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot5s.osadl.org (updated Thu Nov 27, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21604992521,2cyclictest313-21systemd-journal10:40:010
2160699240,23cyclictest313-21systemd-journal09:45:002
2160599241,22cyclictest29199-21apt07:25:011
2160599240,23cyclictest3052-21latency_hist09:50:001
2160599240,23cyclictest18170-21turbostat.cron09:15:001
2160599240,23cyclictest0-21swapper/110:27:361
2160599240,23cyclictest0-21swapper/109:27:361
2160599240,0cyclictest0-21swapper/109:52:381
21604992422,1cyclictest9274-21idleruntime-cro07:49:590
21604992422,1cyclictest8726-21apt-get10:02:390
21611992321,1cyclictest0-21swapper/310:20:023
2160699230,22cyclictest13071-21perf10:09:592
21605992321,1cyclictest0-21swapper/108:37:361
21604992321,1cyclictest0-21swapper/007:20:000
21604992320,2cyclictest3046-21idleruntime-cro09:50:010
2160499230,22cyclictest27669-21apt-get11:47:370
21611992220,1cyclictest10980-21grep09:00:013
2161199220,21cyclictest29213-21munin-run07:25:013
2161199220,21cyclictest24822-21munin-run11:40:003
2161199220,21cyclictest0-21swapper/310:07:343
2161199220,1cyclictest25924-21grep10:35:013
2161199220,1cyclictest21058-21apt09:22:383
21606992220,1cyclictest15913-21apt-get10:17:362
21606992220,1cyclictest13378-21latency_hist09:05:012
21606992219,2cyclictest25466-21idleruntime-cro09:30:002
2160699220,21cyclictest4100-21apt-get07:40:012
2160699220,21cyclictest0-21swapper/211:17:392
2160699220,1cyclictest5987-21apt-get11:02:352
21605992220,1cyclictest31605-21munin-run07:30:011
21605992220,1cyclictest20679-21munin-run10:25:001
21605992220,1cyclictest2010-21awk12:00:001
2160599220,21cyclictest0-21swapper/110:37:381
2160599220,21cyclictest0-21swapper/108:27:361
2160599220,1cyclictest6185-21munin-run08:49:591
2160599220,1cyclictest6185-21munin-run08:49:591
2160599220,1cyclictest593-21dbus-daemon10:00:001
2160599220,1cyclictest15150-21munin-run11:20:001
21604992222,0cyclictest0-21swapper/012:22:360
21604992221,0cyclictest18646-21apt-get09:17:400
21604992221,0cyclictest0-21swapper/008:32:330
2160499220,21cyclictest6162-21cron08:50:000
2160499220,21cyclictest6162-21cron08:50:000
2160499220,21cyclictest4118-21munin-run07:40:000
2160499220,21cyclictest22410-21munin-plugin-st11:34:590
2160499220,21cyclictest20060-21apt-get12:37:370
2160499220,21cyclictest0-21swapper/010:52:370
2160499220,21cyclictest0-21swapper/009:07:340
2160499220,21cyclictest0-21swapper/008:17:360
2160499220,1cyclictest9025-21apt-get08:57:350
2160499220,1cyclictest32661-21turbostat.cron09:45:000
2160499220,0cyclictest27258-21apt-get07:22:360
2160499220,0cyclictest17661-21apt-get12:32:350
2161199210,20cyclictest0-21swapper/308:27:363
2161199210,20cyclictest0-21swapper/307:27:363
2161199210,0cyclictest24244-21apt-get08:22:393
2161199210,0cyclictest0-21swapper/309:17:403
2161199210,0cyclictest0-21swapper/307:12:373
21606992121,0cyclictest6626-21apt-get08:52:342
21606992121,0cyclictest0-21swapper/208:12:382
21606992120,0cyclictest0-21swapper/212:22:362
21606992120,0cyclictest0-21swapper/211:57:392
21606992119,1cyclictest9751-21apt-get07:52:362
21606992119,1cyclictest593-21dbus-daemon11:09:592
21606992119,1cyclictest25904-21cron10:35:012
21606992119,1cyclictest19943-21munin-run11:30:002
2160699210,20cyclictest0-21swapper/211:47:372
2160699210,0cyclictest20394-21apt-get11:32:382
2160699210,0cyclictest0-21swapper/212:02:362
2160699210,0cyclictest0-21swapper/210:02:382
21605992120,1cyclictest15596-21apt-get11:22:371
21605992120,0cyclictest7845-21apt-get10:00:011
21605992120,0cyclictest4869-21apt-get12:07:371
21605992120,0cyclictest0-21swapper/111:37:371
2160599210,21cyclictest0-21swapper/107:47:371
2160599210,0cyclictest0-21swapper/109:22:381
2160599210,0cyclictest0-21swapper/108:22:401
21604992120,1cyclictest13185-21apt-get11:17:390
21604992120,1cyclictest0-21swapper/012:12:370
21604992120,0cyclictest3513-21apt-get09:52:370
21604992119,1cyclictest1967-21cron12:00:000
21604992119,1cyclictest0-21swapper/010:57:350
21604992118,2cyclictest13-21ksoftirqd/008:55:010
2160499210,0cyclictest0-21swapper/007:32:360
21611992020,0cyclictest10780-21apt-get11:12:373
21611992020,0cyclictest0-21swapper/312:32:353
21611992020,0cyclictest0-21swapper/312:22:363
21611992020,0cyclictest0-21swapper/312:12:363
21611992020,0cyclictest0-21swapper/311:22:353
21611992020,0cyclictest0-21swapper/309:27:363
21611992020,0cyclictest0-21swapper/308:47:363
21611992020,0cyclictest0-21swapper/308:47:363
21611992019,1cyclictest13515-21apt-get10:12:363
21611992018,1cyclictest696-21munin-run10:50:003
21611992018,1cyclictest32686-21munin-run09:45:003
21611992018,1cyclictest13376-21latency_hist09:05:013
2161199200,0cyclictest9668-21apt-get12:17:383
2161199200,0cyclictest28308-21apt-get09:37:343
2161199200,0cyclictest0-21swapper/311:17:383
2161199200,0cyclictest0-21swapper/310:02:393
2161199200,0cyclictest0-21swapper/309:57:373
2161199200,0cyclictest0-21swapper/307:47:373
21606992020,0cyclictest0-21swapper/210:52:362
21606992020,0cyclictest0-21swapper/209:32:352
21606992020,0cyclictest0-21swapper/209:07:352
21606992020,0cyclictest0-21swapper/208:04:592
21606992020,0cyclictest0-21swapper/207:27:372
21606992019,0cyclictest0-21swapper/208:47:362
21606992019,0cyclictest0-21swapper/208:47:362
21606992018,1cyclictest21820-21apt-get08:17:362
21606992018,1cyclictest20569-21sh09:20:002
21606992018,1cyclictest18171-21cron09:15:002
21606992018,1cyclictest15472-21cut10:15:002
21606992017,2cyclictest23509-21dump-pmu-power10:30:002
2160699200,20cyclictest0-21swapper/211:37:372
2160699200,20cyclictest0-21swapper/208:42:362
2160699200,20cyclictest0-21swapper/207:37:392
2160699200,0cyclictest28791-21apt-get10:42:392
2160699200,0cyclictest0-21swapper/212:32:352
2160699200,0cyclictest0-21swapper/209:57:362
2160699200,0cyclictest0-21swapper/207:50:002
21605992020,0cyclictest0-21swapper/111:52:391
21605992020,0cyclictest0-21swapper/110:12:371
21605992020,0cyclictest0-21swapper/108:02:361
21605992020,0cyclictest0-21swapper/107:15:021
21605992019,0cyclictest30729-21apt-get09:42:361
21605992018,1cyclictest4111-21perf07:40:001
21605992018,1cyclictest16815-21perf12:30:001
21605992018,1cyclictest0-21swapper/112:30:011
2160599200,20cyclictest0-21swapper/111:07:371
2160599200,0cyclictest0-21swapper/111:42:361
2160599200,0cyclictest0-21swapper/111:12:371
2160599200,0cyclictest0-21swapper/107:12:361
21604992020,0cyclictest21059-21apt-get09:22:380
21604992020,0cyclictest0-21swapper/012:27:370
21604992020,0cyclictest0-21swapper/011:42:360
21604992020,0cyclictest0-21swapper/011:15:000
21604992020,0cyclictest0-21swapper/010:47:390
21604992019,0cyclictest8385-21apt-get11:07:370
2160499200,0cyclictest25910-21apt-get09:32:340
2160499200,0cyclictest23974-21apt-get10:32:370
21611991919,0cyclictest4727-21apt-get07:40:013
21611991919,0cyclictest21570-21apt-get10:27:363
21611991919,0cyclictest0-21swapper/311:32:373
21611991919,0cyclictest0-21swapper/311:27:373
21611991919,0cyclictest0-21swapper/311:02:363
21611991919,0cyclictest0-21swapper/310:37:393
21611991919,0cyclictest0-21swapper/307:52:363
21611991919,0cyclictest0-21swapper/307:15:023
21611991918,1cyclictest31594-21apt-get08:35:023
2161199190,18cyclictest16813-21dump-pmu-power12:30:003
21606991919,0cyclictest0-21swapper/212:37:372
21606991919,0cyclictest0-21swapper/212:17:382
21606991919,0cyclictest0-21swapper/212:07:372
21606991919,0cyclictest0-21swapper/209:47:362
21606991919,0cyclictest0-21swapper/207:32:362
21606991917,1cyclictest676-21cron10:50:002
21605991919,0cyclictest0-21swapper/112:37:361
21605991919,0cyclictest0-21swapper/110:17:361
21605991919,0cyclictest0-21swapper/108:42:361
21605991919,0cyclictest0-21swapper/108:12:381
21605991918,0cyclictest32051-21apt-get07:32:361
21605991918,0cyclictest0-21swapper/110:52:371
21605991917,1cyclictest30737-21munin-run10:44:591
2160599190,0cyclictest0-21swapper/111:45:171
21604991919,0cyclictest0-21swapper/012:02:370
21604991918,0cyclictest5909-21apt-get09:57:370
21604991917,1cyclictest26634-21apt-get08:27:360
2160499190,18cyclictest16964-21apt-get08:07:350
21611991818,0cyclictest16239-21apt-get09:12:353
21611991818,0cyclictest0-21swapper/308:07:353
21606991818,0cyclictest0-21swapper/211:52:392
21606991818,0cyclictest0-21swapper/209:37:342
21606991817,1cyclictest0-21swapper/208:35:012
21606991816,1cyclictest28327-21apt10:40:012
21605991818,0cyclictest0-21swapper/110:32:371
21605991818,0cyclictest0-21swapper/108:07:361
21605991817,1cyclictest12081-21apt-get12:22:361
21604991817,0cyclictest11117-21apt-get10:07:340
21604991816,1cyclictest22043-21apt-get07:12:360
21611991717,0cyclictest25270-21apt-get11:42:363
21611991717,0cyclictest2472-21apt-get12:02:373
21611991717,0cyclictest2046-21apt-get07:37:393
21611991715,1cyclictest23763-21perf08:19:593
21606991717,0cyclictest0-21swapper/208:57:342
21606991717,0cyclictest0-21swapper/208:22:392
21606991717,0cyclictest0-21swapper/207:17:352
21606991716,1cyclictest0-21swapper/209:22:382
21605991717,0cyclictest11445-21apt-get09:02:361
21605991717,0cyclictest0-21swapper/112:12:371
21605991717,0cyclictest0-21swapper/110:07:341
21605991717,0cyclictest0-21swapper/109:37:341
21605991717,0cyclictest0-21swapper/107:57:341
21604991717,0cyclictest0-21swapper/011:37:360
21604991717,0cyclictest0-21swapper/010:42:380
21604991717,0cyclictest0-21swapper/010:17:360
21604991717,0cyclictest0-21swapper/010:15:000
21604991717,0cyclictest0-21swapper/009:02:360
21604991717,0cyclictest0-21swapper/008:22:400
21604991717,0cyclictest0-21swapper/007:57:350
21604991717,0cyclictest0-21swapper/007:52:370
21604991715,1cyclictest0-21swapper/008:15:010
21604991714,2cyclictest3760-21idleruntime-cro08:45:000
21611991616,0cyclictest0-21swapper/311:07:373
21611991616,0cyclictest0-21swapper/310:42:393
21611991616,0cyclictest0-21swapper/310:17:363
21611991614,1cyclictest0-21swapper/308:05:003
2161199160,15cyclictest3071-21munin-run09:50:013
21606991616,0cyclictest16964-21apt-get08:07:342
21606991616,0cyclictest0-21swapper/211:22:362
21606991614,1cyclictest28948-21sed07:22:452
21605991616,0cyclictest0-21swapper/108:32:341
21605991616,0cyclictest0-21swapper/107:52:351
21604991616,0cyclictest0-21swapper/012:17:380
21604991616,0cyclictest0-21swapper/011:02:360
21604991616,0cyclictest0-21swapper/010:25:010
21604991615,1cyclictest0-21swapper/007:42:360
21604991614,1cyclictest17514-21cron11:25:010
21604991614,1cyclictest1338-21sendmail08:39:590
21604991614,1cyclictest0-21swapper/008:02:550
2160499160,15cyclictest30278-21grep07:27:400
2160499160,15cyclictest0-21swapper/009:37:390
21611991515,0cyclictest0-21swapper/311:47:373
21611991515,0cyclictest0-21swapper/309:52:383
21611991515,0cyclictest0-21swapper/308:52:353
21611991515,0cyclictest0-21swapper/308:42:363
21611991515,0cyclictest0-21swapper/308:12:383
21611991514,0cyclictest13838-21apt-get09:07:343
21606991514,1cyclictest0-21swapper/209:52:372
21606991513,1cyclictest26444-21head11:42:432
21606991513,1cyclictest0-21swapper/208:32:392
2160699150,1cyclictest0-21swapper/212:12:422
21605991515,0cyclictest0-21swapper/111:32:381
2160599150,1cyclictest6814-21grep08:52:351
2160599150,1cyclictest0-21swapper/111:27:421
2160599150,1cyclictest0-21swapper/108:17:411
2160599150,15cyclictest0-21swapper/107:43:501
21604991515,0cyclictest0-21swapper/011:27:360
21604991515,0cyclictest0-21swapper/010:22:370
21611991414,0cyclictest0-21swapper/309:31:233
21611991413,1cyclictest0-21swapper/312:07:363
2161199140,14cyclictest0-21swapper/312:35:443
2161199140,14cyclictest0-21swapper/311:50:253
2161199140,14cyclictest0-21swapper/310:52:293
2161199140,14cyclictest0-21swapper/308:30:283
2161199140,14cyclictest0-21swapper/307:55:193
2161199140,0cyclictest0-21swapper/310:55:493
21606991414,0cyclictest0-21swapper/207:55:462
2160699140,14cyclictest0-21swapper/212:25:102
2160699140,14cyclictest0-21swapper/211:10:032
2160699140,0cyclictest0-21swapper/210:56:102
2160699140,0cyclictest0-21swapper/210:20:152
2160699140,0cyclictest0-21swapper/208:25:072
21605991414,0cyclictest0-21swapper/111:00:121
21605991411,2cyclictest15774-21perf09:10:001
2160599140,14cyclictest0-21swapper/108:55:051
2160599140,0cyclictest0-21swapper/112:15:221
2160599140,0cyclictest0-21swapper/112:00:141
2160599140,0cyclictest0-21swapper/110:45:111
2160499140,7cyclictest0-21swapper/009:12:360
2160499140,14cyclictest0-21swapper/012:05:120
2160499140,14cyclictest0-21swapper/011:50:170
2160499140,14cyclictest0-21swapper/009:25:550
21611991313,0cyclictest0-21swapper/307:32:363
21606991312,0cyclictest22043-21apt-get07:12:372
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional