You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-13 - 21:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Wed May 13, 2026 12:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
915599260,25cyclictest6778-21/usr/sbin/munin11:07:121
915899250,24cyclictest4043-21ssh11:34:043
915899250,24cyclictest2488-21diskmemload11:44:523
9156992521,0cyclictest0-21swapper/210:07:322
915699250,24cyclictest2488-21diskmemload10:25:512
915699250,21cyclictest0-21swapper/210:22:462
915699250,1cyclictest0-21swapper/212:01:312
915699250,1cyclictest0-21swapper/209:23:232
9155992523,1cyclictest11174-21ssh12:08:371
9155992523,1cyclictest0-21swapper/111:03:081
915599250,25cyclictest0-21swapper/112:14:051
915599250,24cyclictest0-21swapper/111:14:331
9154992522,2cyclictest21245-21hddtemp_smartct10:52:040
915499250,1cyclictest0-21swapper/011:50:010
915899240,24cyclictest0-21swapper/310:43:413
915899240,23cyclictest6869-21ssh10:38:513
915899240,23cyclictest6476-21rm09:12:403
915899240,23cyclictest4974-21ssh09:40:333
915899240,23cyclictest2488-21diskmemload11:39:153
915899240,23cyclictest2488-21diskmemload11:25:523
915899240,23cyclictest2488-21diskmemload09:57:003
915899240,23cyclictest2488-21diskmemload09:33:193
915899240,23cyclictest16101-21ntpq10:47:053
9156992422,1cyclictest21119-21ssh11:20:522
9156992420,3cyclictest0-21swapper/209:54:212
9156992420,1cyclictest8298-21apt-get12:07:032
915699240,23cyclictest0-21swapper/210:34:552
915699240,23cyclictest0-21swapper/209:41:592
9155992421,2cyclictest4568-21sh09:40:031
915599240,2cyclictest20928-21sh09:54:421
915599240,24cyclictest0-21swapper/111:54:011
915599240,24cyclictest0-21swapper/110:25:181
915599240,23cyclictest0-21swapper/111:57:291
915599240,23cyclictest0-21swapper/110:17:231
915599240,23cyclictest0-21swapper/109:18:521
915599240,1cyclictest0-21swapper/109:58:421
915599240,0cyclictest0-21swapper/110:21:221
915599240,0cyclictest0-21swapper/109:26:361
9154992423,1cyclictest30138-21rm10:59:520
9154992423,1cyclictest0-21swapper/010:16:460
9154992421,2cyclictest0-21swapper/011:31:380
915499240,24cyclictest0-21swapper/011:29:260
915499240,24cyclictest0-21swapper/011:17:380
915499240,23cyclictest7943-21ssh09:14:410
915499240,1cyclictest2488-21diskmemload11:11:500
915499240,1cyclictest2488-21diskmemload09:17:480
9158992323,0cyclictest23287-21ssh11:51:253
9158992323,0cyclictest0-21swapper/310:05:553
9158992323,0cyclictest0-21swapper/309:35:283
9158992321,1cyclictest0-21swapper/312:21:013
915899231,22cyclictest34-21ksoftirqd/312:25:583
915899230,2cyclictest30279-21turbostat.cron11:00:013
915899230,23cyclictest0-21swapper/312:11:583
915899230,23cyclictest0-21swapper/310:31:043
915899230,23cyclictest0-21swapper/309:21:483
915899230,22cyclictest32443-21df_inode11:02:033
915899230,22cyclictest30118-21sendmail09:00:003
915899230,22cyclictest28858-21latency_hist07:50:013
915899230,22cyclictest25626-21rm11:24:373
915899230,22cyclictest2488-21diskmemload09:28:413
915899230,22cyclictest20196-21sh10:51:263
915899230,22cyclictest11957-21diskmemload10:15:293
915899230,0cyclictest2488-21diskmemload11:16:133
915899230,0cyclictest0-21swapper/312:36:123
915899230,0cyclictest0-21swapper/312:07:033
9156992323,0cyclictest2488-21diskmemload11:33:232
9156992323,0cyclictest0-21swapper/212:33:332
9156992323,0cyclictest0-21swapper/210:41:082
9156992323,0cyclictest0-21swapper/210:02:032
9156992323,0cyclictest0-21swapper/209:57:572
9156992323,0cyclictest0-21swapper/208:12:012
9156992320,2cyclictest14743-21latency_hist11:15:002
9156992320,2cyclictest13213-21/usr/sbin/munin11:42:072
915699230,2cyclictest25886-21sh10:56:252
915699230,23cyclictest0-21swapper/212:20:452
915699230,23cyclictest0-21swapper/211:54:522
915699230,23cyclictest0-21swapper/209:10:252
915699230,22cyclictest20763-21ssh09:26:012
915699230,22cyclictest12975-21sh12:10:182
915699230,22cyclictest0-21swapper/211:26:572
915699230,22cyclictest0-21swapper/209:32:002
915699230,1cyclictest32743-21ssh11:59:112
915699230,0cyclictest0-21swapper/212:15:252
915699230,0cyclictest0-21swapper/211:45:072
915699230,0cyclictest0-21swapper/208:17:022
9155992323,0cyclictest2488-21diskmemload10:45:471
9155992323,0cyclictest0-21swapper/112:22:331
9155992323,0cyclictest0-21swapper/112:03:051
9155992322,1cyclictest2488-21diskmemload09:38:101
9155992321,1cyclictest2488-21diskmemload11:24:011
9155992321,1cyclictest0-21swapper/111:38:371
9155992321,1cyclictest0-21swapper/111:27:021
9155992321,1cyclictest0-21swapper/110:30:021
9155992321,1cyclictest0-21swapper/109:20:021
9155992320,2cyclictest6651-21sh10:38:351
915599230,2cyclictest3152-21diskmemload12:30:201
915599230,2cyclictest28567-21diskmemload10:01:571
915599230,2cyclictest26022-21sh09:30:231
915599230,2cyclictest2511-21dump-pmu-power09:10:001
915599230,23cyclictest0-21swapper/111:46:261
915599230,23cyclictest0-21swapper/110:10:511
915599230,23cyclictest0-21swapper/109:12:581
915599230,22cyclictest0-21swapper/110:52:411
915599230,0cyclictest0-21swapper/110:57:111
915599230,0cyclictest0-21swapper/110:42:311
9154992323,0cyclictest62550irq/127-eno110:06:020
9154992323,0cyclictest2488-21diskmemload12:09:080
9154992323,0cyclictest0-21swapper/012:15:290
9154992323,0cyclictest0-21swapper/011:03:010
9154992323,0cyclictest0-21swapper/010:26:270
9154992323,0cyclictest0-21swapper/007:52:010
9154992321,1cyclictest8349-21ssh10:40:310
915499230,2cyclictest30880-21ssh10:32:010
915499230,23cyclictest2488-21diskmemload10:22:470
915499230,23cyclictest0-21swapper/010:35:390
915499230,23cyclictest0-21swapper/010:10:350
915499230,22cyclictest9328-21ssh09:44:080
915499230,22cyclictest6246-21rm11:36:490
915499230,22cyclictest24011-21tr11:52:020
915499230,22cyclictest18476-21apt-get08:37:020
915499230,22cyclictest18326-21ssh09:22:560
915499230,22cyclictest11679-21ssh11:41:260
915499230,0cyclictest2488-21diskmemload12:11:320
9158992222,0cyclictest0-21swapper/311:06:593
9158992221,1cyclictest34-21ksoftirqd/312:16:023
9158992220,1cyclictest14610-21perf11:15:003
915899221,20cyclictest2488-21diskmemload12:31:193
915899220,22cyclictest2488-21diskmemload11:46:153
915899220,21cyclictest28642-21apt-get10:02:013
915899220,21cyclictest1462-21ssh12:00:263
915899220,21cyclictest0-21swapper/310:22:133
915899220,21cyclictest0-21swapper/309:50:203
915899220,21cyclictest0-21swapper/307:41:593
915899220,1cyclictest0-21swapper/311:55:403
915899220,0cyclictest0-21swapper/309:16:173
9156992222,0cyclictest0-21swapper/209:15:292
9156992220,2cyclictest2488-21diskmemload09:46:572
9156992220,1cyclictest28881-21basename07:50:012
915699220,22cyclictest0-21swapper/211:01:382
915699220,22cyclictest0-21swapper/209:35:382
915699220,21cyclictest11803-21ssh10:15:182
915699220,21cyclictest0-21swapper/212:35:112
915699220,21cyclictest0-21swapper/211:35:242
915699220,21cyclictest0-21swapper/208:47:012
915699220,1cyclictest0-21swapper/212:26:072
915699220,1cyclictest0-21swapper/210:52:172
915699220,0cyclictest0-21swapper/210:36:492
9155992222,0cyclictest0-21swapper/111:40:161
9155992220,1cyclictest0-21swapper/112:35:201
915599220,22cyclictest0-21swapper/112:17:091
915599220,21cyclictest3661-21latency_hist08:05:001
915599220,21cyclictest2488-21diskmemload11:30:031
915599220,21cyclictest0-21swapper/108:52:011
915599220,21cyclictest0-21swapper/108:22:011
915599220,21cyclictest0-21swapper/108:17:021
915599220,21cyclictest0-21swapper/107:57:021
915599220,21cyclictest0-21swapper/107:35:011
915599220,1cyclictest32532-21perf09:05:001
915599220,1cyclictest0-21swapper/112:25:431
915599220,1cyclictest0-21swapper/111:15:361
915599220,0cyclictest0-21swapper/109:45:311
9154992222,0cyclictest2488-21diskmemload09:32:000
9154992222,0cyclictest0-21swapper/010:47:240
9154992221,1cyclictest62550irq/127-eno110:02:230
9154992221,1cyclictest15857-21ssh09:50:090
9154992220,1cyclictest1258-21sh12:00:110
9154992220,1cyclictest0-21swapper/009:57:020
915499221,21cyclictest62550irq/127-eno112:25:100
915499220,2cyclictest10123-21sh09:45:020
915499220,21cyclictest32680-21rm09:36:500
915499220,21cyclictest2488-21diskmemload12:20:080
915499220,1cyclictest8612-21sh11:09:300
915499220,1cyclictest3199-21ssh12:30:230
915499220,1cyclictest2488-21diskmemload09:25:590
915499220,1cyclictest21493-21sh11:21:200
915499220,0cyclictest4863-21kworker/u8:112:35:170
915499220,0cyclictest2488-21diskmemload11:56:370
9158992121,0cyclictest0-21swapper/310:26:013
9158992120,1cyclictest9510-21apt-get07:12:013
915899210,21cyclictest0-21swapper/309:46:263
915899210,1cyclictest24065-21awk07:40:013
915899210,0cyclictest8864-21apt-get08:17:023
9156992121,0cyclictest0-21swapper/210:10:182
9156992119,1cyclictest16081-21apt-get08:32:002
915699210,21cyclictest0-21swapper/211:15:312
915699210,20cyclictest14724-21apt-get07:22:012
915599210,20cyclictest0-21swapper/108:57:011
9154992121,0cyclictest0-21swapper/009:01:590
9154992119,1cyclictest1217-21sendmail08:00:000
9154992118,2cyclictest21482-21sh07:35:010
915499210,2cyclictest313-21systemd-journal08:05:000
9158992020,0cyclictest20891-21apt-get08:42:013
9158992020,0cyclictest13680-21apt-get08:27:013
9158992020,0cyclictest0-21swapper/309:01:593
9158992019,1cyclictest31671-21apt-get07:57:023
915899200,20cyclictest25324-21perf08:50:003
9156992020,0cyclictest0-21swapper/208:42:012
9156992020,0cyclictest0-21swapper/208:22:012
9156992018,1cyclictest1671-21apt-get08:02:002
915699200,20cyclictest0-21swapper/207:57:022
915699200,19cyclictest21481-21awk07:35:012
915699200,0cyclictest0-21swapper/207:52:012
915599200,0cyclictest0-21swapper/107:41:591
915499200,19cyclictest13277-21grep08:25:010
9158991919,0cyclictest0-21swapper/307:17:023
9158991918,1cyclictest11287-21apt-get08:22:013
9158991917,1cyclictest31234-21cron07:55:013
9156991919,0cyclictest0-21swapper/208:57:012
9156991919,0cyclictest0-21swapper/208:37:012
9156991919,0cyclictest0-21swapper/207:12:012
9156991918,1cyclictest30540-21apt-get09:01:592
9156991918,0cyclictest0-21swapper/209:07:012
9156991917,1cyclictest27732-21munin-run08:55:002
9156991917,1cyclictest24048-21grep07:40:012
9155991919,0cyclictest0-21swapper/108:37:011
9155991919,0cyclictest0-21swapper/108:32:001
9155991917,1cyclictest16720-21munin-run07:25:001
9154991919,0cyclictest0-21swapper/008:57:010
9154991919,0cyclictest0-21swapper/008:31:590
9154991919,0cyclictest0-21swapper/007:17:010
9154991917,1cyclictest14724-21apt-get07:22:010
9154991917,1cyclictest11500-21latency_hist07:15:000
9158991817,0cyclictest0-21swapper/308:52:013
9155991818,0cyclictest0-21swapper/108:07:021
9154991818,0cyclictest17119-21apt-get07:27:020
9154991816,1cyclictest0-21swapper/008:55:000
9158991717,0cyclictest0-21swapper/308:37:023
9155991717,0cyclictest0-21swapper/107:27:021
9155991716,1cyclictest13680-21apt-get08:27:011
9155991716,1cyclictest0-21swapper/107:15:021
9154991715,1cyclictest26485-21munin-run07:45:010
9158991616,0cyclictest0-21swapper/307:32:023
9155991616,0cyclictest0-21swapper/107:37:011
9154991616,0cyclictest0-21swapper/008:12:010
9154991614,1cyclictest24382-21ntp_states08:47:070
9155991515,0cyclictest0-21swapper/107:12:001
9155991513,1cyclictest28858-21latency_hist07:50:011
915599150,9cyclictest0-21swapper/108:45:001
915899140,14cyclictest0-21swapper/309:05:053
915899140,14cyclictest0-21swapper/308:30:313
915899140,14cyclictest0-21swapper/308:10:173
915899140,14cyclictest0-21swapper/308:06:543
915899140,14cyclictest0-21swapper/307:26:473
915899140,0cyclictest0-21swapper/308:00:203
915899140,0cyclictest0-21swapper/307:21:193
9156991414,0cyclictest0-21swapper/207:26:172
9156991414,0cyclictest0-21swapper/207:15:022
915699140,14cyclictest0-21swapper/208:25:052
915699140,14cyclictest0-21swapper/208:05:332
915699140,0cyclictest0-21swapper/207:40:252
915599140,14cyclictest0-21swapper/108:10:221
915599140,14cyclictest0-21swapper/107:50:211
9154991414,0cyclictest0-21swapper/009:05:210
9154991414,0cyclictest0-21swapper/008:05:370
9154991414,0cyclictest0-21swapper/007:45:180
9154991414,0cyclictest0-21swapper/007:35:200
915499140,14cyclictest0-21swapper/008:40:210
915499140,14cyclictest0-21swapper/008:25:180
915499140,14cyclictest0-21swapper/008:15:030
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional