You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-26 - 22:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Sun Jul 26, 2026 12:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25286992624,1cyclictest21950irq/142-nvme0q312:30:012
2527799263,22cyclictest783-21snmpd11:10:340
2529099251,3cyclictest22397-21apt11:10:003
2529099251,0cyclictest0-21swapper/309:18:593
2529099250,24cyclictest18650-21diskmemload12:19:433
25286992523,1cyclictest0-21swapper/212:03:332
25286992523,1cyclictest0-21swapper/211:52:582
25286992523,1cyclictest0-21swapper/211:20:342
25286992523,1cyclictest0-21swapper/209:13:522
25286992522,2cyclictest622-21gdbus07:20:002
2528699250,24cyclictest0-21swapper/212:11:012
2528699250,24cyclictest0-21swapper/211:28:062
2528699250,24cyclictest0-21swapper/210:08:092
2528199250,24cyclictest17294-21rm11:05:211
2528199250,24cyclictest0-21swapper/111:33:511
2528199250,24cyclictest0-21swapper/111:33:511
2528199250,0cyclictest0-21swapper/110:19:001
25277992523,1cyclictest606-21dbus-daemon08:35:010
25290992423,1cyclictest18207-21ssh11:34:183
25290992423,1cyclictest18207-21ssh11:34:183
25290992423,1cyclictest0-21swapper/309:56:183
25290992422,1cyclictest21045-21idleruntime-cro10:40:003
25290992422,1cyclictest0-21swapper/311:28:093
2529099240,1cyclictest0-21swapper/310:05:263
2529099240,1cyclictest0-21swapper/309:34:563
2528699241,22cyclictest313-21systemd-journal11:10:022
2528699240,23cyclictest18650-21diskmemload10:13:542
2528699240,23cyclictest0-21swapper/209:49:332
2528699240,0cyclictest0-21swapper/210:46:052
2528699240,0cyclictest0-21swapper/210:46:052
25281992421,2cyclictest18650-21diskmemload12:22:221
2528199240,24cyclictest0-21swapper/112:15:031
2528199240,23cyclictest0-21swapper/111:26:041
2528199240,23cyclictest0-21swapper/111:01:581
2528199240,0cyclictest18650-21diskmemload11:59:201
25277992423,0cyclictest29533-21apt-get08:28:090
2527799240,24cyclictest18650-21diskmemload09:33:430
2527799240,24cyclictest0-21swapper/010:08:500
2527799240,23cyclictest18650-21diskmemload12:37:580
2527799240,23cyclictest18650-21diskmemload11:05:480
2527799240,23cyclictest0-21swapper/012:07:170
2527799240,1cyclictest17568-21rm11:33:190
2527799240,1cyclictest17568-21rm11:33:190
25290992323,0cyclictest18650-21diskmemload10:55:193
25290992323,0cyclictest0-21swapper/312:28:273
25290992323,0cyclictest0-21swapper/310:44:073
25290992323,0cyclictest0-21swapper/310:03:143
25290992323,0cyclictest0-21swapper/309:48:093
25290992322,1cyclictest18650-21diskmemload11:20:093
25290992322,1cyclictest18650-21diskmemload10:30:123
25290992322,0cyclictest18650-21diskmemload09:28:333
25290992321,1cyclictest9821-21turbostat07:45:013
25290992321,1cyclictest24944-21ssh11:12:463
25290992321,1cyclictest0-21swapper/312:22:093
2529099230,23cyclictest0-21swapper/310:27:013
2529099230,22cyclictest976-21sh10:50:593
2529099230,22cyclictest9080-21perf08:50:003
2529099230,22cyclictest31922-21ssh10:49:173
2529099230,22cyclictest31922-21ssh10:49:173
2529099230,22cyclictest30605-21sh09:20:383
2529099230,22cyclictest0-21swapper/312:30:153
2529099230,22cyclictest0-21swapper/309:40:563
2529099230,22cyclictest0-21swapper/308:38:093
2529099230,1cyclictest19563-21munin-run09:40:003
2529099230,0cyclictest0-21swapper/309:12:193
25286992323,0cyclictest18650-21diskmemload10:41:252
25286992323,0cyclictest0-21swapper/211:06:422
25286992323,0cyclictest0-21swapper/210:25:442
25286992323,0cyclictest0-21swapper/210:17:512
25286992321,1cyclictest30381-21ssh11:45:322
25286992321,1cyclictest0-21swapper/212:20:212
25286992321,1cyclictest0-21swapper/211:59:252
25286992321,1cyclictest0-21swapper/210:53:132
25286992321,1cyclictest0-21swapper/210:23:112
25286992321,1cyclictest0-21swapper/209:53:132
25286992320,2cyclictest30931-21ssh09:21:122
25286992318,1cyclictest14162-21ssh10:04:182
2528699230,23cyclictest0-21swapper/212:05:412
2528699230,23cyclictest0-21swapper/209:58:492
2528699230,22cyclictest13527-21ssh09:34:232
25281992323,0cyclictest18650-21diskmemload11:17:331
25281992323,0cyclictest18650-21diskmemload10:43:061
25281992323,0cyclictest0-21swapper/110:39:351
25281992323,0cyclictest0-21swapper/109:35:361
25281992322,1cyclictest3330-21ssh09:54:371
25281992322,1cyclictest18650-21diskmemload09:20:321
25281992321,2cyclictest2805-21ssh11:20:291
25281992321,1cyclictest5406-21rm10:25:591
25281992321,1cyclictest31416-21latency_hist08:30:011
25281992321,1cyclictest0-21swapper/110:31:561
2528199230,2cyclictest4290-21sh09:55:341
2528199230,23cyclictest0-21swapper/112:29:581
2528199230,22cyclictest31932-21diskmemload10:20:301
2528199230,22cyclictest21834-21cut08:10:001
2528199230,22cyclictest20311-21ssh12:05:441
2528199230,22cyclictest12674-21grep09:33:201
2528199230,22cyclictest10059-21ssh10:00:591
2528199230,22cyclictest0-21swapper/110:49:361
2528199230,22cyclictest0-21swapper/110:49:361
2528199230,0cyclictest0-21swapper/112:39:591
2528199230,0cyclictest0-21swapper/112:01:081
2528199230,0cyclictest0-21swapper/111:49:271
2528199230,0cyclictest0-21swapper/110:55:281
2528199230,0cyclictest0-21swapper/110:52:441
2528199230,0cyclictest0-21swapper/110:12:181
25277992323,0cyclictest18650-21diskmemload09:56:260
25277992322,1cyclictest18650-21diskmemload09:10:050
25277992322,1cyclictest0-21swapper/011:21:470
25277992322,1cyclictest0-21swapper/009:17:050
25277992321,1cyclictest0-21swapper/011:58:010
25277992320,3cyclictest0-21swapper/009:20:030
25277992320,2cyclictest14611-21ssh09:35:300
25277992320,2cyclictest0-21swapper/011:02:420
25277992320,2cyclictest0-21swapper/010:11:570
2527799230,2cyclictest3934-21sh09:26:050
2527799230,2cyclictest29529-21ssh11:15:500
2527799230,23cyclictest0-21swapper/011:42:460
2527799230,23cyclictest0-21swapper/010:44:380
2527799230,23cyclictest0-21swapper/010:22:070
2527799230,23cyclictest0-21swapper/009:48:260
2527799230,22cyclictest952-21diskmemload10:50:570
2527799230,22cyclictest9477-21ssh10:00:010
2527799230,22cyclictest62550irq/127-eno110:59:110
2527799230,22cyclictest0-21swapper/012:25:180
2527799230,22cyclictest0-21swapper/010:47:120
2527799230,22cyclictest0-21swapper/010:47:120
2527799230,22cyclictest0-21swapper/009:41:240
2527799230,1cyclictest17915-21ssh10:38:100
2527799230,0cyclictest0-21swapper/011:49:130
2527799230,0cyclictest0-21swapper/010:28:440
2527799230,0cyclictest0-21swapper/009:52:370
25290992222,0cyclictest0-21swapper/311:48:563
25290992222,0cyclictest0-21swapper/311:16:013
25290992220,1cyclictest5509-21apt-get07:38:083
25290992220,1cyclictest22191-21sh12:37:123
25290992219,2cyclictest30867-21ssh10:19:453
2529099221,1cyclictest9832-1kworker/u9:111:37:133
2529099220,21cyclictest3124-21ssh11:50:043
2529099220,21cyclictest22115-21sh10:12:383
2529099220,21cyclictest1416-21munin-run08:35:013
2529099220,21cyclictest0-21swapper/311:55:203
2529099220,21cyclictest0-21swapper/309:51:243
2529099220,1cyclictest32729-21sh10:21:303
2529099220,1cyclictest25937-21ssh12:10:523
2529099220,1cyclictest12147-21ssh11:00:593
2529099220,1cyclictest0-21swapper/312:00:013
2529099220,1cyclictest0-21swapper/308:00:003
2529099220,0cyclictest0-21swapper/311:40:253
25286992222,0cyclictest0-21swapper/212:35:492
25286992222,0cyclictest0-21swapper/209:17:142
25286992220,1cyclictest17380-21ssh10:37:202
25286992220,1cyclictest11849-21ssh11:00:372
2528699220,2cyclictest19808-21rm09:40:032
2528699220,21cyclictest32485-21munin-run07:25:002
2528699220,21cyclictest15224-21ssh12:30:112
2528699220,21cyclictest14257-21ssh09:35:042
2528699220,21cyclictest0-21swapper/211:15:492
2528699220,21cyclictest0-21swapper/210:55:422
2528699220,21cyclictest0-21swapper/209:25:432
2528699220,21cyclictest0-21swapper/208:40:012
2528699220,1cyclictest18650-21diskmemload11:37:182
2528699220,1cyclictest0-21swapper/211:41:192
2528699220,0cyclictest14241-21ssh11:31:112
2528699220,0cyclictest14241-21ssh11:31:112
2528699220,0cyclictest0-21swapper/212:17:012
2528699220,0cyclictest0-21swapper/210:30:362
25281992222,0cyclictest0-21swapper/109:26:101
25281992220,1cyclictest24777-21ssh11:40:191
25281992220,1cyclictest24214-21latency_hist08:15:001
25281992220,1cyclictest23431-21ssh11:10:401
2528199220,2cyclictest18900-21rm09:10:011
2528199220,2cyclictest15880-21ssh10:06:211
2528199220,22cyclictest0-21swapper/112:30:121
2528199220,22cyclictest0-21swapper/111:52:421
2528199220,21cyclictest26633-21sh09:46:521
2528199220,21cyclictest19591-21ssh11:35:571
2528199220,21cyclictest1915-21apt-get08:38:091
25277992221,1cyclictest0-21swapper/010:18:460
2527799220,21cyclictest22317-21apt-get08:13:090
2527799220,21cyclictest19120-21diskmemload11:35:070
2527799220,21cyclictest18650-21diskmemload11:50:120
2527799220,21cyclictest0-21swapper/012:15:030
2527799220,21cyclictest0-21swapper/012:10:260
2527799220,21cyclictest0-21swapper/012:00:120
2527799220,21cyclictest0-21swapper/011:25:030
2527799220,1cyclictest6674-21perf08:45:000
2527799220,1cyclictest0-21swapper/009:05:000
25290992121,0cyclictest0-21swapper/312:05:303
25290992119,1cyclictest0-21swapper/307:55:013
25286992120,0cyclictest7173-21apt-get08:48:072
2528699210,0cyclictest0-21swapper/208:08:102
25281992119,1cyclictest7442-21basename07:40:011
25281992118,2cyclictest13895-21basename09:00:001
2528199210,21cyclictest0-21swapper/109:40:251
25277992119,1cyclictest14611-21dump-pmu-power07:55:010
2527799210,21cyclictest0-21swapper/012:30:390
2527799210,0cyclictest0-21swapper/009:10:000
2527799210,0cyclictest0-21swapper/007:43:090
2527799210,0cyclictest0-21swapper/007:33:060
25290992018,1cyclictest29045-21awk08:25:013
25290992018,1cyclictest0-21swapper/308:40:023
2529099200,20cyclictest0-21swapper/307:33:063
25286992020,0cyclictest0-21swapper/208:40:022
25286992020,0cyclictest0-21swapper/208:23:082
25286992020,0cyclictest0-21swapper/207:38:082
2528699200,19cyclictest0-21swapper/208:58:122
25281992020,0cyclictest17526-21apt-get08:03:091
25281992020,0cyclictest0-21swapper/107:48:071
25281992020,0cyclictest0-21swapper/107:28:091
25281992020,0cyclictest0-21swapper/107:25:001
25277992020,0cyclictest0-21swapper/007:28:090
25277992018,1cyclictest3856-21perf08:40:010
25277992018,1cyclictest313-21systemd-journal07:40:010
25290991919,0cyclictest0-21swapper/307:23:093
25290991919,0cyclictest0-21swapper/307:18:083
25290991918,0cyclictest0-21swapper/307:48:073
2529099190,0cyclictest25764-21apt-get07:13:093
2529099190,0cyclictest0-21swapper/309:03:093
25286991919,0cyclictest10316-21apt-get07:48:072
25286991919,0cyclictest0-21swapper/209:08:072
25286991919,0cyclictest0-21swapper/208:18:092
25286991919,0cyclictest0-21swapper/207:58:102
25286991917,1cyclictest16266-21latency_hist09:05:002
25281991919,0cyclictest0-21swapper/109:08:071
25281991919,0cyclictest0-21swapper/108:33:111
25281991919,0cyclictest0-21swapper/107:58:101
25281991919,0cyclictest0-21swapper/107:43:091
2528199190,0cyclictest0-21swapper/108:23:081
25277991919,0cyclictest0-21swapper/008:03:100
25277991918,0cyclictest27139-21apt-get08:23:080
25277991818,0cyclictest0-21swapper/008:58:120
25277991816,1cyclictest0-21swapper/008:50:000
25286991715,1cyclictest31935-21apt-get08:33:112
25277991717,0cyclictest0-21swapper/008:08:100
25277991715,1cyclictest12273-21grep07:50:010
25290991616,0cyclictest0-21swapper/308:28:093
25290991615,1cyclictest16772-21apt-get09:08:073
25290991614,1cyclictest0-21swapper/307:28:093
25290991613,2cyclictest11455-21cron08:55:003
25286991616,0cyclictest0-21swapper/207:43:092
25286991614,1cyclictest0-21swapper/208:25:412
25281991616,0cyclictest0-21swapper/109:03:091
25281991614,1cyclictest7977-21irqstats08:48:111
25277991616,0cyclictest0-21swapper/008:18:080
25277991614,1cyclictest28160-21apt-get07:18:080
25277991614,1cyclictest16829-21users07:58:200
25290991513,1cyclictest26787-21munin-run08:20:013
2529099150,14cyclictest0-21swapper/308:57:193
25286991515,0cyclictest12715-21apt-get07:53:092
25286991515,0cyclictest0-21swapper/208:03:102
25286991515,0cyclictest0-21swapper/207:33:062
25281991515,0cyclictest0-21swapper/107:18:081
25281991513,1cyclictest14623-21munin-run07:55:011
25277991513,1cyclictest0-21swapper/008:53:140
2527799150,1cyclictest0-21swapper/007:13:100
2529099140,14cyclictest0-21swapper/308:06:243
2529099140,0cyclictest0-21swapper/308:10:483
2529099140,0cyclictest0-21swapper/308:00:303
2528699140,14cyclictest0-21swapper/208:10:292
2528699140,14cyclictest0-21swapper/207:25:412
2528699140,14cyclictest0-21swapper/207:13:172
25281991414,0cyclictest0-21swapper/108:52:011
2528199140,14cyclictest0-21swapper/108:15:241
2528199140,14cyclictest0-21swapper/107:10:421
2528199140,0cyclictest0-21swapper/108:40:191
2527799140,14cyclictest0-21swapper/007:20:350
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional