You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-21 - 15:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Tue Apr 21, 2026 12:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3214599280,2cyclictest26064-21munin-run09:40:013
3214599280,27cyclictest26723-21diskmemload09:14:263
3214599280,0cyclictest0-21swapper/309:05:113
32145992722,4cyclictest26723-21diskmemload10:51:283
32145992719,4cyclictest29728-21ssh10:45:413
3214599271,4cyclictest0-21swapper/308:45:103
3214599270,21cyclictest25212-21if_enp1s0.810:10:133
3214599270,0cyclictest0-21swapper/310:42:093
32132992726,1cyclictest62550irq/127-eno112:29:150
32145992620,5cyclictest26723-21diskmemload12:07:133
3214599260,25cyclictest26207-21ssh11:45:343
3214599260,25cyclictest0-21swapper/309:33:143
3214599260,1cyclictest0-21swapper/312:36:443
3214599260,1cyclictest0-21swapper/312:13:303
3214599260,0cyclictest0-21swapper/309:27:383
3213299260,25cyclictest13990-21ssh12:37:010
32145992522,3cyclictest0-21swapper/310:26:393
32145992521,3cyclictest0-21swapper/311:12:033
32145992518,6cyclictest9791-21ssh10:59:273
3214599250,24cyclictest0-21swapper/310:16:183
3214599250,21cyclictest0-21swapper/310:08:483
3214599250,1cyclictest27636-1kworker/u9:112:23:543
3214599250,1cyclictest26723-21diskmemload09:54:333
3214599250,1cyclictest0-21swapper/311:17:453
3214599250,0cyclictest0-21swapper/309:41:003
32140992523,1cyclictest0-21swapper/209:26:292
32140992520,4cyclictest0-21swapper/209:58:232
3214099250,24cyclictest26723-21diskmemload11:58:472
3213699250,1cyclictest6724-21ssh09:51:301
32145992422,2cyclictest0-21swapper/309:45:253
32145992422,1cyclictest9673-21cron07:30:013
32145992421,3cyclictest0-21swapper/311:20:483
32145992421,3cyclictest0-21swapper/310:33:473
32145992421,2cyclictest24320-21cron09:05:013
32145992420,3cyclictest0-21swapper/309:56:113
3214599240,23cyclictest26723-21diskmemload11:50:343
3214599240,20cyclictest0-21swapper/309:24:093
3214599240,1cyclictest0-21swapper/311:09:363
3214599240,0cyclictest0-21swapper/312:16:183
3214599240,0cyclictest0-21swapper/311:58:063
3214599240,0cyclictest0-21swapper/311:43:593
3214599240,0cyclictest0-21swapper/311:31:463
3214599240,0cyclictest0-21swapper/310:03:053
32140992422,1cyclictest27823-21sh12:19:512
3214099242,21cyclictest271rcuc/210:56:302
32140992421,2cyclictest5295-21ssh09:50:172
32140992420,4cyclictest0-21swapper/211:08:402
3214099240,23cyclictest0-21swapper/210:49:092
3214099240,23cyclictest0-21swapper/210:40:152
3214099240,23cyclictest0-21swapper/210:27:152
3214099240,23cyclictest0-21swapper/209:41:272
3214099240,1cyclictest26723-21diskmemload11:12:322
3214099240,0cyclictest0-21swapper/212:36:402
3214099240,0cyclictest0-21swapper/211:02:482
32136992423,1cyclictest26723-21diskmemload11:10:251
32136992423,1cyclictest22-21ksoftirqd/112:37:511
3213699241,2cyclictest22-21ksoftirqd/112:15:181
3213699240,23cyclictest8815-21ssh09:23:051
3213699240,23cyclictest606-21dbus-daemon08:05:011
3213699240,23cyclictest30344-21ssh09:12:341
3213699240,23cyclictest26723-21diskmemload11:05:461
3213699240,23cyclictest0-21swapper/112:12:411
3213699240,23cyclictest0-21swapper/109:59:511
3213699240,22cyclictest313-21systemd-journal10:40:011
3213699240,1cyclictest13408-21ssh11:02:131
3213699240,1cyclictest0-21swapper/111:20:171
3213299240,24cyclictest0-21swapper/011:31:430
3213299240,23cyclictest5205-21ssh09:20:020
3213299240,1cyclictest30011-21sh09:43:060
3213299240,1cyclictest1549-21sh10:19:330
3213299240,0cyclictest26723-21diskmemload09:33:220
3213299240,0cyclictest0-21swapper/011:03:460
32145992323,0cyclictest0-21swapper/310:35:213
32145992320,3cyclictest0-21swapper/312:33:103
32145992320,3cyclictest0-21swapper/311:26:523
3214599230,22cyclictest5094-21perf08:25:003
3214599230,0cyclictest0-21swapper/309:17:023
32140992323,0cyclictest0-21swapper/212:00:522
32140992323,0cyclictest0-21swapper/211:32:502
32140992323,0cyclictest0-21swapper/211:27:172
32140992322,1cyclictest18964-21ssh10:04:542
32140992322,1cyclictest17844-21ssh11:38:302
32140992321,1cyclictest26723-21diskmemload09:30:122
32140992321,1cyclictest12704-21apt-get07:35:022
32140992321,1cyclictest0-21swapper/209:45:112
32140992321,1cyclictest0-21swapper/209:10:122
3214099231,21cyclictest8129-21ssh12:30:432
3214099230,2cyclictest9986-21apt-get09:25:002
3214099230,2cyclictest5734-21sh10:23:032
3214099230,2cyclictest0-21swapper/207:10:112
3214099230,22cyclictest28477-21sh11:16:492
3214099230,22cyclictest2473-21sh09:16:382
3214099230,22cyclictest21203-21ssh11:40:382
3214099230,22cyclictest0-21swapper/210:39:022
3214099230,22cyclictest0-21swapper/210:11:392
3214099230,1cyclictest0-21swapper/212:05:212
3214099230,0cyclictest0-21swapper/212:10:022
3214099230,0cyclictest0-21swapper/211:23:192
32136992323,0cyclictest26723-21diskmemload11:42:391
32136992323,0cyclictest26723-21diskmemload10:01:581
32136992323,0cyclictest0-21swapper/112:21:521
32136992323,0cyclictest0-21swapper/112:06:171
32136992323,0cyclictest0-21swapper/111:37:281
32136992323,0cyclictest0-21swapper/110:07:141
32136992322,1cyclictest9560-21kworker/1:009:45:111
32136992322,0cyclictest24246-21/usr/sbin/munin11:45:151
32136992321,1cyclictest30447-21sh10:46:551
32136992320,2cyclictest1313-21sh10:19:131
3213699230,23cyclictest26723-21diskmemload11:17:071
3213699230,23cyclictest0-21swapper/109:40:251
3213699230,23cyclictest0-21swapper/109:25:581
3213699230,22cyclictest9438-21ssh12:00:301
3213699230,22cyclictest9153-21ssh12:32:231
3213699230,22cyclictest9037-21ssh10:58:191
3213699230,22cyclictest6079-21sh11:25:511
3213699230,22cyclictest3847-21sh10:20:201
3213699230,22cyclictest26889-21rm10:11:231
3213699230,22cyclictest19526-21sh08:55:011
3213699230,22cyclictest15050-21sh10:31:441
3213699230,22cyclictest0-21swapper/110:27:061
32132992321,1cyclictest2674-21sh10:51:080
32132992321,1cyclictest0-21swapper/011:51:210
32132992321,1cyclictest0-21swapper/011:15:590
3213299231,22cyclictest62550irq/127-eno110:02:340
3213299231,22cyclictest62550irq/127-eno109:18:010
3213299231,21cyclictest17944-21sh11:38:400
3213299230,23cyclictest26723-21diskmemload12:13:470
3213299230,23cyclictest26723-21diskmemload12:07:540
3213299230,23cyclictest26723-21diskmemload10:21:030
3213299230,23cyclictest0-21swapper/011:27:310
3213299230,23cyclictest0-21swapper/009:55:240
3213299230,23cyclictest0-21swapper/009:11:130
3213299230,22cyclictest30986-21ssh10:47:430
3213299230,22cyclictest0-21swapper/010:44:110
3213299230,1cyclictest13399-21sh09:26:400
3213299230,1cyclictest0-21swapper/012:30:210
3213299230,0cyclictest26723-21diskmemload09:35:590
3213299230,0cyclictest0-21swapper/011:57:500
3213299230,0cyclictest0-21swapper/010:13:170
32145992220,2cyclictest0-21swapper/310:20:463
32145992220,1cyclictest8085-21sh12:00:133
3214599220,2cyclictest13561-21needreboot08:40:143
3214599220,21cyclictest0-21swapper/311:39:573
3214599220,21cyclictest0-21swapper/311:00:543
3214599220,21cyclictest0-21swapper/307:40:113
3214599220,1cyclictest2289-21grep08:20:013
32140992222,0cyclictest0-21swapper/207:50:102
32140992221,1cyclictest0-21swapper/210:50:132
32140992220,1cyclictest0-21swapper/210:06:582
3214099220,21cyclictest30557-21grep10:15:162
3214099220,21cyclictest30345-21ssh12:20:352
3214099220,21cyclictest14702-21munin-run08:45:002
3214099220,21cyclictest0-21swapper/212:26:102
3214099220,21cyclictest0-21swapper/210:30:312
3214099220,21cyclictest0-21swapper/209:35:252
3214099220,1cyclictest9678-21apt07:30:012
3214099220,1cyclictest30128-21ssh11:50:202
32136992222,0cyclictest0-21swapper/109:38:241
32136992221,1cyclictest0-21swapper/112:25:261
32136992220,1cyclictest0-21swapper/109:15:181
3213699220,21cyclictest26723-21diskmemload10:42:131
3213699220,21cyclictest22552-21apt-get07:55:111
3213699220,21cyclictest17408-21munin-run07:45:001
3213699220,21cyclictest15887-21apt-get09:30:111
3213699220,21cyclictest0-21swapper/111:56:011
3213699220,21cyclictest0-21swapper/111:30:181
3213699220,1cyclictest0-21swapper/111:53:121
3213699220,0cyclictest26723-21diskmemload10:54:291
32132992222,0cyclictest0-21swapper/012:20:320
32132992222,0cyclictest0-21swapper/009:50:100
32132992221,1cyclictest20607-21ssh11:10:070
32132992220,1cyclictest9168-21rm10:25:220
32132992220,1cyclictest31915-21df09:45:120
32132992220,1cyclictest0-21swapper/011:20:160
32132992220,1cyclictest0-21swapper/008:00:000
3213299220,22cyclictest12405-21apt-get10:30:140
3213299220,22cyclictest0-21swapper/012:02:530
3213299220,21cyclictest5359-21apt-get08:25:120
3213299220,21cyclictest26723-21diskmemload11:45:220
3213299220,21cyclictest17752-21ssh11:05:540
3213299220,21cyclictest0-21swapper/010:55:050
3213299220,21cyclictest0-21swapper/010:35:090
3213299220,21cyclictest0-21swapper/007:50:100
3213299220,1cyclictest27440-21ssh12:19:170
3213299220,1cyclictest19427-21apt-get10:05:110
3214599210,21cyclictest0-21swapper/307:45:103
3214599210,20cyclictest2746-21apt-get07:15:103
32140992121,0cyclictest28-21ksoftirqd/208:01:412
32140992120,0cyclictest0-21swapper/208:55:112
3214099210,20cyclictest0-21swapper/208:25:122
3214099210,0cyclictest0-21swapper/211:45:222
32136992121,0cyclictest9955-21apt-get07:30:101
32136992120,0cyclictest32366-21apt-get07:10:111
32136992119,1cyclictest2306-21munin-run08:20:011
3213699210,1cyclictest0-21swapper/108:30:001
3213699210,0cyclictest12080-21apt-get07:35:021
32132992120,1cyclictest17355-21apt-get08:50:100
32132992119,1cyclictest622-21gdbus07:50:000
32132992119,1cyclictest0-21swapper/008:40:000
32132992119,1cyclictest0-21swapper/007:25:010
3213299210,21cyclictest0-21swapper/011:40:060
3213299210,20cyclictest0-21swapper/008:45:100
3213299210,20cyclictest0-21swapper/008:15:010
3213299210,1cyclictest25366-21apt-get08:00:100
3213299210,1cyclictest21905-21cron09:00:010
3213299210,1cyclictest12084-21awk07:35:010
3214599200,0cyclictest0-21swapper/308:55:113
32140992020,0cyclictest0-21swapper/207:30:102
3214099200,0cyclictest0-21swapper/209:05:112
3214099200,0cyclictest0-21swapper/208:35:112
32136992020,0cyclictest24615-21apt-get09:05:111
32136992020,0cyclictest0-21swapper/107:45:101
32136992019,0cyclictest2081-21apt-get07:15:021
32136992018,1cyclictest7278-21cron07:25:011
3213699200,20cyclictest0-21swapper/108:20:121
3213699200,0cyclictest0-21swapper/108:35:111
32132992020,0cyclictest0-21swapper/008:15:090
32132992020,0cyclictest0-21swapper/007:10:110
32145991919,0cyclictest0-21swapper/307:55:113
32145991919,0cyclictest0-21swapper/307:50:103
32145991919,0cyclictest0-21swapper/307:30:103
32145991919,0cyclictest0-21swapper/307:10:113
32145991917,1cyclictest9896-21awk08:35:003
32140991919,0cyclictest7754-21apt-get08:30:112
32140991919,0cyclictest0-21swapper/208:15:092
32140991917,1cyclictest4907-21idleruntime-cro07:20:012
32136991919,0cyclictest0-21swapper/108:30:111
32136991919,0cyclictest0-21swapper/108:10:111
32132991919,0cyclictest0-21swapper/009:05:110
32132991918,1cyclictest22219-21apt-get09:00:110
32140991817,0cyclictest0-21swapper/207:20:102
32140991815,2cyclictest31207-21ntp_states08:10:162
32136991818,0cyclictest12565-21apt-get08:40:091
32136991818,0cyclictest0-21swapper/108:55:111
32132991818,0cyclictest0-21swapper/008:05:100
32140991717,0cyclictest0-21swapper/208:20:022
32136991715,1cyclictest9702-21basename07:30:011
3213699170,16cyclictest22622-21cstates09:00:111
32132991717,0cyclictest0-21swapper/007:40:110
32145991616,0cyclictest0-21swapper/308:05:103
32145991615,0cyclictest0-21swapper/307:20:103
3214599160,0cyclictest0-21swapper/308:52:393
32140991616,0cyclictest0-21swapper/207:45:102
3214099160,13cyclictest783-21snmpd07:55:342
32132991616,0cyclictest0-21swapper/008:40:090
32132991616,0cyclictest0-21swapper/007:25:100
32132991614,1cyclictest5097-21latency_hist08:25:000
32145991515,0cyclictest0-21swapper/308:00:103
32145991514,1cyclictest0-21swapper/308:25:073
3214599150,15cyclictest0-21swapper/308:14:033
3214599150,14cyclictest0-21swapper/307:35:173
32140991515,0cyclictest0-21swapper/207:40:112
32140991514,0cyclictest22219-21apt-get09:00:102
3213299150,0cyclictest0-21swapper/008:34:390
32140991414,0cyclictest0-21swapper/208:45:102
32140991411,2cyclictest18472-21smartctl08:50:172
3214099140,0cyclictest0-21swapper/208:05:502
32136991414,0cyclictest14961-21apt-get08:45:101
3213699140,14cyclictest0-21swapper/107:50:061
3213699140,0cyclictest0-21swapper/108:05:071
32132991414,0cyclictest0-21swapper/007:15:300
3213299140,14cyclictest0-21swapper/007:35:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional