You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-13 - 23:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Wed May 13, 2026 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
210992203167,24sleep20-21swapper/207:06:302
211612202166,24sleep10-21swapper/107:07:161
210522202165,25sleep00-21swapper/007:05:500
210382201165,24sleep30-21swapper/307:05:423
833521370,2sleep00-21swapper/007:50:170
1780921370,3sleep081ktimersoftd/011:52:490
297452660,6sleep12145699cyclictest12:20:151
196482620,4sleep02144899cyclictest08:15:220
6182480,2sleep30-21swapper/307:34:443
304002480,2sleep011-21rcuc/007:25:250
21464994743,3cyclictest0-21swapper/212:00:062
209262430,2sleep30-21swapper/310:45:223
21464994230,6cyclictest0-21swapper/212:10:032
2144899349,11cyclictest0-21swapper/012:15:000
2144899335,13cyclictest663-21munin-run07:35:000
2144899312,13cyclictest22411-21munin-run12:05:010
96362300,4sleep232-21ksoftirqd/207:55:012
21468993013,4cyclictest24685-40dnf09:40:263
166222300,7sleep2311ktimersoftd/210:35:212
2145699299,3cyclictest1340-21latency_hist08:50:021
21464992815,7cyclictest24685-40dnf09:40:272
2144899271,13cyclictest6386-21munin-run10:15:010
2146499268,8cyclictest23438-21timerwakeupswit10:50:232
2146499262,17cyclictest10165-21munin-run09:10:012
2145699266,9cyclictest0-21swapper/112:05:581
2145699263,10cyclictest18859-21smart_sda10:40:221
2145699260,17cyclictest0-21swapper/112:33:081
92162251,3sleep39251-21ntp_states09:05:183
2146499258,3cyclictest13804-21sed11:45:022
2146499251,14cyclictest19578-21perf10:45:012
2145699256,8cyclictest0-21swapper/111:25:431
2145699256,8cyclictest0-21swapper/110:00:401
2145699255,11cyclictest9331-21cut10:20:141
35652240,5sleep02144899cyclictest12:35:120
2146499249,7cyclictest12368-21timerandwakeup10:25:232
2146499243,13cyclictest17986-21latency_hist11:55:022
2146499242,13cyclictest1194-21taskset08:48:142
2146499241,10cyclictest2255-21cpu10:05:112
2145699244,16cyclictest24685-40dnf09:40:251
2145699243,11cyclictest0-21swapper/111:35:271
2145699241,19cyclictest8816-21users11:30:271
2145699240,17cyclictest0-21swapper/109:47:461
2145699240,17cyclictest0-21swapper/109:22:161
2144899242,10cyclictest14565-21munin-run09:20:010
2146899230,6cyclictest0-21swapper/311:42:173
2146899230,15cyclictest0-21swapper/307:37:033
2146499238,2cyclictest26747-21/usr/sbin/munin07:20:172
2146499236,8cyclictest19177-21/usr/sbin/munin09:30:162
2146499235,9cyclictest21473-21idleruntime08:20:172
2146499235,7cyclictest31281-21cat12:25:022
21464992316,4cyclictest5020-21fgrep08:55:222
2146499231,13cyclictest0-21swapper/209:55:202
2146499231,10cyclictest13309-21tr10:30:122
2145699235,8cyclictest0-21swapper/110:31:331
2145699231,18cyclictest23671-21munin-node07:15:011
21456992310,4cyclictest20746-21df12:00:111
2145699230,18cyclictest0-21swapper/111:21:281
2144899237,14cyclictest0-21swapper/007:25:000
2144899233,12cyclictest0-21swapper/010:39:570
2144899231,14cyclictest16184-21kworker/u8:009:44:570
75442220,4sleep121-21rcuc/109:00:261
2146899221,9cyclictest13551-21users08:00:273
2146899221,5cyclictest0-21swapper/310:23:163
2146899220,13cyclictest0-21swapper/310:50:243
2146499226,8cyclictest87650irq/129-enp1s0-12:35:112
2146499221,5cyclictest28607-21/usr/sbin/munin11:05:142
2146499221,12cyclictest27291-21cat08:35:012
2145699225,9cyclictest0-21swapper/110:15:111
2145699225,9cyclictest0-21swapper/110:15:111
2145699223,3cyclictest87550irq/128-enp1s0-08:50:151
2145699221,7cyclictest0-21swapper/109:35:491
2145699221,18cyclictest11717-21cut11:40:121
2145699221,11cyclictest30686-21fschecks_time09:55:151
2145699221,11cyclictest28987-21latency_hist12:20:011
21456992210,2cyclictest3852-21kworker/u8:110:49:151
2144899228,3cyclictest19480-21kworker/0:311:00:430
2144899227,5cyclictest12018-21sshd09:10:440
2144899226,5cyclictest16184-21kworker/u8:009:29:530
2144899223,12cyclictest0-21swapper/011:43:050
2144899222,13cyclictest0-21swapper/010:06:220
2144899222,12cyclictest28121-21kworker/u8:211:11:470
2144899222,12cyclictest0-21swapper/011:45:290
2144899222,12cyclictest0-21swapper/010:53:320
2144899222,12cyclictest0-21swapper/010:48:540
2144899221,14cyclictest0-21swapper/010:27:220
2146899211,6cyclictest0-21swapper/312:16:333
2146899211,6cyclictest0-21swapper/310:26:383
2146899211,4cyclictest0-21swapper/311:15:203
2146899211,11cyclictest5369-21/usr/sbin/munin07:45:153
2146899211,11cyclictest2855-21smart_sda08:50:233
2146499219,2cyclictest808-21snmpd09:48:542
2146499216,7cyclictest14171-21/usr/sbin/munin08:05:232
2146499215,9cyclictest0-21swapper/211:20:182
2146499215,8cyclictest23237-21taskset09:38:442
2146499214,8cyclictest18621-21chrt09:25:562
2146499213,7cyclictest4415-21/usr/sbin/munin10:10:172
2146499213,5cyclictest997-21latency_hist12:30:012
2146499213,5cyclictest32-21ksoftirqd/211:15:112
2146499212,6cyclictest32-21ksoftirqd/207:50:022
2146499212,10cyclictest0-21swapper/211:59:572
2146499211,9cyclictest0-21swapper/211:35:012
2146499211,3cyclictest32-21ksoftirqd/210:15:572
2146499211,3cyclictest32-21ksoftirqd/210:15:572
2146499211,12cyclictest0-21swapper/211:49:082
21464992110,3cyclictest10160-21diskmemload12:32:022
21464992110,3cyclictest0-21swapper/210:56:302
2145699211,9cyclictest579-21systemd-journal11:20:011
2145699211,8cyclictest0-21swapper/109:53:481
2145699211,18cyclictest0-21swapper/108:55:151
2145699211,18cyclictest0-21swapper/107:54:081
2145699211,11cyclictest27824-21sed11:00:261
2145699210,16cyclictest0-21swapper/110:39:411
2144899218,2cyclictest15822-21/usr/sbin/munin09:20:200
2144899212,7cyclictest31194-21munin-run12:25:000
2144899212,7cyclictest22978-21diskstats12:05:130
2144899211,12cyclictest10160-21diskmemload10:24:300
2144899210,13cyclictest24541-21kworker/u8:012:32:490
24382201,3sleep00-21swapper/011:17:590
2146899208,4cyclictest0-21swapper/310:15:173
2146899208,4cyclictest0-21swapper/310:15:173
2146899203,9cyclictest13820-21/usr/sbin/munin11:45:153
2146899202,8cyclictest15755-21users08:05:263
2146899201,8cyclictest0-21swapper/311:05:163
2146899201,5cyclictest0-21swapper/312:39:093
2146899201,5cyclictest0-21swapper/309:21:003
21468992011,3cyclictest87350irq/126-enp1s0-12:20:293
2146899201,10cyclictest3788-21/usr/sbin/munin08:55:233
2146899201,10cyclictest29760-21sshd09:52:453
2146899201,10cyclictest23569-21grep08:25:163
2146499209,5cyclictest0-21swapper/212:15:182
2146499207,2cyclictest0-21swapper/209:00:272
2146499206,7cyclictest12050irq/123-ahci[0007:35:012
2146499201,6cyclictest0-21swapper/209:52:562
2146499201,12cyclictest0-21swapper/211:29:192
2146499201,10cyclictest0-21swapper/209:15:182
2145699209,2cyclictest31806-21needreboot11:10:221
2145699208,2cyclictest23983-21awk10:55:021
2145699206,4cyclictest10365-21gltestperf07:55:151
2145699206,3cyclictest0-21swapper/110:11:461
2145699205,3cyclictest10169-21perf09:10:011
2145699204,3cyclictest87550irq/128-enp1s0-09:25:181
21456992017,2cyclictest0-21swapper/108:15:241
21456992016,3cyclictest0-21swapper/110:09:031
2145699201,3cyclictest191782chrt09:30:111
2145699201,18cyclictest0-21swapper/111:55:441
2145699201,17cyclictest0-21swapper/109:15:381
2145699201,17cyclictest0-21swapper/109:11:111
2145699201,17cyclictest0-21swapper/108:21:051
2145699201,17cyclictest0-21swapper/108:02:201
2145699201,11cyclictest11070-21cpuspeed_turbos10:25:111
2145699201,11cyclictest0-21swapper/107:40:171
2145699201,10cyclictest579-21systemd-journal08:40:221
2145699201,10cyclictest29291-21hwlatdetect11:05:161
2145699201,10cyclictest26674-21sed08:30:251
2145699201,10cyclictest24036-21ntp_states08:25:211
2145699201,10cyclictest16778-21if_enp2s011:50:161
2145699201,10cyclictest15296-21/usr/sbin/munin08:05:211
2144899206,7cyclictest0-21swapper/012:26:180
2144899204,8cyclictest4131-21switchtime11:20:250
2144899202,12cyclictest0-21swapper/010:16:010
2144899202,12cyclictest0-21swapper/010:16:010
2144899202,10cyclictest19806-21users11:55:260
2144899201,4cyclictest32144-21munin-run10:00:000
2144899201,12cyclictest0-21swapper/012:17:540
2144899201,12cyclictest0-21swapper/011:38:380
2144899201,12cyclictest0-21swapper/010:32:300
2144899201,12cyclictest0-21swapper/009:38:210
2144899201,10cyclictest11961-21/usr/sbin/munin08:00:140
2144899200,7cyclictest0-21swapper/011:33:350
9882191,2sleep10-21swapper/112:29:591
2146899198,3cyclictest18564-21/usr/sbin/munin08:15:243
2146899198,3cyclictest10235-21irqstats11:35:183
2146899197,7cyclictest0-21swapper/308:11:503
2146899191,9cyclictest7249-21cpu11:30:123
2146899191,7cyclictest0-21swapper/311:27:363
2146899191,7cyclictest0-21swapper/309:25:163
2146899191,6cyclictest0-21swapper/310:13:253
2146899191,6cyclictest0-21swapper/308:35:103
2146899191,4cyclictest0-21swapper/312:25:363
21468991913,3cyclictest13652-21ntp_states09:15:213
2146899191,11cyclictest0-21swapper/310:01:323
2146899191,10cyclictest25714-21latency12:10:193
2146899191,10cyclictest21592-21kernelversion08:20:203
2146899191,10cyclictest0-21swapper/311:51:573
2146499195,5cyclictest755-21rngd10:20:112
2146499195,2cyclictest9765-21/usr/sbin/munin07:55:242
2146499195,2cyclictest0-21swapper/211:35:272
2146499194,7cyclictest16366-21/usr/sbin/munin08:10:222
2146499192,8cyclictest0-21swapper/207:40:262
2146499192,14cyclictest27043-21gltestperf11:00:152
2146499191,6cyclictest22621-21seq12:05:062
2146499191,6cyclictest12010-21taskset09:10:402
2146499191,10cyclictest0-21swapper/207:27:312
2146499190,8cyclictest87350irq/126-enp1s0-08:35:102
2145699197,3cyclictest32291-21head07:30:221
2145699191,8cyclictest0-21swapper/112:14:211
21456991916,2cyclictest0-21swapper/112:35:471
2145699191,4cyclictest27819-21ntp_states07:20:221
2145699191,17cyclictest0-21swapper/107:16:101
2145699191,16cyclictest0-21swapper/111:45:091
2145699191,16cyclictest0-21swapper/108:35:491
2145699191,16cyclictest0-21swapper/108:10:331
2145699191,16cyclictest0-21swapper/107:36:541
2145699190,5cyclictest0-21swapper/110:55:041
2144899197,7cyclictest87350irq/126-enp1s0-09:47:250
2144899195,4cyclictest28739-21cpuspeed_turbos11:05:120
2144899195,3cyclictest827-21runrttasks10:05:000
2144899192,7cyclictest18129-21fschecks_time10:40:150
2144899191,10cyclictest14171-21/usr/sbin/munin08:05:130
21448991910,2cyclictest87650irq/129-enp1s0-07:42:430
184062191,3sleep318509-21memory10:40:173
93102180,2sleep30-21swapper/307:52:153
39022180,3sleep30-21swapper/310:06:313
2146899189,2cyclictest16184-21kworker/u8:007:56:503
2146899185,3cyclictest23544-21memory12:05:193
2146899183,7cyclictest39-21rcuc/308:45:143
2146899182,9cyclictest0-21swapper/307:10:273
2146899182,7cyclictest28944-21/usr/sbin/munin07:25:223
2146899181,6cyclictest0-21swapper/312:04:303
2146899181,5cyclictest12178-21seq09:12:483
2146899181,5cyclictest0-21swapper/312:32:223
21468991811,4cyclictest31524-21idleruntime-cro08:45:003
2146899181,11cyclictest0-21swapper/309:45:263
2146899181,10cyclictest0-21swapper/311:56:303
2146899181,10cyclictest0-21swapper/307:40:123
2146899180,5cyclictest0-21swapper/309:30:243
2146499188,7cyclictest29905-21diskstats08:40:142
2146499181,4cyclictest0-21swapper/211:10:212
2146499181,15cyclictest0-21swapper/207:16:502
2146499181,10cyclictest19798-21/usr/sbin/munin10:45:112
2145699181,3cyclictest29292-21diskstats07:25:141
2145699180,16cyclictest0-21swapper/107:45:071
2144899188,3cyclictest87550irq/128-enp1s0-08:45:450
2144899186,3cyclictest9765-21/usr/sbin/munin07:55:110
2144899181,9cyclictest808-21snmpd07:38:260
2144899181,6cyclictest4933-21pmu-power08:55:210
2144899181,5cyclictest0-21swapper/008:53:500
2144899181,4cyclictest5983-21/usr/sbin/munin09:00:230
2144899181,3cyclictest0-21swapper/009:52:090
21448991812,3cyclictest6616-21uptime11:25:260
2144899181,10cyclictest0-21swapper/009:32:330
259582171,2sleep30-21swapper/310:57:273
2146899171,6cyclictest0-21swapper/311:02:103
2146899171,5cyclictest6352-21fschecks_count09:00:143
2146899171,15cyclictest0-21swapper/310:35:313
2146899171,15cyclictest0-21swapper/310:30:473
2146899171,15cyclictest0-21swapper/309:57:313
21468991711,3cyclictest30864-21cpu11:10:133
21468991711,3cyclictest25162-21/usr/sbin/munin08:30:243
21468991711,3cyclictest22150-21if_enp2s009:35:193
2146899170,14cyclictest27407-21cut07:20:163
2146499179,5cyclictest22732-21ntp_states07:10:222
2146499178,5cyclictest1060-21ntp_states10:00:212
2146499173,2cyclictest87550irq/128-enp1s0-08:25:262
2146499172,7cyclictest12299-21df08:00:132
2146499171,6cyclictest15855-21ntp_states09:20:222
2146499171,14cyclictest0-21swapper/208:51:302
2146499171,14cyclictest0-21swapper/207:35:022
2144899171,9cyclictest774-21avahi-daemon09:07:120
2144899171,9cyclictest774-21avahi-daemon08:39:010
2144899171,5cyclictest20761-21/usr/sbin/munin08:20:160
2144899171,3cyclictest7336-21idleruntime-cro07:50:000
2144899171,3cyclictest17127-21if_enp2s008:10:170
21448991711,3cyclictest25162-21/usr/sbin/munin08:30:140
21448991711,3cyclictest22966-21/usr/sbin/munin08:25:210
2144899170,12cyclictest0-21swapper/010:59:010
2146899169,4cyclictest24025-21tune2fs07:15:023
2146899165,4cyclictest839-21lldpd11:20:173
2146499167,6cyclictest19627-21ntp_states08:15:212
2144899161,14cyclictest0-21swapper/007:15:560
21448991611,3cyclictest29555-21/usr/sbin/munin08:40:130
21448991611,3cyclictest21730-21wc07:10:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional