You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-29 - 21:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Fri May 29, 2026 12:44:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
56512245154,23sleep00-21swapper/007:05:580
58792240164,24sleep10-21swapper/107:08:561
59192239163,23sleep20-21swapper/207:09:272
59132203166,24sleep30-21swapper/307:09:233
91632640,5sleep07-21ksoftirqd/011:20:250
278532570,4sleep341-21ksoftirqd/310:50:183
17652540,4sleep21767-21sendmail_mailst09:55:242
88042530,3sleep2605899cyclictest12:35:122
310882500,4sleep230-21rcuc/211:00:002
26742500,1sleep10-21swapper/111:05:251
605899400,29cyclictest0-21swapper/209:15:262
604899377,14cyclictest12872-21munin-run09:10:020
6058993625,3cyclictest0-21swapper/211:12:342
605899361,21cyclictest0-21swapper/212:16:542
605899361,21cyclictest0-21swapper/212:16:542
605899341,25cyclictest0-21swapper/211:05:142
605899341,15cyclictest0-21swapper/209:44:352
605899331,22cyclictest0-21swapper/207:59:512
605899331,16cyclictest0-21swapper/210:50:422
604899336,12cyclictest20173-21perf07:55:010
6058993215,3cyclictest0-21swapper/207:26:542
605899321,20cyclictest0-21swapper/208:28:222
605899321,16cyclictest0-21swapper/210:05:502
605899321,16cyclictest0-21swapper/209:36:072
605899321,15cyclictest0-21swapper/212:26:022
605899321,15cyclictest0-21swapper/209:45:072
605899320,16cyclictest0-21swapper/211:27:302
605899311,15cyclictest0-21swapper/212:32:102
605899311,15cyclictest0-21swapper/211:55:592
605899311,14cyclictest0-21swapper/212:06:342
605899311,12cyclictest0-21swapper/211:41:112
605899310,15cyclictest0-21swapper/211:34:262
605899301,21cyclictest0-21swapper/211:45:272
605899301,20cyclictest0-21swapper/207:16:542
605899301,16cyclictest0-21swapper/208:44:102
605899301,15cyclictest0-21swapper/210:35:142
605899301,14cyclictest0-21swapper/209:29:582
605899301,14cyclictest0-21swapper/208:22:222
605899301,13cyclictest0-21swapper/210:33:502
605899301,13cyclictest0-21swapper/209:50:552
605899301,12cyclictest0-21swapper/210:21:502
605899301,12cyclictest0-21swapper/207:39:182
604899303,12cyclictest22924-21perf11:55:000
605899299,9cyclictest0-21swapper/207:51:112
6058992915,9cyclictest0-21swapper/209:06:582
6058992914,10cyclictest0-21swapper/207:43:232
605899291,20cyclictest0-21swapper/207:14:302
605899291,19cyclictest0-21swapper/211:01:022
605899291,19cyclictest0-21swapper/207:30:302
605899291,18cyclictest0-21swapper/208:53:182
605899291,14cyclictest0-21swapper/208:14:302
605899291,14cyclictest0-21swapper/208:08:512
605899291,13cyclictest0-21swapper/211:36:472
605899290,12cyclictest0-21swapper/210:44:222
604899292,13cyclictest23874-21perf09:35:000
6058992818,3cyclictest0-21swapper/208:35:382
605899281,12cyclictest0-21swapper/211:53:512
605899281,11cyclictest0-21swapper/211:23:302
605899281,11cyclictest0-21swapper/211:15:462
605899281,11cyclictest0-21swapper/209:00:222
605899280,15cyclictest0-21swapper/210:45:382
605899280,13cyclictest0-21swapper/208:49:262
605899280,12cyclictest0-21swapper/212:12:302
605899280,11cyclictest0-21swapper/207:46:092
605899271,11cyclictest0-21swapper/212:02:472
605899271,10cyclictest12867-21diskmemload10:28:182
605899271,10cyclictest0-21swapper/210:18:022
605899271,10cyclictest0-21swapper/209:24:542
605899270,14cyclictest0-21swapper/210:04:512
65152260,3sleep339-21rcuc/307:10:113
605899269,8cyclictest0-21swapper/208:03:232
6058992618,3cyclictest0-21swapper/208:19:062
6058992611,5cyclictest0-21swapper/212:20:022
605899261,12cyclictest0-21swapper/209:13:222
6053992613,8cyclictest29629-21pmu-power09:45:251
605899259,4cyclictest9753-21/usr/sbin/munin07:20:222
6058992515,3cyclictest0-21swapper/208:30:062
605899251,12cyclictest0-21swapper/208:59:302
605899251,11cyclictest0-21swapper/209:30:502
605399251,11cyclictest14763-21ntp_kernel_pll_10:20:191
605399251,11cyclictest0-21swapper/107:10:121
604899251,13cyclictest919-21awk11:05:020
604899251,12cyclictest16306-21idleruntime-cro11:40:010
606599241,15cyclictest0-21swapper/309:45:233
606599240,16cyclictest0-21swapper/308:15:223
605899241,16cyclictest0-21swapper/210:14:022
605399244,3cyclictest87550irq/128-enp1s0-10:30:231
6053992414,2cyclictest27996-21seq09:41:531
605399241,17cyclictest0-21swapper/110:16:071
605399241,16cyclictest0-21swapper/109:20:241
6048992415,7cyclictest6445-21kworker/u8:011:45:210
6048992411,3cyclictest232312cpuspeed_turbos11:55:120
606599237,6cyclictest0-21swapper/309:24:313
606599232,9cyclictest0-21swapper/310:20:023
606599230,11cyclictest0-21swapper/310:33:363
605399231,19cyclictest26060-21perf09:40:011
605399231,16cyclictest0-21swapper/112:20:071
605399231,16cyclictest0-21swapper/108:52:281
605399231,14cyclictest0-21swapper/111:12:221
605399231,12cyclictest0-21swapper/109:50:201
605399231,11cyclictest0-21swapper/110:50:061
604899231,11cyclictest1880-21awk08:45:020
605399229,3cyclictest811-21df_inode09:55:131
605399228,12cyclictest24433-21kworker/u8:107:30:531
605399227,4cyclictest0-21swapper/109:27:301
605399222,15cyclictest0-21swapper/108:20:431
605399222,15cyclictest0-21swapper/108:11:521
605399222,14cyclictest0-21swapper/110:48:151
605399222,10cyclictest1235-21perf08:40:001
605399221,5cyclictest0-21swapper/109:33:161
6053992215,4cyclictest10681-21munin-run09:05:011
6053992213,2cyclictest808-21snmpd07:27:251
6053992212,2cyclictest0-21swapper/111:35:191
605399221,15cyclictest0-21swapper/111:44:461
605399221,15cyclictest0-21swapper/111:00:421
605399221,15cyclictest0-21swapper/107:35:361
605399221,11cyclictest0-21swapper/112:30:151
605399221,11cyclictest0-21swapper/111:34:331
605399220,16cyclictest0-21swapper/109:10:061
6048992216,3cyclictest5693-21tr11:15:100
604899221,15cyclictest0-21swapper/009:19:500
606599215,6cyclictest0-21swapper/309:42:473
606599215,4cyclictest16992-21chrt09:16:433
606599215,10cyclictest12465-21expr11:30:143
606599211,11cyclictest14773-21sshd07:35:223
605399218,2cyclictest1309-21sshd12:17:231
605399218,2cyclictest1309-21sshd12:17:231
605399216,3cyclictest12867-21diskmemload12:10:201
6053992116,3cyclictest0-21swapper/110:36:201
605399211,17cyclictest21284-21diskstats11:50:141
605399211,15cyclictest0-21swapper/110:10:461
605399211,15cyclictest0-21swapper/108:18:571
605399211,15cyclictest0-21swapper/107:21:231
605399211,14cyclictest0-21swapper/107:46:301
6053992111,3cyclictest0-21swapper/109:15:191
605399211,12cyclictest26597-21smart_sda12:00:231
605399211,11cyclictest6464-21memory11:15:181
605399211,11cyclictest29178-21diskstats08:25:131
605399211,11cyclictest27851-21expr12:05:121
605399211,11cyclictest0-21swapper/111:21:081
605399211,10cyclictest1875-21perf08:45:011
604899212,9cyclictest19764-21kworker/0:110:33:140
604899211,7cyclictest13506-21sshd10:20:090
604899211,7cyclictest0-21swapper/010:13:080
604899211,6cyclictest29662-21processes09:45:260
604899211,4cyclictest11984-21systemctl10:19:210
6048992111,3cyclictest87550irq/128-enp1s0-12:18:280
6048992111,3cyclictest87550irq/128-enp1s0-12:18:270
606599205,12cyclictest87350irq/126-enp1s0-10:20:133
606599204,7cyclictest24433-21kworker/u8:110:12:233
606599203,3cyclictest87350irq/126-enp1s0-11:22:153
606599203,14cyclictest7098-21head08:55:153
606599202,15cyclictest19379-21head11:45:143
606599202,10cyclictest20317-21/usr/sbin/munin10:35:173
606599201,6cyclictest21343-21df_inode08:00:133
6065992014,4cyclictest23415-21unixbench_multi09:30:253
6065992014,3cyclictest29420-21grep08:25:163
606599201,2cyclictest87350irq/126-enp1s0-09:52:323
606599201,12cyclictest87350irq/126-enp1s0-12:35:193
606599201,12cyclictest0-21swapper/312:17:203
606599201,12cyclictest0-21swapper/312:17:203
606599201,11cyclictest16284-21memory07:40:193
606599201,11cyclictest13093-21/usr/sbin/munin09:10:183
606599201,11cyclictest0-21swapper/311:50:083
606599201,10cyclictest0-21swapper/310:00:253
605399209,3cyclictest16637-21smart_sda07:40:251
605399203,11cyclictest87550irq/128-enp1s0-12:35:131
605399202,16cyclictest0-21swapper/107:55:311
6053992013,2cyclictest0-21swapper/107:51:111
6053992012,3cyclictest0-21swapper/110:41:051
605399201,15cyclictest0-21swapper/108:58:281
605399201,15cyclictest0-21swapper/108:00:471
605399201,14cyclictest0-21swapper/111:25:461
605399201,14cyclictest0-21swapper/110:26:411
605399201,14cyclictest0-21swapper/109:09:251
605399201,14cyclictest0-21swapper/108:32:121
605399201,12cyclictest0-21swapper/111:57:471
605399201,12cyclictest0-21swapper/110:06:071
604899201,8cyclictest26281-21/usr/sbin/munin09:40:250
604899201,4cyclictest0-21swapper/009:13:550
604899201,16cyclictest28608-21munin-run08:25:000
604899201,11cyclictest3967-21/usr/sbin/munin12:25:200
604899201,11cyclictest24714-21/usr/sbin/munin10:45:170
68322191,2sleep00-21swapper/010:09:230
606599196,11cyclictest87350irq/126-enp1s0-12:32:323
606599194,4cyclictest0-21swapper/312:01:513
606599194,12cyclictest39-21rcuc/311:10:243
606599193,6cyclictest0-21swapper/311:42:273
606599192,5cyclictest0-21swapper/310:55:223
606599191,7cyclictest0-21swapper/312:10:303
606599191,6cyclictest2834-21ntp_states12:20:203
606599191,3cyclictest0-21swapper/311:35:203
606599191,11cyclictest0-21swapper/310:44:343
606599191,10cyclictest6776-21gzip11:15:213
606599191,10cyclictest28386-21sshd08:22:223
606599191,10cyclictest13373-21irqstats07:30:173
606599191,10cyclictest10821-21sshd07:20:213
606599191,10cyclictest10806-21latency11:25:173
606599191,10cyclictest0-21swapper/312:25:043
605399196,4cyclictest0-21swapper/110:02:261
6053991912,2cyclictest0-21swapper/110:55:571
6053991912,1cyclictest0-21swapper/107:19:511
605399191,10cyclictest4028-21egrep12:25:111
605399191,10cyclictest24843-21chrt08:05:501
6053991910,2cyclictest0-21swapper/111:45:141
604899199,6cyclictest0-21swapper/011:44:560
604899195,2cyclictest24964-21grep09:35:170
604899191,6cyclictest0-21swapper/012:36:160
604899191,6cyclictest0-21swapper/010:51:490
604899191,6cyclictest0-21swapper/010:29:380
604899191,5cyclictest0-21swapper/009:29:450
604899191,4cyclictest17345-21munin-run07:45:010
604899191,4cyclictest0-21swapper/010:44:590
604899191,10cyclictest2710-21/usr/sbin/munin10:00:180
604899191,10cyclictest2090-21/usr/sbin/munin08:45:130
232302190,2sleep30-21swapper/311:55:123
606599185,4cyclictest0-21swapper/310:05:233
606599181,9cyclictest20090-21sshd07:54:203
606599181,7cyclictest0-21swapper/308:13:203
606599181,6cyclictest0-21swapper/309:55:423
606599181,10cyclictest23593-21fschecks_count08:05:143
606599181,10cyclictest0-21swapper/311:08:133
606599180,10cyclictest0-21swapper/309:00:103
605399181,16cyclictest0-21swapper/108:46:131
604899187,3cyclictest28830-21/usr/sbin/munin08:25:170
604899181,9cyclictest808-21snmpd08:17:330
604899181,6cyclictest0-21swapper/012:33:250
6048991812,3cyclictest481-21cpuspeed_turbos09:55:120
6048991812,3cyclictest25421-21cat12:00:110
6048991811,4cyclictest0-21swapper/010:55:280
315182181,2sleep00-21swapper/012:12:430
606599174,7cyclictest0-21swapper/309:38:163
606599172,3cyclictest12050irq/123-ahci[0008:00:013
606599172,12cyclictest5278-21needreboot08:50:183
606599171,9cyclictest0-21swapper/309:05:173
606599171,6cyclictest18090-21cat07:50:013
606599171,4cyclictest31985-21diskstats08:35:123
6065991712,3cyclictest25781-21ntpq10:45:213
606599171,15cyclictest0-21swapper/312:07:383
606599171,13cyclictest0-21swapper/311:00:053
606599171,13cyclictest0-21swapper/310:25:143
606599171,13cyclictest0-21swapper/309:25:163
604899175,3cyclictest12050irq/123-ahci[0008:40:010
604899171,7cyclictest839-21lldpd11:30:220
604899171,6cyclictest3747-21turbostat12:25:020
604899171,6cyclictest31419-21perf08:35:000
604899171,5cyclictest0-21swapper/009:24:330
6048991712,3cyclictest5604-21wc08:50:240
6048991712,3cyclictest22914-21seq08:04:210
6048991711,3cyclictest8602-21if_err_enp1s007:15:020
6048991710,6cyclictest7-21ksoftirqd/007:10:480
604899170,15cyclictest0-21swapper/011:25:120
38192170,3sleep30-21swapper/308:47:033
178202171,2sleep00-21swapper/007:48:250
606599166,8cyclictest87350irq/126-enp1s0-08:40:223
606599161,14cyclictest0-21swapper/307:26:193
606599160,15cyclictest0-21swapper/307:18:183
604899163,2cyclictest87350irq/126-enp1s0-11:12:110
604899162,3cyclictest87550irq/128-enp1s0-08:55:160
604899161,4cyclictest31846-21ps09:50:210
604899161,4cyclictest27542-21/usr/sbin/munin12:05:210
6048991612,3cyclictest0-21swapper/007:57:070
604899161,14cyclictest0-21swapper/007:39:050
6048991611,3cyclictest25810-21perf08:15:010
6048991611,3cyclictest21386-21ntp_states10:35:220
6048991611,3cyclictest13153-21grep07:30:150
6048991611,3cyclictest12315-21seq07:29:540
6048991611,3cyclictest11318-21head07:20:270
6048991611,3cyclictest10189-21grep09:00:250
6048991610,3cyclictest23314-21tr08:05:110
606599151,9cyclictest0-21swapper/308:30:143
604899151,6cyclictest1132-21/usr/sbin/munin11:05:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional