You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-20 - 16:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Wed May 20, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
152032235162,60sleep00-21swapper/007:09:410
149182233163,57sleep20-21swapper/207:06:012
150392203166,25sleep30-21swapper/307:07:353
150692201165,24sleep10-21swapper/107:07:581
1043221580,3sleep11531799cyclictest11:50:131
1171721430,4sleep21532399cyclictest11:53:312
452821220,4sleep01531199cyclictest07:55:190
15323999083,3cyclictest0-21swapper/211:22:572
15323996460,3cyclictest0-21swapper/211:16:012
212062580,3sleep30-21swapper/312:15:113
212062580,3sleep30-21swapper/312:15:113
168812570,3sleep30-21swapper/310:50:203
114712540,1sleep10-21swapper/108:10:251
3972510,1sleep10-21swapper/109:00:171
1532899402,32cyclictest5278-40dnf07:56:143
1532899398,20cyclictest0-21swapper/307:30:123
1532899381,26cyclictest2430-21latency_hist10:20:013
1532899381,26cyclictest13420-21latency_hist10:45:013
1532899381,24cyclictest29667-21/usr/sbin/munin12:35:253
1532899376,20cyclictest12050irq/123-ahci[0009:40:013
1532899375,21cyclictest12050irq/123-ahci[0012:00:013
1532899374,19cyclictest0-21swapper/309:05:143
1532899361,3cyclictest4384-21pluginstate11:35:233
1532899361,23cyclictest18450-21turbostat.cron12:10:013
1532899351,6cyclictest0-21swapper/309:55:143
1532899351,25cyclictest808-21snmpd09:32:053
1532899351,25cyclictest4854-21chrt10:25:073
1531799351,24cyclictest0-21swapper/109:30:391
1532899347,14cyclictest18052-21/usr/sbin/munin10:55:143
1532899343,3cyclictest8364-21hwlatdetect11:45:163
1532899341,4cyclictest0-21swapper/307:45:203
1532899341,21cyclictest32366-21sshd10:12:113
1532899334,3cyclictest12050irq/123-ahci[0009:15:023
1532899334,2cyclictest27192-21/usr/sbin/munin07:35:253
1532899333,2cyclictest87350irq/126-enp1s0-09:20:013
1532899331,7cyclictest16826-21/usr/sbin/munin08:25:263
1532899331,5cyclictest808-21snmpd09:42:073
1532899331,29cyclictest29051-21/usr/sbin/munin11:20:193
1531799331,24cyclictest0-21swapper/108:47:531
15311993314,16cyclictest3057-21munin-run11:35:010
119812330,4sleep3401ktimersoftd/311:55:023
183992320,3sleep31532899cyclictest07:15:053
1532899324,2cyclictest0-21swapper/311:05:223
1532899324,24cyclictest31975-21sa109:00:023
1532899321,3cyclictest25417-21cpuspeed_turbos12:25:133
1532899321,18cyclictest28430-21/usr/sbin/munin10:05:183
15328993210,19cyclictest5820-21/usr/sbin/munin08:00:153
1532899320,5cyclictest0-21swapper/309:54:323
1531799321,13cyclictest3832-21ps10:20:241
1531799321,13cyclictest0-21swapper/108:01:191
1531799321,12cyclictest23313-21grep07:25:161
1531799320,24cyclictest0-21swapper/108:33:331
1531199326,12cyclictest29178-21idleruntime-cro07:40:010
1531199321,15cyclictest24773-21munin-run07:30:010
1532899315,15cyclictest12426-21/usr/sbin/munin08:15:193
1532899311,18cyclictest4019-21diskmemload10:23:003
15328993110,18cyclictest31355-21sh07:45:013
1532899310,4cyclictest0-21swapper/312:10:223
1532899310,29cyclictest0-21swapper/309:25:113
1531799311,12cyclictest0-21swapper/111:10:051
1531199311,15cyclictest5254-21munin-run11:40:010
1532899308,19cyclictest31245-21/usr/sbin/munin11:25:233
1532899303,23cyclictest2152-21ntp_states11:30:223
1532899302,4cyclictest1774-21diskstats07:50:143
1532899302,3cyclictest21067-21irqstats11:00:193
1532899301,3cyclictest27406-21perl11:15:143
1532899301,21cyclictest27457-21/usr/sbin/munin10:00:223
1532899301,21cyclictest27457-21/usr/sbin/munin10:00:223
1532899301,18cyclictest4019-21diskmemload10:46:233
1531799301,22cyclictest0-21swapper/109:41:111
1531799301,12cyclictest0-21swapper/108:50:171
1531799301,12cyclictest0-21swapper/108:50:171
1531799300,23cyclictest0-21swapper/112:24:191
1532899292,4cyclictest27198-21users08:45:273
1532899292,4cyclictest0-21swapper/310:33:213
1532899292,2cyclictest8643-21/usr/sbin/munin09:20:153
1532899291,4cyclictest26196-21/usr/sbin/munin11:10:263
1532899291,4cyclictest10236-21/usr/sbin/munin08:10:123
1532899291,3cyclictest0-21swapper/312:34:503
1532899291,17cyclictest4019-21diskmemload12:22:223
1532899291,13cyclictest0-21swapper/311:40:163
1532899290,6cyclictest0-21swapper/312:00:233
1532899290,25cyclictest9380-21ls08:05:243
1531799291,26cyclictest0-21swapper/107:35:161
1531799291,11cyclictest0-21swapper/111:36:311
1531799291,10cyclictest0-21swapper/110:27:421
187532281,23sleep018412-21/usr/sbin/munin07:15:150
1532899289,16cyclictest32209-21/usr/sbin/munin09:00:183
1532899289,16cyclictest22992-21expr07:25:133
1532899287,18cyclictest9632-21fschecks_count10:35:123
1532899283,22cyclictest21006-21cat09:45:243
1532899282,3cyclictest14620-21/usr/sbin/munin08:20:233
1532899281,4cyclictest28549-21/usr/sbin/munin08:50:183
1532899281,4cyclictest28549-21/usr/sbin/munin08:50:183
1532899281,2cyclictest21214-21/usr/sbin/munin08:35:193
1532899281,2cyclictest19025-21/usr/sbin/munin08:30:223
1531799281,14cyclictest0-21swapper/112:05:011
1531799281,13cyclictest0-21swapper/112:25:571
1531799281,12cyclictest0-21swapper/108:08:351
1531799281,11cyclictest0-21swapper/107:50:261
15311992810,15cyclictest2438-21munin-run10:20:010
1532899271,2cyclictest20691-21/usr/sbin/munin07:20:113
1532899271,2cyclictest15522-21/usr/sbin/munin07:10:173
1532899270,3cyclictest23417-21/usr/sbin/munin08:40:163
1531799271,9cyclictest0-21swapper/111:20:391
1531799271,16cyclictest0-21swapper/112:06:191
1531799271,11cyclictest0-21swapper/108:55:591
1531799271,11cyclictest0-21swapper/108:26:471
1531799271,10cyclictest0-21swapper/110:37:131
1531799270,9cyclictest0-21swapper/111:40:251
1531799270,11cyclictest0-21swapper/110:30:211
1531799270,11cyclictest0-21swapper/107:21:391
1531799270,10cyclictest0-21swapper/108:23:191
1531199272,8cyclictest17128-21kworker/0:009:42:160
1531199271,20cyclictest9479-21kworker/0:011:56:550
1531799261,9cyclictest0-21swapper/110:05:521
1531799261,9cyclictest0-21swapper/107:58:081
1531799261,12cyclictest0-21swapper/108:35:351
1531799261,10cyclictest4019-21diskmemload09:18:091
1531799261,10cyclictest0-21swapper/109:05:091
1531799261,10cyclictest0-21swapper/107:43:381
1531799260,9cyclictest0-21swapper/107:47:441
1531799260,11cyclictest0-21swapper/108:43:071
1531199268,15cyclictest22575-21munin-run07:25:010
1531199263,11cyclictest8434-21idleruntime-cro09:20:010
1531199262,20cyclictest27070-21kworker/0:112:32:240
1531799251,9cyclictest0-21swapper/110:51:311
15317992512,11cyclictest8356-21kworker/u8:109:50:261
1531799251,12cyclictest27199-21diskstats11:15:131
1531799251,12cyclictest0-21swapper/109:36:171
1531799250,9cyclictest0-21swapper/107:32:521
1531799250,11cyclictest0-21swapper/111:02:131
1531799250,10cyclictest0-21swapper/109:10:071
1531199252,10cyclictest24254-21kworker/0:211:50:250
1531199251,11cyclictest14909-21kworker/0:209:45:180
140422250,7sleep1221ktimersoftd/112:00:001
1531799248,7cyclictest0-21swapper/108:16:131
1531799241,9cyclictest4019-21diskmemload09:29:051
1531799241,9cyclictest32233-21strings11:25:191
1531799241,9cyclictest0-21swapper/110:18:401
1531799241,11cyclictest0-21swapper/112:36:111
1531799241,10cyclictest0-21swapper/107:13:131
1531799240,9cyclictest0-21swapper/111:34:271
1531799240,9cyclictest0-21swapper/111:06:431
1532399231,9cyclictest218642tail08:35:132
1531799239,3cyclictest0-21swapper/110:45:191
1531799238,3cyclictest0-21swapper/110:03:151
1531799238,3cyclictest0-21swapper/110:03:151
1531799236,7cyclictest0-21swapper/112:17:391
1531799236,7cyclictest0-21swapper/112:17:391
1531799231,10cyclictest0-21swapper/109:48:111
1531799230,9cyclictest0-21swapper/110:58:031
1531799229,3cyclictest0-21swapper/109:56:551
1531799228,4cyclictest20378-21munin-run07:20:001
1531799227,3cyclictest9075-21grep09:20:131
1531799221,8cyclictest0-21swapper/110:13:401
1531799220,8cyclictest0-21swapper/112:34:591
1531199224,11cyclictest4019-21diskmemload10:57:250
1531199222,9cyclictest8805-21kworker/0:210:32:330
1531199221,8cyclictest20417-21kworker/0:312:12:250
1531199221,13cyclictest0-21swapper/011:25:190
1531199221,12cyclictest25277-21/usr/sbin/munin12:25:210
1532399211,9cyclictest0-21swapper/207:31:132
1532399211,10cyclictest0-21swapper/212:30:072
1531799211,3cyclictest13042-21users10:40:251
1531799211,12cyclictest0-21swapper/112:10:191
1531799210,10cyclictest0-21swapper/111:46:211
1531199215,14cyclictest16275-21awk12:05:010
1531199213,8cyclictest0-21swapper/009:35:160
263482201,2sleep20-21swapper/212:25:232
1532399202,15cyclictest30615-21grep08:55:162
15323992013,4cyclictest12116-21gltestperf10:40:142
1532399200,6cyclictest0-21swapper/207:15:172
1532399200,13cyclictest0-21swapper/210:20:112
1531199201,9cyclictest19245-21kworker/0:309:52:500
1531199201,8cyclictest31274-21kworker/0:012:36:140
15311992013,4cyclictest6470-21needreboot11:40:190
1531199201,17cyclictest10002-21munin-run08:10:010
1531199201,11cyclictest20825-21sshd08:33:020
1531199201,11cyclictest15522-21/usr/sbin/munin07:10:260
1531199201,11cyclictest13038-21/usr/sbin/munin09:30:230
1531199201,10cyclictest808-21snmpd10:52:220
142662201,3sleep20-21swapper/212:00:062
76712191,2sleep00-21swapper/011:45:080
297562191,3sleep029975-21gltestperf07:40:150
234222190,4sleep230-21rcuc/212:20:122
1532399194,3cyclictest0-21swapper/209:59:472
1532399193,7cyclictest22801-21/usr/sbin/munin07:25:242
1532399191,6cyclictest0-21swapper/211:29:562
1532399191,6cyclictest0-21swapper/209:45:402
1532399191,4cyclictest0-21swapper/210:13:172
1532399190,13cyclictest0-21swapper/210:04:052
1532399190,13cyclictest0-21swapper/210:04:052
1531199195,2cyclictest989-21kworker/0:108:00:330
1531199194,13cyclictest12050irq/123-ahci[0009:00:010
1531199193,14cyclictest0-21swapper/010:10:010
1531199192,7cyclictest17298-21fschecks_time08:25:160
1531199191,6cyclictest0-21swapper/009:11:140
15311991914,3cyclictest21564-21sshd08:35:110
1531199191,3cyclictest0-21swapper/012:22:110
1531199191,12cyclictest0-21swapper/009:22:400
1531199191,11cyclictest10229-21/usr/sbin/munin08:10:250
1531199191,11cyclictest0-21swapper/011:13:360
1531199191,10cyclictest808-21snmpd08:46:560
1531199191,10cyclictest31586-21/usr/sbin/munin07:45:210
1531199191,10cyclictest23417-21/usr/sbin/munin08:40:140
1532399181,7cyclictest0-21swapper/209:35:502
1532399181,6cyclictest30708-21ntpq12:35:202
1532399181,6cyclictest0-21swapper/210:51:222
1532399181,4cyclictest0-21swapper/209:10:212
15323991813,3cyclictest21557-21hddtemp_smartct12:15:152
15323991813,3cyclictest21557-21hddtemp_smartct12:15:152
15323991811,4cyclictest6164-21seq08:00:132
1531199182,7cyclictest19245-21kworker/0:310:13:250
1531199181,7cyclictest0-21swapper/011:20:300
1531199181,15cyclictest15180-21grep10:45:250
1531199181,11cyclictest0-21swapper/011:00:280
1531199181,10cyclictest26851-21/usr/sbin/munin11:15:160
312032171,2sleep20-21swapper/207:43:102
197912171,2sleep20-21swapper/210:57:152
186822171,2sleep20-21swapper/208:28:452
1532399174,4cyclictest19090-21cpuspeed_turbos08:30:122
1532399174,4cyclictest0-21swapper/210:33:092
1532399171,9cyclictest0-21swapper/211:31:122
1532399171,7cyclictest5589-21cpuspeed_turbos11:40:122
1532399171,6cyclictest21324-21ntp_states11:00:252
1532399171,6cyclictest0-21swapper/210:16:142
1532399171,6cyclictest0-21swapper/210:05:162
1532399171,6cyclictest0-21swapper/209:54:372
1532399171,6cyclictest0-21swapper/209:22:162
1532399171,6cyclictest0-21swapper/209:15:102
15323991712,3cyclictest7675-21/usr/sbin/munin11:45:142
15323991712,3cyclictest20605-21/usr/sbin/munin07:20:172
1532399171,15cyclictest0-21swapper/211:56:482
1532399171,15cyclictest0-21swapper/211:36:502
15323991711,3cyclictest13973-21cat10:45:122
15323991711,3cyclictest10846-21acpi09:25:102
1532399171,10cyclictest0-21swapper/208:19:562
1532399170,3cyclictest28218-21/usr/sbin/munin07:35:192
1532399170,10cyclictest0-21swapper/210:25:192
1531199171,9cyclictest0-21swapper/008:51:570
1531199171,9cyclictest0-21swapper/008:51:570
1531199171,8cyclictest0-21swapper/012:05:220
1531199171,7cyclictest22460-21/usr/sbin/munin11:05:230
1531199171,5cyclictest0-21swapper/010:00:590
1531199171,5cyclictest0-21swapper/010:00:590
1531199171,4cyclictest10839-21/usr/sbin/munin09:25:150
15311991712,3cyclictest1792-21fschecks_count07:50:150
15311991711,3cyclictest14620-21/usr/sbin/munin08:20:150
1531199171,11cyclictest0-21swapper/010:22:210
1531199171,10cyclictest0-21swapper/012:18:120
1531199171,10cyclictest0-21swapper/012:18:110
1531199171,10cyclictest0-21swapper/010:41:310
65302160,3sleep00-21swapper/010:26:180
1532399169,4cyclictest28877-21sh08:50:212
1532399169,4cyclictest28877-21sh08:50:212
1532399165,4cyclictest18687-21/usr/sbin/munin12:10:122
1532399164,3cyclictest25697-21cat08:45:112
1532399161,6cyclictest31586-21/usr/sbin/munin07:45:182
1532399161,6cyclictest1-21systemd11:05:122
1532399161,4cyclictest13831-21if_err_enp2s009:30:172
1532399161,2cyclictest0-21swapper/207:51:442
15323991612,3cyclictest0-21swapper/208:10:582
1532399161,14cyclictest0-21swapper/208:44:502
15323991611,3cyclictest3622-21/usr/sbin/munin07:55:262
15323991611,3cyclictest32546-21df09:00:152
15323991611,3cyclictest2679-21gltestperf09:05:162
15323991611,3cyclictest14686-21cpuspeed_turbos08:20:122
1532399161,10cyclictest0-21swapper/211:12:382
15323991610,3cyclictest8976-21missed_timers08:05:182
15323991610,3cyclictest827-21runrttasks12:05:212
1532399160,15cyclictest0-21swapper/210:39:532
1531199167,3cyclictest0-21swapper/007:34:110
1531199164,3cyclictest87350irq/126-enp1s0-09:55:210
1531199161,9cyclictest0-21swapper/009:05:170
1531199161,3cyclictest9255-21/usr/sbin/munin10:35:120
15311991611,3cyclictest1524-21seq09:01:330
15311991610,3cyclictest13064-21gltestperf08:15:160
1532399151,4cyclictest15535-21acpi07:10:132
1532399141,5cyclictest0-21swapper/209:40:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional