You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-31 - 06:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Tue Mar 31, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19412236161,24sleep00-21swapper/019:08:060
324852216164,40sleep20-21swapper/219:05:062
17772204166,25sleep30-21swapper/319:06:003
20042202165,24sleep10-21swapper/119:08:551
2186996563,1cyclictest0-21swapper/320:47:053
2186996158,2cyclictest0-21swapper/323:07:083
2186995653,2cyclictest0-21swapper/322:56:063
2186995450,3cyclictest0-21swapper/300:19:363
250622490,1sleep20-21swapper/222:25:202
2186994642,3cyclictest0-21swapper/323:46:383
217499381,26cyclictest0-21swapper/122:20:151
216699356,13cyclictest5347-21munin-run00:10:010
216699324,13cyclictest31072-21munin-run23:55:010
216699324,13cyclictest11308-21awk23:10:010
218699310,23cyclictest7513-21kworker/3:119:42:513
217499311,14cyclictest0-21swapper/119:21:291
217499311,12cyclictest0-21swapper/100:05:481
217499301,9cyclictest28115-21seq22:33:251
217499291,14cyclictest31126-21ntp_states21:25:211
217499291,12cyclictest0-21swapper/121:24:451
216699294,12cyclictest4082-21munin-run21:40:000
218699280,20cyclictest7513-21kworker/3:119:49:173
218299281,15cyclictest29077-21gltestperf22:35:172
217499281,12cyclictest0-21swapper/123:52:531
217499281,12cyclictest0-21swapper/122:19:051
217499280,11cyclictest0-21swapper/119:10:251
216699280,13cyclictest31430-21idleruntime-cro20:15:010
218699271,18cyclictest7513-21kworker/3:123:19:233
2182992719,5cyclictest31450-21grep23:55:162
217499271,13cyclictest18373-21taskset00:38:191
217499271,11cyclictest24893-21cat23:40:131
217499271,10cyclictest0-21swapper/121:19:151
217499271,10cyclictest0-21swapper/120:17:491
2174992710,9cyclictest269152sleep120:03:491
216699275,14cyclictest15980-21runrttasks00:30:470
216699272,13cyclictest22616-21munin-run19:55:010
216699272,13cyclictest22616-21munin-run19:55:010
217499261,9cyclictest0-21swapper/123:13:331
217499261,12cyclictest0-21swapper/121:45:291
217499261,12cyclictest0-21swapper/100:22:231
217499261,11cyclictest0-21swapper/123:58:011
217499260,9cyclictest0-21swapper/120:41:311
216699268,15cyclictest7869-21idleruntime-cro20:35:020
248732250,4sleep232-21ksoftirqd/223:40:132
218699252,9cyclictest7513-21kworker/3:121:04:273
218699252,16cyclictest7513-21kworker/3:123:38:433
218299253,6cyclictest25771-1kworker/u9:000:20:252
217499258,3cyclictest0-21swapper/120:51:451
217499251,9cyclictest0-21swapper/121:09:151
217499251,9cyclictest0-21swapper/100:15:151
217499251,11cyclictest0-21swapper/121:32:071
217499251,10cyclictest0-21swapper/121:54:531
217499251,10cyclictest0-21swapper/121:39:131
217499250,9cyclictest0-21swapper/119:33:061
217499250,11cyclictest0-21swapper/122:58:251
217499250,10cyclictest0-21swapper/120:30:151
217499250,10cyclictest0-21swapper/100:02:411
216699257,15cyclictest18535-21perf00:40:010
218699242,6cyclictest7513-21kworker/3:120:25:233
217499248,3cyclictest0-21swapper/123:28:111
217499241,9cyclictest0-21swapper/120:05:331
217499241,15cyclictest0-21swapper/122:45:291
217499241,12cyclictest0-21swapper/120:27:291
217499241,11cyclictest0-21swapper/122:01:451
217499241,11cyclictest0-21swapper/121:02:131
217499241,10cyclictest0-21swapper/123:20:291
217499241,10cyclictest0-21swapper/123:20:291
217499241,10cyclictest0-21swapper/123:04:091
217499241,10cyclictest0-21swapper/122:10:411
217499241,10cyclictest0-21swapper/122:10:401
217499240,8cyclictest0-21swapper/100:33:511
217499240,11cyclictest0-21swapper/119:36:491
2186992316,4cyclictest10612-21df20:40:143
218699231,10cyclictest7513-21kworker/3:119:19:233
217499238,3cyclictest23124-21gltestperf23:35:171
217499237,8cyclictest0-21swapper/119:29:421
217499231,9cyclictest0-21swapper/123:46:271
217499231,9cyclictest0-21swapper/123:34:251
217499230,8cyclictest0-21swapper/121:10:311
217499230,7cyclictest0-21swapper/120:46:111
218699228,3cyclictest7513-21kworker/3:120:20:293
218699222,9cyclictest7513-21kworker/3:121:10:043
218699222,9cyclictest7513-21kworker/3:100:09:013
218699222,8cyclictest7513-21kworker/3:120:09:233
218699222,10cyclictest7513-21kworker/3:120:30:033
218699221,6cyclictest0-21swapper/321:50:573
218699221,13cyclictest812-21snmpd23:00:383
2182992216,3cyclictest4873-21grep00:05:292
218299221,5cyclictest0-21swapper/222:20:302
217499227,3cyclictest6311-21sed21:45:011
217499227,3cyclictest0-21swapper/123:07:371
217499226,3cyclictest0-21swapper/122:50:331
217499226,3cyclictest0-21swapper/100:10:511
217499225,3cyclictest0-21swapper/120:36:411
217499225,3cyclictest0-21swapper/119:46:171
217499221,9cyclictest0-21swapper/122:27:291
217499221,15cyclictest0-21swapper/119:55:381
217499221,13cyclictest0-21swapper/100:26:491
217499221,12cyclictest0-21swapper/122:09:211
217499221,12cyclictest0-21swapper/120:22:011
217499221,11cyclictest0-21swapper/120:55:211
217499220,7cyclictest0-21swapper/121:59:511
217499220,11cyclictest0-21swapper/119:51:331
217499220,11cyclictest0-21swapper/119:51:331
216699227,13cyclictest7718-21grep23:00:150
216699225,15cyclictest15697-21awk23:20:010
2166992212,3cyclictest0-21swapper/022:30:530
2166992212,3cyclictest0-21swapper/022:05:360
279252211,3sleep027962-21ntp_states23:45:210
218699218,2cyclictest7513-21kworker/3:121:59:453
218699218,2cyclictest7513-21kworker/3:121:17:253
218699216,2cyclictest7513-21kworker/3:119:23:513
218699213,5cyclictest7513-21kworker/3:122:11:513
218699213,5cyclictest7513-21kworker/3:122:11:513
218699212,3cyclictest0-21swapper/323:59:203
218699212,11cyclictest7513-21kworker/3:119:27:593
218699211,8cyclictest7513-21kworker/3:122:17:473
218699211,6cyclictest0-21swapper/321:44:413
218699211,11cyclictest7513-21kworker/3:120:10:113
2186992110,2cyclictest7513-21kworker/3:122:25:393
218299211,7cyclictest32553-21memory20:15:182
218299211,6cyclictest0-21swapper/222:05:362
218299211,5cyclictest0-21swapper/222:30:532
218299211,5cyclictest0-21swapper/200:33:332
217499215,3cyclictest0-21swapper/122:36:131
217499215,2cyclictest0-21swapper/120:11:131
217499211,8cyclictest0-21swapper/119:17:391
217499211,13cyclictest0-21swapper/122:41:091
217499210,15cyclictest0-21swapper/119:40:251
216699217,5cyclictest0-21swapper/021:22:300
216699215,6cyclictest0-21swapper/022:43:220
216699215,6cyclictest0-21swapper/021:45:460
216699212,16cyclictest528-21grep22:45:120
2166992112,2cyclictest0-21swapper/022:20:300
216699210,9cyclictest0-21swapper/023:23:350
216699210,9cyclictest0-21swapper/023:23:350
52542201,2sleep20-21swapper/222:55:152
24582201,3sleep30-21swapper/322:49:423
218699209,3cyclictest3167-21ntp_states21:35:223
218699209,2cyclictest7513-21kworker/3:123:54:073
218699208,2cyclictest5330-21kworker/3:000:24:513
218699207,3cyclictest7513-21kworker/3:121:29:133
218699207,2cyclictest7513-21kworker/3:120:36:233
218699207,2cyclictest7513-21kworker/3:120:15:293
218699207,2cyclictest7513-21kworker/3:119:56:453
218699202,8cyclictest7513-21kworker/3:122:05:213
218699202,3cyclictest15758-40dnf19:36:493
218699201,9cyclictest7513-21kworker/3:122:35:473
218699201,8cyclictest7513-21kworker/3:123:21:293
218699201,8cyclictest7513-21kworker/3:123:21:293
218699201,11cyclictest26286-21/usr/sbin/munin22:30:273
218699201,10cyclictest7513-21kworker/3:122:52:393
218699201,10cyclictest5330-21kworker/3:000:13:553
218299203,3cyclictest21888-21users23:30:272
218299201,7cyclictest0-21swapper/219:31:182
218299201,4cyclictest0-21swapper/221:49:552
217499201,10cyclictest0-21swapper/123:15:251
216699205,6cyclictest0-21swapper/021:43:150
216699204,3cyclictest3932-21grep19:10:250
216699201,9cyclictest29751-21kworker/u8:322:10:230
216699201,9cyclictest29751-21kworker/u8:322:10:230
2166992013,4cyclictest13047-21unixbench_multi23:10:270
216699201,13cyclictest29078-21gltestperf22:35:170
216699201,11cyclictest30063-21/usr/sbin/munin21:25:250
216699201,10cyclictest0-21swapper/022:50:040
201002201,3sleep220130-21ntpq21:00:212
81112190,3sleep011-21rcuc/000:15:140
56732191,2sleep20-21swapper/221:40:262
29342191,3sleep20-21swapper/200:02:412
254242191,3sleep00-21swapper/021:15:000
222432191,2sleep00-21swapper/023:34:500
218699197,3cyclictest7513-21kworker/3:120:05:013
218699196,2cyclictest7513-21kworker/3:121:32:153
218699194,3cyclictest86550irq/126-enp1s0-00:30:193
218699194,2cyclictest7513-21kworker/3:122:23:513
218699191,9cyclictest7513-21kworker/3:123:25:373
218699191,9cyclictest7513-21kworker/3:119:32:413
218699191,9cyclictest5330-21kworker/3:000:28:173
218699191,9cyclictest0-21swapper/320:51:283
218699191,4cyclictest0-21swapper/323:11:213
218699191,4cyclictest0-21swapper/300:35:323
218699190,9cyclictest7513-21kworker/3:121:20:133
218699190,8cyclictest7513-21kworker/3:100:01:253
218299194,6cyclictest13810-21cpuspeed_turbos23:15:122
218299194,5cyclictest2129-21kworker/u8:022:50:192
218299194,4cyclictest7108-21unixbench_singl00:10:262
218299192,5cyclictest14132-21munin-run00:30:012
218299191,8cyclictest9738-21perf00:20:012
218299191,8cyclictest26014-21diskstats21:15:152
218299191,7cyclictest29215-21crond20:10:012
218299191,7cyclictest0-21swapper/222:41:552
218299191,5cyclictest0-21swapper/220:25:182
2182991914,3cyclictest17431-21wc00:35:202
2182991912,4cyclictest19554-21/usr/sbin/munin19:45:232
216699195,4cyclictest0-21swapper/021:51:410
216699191,5cyclictest0-21swapper/021:16:420
216699191,10cyclictest0-21swapper/021:56:330
216699191,10cyclictest0-21swapper/021:32:530
112072191,3sleep20-21swapper/223:09:072
264672181,2sleep00-21swapper/023:42:310
218699187,2cyclictest7513-21kworker/3:119:52:473
218699187,2cyclictest7513-21kworker/3:119:52:473
218699185,4cyclictest23240-21diskmemload23:30:233
218699185,3cyclictest7513-21kworker/3:122:02:293
218699185,2cyclictest7513-21kworker/3:121:45:193
218699181,8cyclictest7513-21kworker/3:119:11:483
218299186,3cyclictest86850irq/129-enp1s0-21:30:212
218299185,4cyclictest17758-21expr22:10:152
218299185,4cyclictest17758-21expr22:10:152
218299185,4cyclictest1256-21if_enp2s022:45:172
218299184,6cyclictest17415-21tail23:20:232
218299184,6cyclictest17415-21tail23:20:232
218299183,6cyclictest86850irq/129-enp1s0-23:50:162
218299181,7cyclictest7139-21sensors_temp20:30:242
218299181,7cyclictest23739-21sendmail_mailst23:35:242
218299181,7cyclictest14568-21grep22:00:262
218299181,7cyclictest0-21swapper/223:26:102
218299181,7cyclictest0-21swapper/221:20:192
218299181,6cyclictest9686-21needreboot21:50:192
218299181,5cyclictest0-21swapper/219:36:492
2182991814,3cyclictest0-21swapper/219:11:272
218299181,2cyclictest0-21swapper/221:57:212
218299181,10cyclictest0-21swapper/220:01:462
2182991810,5cyclictest0-21swapper/220:20:242
216699187,8cyclictest4316-21gltestperf20:25:170
216699187,3cyclictest31641-21/usr/sbin/munin20:15:230
216699185,3cyclictest22260-21needreboot21:05:210
216699182,8cyclictest1118-21/usr/sbin/munin00:00:170
216699181,6cyclictest0-21swapper/023:39:270
216699181,5cyclictest7244-21perf19:20:010
216699181,3cyclictest0-21swapper/000:14:340
2166991812,3cyclictest9993-21df19:25:140
2166991811,4cyclictest14325-21fgrep22:00:240
216699181,14cyclictest0-21swapper/022:27:010
216699181,10cyclictest23240-21diskmemload00:29:340
216699181,10cyclictest0-21swapper/022:15:180
216699181,10cyclictest0-21swapper/020:00:130
216699180,8cyclictest0-21swapper/023:27:390
115842181,2sleep00-21swapper/000:20:450
225892170,1sleep20-21swapper/221:05:262
218699173,3cyclictest86850irq/129-enp1s0-21:05:433
218699173,3cyclictest7513-21kworker/3:120:56:173
218699171,6cyclictest30822-21grep22:40:133
218699171,15cyclictest0-21swapper/323:40:413
218299174,5cyclictest27820-21missed_timers23:45:192
218299171,9cyclictest0-21swapper/219:51:022
218299171,9cyclictest0-21swapper/219:51:022
218299171,9cyclictest0-21swapper/219:24:142
218299171,8cyclictest0-21swapper/221:35:192
218299171,7cyclictest29467-21cpu20:10:112
218299171,7cyclictest0-21swapper/221:14:292
218299171,7cyclictest0-21swapper/219:40:232
218299171,6cyclictest20499-21irqstats22:15:192
218299171,6cyclictest0-21swapper/223:00:012
218299171,3cyclictest0-21swapper/219:28:002
2182991712,3cyclictest23185-21df_inode19:55:152
2182991711,4cyclictest4962-21grep19:15:042
218299171,14cyclictest0-21swapper/220:35:442
218299171,10cyclictest0-21swapper/220:57:542
2182991710,4cyclictest12098-21perl23:10:172
216699178,2cyclictest32351-21taskset23:55:240
216699177,5cyclictest0-21swapper/022:57:330
216699177,2cyclictest15040-21strings19:35:190
216699172,12cyclictest10279-21/usr/sbin/munin20:40:150
216699171,4cyclictest13981-21unixbench_multi20:45:250
2166991712,3cyclictest8452-21needreboot19:20:180
2166991712,3cyclictest17612-21cat19:40:250
2166991712,3cyclictest17200-21df20:55:130
2166991711,3cyclictest812-21snmpd19:55:020
2166991711,3cyclictest27246-21/usr/sbin/munin20:05:250
2166991711,3cyclictest20015-21sed19:45:270
218299162,5cyclictest11088-21irqstats20:40:182
2182991611,3cyclictest30063-21/usr/sbin/munin21:25:232
2182991611,3cyclictest12472-21/usr/sbin/munin20:45:132
216699165,3cyclictest86750irq/128-enp1s0-19:30:190
216699161,5cyclictest8122-21cpu20:35:130
2166991611,3cyclictest16636-21latency_hist20:55:010
216699161,13cyclictest0-21swapper/020:22:250
216699160,5cyclictest0-21swapper/021:00:120
218299151,5cyclictest0-21swapper/220:50:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional