You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-21 - 02:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Mon Sep 21, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
310392239164,24sleep20-21swapper/219:08:162
310252233167,54sleep30-21swapper/319:08:053
311582217146,59sleep00-21swapper/019:09:460
311542204168,24sleep10-21swapper/119:09:441
312709910298,2cyclictest19963-21diskmemload21:54:182
31270999087,2cyclictest0-21swapper/221:24:222
31270998783,3cyclictest0-21swapper/200:39:282
218802750,4sleep03126099cyclictest22:25:230
31270997471,2cyclictest0-21swapper/222:06:592
31270995854,3cyclictest0-21swapper/200:14:442
51062500,1sleep20-21swapper/220:35:112
75772480,4sleep341-21ksoftirqd/323:05:273
31270994340,2cyclictest0-21swapper/219:11:382
246382400,1sleep00-21swapper/023:45:210
3127599382,10cyclictest19972-21munin-run21:10:013
31275993813,3cyclictest3840-21/usr/sbin/munin23:00:223
3127599362,7cyclictest827-21runrttasks21:03:233
3127599351,2cyclictest0-21swapper/323:55:153
3126099357,13cyclictest26553-21idleruntime-cro21:25:020
150242350,2sleep339-21rcuc/323:25:133
3127599346,3cyclictest1060-21perf19:15:013
31270993431,2cyclictest0-21swapper/223:01:402
76302330,1sleep00-21swapper/019:25:220
3127599334,8cyclictest87350irq/126-enp1s0-23:10:163
3127599334,8cyclictest87350irq/126-enp1s0-23:10:163
3127599333,8cyclictest1253-21processes00:05:223
3127599330,5cyclictest0-21swapper/300:37:593
3127599321,5cyclictest11201-21tr00:30:123
3127599321,16cyclictest0-21swapper/320:30:203
3127599321,10cyclictest0-21swapper/300:18:193
3127599321,10cyclictest0-21swapper/300:18:193
3127599317,3cyclictest8589-21/usr/sbin/munin19:30:243
3127599315,3cyclictest0-21swapper/322:02:093
3127599313,10cyclictest9193-21/usr/sbin/munin20:45:153
3127599312,5cyclictest21768-21diskstats23:40:143
3127599311,11cyclictest0-21swapper/322:54:183
31275993110,3cyclictest28368-21/usr/sbin/munin20:15:243
3126099311,23cyclictest0-21swapper/021:46:570
3127599301,6cyclictest0-21swapper/322:31:543
3127599301,11cyclictest0-21swapper/322:42:283
3127599294,10cyclictest28702-21kworker/u8:222:35:253
31270992916,7cyclictest1235-21processes00:05:232
3126599292,7cyclictest1113-40dnf00:05:241
31265992914,14cyclictest28016-21kworker/u8:019:12:501
3127599281,9cyclictest19871-21gltestperf23:35:183
31275992811,3cyclictest11890-21ls19:35:243
31275992811,3cyclictest11890-21ls19:35:243
3126099289,16cyclictest27171-21munin-run22:40:020
3126099271,13cyclictest18389-21munin-run22:20:010
269982270,4sleep230-21rcuc/221:25:132
3127599266,3cyclictest0-21swapper/322:49:243
3127599261,8cyclictest0-21swapper/322:59:223
3127099265,9cyclictest17626-40dnf21:03:222
3126599261,18cyclictest0-21swapper/100:19:231
3126599261,18cyclictest0-21swapper/100:19:231
3126099268,15cyclictest32207-21munin-run00:05:010
3127599259,3cyclictest0-21swapper/319:40:093
3127599251,10cyclictest0-21swapper/322:18:013
3127599250,9cyclictest28016-21kworker/u8:021:22:483
3126599255,8cyclictest0-21swapper/121:45:171
31265992512,6cyclictest29724-21kworker/u8:200:31:251
31265992512,6cyclictest15454-21kworker/u8:121:08:261
3126599251,14cyclictest0-21swapper/121:39:371
3126099252,11cyclictest8667-21basename00:25:000
3127599249,3cyclictest20029-21fschecks_count19:55:163
3127099240,13cyclictest0-21swapper/222:52:452
3126599247,8cyclictest2775-21pmu-power22:55:221
3126599247,15cyclictest0-21swapper/121:10:251
3126599247,14cyclictest4114-21systemd-journal00:13:211
3126599244,2cyclictest0-21swapper/123:20:141
3126599242,12cyclictest4541-21diskstats19:20:151
3126599241,17cyclictest0-21swapper/123:36:001
3126599241,16cyclictest0-21swapper/123:40:341
3126599241,16cyclictest0-21swapper/123:17:111
15422240,4sleep339-21rcuc/319:15:023
11492240,4sleep121-21rcuc/121:40:131
32112230,4sleep23127099cyclictest22:55:262
3127099236,7cyclictest0-21swapper/200:18:392
3127099236,7cyclictest0-21swapper/200:18:392
3126599239,12cyclictest28016-21kworker/u8:020:00:311
31265992315,3cyclictest0-21swapper/122:34:251
31265992311,3cyclictest8048-21timerandwakeup00:20:231
3127599227,3cyclictest19003-21sleep23:35:013
3127599222,11cyclictest28984-21/usr/sbin/munin21:30:213
3126599228,4cyclictest16517-21cpuspeed_turbos22:15:131
3126599226,3cyclictest0-21swapper/119:46:281
3126599225,3cyclictest88950irq/132-enp2s0-22:10:001
3126599221,5cyclictest0-21swapper/122:51:391
3126599221,16cyclictest27081-21chrt22:39:031
31265992211,3cyclictest24197-21perl23:45:161
3126599221,11cyclictest0-21swapper/123:02:541
3126599221,11cyclictest0-21swapper/122:29:091
3126599221,11cyclictest0-21swapper/121:27:121
3126599221,10cyclictest0-21swapper/122:43:131
3126099226,8cyclictest12740-21crond19:40:000
3126099226,8cyclictest12740-21crond19:40:000
3126099221,5cyclictest0-21swapper/000:05:230
303342221,4sleep230408-21irqstats22:45:202
281412220,6sleep23127099cyclictest20:15:022
271482221,17sleep027396-21gltestperf21:25:160
82492210,4sleep011-21rcuc/020:40:240
3127599219,4cyclictest32227-21ntpq21:35:203
3127599215,3cyclictest0-21swapper/322:21:503
3127599215,3cyclictest0-21swapper/319:27:293
3127599215,3cyclictest0-21swapper/319:24:413
3127599211,9cyclictest0-21swapper/300:02:513
3127599211,13cyclictest0-21swapper/320:27:433
3127099211,6cyclictest5599-21timerwakeupswit19:20:262
31270992111,3cyclictest5187-21latency_hist21:50:022
3127099211,11cyclictest28930-21missed_timers23:55:192
3126599219,4cyclictest827-21runrttasks19:40:011
3126599219,4cyclictest827-21runrttasks19:40:001
3126599216,8cyclictest0-21swapper/100:00:241
3126599212,17cyclictest221ktimersoftd/123:55:241
3126599212,14cyclictest0-21swapper/121:51:311
3126599211,3cyclictest2572-21tune2fs19:15:161
3126599211,18cyclictest0-21swapper/123:50:181
3126599211,14cyclictest0-21swapper/123:14:291
3126599211,14cyclictest0-21swapper/123:14:291
3126599211,14cyclictest0-21swapper/121:31:071
3126599211,14cyclictest0-21swapper/120:50:331
3126599211,14cyclictest0-21swapper/120:48:451
3126099216,2cyclictest11-21rcuc/022:05:150
3126099211,6cyclictest0-21swapper/023:53:000
31260992111,3cyclictest87550irq/128-enp1s0-23:05:310
3127599206,3cyclictest0-21swapper/319:48:213
3127599205,4cyclictest0-21swapper/323:19:303
3127599205,3cyclictest0-21swapper/320:38:133
3127599204,6cyclictest0-21swapper/300:15:003
3127599204,3cyclictest0-21swapper/320:20:333
3127599203,9cyclictest133450kworker/u8:100:26:583
3127599203,3cyclictest19963-21diskmemload21:48:223
3127599203,3cyclictest0-21swapper/321:27:173
3127599201,8cyclictest773-21NetworkManager21:44:063
3127099202,5cyclictest12978-21diskstats23:20:132
3127099201,8cyclictest30337-21sh20:20:012
3127099201,16cyclictest18571-21fschecks_time21:05:162
3127099201,10cyclictest8869-21gltestperf23:10:152
3127099201,10cyclictest8869-21gltestperf23:10:152
3126599205,3cyclictest89050irq/133-enp2s0-21:03:181
3126599204,5cyclictest755-21rngd21:21:151
3126599202,3cyclictest0-21swapper/100:28:431
3126599202,16cyclictest0-21swapper/120:11:141
3126599202,15cyclictest14557-21df_inode22:10:141
31265992016,3cyclictest0-21swapper/120:36:271
3126599201,2cyclictest0-21swapper/121:15:461
3126599201,17cyclictest0-21swapper/122:45:271
3126599201,17cyclictest0-21swapper/120:55:041
3126599201,14cyclictest0-21swapper/122:01:391
3126599201,11cyclictest2428-21cat20:30:001
3126599201,10cyclictest8134-21cat23:10:011
3126599200,2cyclictest87550irq/128-enp1s0-00:37:321
3126599200,18cyclictest0-21swapper/120:16:251
3126099209,3cyclictest14216-21/usr/sbin/munin22:10:180
3126099206,5cyclictest827-21runrttasks22:01:120
3126099203,8cyclictest2982-21timerandwakeup22:55:250
3126099202,10cyclictest15173-21/usr/sbin/munin19:45:180
3126099201,7cyclictest21108-21sshd21:10:200
3126099201,6cyclictest0-21swapper/023:33:020
31260992015,3cyclictest5661-21grep19:20:260
3126099201,16cyclictest20146-21perl19:55:160
3126099201,16cyclictest12119-21ntp_kernel_pll_00:30:200
3126099201,13cyclictest0-21swapper/023:01:520
3126099201,11cyclictest2289-21/usr/sbin/munin00:10:250
38592191,2sleep00-21swapper/019:18:500
3127599195,3cyclictest0-21swapper/322:29:103
3127599194,3cyclictest89250irq/135-enp2s0-00:20:183
3127599193,3cyclictest0-21swapper/320:50:533
3127599193,3cyclictest0-21swapper/320:11:093
3127599193,3cyclictest0-21swapper/320:09:253
3127599193,3cyclictest0-21swapper/319:51:293
3127599192,15cyclictest0-21swapper/321:53:383
31275991913,3cyclictest8529-21memory21:55:203
3127599191,12cyclictest0-21swapper/323:47:313
3127599190,10cyclictest0-21swapper/321:12:003
3127099196,4cyclictest1435-21awk20:25:212
3127099193,5cyclictest9582-21needreboot19:30:222
3127099192,7cyclictest26267-21ntp_states22:35:232
3127099192,3cyclictest0-21swapper/219:27:492
3127099191,8cyclictest27740-21diskstats22:40:132
3127099191,11cyclictest0-21swapper/221:43:312
3126599198,3cyclictest7962-21diskstats21:55:151
3126599192,3cyclictest0-21swapper/119:53:381
3126599192,16cyclictest0-21swapper/120:33:311
3126599192,16cyclictest0-21swapper/119:56:381
3126599192,10cyclictest0-21swapper/123:34:451
31265991915,3cyclictest0-21swapper/119:40:091
3126599191,16cyclictest0-21swapper/120:07:101
3126599191,13cyclictest0-21swapper/122:23:271
3126099194,9cyclictest755-21rngd20:12:430
3126099192,1cyclictest0-21swapper/021:55:170
3126099192,10cyclictest18608-21/usr/sbin/munin22:20:230
3126099191,9cyclictest808-21snmpd23:36:160
3126099191,7cyclictest0-21swapper/019:10:210
3126099191,4cyclictest0-21swapper/023:23:070
3126099191,3cyclictest0-21swapper/023:56:340
3126099191,10cyclictest31836-21smart_sda20:20:250
3126099191,10cyclictest29598-21/usr/sbin/munin22:45:210
3126099191,10cyclictest16696-21memory21:00:170
3126099190,5cyclictest0-21swapper/020:30:240
3127599185,4cyclictest0-21swapper/321:16:583
3127599183,3cyclictest0-21swapper/320:42:013
3127599183,3cyclictest0-21swapper/320:02:333
3127599182,2cyclictest89250irq/135-enp2s0-23:22:303
3127599181,6cyclictest0-21swapper/322:13:033
3127599181,5cyclictest12969-21grep22:05:203
3127099182,7cyclictest17370-21/usr/sbin/munin19:50:212
3127099182,13cyclictest2650-21gltestperf19:15:152
3127099181,6cyclictest0-21swapper/223:25:122
3127099181,6cyclictest0-21swapper/222:02:402
3127099181,5cyclictest12872-21runrttasks00:33:012
3127099181,5cyclictest0-21swapper/223:50:352
3127099181,10cyclictest0-21swapper/221:58:452
3126599185,11cyclictest28016-21kworker/u8:020:20:381
3126599182,7cyclictest16086-21/usr/sbin/munin23:25:241
3126599181,9cyclictest7105-21cpuspeed_turbos20:40:131
3126599181,9cyclictest6968-21perl19:25:161
3126599180,16cyclictest0-21swapper/119:30:031
3126099188,3cyclictest13589-21/usr/sbin/munin20:55:190
3126099184,2cyclictest15172-21kworker/0:120:02:230
3126099182,15cyclictest9-21rcu_preempt20:15:210
3126099181,9cyclictest808-21snmpd21:40:520
3126099181,8cyclictest6088-21sed21:50:180
3126099181,8cyclictest0-21swapper/022:30:150
3126099181,6cyclictest0-21swapper/000:27:220
3126099181,5cyclictest0-21swapper/000:18:270
3126099181,5cyclictest0-21swapper/000:18:270
3126099181,4cyclictest0-21swapper/000:35:450
31260991813,3cyclictest8658-21grep23:10:150
31260991813,3cyclictest8658-21grep23:10:150
3126099181,15cyclictest16797-21munin-plugin-st23:30:010
3126099181,15cyclictest0-21swapper/019:50:250
31260991811,4cyclictest31181-21/usr/sbin/munin21:35:140
130362180,2sleep00-21swapper/020:51:000
73842171,12sleep20-21swapper/223:05:262
3127599177,3cyclictest839-21lldpd20:55:563
31275991711,3cyclictest25801-21seq23:50:013
3127099176,3cyclictest31968-21cat21:35:182
3127099173,6cyclictest0-21swapper/220:30:222
3127099172,5cyclictest14654-21ntp_states20:55:222
3127099171,8cyclictest0-21swapper/222:25:292
3127099171,7cyclictest0-21swapper/222:30:142
3127099171,7cyclictest0-21swapper/221:10:062
3127099171,6cyclictest23588-21cat23:45:022
3127099171,6cyclictest0-21swapper/223:32:592
3127099171,6cyclictest0-21swapper/222:20:242
3127099171,6cyclictest0-21swapper/221:16:562
3127099171,6cyclictest0-21swapper/219:35:212
3127099171,6cyclictest0-21swapper/219:35:212
3127099171,4cyclictest0-21swapper/222:15:152
3127099171,3cyclictest0-21swapper/223:16:362
31270991710,4cyclictest30798-21/usr/sbin/munin00:00:162
3127099170,15cyclictest0-21swapper/220:08:212
3126099178,3cyclictest0-21swapper/021:05:160
3126099171,6cyclictest0-21swapper/022:50:050
3126099171,6cyclictest0-21swapper/022:44:550
3126099171,6cyclictest0-21swapper/021:31:090
3126099171,4cyclictest0-21swapper/023:16:250
31260991712,3cyclictest13787-21irqstats19:40:170
3126099171,10cyclictest0-21swapper/021:16:290
316812161,2sleep20-21swapper/220:20:232
3127099167,4cyclictest763-21dbus-daemon20:00:012
3127099165,3cyclictest87350irq/126-enp1s0-00:20:112
3127099163,4cyclictest23617-21/usr/sbin/munin23:45:172
3127099161,6cyclictest0-21swapper/222:10:282
3127099161,5cyclictest9952-21if_enp2s020:45:162
3127099161,5cyclictest0-21swapper/200:26:342
3127099161,3cyclictest0-21swapper/219:49:412
3127099161,12cyclictest7808-21irqstats20:40:172
31270991610,3cyclictest11861-21fschecks_time20:50:152
3126099161,3cyclictest0-21swapper/019:33:480
31260991612,3cyclictest0-21swapper/020:28:060
31260991612,2cyclictest23975-21/usr/sbin/munin20:05:230
3126099161,14cyclictest0-21swapper/020:45:510
31260991611,3cyclictest5598-21cat20:35:170
3126099160,9cyclictest0-21swapper/023:40:240
296892161,2sleep20-21swapper/221:30:192
3127099151,6cyclictest0-21swapper/223:35:012
31270991511,2cyclictest12974-21/usr/sbin/munin19:40:242
3127099141,6cyclictest0-21swapper/220:00:032
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional