You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-08 - 04:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Mon Jun 08, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1877722820,2sleep30-21swapper/319:10:013
186302239163,24sleep20-21swapper/219:08:092
185012235162,60sleep00-21swapper/019:06:290
186642232163,56sleep10-21swapper/119:08:351
102302460,2sleep20-21swapper/221:15:132
70022410,1sleep00-21swapper/021:05:240
1886499341,12cyclictest0-21swapper/123:47:501
1886499331,20cyclictest0-21swapper/100:17:491
1886499321,13cyclictest0-21swapper/100:01:511
1886499311,22cyclictest0-21swapper/123:36:451
1886499301,22cyclictest0-21swapper/123:03:171
1886499301,21cyclictest0-21swapper/123:41:261
1886499301,12cyclictest0-21swapper/119:38:101
1886499301,10cyclictest0-21swapper/122:03:411
1886499291,11cyclictest0-21swapper/121:29:411
1886899281,2cyclictest0-21swapper/223:48:572
1886899281,12cyclictest5862-21taskset22:18:332
1886499281,17cyclictest0-21swapper/121:12:251
1886499281,17cyclictest0-21swapper/120:11:491
1886499281,12cyclictest15683-21hddtemp_smartct22:40:171
1886499281,10cyclictest0-21swapper/120:15:291
1886499280,6cyclictest0-21swapper/122:53:291
1885699280,13cyclictest29553-21munin-run22:00:010
231662270,2sleep339-21rcuc/320:30:153
202992270,3sleep07-21ksoftirqd/021:35:260
1886899271,10cyclictest0-21swapper/220:18:412
1886499271,9cyclictest0-21swapper/120:42:171
1886499271,16cyclictest0-21swapper/100:34:131
1886499271,11cyclictest0-21swapper/120:23:251
1886499271,10cyclictest15827-21cstates23:55:131
1886499270,12cyclictest0-21swapper/122:48:491
1886499270,12cyclictest0-21swapper/122:29:091
1886499270,12cyclictest0-21swapper/121:43:221
1886499270,11cyclictest0-21swapper/123:51:411
1886499270,11cyclictest0-21swapper/122:20:131
1886499270,10cyclictest0-21swapper/123:33:171
1886899261,10cyclictest0-21swapper/219:28:452
1886499267,2cyclictest0-21swapper/100:10:131
1886499261,9cyclictest0-21swapper/121:50:491
1886499261,9cyclictest0-21swapper/120:59:211
1886499261,9cyclictest0-21swapper/120:59:211
1886499261,11cyclictest0-21swapper/121:08:091
1886499261,11cyclictest0-21swapper/119:12:051
1886499261,10cyclictest0-21swapper/121:34:491
1886499261,10cyclictest0-21swapper/121:23:131
1886499260,4cyclictest0-21swapper/119:30:161
1886499260,20cyclictest0-21swapper/119:43:311
1886499260,17cyclictest0-21swapper/122:08:251
1886499260,11cyclictest0-21swapper/123:18:211
1886499260,11cyclictest0-21swapper/123:18:211
1886499260,11cyclictest0-21swapper/120:25:091
1886499260,10cyclictest0-21swapper/121:19:491
1885699262,11cyclictest3172-21awk21:00:010
1885699262,11cyclictest3172-21awk21:00:010
1885699261,12cyclictest24239-21idleruntime-cro00:15:010
205222250,1sleep20-21swapper/219:10:242
1887499251,6cyclictest0-21swapper/323:25:503
1886499258,3cyclictest7725-21perl19:55:161
1886499256,3cyclictest0-21swapper/120:53:011
1886499251,9cyclictest0-21swapper/122:35:011
1886499251,9cyclictest0-21swapper/122:16:531
1886499251,9cyclictest0-21swapper/119:24:451
1886499251,16cyclictest26353-21df_inode23:05:151
1886499251,11cyclictest0-21swapper/120:34:251
1886499251,10cyclictest0-21swapper/119:27:161
1886499250,9cyclictest0-21swapper/123:11:211
1886499250,9cyclictest0-21swapper/121:56:011
1886499250,9cyclictest0-21swapper/100:06:251
1885699251,12cyclictest6955-21awk19:55:010
1887499246,7cyclictest23803-21/usr/sbin/munin23:00:233
18874992411,5cyclictest87350irq/126-enp1s0-22:10:193
1887499241,12cyclictest0-21swapper/322:49:163
1886899241,6cyclictest0-21swapper/221:53:252
1886899240,10cyclictest0-21swapper/221:37:572
1886899240,10cyclictest0-21swapper/220:00:292
1886499241,9cyclictest3039-21gltestperf23:25:151
1886499241,9cyclictest0-21swapper/122:40:011
1886499241,9cyclictest0-21swapper/122:11:271
1886499241,9cyclictest0-21swapper/121:00:351
1886499241,11cyclictest0-21swapper/120:01:141
1886499241,10cyclictest0-21swapper/119:50:261
1886499240,9cyclictest0-21swapper/122:57:051
1885699247,14cyclictest20157-21perf20:25:010
1885699241,12cyclictest15417-21perf23:55:010
18874992313,2cyclictest87350irq/126-enp1s0-22:30:193
1887499231,15cyclictest0-21swapper/300:09:013
18874992311,3cyclictest87350irq/126-enp1s0-20:50:133
1886899231,6cyclictest0-21swapper/222:41:032
18868992316,4cyclictest3498-21if_enp1s019:45:162
18868992310,4cyclictest0-21swapper/221:10:012
1886499231,7cyclictest0-21swapper/119:48:581
1886499231,16cyclictest88950irq/132-enp2s0-00:25:271
1886499231,16cyclictest88950irq/132-enp2s0-00:25:271
1886499231,10cyclictest0-21swapper/121:46:581
1886499231,10cyclictest0-21swapper/120:46:151
1886499230,9cyclictest0-21swapper/120:06:291
1886499230,7cyclictest0-21swapper/119:18:251
1885699234,16cyclictest19194-21awk22:50:000
1887499229,3cyclictest12050irq/123-ahci[0022:40:213
1887499224,8cyclictest12050irq/123-ahci[0021:50:013
1887499221,9cyclictest401ktimersoftd/323:45:013
18874992215,4cyclictest27380-21/usr/sbin/munin19:25:213
18874992215,4cyclictest14417-21gltestperf20:10:153
1886899228,3cyclictest0-21swapper/222:26:012
1886899221,5cyclictest0-21swapper/223:19:372
1886899221,5cyclictest0-21swapper/223:19:372
1886499226,3cyclictest0-21swapper/100:20:331
1886499221,11cyclictest20766-21perf21:40:011
1886499221,10cyclictest0-21swapper/100:36:311
1886499220,9cyclictest0-21swapper/120:36:251
1885699223,11cyclictest29153-21/usr/sbin/munin20:45:170
158482220,3sleep29-21rcu_preempt20:15:012
1887499219,3cyclictest15967-21/usr/sbin/munin20:15:233
1887499218,3cyclictest12050irq/123-ahci[0021:10:013
1887499214,9cyclictest31344-21missed_timers23:15:183
1887499214,9cyclictest31344-21missed_timers23:15:183
1887499212,6cyclictest39-21rcuc/300:15:013
1887499211,9cyclictest32751-21tr23:20:133
1887499211,11cyclictest0-21swapper/323:55:233
1886899219,3cyclictest0-21swapper/200:38:332
1886899213,7cyclictest87650irq/129-enp1s0-22:05:402
1886899211,7cyclictest0-21swapper/221:26:502
1886899211,6cyclictest0-21swapper/223:04:052
1886899211,6cyclictest0-21swapper/221:45:282
1886899211,5cyclictest0-21swapper/223:23:022
1886899211,5cyclictest0-21swapper/222:47:102
1886899211,4cyclictest0-21swapper/223:27:312
1886899211,11cyclictest0-21swapper/200:00:342
1886899210,9cyclictest0-21swapper/222:12:222
1886899210,5cyclictest16289-21kworker/u8:121:32:212
1885699217,12cyclictest89150irq/134-enp2s0-20:41:130
1885699211,4cyclictest7-21ksoftirqd/023:11:310
18856992113,2cyclictest0-21swapper/023:25:510
1885699211,11cyclictest0-21swapper/023:40:180
18856992110,10cyclictest0-21swapper/023:50:010
1887499209,3cyclictest17833-21/usr/sbin/munin00:00:153
1887499204,3cyclictest9993-21/usr/sbin/munin21:15:183
1887499203,3cyclictest0-21swapper/319:10:143
1887499203,2cyclictest12806-21unixbench_singl23:45:263
1887499202,10cyclictest4639-21/usr/sbin/munin23:30:243
1887499202,10cyclictest21608-21/usr/sbin/munin22:55:193
1887499202,10cyclictest18163-21/usr/sbin/munin20:20:163
1887499201,7cyclictest0-21swapper/321:24:413
1887499201,6cyclictest1241-21cpuspeed_turbos20:55:123
1887499201,6cyclictest1241-21cpuspeed_turbos20:55:123
18874992013,4cyclictest14662-21fgrep23:50:233
1887499201,12cyclictest0-21swapper/300:28:323
1887499201,12cyclictest0-21swapper/300:28:323
1887499201,10cyclictest7792-21/usr/sbin/munin21:10:133
1887499201,10cyclictest12814-21/usr/sbin/munin22:35:253
1887499200,13cyclictest0-21swapper/300:30:133
1886899206,7cyclictest0-21swapper/221:56:212
1886899202,5cyclictest8564-1kworker/u9:121:11:452
1886899201,5cyclictest0-21swapper/223:43:022
1886899201,5cyclictest0-21swapper/200:27:362
1886899201,5cyclictest0-21swapper/200:27:352
1886899200,9cyclictest0-21swapper/221:40:372
1886899200,9cyclictest0-21swapper/200:10:262
1886899200,8cyclictest0-21swapper/219:30:212
1886899200,10cyclictest0-21swapper/223:05:132
18864992015,3cyclictest2224-21awk23:25:011
1885699206,6cyclictest0-21swapper/022:40:360
1885699206,6cyclictest0-21swapper/021:45:170
1885699206,6cyclictest0-21swapper/000:26:150
1885699206,6cyclictest0-21swapper/000:26:150
1885699204,7cyclictest87550irq/128-enp1s0-23:35:260
1885699204,10cyclictest21710-21if_enp1s021:40:170
1885699202,9cyclictest3688-21kworker/0:222:19:190
1885699201,6cyclictest0-21swapper/023:01:500
1885699201,11cyclictest32598-21/usr/sbin/munin23:20:200
1885699201,11cyclictest16587-21/usr/sbin/munin21:30:240
1885699201,11cyclictest13770-21/usr/sbin/munin20:10:160
1885699201,10cyclictest0-21swapper/021:27:400
1887499197,3cyclictest24146-21/usr/sbin/munin19:20:163
1887499196,4cyclictest9662-21mailstats22:25:243
1887499195,8cyclictest41-21ksoftirqd/323:10:223
1887499192,8cyclictest24451-21/usr/sbin/munin00:15:193
1887499192,7cyclictest87350irq/126-enp1s0-19:55:123
1887499192,6cyclictest22039-40dnf20:26:243
1887499191,4cyclictest0-21swapper/322:05:183
1887499191,13cyclictest0-21swapper/322:23:453
1887499191,11cyclictest0-21swapper/319:45:163
1886899193,9cyclictest0-21swapper/200:08:012
1885699198,4cyclictest11-21rcuc/021:16:200
1885699198,3cyclictest30738-21/usr/sbin/munin19:35:250
1885699196,4cyclictest0-21swapper/022:11:270
1885699196,4cyclictest0-21swapper/021:10:340
1885699196,2cyclictest87350irq/126-enp1s0-22:32:500
1885699195,5cyclictest0-21swapper/022:01:150
1885699194,5cyclictest0-21swapper/020:52:490
1885699191,8cyclictest0-21swapper/000:24:020
18856991911,5cyclictest0-21swapper/000:35:160
1885699191,12cyclictest0-21swapper/023:30:150
1885699191,10cyclictest26345-21/usr/sbin/munin19:25:130
1885699191,10cyclictest24761-21/usr/sbin/munin20:35:140
296082180,1sleep00-21swapper/019:30:220
217352181,2sleep00-21swapper/000:06:370
1887499188,3cyclictest808-21snmpd21:55:583
1887499184,3cyclictest0-21swapper/300:24:533
1887499181,9cyclictest0-21swapper/321:36:203
1887499181,7cyclictest0-21swapper/321:28:333
1887499181,6cyclictest0-21swapper/322:50:303
1887499181,4cyclictest18517-21sshd21:34:393
18874991811,5cyclictest22092-21tr19:15:123
1887499181,15cyclictest0-21swapper/323:37:013
18874991811,4cyclictest1392-21grep00:35:153
1887499181,10cyclictest0-21swapper/322:20:013
1886899184,6cyclictest1246-21kworker/2:223:37:072
1886899183,9cyclictest0-21swapper/222:30:572
1886899182,6cyclictest22337-21munin-run20:30:012
1886899181,9cyclictest0-21swapper/222:20:252
1886899181,6cyclictest0-21swapper/223:51:232
18868991813,3cyclictest28851-21cut23:10:162
18868991812,3cyclictest8197-21ntp_kernel_err19:55:202
1885699187,5cyclictest0-21swapper/000:33:540
1885699186,3cyclictest0-21swapper/000:00:550
1885699183,2cyclictest32597-21kworker/0:319:56:050
1885699183,12cyclictest477-21/usr/sbin/munin19:40:150
1885699181,6cyclictest0-21swapper/022:28:400
18856991815,2cyclictest7-21ksoftirqd/023:05:170
18856991812,3cyclictest13182-21grep22:35:150
1885699181,11cyclictest0-21swapper/023:18:070
1885699181,11cyclictest0-21swapper/023:18:070
1885699181,11cyclictest0-21swapper/022:51:550
1885699181,11cyclictest0-21swapper/022:21:110
1885699181,11cyclictest0-21swapper/021:53:300
1885699181,10cyclictest0-21swapper/000:15:510
308382171,2sleep30-21swapper/320:46:353
303352171,3sleep30-21swapper/319:32:543
270842171,2sleep30-21swapper/321:51:433
242762170,3sleep00-21swapper/020:31:500
1887499175,3cyclictest5450-21cat21:05:013
1887499173,4cyclictest30308-21fschecks_time22:00:153
1887499172,5cyclictest22950-21latency_hist21:45:013
1887499172,12cyclictest11575-21/usr/sbin/munin20:05:243
1887499171,6cyclictest0-21swapper/323:05:253
1887499171,4cyclictest887-21diskstats19:40:143
1887499171,4cyclictest31903-21/usr/sbin/munin19:35:213
18874991712,3cyclictest27373-21grep20:40:153
18874991711,3cyclictest25232-21fschecks_time20:35:163
1886899175,3cyclictest0-21swapper/220:07:072
1886899174,4cyclictest2261-21ntp_states20:55:222
1886899174,4cyclictest2261-21ntp_states20:55:222
1886899172,12cyclictest23186-21gltestperf20:30:162
1886899171,7cyclictest0-21swapper/200:16:142
1886899171,6cyclictest29798-21egrep22:00:122
1886899171,6cyclictest21785-21unixbench_multi19:15:042
1886899171,5cyclictest13876-21ntp_states22:35:212
1886899171,4cyclictest0-21swapper/219:50:112
18868991712,3cyclictest31702-21grep00:30:162
18868991712,3cyclictest31685-21df20:50:132
1886899171,15cyclictest0-21swapper/220:21:272
18868991711,4cyclictest26308-21awk20:35:272
1885699178,1cyclictest0-21swapper/019:11:380
1885699176,3cyclictest0-21swapper/022:06:090
1885699171,6cyclictest0-21swapper/021:23:040
1885699171,15cyclictest0-21swapper/023:55:060
1885699170,8cyclictest0-21swapper/022:55:370
222282160,2sleep00-21swapper/020:28:450
1887499162,7cyclictest0-21swapper/320:00:343
1887499160,10cyclictest0-21swapper/319:50:113
1886899161,6cyclictest3397-21basename21:00:122
1886899161,6cyclictest29740-21gltestperf20:45:152
1886899161,6cyclictest26956-21/usr/sbin/munin20:40:162
1886899161,5cyclictest808-21snmpd19:20:382
1886899161,5cyclictest0-21swapper/200:20:062
18868991612,3cyclictest0-21swapper/221:20:092
1886899161,14cyclictest0-21swapper/223:30:452
18868991611,3cyclictest23069-21unixbench-2d22:55:272
18868991611,3cyclictest1146-21grep19:40:162
1886899160,6cyclictest0-21swapper/223:55:252
1885699163,10cyclictest0-21swapper/021:03:390
1885699161,8cyclictest87550irq/128-enp1s0-20:05:250
18856991612,3cyclictest0-21swapper/019:46:490
18856991611,3cyclictest23318-21timerandwakeup19:15:250
18856991611,3cyclictest15967-21/usr/sbin/munin20:15:160
1886899151,5cyclictest19408-21/usr/sbin/munin22:50:162
18868991510,3cyclictest32084-21grep19:35:242
1885699158,4cyclictest24750-21head19:20:170
1885699148,3cyclictest9370-21/usr/sbin/munin20:00:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional