You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-26 - 10:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Jun 26, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
37572240165,25sleep20-21swapper/219:08:502
38032237163,25sleep10-21swapper/119:09:251
37082203165,26sleep00-21swapper/019:08:110
36482203165,25sleep30-21swapper/319:07:263
2947621590,4sleep1393999cyclictest20:05:141
3946998279,2cyclictest0-21swapper/220:05:232
87212560,1sleep00-21swapper/020:30:220
25552470,2sleep00-21swapper/021:30:170
198452460,4sleep011-21rcuc/022:10:150
393999433,17cyclictest25634-40dnf23:35:261
393399392,22cyclictest30112-21idleruntime-cro22:35:020
393399389,14cyclictest9470-21munin-run00:15:010
393999351,17cyclictest0-21swapper/121:42:251
395099341,24cyclictest2168-21grep21:30:143
393399347,12cyclictest26422-21idleruntime-cro23:40:020
395099331,24cyclictest0-21swapper/320:26:103
393999331,24cyclictest0-21swapper/122:25:111
395099321,19cyclictest11274-21irqstats21:50:183
395099321,18cyclictest0-21swapper/319:34:583
395099311,14cyclictest0-21swapper/320:56:573
395099311,14cyclictest0-21swapper/320:13:543
393999311,12cyclictest0-21swapper/121:37:521
257512310,4sleep230-21rcuc/222:25:022
395099301,22cyclictest0-21swapper/320:32:103
393999301,23cyclictest0-21swapper/119:19:041
393399307,11cyclictest27204-21munin-run21:15:020
393399307,11cyclictest21986-21perf23:30:010
393999291,18cyclictest0-21swapper/122:01:351
3939992911,4cyclictest1214-21ntp_states22:40:191
393999291,14cyclictest0-21swapper/100:05:521
393999291,12cyclictest0-21swapper/122:08:411
393999291,11cyclictest0-21swapper/123:58:361
393999291,10cyclictest0-21swapper/119:22:041
3939992910,4cyclictest27717-21memory20:00:171
393399297,10cyclictest7285-21idleruntime-cro00:10:010
395099281,17cyclictest0-21swapper/319:44:143
395099281,13cyclictest0-21swapper/320:07:383
395099281,12cyclictest0-21swapper/321:43:083
395099281,12cyclictest0-21swapper/319:15:153
395099281,11cyclictest0-21swapper/319:14:463
395099280,10cyclictest0-21swapper/322:21:083
393999282,17cyclictest7682-21memory22:55:201
393999281,14cyclictest0-21swapper/100:37:401
393999281,13cyclictest0-21swapper/122:39:031
393999281,12cyclictest0-21swapper/120:56:121
393999281,11cyclictest0-21swapper/119:59:431
393999281,11cyclictest0-21swapper/119:33:041
393399286,2cyclictest30724-21kworker/0:120:09:560
395099271,12cyclictest0-21swapper/319:46:103
395099271,10cyclictest0-21swapper/321:45:183
395099271,10cyclictest0-21swapper/319:57:143
395099270,12cyclictest0-21swapper/323:46:213
393999271,14cyclictest0-21swapper/123:15:001
393999271,14cyclictest0-21swapper/123:14:591
393999271,13cyclictest0-21swapper/121:34:361
393999271,11cyclictest0-21swapper/120:26:081
393999271,10cyclictest0-21swapper/121:17:081
3939992710,3cyclictest8320-21hddtemp_smartct20:30:161
395099261,17cyclictest0-21swapper/321:21:143
395099261,12cyclictest0-21swapper/320:01:023
395099260,17cyclictest0-21swapper/320:16:503
393999261,14cyclictest0-21swapper/122:47:351
3939992611,3cyclictest0-21swapper/123:28:001
393999261,12cyclictest0-21swapper/123:15:041
393999261,12cyclictest0-21swapper/122:19:211
393999261,12cyclictest0-21swapper/121:22:521
393999261,11cyclictest0-21swapper/123:01:191
393999261,11cyclictest0-21swapper/121:46:361
393999261,11cyclictest0-21swapper/119:27:561
393399260,12cyclictest5846-21perf21:40:000
395099252,11cyclictest22243-21cpu23:30:103
395099251,11cyclictest0-21swapper/323:05:203
395099251,10cyclictest0-21swapper/300:08:053
395099250,12cyclictest0-21swapper/300:21:343
395099250,12cyclictest0-21swapper/300:21:343
395099250,11cyclictest0-21swapper/323:41:163
395099250,11cyclictest0-21swapper/322:59:343
393999251,9cyclictest0-21swapper/120:15:401
393999251,13cyclictest0-21swapper/121:52:481
393999251,12cyclictest0-21swapper/122:32:151
393999251,11cyclictest0-21swapper/121:56:561
393999251,11cyclictest0-21swapper/121:56:561
393999251,11cyclictest0-21swapper/120:24:361
393999251,11cyclictest0-21swapper/119:47:231
393999251,11cyclictest0-21swapper/119:40:031
393999251,10cyclictest0-21swapper/120:35:481
393999251,10cyclictest0-21swapper/100:00:421
393399251,17cyclictest0-21swapper/021:40:080
393399251,17cyclictest0-21swapper/000:17:080
395099240,8cyclictest0-21swapper/323:55:153
393999249,3cyclictest13626-21users20:40:241
393999241,15cyclictest0-21swapper/123:34:021
393999241,13cyclictest0-21swapper/100:28:121
393999241,12cyclictest0-21swapper/123:06:511
393999241,12cyclictest0-21swapper/100:31:521
393999241,11cyclictest0-21swapper/123:51:121
393999241,11cyclictest0-21swapper/122:54:551
393999241,11cyclictest0-21swapper/121:27:521
393999241,11cyclictest0-21swapper/119:38:561
393999241,11cyclictest0-21swapper/119:38:551
393399241,17cyclictest0-21swapper/023:05:560
393399241,17cyclictest0-21swapper/021:46:350
393399241,14cyclictest0-21swapper/022:48:540
393399241,12cyclictest1427-21awk21:30:010
393399240,18cyclictest0-21swapper/021:20:130
395099231,9cyclictest0-21swapper/320:41:033
395099230,17cyclictest0-21swapper/322:44:423
395099230,11cyclictest0-21swapper/300:00:593
394699234,5cyclictest0-21swapper/220:34:152
393999239,3cyclictest0-21swapper/122:13:071
393999237,3cyclictest0-21swapper/121:12:441
393999231,12cyclictest0-21swapper/100:25:001
393999231,12cyclictest0-21swapper/100:25:001
393999231,10cyclictest0-21swapper/121:05:281
393999231,10cyclictest0-21swapper/119:11:441
393399231,17cyclictest0-21swapper/000:31:160
393399231,16cyclictest0-21swapper/022:40:580
393399231,16cyclictest0-21swapper/000:23:170
393399231,16cyclictest0-21swapper/000:23:160
393399231,15cyclictest0-21swapper/023:15:430
393399231,15cyclictest0-21swapper/022:02:130
3933992311,3cyclictest0-21swapper/023:04:170
3933992310,5cyclictest0-21swapper/000:04:410
393399230,11cyclictest28619-21perf23:45:020
395099229,2cyclictest0-21swapper/323:16:543
395099221,9cyclictest0-21swapper/321:35:583
3950992213,5cyclictest30509-21cstates22:35:133
395099221,11cyclictest0-21swapper/319:35:103
395099221,11cyclictest0-21swapper/319:35:103
395099220,12cyclictest0-21swapper/319:20:503
395099220,11cyclictest0-21swapper/323:25:383
394699228,3cyclictest0-21swapper/200:30:212
394699223,4cyclictest16061-21munin-run00:30:012
394699222,7cyclictest8147-21grep00:10:172
394699221,7cyclictest12441-21sleep21:55:012
394699221,3cyclictest87650irq/129-enp1s0-00:10:012
394699221,13cyclictest0-21swapper/220:15:162
394699221,13cyclictest0-21swapper/200:35:072
394699221,13cyclictest0-21swapper/200:19:222
393999227,3cyclictest0-21swapper/122:21:231
393999221,14cyclictest0-21swapper/100:19:481
393999221,12cyclictest0-21swapper/121:00:131
393999221,10cyclictest0-21swapper/123:20:361
393999220,9cyclictest0-21swapper/120:13:311
393399229,4cyclictest0-21swapper/022:20:270
393399225,10cyclictest0-21swapper/022:25:230
393399223,13cyclictest87550irq/128-enp1s0-21:55:210
393399223,13cyclictest87550irq/128-enp1s0-21:55:210
393399221,16cyclictest0-21swapper/023:54:340
393399221,15cyclictest0-21swapper/023:32:520
393399221,13cyclictest0-21swapper/022:10:000
393399221,10cyclictest11642-21kworker/0:120:48:420
395099217,3cyclictest0-21swapper/300:37:383
395099216,3cyclictest0-21swapper/320:20:503
395099215,3cyclictest0-21swapper/300:30:423
395099211,9cyclictest0-21swapper/321:55:163
395099211,9cyclictest0-21swapper/321:55:163
395099211,9cyclictest0-21swapper/319:51:203
395099211,6cyclictest0-21swapper/322:18:523
395099211,6cyclictest0-21swapper/322:00:343
395099211,5cyclictest0-21swapper/322:30:043
395099211,11cyclictest0-21swapper/322:07:343
395099210,9cyclictest0-21swapper/323:01:103
395099210,9cyclictest0-21swapper/300:17:173
394699216,3cyclictest5069-21awk00:05:012
394699215,4cyclictest28848-21basename22:30:172
394699212,6cyclictest22803-21latency_hist21:05:022
394699211,8cyclictest5381-21irqstats22:50:192
394699211,7cyclictest0-21swapper/223:00:542
394699211,7cyclictest0-21swapper/220:35:142
393999217,3cyclictest0-21swapper/120:45:281
393999217,11cyclictest30652-21systemd-journal23:48:131
393999211,15cyclictest0-21swapper/123:44:231
393999211,12cyclictest0-21swapper/100:10:161
393399212,10cyclictest22574-21kworker/0:119:52:540
393399211,6cyclictest5560-21strings22:50:210
393399211,12cyclictest0-21swapper/000:39:330
393399211,11cyclictest7831-21ntp_states22:55:230
393399211,10cyclictest0-21swapper/023:45:310
3933992110,3cyclictest0-21swapper/023:55:180
105262210,2sleep09-21rcu_preempt21:50:110
395099208,3cyclictest22811-21awk21:05:003
395099208,2cyclictest25001-21diskmemload23:38:093
395099206,3cyclictest0-21swapper/322:50:583
395099201,7cyclictest0-21swapper/320:47:593
395099201,6cyclictest0-21swapper/323:53:113
395099201,11cyclictest0-21swapper/321:14:463
395099201,10cyclictest0-21swapper/323:10:243
395099201,10cyclictest0-21swapper/323:10:243
395099200,9cyclictest0-21swapper/321:15:563
395099200,11cyclictest0-21swapper/300:12:353
394699201,6cyclictest16166-21diskstats19:35:152
394699201,6cyclictest16166-21diskstats19:35:152
394699201,6cyclictest0-21swapper/221:15:482
3946992013,4cyclictest827-21runrttasks20:53:012
394699201,11cyclictest0-21swapper/222:19:492
394699201,11cyclictest0-21swapper/220:13:072
3939992013,2cyclictest0-21swapper/120:51:351
393399207,5cyclictest0-21swapper/022:39:120
393399205,7cyclictest25000-21crond21:10:010
393399203,12cyclictest0-21swapper/023:21:220
393399201,14cyclictest87350irq/126-enp1s0-22:15:560
393399201,13cyclictest0-21swapper/020:25:150
395099196,3cyclictest0-21swapper/323:20:103
395099191,11cyclictest0-21swapper/322:25:543
395099191,10cyclictest0-21swapper/300:28:173
395099190,12cyclictest0-21swapper/322:12:483
395099190,12cyclictest0-21swapper/321:05:453
394699195,3cyclictest87350irq/126-enp1s0-23:31:192
394699191,5cyclictest15373-21latency_hist23:15:022
394699191,5cyclictest15373-21latency_hist23:15:022
394699191,5cyclictest11376-21ls19:25:012
394699191,12cyclictest0-21swapper/222:05:242
394699191,10cyclictest0-21swapper/221:41:532
393999190,14cyclictest0-21swapper/119:50:131
393399191,5cyclictest15080-21strings00:25:200
393399191,14cyclictest0-21swapper/021:15:260
393399191,12cyclictest0-21swapper/023:12:240
393399191,12cyclictest0-21swapper/023:12:240
393399191,12cyclictest0-21swapper/019:55:120
393399191,10cyclictest28358-21sed20:00:250
32382191,14sleep00-21swapper/020:20:120
395099186,3cyclictest0-21swapper/321:25:023
395099181,8cyclictest0-21swapper/322:45:133
395099181,14cyclictest12106-21hwlatdetect19:25:163
395099181,12cyclictest0-21swapper/320:39:423
394699187,4cyclictest0-21swapper/221:10:232
394699185,3cyclictest1746-21cpuspeed_turbos21:30:112
394699184,3cyclictest87350irq/126-enp1s0-23:20:242
394699183,3cyclictest0-21swapper/221:35:212
394699182,5cyclictest13152-21latency_hist23:10:022
394699182,5cyclictest0-21swapper/223:26:482
394699182,15cyclictest0-21swapper/221:05:272
394699181,7cyclictest0-21swapper/222:43:322
394699181,6cyclictest0-21swapper/222:35:302
394699181,5cyclictest32710-21sed21:25:182
394699181,5cyclictest0-21swapper/223:15:222
394699181,4cyclictest9206-21missed_timers21:45:212
3946991812,3cyclictest24745-21df23:35:122
3946991812,3cyclictest12329-21ls20:40:112
394699181,16cyclictest0-21swapper/200:24:312
394699181,16cyclictest0-21swapper/200:24:312
394699181,15cyclictest0-21swapper/223:41:292
394699181,15cyclictest0-21swapper/219:15:432
394699181,11cyclictest0-21swapper/222:10:202
393399183,3cyclictest20010-21grep19:45:010
393399181,9cyclictest0-21swapper/020:55:170
393399181,4cyclictest17373-21grep19:35:260
393399181,4cyclictest17373-21grep19:35:260
393399181,4cyclictest14226-21head19:30:140
3933991811,4cyclictest20579-21fschecks_count19:45:140
393399181,11cyclictest0-21swapper/020:40:180
393399181,11cyclictest0-21swapper/020:10:480
393399181,11cyclictest0-21swapper/019:10:260
277222181,2sleep20-21swapper/222:27:562
203672181,2sleep20-21swapper/220:57:142
168182180,3sleep20-21swapper/222:04:442
394699179,5cyclictest4337-21turbostat.cron22:50:012
394699175,3cyclictest19189-21ps19:40:222
394699171,9cyclictest0-21swapper/219:56:302
394699171,6cyclictest3239-21/usr/sbin/munin20:20:122
394699171,6cyclictest0-21swapper/223:45:542
394699171,4cyclictest29645-21cpu21:20:122
3946991713,2cyclictest5539-21cpuspeed_turbos20:25:112
3946991711,3cyclictest24373-21latency_hist19:55:022
394699171,10cyclictest0-21swapper/221:56:172
394699171,10cyclictest0-21swapper/221:56:172
3946991710,4cyclictest6907-21tr22:55:142
393399174,4cyclictest22500-21sshd21:01:140
393399172,12cyclictest11-21rcuc/020:35:160
393399171,4cyclictest12412-21head19:25:180
3933991712,3cyclictest17022-21grep20:50:140
393399171,10cyclictest0-21swapper/019:15:170
267942170,3sleep20-21swapper/220:00:072
8172160,2sleep20-21swapper/223:55:082
3950991611,3cyclictest16484-21cpu20:50:103
394699161,6cyclictest4739-21gltestperf19:10:142
3946991611,3cyclictest12081-21cut19:25:152
3946991610,3cyclictest20702-21cut19:45:142
3946991610,3cyclictest13854-21cat19:30:122
394699160,7cyclictest31543-21fschecks_time23:50:142
3933991611,4cyclictest0-21swapper/019:20:160
3933991611,3cyclictest2149-21/usr/sbin/munin20:15:220
394699151,5cyclictest14272-21cpu20:45:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional