You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-12 - 13:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat Sep 12, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
113112241167,23sleep20-21swapper/219:09:472
112272228154,23sleep10-21swapper/119:08:431
109802201164,24sleep30-21swapper/319:05:303
111012200165,24sleep00-21swapper/019:07:050
2055321290,2sleep01141299cyclictest00:20:230
11425991111,108cyclictest0-21swapper/220:27:162
11425998581,3cyclictest0-21swapper/222:59:232
11425996158,2cyclictest0-21swapper/223:13:002
251862580,1sleep10-21swapper/120:50:241
11425995147,3cyclictest0-21swapper/221:40:542
243982500,2sleep20-21swapper/222:05:002
33512480,4sleep121-21rcuc/122:30:131
117282450,3sleep30-21swapper/300:00:243
11425994239,2cyclictest0-21swapper/219:10:472
176122400,2sleep20-21swapper/223:00:242
1141299346,13cyclictest23371-21munin-run00:30:000
1142599331,13cyclictest0-21swapper/222:40:292
11419993327,3cyclictest4127-21diskstats23:45:161
1142599321,22cyclictest0-21swapper/221:49:402
1142599320,12cyclictest0-21swapper/200:12:452
1141299328,11cyclictest28712-21basename22:15:010
1143299300,22cyclictest0-21swapper/323:31:143
1142599300,12cyclictest0-21swapper/200:28:302
1143299291,22cyclictest0-21swapper/321:51:133
1142599291,12cyclictest27885-21cat00:40:012
1142599291,11cyclictest32187-21ntp_states22:20:202
1142599291,11cyclictest32187-21ntp_states22:20:202
1142599291,11cyclictest0-21swapper/200:21:292
1141999291,25cyclictest26377-21ls19:40:191
1141999291,24cyclictest2132-21proc_pri22:25:231
1143299281,20cyclictest0-21swapper/321:20:123
1143299281,20cyclictest0-21swapper/300:26:353
1143299281,18cyclictest0-21swapper/300:13:553
1143299281,11cyclictest0-21swapper/321:45:063
1143299281,10cyclictest0-21swapper/323:44:053
1142599281,11cyclictest0-21swapper/200:15:492
1142599280,13cyclictest0-21swapper/221:50:362
1141299281,13cyclictest3905-21munin-run20:05:010
1143299271,20cyclictest0-21swapper/322:27:253
1143299271,19cyclictest0-21swapper/322:55:143
1143299271,19cyclictest0-21swapper/322:37:383
1143299271,19cyclictest0-21swapper/321:33:523
1143299271,18cyclictest0-21swapper/322:03:223
1142599270,21cyclictest0-21swapper/221:13:212
1142599270,21cyclictest0-21swapper/221:13:212
1142599270,10cyclictest0-21swapper/222:10:242
1141299271,23cyclictest808-21snmpd20:55:150
1141299271,23cyclictest15106-21gltestperf19:15:150
1143299261,18cyclictest0-21swapper/323:17:313
1143299261,18cyclictest0-21swapper/300:39:073
1143299261,17cyclictest0-21swapper/322:48:533
1143299261,16cyclictest13930-21munin-run22:55:013
1143299261,16cyclictest0-21swapper/321:10:383
1143299261,16cyclictest0-21swapper/321:10:383
1143299261,15cyclictest0-21swapper/321:41:003
1143299260,19cyclictest0-21swapper/323:56:423
1142599261,12cyclictest0-21swapper/222:37:172
1142599261,11cyclictest0-21swapper/223:49:452
1142599261,11cyclictest0-21swapper/219:35:122
1142599260,9cyclictest0-21swapper/200:01:012
11419992623,2cyclictest0-21swapper/123:58:381
11419992623,2cyclictest0-21swapper/121:13:481
11419992623,2cyclictest0-21swapper/121:13:481
11419992623,2cyclictest0-21swapper/100:22:481
1141999261,5cyclictest7865-21expr21:25:211
1141999261,22cyclictest17831-21cut19:20:231
1141999261,22cyclictest0-21swapper/122:45:181
1141999261,22cyclictest0-21swapper/122:35:221
1141299261,22cyclictest32733-21strings23:35:180
1141299261,22cyclictest0-21swapper/019:25:250
1141299261,12cyclictest7329-21munin-run22:40:010
317472251,4sleep30-21swapper/321:05:233
1143299251,17cyclictest0-21swapper/323:08:133
1143299251,17cyclictest0-21swapper/321:28:523
1143299251,16cyclictest0-21swapper/300:06:193
1143299251,14cyclictest0-21swapper/323:49:323
1143299251,14cyclictest0-21swapper/321:56:273
1143299251,10cyclictest23265-21grep20:45:263
11432992510,9cyclictest0-21swapper/300:30:243
1143299250,16cyclictest0-21swapper/321:35:263
1142599251,11cyclictest0-21swapper/223:22:452
1142599251,10cyclictest0-21swapper/200:32:412
1142599250,9cyclictest0-21swapper/223:17:412
1142599250,12cyclictest0-21swapper/220:36:172
1142599250,11cyclictest0-21swapper/223:05:212
1142599250,10cyclictest0-21swapper/223:35:132
1141999253,20cyclictest0-21swapper/121:20:001
11419992522,2cyclictest0-21swapper/123:40:061
1141999252,21cyclictest0-21swapper/123:26:141
11419992521,3cyclictest0-21swapper/122:08:061
1141999251,21cyclictest14919-21cpuspeed_turbos00:10:111
1141999251,11cyclictest6311-21/usr/sbin/munin23:50:111
1141299251,21cyclictest11528-21pmu-power00:00:210
1141299251,10cyclictest0-21swapper/022:26:240
1141299250,22cyclictest19319-21if_enp2s023:05:170
1143299242,7cyclictest6099-21munin-run20:10:003
1143299242,7cyclictest6099-21munin-run20:10:003
1143299241,6cyclictest0-21swapper/323:23:413
1143299241,17cyclictest0-21swapper/322:15:313
1143299241,16cyclictest0-21swapper/323:52:073
1143299241,16cyclictest0-21swapper/320:00:223
1143299241,14cyclictest0-21swapper/323:12:273
1143299241,14cyclictest0-21swapper/322:41:043
1143299241,14cyclictest0-21swapper/322:33:013
1143299241,13cyclictest0-21swapper/323:25:073
1143299241,12cyclictest0-21swapper/322:07:383
1143299241,10cyclictest27644-21ls19:45:013
11432992410,8cyclictest19879-21/usr/sbin/munin19:25:193
1142599247,10cyclictest0-21swapper/223:30:172
1142599241,9cyclictest0-21swapper/220:05:132
1142599241,9cyclictest0-21swapper/220:05:132
1142599241,8cyclictest0-21swapper/219:45:202
1142599240,10cyclictest0-21swapper/221:56:562
1141999242,20cyclictest0-21swapper/120:20:531
1141999242,20cyclictest0-21swapper/119:45:241
11419992421,2cyclictest0-21swapper/121:51:351
11419992420,3cyclictest0-21swapper/120:55:201
11419992415,3cyclictest27059-21df22:10:131
1141999241,21cyclictest0-21swapper/123:01:471
1141999241,21cyclictest0-21swapper/121:45:411
1141999241,21cyclictest0-21swapper/121:31:591
1141999241,21cyclictest0-21swapper/121:01:301
1141999241,21cyclictest0-21swapper/119:25:151
1141999241,21cyclictest0-21swapper/119:19:451
1141999241,11cyclictest12360-21latency_hist00:05:021
1141999241,10cyclictest14446-21sshd21:40:181
1141999240,4cyclictest0-21swapper/123:15:261
1141999240,3cyclictest0-21swapper/122:18:061
1141999240,22cyclictest0-21swapper/121:05:191
1141299241,7cyclictest32483-21grep21:10:000
1141299241,7cyclictest24092-21awk23:15:220
1141299241,21cyclictest0-21swapper/022:15:180
1141299241,21cyclictest0-21swapper/021:45:130
1141299241,21cyclictest0-21swapper/021:10:270
1141299241,21cyclictest0-21swapper/021:10:270
1141299241,21cyclictest0-21swapper/019:50:100
1141299241,20cyclictest12102-21if_enp2s021:35:180
1141299241,10cyclictest0-21swapper/019:30:120
1141299240,22cyclictest0-21swapper/020:05:170
1141299240,22cyclictest0-21swapper/020:05:170
1143299234,9cyclictest87350irq/126-enp1s0-20:00:003
1143299231,8cyclictest18432-21wc00:15:253
11432992315,3cyclictest0-21swapper/323:35:363
1143299231,16cyclictest0-21swapper/321:16:103
1143299231,16cyclictest0-21swapper/320:30:133
1143299231,15cyclictest0-21swapper/322:21:053
1143299231,15cyclictest0-21swapper/322:21:043
1143299231,14cyclictest0-21swapper/322:11:423
1143299231,14cyclictest0-21swapper/300:21:123
1143299231,11cyclictest0-21swapper/323:02:023
11425992310,3cyclictest0-21swapper/219:17:132
1142599230,9cyclictest0-21swapper/221:15:452
1142599230,5cyclictest9880-21kworker/u8:000:00:022
11419992315,5cyclictest23002-21hddtemp_smartct22:00:181
11419992312,3cyclictest0-21swapper/120:06:271
11419992312,3cyclictest0-21swapper/120:06:271
1141999231,21cyclictest0-21swapper/123:30:251
1141999231,21cyclictest0-21swapper/122:51:151
1141999231,21cyclictest0-21swapper/119:55:281
1141999231,20cyclictest0-21swapper/123:39:441
1141999231,20cyclictest0-21swapper/120:35:191
1141999231,20cyclictest0-21swapper/120:30:311
1141999231,20cyclictest0-21swapper/100:15:131
1141999231,20cyclictest0-21swapper/100:09:401
1141999230,5cyclictest0-21swapper/121:55:121
1141999230,4cyclictest0-21swapper/122:22:461
1141999230,4cyclictest0-21swapper/122:22:461
1141999230,4cyclictest0-21swapper/121:22:041
1141999230,2cyclictest0-21swapper/119:15:011
1141999230,22cyclictest0-21swapper/119:50:101
1141999230,20cyclictest2004-21cat20:00:131
1141999230,15cyclictest0-21swapper/100:30:021
1141299231,6cyclictest26374-21gzip23:20:220
1141299231,5cyclictest0-21swapper/023:42:350
1141299231,20cyclictest0-21swapper/022:30:140
1141299231,20cyclictest0-21swapper/020:10:240
1141299231,20cyclictest0-21swapper/019:55:260
1141299231,11cyclictest19409-21grep21:50:250
1141299231,10cyclictest13100-21pmu-power22:50:210
1141299230,21cyclictest4356-21kworker/0:221:20:230
1141299230,21cyclictest0-21swapper/020:40:210
1141299230,21cyclictest0-21swapper/019:35:180
1141299230,21cyclictest0-21swapper/000:30:170
1143299228,9cyclictest0-21swapper/320:15:193
1143299228,8cyclictest41-21ksoftirqd/320:50:113
1143299223,10cyclictest30552-21smartctl19:50:143
1143299222,11cyclictest25261-21latency_hist19:40:023
1143299221,9cyclictest13912-21grep20:25:183
1143299221,8cyclictest0-21swapper/320:35:143
1143299221,13cyclictest0-21swapper/319:45:233
1143299221,12cyclictest21101-21users20:40:253
1142599222,4cyclictest10130-21kworker/u8:121:24:412
1142599221,9cyclictest0-21swapper/222:34:532
1142599221,6cyclictest0-21swapper/223:42:352
1142599220,9cyclictest0-21swapper/223:27:572
1142599220,12cyclictest0-21swapper/222:17:482
1142599220,11cyclictest0-21swapper/223:54:412
1142599220,10cyclictest0-21swapper/222:26:452
1142599220,10cyclictest0-21swapper/220:31:192
11419992219,2cyclictest0-21swapper/122:57:341
1141999221,20cyclictest0-21swapper/120:48:381
1141999221,20cyclictest0-21swapper/120:14:301
1141999220,6cyclictest0-21swapper/123:11:211
1141999220,20cyclictest0-21swapper/120:15:241
1141299221,2cyclictest0-21swapper/022:05:260
1141299221,20cyclictest0-21swapper/021:25:200
1141299221,18cyclictest8933-21if_enp2s023:55:160
1141299221,18cyclictest7938-21fschecks_count22:40:150
1141299221,11cyclictest31212-21sshd23:31:170
1141299220,21cyclictest0-21swapper/023:10:250
1141299220,20cyclictest0-21swapper/021:00:110
1143299215,3cyclictest12050irq/123-ahci[0019:25:023
11432992117,3cyclictest0-21swapper/320:12:243
1143299211,11cyclictest12760-21grep19:10:213
1142599216,3cyclictest0-21swapper/221:33:032
1142599211,8cyclictest8914-21perf21:30:012
1142599211,5cyclictest0-21swapper/221:35:362
1142599210,9cyclictest0-21swapper/222:45:052
1141999219,3cyclictest0-21swapper/119:35:471
11419992118,1cyclictest0-21swapper/121:35:261
11419992117,3cyclictest0-21swapper/123:22:321
1141999211,5cyclictest26101-21expr00:35:131
1141999211,19cyclictest0-21swapper/120:43:581
1141999211,18cyclictest0-21swapper/122:40:131
1141999211,18cyclictest0-21swapper/119:30:131
1141299218,2cyclictest15182-21fschecks_count00:10:130
1141299216,3cyclictest3766-21rt-features21:15:230
1141299211,16cyclictest13833-21memory20:25:190
1141299210,19cyclictest0-21swapper/022:00:200
1141299210,19cyclictest0-21swapper/020:20:170
1141299210,18cyclictest27421-21cpuspeed_turbos23:25:120
169642200,1sleep20-21swapper/219:20:152
1143299201,7cyclictest21817-21basename19:30:173
1143299201,6cyclictest0-21swapper/320:20:143
1142599201,3cyclictest11458-21kworker/u8:222:52:152
1141999202,17cyclictest0-21swapper/100:25:241
11419992013,3cyclictest19696-21pmu-power23:05:221
1141999200,18cyclictest0-21swapper/120:27:291
1141299207,7cyclictest0-21swapper/023:45:160
1141299206,3cyclictest15631-21sed22:55:260
1141299206,2cyclictest32036-21memory22:20:190
1141299206,2cyclictest32036-21memory22:20:190
1141299201,7cyclictest0-21swapper/019:45:210
11412992012,2cyclictest10207-21ntp_states21:30:200
1141299200,19cyclictest0-21swapper/023:00:170
299692191,2sleep30-21swapper/321:01:103
1143299191,5cyclictest26682-21perl20:55:163
1143299191,4cyclictest16481-21perf19:20:013
1142599194,3cyclictest87550irq/128-enp1s0-00:08:492
1141299197,3cyclictest0-21swapper/023:50:170
1141299194,3cyclictest26208-21basename19:40:160
1141299194,3cyclictest0-21swapper/019:21:460
1141299192,14cyclictest11660-21sshd22:49:170
1141299191,7cyclictest0-21swapper/000:05:100
1141299191,6cyclictest8517-21/usr/sbin/munin20:15:120
1141299191,3cyclictest87550irq/128-enp1s0-21:40:140
1141299191,17cyclictest0-21swapper/020:32:420
1141299191,10cyclictest0-21swapper/020:38:500
14742181,2sleep20-21swapper/219:57:362
1142599181,7cyclictest30611-21cpuspeed_turbos21:05:122
1142599181,7cyclictest26003-21cat20:55:012
1142599181,5cyclictest8564-1kworker/u9:120:19:142
11425991814,2cyclictest19697-40dnf19:25:192
11425991812,3cyclictest25491-21/usr/sbin/munin19:40:182
1141299181,5cyclictest0-21swapper/021:57:140
1141299181,4cyclictest24742-21sshd20:50:170
1141299180,5cyclictest0-21swapper/000:15:250
217062171,2sleep20-21swapper/220:45:122
1142599175,4cyclictest5750irq/9-acpi20:55:162
1142599172,7cyclictest6322-21/usr/sbin/munin20:10:252
11425991712,3cyclictest1-21systemd20:40:172
11425991711,3cyclictest21098-21/usr/sbin/munin19:30:132
1141299171,4cyclictest0-21swapper/000:39:440
1141299171,2cyclictest0-21swapper/019:10:180
1142599168,6cyclictest2076-21cpuspeed_turbos20:00:132
1142599164,3cyclictest0-21swapper/222:05:132
1142599162,5cyclictest29069-21if_err_enp1s021:00:172
1142599161,6cyclictest10715-21/usr/sbin/munin20:20:182
1142599161,14cyclictest0-21swapper/219:54:552
1141299169,4cyclictest21828-21cpuspeed_turbos20:45:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional