You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-22 - 05:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri May 22, 2026 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
216932239162,24sleep00-21swapper/019:07:260
216252239162,24sleep20-21swapper/219:06:342
217612234164,56sleep10-21swapper/119:08:181
202882203167,24sleep30-21swapper/319:05:153
2764821880,5sleep32199199cyclictest23:00:173
1710621770,4sleep22198799cyclictest22:35:242
238621310,3sleep339-21rcuc/319:35:183
228322650,1sleep20-21swapper/219:10:162
21987996158,2cyclictest0-21swapper/220:52:262
124642440,1sleep30-21swapper/320:00:013
21987993734,2cyclictest0-21swapper/222:31:412
21974993610,13cyclictest5667-21munin-run19:45:010
2197499342,13cyclictest8327-21kworker/0:121:05:230
2198799320,13cyclictest0-21swapper/222:05:292
2197499326,12cyclictest18879-21perf20:15:020
2197499322,24cyclictest27146-21kworker/0:100:12:480
2198799311,13cyclictest0-21swapper/200:01:372
2199199301,17cyclictest0-21swapper/323:16:093
2198799301,10cyclictest0-21swapper/222:44:262
2198799300,12cyclictest0-21swapper/221:46:452
2198099300,24cyclictest0-21swapper/122:23:081
2197499302,12cyclictest1255-21kworker/0:100:29:110
2199199291,20cyclictest0-21swapper/321:44:173
2199199291,19cyclictest0-21swapper/323:28:223
2199199291,16cyclictest11703-21kworker/u8:223:40:173
2198799291,22cyclictest0-21swapper/223:26:132
2198099292,25cyclictest0-21swapper/119:17:121
2198099291,26cyclictest0-21swapper/119:10:461
21974992919,3cyclictest9501-21kworker/0:023:45:200
21974992915,3cyclictest9974-21kworker/0:319:55:160
112252290,2sleep30-21swapper/323:35:253
112252290,2sleep30-21swapper/323:35:253
2199199281,19cyclictest0-21swapper/300:09:133
2199199281,18cyclictest0-21swapper/322:30:473
2199199280,17cyclictest0-21swapper/323:57:053
2198799281,13cyclictest0-21swapper/223:44:532
2198799281,12cyclictest0-21swapper/219:15:102
2198799281,11cyclictest0-21swapper/221:56:172
2198099289,10cyclictest8070-21cat22:15:201
2198099282,13cyclictest32055-21munin-run20:45:021
2198099281,26cyclictest0-21swapper/123:46:281
2198099281,25cyclictest0-21swapper/123:51:011
2198099281,25cyclictest0-21swapper/123:24:101
2197499281,22cyclictest26713-21kworker/0:123:03:230
2197499281,10cyclictest26713-21kworker/0:123:16:230
2199199271,18cyclictest0-21swapper/321:53:593
2199199270,18cyclictest0-21swapper/300:37:123
2199199270,17cyclictest0-21swapper/323:30:363
2198799271,12cyclictest27261-21df23:00:142
2198799270,11cyclictest0-21swapper/222:14:532
2198799270,11cyclictest0-21swapper/221:40:412
2198099272,16cyclictest0-21swapper/123:01:441
2198099271,24cyclictest0-21swapper/123:57:121
2198099271,24cyclictest0-21swapper/123:29:001
2198099271,24cyclictest0-21swapper/119:38:261
2198099271,24cyclictest0-21swapper/100:18:411
2198099270,25cyclictest0-21swapper/123:18:481
2198099270,25cyclictest0-21swapper/119:26:181
2197499272,9cyclictest17604-21kworker/0:022:39:110
307432260,2sleep30-21swapper/320:40:143
2199199264,13cyclictest0-21swapper/323:21:023
21991992612,6cyclictest0-21swapper/322:19:343
2199199261,17cyclictest0-21swapper/321:26:173
2198799261,2cyclictest0-21swapper/200:06:412
2198799260,13cyclictest0-21swapper/222:57:062
2198799260,12cyclictest0-21swapper/222:49:102
2198799260,11cyclictest0-21swapper/223:48:052
2198099262,15cyclictest0-21swapper/120:17:271
21980992614,3cyclictest0-21swapper/122:47:421
2198099261,16cyclictest0-21swapper/120:50:161
2198099260,24cyclictest0-21swapper/100:30:441
2198099260,24cyclictest0-21swapper/100:11:101
2197499262,9cyclictest9974-21kworker/0:319:55:020
2197499262,11cyclictest5036-21kworker/0:323:29:030
2197499262,11cyclictest17604-21kworker/0:022:43:190
2197499262,11cyclictest10667-21kworker/0:021:17:110
2197499262,10cyclictest14023-21kworker/0:323:52:160
2199199259,5cyclictest0-21swapper/321:20:033
2199199251,18cyclictest0-21swapper/323:48:263
2199199251,18cyclictest0-21swapper/322:36:413
2199199251,17cyclictest0-21swapper/300:15:343
2199199251,16cyclictest20738-21kworker/u8:200:20:413
2199199251,15cyclictest653-21kworker/u8:023:14:563
2199199250,17cyclictest0-21swapper/322:11:063
2198799251,9cyclictest0-21swapper/223:37:172
2198799251,9cyclictest0-21swapper/223:37:172
2198799251,11cyclictest0-21swapper/223:05:462
2198799250,11cyclictest0-21swapper/222:03:172
2198099252,15cyclictest0-21swapper/121:23:101
21980992514,3cyclictest0-21swapper/100:06:341
2198099251,22cyclictest0-21swapper/123:43:021
2198099251,16cyclictest0-21swapper/122:37:041
2198099251,16cyclictest0-21swapper/122:28:541
2198099251,16cyclictest0-21swapper/121:44:351
2198099251,16cyclictest0-21swapper/121:02:231
2198099251,15cyclictest0-21swapper/121:34:271
2198099251,15cyclictest0-21swapper/120:37:231
2198099251,15cyclictest0-21swapper/100:04:261
2198099250,15cyclictest0-21swapper/121:50:251
2197499252,9cyclictest9501-21kworker/0:023:36:000
2197499252,9cyclictest9501-21kworker/0:023:36:000
2197499251,9cyclictest9501-21kworker/0:023:33:200
2197499251,9cyclictest29474-21kworker/0:300:38:590
2197499251,18cyclictest29389-21kworker/0:000:22:040
2197499251,10cyclictest30293-21kworker/0:121:55:240
2197499251,10cyclictest10667-21kworker/0:021:33:190
21974992510,3cyclictest24445-21kworker/0:022:58:510
19902250,2sleep20-21swapper/200:30:132
21991992414,3cyclictest0-21swapper/322:40:053
2199199241,16cyclictest0-21swapper/321:31:243
2199199241,15cyclictest0-21swapper/300:30:193
21991992410,5cyclictest0-21swapper/322:51:373
2199199240,5cyclictest17087-21kworker/u8:121:35:163
2199199240,17cyclictest0-21swapper/300:27:133
2199199240,16cyclictest0-21swapper/321:57:263
2198799241,9cyclictest0-21swapper/223:17:222
2198799241,9cyclictest0-21swapper/219:23:182
2198799241,10cyclictest0-21swapper/222:22:542
2198799240,10cyclictest0-21swapper/223:21:182
21980992414,2cyclictest0-21swapper/122:56:591
21980992413,3cyclictest0-21swapper/122:02:571
21980992413,3cyclictest0-21swapper/121:17:121
21980992412,3cyclictest0-21swapper/119:55:151
2198099241,22cyclictest0-21swapper/100:24:031
2198099241,21cyclictest0-21swapper/119:44:531
2198099241,15cyclictest0-21swapper/122:52:201
2198099241,15cyclictest0-21swapper/122:14:181
2198099241,14cyclictest0-21swapper/122:44:121
2198099241,11cyclictest32621-21diskstats20:45:131
2198099240,15cyclictest0-21swapper/120:12:371
2197499249,3cyclictest30293-21kworker/0:122:12:160
2197499242,9cyclictest23839-21kworker/0:321:43:520
2197499242,10cyclictest9501-21kworker/0:023:40:040
2197499241,11cyclictest23839-21kworker/0:321:53:320
2197499241,10cyclictest5036-21kworker/0:323:24:110
2197499241,10cyclictest20386-21kworker/0:000:05:480
18332241,3sleep21846-21diskstats19:35:132
2199199239,6cyclictest0-21swapper/323:52:303
2199199232,6cyclictest27039-21munin-run19:20:023
2199199232,14cyclictest26677-21kworker/u8:123:07:153
2199199231,14cyclictest0-21swapper/300:11:553
2198799230,10cyclictest0-21swapper/222:50:102
21980992313,3cyclictest0-21swapper/120:20:531
21980992313,2cyclictest0-21swapper/123:06:081
21980992312,2cyclictest10691-21diskmemload22:05:581
21980992312,2cyclictest0-21swapper/120:57:131
2198099231,19cyclictest0-21swapper/119:45:231
2198099231,15cyclictest0-21swapper/121:13:261
2198099231,14cyclictest0-21swapper/121:48:491
2198099231,14cyclictest0-21swapper/120:04:171
2198099230,15cyclictest0-21swapper/119:53:461
2197499232,12cyclictest11255-21kworker/0:122:26:310
2197499231,9cyclictest29474-21kworker/0:300:32:520
21974992311,3cyclictest10667-21kworker/0:021:38:480
2197499231,10cyclictest527-21kworker/0:023:14:230
2197499231,10cyclictest25126-21kworker/0:020:33:030
21974992310,3cyclictest8976-21kworker/0:022:21:230
157602230,1sleep30-21swapper/320:05:193
157602230,1sleep30-21swapper/320:05:193
2199199222,15cyclictest87350irq/126-enp1s0-22:45:473
2199199222,14cyclictest87350irq/126-enp1s0-21:45:033
2199199221,16cyclictest0-21swapper/322:06:123
2199199221,15cyclictest0-21swapper/322:02:353
2199199221,15cyclictest0-21swapper/321:10:533
2199199221,14cyclictest0-21swapper/300:00:573
2199199221,10cyclictest9-21rcu_preempt22:55:213
21987992219,2cyclictest0-21swapper/200:26:312
2198099221,3cyclictest27603-21df19:20:121
21980992212,2cyclictest0-21swapper/121:07:391
2198099221,14cyclictest0-21swapper/121:26:191
2198099221,10cyclictest32648-21cat21:59:561
2197499229,3cyclictest23839-21kworker/0:321:45:320
2197499229,3cyclictest10667-21kworker/0:021:24:470
2197499228,3cyclictest61292cstates19:45:130
2197499228,3cyclictest22619-21kworker/0:100:01:200
2197499228,2cyclictest12174-21kworker/0:120:06:280
2197499228,2cyclictest12174-21kworker/0:120:06:280
2197499227,3cyclictest26713-21kworker/0:123:05:510
2197499222,9cyclictest8327-21kworker/0:121:10:350
2197499222,8cyclictest12954-21kworker/0:019:15:110
2197499221,9cyclictest24882-21kworker/0:200:16:040
2197499221,10cyclictest6686-21kworker/0:222:18:470
21974992210,3cyclictest30293-21kworker/0:122:07:520
2199199216,12cyclictest41-21ksoftirqd/319:45:173
2199199212,7cyclictest31438-21munin-run19:30:023
2199199211,9cyclictest0-21swapper/322:21:463
2199199211,14cyclictest0-21swapper/322:25:383
2198799211,9cyclictest0-21swapper/221:10:242
2198799211,11cyclictest0-21swapper/223:30:052
2198799211,11cyclictest0-21swapper/200:18:452
2198099211,3cyclictest0-21swapper/123:36:411
2198099211,3cyclictest0-21swapper/123:36:411
2198099211,18cyclictest0-21swapper/119:32:301
2198099211,17cyclictest14692-21/usr/sbin/munin20:05:211
2198099211,17cyclictest14692-21/usr/sbin/munin20:05:211
21980992111,2cyclictest0-21swapper/123:10:331
2197499217,2cyclictest27393-21kworker/0:120:45:030
2197499216,4cyclictest20995-21kworker/0:120:20:280
2197499216,4cyclictest17604-21kworker/0:022:46:430
2197499216,4cyclictest1210-21kworker/0:019:37:440
2197499213,10cyclictest29467-21/usr/sbin/munin19:25:200
21974992110,2cyclictest8327-21kworker/0:121:03:110
2197499210,7cyclictest16514-21kworker/0:320:15:360
2199199202,5cyclictest10057-21sh19:55:003
2199199202,10cyclictest13117-21/usr/sbin/munin21:15:193
2199199201,11cyclictest12487-21/usr/sbin/munin20:00:243
2198099202,16cyclictest0-21swapper/120:26:561
2198099201,3cyclictest5650-21runrttasks00:36:051
2197499208,2cyclictest20386-21kworker/0:023:56:560
2197499206,2cyclictest13524-21kworker/0:222:33:270
2197499205,4cyclictest1602-21kworker/0:020:59:070
2197499201,9cyclictest30293-21kworker/0:122:00:360
2197499201,10cyclictest28913-21kworker/0:119:21:430
2199199198,8cyclictest8413-21sshd21:04:083
2199199193,3cyclictest87350irq/126-enp1s0-20:25:013
2199199191,6cyclictest21007-21taskset20:19:303
2199199191,5cyclictest10676-21crond21:10:013
2199199191,10cyclictest4005-21sshd20:54:083
2198799198,3cyclictest156412expr21:20:132
2198799191,9cyclictest0-21swapper/221:01:422
2198799191,5cyclictest0-21swapper/223:13:582
2198799191,2cyclictest0-21swapper/222:15:302
2198799191,14cyclictest0-21swapper/220:15:372
2198799191,10cyclictest0-21swapper/221:35:452
2198099191,2cyclictest0-21swapper/120:32:401
2198099191,2cyclictest0-21swapper/100:28:361
2198099191,16cyclictest0-21swapper/123:30:171
2198099191,10cyclictest10691-21diskmemload21:35:211
2197499199,2cyclictest27393-21kworker/0:120:35:110
2197499196,3cyclictest29114-21kworker/0:319:31:410
2197499195,8cyclictest9974-21kworker/0:320:01:120
2197499192,10cyclictest19201-21kworker/0:221:27:310
21974991912,3cyclictest22171-21kworker/0:222:50:230
2197499191,10cyclictest27393-21kworker/0:120:51:150
2197499191,10cyclictest23490-21/usr/sbin/munin20:25:160
304732180,4sleep230-21rcuc/220:40:142
2199199184,12cyclictest39-21rcuc/319:30:123
2199199184,11cyclictest26513-21kernelversion20:30:173
2199199181,6cyclictest17373-21fschecks_time20:10:133
2199199181,4cyclictest24147-21latency_hist19:15:013
21991991812,3cyclictest1041-21pmu-power20:45:213
2199199181,15cyclictest0-21swapper/319:20:093
2198799181,8cyclictest0-21swapper/221:52:572
2198799181,8cyclictest0-21swapper/220:57:542
2198799181,7cyclictest16248-21sed20:05:232
2198799181,7cyclictest16248-21sed20:05:232
2198799181,7cyclictest0-21swapper/200:38:022
2198799181,4cyclictest909-21sssd_nss21:10:012
21987991813,3cyclictest18138-21users23:50:272
21987991812,3cyclictest13769-21grep21:15:182
21987991811,4cyclictest24135-21sed20:25:152
2198799181,12cyclictest0-21swapper/222:26:522
2198799181,11cyclictest0-21swapper/220:23:412
2198099181,14cyclictest153752sleep122:30:561
2197499181,9cyclictest27393-21kworker/0:120:41:030
2199199174,11cyclictest29027-21head20:35:213
2199199171,9cyclictest0-21swapper/319:40:233
21991991712,3cyclictest23576-21cpuspeed_turbos20:25:113
2198799175,3cyclictest25916-21fschecks_time00:10:152
2198799174,3cyclictest31442-21perf19:30:012
2198799171,9cyclictest0-21swapper/220:45:132
2198799171,6cyclictest5655-21crond19:45:002
2198799171,6cyclictest1-21systemd00:00:012
2198799171,5cyclictest0-21swapper/221:28:082
2198799171,3cyclictest0-21swapper/200:20:112
21987991712,3cyclictest32475-21irqstats19:30:172
2198799171,15cyclictest0-21swapper/219:49:332
21987991710,4cyclictest13744-21sensors_temp20:00:222
2197499170,10cyclictest22965-21kworker/0:119:11:520
2199199160,9cyclictest0-21swapper/320:55:103
2198799163,8cyclictest18622-21taskset20:12:042
2198799161,6cyclictest10639-21diskstats19:55:142
21987991611,4cyclictest0-21swapper/220:36:272
2198799161,14cyclictest0-21swapper/220:33:382
21987991611,3cyclictest9335-21sendmail_mailst19:50:212
21987991610,3cyclictest19723-21cpu21:30:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional