You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-26 - 03:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sun Apr 26, 2026 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
272122235164,58sleep20-21swapper/219:05:122
288562227151,25sleep00-21swapper/019:05:580
288342201164,24sleep30-21swapper/319:05:413
290922200164,24sleep10-21swapper/119:09:011
783421570,5sleep02925499cyclictest23:10:260
2170921510,4sleep02925499cyclictest23:45:120
311632600,4sleep2311ktimersoftd/200:05:182
277982600,8sleep07-21ksoftirqd/020:15:250
186572550,3sleep30-21swapper/319:55:213
188202500,1sleep20-21swapper/221:10:172
175802500,2sleep00-21swapper/023:35:130
93482450,1sleep20-21swapper/200:30:142
2925499399,14cyclictest31161-21munin-run21:40:000
2926999371,26cyclictest0-21swapper/323:48:313
2926999321,3cyclictest8167-21diskstats22:00:153
2926499321,10cyclictest0-21swapper/222:20:072
2925499327,12cyclictest7605-21idleruntime-cro22:00:000
2925499326,13cyclictest5395-21munin-run21:55:000
2926999311,13cyclictest0-21swapper/323:34:473
2926999311,12cyclictest0-21swapper/323:54:233
2926499311,8cyclictest0-21swapper/200:10:152
2926499311,7cyclictest0-21swapper/223:48:312
2926499311,10cyclictest0-21swapper/222:14:392
2926499310,9cyclictest0-21swapper/221:20:192
2925499314,12cyclictest12960-21awk19:45:000
2926999301,19cyclictest0-21swapper/322:52:473
2926999301,19cyclictest0-21swapper/322:52:473
2926999301,13cyclictest0-21swapper/300:14:113
2926999301,12cyclictest0-21swapper/320:59:513
2926999301,12cyclictest0-21swapper/320:36:113
2926999301,12cyclictest0-21swapper/320:20:233
2926999300,23cyclictest0-21swapper/321:10:063
2925499305,12cyclictest8861-21munin-run00:30:020
29254993011,16cyclictest3199-21munin-run21:50:000
2926999291,12cyclictest1571-21users19:15:273
2926999291,12cyclictest0-21swapper/322:34:313
2926999291,12cyclictest0-21swapper/300:07:573
2926999290,13cyclictest0-21swapper/323:29:583
2926999290,13cyclictest0-21swapper/322:23:473
2926999290,13cyclictest0-21swapper/320:00:113
2926999290,11cyclictest0-21swapper/322:45:273
2925999291,11cyclictest0-21swapper/123:49:291
2925999291,10cyclictest0-21swapper/123:05:491
2925999291,10cyclictest0-21swapper/122:07:191
2925499292,13cyclictest10750-21munin-run19:40:010
2926999281,13cyclictest0-21swapper/322:27:313
2926999281,12cyclictest0-21swapper/321:44:073
2926999281,10cyclictest0-21swapper/323:17:183
2926999281,10cyclictest0-21swapper/322:15:473
2926999280,12cyclictest0-21swapper/300:20:353
2926999280,10cyclictest0-21swapper/321:53:353
2926499281,8cyclictest0-21swapper/200:21:272
2925999281,18cyclictest827-21runrttasks00:15:241
2925499283,12cyclictest23940-21munin-run20:10:010
2925499281,8cyclictest1860-21kworker/0:019:30:510
2926999271,13cyclictest0-21swapper/323:02:133
2926999270,12cyclictest0-21swapper/321:15:063
2926499271,8cyclictest15374-21users21:00:262
2925999271,9cyclictest0-21swapper/121:30:321
2925999271,6cyclictest0-21swapper/100:32:181
2925999271,3cyclictest0-21swapper/122:55:241
2926999261,8cyclictest0-21swapper/322:41:593
29269992615,4cyclictest0-21swapper/300:38:023
2926999261,13cyclictest0-21swapper/319:52:233
2926999261,12cyclictest0-21swapper/319:36:073
2926999261,11cyclictest0-21swapper/322:13:353
2926999261,10cyclictest0-21swapper/321:55:393
2926999261,10cyclictest0-21swapper/321:05:463
2926999261,10cyclictest0-21swapper/321:01:033
2926999260,7cyclictest0-21swapper/322:59:393
2926499261,10cyclictest0-21swapper/221:53:112
2926499260,12cyclictest0-21swapper/222:46:382
2925999262,10cyclictest2569-21latency_hist20:35:011
2925499262,9cyclictest30810-21kworker/0:122:11:430
2925499261,14cyclictest20169-21munin-run21:15:020
2926999251,9cyclictest0-21swapper/323:06:543
2926999251,8cyclictest0-21swapper/300:32:393
2926999251,12cyclictest0-21swapper/323:56:393
2926999251,12cyclictest0-21swapper/323:56:393
2926999251,12cyclictest0-21swapper/323:22:223
2926999251,12cyclictest0-21swapper/319:20:463
2926999251,11cyclictest0-21swapper/323:36:213
2926499251,9cyclictest0-21swapper/223:04:222
2926499251,9cyclictest0-21swapper/221:31:432
2926499250,21cyclictest0-21swapper/222:26:472
2925999253,13cyclictest9078-21smart_sda22:00:241
2925999252,16cyclictest21-21rcuc/120:47:351
2925999251,7cyclictest23-21ksoftirqd/122:45:131
2925999251,17cyclictest358-21timerwakeupswit21:40:251
2925999251,16cyclictest0-21swapper/119:50:151
2925499257,15cyclictest13248-21munin-run00:40:020
2925499252,9cyclictest27958-21kworker/0:100:17:110
2925499252,6cyclictest6019-21munin-run23:10:010
2925499252,11cyclictest17130-21kworker/0:119:55:180
2925499251,12cyclictest12601-21idleruntime-cro23:25:020
29269992415,3cyclictest25419-21tail21:25:153
2926999241,3cyclictest0-21swapper/321:24:503
2926999241,14cyclictest0-21swapper/323:10:223
2926999241,10cyclictest0-21swapper/321:46:113
29269992410,3cyclictest0-21swapper/319:27:423
2926999240,10cyclictest0-21swapper/300:27:113
2926499241,9cyclictest0-21swapper/219:21:512
2926499241,11cyclictest0-21swapper/222:43:262
2926499241,10cyclictest0-21swapper/223:06:542
2926499241,10cyclictest0-21swapper/220:20:192
2926499240,9cyclictest0-21swapper/223:18:592
2925999244,9cyclictest3444-21cpu21:50:131
2925999243,13cyclictest32503-40dnf20:27:121
29259992418,3cyclictest21747-21ls20:05:021
2925999241,6cyclictest0-21swapper/123:41:371
2925999241,16cyclictest0-21swapper/100:20:241
2925999241,13cyclictest0-21swapper/121:37:511
2925999241,13cyclictest0-21swapper/100:11:221
2925499248,2cyclictest11096-21kworker/0:222:15:490
2925499242,9cyclictest10699-21kworker/0:200:33:350
2925499242,10cyclictest13511-21kworker/0:121:05:470
2925499241,12cyclictest21751-21munin-run20:05:010
2926999236,14cyclictest2294-21sshd20:31:343
2926999231,6cyclictest0-21swapper/321:32:073
29269992316,2cyclictest0-21swapper/323:40:243
2926999231,11cyclictest22646-21hddtemp_smartct20:05:173
2926999231,11cyclictest0-21swapper/320:48:553
2926999231,10cyclictest0-21swapper/319:10:113
2926499231,9cyclictest0-21swapper/221:46:272
2926499230,10cyclictest0-21swapper/221:28:432
2925999234,4cyclictest0-21swapper/123:36:371
2925999233,6cyclictest6083-21grep21:55:131
2925999232,6cyclictest0-21swapper/123:21:021
2925999232,15cyclictest0-21swapper/123:33:261
2925999232,15cyclictest0-21swapper/120:14:391
2925999231,17cyclictest0-21swapper/100:38:121
2925999231,16cyclictest0-21swapper/123:10:391
2925999231,12cyclictest0-21swapper/122:24:301
2925999231,11cyclictest0-21swapper/123:15:431
2925999231,10cyclictest0-21swapper/122:15:001
2925999231,10cyclictest0-21swapper/121:23:301
29259992310,3cyclictest4161-21munin-run19:25:011
2925499232,11cyclictest31310-21kworker/0:000:09:070
2925499232,11cyclictest25736-21kworker/0:223:58:470
2925499232,11cyclictest25736-21kworker/0:223:58:470
2925499231,6cyclictest0-21swapper/022:37:380
2925499231,5cyclictest0-21swapper/023:40:110
2925499231,5cyclictest0-21swapper/021:29:470
2925499231,10cyclictest9810-21perf22:05:000
2925499231,10cyclictest21074-21kworker/0:023:53:020
122142230,4sleep22926499cyclictest20:55:152
2926999221,14cyclictest0-21swapper/300:04:133
2926999221,11cyclictest12822-21sshd19:43:343
2926499226,3cyclictest87350irq/126-enp1s0-23:10:222
2926499221,9cyclictest0-21swapper/221:43:512
2926499221,9cyclictest0-21swapper/220:35:152
2926499221,9cyclictest0-21swapper/219:26:292
2925999227,2cyclictest23967-21tr23:50:111
2925999226,7cyclictest30335-21sed00:05:011
2925999226,6cyclictest8750-21cat19:35:021
2925999222,15cyclictest0-21swapper/119:12:101
2925999222,10cyclictest24080-21cat20:10:001
2925999221,6cyclictest0-21swapper/123:28:231
2925999221,3cyclictest827-21runrttasks21:12:131
2925999221,18cyclictest13985-21irqstats19:45:191
2925999221,13cyclictest0-21swapper/122:27:251
2925999221,11cyclictest0-21swapper/121:47:121
2925999220,6cyclictest0-21swapper/122:40:301
2925999220,6cyclictest0-21swapper/100:05:401
2925999220,15cyclictest0-21swapper/100:25:021
2925499229,2cyclictest14825-21kworker/0:223:31:510
2925499222,5cyclictest22305-1kworker/u9:222:07:260
2925499221,5cyclictest0-21swapper/022:30:300
2925499221,5cyclictest0-21swapper/000:10:190
2925499221,4cyclictest0-21swapper/000:02:320
38042210,4sleep230-21rcuc/200:15:252
2926999217,3cyclictest24121-21memory22:35:193
2926999211,9cyclictest0-21swapper/321:38:073
2926999211,12cyclictest0-21swapper/320:26:473
2926999211,11cyclictest0-21swapper/320:13:073
2926499216,3cyclictest87650irq/129-enp1s0-19:59:352
2926499216,3cyclictest0-21swapper/200:36:442
2926499213,3cyclictest0-21swapper/222:51:442
2926499213,3cyclictest0-21swapper/222:51:442
2926499212,6cyclictest26363-21/usr/sbin/munin20:15:142
2926499211,4cyclictest0-21swapper/223:55:042
2926499211,4cyclictest0-21swapper/223:55:042
2926499211,2cyclictest0-21swapper/223:44:032
2925999216,9cyclictest87550irq/128-enp1s0-21:27:151
2925999216,9cyclictest0-21swapper/119:55:241
2925999212,3cyclictest0-21swapper/120:56:181
2925999211,6cyclictest0-21swapper/122:36:371
2925999211,3cyclictest14416-21gltestperf21:00:171
2925999211,3cyclictest0-21swapper/121:15:281
29259992113,3cyclictest0-21swapper/119:19:441
2925999211,18cyclictest0-21swapper/119:43:271
2925999211,15cyclictest0-21swapper/123:01:521
2925999211,15cyclictest0-21swapper/120:41:591
2925999211,15cyclictest0-21swapper/120:22:301
2925999211,15cyclictest0-21swapper/120:15:531
29259992111,2cyclictest87550irq/128-enp1s0-22:15:171
2925999211,11cyclictest0-21swapper/122:50:471
2925999211,11cyclictest0-21swapper/122:50:461
2925999210,16cyclictest0-21swapper/122:32:111
2925499212,8cyclictest16140-21kworker/0:022:48:590
2925499212,11cyclictest12834-21/usr/sbin/munin23:25:260
2925499211,5cyclictest0-21swapper/022:56:080
2925499210,10cyclictest5846-21kworker/0:023:16:550
2926999206,4cyclictest3399-21memory00:15:193
2926999201,13cyclictest0-21swapper/320:18:073
2926499207,2cyclictest0-21swapper/222:00:192
2926499205,3cyclictest87650irq/129-enp1s0-19:15:262
2926499201,5cyclictest0-21swapper/223:33:392
2926499201,5cyclictest0-21swapper/223:29:432
2926499201,4cyclictest0-21swapper/223:22:362
2926499201,4cyclictest0-21swapper/200:27:592
2926499201,12cyclictest0-21swapper/221:16:112
2926499201,11cyclictest0-21swapper/219:51:432
2926499201,10cyclictest0-21swapper/220:29:112
29259992011,2cyclictest0-21swapper/123:55:211
29259992011,2cyclictest0-21swapper/123:55:211
2925499205,3cyclictest7188-21kworker/0:220:50:290
2925499202,8cyclictest16140-21kworker/0:022:41:510
2925499202,5cyclictest17677-21ntp_states22:20:220
2925499202,15cyclictest9159-21idleruntime-cro20:50:010
2925499202,11cyclictest4708-21kworker/0:120:45:010
2925499201,6cyclictest0-21swapper/021:15:520
2925499201,4cyclictest0-21swapper/022:51:190
2925499201,4cyclictest0-21swapper/022:51:190
2925499201,16cyclictest31434-21munin-run19:15:000
2925499201,11cyclictest509-21/usr/sbin/munin20:30:140
2925499201,11cyclictest0-21swapper/022:25:030
2925499201,10cyclictest4386-21/usr/sbin/munin19:25:250
2926999191,15cyclictest14087-21memory19:45:203
2926999190,18cyclictest0-21swapper/322:05:343
2926999190,10cyclictest0-21swapper/319:32:023
2926499194,4cyclictest25519-21NetworkManager23:51:392
2926499194,4cyclictest22187-21cstates20:05:142
2926499191,5cyclictest7194-21/usr/sbin/munin20:45:172
2926499191,3cyclictest0-21swapper/220:10:132
29264991913,3cyclictest10792-21if_err_enp1s022:05:192
2926499191,15cyclictest11341-21taskset20:54:542
2926499191,10cyclictest0-21swapper/221:57:392
2925999198,3cyclictest8962-21cstates19:35:141
2925999191,10cyclictest3868-21ntp_states20:35:191
2925999190,9cyclictest17967-21munin-run21:10:021
2925999190,14cyclictest0-21swapper/119:26:271
2925499199,3cyclictest87550irq/128-enp1s0-21:30:500
2925499192,9cyclictest808-21snmpd20:29:570
2925499192,10cyclictest1845-21/usr/sbin/munin23:00:200
2925499191,6cyclictest0-21swapper/000:23:100
2926499185,3cyclictest87650irq/129-enp1s0-23:35:202
2926499184,3cyclictest21797-21cat20:05:022
2926499183,7cyclictest87550irq/128-enp1s0-22:30:252
2926499181,9cyclictest0-21swapper/219:30:192
2926499181,6cyclictest0-21swapper/219:45:022
2926499181,3cyclictest0-21swapper/222:36:212
29264991812,3cyclictest664-21sed22:55:222
2925999189,6cyclictest10391-21netstat20:50:201
2925499189,2cyclictest32349-21/usr/sbin/munin19:15:190
2925499188,3cyclictest13788-21/usr/sbin/munin21:00:180
2925499181,9cyclictest808-21snmpd19:54:490
2925499181,4cyclictest22181-21kworker/0:321:20:570
29254991811,4cyclictest26142-21munin-run20:15:010
149162181,3sleep20-21swapper/219:47:152
2926499171,6cyclictest0-21swapper/222:15:302
2926499171,6cyclictest0-21swapper/200:04:412
2926499170,15cyclictest0-21swapper/221:35:082
2925499175,3cyclictest0-21swapper/021:40:130
2925499170,5cyclictest0-21swapper/019:45:130
34132160,5sleep02925499cyclictest19:20:230
2926999167,7cyclictest10011-21gltestperf20:50:163
29269991611,3cyclictest5608-21grep20:40:153
2926499161,6cyclictest0-21swapper/219:35:172
2926499161,3cyclictest0-21swapper/221:09:252
2926499161,2cyclictest0-21swapper/219:11:592
29264991611,3cyclictest7167-21sed20:45:012
2925499162,5cyclictest3599-21/usr/sbin/munin20:35:170
2925499161,6cyclictest11588-21/usr/sbin/munin20:55:150
29254991611,3cyclictest28904-21sh20:20:140
29264991510,3cyclictest956-21fschecks_count20:30:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional