You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-16 - 14:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Mar 16, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
44462241163,25sleep10-21swapper/107:08:251
44472236163,25sleep20-21swapper/207:08:262
42602233160,24sleep00-21swapper/007:05:590
45632222193,20sleep30-21swapper/307:09:563
1447721850,2sleep0465999cyclictest07:30:110
4676999794,2cyclictest0-21swapper/308:57:453
4676999086,3cyclictest0-21swapper/310:11:313
4676996865,2cyclictest0-21swapper/312:01:323
69762600,5sleep3467699cyclictest07:15:013
166412510,4sleep016644-21cpuspeed_turbos07:35:130
4676994946,2cyclictest0-21swapper/312:12:363
100262460,2sleep10-21swapper/112:15:011
4676994441,2cyclictest0-21swapper/309:36:433
465999379,13cyclictest18793-21perf12:35:010
4659993724,7cyclictest19461-40dnf10:06:350
4659993711,14cyclictest6647-21cut12:05:170
4659993711,14cyclictest6647-21cut12:05:170
4659993610,8cyclictest4574-21/usr/sbin/munin09:35:160
465999356,14cyclictest28540-21munin-run10:30:010
4659993512,9cyclictest12994-21perl12:20:150
4659993414,8cyclictest23294-21pmu-power10:15:220
4659993411,11cyclictest13997-21/usr/sbin/munin11:10:220
465999336,15cyclictest8968-21chrt09:45:070
465999336,12cyclictest21536-21/usr/sbin/munin09:00:170
4659993319,5cyclictest14530-21/usr/sbin/munin09:55:220
4659993313,5cyclictest1337-21cat11:55:010
4659993311,7cyclictest24711-21diskstats10:20:140
4659993310,15cyclictest31925-21/usr/sbin/munin08:10:160
4676993229,2cyclictest0-21swapper/309:13:583
4676993228,3cyclictest0-21swapper/309:48:453
465999329,7cyclictest580-21systemd-journal08:35:330
465999327,12cyclictest31352-21munin-run11:50:010
4659993212,7cyclictest2231-21cat09:30:010
4659993210,7cyclictest13765-21latency_hist11:10:010
4676993128,2cyclictest0-21swapper/307:43:283
465999319,9cyclictest25630-21head11:35:150
465999319,10cyclictest14735-21perf08:45:010
465999318,9cyclictest22136-21head11:25:260
4659993113,6cyclictest18159-21latency_hist11:20:030
4659993111,8cyclictest19656-21swap11:20:240
4659993010,5cyclictest86550irq/126-enp1s0-10:15:020
465999299,7cyclictest28498-21needreboot08:00:190
465999299,11cyclictest10079-21latency_hist12:15:010
465999298,8cyclictest28978-21cstates10:30:120
465999297,13cyclictest11963-21if_err_enp2s009:50:170
465999296,9cyclictest0-21swapper/008:45:260
4659992913,6cyclictest3612-21/usr/sbin/munin12:00:260
4659992912,14cyclictest25729-21sleep09:10:010
100492290,7sleep2311ktimersoftd/209:45:212
465999289,16cyclictest14607-21/usr/sbin/munin12:25:220
465999286,11cyclictest0-21swapper/011:40:260
4659992811,14cyclictest8214-21tune2fs07:15:140
465999278,16cyclictest12203-21idleruntime-cro12:20:010
465999277,9cyclictest7394-21/usr/sbin/munin10:55:210
465999277,9cyclictest7394-21/usr/sbin/munin10:55:210
465999277,6cyclictest30339-21/usr/sbin/munin09:20:180
465999275,12cyclictest812-21snmpd09:12:500
465999274,14cyclictest6160-21/usr/sbin/munin08:25:190
4659992714,4cyclictest5675-21/usr/sbin/munin07:10:180
4659992712,7cyclictest20847-21sshd12:38:330
4659992712,4cyclictest21307-21latency_hist09:00:020
4659992710,4cyclictest9591-21/usr/sbin/munin11:00:140
465999270,12cyclictest25717-21diskmemload10:55:010
466499262,7cyclictest771-21rngd11:11:141
466499261,12cyclictest0-21swapper/112:22:201
465999269,14cyclictest1383-21grep10:40:160
465999268,11cyclictest25717-21diskmemload10:03:290
465999264,5cyclictest86550irq/126-enp1s0-11:55:560
4659992618,5cyclictest24193-21ntp_states07:50:210
4659992617,6cyclictest10413-21/usr/sbin/munin07:20:140
4659992616,7cyclictest19839-21/usr/sbin/munin07:40:240
4659992610,6cyclictest5719-1kworker/u9:211:30:590
467699253,3cyclictest0-21swapper/309:56:083
4676992512,3cyclictest25794-21irqstats11:35:173
467699251,12cyclictest31925-21/usr/sbin/munin08:10:233
467699251,11cyclictest27518-21/usr/sbin/munin08:00:133
466499255,3cyclictest0-21swapper/111:44:541
466499255,2cyclictest10932-21kworker/u8:309:55:221
466499253,2cyclictest17823-21kworker/u8:111:18:571
466499251,9cyclictest0-21swapper/110:12:461
466499251,11cyclictest0-21swapper/111:51:581
466499251,11cyclictest0-21swapper/110:50:291
465999257,7cyclictest2560-21cstates09:30:130
465999256,12cyclictest31520-21fschecks_time10:35:140
4659992517,6cyclictest8357-21/usr/sbin/munin08:30:190
4659992517,6cyclictest8357-21/usr/sbin/munin08:30:190
4659992516,6cyclictest6838-21/usr/sbin/munin09:40:120
4659992516,6cyclictest6838-21/usr/sbin/munin09:40:120
4659992516,6cyclictest25321-21/usr/sbin/munin07:55:170
4659992516,6cyclictest20930-21/usr/sbin/munin07:45:250
4659992515,7cyclictest30070-21diskstats08:05:130
467699243,13cyclictest0-21swapper/310:47:113
467699242,10cyclictest5824-21/usr/sbin/munin12:05:123
467699242,10cyclictest5824-21/usr/sbin/munin12:05:123
467699241,6cyclictest0-21swapper/311:07:133
467699241,4cyclictest0-21swapper/310:07:453
4676992412,3cyclictest0-21swapper/312:33:463
467699241,11cyclictest22785-21/usr/sbin/munin11:30:113
467699240,3cyclictest0-21swapper/309:50:333
466499246,9cyclictest19198-21kworker/u8:207:30:041
466499246,6cyclictest0-21swapper/110:20:541
466499241,3cyclictest6556-21munin-run09:40:021
466499241,17cyclictest0-21swapper/111:26:221
466499241,16cyclictest0-21swapper/109:52:191
466499241,15cyclictest0-21swapper/109:32:341
466499241,12cyclictest0-21swapper/109:26:421
465999249,12cyclictest3727-21latency_hist08:20:010
4659992417,4cyclictest3342-21/usr/sbin/munin10:45:130
4659992417,4cyclictest28495-21/usr/sbin/munin09:15:140
4659992416,6cyclictest12141-21/usr/sbin/munin07:25:120
4659992415,6cyclictest17143-21/usr/sbin/munin08:50:140
4659992411,5cyclictest4307-21diskstats08:20:140
467699233,10cyclictest11164-21unixbench_singl11:00:253
467699232,3cyclictest0-21swapper/309:18:393
467699231,6cyclictest0-21swapper/310:31:503
4676992311,3cyclictest0-21swapper/311:44:543
467699231,10cyclictest13997-21/usr/sbin/munin11:10:193
4676992310,3cyclictest1524-21if_enp2s010:40:173
467699230,6cyclictest0-21swapper/310:38:353
466499233,4cyclictest0-21swapper/110:25:401
466499233,11cyclictest4967-21cpuspeed_turbos07:10:131
466499231,6cyclictest0-21swapper/111:45:321
466499231,15cyclictest0-21swapper/110:44:231
466499231,14cyclictest0-21swapper/110:09:181
466499231,13cyclictest0-21swapper/109:22:401
466499231,11cyclictest0-21swapper/111:33:081
466499231,11cyclictest0-21swapper/111:20:281
467699221,4cyclictest0-21swapper/310:17:543
467699221,11cyclictest18631-21ls11:20:123
467699221,10cyclictest3368-21grep09:30:193
467699221,10cyclictest31574-21/usr/sbin/munin11:50:233
467699221,10cyclictest11433-21ps12:15:233
466999221,6cyclictest0-21swapper/209:57:082
466999221,5cyclictest0-21swapper/212:10:362
466999221,4cyclictest0-21swapper/209:37:082
466499222,11cyclictest5204-21users12:00:281
4664992212,3cyclictest0-21swapper/108:45:341
466499221,15cyclictest0-21swapper/110:03:471
466499221,15cyclictest0-21swapper/109:43:451
466499221,15cyclictest0-21swapper/109:43:441
466499221,15cyclictest0-21swapper/109:08:261
466499221,15cyclictest0-21swapper/108:39:011
466499221,15cyclictest0-21swapper/107:52:201
466499221,15cyclictest0-21swapper/107:41:551
466499221,15cyclictest0-21swapper/107:37:511
466499221,13cyclictest0-21swapper/111:37:431
466499221,13cyclictest0-21swapper/109:46:421
466499221,11cyclictest31091-21grep10:35:111
466499221,11cyclictest11578-21munin-run11:05:011
466499221,10cyclictest16688-21sort12:30:011
4664992210,3cyclictest2999-21users11:55:251
466499220,16cyclictest0-21swapper/109:04:281
467699219,2cyclictest2581-21ps11:55:223
467699216,3cyclictest16188-21/usr/sbin/munin11:15:133
467699212,2cyclictest0-21swapper/309:24:063
467699211,5cyclictest18214-21ntp_states08:50:213
467699211,4cyclictest0-21swapper/309:28:073
4676992111,3cyclictest0-21swapper/310:25:413
467699211,13cyclictest0-21swapper/310:03:133
466999212,4cyclictest4687-21ntp_states12:00:222
466999211,5cyclictest0-21swapper/211:29:162
466499212,11cyclictest13221-21open_files07:25:221
466499211,6cyclictest0-21swapper/111:07:301
4664992114,1cyclictest0-21swapper/107:24:381
4664992113,3cyclictest0-21swapper/107:58:391
4664992113,2cyclictest0-21swapper/108:58:161
466499211,15cyclictest0-21swapper/112:17:391
466499211,15cyclictest0-21swapper/112:07:191
466499211,15cyclictest0-21swapper/112:07:191
466499211,15cyclictest0-21swapper/110:55:441
466499211,15cyclictest0-21swapper/110:55:441
466499211,15cyclictest0-21swapper/108:54:041
466499211,15cyclictest0-21swapper/108:42:251
466499211,15cyclictest0-21swapper/108:05:261
466499211,11cyclictest0-21swapper/112:38:341
466499210,5cyclictest0-21swapper/109:12:501
466499210,15cyclictest0-21swapper/108:15:441
466499210,13cyclictest0-21swapper/107:45:151
244592211,3sleep224628-21tail07:50:252
187142210,5sleep230-21rcuc/211:20:122
467699209,4cyclictest86850irq/129-enp1s0-08:35:143
467699202,6cyclictest9-21rcu_preempt12:36:163
467699202,5cyclictest8462-21chrt09:41:293
467699202,5cyclictest8462-21chrt09:41:293
467699202,4cyclictest8120-21kworker/3:008:45:173
467699201,6cyclictest0-21swapper/310:58:493
467699201,6cyclictest0-21swapper/310:58:493
467699201,6cyclictest0-21swapper/310:51:113
4676992014,3cyclictest23238-21cpuspeed_turbos07:50:123
467699201,16cyclictest13082-21cut12:20:153
467699201,11cyclictest4642-21kworker/u8:211:30:013
467699201,10cyclictest25655-21switchtime10:20:243
466999205,5cyclictest5719-1kworker/u9:210:22:512
466999204,3cyclictest5719-1kworker/u9:209:44:332
466999204,3cyclictest5719-1kworker/u9:209:44:332
466999203,11cyclictest25717-21diskmemload10:03:562
466999201,5cyclictest13997-21/usr/sbin/munin11:10:262
4669992015,3cyclictest6662-21grep10:50:262
466999201,10cyclictest6583-21if_enp2s012:05:162
466999201,10cyclictest6583-21if_enp2s012:05:162
466499209,3cyclictest8812-21ntp_states07:15:201
466499209,2cyclictest0-21swapper/108:10:141
466499208,3cyclictest4050-21cpuspeed_turbos08:20:131
466499201,9cyclictest27677-21cpuspeed_turbos08:00:121
466499201,17cyclictest0-21swapper/112:30:021
4664992011,2cyclictest0-21swapper/108:25:131
466499201,11cyclictest29811-21ntpq10:30:201
466499201,11cyclictest0-21swapper/110:45:141
466499201,10cyclictest29309-21proc_pri09:15:211
466499200,15cyclictest0-21swapper/108:31:261
466499200,15cyclictest0-21swapper/108:31:261
467699199,3cyclictest28967-21kworker/u8:111:45:323
466999195,7cyclictest24982-21/usr/sbin/munin11:35:222
466999194,9cyclictest19877-21latency12:35:182
466999193,8cyclictest2722-21df_inode09:30:132
466999191,5cyclictest28139-21/usr/sbin/munin09:15:252
466999191,5cyclictest19338-21/usr/sbin/munin08:55:132
466999191,5cyclictest0-21swapper/211:17:542
466999191,4cyclictest27184-21/usr/sbin/munin11:40:212
466999191,4cyclictest19970-21/usr/sbin/munin10:10:172
466999191,4cyclictest0-21swapper/211:00:592
466999191,12cyclictest0-21swapper/210:58:392
466999191,12cyclictest0-21swapper/210:58:392
466499191,9cyclictest0-21swapper/110:16:351
30382191,2sleep20-21swapper/211:55:452
159362190,4sleep230-21rcuc/207:30:252
72872181,2sleep20-21swapper/208:25:242
467699183,12cyclictest12141-21/usr/sbin/munin07:25:223
467699181,6cyclictest0-21swapper/307:35:183
467699181,3cyclictest0-21swapper/307:45:183
4676991813,3cyclictest0-21swapper/312:25:123
467699181,16cyclictest4562-21tail08:20:163
466999188,3cyclictest31951-21sshd09:20:342
466999186,3cyclictest26226-21/usr/sbin/munin07:55:182
466999184,7cyclictest0-21swapper/211:05:482
466999183,3cyclictest5719-1kworker/u9:210:40:022
466999182,6cyclictest812-21snmpd09:12:492
466999182,5cyclictest3022-21smart_sda08:15:252
466999182,3cyclictest86850irq/129-enp1s0-10:39:032
466999182,3cyclictest86550irq/126-enp1s0-10:46:502
466999181,9cyclictest23515-21turbostat09:09:562
466999181,7cyclictest0-21swapper/209:51:042
466999181,6cyclictest15389-21if_err_enp2s012:25:152
466999181,6cyclictest0-21swapper/210:15:052
466999181,6cyclictest0-21swapper/209:26:222
466999181,5cyclictest29823-21ntp_states10:30:212
466999181,5cyclictest0-21swapper/212:24:302
466999181,4cyclictest3959-21/usr/sbin/munin08:20:142
466999181,4cyclictest0-21swapper/212:17:022
466999181,3cyclictest86850irq/129-enp1s0-08:54:082
466999181,3cyclictest13449-21idleruntime08:40:162
466999181,15cyclictest0-21swapper/208:00:032
466999181,11cyclictest0-21swapper/211:48:522
466999181,11cyclictest0-21swapper/210:07:442
99722171,2sleep30-21swapper/308:30:353
99722171,2sleep30-21swapper/308:30:353
467699171,6cyclictest23743-21/usr/sbin/munin09:05:223
467699171,5cyclictest0-21swapper/307:30:183
467699171,3cyclictest29926-21expr08:05:123
4676991711,4cyclictest851-21sssd_nss07:20:193
467699171,13cyclictest12752-21/usr/sbin/munin08:40:133
466999178,7cyclictest32-21ksoftirqd/207:24:582
466999176,3cyclictest215362/usr/sbin/munin09:00:182
466999175,2cyclictest86750irq/128-enp1s0-11:30:222
466999172,5cyclictest17146-21df_inode12:30:122
466999172,4cyclictest851-21sssd_nss08:05:132
466999172,4cyclictest32637-21ntp_states11:50:202
466999171,7cyclictest13208-21ntp_states07:25:222
466999171,6cyclictest21269-21chrt07:45:142
466999171,5cyclictest19120-21fschecks_count07:40:162
466999171,14cyclictest0-21swapper/210:26:192
466999171,14cyclictest0-21swapper/208:35:062
4669991710,4cyclictest9430-21ntpq08:30:202
4669991710,4cyclictest9430-21ntpq08:30:202
466999170,3cyclictest5929-21ntpq07:10:212
467699165,3cyclictest86850irq/129-enp1s0-08:19:333
467699162,2cyclictest86550irq/126-enp1s0-09:00:113
4676991612,3cyclictest0-21swapper/308:28:123
4676991611,3cyclictest8722-21sed07:15:193
466999161,5cyclictest8045-21cstates07:15:132
466999161,2cyclictest86850irq/129-enp1s0-07:35:122
4669991611,3cyclictest32117-21expr08:10:142
4676991510,3cyclictest833-21runrttasks07:55:023
466999150,2cyclictest0-21swapper/208:45:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional