You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-09 - 18:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Dec 09, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
415226946,2674sleep3414-21ntp_states11:28:073
231052104016,1007sleep023115-21/usr/sbin/munin07:32:400
32282284814,817sleep032345-21/usr/sbin/munin09:42:580
27888280213,771sleep027990-21/usr/sbin/munin09:28:120
2905427517,731sleep0444-1kworker/0:2H09:33:090
11015267824,641sleep311059-21ntpq10:18:073
2168525688,550sleep35666-21snmpd09:08:363
187102512318,129sleep30-21swapper/312:27:403
31224232541,246sleep331281-21grep07:57:593
27130231119,197sleep327273-21unixbench_multi07:43:093
27130231119,197sleep327273-21unixbench_multi07:43:093
20906230927,243sleep321039-21tail07:23:103
20906230927,243sleep321039-21tail07:23:103
22269229383,151sleep322411-21unixbench_singl10:53:133
19571229351,59sleep30-21swapper/309:03:153
19571229351,59sleep30-21swapper/309:03:153
2269922658,216sleep322736-21sed09:13:043
2269922658,216sleep322736-21sed09:13:043
24232224812,222sleep224261-21users07:33:152
15830224424,18sleep30-21swapper/308:52:403
15830224424,18sleep30-21swapper/308:52:403
847222388,194sleep38480-21awk11:53:073
10913221512,189sleep210922-21unin-run12:02:392
4689221417,151sleep34833-21latency_hist11:42:413
32121221086,84sleep332132-21modprobe11:27:383
1556922098,163sleep315618-21systemd-run12:17:393
10316219952,133sleep310335-21kworker/u8:310:17:393
321321979,147sleep33389-21apt-key08:12:423
2491219735,76sleep32518-21idleruntime-cro09:52:383
25004219447,50sleep30-21swapper/307:38:003
3934219214,157sleep227448-21kworker/u8:011:39:002
1753121906,16sleep00-21swapper/007:13:290
19339217851,54sleep30-21swapper/307:18:133
19339217851,54sleep30-21swapper/307:18:133
24333217533,69sleep324352-21modprobe11:02:383
11549217048,59sleep30-21swapper/312:03:133
13955216112,109sleep313331-21/usr/sbin/munin07:03:103
1347521608,45sleep313474-21munin-run10:27:403
15666215821,53sleep30-21swapper/307:08:033
336215140,55sleep30-21swapper/309:43:093
28087215121,26sleep00-21swapper/007:47:590
14667214922,37sleep20-21swapper/207:07:422
28477214811,121sleep328536-21lockfile-create09:32:393
7605214523,16sleep30-21swapper/308:23:123
13989214041,61sleep314005-21modprobe12:12:383
5078213842,17sleep30-21swapper/308:17:433
7099213756,24sleep30-21swapper/311:48:103
29260213629,57sleep329263-21cron07:52:383
25312213537,79sleep30-21swapper/309:22:413
17432213510,110sleep317449-21latency_hist08:57:453
1196121299,81sleep312042-21sed08:38:103
3766212515,19sleep30-21swapper/309:53:403
31400212210,98sleep231473-21sensors_temp11:23:092
601121208,86sleep30-21swapper/310:03:013
1912211910,71sleep00-21swapper/011:33:050
5254211840,36sleep30-21swapper/311:43:053
2478521177,19sleep30-21swapper/311:02:583
8028211551,47sleep38094-21switchtime10:08:103
22089211511,91sleep122158-21cat10:53:111
8493211310,51sleep30-21swapper/308:28:013
1203121130,8sleep112-21rcu_sched10:22:411
16743211014,80sleep116845-21sed08:53:191
434821098,52sleep30-21swapper/309:57:523
16224210928,42sleep30-21swapper/312:18:033
23735210813,73sleep023854-21/usr/sbin/munin10:58:110
162721087,42sleep30-21swapper/311:33:003
4920210724,64sleep10-21swapper/108:17:411
3006421036,63sleep35666-21snmpd09:34:413
871210242,39sleep30-21swapper/308:03:173
15950210216,71sleep015775-21latency_hist08:52:540
15950210216,71sleep015775-21latency_hist08:52:540
29405210113,72sleep229450-21fschecks_count11:18:002
21278210110,44sleep30-21swapper/310:52:403
4556210011,74sleep04609-21/usr/sbin/munin09:58:010
29023210025,51sleep229025-21apt-get09:32:502
10021210010,24sleep010028-21ntp_states11:58:050
939229918,64sleep19525-21modprobe11:57:411
379729834,22sleep30-21swapper/308:13:513
1800229811,38sleep30-21swapper/308:58:063
906029724,39sleep10-21swapper/110:12:521
2396629619,56sleep124045-21fschecks_time09:17:591
170932966,46sleep30-21swapper/307:12:483
149829616,58sleep11533-21munin-plugin-st08:07:391
2907429516,29sleep329202-21latency_hist11:17:413
1478029510,72sleep05666-21snmpd10:30:080
1027629417,36sleep30-21swapper/308:33:053
2674029318,17sleep30-21swapper/309:25:163
2642929322,32sleep30-21swapper/311:08:033
2045329317,60sleep220526-21/usr/sbin/munin10:48:112
142052937,17sleep30-21swapper/308:47:393
142052937,17sleep30-21swapper/308:47:393
300032927,39sleep30-21swapper/311:18:133
1408629243,34sleep30-21swapper/310:28:053
1248029224,41sleep012635-21grep06:58:110
953329122,37sleep30-21swapper/311:57:483
2364529128,27sleep10-21swapper/109:17:431
2364529128,27sleep10-21swapper/109:17:431
299782907,49sleep30-21swapper/307:53:073
143802909,65sleep00-21swapper/012:12:550
701128916,56sleep27098-21switchtime11:48:092
333028818,50sleep03313-21apt-get11:37:570
318982888,42sleep332009-21date09:42:413
2520928823,33sleep20-21swapper/209:22:402
1109628818,48sleep011124-21modprobe08:37:390
899328521,43sleep09121-21unixbench_singl08:28:140
272782850,10sleep00-21swapper/007:43:090
272782850,10sleep00-21swapper/007:43:090
1839928527,39sleep30-21swapper/310:42:553
1839928527,39sleep30-21swapper/310:42:553
200062845,13sleep30-21swapper/312:29:553
1810128411,59sleep318133-21awk10:42:393
3059228317,49sleep030586-21apt-get09:37:570
755928218,39sleep00-21swapper/010:07:580
2465228228,43sleep324642-21munin-run07:37:403
172082829,53sleep10-21swapper/107:12:581
1592828125,42sleep316049-21ls10:33:103
1342428120,31sleep10-21swapper/110:27:381
1184628114,52sleep111883-21grep08:38:081
561128016,49sleep15617-21ntp_states08:18:051
2201128019,38sleep222031-21apt09:12:392
1993627917,48sleep119966-21latency_hist07:22:391
1993627917,48sleep119966-21latency_hist07:22:391
172622790,9sleep10-21swapper/110:38:031
1694327912,51sleep216589-21apt-get07:12:432
1036927916,48sleep210401-21sed10:17:422
820827817,45sleep18340-21unixbench_singl10:08:121
2050727814,49sleep220563-21latency_hist09:07:422
2050727814,49sleep220563-21latency_hist09:07:422
1717327523,35sleep117318-21latency_hist12:22:421
1407027516,43sleep114151-21latency_hist12:12:401
376127416,43sleep2225-21systemd-journal09:53:392
3268827411,28sleep20-21swapper/208:02:552
2823927415,28sleep00-21swapper/011:13:090
2581727417,21sleep00-21swapper/009:23:190
131372748,48sleep30-21swapper/307:02:423
561827316,29sleep20-21swapper/208:18:062
2722027315,39sleep127192-21/usr/sbin/munin09:27:561
1973227316,38sleep019862-21/usr/sbin/munin12:28:100
1263927316,25sleep10-21swapper/106:58:121
1584927218,35sleep20-21swapper/210:33:092
49527117,26sleep20-21swapper/211:28:102
250132716,11sleep00-21swapper/009:21:050
2466427117,28sleep10-21swapper/107:37:421
1890527110,48sleep118981-21systemctl10:43:091
1890527110,48sleep118981-21systemctl10:43:091
1457727116,37sleep114663-21sed12:13:001
854727020,29sleep10-21swapper/111:53:091
2872627020,21sleep20-21swapper/209:32:422
205727014,22sleep00-21swapper/008:08:020
2029627016,38sleep220400-21date12:32:402
1303227017,25sleep00-21swapper/008:43:000
1303227017,25sleep00-21swapper/008:43:000
1557326925,21sleep20-21swapper/207:08:022
1264926915,37sleep212702-21/usr/sbin/munin10:23:112
1114626917,27sleep20-21swapper/208:37:412
1059026913,41sleep010609-21apt-get10:17:520
680226812,38sleep06831-21/usr/sbin/munin11:48:030
100426816,19sleep00-21swapper/008:03:180
475726711,42sleep00-21swapper/008:16:140
1334026716,22sleep10-21swapper/108:43:051
1334026716,22sleep10-21swapper/108:43:051
132826716,28sleep00-21swapper/009:47:590
2185726620,32sleep30-21swapper/307:27:583
2119826613,28sleep10-21swapper/110:52:381
1430726615,30sleep20-21swapper/212:12:532
90462657,17sleep00-21swapper/011:54:420
515526516,19sleep10-21swapper/111:43:031
38022656,15sleep00-21swapper/009:54:050
2161126514,33sleep221667-21tune2fs10:53:002
2049426523,26sleep220559-21grep07:23:012
2049426523,26sleep220559-21grep07:23:012
1987526512,20sleep00-21swapper/009:03:280
1987526512,20sleep00-21swapper/009:03:280
1916326513,34sleep119213-21/usr/sbin/munin12:27:591
1666926512,23sleep00-21swapper/012:18:120
1515026518,19sleep00-21swapper/007:07:480
2594526420,22sleep10-21swapper/111:07:411
246926420,24sleep10-21swapper/108:08:111
2438726420,24sleep10-21swapper/111:02:401
1566426417,23sleep00-21swapper/010:33:030
485926323,18sleep10-21swapper/109:58:101
3019826322,21sleep20-21swapper/207:53:142
2744326317,27sleep227471-21munin-run11:12:392
2627726323,21sleep10-21swapper/111:08:011
2356926313,21sleep20-21swapper/210:58:092
1898226314,25sleep218374-21/usr/sbin/munin10:43:102
1898226314,25sleep218374-21/usr/sbin/munin10:43:102
1372426313,34sleep113757-21/usr/sbin/munin07:03:041
1178926321,22sleep20-21swapper/212:03:202
338326217,24sleep10-21swapper/111:38:311
326072626,41sleep10-21swapper/111:28:011
3067826219,25sleep130772-21fschecks_time09:38:021
2618326215,27sleep026173-21apt-get11:07:560
2278626212,34sleep022811-21awk10:57:380
192526216,28sleep12071-21unixbench_multi09:48:121
1821126218,24sleep00-21swapper/008:58:310
1569326216,27sleep10-21swapper/110:33:041
1493426220,22sleep10-21swapper/107:07:451
145972628,11sleep00-21swapper/007:07:230
73682616,10sleep10-21swapper/111:48:291
2971626119,21sleep00-21swapper/011:18:060
2861326116,26sleep20-21swapper/207:48:132
275852616,16sleep10-21swapper/107:46:061
275852616,16sleep10-21swapper/107:46:061
2592626118,23sleep00-21swapper/011:07:400
2209226114,33sleep022167-21latency_hist09:12:410
1873126126,18sleep00-21swapper/012:27:410
1346126118,23sleep00-21swapper/012:08:110
2836926015,29sleep128402-21/usr/sbin/munin07:48:041
2362826013,30sleep023720-21load07:33:040
184982609,14sleep20-21swapper/212:25:252
38302594,42sleep10-21swapper/109:54:261
3240325917,18sleep20-21swapper/209:43:002
3212125910,37sleep05666-21snmpd07:59:100
3132725914,28sleep00-21swapper/011:23:070
2935725913,17sleep10-21swapper/109:34:071
1975925918,17sleep00-21swapper/010:47:410
3085725817,25sleep130854-21apt-get11:22:521
227962586,10sleep20-21swapper/207:29:352
1628925818,19sleep00-21swapper/008:53:060
1594325814,19sleep20-21swapper/212:17:582
963425715,18sleep10-21swapper/108:32:431
903825714,25sleep09057-21apt-get10:12:510
736125714,25sleep17471-21sed08:23:101
610325619,17sleep00-21swapper/010:03:020
3140725614,27sleep20-21swapper/207:58:032
2567125614,24sleep125665-21apt-get09:22:561
2536525613,23sleep00-21swapper/007:38:090
2490225616,18sleep20-21swapper/207:37:552
2232925614,25sleep022324-21apt-get09:12:530
2232925614,25sleep022324-21apt-get09:12:530
1995125619,19sleep00-21swapper/010:47:570
1866925611,20sleep10-21swapper/112:27:381
156242565,10sleep20-21swapper/208:51:022
156242565,10sleep20-21swapper/208:51:022
1216725612,18sleep20-21swapper/206:58:052
1135925615,28sleep011365-21acpi08:37:510
728425519,21sleep07359-21switchtime08:23:090
683925512,25sleep25901-21/usr/sbin/munin10:03:162
2958725514,20sleep10-21swapper/111:18:031
2459825513,29sleep224619-21dpkg11:02:492
1876825514,17sleep00-21swapper/010:43:040
1790325519,19sleep10-21swapper/108:58:041
153725515,25sleep21531-21apt-get11:32:542
1518125510,30sleep115306-21cat08:48:181
1518125510,30sleep115306-21cat08:48:181
131225517,18sleep20-21swapper/209:47:572
1186525510,17sleep20-21swapper/210:22:392
2958425411,26sleep129577-21apt-get07:52:571
2254725412,18sleep10-21swapper/107:28:131
1883225416,17sleep00-21swapper/007:18:020
1883225416,17sleep00-21swapper/007:18:020
1840325415,16sleep20-21swapper/208:58:332
795825313,23sleep28027-21systemctl10:08:092
496225315,19sleep20-21swapper/208:17:422
3248225312,17sleep00-21swapper/011:27:560
2286825313,18sleep20-21swapper/209:13:112
2286825313,18sleep20-21swapper/209:13:112
1996725317,20sleep20-21swapper/207:22:402
1996725317,20sleep20-21swapper/207:22:402
1619825321,18sleep20-21swapper/208:53:042
1417425313,17sleep20-21swapper/210:28:082
1379725311,18sleep10-21swapper/110:27:591
129772537,15sleep20-21swapper/212:08:002
1702825213,22sleep017086-21if_err_eth010:37:590
1145125215,22sleep111453-21irqstats12:03:111
61392519,16sleep00-21swapper/011:47:390
58432516,10sleep20-21swapper/211:43:492
445725112,23sleep24462-21/usr/sbin/munin09:57:592
2612325115,22sleep226147-21ntp_states09:23:252
2539125115,18sleep10-21swapper/107:38:101
1745325117,16sleep20-21swapper/210:38:102
3144125012,21sleep231494-21users09:38:162
290202505,34sleep20-21swapper/211:17:362
2100725017,16sleep10-21swapper/109:08:011
178325011,24sleep21780-21apt-get08:07:522
1274525011,17sleep20-21swapper/208:42:422
1029625015,17sleep10-21swapper/108:33:071
592424911,22sleep15919-21apt-get10:02:541
523724911,22sleep05242-21diskstats08:17:560
2064024910,21sleep120677-21ntp_states07:23:041
2064024910,21sleep120677-21ntp_states07:23:041
1242424912,20sleep012443-21munin-run12:07:380
927424816,18sleep20-21swapper/211:57:382
2958324811,20sleep00-21swapper/007:52:560
2776224811,21sleep127756-21apt-get11:12:541
237924810,24sleep11513-21/usr/sbin/munin11:33:141
122562489,13sleep00-21swapper/010:23:010
671124710,14sleep20-21swapper/208:22:552
2671824712,22sleep226757-21expr07:43:002
2671824712,22sleep226757-21expr07:43:002
3164724612,18sleep10-21swapper/107:58:101
2039124515,16sleep00-21swapper/007:22:590
2039124515,16sleep00-21swapper/007:22:590
1919024410,18sleep10-21swapper/109:02:521
1919024410,18sleep10-21swapper/109:02:521
82552429,17sleep20-21swapper/208:27:512
3262924214,13sleep10-21swapper/109:43:071
130122398,15sleep20-21swapper/208:42:582
130122398,15sleep20-21swapper/208:42:582
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional