You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-14 - 09:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Tue Apr 14, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24399265626,6546sleep121405-21kworker/1:119:32:191
174822314811,3125sleep34671-21kworker/3:020:54:193
3321223447,2303sleep1676-21kworker/1:020:10:191
66302111857,1049sleep16674-21ntpq23:46:471
7015269415,659sleep07151-21/usr/sbin/munin23:46:550
143225998,576sleep11468-21ntpq21:46:461
17457258611,530sleep117609-21latency_hist00:21:241
19188255611,528sleep119338-21grep22:41:581
19188255611,528sleep119338-21grep22:41:581
30789237212,342sleep330801-21/usr/sbin/munin21:36:473
5505229266,143sleep10-21swapper/122:01:241
20118228821,253sleep00-21swapper/019:21:240
30008228482,70sleep10-21swapper/119:51:421
8986228218,213sleep19086-21timerandwakeup20:26:551
10229226989,138sleep110281-21switchtime23:57:131
17939222739,135sleep118022-21latency00:21:441
17939222739,135sleep118022-21latency00:21:441
2776222076,188sleep127769-21ntpq21:26:511
21390219920,51sleep10-21swapper/100:31:541
20452218517,153sleep020476-21proc_pri22:46:500
20452218517,153sleep020476-21proc_pri22:46:500
4910218416,144sleep05049-21sed21:56:550
20476218148,68sleep10-21swapper/100:31:211
20476218148,68sleep10-21swapper/100:31:211
2989921779,153sleep129974-21mailstats23:16:511
2989921779,153sleep129974-21mailstats23:16:511
1426221718,148sleep114337-21awk22:26:561
1426221718,148sleep114337-21awk22:26:561
16751216849,75sleep10-21swapper/120:51:461
29117216016,119sleep10-21swapper/121:31:441
15762215510,134sleep013441-21snmpd00:15:500
11254215446,49sleep10-21swapper/120:36:241
2494421507,57sleep10-21swapper/123:01:491
2494421507,57sleep10-21swapper/123:01:491
20067214910,21sleep10-21swapper/121:01:521
20067214910,21sleep10-21swapper/121:01:521
17118214910,23sleep10-21swapper/122:36:491
419021477,125sleep14268-21mailstats20:11:531
419021477,125sleep14268-21mailstats20:11:531
951021449,71sleep10-21swapper/122:11:541
1280221448,98sleep112822-21latency_hist20:41:251
6894214122,106sleep37012-21sed23:46:543
12444214073,21sleep10-21swapper/100:06:221
5314213938,86sleep15408-21timerandwakeup23:41:551
18683213918,37sleep30-21swapper/322:41:483
18683213918,37sleep30-21swapper/322:41:483
13532213730,83sleep313588-21fschecks_count19:01:413
3622213610,113sleep13650-21proc_pri23:36:501
26583213310,99sleep10-21swapper/119:41:241
26583213310,99sleep10-21swapper/119:41:241
1587213273,43sleep10-21swapper/123:31:371
22175213141,73sleep10-21swapper/121:09:191
22211213020,83sleep20-21swapper/222:51:532
22211213020,83sleep20-21swapper/222:51:532
25838212813,102sleep025841-21entropy21:21:420
20366212647,61sleep120378-21cpu19:21:371
17423212619,88sleep00-21swapper/000:21:230
26442212421,85sleep226482-21cat21:22:042
21298212463,18sleep10-21swapper/122:51:231
21298212463,18sleep10-21swapper/122:51:231
1899121246,34sleep10-21swapper/120:58:211
9363212316,89sleep29545-21apt-key23:56:262
15794212323,40sleep10-21swapper/122:32:011
2932621226,11sleep10-21swapper/119:47:301
2932621226,11sleep10-21swapper/119:47:301
2259921225,12sleep10-21swapper/121:11:331
541212136,65sleep10-21swapper/123:26:531
24719212123,82sleep024731-21modprobe19:36:220
15602212146,16sleep10-21swapper/119:06:551
15602212146,16sleep10-21swapper/119:06:551
362121195,10sleep10-21swapper/121:53:121
3197121188,39sleep10-21swapper/121:41:371
2746921185,10sleep10-21swapper/123:10:411
2746921185,10sleep10-21swapper/123:10:411
20129211827,77sleep10-21swapper/119:21:251
22400211553,47sleep10-21swapper/119:26:541
9230211453,50sleep19360-21apt-key23:56:251
20526211423,74sleep320616-21latency_hist00:31:233
20526211423,74sleep320616-21latency_hist00:31:233
1585421144,16sleep00-21swapper/020:48:180
637821095,91sleep10-21swapper/120:19:201
637821095,91sleep10-21swapper/120:19:201
11972210930,41sleep112006-21cpu22:21:391
7962210851,18sleep18083-21cat22:06:531
19877210813,77sleep319886-21/usr/sbin/munin21:01:463
19877210813,77sleep319886-21/usr/sbin/munin21:01:463
9546210611,77sleep39082-21apt-get23:56:273
3162210615,70sleep22586-21/usr/sbin/munin21:51:522
22478210513,74sleep00-21swapper/022:52:190
22478210513,74sleep00-21swapper/022:52:190
1060521058,76sleep10-21swapper/122:16:421
331210443,45sleep10-21swapper/120:01:231
331210443,45sleep10-21swapper/120:01:231
3246210315,70sleep03256-21/usr/sbin/munin23:36:410
10515210318,66sleep010604-21fschecks_time22:16:410
25756210227,62sleep125759-21grep21:21:381
166321025,53sleep10-21swapper/120:03:531
166321025,53sleep10-21swapper/120:03:531
7294210112,74sleep27353-21processes20:21:512
29924210117,61sleep20-21swapper/219:51:392
2418421017,44sleep124203-21apt-get21:16:341
9601210012,70sleep215044-1kworker/2:0H23:56:342
808529917,36sleep10-21swapper/120:26:221
3158429921,58sleep231640-21tune2fs19:56:432
3158429921,58sleep231640-21tune2fs19:56:432
2749429913,70sleep327607-21timerwakeupswit19:41:543
705529811,73sleep07093-21latency_hist22:06:240
2858129835,19sleep10-21swapper/123:11:551
2470529813,69sleep024785-21sendmail_mailtr21:16:530
134712987,34sleep10-21swapper/120:41:491
2132129713,64sleep021387-21cat00:31:530
135212977,62sleep00-21swapper/022:26:390
135212977,62sleep00-21swapper/022:26:390
1282329719,53sleep212971-21latency_hist20:41:262
3276129624,47sleep00-21swapper/020:01:220
3276129624,47sleep00-21swapper/020:01:220
1535329615,69sleep315412-21latency22:31:513
1529929616,65sleep215306-21open_files20:46:512
2956929517,56sleep229702-21/usr/sbin/munin21:31:532
129729520,50sleep21317-21modprobe23:31:222
1609529435,18sleep10-21swapper/120:51:221
933529321,44sleep38816-21/usr/sbin/munin22:11:523
2289229221,51sleep10-21swapper/122:56:241
2289229221,51sleep10-21swapper/122:56:241
65592910,10sleep20-21swapper/220:21:222
65592910,10sleep20-21swapper/220:21:222
2524329119,53sleep324741-21apt-get19:36:333
2524329119,53sleep324741-21apt-get19:36:333
3128028918,52sleep00-21swapper/019:56:250
1341528821,45sleep013565-21tail00:06:530
1167028813,61sleep211747-21basename00:01:512
837728725,17sleep20-21swapper/220:26:402
3099328724,16sleep10-21swapper/121:36:531
1769428712,58sleep017700-21ntp_states19:11:550
2846528613,55sleep028580-21timerwakeupswit23:11:540
1679428611,18sleep10-21swapper/119:11:281
1249528622,41sleep312472-21latency_hist00:06:243
105928613,55sleep0566-21/usr/sbin/munin20:01:500
105928613,55sleep0566-21/usr/sbin/munin20:01:500
3115128514,58sleep031203-21cat23:21:430
3065528417,48sleep3222-21systemd-journal23:21:213
3065528417,48sleep3222-21systemd-journal23:21:213
2422528419,10sleep10-21swapper/122:59:231
1320028436,35sleep10-21swapper/100:06:501
1198028415,44sleep00-21swapper/020:36:510
601928317,45sleep35994-21latency_hist23:46:253
2476028320,42sleep224846-21fschecks_time23:01:462
2476028320,42sleep224846-21fschecks_time23:01:462
277232827,64sleep127733-21awk19:41:571
166272827,11sleep00-21swapper/022:36:320
1409328210,56sleep314168-21grep22:26:543
1409328210,56sleep314168-21grep22:26:543
1551428119,10sleep10-21swapper/100:12:481
991128029,33sleep09929-21kernelversion23:57:000
574328022,41sleep10-21swapper/122:01:361
1927928014,52sleep219386-21cat21:01:242
71827926,22sleep30-21swapper/323:26:553
3196527918,9sleep10-21swapper/123:23:211
269627919,45sleep32708-21fschecks_count21:51:413
226827914,48sleep22423-21unixbench_multi23:31:542
189882794,11sleep00-21swapper/020:58:200
709427813,18sleep00-21swapper/020:21:450
2836927811,43sleep20-21swapper/219:46:382
2836927811,43sleep20-21swapper/219:46:382
2820127819,36sleep227788-21/usr/sbin/munin23:11:472
2289727814,27sleep20-21swapper/221:11:462
1962227717,40sleep019663-21awk00:26:480
1962227717,40sleep019663-21awk00:26:480
1802327613,48sleep217645-21/usr/sbin/munin00:21:452
1802327613,48sleep217645-21/usr/sbin/munin00:21:452
1646127620,25sleep30-21swapper/319:11:253
1646127620,25sleep30-21swapper/319:11:253
941527515,47sleep09509-21switchtime22:11:530
40832757,12sleep30-21swapper/323:37:043
2125227518,40sleep321318-21run-parts00:31:523
17972750,8sleep30-21swapper/323:31:423
1737927517,42sleep31-21systemd00:21:223
1459027521,31sleep20-21swapper/220:46:262
99662745,19sleep20-21swapper/222:14:332
50127418,29sleep00-21swapper/023:26:520
2602727419,22sleep00-21swapper/023:06:240
2602727419,22sleep00-21swapper/023:06:240
2063427417,41sleep020665-21latency19:21:440
2014327415,46sleep220269-21cat21:01:532
2014327415,46sleep220269-21cat21:01:532
1934127412,48sleep219350-21/usr/sbin/munin22:41:592
1934127412,48sleep219350-21/usr/sbin/munin22:41:592
1930727414,46sleep219343-21expr00:26:422
1930727414,46sleep219343-21expr00:26:422
1470727426,25sleep10-21swapper/119:06:231
2965727321,29sleep20-21swapper/223:16:442
2965727321,29sleep20-21swapper/223:16:442
2077027311,48sleep220817-21grep00:31:382
1970927317,41sleep019493-21/usr/sbin/munin21:01:430
1970927317,41sleep019493-21/usr/sbin/munin21:01:430
1641527326,24sleep00-21swapper/022:36:240
103927315,43sleep01058-21apt-get21:46:330
816927215,14sleep30-21swapper/320:26:243
789927220,27sleep30-21swapper/323:51:423
2739727218,40sleep227492-21switchtime19:41:532
2421327223,32sleep30-21swapper/321:16:403
1580227216,32sleep315844-21awk00:16:213
3140427119,31sleep30-21swapper/323:21:503
1359227122,26sleep013659-21ls19:01:420
1227827119,29sleep30-21swapper/320:36:543
426927027,26sleep24345-21switchtime20:11:542
426927027,26sleep24345-21switchtime20:11:542
3214427024,25sleep20-21swapper/221:41:422
237426918,25sleep30-21swapper/321:51:243
1300126923,27sleep30-21swapper/300:06:443
574426821,22sleep20-21swapper/222:01:372
517026815,37sleep25183-21/usr/sbin/munin23:41:522
274126819,31sleep02821-21find20:06:510
366926714,19sleep30-21swapper/320:11:343
366926714,19sleep30-21swapper/320:11:343
2524726720,23sleep00-21swapper/019:36:340
2524726720,23sleep00-21swapper/019:36:340
2253726718,28sleep322664-21wc19:26:563
1635726716,29sleep216458-21dpkg19:11:242
1635726716,29sleep216458-21dpkg19:11:242
2246926621,18sleep20-21swapper/219:26:552
1825926615,35sleep218263-21modprobe19:13:222
101326611,30sleep21049-21ntp_states20:01:482
101326611,30sleep21049-21ntp_states20:01:482
3168226522,24sleep20-21swapper/221:41:222
2379526521,19sleep00-21swapper/022:56:550
2190226510,33sleep30-21swapper/322:51:463
2190226510,33sleep30-21swapper/322:51:463
268172648,18sleep00-21swapper/019:41:350
2319626410,27sleep20-21swapper/222:56:412
1717826414,28sleep2281-21dbus-daemon22:36:502
1714026420,28sleep317142-21ntp_states19:11:343
860326324,23sleep00-21swapper/022:11:240
2968226316,26sleep00-21swapper/019:51:240
2968226316,26sleep00-21swapper/019:51:240
2484426314,23sleep20-21swapper/221:17:112
2353626314,33sleep323542-21diskstats19:31:413
2299026312,22sleep30-21swapper/321:11:473
2023826315,19sleep20-21swapper/222:46:442
2023826315,19sleep20-21swapper/222:46:442
160532635,12sleep20-21swapper/222:32:472
2777026220,25sleep227811-21cut21:26:522
2435526226,19sleep20-21swapper/219:31:562
1935126214,31sleep30-21swapper/319:16:513
1796726220,17sleep30-21swapper/322:41:243
1313526218,20sleep20-21swapper/200:06:472
729626121,21sleep20-21swapper/222:06:382
2938426118,21sleep00-21swapper/021:31:510
2401326118,22sleep00-21swapper/021:16:250
236326117,29sleep22373-21munin-run21:51:232
2166926114,31sleep321703-21latency_hist19:26:243
989426013,23sleep00-21swapper/020:31:340
622526015,24sleep30-21swapper/322:01:503
596526015,31sleep05996-21munin-run23:46:220
306092605,11sleep00-21swapper/021:36:440
2757126018,23sleep227634-21sed23:11:232
2757126018,23sleep227634-21sed23:11:232
2633726018,25sleep326343-21df_inode23:06:403
2633726018,25sleep326343-21df_inode23:06:403
132222609,13sleep00-21swapper/020:41:440
464225919,22sleep34648-21ntp_states21:56:503
3010825921,20sleep30-21swapper/321:36:223
297825916,21sleep00-21swapper/021:51:460
294362594,18sleep30-21swapper/319:48:533
294362594,18sleep30-21swapper/319:48:533
2838325920,19sleep30-21swapper/323:11:533
1803925915,19sleep30-21swapper/300:21:463
1803925915,19sleep30-21swapper/300:21:463
6802585,39sleep30-21swapper/320:01:413
6802585,39sleep30-21swapper/320:01:413
1659225811,30sleep20-21swapper/220:51:432
1433925715,18sleep30-21swapper/320:46:233
1257525713,29sleep012663-21timerandwakeup22:21:540
82525616,19sleep30-21swapper/321:46:243
613125614,25sleep06184-21users20:16:570
613125614,25sleep06184-21users20:16:570
2238625617,24sleep022497-21latency_hist21:11:240
2116325617,19sleep20-21swapper/221:06:422
185525618,24sleep00-21swapper/023:31:440
1635825621,19sleep00-21swapper/020:51:330
1475925618,19sleep20-21swapper/219:06:242
1042025619,21sleep20-21swapper/220:31:532
840925512,18sleep30-21swapper/320:26:413
681025519,18sleep20-21swapper/223:46:532
3124825516,18sleep30-21swapper/319:56:243
1999025513,24sleep319985-21apt-get22:46:373
1999025513,24sleep319985-21apt-get22:46:373
1808625514,17sleep00-21swapper/000:21:470
1808625514,17sleep00-21swapper/000:21:470
1211725514,17sleep20-21swapper/222:21:442
892025410,22sleep00-21swapper/020:26:540
415825414,22sleep00-21swapper/020:11:520
415825414,22sleep00-21swapper/020:11:520
221425416,23sleep32221-21df_inode20:06:383
2050125415,18sleep20-21swapper/219:21:422
1967125415,23sleep219718-21users19:16:542
1796125413,24sleep317956-21apt-get20:56:373
1483525414,23sleep314832-21apt-get20:46:363
1173125416,14sleep20-21swapper/222:21:242
1035425419,20sleep310373-21ntp_states20:31:503
3197025313,24sleep031964-21apt-get21:41:360
3102825315,18sleep20-21swapper/223:21:412
2770425318,16sleep30-21swapper/321:26:493
261125322,19sleep22649-21processes20:06:492
2320025321,18sleep30-21swapper/322:56:423
1118525312,24sleep011179-21apt-get00:01:360
443025217,21sleep24459-21irqstats21:56:452
2939925211,19sleep00-21swapper/023:16:380
2939925211,19sleep00-21swapper/023:16:380
2730425211,24sleep027298-21apt-get21:26:380
1844325210,25sleep018433-21apt-get22:41:370
1844325210,25sleep018433-21apt-get22:41:370
1612125213,23sleep216115-21apt-get00:16:372
1469325213,17sleep20-21swapper/200:12:212
334425114,21sleep23138-21/usr/sbin/munin23:36:432
2198525113,15sleep00-21swapper/019:26:410
1359925112,23sleep213494-21/usr/sbin/munin22:26:412
1359925112,23sleep213494-21/usr/sbin/munin22:26:412
914025010,22sleep09229-21gpgv23:56:230
2653325014,16sleep00-21swapper/023:06:450
2653325014,16sleep00-21swapper/023:06:450
2531425014,21sleep225264-21apt-get19:37:092
2531425014,21sleep225264-21apt-get19:37:092
2107925013,23sleep321148-21df21:06:383
733624912,21sleep37367-21df22:06:393
2467624912,21sleep324746-21df23:01:433
2467624912,21sleep324746-21df23:01:433
315002489,17sleep30-21swapper/319:56:403
315002489,17sleep30-21swapper/319:56:403
527224712,20sleep35275-21apt-get20:16:353
527224712,20sleep35275-21apt-get20:16:353
261692478,16sleep30-21swapper/321:21:493
1552324712,16sleep00-21swapper/019:06:540
1552324712,16sleep00-21swapper/019:06:540
1178624714,16sleep30-21swapper/322:21:253
120442409,16sleep30-21swapper/322:21:413
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional