You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-19 - 10:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Fri Jun 19, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
228282357620,3543sleep119615-21kworker/1:021:50:281
3828276810,742sleep33881-21/usr/sbin/munin22:32:493
3828276810,742sleep33881-21/usr/sbin/munin22:32:493
22784259119,555sleep222774-21apt-get20:07:342
22784259119,555sleep222774-21apt-get20:07:342
29011246529,420sleep029087-21awk23:52:500
6222241161,335sleep26225-21df_inode20:57:362
6222241161,335sleep26225-21df_inode20:57:362
16860237027,305sleep017001-21sed21:32:230
9997232519,261sleep010141-21head21:07:520
9997232519,261sleep010141-21head21:07:520
1360022998,255sleep013641-21ntpq19:37:430
281982276102,159sleep028345-21latency_hist23:52:210
6592226583,161sleep00-21swapper/000:27:180
7026225711,179sleep00-21swapper/019:17:480
32227225770,19sleep00-21swapper/000:02:510
4689224749,183sleep04702-21awk00:17:530
16513224567,162sleep016587-21switchtime23:12:480
2277223941,68sleep02744-21cyclictest00:12:410
25692222471,88sleep00-21swapper/023:42:440
17136222242,58sleep00-21swapper/021:32:390
25300221624,16sleep00-21swapper/021:57:460
10406221421,32sleep00-21swapper/019:27:460
3044622138,163sleep00-21swapper/022:13:290
1910522127,190sleep019189-21grep21:37:480
26150220315,174sleep226183-21latency_hist22:02:202
941621998,176sleep09566-21grep00:32:520
941621998,176sleep09566-21grep00:32:520
5834219922,158sleep25927-21timerandwakeup19:13:262
13763219310,89sleep00-21swapper/021:22:210
29752219010,164sleep029893-21grep20:27:510
27074218618,144sleep327203-21unixbench_multi22:02:503
2408721769,152sleep024123-21latency_hist20:12:200
2408721769,152sleep024123-21latency_hist20:12:200
4179217358,69sleep04049-21apt-get20:52:220
5171216267,51sleep05168-21apt-get19:12:310
8943216160,54sleep09086-21unixbench_singl22:48:090
2339021617,99sleep0363-21snmpd23:36:340
20849216111,135sleep120615-21/usr/sbin/munin23:27:391
6635216064,53sleep00-21swapper/022:42:390
6635216064,53sleep00-21swapper/022:42:390
6235216029,28sleep00-21swapper/020:57:380
6235216029,28sleep00-21swapper/020:57:380
14127216066,51sleep014141-21latency_hist23:07:190
2195521570,49sleep00-21swapper/020:03:310
2195521570,49sleep00-21swapper/020:03:310
31283215657,50sleep00-21swapper/022:17:410
11182215357,52sleep00-21swapper/021:12:450
16289215111,124sleep116290-21/usr/sbin/munin19:47:301
14329215013,54sleep00-21swapper/021:22:460
5230214961,45sleep05280-21ntp_kernel_pll_22:37:430
5230214961,45sleep05280-21ntp_kernel_pll_22:37:430
27259213913,112sleep327301-21awk23:47:443
24129213748,75sleep00-21swapper/023:37:440
11995213534,78sleep00-21swapper/022:57:500
22079213448,22sleep00-21swapper/021:47:410
1107621289,106sleep011110-21date22:57:210
7872212737,64sleep00-21swapper/000:28:150
7872212737,64sleep00-21swapper/000:28:150
26928212713,25sleep10-21swapper/122:02:481
18857212663,24sleep018995-21latency_hist23:22:210
19068212516,47sleep00-21swapper/023:22:290
17625212412,47sleep00-21swapper/019:52:210
2016521237,54sleep00-21swapper/021:42:320
9449212215,87sleep29464-21latency_hist22:52:192
3466212210,15sleep20-21swapper/200:17:192
1481212222,37sleep00-21swapper/000:07:520
1481212222,37sleep00-21swapper/000:07:520
969321218,101sleep0363-21snmpd19:25:480
19121211924,78sleep119169-21sendmail19:57:191
9234211521,72sleep19379-21sed19:22:501
23838211510,43sleep00-21swapper/021:52:500
3125021146,17sleep00-21swapper/000:02:150
3125021146,17sleep00-21swapper/000:02:150
1962421149,39sleep00-21swapper/019:57:380
15333211411,43sleep00-21swapper/019:42:480
15333211411,43sleep00-21swapper/019:42:480
2717211216,80sleep32850-21cut19:07:213
11917211116,32sleep00-21swapper/019:32:430
16185211024,57sleep316346-21tail21:27:543
32708210915,79sleep1322-21grep22:22:381
21227210822,66sleep120736-21apt-get20:02:321
21227210822,66sleep120736-21apt-get20:02:321
1969210848,16sleep00-21swapper/020:42:530
19065210817,77sleep319066-21sshd23:22:283
6997210718,73sleep36467-21apt-get19:17:313
17490210712,81sleep317500-21ntpq21:32:463
32130210432,57sleep02744-21cyclictest20:37:320
366421039,26sleep20-21swapper/222:32:442
366421039,26sleep20-21swapper/222:32:442
28759210222,62sleep228778-21latency_hist20:27:202
1834210210,76sleep11835-21sshd00:11:281
1834210210,76sleep11835-21sshd00:11:281
20914210119,62sleep321005-21/usr/sbin/munin23:27:413
20350210021,58sleep120374-21basename23:27:191
2957529920,61sleep229637-21grep22:12:382
711729820,56sleep16609-21apt-get00:27:311
711729820,56sleep16609-21apt-get00:27:311
1019429814,68sleep210203-21ntpq22:52:472
2966329728,45sleep129692-21munin-run23:57:191
216882955,80sleep00-21swapper/023:29:360
3055429413,19sleep20-21swapper/220:32:332
1680829416,60sleep316887-21/usr/sbin/munin19:47:483
441229320,50sleep34612-21dpkg19:12:223
441229320,50sleep34612-21dpkg19:12:223
21932939,63sleep00-21swapper/022:27:580
611029118,55sleep36136-21ls22:42:183
611029118,55sleep36136-21ls22:42:183
7482906,50sleep00-21swapper/022:22:490
2483929016,60sleep324834-21apt-get21:57:333
2269429022,46sleep222787-21mailstats23:32:492
528928721,51sleep15316-21fschecks_count19:12:571
177728711,61sleep21841-21/usr/sbin/munin20:42:512
1395228627,26sleep20-21swapper/221:22:362
875828519,51sleep38790-21tune2fs00:32:383
875828519,51sleep38790-21tune2fs00:32:383
355828515,55sleep23650-21/usr/sbin/munin19:07:492
355828515,55sleep23650-21/usr/sbin/munin19:07:492
2908528518,10sleep00-21swapper/022:10:500
1602528518,31sleep30-21swapper/319:47:193
1602528518,31sleep30-21swapper/319:47:193
153432855,49sleep00-21swapper/023:08:040
895528419,11sleep00-21swapper/021:07:110
902828330,22sleep10-21swapper/121:07:201
3178828318,11sleep00-21swapper/020:36:590
2559728317,49sleep225638-21basename20:17:192
409528218,28sleep00-21swapper/019:10:290
409528218,28sleep00-21swapper/019:10:290
275482829,58sleep00-21swapper/020:22:380
2071928214,52sleep220726-21/usr/sbin/munin23:27:362
1248628224,32sleep10-21swapper/121:17:381
594028112,55sleep16017-21/usr/sbin/munin00:22:511
483228154,12sleep00-21swapper/020:52:500
2855428122,31sleep30-21swapper/322:07:503
2702928118,10sleep00-21swapper/020:20:280
2403828118,47sleep324127-21/usr/sbin/munin23:37:433
1763428114,53sleep217706-21fschecks_time23:17:392
3165028014,44sleep10-21swapper/122:17:501
1368328019,44sleep313698-21/usr/sbin/munin19:37:463
1161928016,45sleep311636-21/usr/sbin/munin19:32:293
779027923,24sleep10-21swapper/122:47:211
779027923,24sleep10-21swapper/122:47:211
274522799,12sleep00-21swapper/022:04:450
2528327918,11sleep00-21swapper/020:13:250
1334827919,43sleep313406-21/usr/sbin/munin23:02:503
175482787,12sleep20-21swapper/219:51:592
124427815,48sleep21321-21/usr/sbin/munin00:07:502
124427815,48sleep21321-21/usr/sbin/munin00:07:502
3134727716,46sleep331495-21cat00:02:223
2412427720,25sleep10-21swapper/120:12:211
2412427720,25sleep10-21swapper/120:12:211
2122827720,38sleep221224-21apt-get20:02:332
2122827720,38sleep221224-21apt-get20:02:332
442127615,9sleep00-21swapper/022:35:490
442127615,9sleep00-21swapper/022:35:490
171127626,35sleep11776-21awk20:42:501
277727519,26sleep30-21swapper/300:12:523
1887627515,11sleep00-21swapper/019:54:110
112692756,49sleep30-21swapper/321:12:483
98527415,43sleep11074-21ls20:42:211
39172746,18sleep30-21swapper/300:17:403
3116027424,28sleep20-21swapper/222:17:392
2732227416,28sleep227363-21grep23:47:472
2694627425,21sleep10-21swapper/123:47:381
266872740,7sleep10-21swapper/123:47:201
1775727415,38sleep117019-21/usr/sbin/munin21:32:521
1382727416,37sleep20-21swapper/219:37:492
2985627326,27sleep30-21swapper/322:12:433
2837627320,32sleep30-21swapper/323:52:283
1930427316,39sleep219360-21/usr/sbin/munin23:22:392
1525827320,26sleep30-21swapper/321:27:213
583027214,43sleep15888-21sendmail20:57:191
2666327220,25sleep20-21swapper/222:02:412
2400627219,32sleep224142-21sed21:52:522
1816027214,35sleep118264-21timerandwakeup23:17:501
591027115,43sleep35987-21latency_hist20:57:213
3265827119,24sleep30-21swapper/320:37:473
1890227115,28sleep20-21swapper/221:37:422
1658927115,30sleep10-21swapper/123:12:491
1211427114,28sleep20-21swapper/219:32:492
896327015,37sleep38989-21modprobe21:07:183
843327026,24sleep20-21swapper/200:32:212
843327026,24sleep20-21swapper/200:32:212
691126824,25sleep30-21swapper/300:27:213
691126824,25sleep30-21swapper/300:27:213
3083626818,35sleep130781-21latency_hist22:17:211
264526820,27sleep12679-21munin-run19:07:191
2227126813,38sleep322274-21/usr/sbin/munin23:32:383
1232726813,41sleep112455-21/usr/sbin/munin19:32:521
1047426816,29sleep10-21swapper/119:27:471
2877926720,32sleep328924-21sed20:27:213
2618426722,19sleep30-21swapper/320:17:403
2255726715,36sleep122591-21/usr/sbin/munin23:32:441
1979726712,37sleep319807-21needreboot19:57:413
1128526721,28sleep311304-21apt-get22:57:283
3055326622,26sleep130550-21apt-get20:32:321
295612660,7sleep20-21swapper/220:27:482
2717126626,24sleep127178-21modprobe20:22:181
40226521,26sleep1544-21latency_hist00:07:211
347826515,34sleep13557-21systemctl19:07:481
347826515,34sleep13557-21systemctl19:07:481
298326517,24sleep10-21swapper/122:32:191
2237826519,27sleep312-21rcu_sched21:47:483
196926528,20sleep20-21swapper/219:03:172
953826414,34sleep29606-21ls21:07:422
953826414,34sleep29606-21ls21:07:422
820626414,25sleep10-21swapper/122:47:451
533026416,27sleep30-21swapper/319:13:033
303326413,24sleep30-21swapper/322:32:213
139126417,24sleep00-21swapper/019:02:340
1184626421,19sleep20-21swapper/222:57:482
616126313,25sleep10-21swapper/122:42:201
616126313,25sleep10-21swapper/122:42:201
601826322,19sleep20-21swapper/200:22:522
1987726317,24sleep30-21swapper/321:42:193
1970926320,22sleep20-21swapper/219:57:402
1550126320,24sleep20-21swapper/221:27:362
1497026323,18sleep20-21swapper/223:07:492
1487326313,36sleep314878-21ntp_states23:07:463
146726316,35sleep38970-21kworker/u8:222:27:203
988226216,27sleep30-21swapper/321:07:513
988226216,27sleep30-21swapper/321:07:513
765826219,25sleep37711-21apt-config22:47:193
765826219,25sleep37711-21apt-config22:47:193
3037526213,29sleep230414-21/usr/sbin/munin23:57:482
3037526213,29sleep230414-21/usr/sbin/munin23:57:482
2940426220,26sleep10-21swapper/120:27:441
2909326213,21sleep30-21swapper/320:27:373
26602626,17sleep20-21swapper/222:28:162
2488126220,22sleep20-21swapper/223:40:282
1723026226,18sleep30-21swapper/323:17:193
1171126213,25sleep10-21swapper/122:57:431
915726116,25sleep10-21swapper/100:32:481
915726116,25sleep10-21swapper/100:32:481
324226119,19sleep30-21swapper/320:47:483
3063926114,28sleep30-21swapper/320:32:383
2999426115,24sleep10-21swapper/123:57:351
2999426115,24sleep10-21swapper/123:57:351
2646726013,18sleep20-21swapper/220:17:472
1920026023,18sleep30-21swapper/319:57:213
1888526017,24sleep10-21swapper/121:37:411
1755026015,29sleep217638-21basename21:32:492
1366026019,23sleep20-21swapper/221:22:192
118692607,11sleep10-21swapper/121:14:351
700525920,23sleep16993-21apt-get19:17:331
399625919,18sleep10-21swapper/122:32:521
399625919,18sleep10-21swapper/122:32:521
323425920,21sleep20-21swapper/220:47:472
3205625916,16sleep20-21swapper/200:02:492
2504925915,28sleep125106-21irqstats21:57:391
2025225916,23sleep20-21swapper/221:42:382
1062925915,18sleep20-21swapper/221:12:222
100825914,25sleep30-21swapper/300:07:433
100825914,25sleep30-21swapper/300:07:433
778325814,27sleep27789-21diskstats21:02:372
2929125817,21sleep20-21swapper/222:12:222
1571625816,25sleep115725-21munin-run23:12:211
1395125814,27sleep113945-21apt-get21:22:351
1314325810,20sleep10-21swapper/123:02:441
1000625815,18sleep10-21swapper/122:52:421
892125715,25sleep28966-21awk19:22:432
710825717,20sleep20-21swapper/222:42:502
710825717,20sleep20-21swapper/222:42:502
661025717,20sleep10-21swapper/120:57:481
661025717,20sleep10-21swapper/120:57:481
456025714,17sleep20-21swapper/220:52:322
2656925716,24sleep226574-21modprobe23:47:182
265525719,24sleep12697-21ps00:12:501
2214225714,26sleep222164-21ntp_states21:47:432
2074025727,18sleep120814-21cat21:42:491
818225616,17sleep30-21swapper/322:47:313
672225616,23sleep36791-21/usr/sbin/munin20:57:503
672225616,23sleep36791-21/usr/sbin/munin20:57:503
548625616,24sleep25601-21expr22:37:492
548625616,24sleep25601-21expr22:37:492
225825611,27sleep30-21swapper/320:44:293
2186325616,25sleep321950-21sed20:03:313
2186325616,25sleep321950-21sed20:03:313
2031325617,23sleep30-21swapper/321:42:393
1552725615,26sleep215663-21/usr/sbin/munin19:42:502
1552725615,26sleep215663-21/usr/sbin/munin19:42:502
1541025617,19sleep10-21swapper/119:42:491
1541025617,19sleep10-21swapper/119:42:491
1417925614,17sleep20-21swapper/223:07:212
387225517,20sleep20-21swapper/200:17:392
3152025515,26sleep331522-21/usr/sbin/munin00:02:303
2840825511,26sleep228478-21df23:52:362
2529025518,22sleep325309-21apt-get23:42:313
2432825514,25sleep324324-21apt-get20:12:313
2324625515,22sleep30-21swapper/320:07:473
2324625515,22sleep30-21swapper/320:07:473
1786225515,23sleep217858-21apt-get19:52:312
141252556,36sleep10-21swapper/119:37:521
1281125511,19sleep30-21swapper/321:17:443
533425415,22sleep35331-21apt-get00:22:333
2323825412,25sleep123257-21apt-get21:52:301
1991725414,19sleep10-21swapper/119:57:471
922625316,16sleep20-21swapper/200:32:492
922625316,16sleep20-21swapper/200:32:492
421625314,23sleep14239-21ntp_states00:17:461
2376525313,22sleep123762-21apt-get23:37:321
950525210,17sleep10-21swapper/121:07:411
950525210,17sleep10-21swapper/121:07:411
288752525,36sleep10-21swapper/122:08:091
2745825214,15sleep10-21swapper/120:22:311
2365825212,27sleep323698-21ntpq21:52:443
1675525211,21sleep10-21swapper/121:32:201
96925112,21sleep21067-21users22:22:512
312052519,24sleep331281-21fschecks_time22:17:403
1670525115,16sleep20-21swapper/223:12:502
110172518,31sleep3363-21snmpd19:27:573
974125010,22sleep30-21swapper/319:26:283
3259625013,16sleep20-21swapper/220:37:462
973624912,21sleep39734-21apt-get22:52:323
30292498,16sleep10-21swapper/120:47:421
2999124812,21sleep329987-21apt-get23:57:333
2433124814,19sleep10-21swapper/120:12:331
175512477,15sleep30-21swapper/323:17:353
709824510,21sleep27107-21diskstats19:17:502
113572436,14sleep20-21swapper/219:32:202
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional