You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-22 - 13:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Wed Jul 22, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
287932194713,1917sleep228798-21ntp_states20:48:442
28995259314,565sleep329123-21ls20:48:493
284102429387,27sleep128418-21sshd20:48:351
7586242522,386sleep17645-21/usr/sbin/munin23:08:401
8953235037,209sleep39016-21fschecks_time23:13:383
293502254109,30sleep30-21swapper/319:08:483
3896224916,17sleep30-21swapper/322:58:173
31712224823,215sleep3364-21snmpd19:14:413
31712224823,215sleep3364-21snmpd19:14:413
5285222533,70sleep25304-21apt-get21:18:312
5285222533,70sleep25304-21apt-get21:18:312
1740822169,192sleep317419-21sort20:13:303
29488220615,179sleep129604-21sed22:33:491
2402721958,175sleep324066-21ntpq20:33:403
23101219411,168sleep323142-21grep22:13:453
22817219422,147sleep117166-1kworker/1:1H23:58:161
9742219216,27sleep30-21swapper/319:48:353
29493219139,24sleep30-21swapper/300:18:363
29493219139,24sleep30-21swapper/300:18:363
5186219054,31sleep30-21swapper/319:34:193
1175921878,164sleep311902-21latency_hist23:23:193
31572218518,25sleep30-21swapper/300:23:493
15487218513,152sleep30-21swapper/323:34:213
24429217919,25sleep30-21swapper/322:18:373
13313217714,130sleep30-21swapper/321:43:303
616521758,151sleep36201-21ntpq23:03:423
30213217511,34sleep330345-21latency_hist22:38:193
26233216931,114sleep30-21swapper/322:23:463
19133216718,130sleep20-21swapper/220:18:372
10425216618,126sleep30-21swapper/321:33:543
26871215813,18sleep30-21swapper/320:43:373
19106215214,120sleep219232-21sed23:43:482
1502821529,16sleep30-21swapper/323:33:203
8823215019,23sleep30-21swapper/319:43:513
1901321508,17sleep30-21swapper/320:18:303
1257021458,42sleep312659-21switchtime23:23:483
6207214412,118sleep36222-21awk21:18:523
6207214412,118sleep36222-21awk21:18:523
3268921447,123sleep032725-21ntpq19:18:420
3268921447,123sleep032725-21ntpq19:18:420
211962140106,17sleep30-21swapper/320:23:513
12162214021,41sleep30-21swapper/319:54:253
15469213916,23sleep30-21swapper/321:48:473
708021387,17sleep30-21swapper/321:23:403
708021387,17sleep30-21swapper/321:23:403
8524213720,60sleep30-21swapper/323:12:383
6947213715,104sleep17034-21fschecks_time21:23:381
6947213715,104sleep17034-21fschecks_time21:23:381
3145421335,11sleep30-21swapper/322:40:013
2467521335,10sleep30-21swapper/300:03:273
250121319,110sleep32542-21ntp_kernel_err21:08:433
250121319,110sleep32542-21ntp_kernel_err21:08:433
2448821319,104sleep30-21swapper/300:03:193
2318721315,10sleep30-21swapper/320:30:573
899212817,66sleep30-21swapper/322:48:203
69662128100,16sleep30-21swapper/319:38:423
1815021285,10sleep30-21swapper/323:43:013
1372821285,11sleep30-21swapper/319:59:173
966221269,77sleep39682-21munin-run21:33:173
15536212621,10sleep30-21swapper/320:07:323
376021258,17sleep30-21swapper/321:13:353
376021258,17sleep30-21swapper/321:13:353
26388212519,89sleep226492-21ls22:23:492
189021258,20sleep32030-21sed21:08:193
189021258,20sleep32030-21sed21:08:193
112221248,13sleep312-21rcu_sched19:23:173
112221248,13sleep312-21rcu_sched19:23:173
7245212225,78sleep27386-21sed19:38:502
1882821228,17sleep30-21swapper/323:43:413
1173221229,15sleep30-21swapper/321:38:263
28451212140,54sleep128474-21latency_hist19:08:171
27248212116,31sleep00-21swapper/020:43:460
168621218,42sleep30-21swapper/319:23:413
168621218,42sleep30-21swapper/319:23:413
23904212020,82sleep223956-21sendmail22:18:162
30521198,18sleep3315-21munin-run21:03:183
30521198,18sleep3315-21munin-run21:03:183
2907621189,17sleep329142-21modprobe00:18:163
2907621189,17sleep329142-21modprobe00:18:163
1588521188,17sleep30-21swapper/320:08:333
2989321178,13sleep30-21swapper/320:53:343
2234921177,99sleep322377-21munin-run22:13:173
2016021178,16sleep30-21swapper/323:48:383
2155721167,13sleep30-21swapper/323:53:303
1776821168,94sleep317924-21date21:58:193
155721166,12sleep30-21swapper/322:48:443
313821136,13sleep30-21swapper/319:28:213
313821136,13sleep30-21swapper/319:28:213
1938621125,11sleep30-21swapper/322:03:203
1656211010,62sleep30-21swapper/300:33:343
32368210916,76sleep132394-21netstat22:43:421
3168210811,17sleep20-21swapper/219:28:282
3168210811,17sleep20-21swapper/219:28:282
21464210810,18sleep10-21swapper/120:23:531
10035210817,73sleep010078-21ntp_kernel_pll_19:48:400
7290210710,84sleep27309-21apt-get23:08:292
28881210715,77sleep128977-21fschecks_time19:08:381
2850221075,10sleep30-21swapper/322:32:183
2850221075,10sleep30-21swapper/322:32:183
119621075,11sleep30-21swapper/300:32:003
8620210612,77sleep08678-21irqstats21:28:420
7077210623,57sleep07165-21latency_hist23:08:190
30831210623,64sleep230926-21sort19:13:392
30831210623,64sleep230926-21sort19:13:392
2523421067,61sleep30-21swapper/320:38:323
29865210520,24sleep00-21swapper/000:18:460
29865210520,24sleep00-21swapper/000:18:460
1064210516,75sleep21076-21cat22:48:272
12083210419,70sleep012100-21uniq23:23:360
10293210412,71sleep210422-21timerwakeupswit19:48:472
27725210214,70sleep027801-21/usr/sbin/munin19:03:450
27698210218,64sleep00-21swapper/022:28:430
27698210218,64sleep00-21swapper/022:28:430
25328210220,64sleep225413-21tune2fs20:38:392
1912721015,42sleep30-21swapper/322:01:383
53672995,42sleep30-21swapper/323:01:483
3054729922,54sleep030637-21grep22:38:360
273562995,41sleep30-21swapper/300:11:183
273562995,41sleep30-21swapper/300:11:183
102122990,13sleep10-21swapper/119:48:451
2999929714,68sleep030000-21sshd22:36:340
2491729721,60sleep0225-21systemd-journal20:38:170
2992629514,63sleep130000-21sensors_temp00:18:471
2992629514,63sleep130000-21sensors_temp00:18:471
316729312,63sleep12683-21apt-get19:28:271
316729312,63sleep12683-21apt-get19:28:271
1355229320,54sleep213657-21grep19:58:522
3208629218,58sleep031975-21/usr/sbin/munin22:43:370
125442899,64sleep212649-21cat19:58:192
2374328813,53sleep223827-21cut20:33:352
1822428827,28sleep10-21swapper/123:43:191
1677828813,59sleep116810-21awk21:53:411
452728730,28sleep00-21swapper/019:33:190
452728730,28sleep00-21swapper/019:33:190
1131628718,54sleep011431-21apt-key21:38:180
187628616,50sleep01912-21proc_pri19:23:460
187628616,50sleep01912-21proc_pri19:23:460
1875228628,31sleep10-21swapper/121:58:501
912928521,42sleep29273-21unixbench_singl21:28:522
2766328522,31sleep30-21swapper/319:03:443
1195628520,43sleep112099-21head19:53:541
281628221,17sleep10-21swapper/119:28:191
281628221,17sleep10-21swapper/119:28:191
3010228114,50sleep130152-21if_eth020:53:401
2392828122,33sleep10-21swapper/120:33:381
43527924,29sleep20-21swapper/222:43:512
3114927917,41sleep231204-21grep20:58:172
3048227925,22sleep00-21swapper/019:13:250
3048227925,22sleep00-21swapper/019:13:250
3007527917,44sleep230092-21cat19:13:192
2510027821,37sleep225139-21ntpq00:03:422
2290727816,46sleep122921-21kernelversion22:13:391
205912788,17sleep00-21swapper/022:05:350
293427718,40sleep23137-21sed19:28:202
293427718,40sleep23137-21sed19:28:202
238692779,51sleep024016-21unixbench_singl23:58:520
1246427716,45sleep012508-21run-parts19:58:170
2886427616,40sleep028879-21fschecks_count19:08:370
240227621,34sleep12527-21grep22:53:181
1925627623,30sleep00-21swapper/022:03:170
1120827623,33sleep211219-21modprobe21:38:162
3155527516,32sleep20-21swapper/220:58:372
3155527516,32sleep20-21swapper/220:58:372
1945127510,14sleep00-21swapper/020:18:430
11602756,53sleep00-21swapper/022:48:330
656227421,32sleep06557-21apt-get19:38:310
1417527415,25sleep20-21swapper/223:28:472
713127320,29sleep00-21swapper/021:23:410
713127320,29sleep00-21swapper/021:23:410
3257527322,29sleep00-21swapper/000:28:340
1013127325,31sleep110136-21modprobe23:18:171
3258927222,22sleep20-21swapper/219:18:402
3258927222,22sleep20-21swapper/219:18:402
2793127224,29sleep228086-21uptime19:03:472
2584527218,37sleep125939-21sed20:38:501
1409027215,36sleep1278-21dbus-daemon20:03:191
117427212,44sleep11276-21sed19:23:191
117427212,44sleep11276-21sed19:23:191
2804327014,26sleep20-21swapper/200:13:392
2804327014,26sleep20-21swapper/200:13:392
2740327029,23sleep20-21swapper/222:28:372
2740327029,23sleep20-21swapper/222:28:372
213027012,30sleep12124-21apt-get21:08:331
213027012,30sleep12124-21apt-get21:08:331
971526921,27sleep10-21swapper/121:33:191
2256326911,42sleep022565-21ntp_states20:28:440
216462699,46sleep021650-21entropy23:53:350
167126918,28sleep10-21swapper/100:33:361
31726814,26sleep00-21swapper/021:03:190
31726814,26sleep00-21swapper/021:03:190
262562689,13sleep00-21swapper/022:23:470
206026818,26sleep20-21swapper/200:33:452
1952126813,39sleep20-21swapper/222:03:312
1901626815,28sleep10-21swapper/120:18:321
1780326816,34sleep017831-21/usr/sbin/munin20:13:430
1386226815,22sleep00-21swapper/021:43:470
572126719,19sleep20-21swapper/219:34:302
441326714,21sleep00-21swapper/021:13:490
441326714,21sleep00-21swapper/021:13:490
2715626713,35sleep127203-21/usr/sbin/munin20:43:431
2672026710,43sleep026784-21ps00:08:450
2672026710,43sleep026784-21ps00:08:450
2516626718,36sleep125203-21ps00:03:451
1894926720,19sleep00-21swapper/023:43:460
185426719,30sleep01918-21cat00:33:390
2564226610,13sleep00-21swapper/020:38:450
1204826614,24sleep10-21swapper/121:38:481
3030426511,39sleep130316-21fschecks_time19:13:221
3030426511,39sleep130316-21fschecks_time19:13:221
3000126522,24sleep20-21swapper/200:18:482
3000126522,24sleep20-21swapper/200:18:482
2685726517,24sleep20-21swapper/200:08:472
2685726517,24sleep20-21swapper/200:08:472
2284926518,23sleep00-21swapper/022:13:380
2132226517,24sleep00-21swapper/023:53:190
83042647,14sleep00-21swapper/019:43:400
65472647,10sleep10-21swapper/121:23:021
65472647,10sleep10-21swapper/121:23:021
623126325,23sleep20-21swapper/223:03:452
1405326323,19sleep20-21swapper/221:43:502
2553326217,19sleep00-21swapper/022:23:190
205322628,43sleep00-21swapper/020:23:240
1792126218,19sleep10-21swapper/120:13:481
1609426219,25sleep10-21swapper/120:08:391
835626113,33sleep18385-21irqstats19:43:411
36582616,12sleep20-21swapper/222:55:132
276526119,24sleep12849-21perl22:53:391
2240026115,33sleep122456-21sed22:13:191
1838526120,21sleep218419-21ntp_kernel_pll_21:58:432
1691426114,32sleep216993-21mailstats21:53:462
1564626111,36sleep215719-21cat20:08:202
1558026116,18sleep10-21swapper/123:34:231
992526016,31sleep29928-21sshd23:15:392
548126013,32sleep25516-21idleruntime-cro23:03:162
2787326017,24sleep10-21swapper/122:28:491
2787326017,24sleep10-21swapper/122:28:491
2702726025,20sleep10-21swapper/122:28:161
1330626019,21sleep20-21swapper/223:28:192
1044726011,28sleep210444-21apt-get23:18:302
75625915,29sleep1796-21sort21:03:411
75625915,29sleep1796-21sort21:03:411
664925914,17sleep10-21swapper/119:38:361
576925912,27sleep10-21swapper/123:03:311
39925924,17sleep10-21swapper/100:28:391
3268125915,22sleep20-21swapper/200:28:362
2923725916,20sleep20-21swapper/222:33:422
1325525917,27sleep1278-21dbus-daemon23:28:181
1307825914,18sleep00-21swapper/021:43:190
64725813,16sleep20-21swapper/221:03:372
64725813,16sleep20-21swapper/221:03:372
63272588,18sleep10-21swapper/119:38:201
54125817,21sleep00-21swapper/021:03:310
54125817,21sleep00-21swapper/021:03:310
322522589,33sleep032368-21sed00:28:180
2817925814,26sleep128223-21ntpq00:13:421
2817925814,26sleep128223-21ntpq00:13:421
2422225826,19sleep20-21swapper/222:18:322
220525817,25sleep02212-21df_inode21:08:360
220525817,25sleep02212-21df_inode21:08:360
2157625815,25sleep221632-21df23:53:332
2083725815,28sleep220896-21latency20:23:422
180702585,11sleep20-21swapper/223:41:592
1483725810,27sleep00-21swapper/021:48:280
319832578,37sleep20-21swapper/200:25:302
3113025713,26sleep10-21swapper/100:23:391
2912625710,19sleep00-21swapper/020:48:500
2489825713,27sleep125025-21/usr/sbin/munin22:18:481
1143225714,26sleep111229-21apt-get21:38:191
557225614,17sleep00-21swapper/019:34:280
553225619,17sleep00-21swapper/023:03:180
421425614,25sleep14211-21apt-get22:58:311
211492566,10sleep20-21swapper/223:51:432
2061825621,17sleep10-21swapper/123:48:481
1656325610,35sleep20-21swapper/223:37:512
811325512,24sleep28107-21apt-get19:43:342
502625515,25sleep15039-21munin-run21:18:181
502625515,25sleep15039-21munin-run21:18:181
367425514,24sleep23670-21apt-get21:13:302
367425514,24sleep23670-21apt-get21:13:302
3176025517,22sleep20-21swapper/222:43:182
2220325516,25sleep122276-21fschecks_time20:28:371
1641025512,26sleep016404-21apt-get21:53:320
1588425514,25sleep215880-21apt-get20:08:322
1128525518,20sleep10-21swapper/123:18:491
723025416,18sleep20-21swapper/221:23:432
723025416,18sleep20-21swapper/221:23:432
2109425411,17sleep00-21swapper/022:08:330
3155025314,23sleep131554-21diskstats20:58:361
3155025314,23sleep131554-21diskstats20:58:361
2678225314,23sleep226778-21apt-get20:43:322
257072536,38sleep00-21swapper/000:05:010
2314525312,24sleep123139-21apt-get23:58:331
221825316,18sleep20-21swapper/221:08:382
221825316,18sleep20-21swapper/221:08:382
1691025313,23sleep016905-21apt-get23:38:300
1354125314,22sleep013537-21apt-get23:28:290
1302625317,18sleep20-21swapper/221:43:172
556125214,22sleep05583-21irqstats21:18:410
556125214,22sleep05583-21irqstats21:18:410
3049825213,16sleep10-21swapper/122:38:331
2989525212,19sleep00-21swapper/020:53:350
2360925214,23sleep223610-21ntp_states23:58:462
2210725212,24sleep222104-21apt-get20:28:302
1718325216,19sleep00-21swapper/020:13:190
165452525,37sleep00-21swapper/023:37:370
309725113,22sleep03103-21ntp_states22:53:460
1396125111,22sleep113997-21ntp_states23:28:421
863925017,16sleep00-21swapper/023:13:190
2780125016,16sleep00-21swapper/000:13:330
2780125016,16sleep00-21swapper/000:13:330
1751324912,22sleep217519-21diskstats20:13:372
1486024911,20sleep214856-21apt-get21:48:302
1441124913,20sleep114422-21diskstats20:03:351
2109124810,18sleep20-21swapper/222:08:312
1870624810,21sleep118735-21irqstats23:43:391
1523224811,20sleep10-21swapper/121:48:411
1278524811,21sleep012778-21apt-get19:58:330
1482424716,13sleep00-21swapper/020:03:460
886324610,18sleep00-21swapper/023:13:310
1962624612,17sleep10-21swapper/122:03:381
1340224614,19sleep113425-21fschecks_count21:43:361
109102469,16sleep00-21swapper/023:18:440
2665424511,22sleep126224-21/usr/sbin/munin00:08:421
2665424511,22sleep126224-21/usr/sbin/munin00:08:421
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional