You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-12 - 15:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Wed Nov 12, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
235822121910,1192sleep18967-1kworker/1:0H10:09:251
30402288412,858sleep24504-1kworker/2:0H12:14:292
30402288412,858sleep24504-1kworker/2:0H12:14:292
28248259494,412sleep328388-21unixbench_singl08:39:433
28248259494,412sleep328388-21unixbench_singl08:39:433
23689247415,442sleep023759-21sshd08:24:460
10670238321,348sleep210663-21apt-get09:29:292
10670238321,348sleep210663-21apt-get09:29:292
17222232044,198sleep317291-21fschecks_time09:49:323
17651229513,268sleep03345-21kworker/u8:111:34:220
19908227310,242sleep320010-21grep11:39:463
19556226913,241sleep319710-21modprobe08:14:153
23064225718,221sleep123203-21head11:49:501
15825225351,36sleep30-21swapper/309:44:443
5483223214,204sleep35503-21uname10:54:413
5483223214,204sleep35503-21uname10:54:413
32301222023,101sleep332348-21fschecks_count08:54:333
27109222055,153sleep327199-21fschecks_time12:04:333
8438221933,119sleep38453-21fschecks_count11:04:343
32022221651,15sleep30-21swapper/312:19:313
14539221414,137sleep314602-21unin-run07:59:133
2219221018,171sleep32232-21fschecks_count07:19:363
22559220215,108sleep322581-21awk08:24:133
22559220215,108sleep322581-21awk08:24:133
1441122019,181sleep0362-21snmpd07:57:370
2687220029,110sleep32758-21fschecks_time12:29:323
2083219618,93sleep30-21swapper/310:44:313
2083219618,93sleep30-21swapper/310:44:313
9798218684,31sleep39967-21dpkg07:44:153
31917218374,94sleep331981-21find07:09:433
30365218313,154sleep129864-21apt-get12:14:201
30365218313,154sleep129864-21apt-get12:14:201
2122421839,162sleep321235-21ntpq11:44:393
975217611,36sleep30-21swapper/310:39:393
23990217354,105sleep324020-21grep11:54:323
28007217115,134sleep028067-21sendmail_mailqu08:39:400
28007217115,134sleep028067-21sendmail_mailqu08:39:400
26063216917,137sleep026117-21cut08:34:310
26063216917,137sleep026117-21cut08:34:310
26722216540,113sleep326767-21systemd-run12:04:133
31898216054,88sleep30-21swapper/308:54:133
28565215630,81sleep328603-21latency10:24:373
3048221537,41sleep30-21swapper/312:14:343
3048221537,41sleep30-21swapper/312:14:343
765621527,128sleep30-21swapper/309:19:323
3255214631,103sleep33300-21awk09:04:383
2334321455,11sleep30-21swapper/311:51:013
9581214337,19sleep30-21swapper/309:24:413
18533214240,43sleep30-21swapper/308:09:353
28166214124,90sleep00-21swapper/006:59:350
21579214132,85sleep321634-21latency_hist10:04:143
995221398,40sleep30-21swapper/311:09:353
995221398,40sleep30-21swapper/311:09:353
30216213915,32sleep330349-21unixbench_singl07:04:473
844213853,17sleep30-21swapper/307:14:363
31693213815,110sleep331713-21uname10:34:343
31693213815,110sleep331713-21uname10:34:343
7352213620,73sleep30-21swapper/307:34:423
15028213514,100sleep015087-21sendmail_mailqu11:24:410
5229213316,28sleep30-21swapper/307:29:303
5229213316,28sleep30-21swapper/307:29:303
5997213214,104sleep15998-21/usr/sbin/munin09:14:261
7267213013,102sleep17288-21awk07:34:401
4891213044,19sleep30-21swapper/310:54:123
4891213044,19sleep30-21swapper/310:54:123
30596213022,31sleep30-21swapper/310:29:463
30024212945,46sleep330078-21munin-run12:14:143
1137212841,41sleep30-21swapper/312:24:313
676821279,23sleep36764-21apt-get10:59:293
6531212312,91sleep30-21swapper/309:14:443
28649212321,80sleep128665-21fschecks_count12:09:321
4630212035,63sleep30-21swapper/309:09:353
1854321190,33sleep218550-21unixbench_singl11:34:442
16473211928,23sleep30-21swapper/311:29:373
13850211710,14sleep30-21swapper/311:19:443
12994211621,73sleep213146-21grep09:34:452
12994211621,73sleep213146-21grep09:34:452
10113211626,20sleep30-21swapper/307:44:233
24609211512,24sleep30-21swapper/308:29:343
24105211515,63sleep324170-21processes10:09:593
31712211313,16sleep10-21swapper/107:09:371
13397211317,24sleep00-21swapper/009:37:460
13397211317,24sleep00-21swapper/009:37:460
8819211212,84sleep18832-21/usr/sbin/munin07:39:411
3071821129,18sleep00-21swapper/008:49:330
2565821109,29sleep30-21swapper/310:14:443
17834211022,75sleep30-21swapper/308:08:423
1083211014,80sleep01135-21ls08:59:140
11999210510,14sleep30-21swapper/309:34:143
11999210510,14sleep30-21swapper/309:34:143
9573210415,70sleep29580-21/usr/sbin/munin09:24:402
26249210429,59sleep30-21swapper/308:34:343
26249210429,59sleep30-21swapper/308:34:343
3860210216,65sleep23881-21modprobe12:34:112
28391210217,70sleep028441-21cut10:24:340
22285210215,74sleep322340-21processes10:04:433
22285210215,74sleep322340-21processes10:04:433
20814210214,72sleep020882-21/usr/sbin/munin09:59:430
15467210220,65sleep115511-21ntpq07:59:521
10783210015,65sleep010890-21timerandwakeup07:45:050
2250129921,56sleep222648-21unixbench_multi10:04:462
2250129921,56sleep222648-21unixbench_multi10:04:462
2194929916,17sleep00-21swapper/008:19:460
2194929916,17sleep00-21swapper/008:19:460
313129810,13sleep10-21swapper/112:29:421
1121729813,67sleep211350-21latency_hist11:14:142
1121729813,67sleep211350-21latency_hist11:14:142
878929714,67sleep08847-21processes11:04:430
404329724,54sleep30-21swapper/307:24:373
334029711,72sleep03387-21sh10:49:130
334029711,72sleep03387-21sh10:49:130
276902975,28sleep30-21swapper/310:20:093
2099029713,22sleep221124-21unixbench_singl09:59:452
1750529716,67sleep017545-21ntp_kernel_err09:49:370
315329614,66sleep13218-21grep09:04:361
1923629510,69sleep019297-21cpuspeed11:39:310
14172959,64sleep30-21swapper/308:59:343
119742955,12sleep00-21swapper/007:49:410
2573929412,60sleep025814-21latency_hist08:34:150
1265229424,17sleep30-21swapper/309:34:383
1265229424,17sleep30-21swapper/309:34:383
1368729222,46sleep213846-21grep11:19:432
294952918,37sleep30-21swapper/308:44:403
2620729117,51sleep126351-21grep11:59:471
1475729118,62sleep30-21swapper/309:40:203
2415028911,43sleep324158-21systemd-run08:29:133
234228914,59sleep12393-21if_eth007:19:381
2179228918,56sleep221863-21mailstats08:19:442
2179228918,56sleep221863-21mailstats08:19:442
1581128912,61sleep215824-21/usr/sbin/munin09:44:432
3073628717,48sleep130760-21munin-plugin-st07:09:131
126212875,31sleep30-21swapper/311:17:583
1043228723,34sleep00-21swapper/009:29:140
982328624,36sleep29778-21apt-get11:09:292
3185828613,19sleep00-21swapper/007:09:400
1045228619,53sleep110528-21switchtime11:09:491
1045228619,53sleep110528-21switchtime11:09:491
888128519,45sleep19014-21latency_hist09:24:151
1994628411,61sleep019948-21sshd09:58:460
1994628411,61sleep019948-21sshd09:58:460
1188528314,54sleep211943-21if_err_eth007:49:392
2048928222,38sleep120512-21modprobe11:44:121
1986128218,54sleep30-21swapper/309:57:403
1986128218,54sleep30-21swapper/309:57:403
127872827,12sleep30-21swapper/307:52:073
2835528125,20sleep10-21swapper/106:59:411
1570428011,20sleep30-21swapper/307:59:583
2922027934,29sleep329262-21cyclictest07:04:143
264727917,28sleep00-21swapper/009:04:150
872627819,31sleep10-21swapper/111:04:401
2720427818,44sleep027263-21if_eth012:04:340
184027815,47sleep24504-1kworker/2:0H10:44:142
1581327810,31sleep30-21swapper/311:29:133
648127721,35sleep16511-21apt07:34:121
648127721,35sleep16511-21apt07:34:121
518027722,31sleep25322-21sshd09:09:462
3189427723,29sleep10-21swapper/110:34:401
3189427723,29sleep10-21swapper/110:34:401
1595927712,44sleep116062-21/usr/sbin/munin09:44:461
1665227624,27sleep00-21swapper/011:29:420
1143327611,50sleep011452-21apt-get11:14:240
656227514,46sleep16614-21latency_hist10:59:151
656227514,46sleep16614-21latency_hist10:59:151
2948127520,29sleep00-21swapper/007:04:270
2476927514,46sleep224800-21nfs_client08:29:372
1764027517,39sleep217721-21cut09:49:522
3245627411,24sleep20-21swapper/212:19:462
310327414,41sleep00-21swapper/009:04:350
1335627414,44sleep213363-21/usr/sbin/munin07:54:322
570227319,31sleep00-21swapper/009:14:120
394227316,31sleep00-21swapper/012:34:130
2916027312,45sleep029167-21diskstats08:44:330
2915127317,41sleep229219-21systemctl07:04:132
2758927322,28sleep10-21swapper/112:04:431
1839827315,43sleep118541-21unixbench_singl11:34:441
183027315,44sleep011-21ksoftirqd/007:19:130
961127220,35sleep09626-21apt-get07:44:120
724027213,43sleep27274-21ps10:59:492
1660927220,31sleep00-21swapper/009:49:130
269912716,12sleep20-21swapper/210:19:352
2404127114,41sleep024088-21ntpq10:09:560
186432717,11sleep30-21swapper/311:35:223
97927016,23sleep20-21swapper/207:14:392
76227023,30sleep0788-21awk12:24:120
3009727019,21sleep20-21swapper/207:04:462
2208827020,25sleep20-21swapper/211:49:142
1708127012,42sleep017088-21ntpq08:04:440
159427016,29sleep20-21swapper/208:59:372
118062698,12sleep00-21swapper/009:32:370
118062698,12sleep00-21swapper/009:32:370
68372688,14sleep20-21swapper/209:14:472
497626817,37sleep04986-21/usr/sbin/munin07:29:140
379426816,35sleep23817-21tune2fs07:24:322
1994326826,22sleep10-21swapper/108:14:381
383026720,23sleep10-21swapper/110:49:461
383026720,23sleep10-21swapper/110:49:461
3172026715,35sleep031795-21memory10:34:350
3172026715,35sleep031795-21memory10:34:350
2967426713,36sleep1225-21systemd-journal10:29:151
2780126725,24sleep20-21swapper/208:39:342
2780126725,24sleep20-21swapper/208:39:342
2711026718,24sleep00-21swapper/010:19:370
2051726714,23sleep00-21swapper/008:14:490
2031526718,36sleep220385-21expr08:14:472
1911126719,31sleep119153-21ntpq09:54:391
1911126719,31sleep119153-21ntpq09:54:391
1360626720,28sleep10-21swapper/111:19:421
3047226613,19sleep20-21swapper/210:29:452
2992726621,22sleep20-21swapper/212:14:132
2495526620,26sleep20-21swapper/210:14:142
1472126623,19sleep10-21swapper/107:59:151
394426514,37sleep13989-21latency07:24:351
162426512,20sleep20-21swapper/212:24:432
3129026415,26sleep231393-21kernelversion07:09:222
2522526413,27sleep10-21swapper/111:59:141
2448526427,21sleep10-21swapper/108:29:321
1093826416,22sleep10-21swapper/109:29:361
1093826416,22sleep10-21swapper/109:29:361
78682636,45sleep00-21swapper/011:02:090
57626314,36sleep1645-21df07:14:301
452026319,30sleep14545-21fschecks_count09:09:331
3165926314,31sleep031697-21modprobe12:19:120
3165926314,31sleep031697-21modprobe12:19:120
2882926318,29sleep228900-21sensors_temp10:24:442
2723726312,28sleep10-21swapper/108:39:131
2723726312,28sleep10-21swapper/108:39:131
2242226310,38sleep122500-21grep10:04:451
2242226310,38sleep122500-21grep10:04:451
1548626318,26sleep10-21swapper/111:25:031
1470826316,28sleep113770-21/usr/sbin/munin09:39:451
81892628,38sleep28260-21latency_hist07:39:182
775026219,24sleep00-21swapper/009:19:330
326032627,11sleep20-21swapper/210:38:042
326032627,11sleep20-21swapper/210:38:042
2437026210,38sleep024407-21ps11:54:410
1276526221,25sleep10-21swapper/111:19:131
738626113,25sleep20-21swapper/209:19:152
673326116,22sleep00-21swapper/010:59:260
56172615,15sleep20-21swapper/210:54:442
56172615,15sleep20-21swapper/210:54:442
2956026120,26sleep229624-21ps08:44:432
2866526115,26sleep10-21swapper/110:24:391
1655626116,28sleep116562-21ntp_states11:29:391
1029426123,21sleep10-21swapper/107:44:541
2608026013,20sleep00-21swapper/011:59:460
2517226016,22sleep20-21swapper/210:14:262
2294926020,19sleep00-21swapper/011:49:490
140872605,44sleep00-21swapper/011:21:550
1386026013,32sleep013870-21df_inode09:39:280
1302926019,19sleep10-21swapper/107:54:151
113726014,30sleep11281-21cat08:59:151
3195325915,27sleep131934-21apt-get12:19:291
3119325915,24sleep00-21swapper/010:34:150
2883225920,20sleep10-21swapper/108:44:131
2883225920,20sleep10-21swapper/108:44:131
264552598,13sleep00-21swapper/010:19:130
244872595,12sleep20-21swapper/211:54:432
2205925919,22sleep10-21swapper/108:19:471
2205925919,22sleep10-21swapper/108:19:471
20682596,41sleep00-21swapper/012:26:020
1843225916,27sleep118448-21/usr/sbin/munin08:09:331
1545225917,23sleep00-21swapper/007:59:510
938725816,21sleep10-21swapper/109:24:351
802625813,23sleep18057-21munin-run11:04:121
3064225814,20sleep20-21swapper/208:49:312
2546325812,32sleep2362-21snmpd11:59:272
968725718,23sleep29797-21apt-key07:44:142
62732575,12sleep20-21swapper/207:31:312
62732575,12sleep20-21swapper/207:31:312
304825712,27sleep22105-21/usr/sbin/munin07:19:522
269325716,19sleep20-21swapper/210:44:472
269325716,19sleep20-21swapper/210:44:472
2619025715,25sleep226248-21irqstats08:34:332
2619025715,25sleep226248-21irqstats08:34:332
165632579,26sleep20-21swapper/211:29:402
307252569,34sleep130764-21fschecks_count08:49:341
2977425618,21sleep10-21swapper/108:44:461
279092567,18sleep10-21swapper/110:22:571
120825620,16sleep00-21swapper/010:39:440
602225514,24sleep06017-21apt-get09:14:290
551025513,17sleep00-21swapper/010:54:420
370425515,24sleep03710-21df_inode10:49:290
370425515,24sleep03710-21df_inode10:49:290
2014725510,22sleep20-21swapper/211:39:492
1951125516,22sleep20-21swapper/208:14:142
1515825516,20sleep20-21swapper/211:24:432
144312558,31sleep214557-21sed09:39:422
133522545,12sleep10-21swapper/109:37:111
133522545,12sleep10-21swapper/109:37:111
680525313,23sleep06799-21apt-get07:34:270
522625315,22sleep05222-21apt-get07:29:270
522625315,22sleep05222-21apt-get07:29:270
288525313,24sleep22878-21apt-get09:04:292
2728625312,23sleep227380-21memory12:04:362
2307125313,24sleep123124-21irqstats08:24:351
22412539,27sleep00-21swapper/007:19:370
2035425317,17sleep10-21swapper/109:59:321
235425217,19sleep10-21swapper/112:29:141
1036525212,23sleep010451-21mailstats11:09:480
1036525212,23sleep010451-21mailstats11:09:480
89625113,19sleep10-21swapper/110:39:371
3015725114,18sleep00-21swapper/012:14:150
1788325115,21sleep20-21swapper/208:09:142
363125011,22sleep20-21swapper/210:49:272
363125011,22sleep20-21swapper/210:49:272
3229625011,22sleep232300-21diskstats08:54:322
2123925014,18sleep10-21swapper/111:44:411
1823825014,22sleep018244-21df_inode08:09:280
139925014,21sleep11406-21diskstats08:59:321
2107824910,22sleep021145-21memory11:44:360
2519724812,22sleep125192-21apt-get10:14:301
2163824814,17sleep00-21swapper/010:04:150
2163824814,17sleep00-21swapper/010:04:150
1720924812,21sleep117216-21df_inode09:49:301
1674424811,22sleep116600-21/usr/sbin/munin08:04:371
1200924811,26sleep212085-21sed11:14:422
1054324813,21sleep210584-21ntp_kernel_err07:44:592
231924710,20sleep12352-21irqstats10:44:381
231924710,20sleep12352-21irqstats10:44:381
161682475,34sleep20-21swapper/208:02:112
2293624610,21sleep222940-21diskstats08:24:322
223382459,20sleep222407-21df11:49:342
1998124412,14sleep20-21swapper/209:59:122
1998124412,14sleep20-21swapper/209:59:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional