You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-15 - 21:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Sep 15, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2103521104415,11005sleep220352-21kworker/u8:109:08:162
68762396812,3945sleep2811-21kworker/2:210:04:522
2275272817,695sleep02313-21/usr/sbin/munin08:08:020
268342518327,167sleep326855-21awk09:27:353
28096247614,447sleep228100-21ntpq11:13:032
28096247614,447sleep228100-21ntpq11:13:032
8090239827,238sleep38237-21grep10:08:083
20456236532,259sleep320547-21sed10:48:053
32122232765,245sleep332051-21latency_hist11:27:373
2738123206,297sleep06510-1kworker/0:2H09:28:000
98612299128,41sleep3225-21systemd-journal10:13:253
11442271169,25sleep30-21swapper/309:47:503
1053922660,13sleep110541-21grep12:02:501
1053922660,13sleep110541-21grep12:02:501
286272228100,46sleep30-21swapper/309:32:353
456222444,167sleep332747-21/usr/sbin/munin08:03:253
4638221848,146sleep30-21swapper/311:42:393
4638221848,146sleep30-21swapper/311:42:393
27526220811,183sleep227528-21fschecks_count09:28:222
5486220748,81sleep35616-21grep11:43:153
5486220748,81sleep35616-21grep11:43:153
17567220031,147sleep30-21swapper/308:57:553
28101219718,20sleep328141-21processes11:13:043
28101219718,20sleep328141-21processes11:13:043
16051219619,32sleep30-21swapper/307:12:373
15272219329,147sleep315380-21fschecks_time10:32:553
15272219329,147sleep315380-21fschecks_time10:32:553
21089219210,167sleep321112-21latency_hist07:27:353
6834218916,157sleep36895-21latency08:22:593
6834218916,157sleep36895-21latency08:22:593
16937218715,150sleep316951-21fschecks_count12:22:563
23176217920,33sleep323227-21cpuspeed07:32:523
14820217510,20sleep314850-21ps08:48:063
907217411,21sleep30-21swapper/309:47:373
1410321749,150sleep314245-21cat08:47:383
9087217016,22sleep30-21swapper/308:28:523
9087217016,22sleep30-21swapper/308:28:523
27190215714,100sleep327206-21vmstat07:43:063
25123215716,31sleep325167-21ntp_kernel_err07:38:013
25797215410,17sleep30-21swapper/311:07:353
15172215010,122sleep00-21swapper/010:32:480
15172215010,122sleep00-21swapper/010:32:480
22299214846,40sleep322314-21awk10:53:103
122952147100,25sleep30-21swapper/312:08:393
32748214521,100sleep132275-21apt-get08:02:461
30858214410,116sleep30-21swapper/309:38:003
29906214212,115sleep330033-21unixbench_multi11:18:083
3431214080,46sleep33456-21cpu08:12:543
1710221366,41sleep30-21swapper/308:56:503
32289213369,51sleep332356-21cat08:02:363
674321320,12sleep10-21swapper/108:22:561
674321320,12sleep10-21swapper/108:22:561
528721315,40sleep30-21swapper/309:59:203
2369221318,40sleep30-21swapper/309:17:253
2369221318,40sleep30-21swapper/309:17:253
6846212883,23sleep30-21swapper/311:48:043
21735212812,97sleep221774-21cut07:28:152
30187212710,80sleep330256-21unin-run09:37:363
18659212710,97sleep30-21swapper/312:27:573
18659212710,97sleep30-21swapper/312:27:573
1061921258,80sleep310722-21perl10:17:533
2162421247,82sleep321614-21apt-get07:27:523
1877021208,64sleep30-21swapper/310:43:083
20856211921,78sleep120364-21apt-get09:07:461
29320211817,78sleep229426-21fschecks_time11:17:552
10136211883,16sleep30-21swapper/312:02:373
31820211719,78sleep031970-21wc09:42:380
984921146,86sleep00-21swapper/008:32:570
3187211412,64sleep30-21swapper/308:12:373
16846211418,78sleep216937-21/usr/sbin/munin10:37:552
16002211412,88sleep216050-21unin-run07:12:362
11086211424,71sleep011241-21head12:03:030
11086211424,71sleep011241-21head12:03:030
20180211312,22sleep020219-21memory10:47:580
25874211212,85sleep225875-21sshd07:40:512
2704621110,6sleep20-21swapper/209:27:372
15261211114,81sleep215271-21/usr/sbin/munin10:32:542
15261211114,81sleep215271-21/usr/sbin/munin10:32:542
12478211118,72sleep212514-21latency12:08:422
314210913,79sleep232766-21apt-get08:02:512
12196210721,66sleep112143-21apt-get12:08:251
1880721059,25sleep318903-21mailstats07:18:193
1654210515,21sleep00-21swapper/011:32:470
11694210417,66sleep211851-21latency_hist12:07:372
11694210417,66sleep211851-21latency_hist12:07:372
29074210327,42sleep30-21swapper/307:52:063
29074210327,42sleep30-21swapper/307:52:063
17858210321,11sleep30-21swapper/310:40:343
3123721028,73sleep20-21swapper/211:23:042
269221029,15sleep30-21swapper/309:52:503
13274210223,11sleep30-21swapper/310:26:533
1124721028,16sleep30-21swapper/308:37:533
10276210217,66sleep010424-21latency_hist10:17:380
2780210121,56sleep02811-21fschecks_count09:52:550
27132210138,52sleep30-21swapper/311:09:493
2096221015,33sleep30-21swapper/312:34:233
32360210018,66sleep032522-21apt-key08:02:370
23704210019,67sleep023725-21latency_hist09:17:340
23704210019,67sleep023725-21latency_hist09:17:340
2034521008,16sleep320413-21apt-config09:07:363
229842997,14sleep30-21swapper/310:57:543
294792986,13sleep30-21swapper/307:52:553
294792986,13sleep30-21swapper/307:52:553
2128229814,60sleep30-21swapper/309:08:213
2429729721,60sleep124000-21/usr/sbin/munin09:18:001
1654529721,43sleep30-21swapper/312:22:243
689929623,41sleep30-21swapper/310:05:093
2218329616,19sleep00-21swapper/007:28:250
1810729617,64sleep1313-21rs:main1
16252967,14sleep30-21swapper/311:32:423
1281929526,45sleep013070-21apt-key07:02:380
1069829421,41sleep30-21swapper/308:34:423
1043929450,12sleep30-21swapper/312:02:493
1043929450,12sleep30-21swapper/312:02:493
765029313,64sleep27744-21memory10:07:592
3017829323,40sleep230184-21modprobe09:37:352
1956229225,44sleep1225-21systemd-journal10:47:341
1715029249,29sleep30-21swapper/307:13:023
606729122,41sleep30-21swapper/308:19:563
92722905,31sleep30-21swapper/311:55:593
558929018,50sleep15557-21latency_hist10:02:371
2609329021,48sleep10-21swapper/111:07:531
2195729021,44sleep022023-21cat10:53:060
309832896,12sleep30-21swapper/311:22:573
2804028916,55sleep227828-21/usr/sbin/munin07:47:582
2804028916,55sleep227828-21/usr/sbin/munin07:47:582
2431028917,52sleep224461-21cat11:02:372
1472528947,10sleep30-21swapper/312:14:243
251312885,10sleep30-21swapper/309:20:363
1595128813,59sleep216073-21timerwakeupswit12:18:072
140732887,14sleep30-21swapper/310:28:023
2690628714,55sleep127045-21sed09:27:371
2274028723,43sleep222832-21latency_hist10:57:362
3125428613,60sleep231308-21sed07:57:592
1438628621,45sleep014553-21unixbench_multi12:13:080
1287328522,37sleep20-21swapper/208:42:522
890428426,34sleep28923-21apt-get10:12:472
1759228420,46sleep017735-21grep12:23:090
778528319,35sleep00-21swapper/011:52:370
789528211,19sleep20-21swapper/208:27:382
789528211,19sleep20-21swapper/208:27:382
278662820,7sleep00-21swapper/011:12:560
278662820,7sleep00-21swapper/011:12:560
52742818,55sleep00-21swapper/011:43:120
52742818,55sleep00-21swapper/011:43:120
785628013,53sleep07918-21grep10:08:050
3213327926,26sleep10-21swapper/109:42:551
32127915,46sleep2343-21/usr/sbin/munin11:28:052
70742789,13sleep00-21swapper/010:07:240
501427823,36sleep15082-21df08:17:521
2051427823,32sleep120720-21cut09:07:381
245527714,41sleep22581-21latency_hist09:52:372
77982764,61sleep00-21swapper/008:27:160
77982764,61sleep00-21swapper/008:27:160
3025827612,50sleep030317-21latency_hist09:37:370
1957427519,39sleep219612-21munin-run10:47:352
1364627521,24sleep30-21swapper/307:03:213
1345427520,35sleep213548-21memory07:03:162
1215727520,29sleep10-21swapper/110:22:561
1815227418,27sleep20-21swapper/212:27:362
132422749,45sleep10-21swapper/108:42:591
1260927419,28sleep00-21swapper/008:42:380
303302737,54sleep00-21swapper/011:20:360
2755827325,27sleep00-21swapper/007:47:350
249852737,12sleep00-21swapper/009:18:430
2222627314,43sleep022350-21latency_hist09:12:380
187662738,15sleep10-21swapper/112:27:591
187662738,15sleep10-21swapper/112:27:591
161342738,13sleep00-21swapper/007:12:390
83227219,30sleep00-21swapper/008:03:340
2385227218,29sleep20-21swapper/207:33:082
2317227210,45sleep10-21swapper/107:32:501
2031827217,40sleep120384-21awk10:48:031
1482327219,20sleep00-21swapper/007:07:540
1362027216,43sleep013679-21cpuspeed10:27:510
209242715,11sleep10-21swapper/112:33:561
161327116,23sleep00-21swapper/009:48:030
1443027119,24sleep10-21swapper/108:47:561
904727023,30sleep20-21swapper/211:53:202
324627013,42sleep03237-21apt-get11:37:520
324627013,42sleep03237-21apt-get11:37:520
2832027011,12sleep10-21swapper/107:48:051
2832027011,12sleep10-21swapper/107:48:051
2252527019,36sleep122561-21/usr/sbin/munin09:12:551
2252527019,36sleep122561-21/usr/sbin/munin09:12:551
1661227017,27sleep00-21swapper/012:22:370
3111026916,36sleep231220-21timerandwakeup09:38:082
2585026916,38sleep125958-21sed11:07:371
1267626913,42sleep012703-21proc_pri12:08:480
454326817,22sleep00-21swapper/009:58:000
167426816,22sleep10-21swapper/109:48:041
842026716,25sleep00-21swapper/011:53:100
760426714,32sleep21-21systemd11:52:352
350426724,21sleep10-21swapper/108:12:561
2600926714,32sleep126027-21idleruntime-cro07:42:361
1835326710,41sleep218357-21df07:17:532
3073826617,25sleep00-21swapper/007:57:360
3073826617,25sleep00-21swapper/007:57:360
300026612,40sleep13035-21latency_hist11:37:361
22332667,38sleep20-21swapper/208:08:002
2202626621,22sleep10-21swapper/110:53:071
1598926622,26sleep00-21swapper/008:52:560
501026513,36sleep20-21swapper/208:17:492
2617626511,21sleep00-21swapper/011:07:560
2130226515,33sleep221503-21apt-config07:27:392
2111626515,32sleep021160-21munin-run07:27:360
1864226520,25sleep10-21swapper/107:18:011
3266226412,36sleep0323-21sed09:43:060
316026414,24sleep20-21swapper/209:53:052
308622649,13sleep10-21swapper/111:22:551
1775026417,25sleep20-21swapper/208:57:582
1565426414,25sleep20-21swapper/208:52:372
1461926422,23sleep00-21swapper/008:47:590
117326417,26sleep11242-21apt-key11:32:361
1021826414,32sleep110288-21wc08:33:061
254532636,12sleep00-21swapper/011:03:120
1011326310,35sleep210135-21sort12:02:362
3259326215,30sleep032659-21latency11:27:580
1724026210,37sleep117353-21cat08:57:371
2912626119,22sleep10-21swapper/109:32:581
257462615,43sleep00-21swapper/007:39:110
2128826118,18sleep20-21swapper/212:37:372
2041626120,26sleep020513-21cut09:07:370
1587326125,21sleep10-21swapper/112:18:061
947926020,17sleep10-21swapper/110:13:071
872726010,34sleep28831-21wc08:28:072
872726010,34sleep28831-21wc08:28:072
364426015,27sleep20-21swapper/211:38:022
364426015,27sleep20-21swapper/211:38:022
281426015,16sleep10-21swapper/109:52:561
2357326015,29sleep023616-21ntpq07:33:020
2268026012,33sleep222713-21irqstats09:13:002
2268026012,33sleep222713-21irqstats09:13:002
2044826016,20sleep00-21swapper/012:33:070
1890026015,29sleep218978-21mailstats12:28:042
1890026015,29sleep218978-21mailstats12:28:042
1369726011,20sleep20-21swapper/210:27:532
942525910,34sleep09536-21date08:32:370
942525910,34sleep09536-21date08:32:370
604225912,20sleep10-21swapper/110:03:011
501325916,26sleep05007-21apt-get08:17:510
478325917,26sleep14778-21apt-get11:42:491
478325917,26sleep14778-21apt-get11:42:491
2831325915,18sleep00-21swapper/007:48:040
2831325915,18sleep00-21swapper/007:48:040
2038525910,27sleep20-21swapper/210:48:042
1675225915,27sleep016747-21apt-get10:37:480
1506125917,18sleep10-21swapper/112:17:371
91572584,11sleep10-21swapper/108:29:471
91572584,11sleep10-21swapper/108:29:471
584225819,23sleep10-21swapper/111:44:511
584225819,23sleep10-21swapper/111:44:511
44462585,15sleep20-21swapper/208:14:302
404125820,21sleep00-21swapper/008:13:110
2704125811,15sleep20-21swapper/207:43:052
2557825815,26sleep025572-21apt-get09:22:510
2170525817,25sleep121713-21diskstats07:27:581
123882584,18sleep10-21swapper/108:41:061
2901925719,18sleep10-21swapper/111:17:381
1855625717,23sleep00-21swapper/007:18:000
1410425716,24sleep114174-21mailstats10:28:041
1253225718,18sleep20-21swapper/210:23:052
1121925717,21sleep20-21swapper/208:37:482
8422564,41sleep20-21swapper/209:47:242
800225612,24sleep17625-21apt-get11:52:381
634225613,25sleep16336-21apt-get11:47:501
2949425617,21sleep10-21swapper/111:17:581
1848425612,23sleep00-21swapper/012:27:540
1848425612,23sleep00-21swapper/012:27:540
1648225613,19sleep20-21swapper/207:12:452
1642725615,25sleep116481-21memory07:12:441
1532125617,19sleep10-21swapper/107:08:071
137612568,39sleep00-21swapper/008:43:510
1059225610,19sleep10-21swapper/110:17:511
745325512,19sleep10-21swapper/110:07:541
2480625518,18sleep20-21swapper/211:02:572
2348825513,26sleep023578-21mailstats10:58:070
1951625513,16sleep20-21swapper/209:03:032
1884725517,17sleep10-21swapper/109:02:381
1307225513,23sleep112644-21apt-get07:02:391
1795625421,19sleep00-21swapper/008:58:040
507625315,18sleep20-21swapper/211:43:062
507625315,18sleep20-21swapper/211:43:062
2937825319,18sleep10-21swapper/107:52:491
2937825319,18sleep10-21swapper/107:52:491
2638025313,27sleep126390-21cat07:42:521
2116925316,16sleep10-21swapper/107:27:381
1410825313,22sleep214157-21ntp_states12:13:022
1051325311,20sleep00-21swapper/010:17:460
432025213,22sleep24346-21df09:57:542
3103825215,17sleep10-21swapper/109:38:071
195512525,34sleep00-21swapper/010:47:250
1560625214,22sleep115629-21ntp_states10:33:011
1560625214,22sleep115629-21ntp_states10:33:011
1112425217,20sleep20-21swapper/210:18:042
3077425114,22sleep030769-21apt-get11:22:500
2790625115,19sleep10-21swapper/111:12:581
2435725112,22sleep224395-21memory09:18:012
2324325114,22sleep123259-21uname10:58:001
185925113,22sleep11871-21apt-get08:07:471
1011925115,21sleep210125-21ntp_states08:33:032
2669925015,19sleep20-21swapper/211:08:102
178725012,22sleep21791-21diskstats11:33:252
1611325012,22sleep216171-21irqstats08:52:582
3105324912,21sleep131062-21df_inode07:57:541
2826124912,20sleep128313-21users09:28:381
295252479,14sleep20-21swapper/207:52:582
295252479,14sleep20-21swapper/207:52:582
1206924711,20sleep012064-21apt-get10:22:510
1730824614,19sleep10-21swapper/110:38:061
95812397,12sleep00-21swapper/011:57:380
100732378,15sleep10-21swapper/112:02:351
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional