You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 06:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Nov 28, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29102282195,8202sleep228517-21kworker/2:223:23:532
29102282195,8202sleep228517-21kworker/2:223:23:532
11440297122,934sleep011444-21diskstats00:13:380
11440297122,934sleep011444-21diskstats00:13:380
12173282519,789sleep112242-21munin-node19:08:221
20819247816,445sleep120832-21/usr/sbin/munin22:58:531
20819247816,445sleep120832-21/usr/sbin/munin22:58:531
15113235612,329sleep015149-21ntpq00:23:590
15113235612,329sleep015149-21ntpq00:23:590
22103235313,322sleep022179-21/usr/sbin/munin19:38:380
21039230613,175sleep221053-21fschecks_count21:18:412
25389229914,261sleep025530-21latency_hist21:33:210
22734229914,224sleep222792-21dpkg23:08:192
22897221717,180sleep023107-21latency_hist23:08:210
21180220311,128sleep221188-21kworker/u8:123:03:182
21180220311,128sleep221188-21kworker/u8:123:03:182
651222019,133sleep26579-21date23:58:212
651222019,133sleep26579-21date23:58:212
11109218710,68sleep21-21systemd00:13:192
11109218710,68sleep21-21systemd00:13:192
26942218322,139sleep00-21swapper/021:38:220
22750218012,17sleep30-21swapper/321:23:433
12700217611,88sleep20-21swapper/219:08:402
1060216012,80sleep20-21swapper/223:38:512
7030215974,18sleep20-21swapper/223:58:462
7030215974,18sleep20-21swapper/223:58:462
25072215914,131sleep025122-21cpuspeed19:48:350
25072215914,131sleep025122-21cpuspeed19:48:350
3212421578,31sleep20-21swapper/221:53:542
11349215452,87sleep211498-21latency_hist20:48:212
31942215344,95sleep232076-21timerwakeupswit23:33:502
32127215214,125sleep332150-21grep21:53:553
7851215167,63sleep20-21swapper/222:18:452
17378214950,84sleep217412-21cut19:23:382
30201214646,74sleep20-21swapper/223:28:462
20714214538,21sleep20-21swapper/221:18:212
3371214354,67sleep20-21swapper/222:03:512
156352143101,23sleep20-21swapper/222:43:442
27951213944,70sleep20-21swapper/223:23:212
24533213927,92sleep124556-21sh23:13:181
29705213852,74sleep229742-21idleruntime-cro20:03:192
29705213852,74sleep229742-21idleruntime-cro20:03:192
22238213712,42sleep222274-21latency19:38:402
28489213652,18sleep20-21swapper/221:43:202
19181213515,99sleep019328-21cat21:13:220
8543213211,101sleep28633-21fschecks_time20:38:402
6907213161,21sleep26918-21cpu20:33:352
14665213039,71sleep20-21swapper/200:23:292
14665213039,71sleep20-21swapper/200:23:292
22681212932,66sleep30-21swapper/319:38:493
3135321289,103sleep031504-21latency_hist20:08:210
2010421289,97sleep220132-21munin-run19:33:192
25466212716,89sleep125504-21ntp_kernel_pll_19:48:441
25466212716,89sleep125504-21ntp_kernel_pll_19:48:441
1432421278,104sleep214438-21dpkg00:23:212
8415212511,15sleep20-21swapper/200:03:552
24782212511,100sleep024813-21awk19:48:190
24782212511,100sleep024813-21awk19:48:190
13777212418,86sleep20-21swapper/222:38:392
30079212241,16sleep20-21swapper/221:48:212
2239212211,93sleep21944-21/usr/sbin/munin23:43:442
30646212015,87sleep030673-21awk21:48:430
25053212014,89sleep324553-21apt-get23:13:243
1704821200,9sleep10-21swapper/119:23:191
17328211928,25sleep20-21swapper/221:05:532
5411211823,19sleep20-21swapper/220:28:382
5411211823,19sleep20-21swapper/220:28:382
23608211717,77sleep20-21swapper/219:43:402
23608211717,77sleep20-21swapper/219:43:402
19307211624,72sleep019342-21proc_pri19:28:480
22893211518,20sleep20-21swapper/221:23:462
10824211410,91sleep210837-21kernelversion22:28:422
26735211224,62sleep126825-21fschecks_time23:18:391
18314211232,17sleep20-21swapper/221:08:482
26318211121,25sleep00-21swapper/023:17:540
14850211115,70sleep114680-21/usr/sbin/munin20:58:421
19685211021,28sleep219806-21latency_hist22:58:212
9342210715,75sleep29047-21/usr/sbin/munin18:58:432
2733210615,74sleep12751-21apt22:03:301
11333210663,21sleep20-21swapper/219:03:512
31306210412,75sleep320111-21kworker/u8:020:08:203
1024221045,12sleep20-21swapper/222:28:062
14472210318,69sleep314566-21memory19:13:453
25061210213,74sleep125083-21apt23:13:301
15961210212,74sleep315962-21cron00:28:183
15961210212,74sleep315962-21cron00:28:183
3027721018,75sleep20-21swapper/220:03:432
21579210116,62sleep321671-21fschecks_time23:03:393
1550721018,79sleep215472-21latency_hist19:18:212
1377229916,25sleep10-21swapper/122:38:381
435229710,75sleep24440-21switchtime20:23:502
435229710,75sleep24440-21switchtime20:23:502
13942977,12sleep20-21swapper/221:58:432
1372529710,25sleep20-21swapper/220:53:542
3076329616,66sleep130841-21basename21:48:481
269222966,25sleep20-21swapper/219:53:422
269222966,25sleep20-21swapper/219:53:422
1260729613,63sleep00-21swapper/019:08:340
953129531,39sleep39569-21grep18:58:483
1845529518,56sleep318510-21fschecks_time22:53:383
167772955,17sleep20-21swapper/219:20:272
1470329527,43sleep314699-21apt-get20:58:353
1466229549,25sleep00-21swapper/019:13:500
316412947,30sleep231747-21tune2fs20:08:402
2349929426,25sleep30-21swapper/319:43:333
2349929426,25sleep30-21swapper/319:43:333
2168229412,25sleep21-21systemd19:38:192
3264829318,52sleep00-21swapper/023:38:210
156222936,75sleep20-21swapper/220:58:592
267192918,13sleep20-21swapper/221:36:142
872629016,60sleep18796-21latency20:38:431
44242908,62sleep20-21swapper/223:48:532
171922907,70sleep217238-21ntpq22:48:452
1335228838,40sleep20-21swapper/222:37:422
59672878,57sleep20-21swapper/222:13:362
523328766,10sleep20-21swapper/222:09:002
315628716,18sleep00-21swapper/022:03:450
1682128617,17sleep00-21swapper/000:28:480
2324028510,55sleep20-21swapper/223:08:312
1425228517,46sleep114177-21/usr/sbin/munin19:13:391
1102428516,18sleep30-21swapper/319:03:443
170132837,16sleep20-21swapper/200:28:502
1343528314,55sleep313512-21mailstats00:19:023
477128220,42sleep34815-40ogrotate23:53:183
678128113,53sleep16820-21fschecks_time23:58:411
678128113,53sleep16820-21fschecks_time23:58:411
2897228016,45sleep329050-21mailstats19:59:043
2897228016,45sleep329050-21mailstats19:59:043
246512800,10sleep324607-21apt-key23:13:193
2141528019,29sleep30-21swapper/321:18:503
179228022,12sleep20-21swapper/220:16:272
1250028022,11sleep20-21swapper/220:49:102
99027916,26sleep10-21swapper/123:38:501
94842794,65sleep00-21swapper/020:40:230
439627917,49sleep04414-21fschecks_count22:08:380
1672427918,22sleep30-21swapper/321:03:483
947027825,33sleep10-21swapper/100:08:181
3147327824,42sleep20-21swapper/221:52:292
251152786,59sleep20-21swapper/221:30:532
708827716,38sleep37134-21basename20:33:403
508927716,45sleep15165-21sed20:28:211
508927716,45sleep15165-21sed20:28:211
645427614,41sleep16511-21sort23:58:201
645427614,41sleep16511-21sort23:58:201
2629927622,42sleep20-21swapper/219:52:532
2629927622,42sleep20-21swapper/219:52:532
2053027622,34sleep10-21swapper/121:16:381
332927523,42sleep20-21swapper/220:21:152
1739327516,33sleep30-21swapper/322:48:503
1477527513,47sleep114782-21df_inode00:23:401
1477527513,47sleep114782-21df_inode00:23:401
126692757,13sleep20-21swapper/200:18:202
126692757,13sleep20-21swapper/200:18:202
571127413,20sleep30-21swapper/322:13:213
311982745,18sleep30-21swapper/321:48:593
1725627412,47sleep117320-21ps22:48:481
1015327311,47sleep09733-21/usr/sbin/munin00:09:010
1015327311,47sleep09733-21/usr/sbin/munin00:09:010
312552728,14sleep10-21swapper/120:08:181
2843727216,42sleep028468-21modprobe21:43:180
1601427225,25sleep30-21swapper/319:18:423
1290427222,24sleep10-21swapper/120:53:211
1153827212,45sleep311542-21find20:48:303
973227117,28sleep30-21swapper/300:08:323
900027116,24sleep00-21swapper/022:23:350
2876227117,38sleep128831-21switchtime23:23:481
2876227117,38sleep128831-21switchtime23:23:481
263052716,11sleep20-21swapper/223:17:432
544027022,32sleep35507-21fschecks_time20:28:393
544027022,32sleep35507-21fschecks_time20:28:393
387627013,42sleep03981-21fschecks_time20:23:400
387627013,42sleep03981-21fschecks_time20:23:400
2945127011,45sleep329483-21munin-run23:28:183
2945127011,45sleep329483-21munin-run23:28:183
2022927015,39sleep320305-21/usr/sbin/munin21:13:543
750926920,31sleep17593-21wc20:33:501
2259526921,22sleep10-21swapper/121:23:411
2103226916,38sleep121038-21diskstats21:18:401
184126916,20sleep00-21swapper/021:58:530
44226811,44sleep3540-21cat20:13:213
1589326818,20sleep00-21swapper/022:43:500
128526814,37sleep31294-21diskstats21:58:403
635826714,15sleep10-21swapper/122:13:471
1062226719,27sleep10-21swapper/120:43:521
34322666,11sleep00-21swapper/020:22:350
2979226622,22sleep10-21swapper/120:03:221
2979226622,22sleep10-21swapper/120:03:221
2135026617,32sleep021387-21sed21:18:470
136922669,29sleep30-21swapper/322:38:363
753526518,28sleep07579-21fschecks_count22:18:390
47402658,13sleep30-21swapper/322:08:453
2982426516,36sleep129831-21forks23:28:371
2796126514,19sleep10-21swapper/121:38:511
2420726515,28sleep00-21swapper/023:09:300
2088026512,39sleep120888-21open_files19:33:461
1737126519,29sleep117377-21diskstats19:23:371
1222226518,26sleep10-21swapper/122:33:381
657426415,22sleep00-21swapper/022:13:500
2602526416,31sleep30-21swapper/321:33:443
1857326419,26sleep10-21swapper/122:53:401
168642646,45sleep00-21swapper/019:21:300
152422647,12sleep10-21swapper/122:43:351
129982649,12sleep30-21swapper/319:08:453
1253526416,24sleep30-21swapper/322:33:443
1239826422,21sleep00-21swapper/022:33:410
1136026416,32sleep10-21swapper/100:13:351
1136026416,32sleep10-21swapper/100:13:351
1040126419,25sleep00-21swapper/019:03:210
467826318,24sleep10-21swapper/122:08:431
312626310,36sleep33154-21/usr/sbin/munin22:03:443
2725526325,18sleep00-21swapper/023:18:500
1804826323,22sleep00-21swapper/022:53:190
121326318,28sleep11248-21grep21:58:381
863426213,14sleep30-21swapper/320:38:413
327302628,44sleep10-21swapper/120:12:141
2858826213,12sleep00-21swapper/023:23:430
2858826213,12sleep00-21swapper/023:23:430
2413526224,19sleep30-21swapper/321:28:403
196626212,25sleep00-21swapper/023:43:340
6572615,12sleep00-21swapper/021:55:010
278512617,14sleep30-21swapper/323:22:413
2689026122,24sleep126918-21munin-run21:38:191
263112615,11sleep30-21swapper/319:53:023
263112615,11sleep30-21swapper/319:53:023
2066526115,23sleep00-21swapper/022:58:520
2066526115,23sleep00-21swapper/022:58:520
194326114,26sleep31962-21apt-get23:43:323
194126117,28sleep11-21systemd20:18:201
1523926115,24sleep315235-21apt-get22:43:333
1070926119,20sleep00-21swapper/019:03:370
88126013,32sleep1882-21sshd21:57:531
574626015,23sleep30-21swapper/323:53:543
574626015,23sleep30-21swapper/323:53:543
528826017,19sleep00-21swapper/023:53:430
528826017,19sleep00-21swapper/023:53:430
3107626025,19sleep00-21swapper/023:33:200
2409826023,19sleep10-21swapper/119:43:511
2409826023,19sleep10-21swapper/119:43:511
2055126016,27sleep0225-21systemd-journal21:16:540
1673326015,27sleep316818-21cut00:28:473
1314926017,27sleep313144-21apt-get20:53:393
1053326011,19sleep30-21swapper/322:28:313
38792596,12sleep00-21swapper/023:48:420
377925918,25sleep33774-21apt-get20:23:353
377925918,25sleep33774-21apt-get20:23:353
2796725916,26sleep01-21systemd19:58:200
2796725916,26sleep01-21systemd19:58:200
2723025922,19sleep10-21swapper/119:53:491
2723025922,19sleep10-21swapper/119:53:491
239825917,25sleep12421-21ntp_states23:43:471
194522598,15sleep00-21swapper/022:56:010
1423225916,23sleep10-21swapper/100:23:201
306822584,10sleep30-21swapper/323:29:153
211662588,37sleep30-21swapper/323:03:073
211662588,37sleep30-21swapper/323:03:073
125492586,42sleep30-21swapper/300:17:243
125492586,42sleep30-21swapper/300:17:243
1236725812,28sleep012465-21users20:48:510
81732577,38sleep00-21swapper/000:03:280
81732577,38sleep00-21swapper/000:03:280
771725713,28sleep37758-21ls22:18:423
670225712,25sleep00-21swapper/020:33:210
1727025716,24sleep317288-21apt-get19:23:303
115742576,38sleep10-21swapper/122:30:151
1091525718,24sleep110634-21/usr/sbin/munin19:03:421
101242576,38sleep10-21swapper/119:00:431
3262025617,17sleep30-21swapper/323:38:203
1793525617,21sleep10-21swapper/121:08:391
1633325615,25sleep116338-21/usr/sbin/munin00:28:371
1112225613,17sleep30-21swapper/300:13:203
1112225613,17sleep30-21swapper/300:13:203
1087825614,23sleep310916-21users20:43:543
90425519,16sleep00-21swapper/023:38:490
78325515,25sleep3810-21netstat23:38:443
758025514,23sleep17658-21fw_packets22:18:401
68025515,25sleep0675-21apt-get20:13:340
485925516,16sleep10-21swapper/123:53:201
2042025514,23sleep020414-21apt-get19:33:330
1604225518,21sleep016187-21wc21:03:210
104925511,26sleep31078-21/usr/sbin/munin20:13:453
962825417,18sleep30-21swapper/322:23:513
532625415,23sleep15321-21apt-get20:28:331
532625415,23sleep15321-21apt-get20:28:331
1884425414,25sleep118840-21apt-get19:28:331
1634725416,23sleep116357-21/usr/sbin/munin21:03:381
107052548,12sleep00-21swapper/022:28:400
2894925313,24sleep028998-21irqstats21:43:430
1789325316,21sleep317897-21diskstats21:08:373
1626225313,24sleep016257-21apt-get21:03:330
1410725313,24sleep014143-21ntp_states22:38:450
1031025316,22sleep110418-21latency_hist22:28:211
850425218,21sleep00-21swapper/000:03:570
2918325215,22sleep129189-21ntp_states21:43:481
2725725215,21sleep327262-21diskstats21:38:373
247725212,15sleep30-21swapper/320:18:423
237582526,35sleep00-21swapper/021:28:160
1361425215,16sleep00-21swapper/020:53:520
1354725215,18sleep10-21swapper/120:53:491
317432519,19sleep10-21swapper/123:33:451
2887725116,19sleep30-21swapper/321:43:423
220625111,25sleep12225-21apt-get20:18:321
2155025110,15sleep10-21swapper/123:03:371
1893025115,16sleep30-21swapper/319:28:393
1466325112,24sleep014186-21apt-get00:23:270
1127525121,17sleep00-21swapper/020:48:190
3162525011,16sleep00-21swapper/020:08:380
2745525011,15sleep30-21swapper/319:53:513
2745525011,15sleep30-21swapper/319:53:513
204942509,16sleep30-21swapper/319:33:363
1736325016,18sleep00-21swapper/019:23:360
1020025012,23sleep110236-21ntp_states00:09:021
1020025012,23sleep110236-21ntp_states00:09:021
532524817,16sleep00-21swapper/020:28:320
2218024812,21sleep122237-21irqstats19:38:391
2872424714,20sleep028737-21irqstats19:58:570
2872424714,20sleep028737-21irqstats19:58:570
791824310,15sleep30-21swapper/300:03:193
791824310,15sleep30-21swapper/300:03:193
69052439,18sleep06900-21apt-get20:33:330
241002377,12sleep10-21swapper/121:28:381
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional