You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-15 - 22:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Tue Jul 15, 2025 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32341298310,959sleep015709-1kworker/0:2H10:29:060
19923284318,807sleep119953-21/usr/sbin/munin09:49:111
2773627068,648sleep227783-21ntpq10:14:112
28953251519,481sleep129021-21/usr/sbin/munin08:34:181
3272249610,472sleep03308-21ntpq10:39:110
3546248217,449sleep35014-21kworker/u8:310:39:193
317922437401,22sleep331795-21sshd07:03:193
22433233943,281sleep322466-21irqstats08:14:213
112822920,8sleep00-21swapper/007:08:480
1159722607,237sleep211745-21modprobe09:23:482
1159722607,237sleep211745-21modprobe09:23:482
20501225311,225sleep220520-21pluginstate11:34:202
24790223714,207sleep024961-21latency_hist11:48:470
23812222965,23sleep20-21swapper/208:19:042
28980221916,180sleep229042-21fschecks_time10:19:072
6009221413,141sleep26027-21apt-get10:48:582
29188221016,160sleep20-21swapper/206:54:182
27131221028,72sleep20-21swapper/211:54:142
27131221028,72sleep20-21swapper/211:54:142
2915220015,113sleep23064-21unixbench_singl08:54:172
29875219617,161sleep00-21swapper/012:04:010
31865219510,29sleep232000-21latency_hist10:28:482
1361219412,164sleep21464-21timerandwakeup12:14:162
1361219412,164sleep21464-21timerandwakeup12:14:162
8888219210,166sleep28857-21latency_hist10:58:472
18597219026,21sleep20-21swapper/208:03:462
18597219026,21sleep20-21swapper/208:03:462
21947218519,142sleep00-21swapper/009:54:180
21947218519,142sleep00-21swapper/009:54:180
9233217711,151sleep29307-21cat09:14:152
30800217112,72sleep20-21swapper/208:39:192
13933217068,28sleep20-21swapper/209:29:182
666821678,20sleep20-21swapper/207:24:182
30149216528,114sleep030191-21modprobe10:23:450
18463216522,126sleep118469-21/usr/sbin/munin09:44:141
1957321649,139sleep219720-21head08:04:192
1957321649,139sleep219720-21head08:04:192
32039216320,80sleep20-21swapper/212:09:202
32039216320,80sleep20-21swapper/212:09:202
29650215710,87sleep229803-21sort12:03:472
1953215717,27sleep21924-21latency_hist08:53:492
1380721529,64sleep213813-21cpu11:14:012
32224214913,71sleep20-21swapper/210:28:562
31662214918,111sleep031715-21cpuspeed12:09:020
31662214918,111sleep031715-21cpuspeed12:09:020
24242214915,73sleep20-21swapper/210:04:012
14260214749,34sleep20-21swapper/207:49:132
14260214749,34sleep20-21swapper/207:49:132
38821439,82sleep2544-21latency_hist08:48:482
21828214215,68sleep222046-21apt-config08:13:482
1202221407,86sleep216876-1kworker/2:2H07:43:232
1202221407,86sleep216876-1kworker/2:2H07:43:232
1260213914,108sleep31394-21sed08:49:173
578121388,72sleep20-21swapper/210:48:462
27084213830,53sleep20-21swapper/210:13:462
18236213712,109sleep218369-21latency_hist11:28:472
18236213712,109sleep218369-21latency_hist11:28:472
11868213722,22sleep30-21swapper/309:24:063
11868213722,22sleep30-21swapper/309:24:063
28369213615,22sleep10-21swapper/111:59:061
455821358,12sleep00-21swapper/010:44:040
826221317,77sleep2370-21snmpd09:12:222
23897213150,66sleep223969-21mailstats11:44:172
2752421308,71sleep227651-21sshd08:29:182
2752421308,71sleep227651-21sshd08:29:182
17020212814,80sleep30-21swapper/307:58:473
10713212417,90sleep010793-21systemctl09:19:180
10713212417,90sleep010793-21systemctl09:19:180
13568212319,80sleep00-21swapper/011:13:470
912521218,55sleep20-21swapper/210:59:002
27021212139,45sleep00-21swapper/006:48:590
30302212060,45sleep20-21swapper/212:04:112
23412212036,12sleep20-21swapper/208:18:282
15934212034,70sleep20-21swapper/211:19:182
964621198,60sleep29691-21ntpq07:34:112
964621198,60sleep29691-21ntpq07:34:112
2006421169,56sleep220136-21mailstats09:49:162
25222211417,58sleep325173-21apt-get11:49:273
25222211417,58sleep325173-21apt-get11:49:273
25757211225,32sleep225832-21/usr/sbin/munin11:49:462
25757211225,32sleep225832-21/usr/sbin/munin11:49:462
22751211238,18sleep20-21swapper/209:59:092
12640211216,78sleep312671-21kernelversion07:44:093
28377211143,56sleep228401-21apt-get08:33:582
25304211138,56sleep20-21swapper/208:24:012
22930211013,83sleep123053-21unixbench_singl08:14:311
19079210713,76sleep019189-21ls11:29:130
4946210667,16sleep20-21swapper/207:19:172
31681210518,12sleep20-21swapper/207:01:532
3560210430,52sleep13572-21modprobe12:23:441
3560210430,52sleep13572-21modprobe12:23:441
25305210429,49sleep325299-21apt-get08:24:023
22344210444,49sleep20-21swapper/209:58:322
22344210444,49sleep20-21swapper/209:58:322
29574210313,76sleep329700-21/usr/sbin/munin10:19:203
26443210213,72sleep126485-21systemd-run11:53:451
26443210213,72sleep126485-21systemd-run11:53:451
2138221029,77sleep021383-21sshd08:10:180
15923210113,14sleep00-21swapper/007:54:070
12443210134,51sleep20-21swapper/207:44:042
19191210019,63sleep119343-21unixbench_singl11:29:141
246529910,74sleep02467-21sshd07:13:220
796829735,48sleep20-21swapper/207:29:222
796829735,48sleep20-21swapper/207:29:222
961029610,70sleep19673-21sendmail_mailst10:59:151
382629636,10sleep20-21swapper/207:15:582
2464129612,64sleep024684-21ntp_offset10:04:120
1029129612,70sleep110295-21diskstats09:19:071
1029129612,70sleep110295-21diskstats09:19:071
2860129224,52sleep028616-21ls10:18:450
1739829231,13sleep20-21swapper/209:40:072
92129131,11sleep20-21swapper/207:08:232
287229016,48sleep32887-21fschecks_count07:14:053
11452909,13sleep00-21swapper/012:14:100
11452909,13sleep00-21swapper/012:14:100
1535828914,18sleep00-21swapper/011:19:040
321432887,38sleep10-21swapper/108:44:191
200428828,11sleep20-21swapper/207:09:202
149532887,14sleep00-21swapper/007:49:310
149532887,14sleep00-21swapper/007:49:310
357728751,24sleep21-21systemd12:23:452
357728751,24sleep21-21systemd12:23:452
184742878,66sleep218543-21df11:28:592
107192877,56sleep20-21swapper/211:04:042
996728618,16sleep10-21swapper/109:18:471
1555528612,51sleep015714-21sed09:34:160
1018428610,20sleep20-21swapper/209:19:002
1018428610,20sleep20-21swapper/209:19:002
2776228528,31sleep20-21swapper/206:49:182
2230428517,51sleep20-21swapper/211:39:332
2230428517,51sleep20-21swapper/211:39:332
217872859,58sleep321940-21awk09:54:173
217872859,58sleep321940-21awk09:54:173
1571528512,59sleep115776-21users09:34:171
579228415,52sleep35923-21cat10:48:473
508728414,19sleep00-21swapper/009:03:480
355728431,26sleep13702-21modprobe08:58:481
3113028414,56sleep331146-21apt-get12:08:443
304802840,8sleep30-21swapper/308:39:153
159962849,16sleep20-21swapper/207:54:102
1926428314,55sleep019312-21unin-run09:48:450
1216328317,46sleep112207-21ntp_kernel_pll_09:24:121
1216328317,46sleep112207-21ntp_kernel_pll_09:24:121
1202028312,57sleep00-21swapper/011:08:460
298128211,16sleep20-21swapper/210:39:052
1369928219,33sleep00-21swapper/009:29:110
654328110,18sleep20-21swapper/209:08:182
654328110,18sleep20-21swapper/209:08:182
25602815,65sleep20-21swapper/210:38:082
193152817,15sleep20-21swapper/209:48:472
559227912,48sleep35623-21/usr/sbin/munin09:04:073
559227912,48sleep35623-21/usr/sbin/munin09:04:073
47742795,64sleep20-21swapper/209:00:472
3230627914,48sleep132421-21wc12:09:271
3230627914,48sleep132421-21wc12:09:271
2178227925,37sleep321297-21apt-get11:38:573
2178227925,37sleep321297-21apt-get11:38:573
2165027917,44sleep321661-21modprobe08:13:453
278627811,51sleep10-21swapper/107:13:591
2128227823,36sleep321300-21awk11:38:453
925027714,44sleep19317-21/usr/sbin/munin07:34:021
925027714,44sleep19317-21/usr/sbin/munin07:34:021
65532778,57sleep015709-1kworker/0:2H09:08:240
65532778,57sleep015709-1kworker/0:2H09:08:240
2728827730,22sleep10-21swapper/106:49:091
2702427725,24sleep30-21swapper/306:49:023
1776027713,49sleep317904-21latency_hist09:43:483
3044627612,52sleep130465-21apt-get10:23:581
2605727626,26sleep00-21swapper/010:09:080
1487127637,23sleep20-21swapper/209:33:582
125292767,16sleep20-21swapper/211:09:092
2058727514,42sleep120373-21/usr/sbin/munin08:09:101
1511027514,43sleep30-21swapper/311:18:473
91002748,45sleep30-21swapper/310:58:563
836927418,40sleep08402-21munin-run09:13:450
3120827425,29sleep00-21swapper/008:43:450
2683127412,44sleep126926-21fschecks_time11:54:091
2683127412,44sleep126926-21fschecks_time11:54:091
264927416,30sleep30-21swapper/310:38:463
187702740,8sleep00-21swapper/008:03:470
187702740,8sleep00-21swapper/008:03:470
1463427433,24sleep30-21swapper/309:33:473
108802749,48sleep110884-21/usr/sbin/munin07:39:061
108802749,48sleep110884-21/usr/sbin/munin07:39:061
2312027312,45sleep323159-21ps09:59:183
770427216,39sleep37705-21cpu07:29:033
770427216,39sleep37705-21cpu07:29:033
306927227,22sleep30-21swapper/308:54:183
142427119,27sleep30-21swapper/307:09:093
969327013,44sleep39703-21ntp_states07:34:123
969327013,44sleep39703-21ntp_states07:34:123
338927018,26sleep10-21swapper/110:39:171
2661127012,27sleep00-21swapper/008:28:470
772926915,37sleep37821-21/usr/sbin/munin09:09:403
738126918,22sleep10-21swapper/107:28:491
2907526912,22sleep00-21swapper/006:54:160
2134726919,31sleep121434-21grep11:38:471
3269726823,19sleep30-21swapper/307:04:163
2916326814,29sleep10-21swapper/110:19:101
2081126816,34sleep320829-21apt09:53:453
1618126823,22sleep30-21swapper/309:38:473
1739126713,35sleep117413-21/usr/sbin/munin11:24:131
1739126713,35sleep117413-21/usr/sbin/munin11:24:131
1385726720,29sleep113932-21/usr/sbin/munin09:29:171
324926612,40sleep03286-21ps07:14:140
2741226626,20sleep30-21swapper/310:14:033
2694126617,23sleep30-21swapper/308:29:073
2694126617,23sleep30-21swapper/308:29:073
2549326619,30sleep125515-21awk10:08:451
2505526619,30sleep00-21swapper/008:23:470
1650926613,39sleep116564-21fschecks_count09:39:051
1424426612,39sleep114253-21diskstats07:49:081
1424426612,39sleep114253-21diskstats07:49:081
75832657,15sleep10-21swapper/110:54:021
755626516,29sleep00-21swapper/010:53:570
43022659,12sleep00-21swapper/007:18:510
26452657,15sleep10-21swapper/108:54:121
1602826520,19sleep30-21swapper/311:19:193
862726410,12sleep30-21swapper/310:56:023
3189926413,24sleep30-21swapper/308:44:133
2421726413,26sleep10-21swapper/111:44:201
1696926422,23sleep116993-21munin-run07:58:451
133302649,10sleep30-21swapper/311:11:203
991926313,28sleep00-21swapper/009:18:460
767026313,35sleep07188-21apt-get07:28:560
767026313,35sleep07188-21apt-get07:28:560
696626311,22sleep10-21swapper/109:08:491
3249026318,28sleep332546-21interrupts10:29:213
295492637,18sleep00-21swapper/008:37:190
2470126314,31sleep324785-21gpgv11:48:463
196412638,13sleep00-21swapper/009:49:060
1214926317,25sleep011758-21/usr/sbin/munin09:24:110
1214926317,25sleep011758-21/usr/sbin/munin09:24:110
2879326218,20sleep00-21swapper/011:59:170
195952627,43sleep30-21swapper/311:31:593
153726215,26sleep10-21swapper/110:34:081
646426117,23sleep00-21swapper/007:24:160
2060126118,20sleep00-21swapper/011:34:220
1205426119,24sleep312056-21cron07:43:453
1205426119,24sleep312056-21cron07:43:453
244172607,13sleep30-21swapper/308:19:183
2186726016,26sleep00-21swapper/011:39:060
2186726016,26sleep00-21swapper/011:39:060
1283926015,22sleep10-21swapper/107:44:151
1206826016,20sleep00-21swapper/007:43:460
1206826016,20sleep00-21swapper/007:43:460
70542597,11sleep00-21swapper/010:50:420
3250725917,22sleep10-21swapper/107:04:101
3094325917,27sleep331003-21processes10:24:173
2992025918,26sleep329933-21munin-node06:58:473
181812597,38sleep00-21swapper/011:28:400
181812597,38sleep00-21swapper/011:28:400
146725917,22sleep30-21swapper/312:14:173
146725917,22sleep30-21swapper/312:14:173
992325814,26sleep010049-21/usr/sbin/munin07:34:170
992325814,26sleep010049-21/usr/sbin/munin07:34:170
931025815,24sleep39439-21sort09:14:163
3035925819,17sleep10-21swapper/112:04:141
251825812,28sleep32567-21irqstats12:19:063
251825812,28sleep32567-21irqstats12:19:063
2507025815,21sleep10-21swapper/108:23:481
193272588,13sleep30-21swapper/308:04:163
193272588,13sleep30-21swapper/308:04:163
1861225812,22sleep30-21swapper/308:03:473
1861225812,22sleep30-21swapper/308:03:473
1098725814,28sleep111033-21sed11:04:111
641025710,24sleep30-21swapper/310:49:123
60725715,25sleep0626-21apt-get08:48:580
3122725723,17sleep10-21swapper/108:43:461
235832575,34sleep10-21swapper/109:59:321
1152825715,24sleep311622-21users11:04:213
524725616,18sleep10-21swapper/110:44:171
411025613,28sleep34159-21sed08:59:103
316292567,38sleep00-21swapper/010:27:290
559425514,24sleep35621-21munin-run07:23:463
349925514,19sleep00-21swapper/008:58:470
2852725521,18sleep30-21swapper/308:34:073
2827925517,23sleep328276-21apt-get11:58:593
280602557,36sleep00-21swapper/008:33:110
280602557,36sleep00-21swapper/008:33:110
2789325519,20sleep10-21swapper/110:14:181
230025513,25sleep02294-21apt-get12:18:590
230025513,25sleep02294-21apt-get12:18:590
21722556,39sleep10-21swapper/107:09:351
3182925417,18sleep10-21swapper/107:03:461
2709425413,17sleep30-21swapper/310:13:473
2591125416,20sleep00-21swapper/011:49:480
2591125416,20sleep00-21swapper/011:49:480
25432546,36sleep00-21swapper/010:37:540
1544925411,25sleep30-21swapper/307:53:463
1544925411,25sleep30-21swapper/307:53:463
1305725420,16sleep00-21swapper/007:44:180
118325413,19sleep10-21swapper/112:14:111
118325413,19sleep10-21swapper/112:14:111
648925314,24sleep16495-21ntp_states10:49:141
3224025314,22sleep132252-21apt-get10:28:591
2727925312,26sleep127302-21ntp_states08:29:131
2727925312,26sleep127302-21ntp_states08:29:131
2612725314,23sleep126155-21memory10:09:091
2010425312,21sleep00-21swapper/008:08:450
2010425312,21sleep00-21swapper/008:08:450
1945925315,20sleep10-21swapper/108:04:181
1945925315,20sleep10-21swapper/108:04:181
3215725219,18sleep30-21swapper/312:09:253
3215725219,18sleep30-21swapper/312:09:253
2731025217,16sleep30-21swapper/311:54:193
2731025217,16sleep30-21swapper/311:54:193
230702526,16sleep10-21swapper/111:43:431
230702526,16sleep10-21swapper/111:43:431
1798925215,19sleep10-21swapper/107:59:171
1798925215,19sleep10-21swapper/107:59:171
1433925211,20sleep00-21swapper/011:14:150
1225725214,22sleep112252-21apt-get11:09:001
100425212,19sleep10-21swapper/108:49:111
767125112,20sleep10-21swapper/107:28:571
767125112,20sleep10-21swapper/107:28:571
3136725112,22sleep031466-21users06:59:210
2739125111,23sleep027503-21cat11:54:200
2739125111,23sleep027503-21cat11:54:200
2551525112,23sleep125572-21irqstats08:24:081
2265525113,23sleep022650-21apt-get09:59:030
1748225116,15sleep30-21swapper/311:24:163
1748225116,15sleep30-21swapper/311:24:163
1407525119,18sleep10-21swapper/111:14:081
2347725012,22sleep023486-21munin-run08:18:460
2241225011,23sleep122501-21ls09:58:481
2241225011,23sleep122501-21ls09:58:481
1669425011,18sleep00-21swapper/009:39:090
617124911,21sleep16224-21users09:04:171
617124911,21sleep16224-21users09:04:171
1807124911,22sleep318080-21df_inode09:44:043
568024814,18sleep10-21swapper/107:23:481
3194724812,18sleep00-21swapper/008:44:140
1020224813,19sleep310204-21apt-get09:19:013
1020224813,19sleep310204-21apt-get09:19:013
987024716,15sleep00-21swapper/010:59:180
2471524713,20sleep30-21swapper/310:04:193
720524611,18sleep30-21swapper/307:28:473
1542624612,17sleep10-21swapper/111:19:051
286102419,15sleep10-21swapper/106:54:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional