You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 16:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Dec 05, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6947235406,3522sleep25965-21kworker/2:012:25:212
28631291630,874sleep228633-21sshd11:52:212
1675268415,653sleep31736-21/usr/sbin/munin12:08:223
1285260429,527sleep21352-21fschecks_time12:08:122
6054255017,519sleep36056-21sshd08:52:223
849250126,463sleep2890-21ntpq10:23:202
260212233104,19sleep20-21swapper/207:48:202
260212233104,19sleep20-21swapper/207:48:202
14219223249,124sleep20-21swapper/211:07:542
32659222531,179sleep2304-21date10:22:552
4871221516,142sleep24890-21latency_hist10:37:542
26666221512,172sleep20-21swapper/210:03:082
983221116,71sleep01196-21dpkg08:27:560
26303221016,180sleep326427-21sed11:43:263
8347220713,177sleep08433-21fw_packets10:48:130
8692219919,125sleep28716-21modprobe12:32:522
10538219915,123sleep20-21swapper/209:12:522
23898219820,56sleep20-21swapper/211:37:552
24916219715,164sleep225061-21latency_hist09:57:552
25816219213,57sleep025874-21if_eth011:43:140
26913218848,118sleep226928-21apt-key11:47:532
8225218417,149sleep38220-21apt-get10:48:083
5811218025,133sleep15958-21grep10:38:251
31265217631,28sleep20-21swapper/211:58:272
31265217631,28sleep20-21swapper/211:58:272
20414217011,143sleep220499-21cut07:18:242
16455216919,130sleep316558-21timerandwakeup09:29:293
816215512,125sleep2873-21apt-config08:27:542
32309215511,132sleep05666-21snmpd10:19:020
30480215527,115sleep230532-21latency_hist08:12:552
8980215312,125sleep01-21systemd09:07:530
29721214957,73sleep229867-21unixbench_singl11:53:222
2955821487,119sleep20-21swapper/210:12:542
2955821487,119sleep20-21swapper/210:12:542
9885214729,77sleep20-21swapper/210:53:122
18592214752,74sleep218622-21awk09:37:522
2731214622,97sleep32868-21head10:28:263
31109214566,64sleep231197-21latency_hist10:17:552
7232214228,73sleep20-21swapper/210:43:232
345521408,112sleep20-21swapper/212:13:252
31096213712,108sleep031163-21swap11:58:250
31096213712,108sleep031163-21swap11:58:250
19775213635,63sleep20-21swapper/211:23:252
16518213417,102sleep316542-21proc_pri11:13:213
16518213417,102sleep316542-21proc_pri11:13:213
24652213312,109sleep224736-21cat11:38:242
24652213312,109sleep224736-21cat11:38:242
3135921329,18sleep20-21swapper/208:18:082
602821285,10sleep20-21swapper/208:52:002
2963521285,11sleep20-21swapper/208:06:252
928021275,28sleep20-21swapper/210:50:182
2191421269,100sleep20-21swapper/207:27:572
1886121255,11sleep20-21swapper/207:11:372
1505421225,43sleep20-21swapper/209:25:492
29480212021,82sleep029484-21cron10:12:520
29480212021,82sleep029484-21cron10:12:520
2484212013,93sleep02546-21sendmail_mailst10:28:230
21253212062,42sleep20-21swapper/211:28:222
1773121196,28sleep20-21swapper/207:05:402
2720321185,27sleep20-21swapper/207:53:252
1693921185,28sleep20-21swapper/209:31:552
16322211642,49sleep216349-21apt07:02:532
30071211516,83sleep330167-21/usr/sbin/munin10:13:153
19588211414,85sleep119589-21sshd07:17:221
24174211216,81sleep324242-21latency_hist07:37:593
2278211216,81sleep32324-21sed08:33:163
13714211152,20sleep213754-21sed09:22:542
2400121105,27sleep20-21swapper/207:37:222
1731421097,36sleep20-21swapper/211:17:522
1731421097,36sleep20-21swapper/211:17:522
11362210924,16sleep20-21swapper/210:58:092
2978121080,13sleep1225-21systemd-journal08:07:531
19341210812,81sleep119348-21sh09:38:201
241121077,64sleep22459-21ntpq08:33:192
1446021079,56sleep20-21swapper/211:08:072
1446021079,56sleep20-21swapper/211:08:072
11332210516,71sleep117975-21munin-node10:58:041
903021047,61sleep20-21swapper/209:07:552
527321048,61sleep25342-21basename10:38:142
31153210423,14sleep20-21swapper/208:17:562
21182210222,60sleep321218-21munin-run07:22:533
2968210019,59sleep32760-21/usr/sbin/munin12:13:143
27992210016,69sleep028064-21switchtime11:48:230
2356229929,44sleep023558-21apt-get09:53:090
2340329810,72sleep123477-21timerandwakeup07:33:291
243512975,26sleep20-21swapper/207:38:352
263192935,33sleep20-21swapper/210:02:112
1886429248,27sleep10-21swapper/111:22:521
173572929,12sleep10-21swapper/109:33:231
357429118,49sleep33569-21apt-get10:33:063
643429015,50sleep35859-21/usr/sbin/munin12:23:243
2186929013,56sleep00-21swapper/007:27:550
1959629047,23sleep20-21swapper/207:17:272
2366128916,57sleep323761-21fschecks_time09:53:163
183272890,6sleep30-21swapper/307:07:583
3246928714,57sleep032480-21netstat12:03:180
3091428714,56sleep130923-21/usr/sbin/munin11:58:181
3091428714,56sleep130923-21/usr/sbin/munin11:58:181
1403028716,53sleep30-21swapper/309:23:113
1995728664,12sleep20-21swapper/209:40:312
1366628610,63sleep113713-21systemctl09:22:531
958528510,59sleep19736-21latency_hist10:52:551
2514628518,56sleep20-21swapper/207:45:332
53428414,17sleep00-21swapper/010:23:140
399128420,47sleep20-21swapper/208:42:562
399128420,47sleep20-21swapper/208:42:562
3111728414,56sleep131152-21cut08:17:551
299092840,8sleep00-21swapper/010:13:110
1365428318,40sleep013803-21grep11:03:240
219762828,14sleep20-21swapper/209:48:062
1620228222,31sleep00-21swapper/009:29:220
668428111,59sleep00-21swapper/008:56:430
652428118,51sleep20-21swapper/208:54:392
17522817,62sleep21763-21munin-run08:32:542
705928020,37sleep10-21swapper/110:43:181
472828019,47sleep34764-21ntpq12:18:193
2607828023,38sleep026084-21ntp_states07:48:220
2607828023,38sleep026084-21ntp_states07:48:220
227028050,15sleep20-21swapper/210:28:172
2742427920,31sleep30-21swapper/310:03:263
2570827915,45sleep125800-21/usr/sbin/munin09:58:271
1848227919,25sleep10-21swapper/107:08:011
13512798,12sleep00-21swapper/008:28:320
323452787,13sleep20-21swapper/212:03:162
2196327822,38sleep021984-21apt-get11:32:520
72532778,56sleep10-21swapper/109:00:481
3101727711,46sleep10-21swapper/110:17:211
2791127716,37sleep327990-21/usr/sbin/munin11:48:223
2552627718,44sleep025576-21grep09:58:220
204372776,14sleep20-21swapper/209:43:092
121172779,30sleep20-21swapper/209:17:532
302927629,29sleep0369-21sshd12:13:150
2688327619,37sleep126910-21basename11:47:521
194802768,13sleep00-21swapper/009:38:230
1798727614,48sleep318029-21sed11:18:173
3133127516,44sleep131348-21apt08:18:031
2860927510,52sleep10-21swapper/111:52:041
2825427512,44sleep028382-21latency_hist08:02:560
2820027517,31sleep10-21swapper/108:02:531
279082758,11sleep20-21swapper/207:59:102
2004527517,45sleep020056-21grep07:18:140
1895527420,28sleep00-21swapper/011:22:550
182882746,12sleep20-21swapper/211:18:252
42982735,10sleep20-21swapper/212:17:582
2296327316,40sleep123003-21ntp_kernel_pll_11:34:031
221122737,13sleep20-21swapper/207:28:212
1200727319,28sleep00-21swapper/010:58:270
1193027320,25sleep30-21swapper/310:58:263
80527213,45sleep1815-21kworker/u8:108:27:521
31922729,12sleep00-21swapper/008:36:560
1525927222,29sleep10-21swapper/109:27:541
99227114,41sleep11064-21/usr/sbin/munin10:23:271
935327112,26sleep10-21swapper/109:08:111
74052716,10sleep20-21swapper/209:02:472
1016727118,33sleep39773-21/usr/sbin/munin10:53:173
2118827026,24sleep10-21swapper/111:28:211
779426914,37sleep07825-21/usr/sbin/munin12:28:140
70022699,50sleep00-21swapper/012:26:030
243172694,12sleep10-21swapper/107:38:091
1968726921,21sleep30-21swapper/307:17:563
42372686,10sleep20-21swapper/208:43:532
231422686,11sleep10-21swapper/109:51:361
2069426822,24sleep10-21swapper/107:18:271
1939226810,26sleep30-21swapper/309:38:223
460126714,33sleep00-21swapper/008:47:540
2953526716,31sleep029602-21/usr/sbin/munin11:53:200
246526718,31sleep12473-21modprobe12:12:521
1952826716,36sleep319559-21/usr/sbin/munin11:23:183
1635826722,24sleep316391-21munin-run07:02:543
36382667,16sleep30-21swapper/308:39:253
36382667,16sleep30-21swapper/308:39:253
3267026611,39sleep0325-21latency_hist08:22:550
2975126618,32sleep029776-21munin-run08:07:530
2599126612,37sleep126020-21/usr/sbin/munin07:48:191
2599126612,37sleep126020-21/usr/sbin/munin07:48:191
2251026611,27sleep30-21swapper/311:33:043
1630926624,27sleep116321-21kernelversion11:13:151
1630926624,27sleep116321-21kernelversion11:13:151
1375526621,19sleep30-21swapper/309:22:553
1002226614,39sleep010067-21irqstats10:53:140
808326513,35sleep18112-21/usr/sbin/munin09:03:141
336626513,29sleep10-21swapper/112:13:241
277342657,49sleep00-21swapper/010:05:250
266422658,11sleep30-21swapper/307:49:503
266422658,11sleep30-21swapper/307:49:503
2487526516,26sleep10-21swapper/109:57:541
42026419,18sleep30-21swapper/310:23:043
2839626418,24sleep10-21swapper/110:08:121
2839626418,24sleep10-21swapper/110:08:121
2219626414,21sleep30-21swapper/309:48:153
1915826420,28sleep00-21swapper/007:12:570
1217026314,20sleep10-21swapper/109:17:561
609826215,30sleep16127-21modprobe08:52:521
2473726216,21sleep30-21swapper/311:38:253
2473726216,21sleep30-21swapper/311:38:253
2052126220,27sleep10-21swapper/109:43:121
1380726212,24sleep10-21swapper/111:03:251
704126117,26sleep00-21swapper/010:43:170
231622619,40sleep00-21swapper/009:51:520
1798726122,21sleep10-21swapper/107:07:571
1582426113,28sleep115973-21latency_hist11:12:541
1582426113,28sleep115973-21latency_hist11:12:541
1446326119,24sleep00-21swapper/011:08:090
1446326119,24sleep00-21swapper/011:08:090
1146726115,19sleep30-21swapper/309:13:223
54222608,11sleep10-21swapper/112:20:391
51242608,40sleep3360-21ntpd10:38:073
2910526018,19sleep30-21swapper/308:03:233
1956126016,20sleep00-21swapper/011:23:190
187842604,12sleep30-21swapper/307:10:373
323592595,10sleep30-21swapper/308:19:353
1761525916,22sleep30-21swapper/309:33:293
1710125913,26sleep017107-21ntp_states07:03:190
130925916,22sleep10-21swapper/108:28:051
808525817,18sleep30-21swapper/312:28:213
2960325816,23sleep129710-21sshd11:53:211
2761125814,20sleep00-21swapper/007:57:550
2174225814,18sleep30-21swapper/309:47:553
1736625813,21sleep00-21swapper/011:17:540
1736625813,21sleep00-21swapper/011:17:540
1571325816,28sleep115247-21apt-get09:28:021
141925813,18sleep00-21swapper/012:08:140
792525713,27sleep18005-21/usr/sbin/munin12:28:191
2760025716,25sleep327609-21munin-run07:57:543
2333125715,26sleep023402-21sensors07:33:280
1581625713,23sleep115456-21apt-get06:57:561
1511425714,25sleep30-21swapper/311:08:243
1511425714,25sleep30-21swapper/311:08:243
87525618,21sleep30-21swapper/308:27:553
3237425615,19sleep30-21swapper/312:03:173
2530125612,28sleep025303-21sshd07:47:330
1121025615,24sleep011224-21needreboot09:13:150
730625518,21sleep30-21swapper/310:43:243
2537225514,25sleep125382-21munin-run07:47:551
2295025521,18sleep00-21swapper/011:34:020
176425515,19sleep30-21swapper/308:32:553
719425412,19sleep10-21swapper/112:27:541
512625413,24sleep05121-21apt-get10:38:080
302472549,33sleep35666-21snmpd11:57:433
2693725415,18sleep10-21swapper/110:03:161
2537625422,19sleep00-21swapper/011:42:540
2537625422,19sleep00-21swapper/011:42:540
207525414,24sleep12082-21diskstats10:28:111
1916125413,25sleep119152-21apt-get11:23:081
1735425411,25sleep017328-21apt-get09:33:220
1040725411,24sleep10-21swapper/110:53:241
3054825312,22sleep330543-21apt-get11:58:083
3054825312,22sleep330543-21apt-get11:58:083
2829625314,24sleep328299-21/usr/sbin/munin10:08:103
2829625314,24sleep328299-21/usr/sbin/munin10:08:103
1318025312,17sleep30-21swapper/311:03:153
741625214,25sleep01-21systemd09:02:530
489425217,18sleep00-21swapper/012:18:240
396025215,23sleep13990-21sed08:42:551
396025215,23sleep13990-21sed08:42:551
253072526,34sleep30-21swapper/307:47:363
223252526,35sleep00-21swapper/007:31:060
1010425211,20sleep30-21swapper/309:08:253
3181925116,21sleep10-21swapper/110:18:221
301412516,15sleep30-21swapper/308:09:413
1440025111,23sleep014411-21ntp_states09:23:200
779325013,21sleep37799-21diskstats09:03:083
65852507,34sleep10-21swapper/108:55:271
2147325013,23sleep321474-21sshd07:23:213
1248624912,22sleep012492-21diskstats09:18:110
43692489,14sleep00-21swapper/010:33:230
3248124811,18sleep10-21swapper/112:03:191
2559224810,22sleep325627-21ps09:58:253
2132124813,15sleep30-21swapper/311:28:233
1061724813,17sleep00-21swapper/009:12:540
688524711,21sleep30-21swapper/308:58:073
3180524712,20sleep331811-21ntp_states10:18:203
2088224711,14sleep00-21swapper/009:43:200
200024711,21sleep12001-21apt-get08:33:051
1593824711,19sleep00-21swapper/006:58:030
3003724513,15sleep00-21swapper/008:08:210
279572455,31sleep30-21swapper/307:59:483
364424410,13sleep10-21swapper/110:33:081
2407224310,16sleep10-21swapper/111:38:061
2407224310,16sleep10-21swapper/111:38:061
483924210,16sleep00-21swapper/008:48:070
2246324212,18sleep30-21swapper/307:32:503
3047124010,16sleep10-21swapper/108:12:541
3111023812,14sleep00-21swapper/008:17:540
3106723812,13sleep30-21swapper/308:17:533
244872377,12sleep30-21swapper/307:40:213
39492369,13sleep00-21swapper/008:42:540
39492369,13sleep00-21swapper/008:42:540
214542366,13sleep10-21swapper/107:23:151
196352368,14sleep00-21swapper/007:17:530
41372359,13sleep30-21swapper/308:42:573
246592347,15sleep00-21swapper/007:42:330
227082348,12sleep10-21swapper/107:33:011
43492337,13sleep10-21swapper/108:45:211
273142317,15sleep10-21swapper/107:54:451
70062275,11sleep30-21swapper/308:58:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional