You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-19 - 07:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Thu Mar 19, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29007251315,397sleep229048-21grep00:03:202
29007251315,397sleep229048-21grep00:03:202
3108238158,147sleep23140-21idleruntime-cro22:42:532
17955235131,205sleep217929-21/usr/sbin/munin20:03:092
17955235131,205sleep217929-21/usr/sbin/munin20:03:092
9817230741,204sleep210028-21latency_hist19:37:572
1955822987,274sleep218685-1kworker/2:1H23:33:192
2375229014,259sleep12524-21latency_hist00:22:571
11439228811,139sleep211455-21idleruntime-cro19:42:542
2718322589,104sleep227243-21if_eth023:58:152
13085222738,63sleep20-21swapper/223:13:162
2407221622,176sleep32496-21mailstats19:13:233
2407221622,176sleep32496-21mailstats19:13:233
4677221513,146sleep24756-21switchtime00:28:232
24264221216,178sleep224278-21fschecks_count20:23:122
30395220933,115sleep20-21swapper/220:42:552
30395220933,115sleep20-21swapper/220:42:552
21256220810,152sleep221310-21if_eth020:13:152
14584220729,125sleep214629-21latency_hist19:52:552
31352220314,125sleep231362-21modprobe00:12:532
31352220314,125sleep231362-21modprobe00:12:532
25886219752,133sleep225947-21grep23:53:222
25886219752,133sleep225947-21grep23:53:222
8189219515,124sleep28305-21timerandwakeup21:13:232
8189219515,124sleep28305-21timerandwakeup21:13:232
9587217546,108sleep20-21swapper/221:18:402
9587217546,108sleep20-21swapper/221:18:402
193272174108,23sleep20-21swapper/221:48:232
24868217319,135sleep325012-21tail20:23:253
15097217111,123sleep215132-21latency19:53:162
12963217110,144sleep013003-21ps21:28:230
12963217110,144sleep013003-21ps21:28:230
28546216835,15sleep20-21swapper/222:18:232
10734216611,137sleep110800-21mailstats19:39:131
21381216535,109sleep20-21swapper/223:38:242
2946215724,47sleep20-21swapper/220:58:132
14770215719,114sleep20-21swapper/223:18:212
21312215514,124sleep121380-21awk23:38:231
1557214410,37sleep20-21swapper/220:53:162
6477214247,71sleep36693-21cut19:27:563
27962214116,76sleep228120-21unixbench_multi20:33:262
5351213711,22sleep20-21swapper/222:48:182
1871821375,120sleep00-21swapper/023:30:300
15373213714,109sleep315325-21latency_hist21:37:553
24421213311,73sleep224500-21switchtime23:48:212
16016213329,54sleep216051-21ntpq21:38:182
31252213132,17sleep20-21swapper/220:43:232
31255212936,75sleep231274-21fschecks_count22:28:112
32729212813,17sleep20-21swapper/222:33:082
32729212813,17sleep20-21swapper/222:33:082
7031212739,18sleep20-21swapper/200:37:562
30112212519,86sleep20-21swapper/222:23:232
9622212423,16sleep20-21swapper/223:03:072
30936212412,92sleep331018-21latency_hist22:27:563
29081212220,76sleep128569-21apt-get20:38:051
29081212220,76sleep128569-21apt-get20:38:051
27229212118,83sleep327382-21sed22:13:233
25112212110,16sleep20-21swapper/222:08:132
22314212032,60sleep122307-21apt-get23:43:071
9977211815,86sleep110046-21sensors_temp21:18:511
9977211815,86sleep110046-21sensors_temp21:18:511
21624211842,26sleep20-21swapper/221:57:562
1144521188,60sleep20-21swapper/223:08:162
6359211717,41sleep20-21swapper/219:27:552
30188211414,81sleep130286-21fschecks_time00:08:111
30188211414,81sleep130286-21fschecks_time00:08:111
768121098,90sleep213441-21snmpd22:56:582
11221210915,74sleep111310-21/usr/sbin/munin21:23:151
23250210747,14sleep20-21swapper/220:18:242
11311210710,44sleep211324-21needreboot21:23:162
31631210617,69sleep331673-21ps22:28:213
1628921049,78sleep016405-21sed21:38:240
4306210313,77sleep14361-21if_err_eth000:28:141
1845021037,84sleep218457-21munin-run21:47:552
4901210246,42sleep20-21swapper/221:03:222
23394210147,14sleep20-21swapper/222:03:092
12099210124,28sleep20-21swapper/219:43:182
2910629924,57sleep329075-21apt-get20:38:193
2910629924,57sleep329075-21apt-get20:38:193
1654629916,62sleep116065-21/usr/sbin/munin23:23:211
68312988,78sleep26806-21apt-get19:28:162
970729715,68sleep39719-21fschecks_count23:03:123
873629717,22sleep00-21swapper/019:33:190
3240829710,36sleep20-21swapper/219:08:092
2621429766,19sleep20-21swapper/220:28:212
2291629711,69sleep022991-21find23:43:230
145029635,15sleep20-21swapper/200:18:262
14092956,33sleep20-21swapper/219:12:572
14092956,33sleep20-21swapper/219:12:572
2336729422,45sleep323513-21grep20:18:253
84852937,37sleep20-21swapper/222:58:192
1122329320,51sleep011148-21/usr/sbin/munin23:08:100
2754929114,62sleep327629-21cat23:58:243
1431529116,56sleep014019-21/usr/sbin/munin21:33:150
111929114,35sleep00-21swapper/000:18:200
2980529024,47sleep329827-21awk00:07:533
2980529024,47sleep329827-21awk00:07:533
1470529019,49sleep314282-21/usr/sbin/munin23:18:183
103392907,63sleep00-21swapper/019:39:040
185128917,30sleep10-21swapper/122:38:071
3208028829,36sleep332082-21cron19:07:533
30328810,65sleep2407-21switchtime20:48:242
275332885,36sleep20-21swapper/222:14:592
1838528723,43sleep318516-21unixbench_singl23:28:273
1544228717,52sleep315553-21sed19:53:253
326528621,46sleep13418-21latency_hist19:17:571
326528621,46sleep13418-21latency_hist19:17:571
3233228623,45sleep032472-21sed00:13:230
2230428626,45sleep20-21swapper/223:43:042
1960428613,56sleep30-21swapper/323:33:203
173232868,28sleep20-21swapper/219:58:332
173232868,28sleep20-21swapper/219:58:332
1200028617,52sleep012076-21latency19:43:160
293228520,32sleep00-21swapper/020:58:110
2023928519,43sleep120385-21head20:08:251
2023928519,43sleep120385-21head20:08:251
1563728516,57sleep015767-21apt-key23:22:550
512428413,57sleep05190-21fschecks_time19:23:170
3082328419,34sleep20-21swapper/219:03:082
2619028425,37sleep126212-21sshd20:28:201
705428318,52sleep37135-21sed22:53:233
44122837,29sleep20-21swapper/219:18:392
316028311,52sleep03197-21sed22:42:550
1687928321,42sleep116903-21modprobe21:42:531
70272825,28sleep20-21swapper/221:09:322
1848028221,33sleep00-21swapper/021:47:570
45728113,19sleep20-21swapper/200:17:532
330428114,16sleep10-21swapper/120:58:201
1347828121,10sleep20-21swapper/221:29:182
1347828121,10sleep20-21swapper/221:29:182
408028015,49sleep20-21swapper/222:43:252
1919528016,42sleep00-21swapper/021:48:210
1880028021,42sleep318893-21grep21:48:123
2666627923,38sleep126733-40mandb23:57:531
2666627923,38sleep126733-40mandb23:57:531
2072127929,37sleep320757-21ps21:53:213
1143127916,46sleep111444-21/usr/sbin/munin23:08:151
310627810,55sleep13176-21/usr/sbin/munin00:23:211
1945927818,29sleep00-21swapper/023:33:170
1685027815,48sleep316862-21ntpq19:58:213
1685027815,48sleep316862-21ntpq19:58:213
543127727,26sleep30-21swapper/300:32:563
3064727720,26sleep30-21swapper/300:08:213
3064727720,26sleep30-21swapper/300:08:213
103862776,60sleep00-21swapper/021:19:190
103862776,60sleep00-21swapper/021:19:190
10222777,59sleep00-21swapper/019:10:250
2675427517,37sleep326823-21latency_hist23:57:553
2675427517,37sleep326823-21latency_hist23:57:553
246212755,61sleep00-21swapper/022:07:270
188492756,57sleep20-21swapper/223:32:142
171112757,57sleep213441-21snmpd23:24:472
962127413,45sleep10-21swapper/123:03:061
521327422,23sleep30-21swapper/322:48:153
2788427424,27sleep10-21swapper/120:33:251
2682527315,43sleep026954-21latency_hist23:57:560
2404327316,42sleep124096-21if_err_eth023:48:121
1627627313,47sleep316319-21basename23:23:153
1522627319,43sleep20-21swapper/221:36:532
885627224,32sleep18916-21sendmail_mailst19:33:241
854727221,29sleep10-21swapper/122:58:221
2868227218,21sleep00-21swapper/020:37:560
1380927216,30sleep10-21swapper/121:32:561
1380927216,30sleep10-21swapper/121:32:561
53327118,29sleep3625-21apt-config00:17:543
486327123,27sleep00-21swapper/021:03:200
2971027116,36sleep029602-21/usr/sbin/munin22:23:130
2727227111,43sleep30-21swapper/320:33:063
1857427121,28sleep10-21swapper/120:03:241
1857427121,28sleep10-21swapper/120:03:241
180712719,48sleep318083-21sshd21:43:283
156992719,15sleep10-21swapper/119:53:271
3100227015,38sleep031049-21awk20:43:170
2286527018,43sleep20-21swapper/221:59:462
2085127025,25sleep30-21swapper/323:38:133
2081127017,29sleep30-21swapper/320:12:553
2081127017,29sleep30-21swapper/320:12:553
2064227018,41sleep20-21swapper/220:11:212
2064227018,41sleep20-21swapper/220:11:212
938726921,34sleep39459-21date23:02:563
878126920,31sleep08925-21/usr/sbin/munin22:58:250
699926918,27sleep10-21swapper/100:37:551
363026920,29sleep10-21swapper/119:18:131
360726912,42sleep03665-21irqstats22:43:150
349926917,42sleep20-21swapper/200:23:392
2862726921,23sleep00-21swapper/000:03:090
2862726921,23sleep00-21swapper/000:03:090
2417826920,30sleep024174-21apt-get20:23:060
2231826911,44sleep10-21swapper/120:17:541
161426918,31sleep10-21swapper/122:37:551
161426918,31sleep10-21swapper/122:37:551
1385626921,29sleep30-21swapper/319:48:243
127026921,23sleep10-21swapper/119:12:561
1181726919,31sleep111902-21fschecks_time19:43:131
70326813,40sleep0725-21ntp_states22:33:180
70326813,40sleep0725-21ntp_states22:33:180
394426815,32sleep03457-21/usr/sbin/munin19:18:200
2413126818,29sleep30-21swapper/323:48:143
2184026820,28sleep321832-21apt-get21:58:093
292862676,50sleep00-21swapper/022:22:160
260582677,49sleep00-21swapper/022:11:020
2184226712,27sleep00-21swapper/021:58:100
1673326719,22sleep00-21swapper/019:58:180
1673326719,22sleep00-21swapper/019:58:180
151972677,12sleep00-21swapper/019:53:180
99626622,27sleep30-21swapper/320:52:533
479126614,28sleep30-21swapper/319:22:563
2281926622,21sleep30-21swapper/323:43:223
2159826616,32sleep121547-21latency_hist21:57:551
1394126629,19sleep014059-21grep19:48:250
900926523,23sleep30-21swapper/321:17:563
900926523,23sleep30-21swapper/321:17:563
733826519,21sleep30-21swapper/321:12:563
676526516,26sleep00-21swapper/022:53:160
492026512,33sleep00-21swapper/022:48:030
2876026516,20sleep10-21swapper/122:18:261
128826520,25sleep30-21swapper/320:53:093
775326419,28sleep17778-21munin-run22:57:531
2938626421,25sleep30-21swapper/322:22:563
2422226421,23sleep00-21swapper/023:48:150
2300626412,23sleep00-21swapper/020:18:170
2223926423,34sleep013591-1kworker/0:2H20:16:520
1991626418,22sleep10-21swapper/123:33:261
1842726417,26sleep10-21swapper/121:47:541
1269226418,23sleep10-21swapper/123:13:031
107572644,49sleep00-21swapper/023:06:160
504026318,26sleep10-21swapper/119:23:141
487726318,26sleep14898-21apt-get21:03:211
322926313,23sleep30-21swapper/320:58:183
2570026313,20sleep00-21swapper/020:28:030
1844226313,20sleep30-21swapper/320:03:223
1844226313,20sleep30-21swapper/320:03:223
1439126323,21sleep10-21swapper/123:18:121
1398926320,18sleep30-21swapper/323:17:543
864026211,33sleep38339-21/usr/sbin/munin19:33:183
494626219,25sleep35038-21sendmail_mailtr21:03:233
448526217,25sleep00-21swapper/000:28:170
318712625,44sleep10-21swapper/120:47:341
2972326214,27sleep10-21swapper/122:23:141
734326116,27sleep3281-21dbus-daemon19:28:373
3140426114,25sleep00-21swapper/000:12:550
3140426114,25sleep00-21swapper/000:12:550
3137126119,30sleep331513-21unixbench_singl20:43:243
2696926116,28sleep126989-21apt-get23:58:061
225626119,21sleep30-21swapper/322:38:183
1687326115,30sleep116914-21cut19:58:241
1687326115,30sleep116914-21cut19:58:241
104826118,19sleep10-21swapper/120:52:551
2854626020,24sleep128539-21apt-get00:03:061
2854626020,24sleep128539-21apt-get00:03:061
112362606,10sleep30-21swapper/319:40:163
257025915,27sleep32562-21apt-get00:23:073
1619625917,29sleep316287-21sed21:38:233
1338825916,24sleep10-21swapper/119:48:141
684325816,27sleep16870-21cpu19:28:231
602925815,23sleep10-21swapper/121:08:091
386525813,21sleep30-21swapper/322:43:223
1298625816,22sleep00-21swapper/023:13:140
1180125816,26sleep311807-21df_inode19:43:113
1137325812,30sleep011402-21ntp_states21:23:180
629725718,17sleep00-21swapper/021:08:160
2549125714,28sleep125567-21sed20:27:551
129325713,19sleep00-21swapper/020:53:100
1132625720,23sleep311370-21ntp_kernel_pll_21:23:173
3127825615,24sleep131284-21ntp_states19:03:201
2494425615,24sleep124938-21apt-get22:08:081
2339025619,21sleep323387-21apt-get22:03:063
1895025613,28sleep118984-21/usr/sbin/munin21:48:141
103225613,26sleep31011-21apt-get00:18:153
493925516,23sleep14940-21apt-get22:48:051
3142125517,18sleep00-21swapper/019:03:230
3124825512,20sleep10-21swapper/122:28:101
2648625515,24sleep126481-21apt-get22:13:051
2624125513,26sleep326251-21/usr/sbin/munin22:12:553
2602225517,16sleep00-21swapper/023:53:240
2402525517,20sleep10-21swapper/122:03:241
2104725513,25sleep321043-21apt-get20:13:073
206522555,11sleep00-21swapper/020:11:270
206522555,11sleep00-21swapper/020:11:270
121125517,22sleep31243-21munin-run19:12:543
962525417,20sleep30-21swapper/321:18:413
962525417,20sleep30-21swapper/321:18:413
45092546,39sleep10-21swapper/122:45:591
2967125411,19sleep00-21swapper/020:38:410
2967125411,19sleep00-21swapper/020:38:410
2726625414,24sleep027360-21memory23:58:170
2315725415,23sleep123251-21latency_hist22:02:551
2076225415,23sleep020759-21apt-get23:38:060
182625417,17sleep10-21swapper/120:53:241
1402025413,22sleep114038-21apt-get21:33:041
633425316,17sleep30-21swapper/300:33:253
2599525212,25sleep326088-21cut20:28:143
2152425217,19sleep00-21swapper/021:57:540
1926725214,15sleep00-21swapper/020:07:560
1926725214,15sleep00-21swapper/020:07:560
1787125212,22sleep117912-21irqstats23:28:171
1230825219,18sleep30-21swapper/323:11:203
3211025115,18sleep10-21swapper/100:13:201
271725113,21sleep00-21swapper/000:23:120
1605725114,21sleep015581-21apt-get23:22:590
884625014,19sleep18923-21apt-config21:17:541
884625014,19sleep18923-21apt-config21:17:541
684225014,18sleep00-21swapper/019:28:220
1277225012,21sleep312865-21memory21:28:183
1277225012,21sleep312865-21memory21:28:183
101225016,19sleep00-21swapper/020:52:540
613324915,20sleep00-21swapper/000:33:210
45824911,17sleep30-21swapper/322:33:133
45824911,17sleep30-21swapper/322:33:133
323624912,24sleep03263-21latency_hist19:17:560
323624912,24sleep03263-21latency_hist19:17:560
2787724914,16sleep00-21swapper/022:17:560
2270624914,20sleep122712-21df_inode20:18:101
3124024811,22sleep031247-21df_inode22:28:090
75832466,15sleep00-21swapper/021:13:090
75832466,15sleep00-21swapper/021:13:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional