You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-18 - 11:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Mar 18, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1549728659,808sleep115546-21unin-run23:07:581
1549728659,808sleep115546-21unin-run23:07:581
1589227695,750sleep122524-1kworker/1:1H19:43:101
5724270211,672sleep25800-21/usr/sbin/munin20:53:262
13858261416,578sleep313931-21/usr/sbin/munin21:18:263
14559253511,508sleep014569-21ntpq19:38:240
8340241813,389sleep18418-21mailstats22:44:211
8340241813,389sleep18418-21mailstats22:44:211
7377240957,206sleep17576-21latency_hist22:42:591
7377240957,206sleep17576-21latency_hist22:42:591
7426230713,277sleep27465-21ntpq20:58:352
17210229915,261sleep317292-21latency_hist19:48:003
1638622908,239sleep116409-21grep21:28:121
27013228514,212sleep127057-21fschecks_count23:43:171
21519227578,179sleep1222-21systemd-journal23:25:171
25962227163,184sleep126116-21unixbench_singl20:13:291
32009222959,36sleep131581-21/usr/sbin/munin23:58:221
32009222959,36sleep131581-21/usr/sbin/munin23:58:221
25561221612,182sleep125612-21sshd23:38:161
13940216719,122sleep014063-21timerwakeupswit21:18:270
1050721637,22sleep15423-21kworker/1:219:28:001
19316215620,24sleep30-21swapper/321:37:593
29344215411,68sleep10-21swapper/122:08:211
12462215422,111sleep012146-21apt-get22:58:000
3120221489,88sleep131227-21cat20:32:591
473021445,38sleep10-21swapper/120:50:191
29612214423,101sleep029634-21awk20:27:570
19058214211,16sleep00-21swapper/019:53:140
18015214018,97sleep318161-21unixbench_multi23:13:293
949221396,38sleep10-21swapper/100:29:381
5669213910,69sleep10-21swapper/100:18:201
3201213919,97sleep33334-21unixbench_multi19:03:283
8202213611,102sleep18205-21cron00:27:561
451221339,54sleep10-21swapper/119:08:221
28082213346,30sleep10-21swapper/122:03:281
8261213111,60sleep10-21swapper/119:18:271
24155213110,66sleep124218-21sed21:52:591
11298212810,103sleep111369-21gpgv00:37:571
8973212616,85sleep29024-21ntpq21:03:232
484321265,109sleep00-21swapper/020:51:470
2467621266,40sleep10-21swapper/120:09:381
6190212528,81sleep10-21swapper/120:54:141
32292212322,85sleep232373-21fschecks_time22:18:142
1786321239,95sleep10-21swapper/119:48:231
17094212310,64sleep117159-21cat23:12:581
17094212310,64sleep117159-21cat23:12:581
30281211911,24sleep030328-21tr20:28:220
25122211971,27sleep125279-21unixbench_singl21:53:271
7139211710,54sleep17163-21fschecks_count20:58:261
2558211722,74sleep02588-21idleruntime-cro22:27:570
3185021159,94sleep131885-21ntpq20:33:231
2429021157,15sleep30-21swapper/323:33:263
2243921155,37sleep10-21swapper/121:46:251
8206211417,83sleep08230-21netstat22:44:160
8206211417,83sleep08230-21netstat22:44:160
24143211422,73sleep324184-21processes20:08:273
21438211422,73sleep321458-21fschecks_count21:43:323
3102321138,92sleep131067-21ntpq22:13:211
1784121129,86sleep10-21swapper/123:13:271
9908211143,56sleep19920-21latency_hist21:07:581
9613211113,78sleep09623-21/usr/sbin/munin22:48:360
1421921115,17sleep10-21swapper/121:18:441
1151721118,46sleep10-21swapper/119:28:281
10828211118,69sleep010917-21fschecks_time19:28:150
20657211014,77sleep320752-21fschecks_time19:58:173
22212210836,53sleep10-21swapper/120:03:161
723210715,77sleep3724-21sshd22:19:163
2720721078,76sleep10-21swapper/120:18:221
3052421058,84sleep130613-21tr23:53:261
2897210557,31sleep12948-21ntp_kernel_pll_19:03:211
13949210536,57sleep113967-21munin-run23:02:571
10575210549,39sleep110588-21cron22:52:561
17430210412,60sleep00-21swapper/023:13:170
16095210414,74sleep216112-21idleruntime-cro21:27:572
6442210316,73sleep25955-21/usr/sbin/munin22:38:242
6442210316,73sleep25955-21/usr/sbin/munin22:38:242
24774210362,19sleep10-21swapper/123:34:171
15315210329,24sleep10-21swapper/119:42:171
12199210361,20sleep10-21swapper/121:13:231
21299210220,65sleep021352-21cpuspeed21:43:130
12912210227,13sleep10-21swapper/119:34:061
4155210114,15sleep10-21swapper/122:32:581
8305210014,72sleep08317-21sh21:02:570
798210021,62sleep3894-21sort00:03:163
798210021,62sleep3894-21sort00:03:163
1421029915,64sleep314211-21/usr/sbin/munin23:03:073
1421029915,64sleep314211-21/usr/sbin/munin23:03:073
59812988,41sleep16006-21timerandwakeup19:13:061
59812988,41sleep16006-21timerandwakeup19:13:061
1418329815,67sleep214086-21/usr/sbin/munin19:38:142
2893529712,67sleep328958-21/usr/sbin/munin23:48:233
2674229714,68sleep026656-21sh23:43:000
1848429763,19sleep10-21swapper/121:33:251
1206829741,10sleep10-21swapper/122:57:121
111522975,80sleep10-21swapper/121:09:361
37332967,11sleep20-21swapper/219:07:182
22942967,13sleep10-21swapper/122:24:351
1615829612,69sleep116313-21cat21:28:001
1040229617,64sleep310403-21sshd19:27:173
28662959,67sleep00-21swapper/000:08:260
849629421,56sleep38550-21sort00:28:153
670429413,64sleep06858-21latency_hist00:22:590
161512946,12sleep30-21swapper/323:08:203
161512946,12sleep30-21swapper/323:08:203
202629349,25sleep12182-21latency_hist00:07:591
202629349,25sleep12182-21latency_hist00:07:591
2025729318,57sleep10-21swapper/123:22:591
174729332,13sleep10-21swapper/120:41:551
174729332,13sleep10-21swapper/120:41:551
1062729322,49sleep310714-21latency_hist22:52:583
2057029126,15sleep10-21swapper/119:58:111
1614829022,48sleep016155-21cat21:27:590
563628917,58sleep05654-21unixbench_multi19:13:010
563628917,58sleep05654-21unixbench_multi19:13:010
552328926,18sleep10-21swapper/122:36:091
2823628914,59sleep328353-21latency_hist23:47:593
776428830,33sleep27756-21apt-get22:43:132
776428830,33sleep27756-21apt-get22:43:132
50712885,17sleep10-21swapper/100:17:511
1753228833,32sleep30-21swapper/319:48:163
322062879,19sleep10-21swapper/122:18:091
2831728728,10sleep10-21swapper/120:23:021
2647728710,63sleep126559-21mailstats21:58:281
2333128727,11sleep10-21swapper/123:30:341
155828716,55sleep31560-21sshd20:39:313
155828716,55sleep31560-21sshd20:39:313
2080928611,52sleep10-21swapper/121:42:581
2806228511,59sleep028093-21idleruntime-cro20:22:570
2611728417,48sleep226126-21vmstat20:13:302
1076428415,49sleep210866-21timerandwakeup21:08:282
351128323,41sleep23525-21modprobe00:12:562
43312820,6sleep30-21swapper/322:32:593
2888728215,49sleep228958-21sensors_temp20:23:272
3128028120,40sleep031305-21munin-run23:57:570
471528015,44sleep24799-21grep19:08:272
38692808,16sleep30-21swapper/300:13:133
1304728016,46sleep313056-21/usr/sbin/munin22:58:233
2273327915,15sleep00-21swapper/020:03:270
1586727925,37sleep20-21swapper/223:08:152
1586727925,37sleep20-21swapper/223:08:152
1331027913,50sleep213435-21expr22:58:302
1297027922,37sleep312997-21munin-run21:17:573
22292786,15sleep10-21swapper/100:08:071
1200127819,32sleep20-21swapper/219:32:572
2656727713,15sleep30-21swapper/320:17:593
2514427715,48sleep00-21swapper/023:37:590
2407727717,28sleep00-21swapper/020:08:240
189882778,15sleep10-21swapper/123:18:151
1618027716,44sleep216239-21latency19:43:272
2492627613,17sleep30-21swapper/321:53:253
200627628,27sleep00-21swapper/000:07:580
200627628,27sleep00-21swapper/000:07:580
194242766,13sleep10-21swapper/119:53:231
924127514,38sleep09373-21grep21:03:300
908327522,26sleep30-21swapper/322:48:023
908327522,26sleep30-21swapper/322:48:023
3202927520,32sleep231423-21/usr/sbin/munin20:33:282
284722757,14sleep10-21swapper/123:48:091
1131627527,23sleep30-21swapper/319:28:263
1086627518,28sleep00-21swapper/022:53:120
754627417,43sleep37627-21timerandwakeup00:23:263
664727426,27sleep30-21swapper/320:58:003
2435827416,17sleep20-21swapper/220:08:302
440127317,43sleep24412-21grep22:33:122
2998527322,23sleep20-21swapper/220:28:162
1938827329,26sleep00-21swapper/023:18:260
1938827329,26sleep00-21swapper/023:18:260
83027217,28sleep20-21swapper/220:38:172
83027217,28sleep20-21swapper/220:38:172
29762726,14sleep10-21swapper/120:43:291
2584127211,46sleep025865-21proc_pri23:38:240
2583527220,30sleep00-21swapper/020:13:280
222212727,13sleep30-21swapper/323:28:053
1556327220,33sleep315717-21unixbench_singl21:23:263
1249927218,36sleep012507-21/usr/sbin/munin19:33:080
887427120,24sleep00-21swapper/000:28:240
2944427110,12sleep30-21swapper/322:08:233
2883627127,19sleep00-21swapper/022:07:590
3038827015,31sleep030540-21latency_hist22:13:000
298827019,28sleep00-21swapper/019:03:250
1094827013,37sleep310960-21/usr/sbin/munin22:53:153
101252709,15sleep00-21swapper/000:33:160
477926927,23sleep30-21swapper/322:33:213
2175926917,30sleep321804-21munin-run23:27:573
417426824,27sleep24330-21wc22:32:592
1141326815,27sleep30-21swapper/321:12:573
1367326725,19sleep20-21swapper/221:18:212
3270626615,30sleep3300-21sendmail-msp20:37:573
2184526624,17sleep00-21swapper/023:27:580
2172426615,30sleep221757-21awk23:27:562
208826619,18sleep00-21swapper/020:43:070
1996226617,31sleep219483-21/usr/sbin/munin21:38:232
1206726618,25sleep00-21swapper/021:13:180
1062726615,34sleep010691-21sendmail_mailqu21:08:260
3221226524,25sleep20-21swapper/223:58:272
3221226524,25sleep20-21swapper/223:58:272
930426417,29sleep29259-21apt-get22:48:092
550426418,21sleep30-21swapper/319:13:003
550426418,21sleep30-21swapper/319:13:003
314422648,40sleep30-21swapper/323:58:003
314422648,40sleep30-21swapper/323:58:003
275282649,42sleep30-21swapper/322:03:143
1966826415,21sleep20-21swapper/219:53:282
928226318,20sleep20-21swapper/219:23:162
762226312,15sleep20-21swapper/219:18:082
2896026315,31sleep00-21swapper/023:48:240
2811226315,22sleep20-21swapper/220:22:592
238226322,19sleep30-21swapper/320:43:183
2077026314,29sleep00-21swapper/021:42:570
1839226314,19sleep30-21swapper/321:33:233
174542638,18sleep20-21swapper/221:30:272
1715826322,23sleep00-21swapper/019:47:570
741126219,18sleep20-21swapper/219:18:002
741126219,18sleep20-21swapper/219:18:002
498326219,22sleep30-21swapper/320:52:593
331126213,21sleep00-21swapper/022:28:250
3070126226,20sleep30-21swapper/323:53:283
2302026219,22sleep00-21swapper/021:48:180
185342626,11sleep00-21swapper/019:50:440
973226117,19sleep00-21swapper/019:23:270
380326114,30sleep23868-21/usr/sbin/munin00:13:122
3191226113,32sleep031949-21ps20:33:260
3061426113,32sleep230698-21threads23:53:272
2979826116,20sleep20-21swapper/222:08:302
2668926119,22sleep30-21swapper/323:42:593
2225626117,20sleep20-21swapper/223:28:132
1945426116,23sleep30-21swapper/319:53:253
530526011,32sleep05317-21/usr/sbin/munin20:53:160
1662226015,25sleep316656-21irqstats21:28:193
1419326022,21sleep30-21swapper/319:38:153
1145726018,21sleep30-21swapper/300:37:593
723825919,20sleep30-21swapper/322:42:573
723825919,20sleep30-21swapper/322:42:573
457925916,27sleep00-21swapper/019:08:250
108725916,21sleep00-21swapper/022:23:010
2812525818,24sleep30-21swapper/320:23:003
1771725819,18sleep20-21swapper/219:48:192
1646925816,24sleep00-21swapper/021:28:150
154502584,10sleep00-21swapper/023:07:200
154502584,10sleep00-21swapper/023:07:200
691625717,20sleep00-21swapper/000:23:070
497425718,19sleep00-21swapper/022:33:260
2752725717,22sleep227521-21apt-get22:03:132
2054525725,15sleep20-21swapper/219:58:082
170925715,26sleep31754-21ntp_states22:23:243
747725615,19sleep20-21swapper/200:23:252
69302565,39sleep00-21swapper/022:38:590
69302565,39sleep00-21swapper/022:38:590
558025612,19sleep30-21swapper/300:18:183
2261025619,15sleep20-21swapper/221:47:592
917025518,20sleep30-21swapper/321:03:283
849525513,25sleep28489-21apt-get00:28:142
2847325511,24sleep228467-21apt-get23:48:102
2389325514,22sleep223986-21perl23:33:172
329025410,18sleep30-21swapper/300:10:073
3109025415,21sleep00-21swapper/022:13:240
294662545,11sleep30-21swapper/320:26:073
2597025416,23sleep225966-21apt-get21:58:122
2439325415,23sleep224460-21df21:53:122
179362548,17sleep00-21swapper/021:33:120
1253225415,22sleep312519-21apt-get19:33:153
3172825313,22sleep00-21swapper/023:58:170
3172825313,22sleep00-21swapper/023:58:170
269142538,36sleep00-21swapper/021:58:550
2580825315,22sleep225832-21ntp_states23:38:222
2439125314,24sleep024386-21apt-get21:53:100
1926825316,18sleep20-21swapper/221:37:582
1891925313,24sleep318914-21apt-get23:18:133
1891925313,24sleep318914-21apt-get23:18:133
1733325313,23sleep217329-21apt-get23:13:112
1431625310,15sleep20-21swapper/223:03:152
1431625310,15sleep20-21swapper/223:03:152
575925216,19sleep20-21swapper/200:18:212
2330825213,16sleep30-21swapper/321:48:253
125642528,19sleep20-21swapper/219:33:382
480225111,20sleep20-21swapper/220:51:162
221962518,28sleep322205-21df_inode20:03:143
1899625113,22sleep219002-21diskstats23:18:162
1899625113,22sleep219002-21diskstats23:18:162
1584825113,18sleep00-21swapper/023:08:130
1584825113,18sleep00-21swapper/023:08:130
118512517,32sleep20-21swapper/222:54:272
1011125115,18sleep20-21swapper/200:33:142
894625011,22sleep08956-21munin-run19:22:580
79025010,18sleep20-21swapper/200:03:152
79025010,18sleep20-21swapper/200:03:152
3062125014,20sleep230615-21apt-get22:13:102
167125018,17sleep20-21swapper/222:23:232
1194525012,22sleep211968-21irqstats21:13:162
3196924917,18sleep332034-21latency_hist22:17:593
2356624912,25sleep023535-21latency_hist23:32:590
2142824915,16sleep00-21swapper/019:58:310
746324812,13sleep30-21swapper/319:18:013
53424814,17sleep00-21swapper/020:38:070
53424814,17sleep00-21swapper/020:38:070
264924811,20sleep22720-21sensors_temp20:43:252
2311724814,15sleep20-21swapper/221:48:202
388124712,20sleep03887-21diskstats00:13:140
2211624710,21sleep222119-21apt-get20:03:092
700724611,20sleep36503-21apt-get20:58:083
41102458,19sleep30-21swapper/319:08:123
2597124510,18sleep30-21swapper/321:58:133
2990124310,13sleep30-21swapper/320:28:133
2729623911,12sleep20-21swapper/223:43:222
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional