You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-07 - 10:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Tue Apr 07, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
68062117916,1141sleep326257-21kworker/u8:020:42:053
2069269512,669sleep02161-21sshd23:52:100
1904725016,456sleep219082-21ntpq23:02:122
31688242547,326sleep231837-21grep20:17:202
29071241535,257sleep2222-21systemd-journal20:11:472
17749239818,361sleep117765-21fschecks_count21:17:081
1960228315,191sleep22102-21head22:07:212
4474227719,238sleep04526-21fschecks_time22:17:080
17421227031,202sleep217381-21latency_hist21:16:492
933225430,195sleep1958-21modprobe22:06:471
3342223718,205sleep13416-21cat22:12:161
3051222945,133sleep20-21swapper/220:31:502
21065221926,63sleep221104-21modprobe19:46:472
884321828,135sleep20-21swapper/200:12:112
23560217829,128sleep323555-21apt-get23:16:593
3306217252,65sleep23394-21apt-key23:56:482
12122216718,136sleep112127-21cat19:17:061
30017216440,71sleep230069-21munin-plugin-st21:56:472
9218216318,127sleep29326-21unixbench_multi19:07:182
629621628,140sleep36332-21ntpq22:22:113
13053216243,104sleep213067-21fschecks_count21:02:072
8073215912,55sleep20-21swapper/222:27:152
10820215826,118sleep213441-21snmpd22:37:012
10820215826,118sleep213441-21snmpd22:37:012
4117215612,81sleep20-21swapper/222:16:502
4076215416,123sleep04151-21timerandwakeup20:33:340
7172215132,73sleep20-21swapper/200:07:102
29996215130,80sleep230120-21timerwakeupswit20:12:202
14952215114,119sleep20-21swapper/200:32:102
14137215130,70sleep20-21swapper/219:22:162
988214916,112sleep01112-21latency_hist22:06:500
5901214431,70sleep25953-21cpuspeed22:22:022
1869421446,125sleep2191-21jbd2/mmcblk2p2-21:18:552
15342214418,93sleep10-21swapper/119:27:081
9813214245,53sleep20-21swapper/220:51:572
1449321399,15sleep20-21swapper/200:31:492
2635221356,72sleep20-21swapper/220:02:062
4630213412,70sleep24645-21modprobe20:36:472
20171213217,45sleep20-21swapper/223:06:572
25937213011,66sleep226140-21apt-config20:01:502
16444213017,96sleep316418-21latency_hist19:31:503
32106212954,47sleep20-21swapper/222:02:002
31290212944,60sleep20-21swapper/221:57:442
27971212819,87sleep328022-21latency_hist23:31:493
381321266,107sleep23788-21apt-get23:57:122
11095212221,83sleep3222-21systemd-journal20:56:473
25636212011,53sleep20-21swapper/221:41:572
2721321197,96sleep20-21swapper/221:47:012
25737211819,80sleep3292-21in:imuxsock20:01:473
21444211816,27sleep121535-21perl19:47:071
26872211749,21sleep20-21swapper/223:27:082
19818211738,20sleep20-21swapper/221:22:182
11238211729,74sleep211099-21cron20:56:502
9814211511,22sleep00-21swapper/019:11:480
29336211514,85sleep129428-21sed21:52:171
2453021146,29sleep20-21swapper/223:18:072
29919211315,84sleep129993-21switchtime20:12:191
6077211221,64sleep36221-21grep00:02:183
1366821126,12sleep10-21swapper/100:27:161
780621115,30sleep10-21swapper/100:07:211
12133211111,83sleep212203-21switchtime00:22:162
25651211015,74sleep325722-21/usr/sbin/munin23:22:163
157721105,29sleep20-21swapper/223:50:222
1641121087,11sleep20-21swapper/222:52:432
15159210818,70sleep015224-21grep00:32:160
5777210715,71sleep15846-21/usr/sbin/munin20:37:181
12848210616,75sleep012910-21sendmail_mailqu22:42:160
16431210515,72sleep116446-21/usr/sbin/munin21:12:111
12978210324,57sleep20-21swapper/222:42:182
18685210210,80sleep018687-21sshd21:18:490
27014210115,70sleep327160-21cat21:46:503
30699210013,74sleep330718-21latency_hist20:16:493
2864421006,29sleep20-21swapper/223:32:132
451229922,56sleep03768-21/usr/sbin/munin23:57:470
2183729953,12sleep20-21swapper/219:47:162
1133729947,12sleep20-21swapper/200:21:482
66829813,31sleep3695-21irqstats23:47:083
456729846,14sleep20-21swapper/222:17:102
2626829823,62sleep326295-21grep20:02:033
2211329812,69sleep022152-21ntpq23:12:120
2211329812,69sleep022152-21ntpq23:12:120
515429736,25sleep20-21swapper/220:37:072
843729611,68sleep08506-21idleruntime20:47:070
93202949,18sleep30-21swapper/322:32:053
93202949,18sleep30-21swapper/322:32:053
3229229411,65sleep332350-21idleruntime22:02:093
2644329423,44sleep026592-21latency_hist23:26:500
2422529425,58sleep224235-21munin-run19:56:492
2422529425,58sleep224235-21munin-run19:56:492
118229416,61sleep11323-21uptime23:47:181
982729220,45sleep19831-21munin-node19:11:491
942729125,49sleep20-21swapper/222:32:082
942729125,49sleep20-21swapper/222:32:082
3154429124,15sleep28270-21cyclictest23:42:102
2666529116,33sleep00-21swapper/023:27:020
2408429138,14sleep20-21swapper/221:36:522
226592919,65sleep322717-21cat19:51:493
789428915,63sleep20-21swapper/220:45:312
339628820,48sleep33492-21sed23:56:493
323228828,40sleep13255-21munin-run23:56:471
562628720,40sleep05774-21grep20:37:170
1834028715,59sleep118360-21grep19:37:051
992328619,13sleep20-21swapper/219:11:502
493128615,58sleep35024-21cat22:17:193
3152228613,24sleep10-21swapper/121:57:471
2073228515,53sleep120750-21apt21:26:571
192392856,68sleep20-21swapper/219:38:152
740828427,34sleep00-21swapper/019:02:160
3027628419,52sleep130353-21mailstats23:37:171
3003528422,13sleep20-21swapper/223:37:092
2052028423,33sleep120606-21latency_hist21:26:491
1794128413,45sleep20-21swapper/222:57:202
2488128344,16sleep00-21swapper/023:21:490
172882839,63sleep013441-21snmpd21:16:020
264142827,19sleep20-21swapper/223:26:482
169512829,16sleep20-21swapper/219:32:102
1466428223,42sleep314816-21sed22:47:193
1112028211,57sleep011172-21unin-run20:56:480
656028113,55sleep16674-21timerwakeupswit22:22:181
569428121,27sleep10-21swapper/122:21:491
307132818,59sleep313441-21snmpd21:57:203
2273528128,18sleep213441-21snmpd23:13:232
2273528128,18sleep213441-21snmpd23:13:232
673528012,54sleep00-21swapper/020:42:020
140042809,18sleep20-21swapper/222:47:062
670527929,25sleep10-21swapper/120:41:591
1793827916,43sleep317956-21apt19:36:473
121282797,14sleep20-21swapper/219:17:072
960327813,48sleep39722-21latency_hist20:51:503
2001327817,41sleep319779-21/usr/sbin/munin19:42:083
1372427815,44sleep313865-21uptime21:02:203
1321927812,51sleep313295-21cut00:27:063
88932775,20sleep20-21swapper/220:47:172
3092127719,33sleep10-21swapper/120:16:591
2808227718,43sleep128098-21kernelversion20:07:101
1865127713,51sleep318692-21ntp_kernel_pll_19:37:113
1573627724,15sleep20-21swapper/219:27:172
2721127625,34sleep127206-21apt-get21:47:001
1991027617,39sleep119960-21munin-run23:06:481
1044427623,30sleep10-21swapper/120:52:171
441927515,17sleep30-21swapper/323:57:463
2536927517,43sleep125382-21vmstat19:57:201
2326527517,39sleep023296-21/usr/sbin/munin19:52:110
2326527517,39sleep023296-21/usr/sbin/munin19:52:110
1520627523,22sleep00-21swapper/021:07:180
764327421,38sleep37788-21unixbench_singl00:07:193
307232749,49sleep030871-21wc20:16:500
291027424,23sleep10-21swapper/120:31:491
163227415,23sleep30-21swapper/322:07:133
1555127411,16sleep00-21swapper/022:52:050
13482749,26sleep21484-21date20:26:492
1682627319,28sleep10-21swapper/122:56:501
119927314,39sleep11200-21/usr/sbin/munin22:06:591
2503527224,30sleep124095-21/usr/sbin/munin21:37:231
1745827215,25sleep30-21swapper/321:16:503
1050527222,30sleep010534-21modprobe22:36:460
1050527222,30sleep010534-21modprobe22:36:460
1836127118,33sleep118386-21munin-run23:01:471
1375427114,16sleep10-21swapper/119:22:071
1204927115,41sleep012069-21open_inodes00:22:140
1105527127,21sleep00-21swapper/019:12:170
882227014,41sleep18833-21netstat00:12:101
3184127016,36sleep331850-21vmstat20:17:213
2756727026,25sleep0281-21dbus-daemon20:06:480
208432709,42sleep20-21swapper/221:27:062
1149927017,26sleep30-21swapper/320:57:073
393026915,40sleep13938-21awk20:33:311
2885626910,43sleep328952-21fschecks_time21:52:073
1289726916,37sleep312993-21latency_hist00:26:493
960826819,22sleep10-21swapper/122:32:111
960826819,22sleep10-21swapper/122:32:111
95192685,12sleep10-21swapper/120:51:241
42126810,42sleep00-21swapper/020:22:110
2830326817,33sleep028395-21sort23:32:070
223032689,15sleep20-21swapper/221:31:592
1602226822,22sleep10-21swapper/100:36:491
1098226820,28sleep310425-21/usr/sbin/munin19:12:163
2267026717,36sleep322709-21ntpq21:32:083
1012726718,20sleep00-21swapper/000:17:050
1989026619,23sleep10-21swapper/119:42:061
227172659,13sleep00-21swapper/021:32:090
1130026515,35sleep111336-21idleruntime-cro00:21:471
904126423,21sleep30-21swapper/319:07:153
818026422,19sleep30-21swapper/319:06:473
625826422,23sleep36348-21/usr/sbin/munin20:41:483
3189626418,20sleep00-21swapper/022:01:490
2489226412,22sleep30-21swapper/321:37:213
1841126423,20sleep30-21swapper/323:01:493
178226411,36sleep01930-21sed23:51:490
145982647,16sleep20-21swapper/221:07:032
1336426415,26sleep013390-21ntpq21:02:130
111862647,13sleep30-21swapper/322:37:103
111862647,13sleep30-21swapper/322:37:103
900626322,23sleep30-21swapper/322:31:493
692226317,27sleep26918-21apt-get19:02:022
2875826319,28sleep028754-21apt-get21:52:000
284026314,33sleep32865-21munin-run20:31:473
1308426315,33sleep313230-21sed22:42:193
1213026315,29sleep10-21swapper/122:41:481
1213026315,29sleep10-21swapper/122:41:481
820526220,19sleep00-21swapper/000:11:480
49126217,28sleep3514-21/usr/sbin/munin20:22:143
2329726214,13sleep10-21swapper/119:52:121
2329726214,13sleep10-21swapper/119:52:121
2183126221,18sleep30-21swapper/323:12:073
2183126221,18sleep30-21swapper/323:12:073
1939526219,20sleep00-21swapper/021:22:080
1158526220,24sleep111582-21apt-get00:21:591
1030926215,28sleep310341-21/usr/sbin/munin00:17:083
59632618,16sleep20-21swapper/200:02:172
325426114,19sleep30-21swapper/320:31:513
2461026112,33sleep024693-21grep21:37:180
2135326118,26sleep021347-21apt-get19:47:020
1334326121,23sleep313420-21sed19:21:493
1231326119,22sleep10-21swapper/120:57:211
833226015,28sleep00-21swapper/022:27:210
3220026012,20sleep00-21swapper/020:21:470
2870526016,16sleep10-21swapper/123:32:161
2627026015,24sleep10-21swapper/121:42:161
238472606,11sleep30-21swapper/319:52:503
238472606,11sleep30-21swapper/319:52:503
2355726019,24sleep123043-21apt-get23:16:571
1422126019,17sleep30-21swapper/319:22:173
1352226011,24sleep10-21swapper/121:02:181
3104525916,25sleep331077-21munin-run23:41:473
2767425914,20sleep30-21swapper/321:47:153
211492597,12sleep00-21swapper/021:27:130
1017825916,21sleep10-21swapper/119:11:541
36852587,39sleep00-21swapper/022:12:230
2912625820,18sleep00-21swapper/020:11:490
2634125817,23sleep126351-21diskstats20:02:051
2557825812,27sleep10-21swapper/123:22:141
2522925813,22sleep00-21swapper/019:57:190
164952586,18sleep30-21swapper/322:53:493
141872586,44sleep00-21swapper/000:29:090
583525712,28sleep05895-21sendmail_mailst00:02:150
3254325719,21sleep10-21swapper/123:46:091
2046925714,27sleep320487-21munin-run21:26:473
150502576,11sleep00-21swapper/022:50:070
1460525714,26sleep314612-21/usr/sbin/munin21:07:043
845625613,26sleep08450-21apt-get00:12:000
768925614,23sleep17683-21apt-get22:27:021
597725614,17sleep00-21swapper/022:22:040
3142225612,31sleep031435-21fschecks_count23:42:080
151632565,40sleep10-21swapper/122:51:331
1514325611,21sleep30-21swapper/300:32:153
429825512,30sleep14338-21ps23:57:441
248125514,28sleep02510-21awk22:11:470
2056425515,25sleep320605-21ntpq23:07:103
1825425515,24sleep018249-21apt-get19:37:000
1524925515,23sleep015341-21perl19:27:070
1369825516,23sleep113840-21latency_hist22:46:491
325622548,35sleep30-21swapper/323:46:233
2806925412,26sleep028116-21users21:47:200
2788425418,20sleep30-21swapper/320:07:043
2666625415,22sleep126733-21df23:27:031
1988425414,24sleep019889-21diskstats19:42:050
1873525414,19sleep00-21swapper/023:02:060
1209825411,27sleep012129-21unin-run22:41:470
1209825411,27sleep012129-21unin-run22:41:470
1203625413,24sleep012030-21apt-get19:17:010
313725317,18sleep30-21swapper/322:12:103
287225315,22sleep02896-21apt-key20:31:480
2227725311,16sleep10-21swapper/123:12:171
2227725311,16sleep10-21swapper/123:12:171
1499125316,19sleep10-21swapper/121:07:151
2026725214,16sleep00-21swapper/023:07:030
1749925214,22sleep017582-21users19:32:200
146602526,37sleep00-21swapper/019:24:240
535125114,21sleep15346-21apt-get00:02:001
2866425114,21sleep328670-21ntp_states23:32:143
1816225116,17sleep30-21swapper/321:17:183
1801725116,18sleep10-21swapper/119:36:491
935925011,19sleep30-21swapper/300:12:203
3273825013,22sleep132762-21irqstats20:22:081
2238625014,20sleep122387-21diskstats21:32:021
2019625011,23sleep120191-21apt-get23:07:001
1271925013,19sleep00-21swapper/021:01:480
2566124913,21sleep025655-21apt-get21:41:590
1768424912,17sleep00-21swapper/022:57:180
734224810,16sleep10-21swapper/119:02:131
3108624313,16sleep00-21swapper/023:41:480
190842429,18sleep319103-21ntp_states23:02:133
220152419,14sleep30-21swapper/321:31:483
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional