You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-05 - 06:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue May 05, 2026 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8718216879,1666sleep230894-1kworker/2:0H21:15:212
22486239415,366sleep222488-21sshd21:57:532
32306229075,196sleep00-21swapper/022:30:020
24794228715,188sleep024817-21kernelversion22:05:220
5993227719,236sleep36125-21unixbench_singl00:30:313
3734226778,174sleep03775-21fschecks_time21:00:410
29981226692,21sleep00-21swapper/000:05:200
28047223856,107sleep028069-21munin-run23:59:590
11202223773,81sleep011267-21awk19:41:300
8805222610,201sleep08837-21tune2fs21:15:270
67612223152,15sleep00-21swapper/022:50:250
565722079,91sleep05697-21awk00:30:240
3078220724,159sleep00-21swapper/019:15:300
15020219711,126sleep015039-21fschecks_count21:35:180
1360219522,152sleep11393-21grep22:34:591
25005218912,119sleep025149-21latency_hist23:50:010
29474218213,101sleep029416-21/usr/sbin/munin22:20:330
29474218213,101sleep029416-21/usr/sbin/munin22:20:330
12215217916,80sleep00-21swapper/021:25:250
1650421769,106sleep016589-21switchtime23:20:290
2145217015,68sleep00-21swapper/022:35:270
2145217015,68sleep00-21swapper/022:35:270
6117215913,89sleep06226-21latency_hist22:50:010
11758215764,22sleep00-21swapper/023:05:310
11758215764,22sleep00-21swapper/023:05:310
30202215457,41sleep030319-21sort20:40:300
30202215457,41sleep030319-21sort20:40:300
28889215210,82sleep01-21systemd22:20:000
28889215210,82sleep01-21systemd22:20:000
2100521495,10sleep00-21swapper/021:53:390
1652221488,101sleep016684-21apt-key21:40:020
16951214714,116sleep317004-21if_err_eth021:40:303
26598214313,118sleep326663-21cat20:30:203
25324214361,62sleep00-21swapper/023:50:180
689221427,15sleep00-21swapper/000:35:160
5215214024,100sleep05305-21perl21:05:220
31931214027,26sleep00-21swapper/019:05:290
2989213815,108sleep22937-21latency_hist22:40:012
2989213815,108sleep22937-21latency_hist22:40:012
26219213810,56sleep00-21swapper/022:10:190
2011021385,122sleep00-21swapper/023:32:490
6335213720,27sleep00-21swapper/019:25:310
380721378,111sleep13809-21cron19:19:591
12452213754,63sleep00-21swapper/019:45:220
31686213516,104sleep331743-21irqstats00:10:203
31686213516,104sleep331743-21irqstats00:10:203
3192521348,64sleep00-21swapper/000:10:250
3192521348,64sleep00-21swapper/000:10:250
1905212713,87sleep00-21swapper/019:15:010
1905212713,87sleep00-21swapper/019:15:010
24835212619,88sleep224353-21apt-get20:25:102
4768212514,96sleep3225-21systemd-journal00:27:203
11593212549,64sleep011602-21munin-run21:25:000
3383212312,93sleep23019-21apt-get21:00:032
1715212223,34sleep30-21swapper/320:55:113
6926212135,67sleep00-21swapper/021:10:180
22732212111,91sleep022815-21latency_hist22:00:010
1645321218,103sleep0363-21snmpd19:58:590
2337321209,87sleep023406-21ntpq22:00:260
20542212013,91sleep220543-21/usr/sbin/munin23:35:092
20542212013,91sleep220543-21/usr/sbin/munin23:35:092
31607211819,72sleep231450-21/usr/sbin/munin00:10:192
31607211819,72sleep231450-21/usr/sbin/munin00:10:192
29301211843,13sleep00-21swapper/020:40:020
29301211843,13sleep00-21swapper/020:40:020
14145211516,81sleep214285-21/usr/sbin/munin21:30:312
31923211417,36sleep331929-21ntp_states19:05:283
1685121149,74sleep00-21swapper/020:00:160
1685121149,74sleep00-21swapper/020:00:160
2293211217,22sleep10-21swapper/120:55:291
18785211223,62sleep318911-21latency_hist23:30:013
4636211034,15sleep00-21swapper/000:25:350
26992211017,21sleep00-21swapper/020:30:290
32637210942,35sleep00-21swapper/020:50:150
28826210937,40sleep00-21swapper/000:00:270
12687210911,80sleep30-21swapper/319:45:293
30254210815,63sleep10-21swapper/122:20:501
30254210815,63sleep10-21swapper/122:20:501
5207210712,81sleep35213-21cat21:05:213
1411210516,69sleep31498-21latency_hist22:35:013
9150210410,37sleep09196-21ntp_kernel_pll_19:35:250
8346210414,73sleep38192-21latency_hist21:15:023
3671210437,32sleep00-21swapper/022:40:230
3671210437,32sleep00-21swapper/022:40:230
2171921046,87sleep00-21swapper/023:38:350
2171921046,87sleep00-21swapper/023:38:350
13930210423,70sleep0363-21snmpd23:13:230
13164210413,71sleep312770-21/usr/sbin/munin23:10:233
23402210210,18sleep20-21swapper/223:44:592
23402210210,18sleep20-21swapper/223:44:592
2003421028,27sleep00-21swapper/020:10:180
8585210118,70sleep38626-21cut22:55:293
8585210118,70sleep38626-21cut22:55:293
31087210125,42sleep00-21swapper/020:45:150
31087210125,42sleep00-21swapper/020:45:150
8746210022,50sleep38765-21apt-get19:35:113
2307929930,42sleep223075-21apt-get20:20:132
2307929930,42sleep223075-21apt-get20:20:132
1971429963,15sleep00-21swapper/020:10:010
139452999,18sleep00-21swapper/021:30:290
821229817,66sleep18350-21cut00:40:021
223429812,17sleep30-21swapper/300:20:133
264029710,75sleep02653-21awk20:55:320
1475929721,14sleep00-21swapper/023:15:250
291329615,63sleep12988-21systemctl22:40:001
291329615,63sleep12988-21systemctl22:40:001
2668929512,68sleep126783-21/usr/sbin/munin20:30:221
2256129521,14sleep00-21swapper/023:40:240
2256129521,14sleep00-21swapper/023:40:240
2177129512,16sleep00-21swapper/020:15:220
2322729360,19sleep00-21swapper/020:20:190
2322729360,19sleep00-21swapper/020:20:190
16922935,28sleep00-21swapper/000:18:000
769229216,19sleep00-21swapper/019:30:280
769229216,19sleep00-21swapper/019:30:280
2354829020,45sleep123623-21grep22:00:311
182972898,17sleep00-21swapper/021:45:110
433528815,49sleep14407-21unixbench_multi21:00:551
102992889,66sleep010384-21timerandwakeup23:00:310
102992889,66sleep010384-21timerandwakeup23:00:310
1914628724,40sleep319283-21unixbench_singl20:05:323
3118428619,51sleep2225-21systemd-journal00:09:582
2678428614,54sleep226807-21/usr/sbin/munin20:30:232
1546728621,52sleep115525-21if_eth019:55:201
2191528512,54sleep321951-21ntp_states20:15:253
108322850,7sleep20-21swapper/219:41:112
5332848,14sleep00-21swapper/022:30:260
253482849,37sleep00-21swapper/020:25:360
1593028417,49sleep216000-21df23:20:152
820228320,47sleep28260-21tune2fs22:55:202
820228320,47sleep28260-21tune2fs22:55:202
43892836,13sleep00-21swapper/019:20:220
2362528315,29sleep20-21swapper/222:00:322
449328218,45sleep24621-21/usr/sbin/munin00:25:332
2498828218,47sleep225028-21ps22:05:282
1199428227,33sleep111917-21latency_hist19:45:031
2120728119,46sleep121236-21munin-plugin-st20:14:591
1101028115,48sleep311150-21sed23:05:023
1101028115,48sleep311150-21sed23:05:023
504928027,30sleep10-21swapper/122:45:211
2817428025,28sleep30-21swapper/300:00:023
2738228025,23sleep20-21swapper/222:15:012
2427928015,47sleep324300-21latency_hist22:05:013
871727915,45sleep18797-21awk22:55:311
871727915,45sleep18797-21awk22:55:311
3088827915,48sleep331036-21cut20:45:013
3088827915,48sleep331036-21cut20:45:013
2556127920,29sleep30-21swapper/320:25:393
992127820,35sleep29918-21apt-get19:38:212
2460127823,27sleep30-21swapper/320:25:033
2460127823,27sleep30-21swapper/320:25:033
55927614,28sleep10-21swapper/120:50:251
21642767,13sleep00-21swapper/000:20:100
1639127615,45sleep116394-21ntpq23:20:261
2035827515,29sleep30-21swapper/320:10:253
1306527519,26sleep10-21swapper/123:10:211
368627418,40sleep13693-21sh22:40:241
368627418,40sleep13693-21sh22:40:241
3123827419,28sleep10-21swapper/100:10:011
2905527416,30sleep10-21swapper/100:00:321
1426327414,23sleep30-21swapper/319:50:303
3121127316,23sleep30-21swapper/322:25:243
2071327320,34sleep320777-21sshd23:35:183
2071327320,34sleep320777-21sshd23:35:183
147472737,13sleep00-21swapper/019:51:560
60027221,32sleep30-21swapper/319:10:163
60027221,32sleep30-21swapper/319:10:163
39427215,43sleep232612-21/usr/sbin/munin20:50:222
2905227215,39sleep229230-21date22:20:022
2905227215,39sleep229230-21date22:20:022
1709327211,15sleep20-21swapper/220:00:222
1709327211,15sleep20-21swapper/220:00:222
1703427216,25sleep10-21swapper/120:00:211
1703427216,25sleep10-21swapper/120:00:211
1163427226,21sleep20-21swapper/221:25:022
185722717,10sleep00-21swapper/023:27:570
1038327110,42sleep30-21swapper/321:20:193
637727014,35sleep21-21systemd21:10:002
617127018,33sleep26249-21/usr/sbin/munin19:25:292
2954927012,14sleep20-21swapper/222:20:352
2954927012,14sleep20-21swapper/222:20:352
2495027027,24sleep10-21swapper/123:49:581
249272700,8sleep30-21swapper/322:05:243
223342706,10sleep10-21swapper/121:55:541
1993527024,21sleep20-21swapper/221:50:182
161792708,52sleep216127-21kworker/2:221:39:482
1561027017,31sleep215642-21munin-run23:19:592
93542696,10sleep00-21swapper/022:59:420
93542696,10sleep00-21swapper/022:59:420
1817326913,26sleep20-21swapper/223:25:322
422426811,40sleep24318-21fschecks_time19:20:202
373326821,22sleep30-21swapper/322:40:273
373326821,22sleep30-21swapper/322:40:273
2941526811,42sleep328909-21apt-get22:20:113
2941526811,42sleep328909-21apt-get22:20:113
192926814,37sleep12067-21cut19:15:021
192926814,37sleep12067-21cut19:15:021
1218226812,39sleep112249-21/usr/sbin/munin19:45:151
1122426811,26sleep20-21swapper/223:05:132
1122426811,26sleep20-21swapper/223:05:132
977826723,21sleep10-21swapper/123:00:191
977826723,21sleep10-21swapper/123:00:191
694426711,35sleep16909-21apt-get21:10:311
296362679,15sleep30-21swapper/300:04:593
2884126721,27sleep20-21swapper/200:00:292
235726726,21sleep20-21swapper/220:55:302
1674026721,24sleep10-21swapper/121:40:121
1318526723,28sleep213333-21latency_hist21:30:032
1220426716,24sleep30-21swapper/321:25:243
762726614,39sleep17671-21cat19:30:251
762726614,39sleep17671-21cat19:30:251
603926620,21sleep20-21swapper/222:49:582
2896426624,18sleep10-21swapper/122:20:011
2896426624,18sleep10-21swapper/122:20:011
2653926622,24sleep30-21swapper/323:55:013
263026615,33sleep20-21swapper/219:15:202
201826620,27sleep30-21swapper/322:35:223
201826620,27sleep30-21swapper/322:35:223
1693326621,20sleep30-21swapper/320:00:193
1693326621,20sleep30-21swapper/320:00:193
119526622,25sleep10-21swapper/100:15:311
932526513,37sleep19397-21switchtime19:35:301
2719126518,18sleep10-21swapper/123:55:241
2705526522,20sleep20-21swapper/223:55:202
231142659,14sleep10-21swapper/123:41:181
231142659,14sleep10-21swapper/123:41:181
1866626516,30sleep10-21swapper/120:05:221
1856826520,29sleep10-21swapper/121:45:201
1635226521,21sleep30-21swapper/321:40:013
6872649,41sleep2692-21entropy19:10:192
6872649,41sleep2692-21entropy19:10:192
347726423,17sleep20-21swapper/200:25:012
2960826413,30sleep20-21swapper/220:40:162
2960826413,30sleep20-21swapper/220:40:162
2155726414,30sleep321615-21/usr/sbin/munin21:55:203
1345926422,21sleep30-21swapper/321:30:163
83926314,20sleep20-21swapper/222:30:322
2572026319,28sleep125780-21grep23:50:271
2361526320,21sleep10-21swapper/120:20:281
2361526320,21sleep10-21swapper/120:20:281
1774826320,21sleep10-21swapper/123:25:231
657826227,17sleep10-21swapper/100:35:011
3145526213,35sleep131062-21/usr/sbin/munin20:45:241
3145526213,35sleep131062-21/usr/sbin/munin20:45:241
2636526219,26sleep20-21swapper/223:53:192
2021826211,33sleep320227-21needreboot21:50:233
1619326215,31sleep116258-21sendmail-msp21:39:591
1347526220,20sleep10-21swapper/121:30:181
1191926215,28sleep10-21swapper/121:25:171
354826110,17sleep30-21swapper/321:00:123
2810226117,25sleep328170-21systemctl22:15:273
2810226117,25sleep328170-21systemctl22:15:273
225026122,23sleep10-21swapper/100:20:151
1910826119,19sleep10-21swapper/123:30:201
1877326112,20sleep30-21swapper/321:45:263
1132926115,29sleep111336-21/usr/sbin/munin23:05:201
1132926115,29sleep111336-21/usr/sbin/munin23:05:201
724126020,19sleep20-21swapper/221:11:012
3184026024,19sleep10-21swapper/119:05:261
3156026021,21sleep10-21swapper/100:10:181
3156026021,21sleep10-21swapper/100:10:181
2990526015,27sleep329924-21apt-get00:05:153
2617926022,20sleep10-21swapper/120:30:011
1653026014,30sleep316559-21idleruntime-cro19:59:593
1503526023,18sleep20-21swapper/219:55:012
810925915,27sleep3284-21dbus-daemon00:40:003
588725912,25sleep20-21swapper/200:30:302
307712596,10sleep10-21swapper/120:44:391
307712596,10sleep10-21swapper/120:44:391
2831325912,27sleep10-21swapper/120:35:221
2831325912,27sleep10-21swapper/120:35:221
1083025915,18sleep10-21swapper/121:20:291
807425814,29sleep28106-21munin-plugin-st00:39:592
3189025814,24sleep231922-21ntp_states19:05:272
3187225815,21sleep10-21swapper/122:25:591
224282585,10sleep10-21swapper/120:15:481
99392577,13sleep30-21swapper/321:19:303
545025716,19sleep30-21swapper/319:25:023
388525718,20sleep30-21swapper/319:20:013
2498425716,26sleep124987-21ntp_states22:05:271
246382576,11sleep20-21swapper/223:46:002
2234225714,17sleep20-21swapper/220:15:332
1419125720,19sleep10-21swapper/119:50:281
1004725718,21sleep10-21swapper/121:20:001
693325612,18sleep30-21swapper/322:50:283
478325614,25sleep34850-21df22:45:153
380525615,17sleep10-21swapper/100:25:201
2035725615,21sleep10-21swapper/121:50:291
1276925615,25sleep212788-21apt-get23:10:102
1250625614,29sleep312542-21idleruntime-cro23:09:583
1250625614,29sleep312542-21idleruntime-cro23:09:583
730925512,25sleep37365-21tune2fs19:30:193
730925512,25sleep37365-21tune2fs19:30:193
625025515,23sleep30-21swapper/319:25:303
339425515,28sleep23449-21idleruntime22:40:172
339425515,28sleep23449-21idleruntime22:40:172
1806225515,24sleep218039-21latency_hist21:45:012
1006325511,27sleep210097-21/usr/sbin/munin23:00:242
1006325511,27sleep210097-21/usr/sbin/munin23:00:242
322972548,36sleep30-21swapper/320:49:223
322972548,36sleep30-21swapper/320:49:223
2614325414,24sleep126152-21df_inode22:10:161
2033225417,18sleep20-21swapper/223:35:012
1923225411,20sleep30-21swapper/323:30:223
697725317,16sleep10-21swapper/119:30:001
597725312,22sleep16024-21users21:05:351
3148425317,20sleep20-21swapper/220:45:252
3148425317,20sleep20-21swapper/220:45:252
3112425318,19sleep20-21swapper/222:25:232
1534925313,24sleep215355-21diskstats19:55:172
153172536,18sleep30-21swapper/323:16:213
421125214,21sleep14223-21sshd19:20:191
2803525214,22sleep328044-21diskstats20:35:163
2803525214,22sleep328044-21diskstats20:35:163
2190425218,17sleep10-21swapper/123:40:011
2190425218,17sleep10-21swapper/123:40:011
1723525215,23sleep317275-21sed23:25:013
12872525,11sleep30-21swapper/320:52:533
730425112,17sleep20-21swapper/219:30:182
730425112,17sleep20-21swapper/219:30:182
653925114,23sleep10-21swapper/122:50:101
1994725112,23sleep219944-21apt-get20:10:112
1063725111,24sleep310624-21apt-get19:40:163
2607325013,21sleep326067-21apt-get22:10:143
104425013,21sleep20-21swapper/200:15:282
1298024912,20sleep213083-21users19:45:322
2368824811,21sleep323684-21apt-get23:45:123
2262024813,15sleep30-21swapper/323:40:273
2262024813,15sleep30-21swapper/323:40:273
1868424813,17sleep20-21swapper/220:05:232
1827324811,22sleep218292-21apt-get21:45:092
1495624819,15sleep10-21swapper/123:15:301
143724813,19sleep30-21swapper/300:15:333
684924612,19sleep26546-21/usr/sbin/munin22:50:272
2813924616,17sleep20-21swapper/220:35:192
2813924616,17sleep20-21swapper/220:35:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional