You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-24 - 12:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sun May 24, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
265562722014,7194sleep125008-21kworker/1:023:11:571
31532256318,528sleep031566-21/usr/sbin/munin21:44:210
31532256318,528sleep031566-21/usr/sbin/munin21:44:210
16638256016,525sleep216708-21/usr/sbin/munin00:24:082
21063254616,513sleep121011-21latency_hist00:38:531
25303243139,377sleep325341-21ntpq21:24:183
29159240910,385sleep329249-21cat23:19:223
18161228826,237sleep318225-21ps19:19:203
2119422667,245sleep321240-21ntpq19:29:173
13230226628,202sleep312718-21/usr/sbin/munin22:29:253
23898225547,126sleep30-21swapper/321:19:213
32334225315,220sleep232379-21/usr/sbin/munin23:29:462
15043223540,178sleep315183-21unixbench_singl22:34:253
5654223018,86sleep35804-21unixbench_singl22:04:263
17182221940,117sleep3363-21snmpd22:43:543
29275221524,166sleep029423-21grep19:54:250
7108220128,112sleep30-21swapper/320:29:113
9829219951,129sleep39866-21systemctl20:38:523
19491219816,137sleep319590-21latency_hist00:33:553
31926219613,64sleep30-21swapper/323:29:073
23628219210,166sleep023711-21sed21:19:140
1192021879,154sleep00-21swapper/000:09:040
11732217913,150sleep111880-21latency_hist20:43:551
31240217512,122sleep331332-21fschecks_time21:44:163
31240217512,122sleep331332-21fschecks_time21:44:163
9325217122,123sleep07881-1kworker/0:1H22:18:520
19913217017,134sleep219713-21latency_hist21:08:552
2339216625,130sleep32386-21grep20:14:103
27001216551,62sleep30-21swapper/323:14:093
20551216542,106sleep30-21swapper/322:53:553
21036216411,137sleep30-21swapper/322:54:153
31403215845,98sleep331424-21awk20:03:523
1678421577,25sleep30-21swapper/319:14:243
99321539,133sleep31013-21dpkg23:34:033
31525215341,62sleep30-21swapper/323:28:543
31525215341,62sleep30-21swapper/323:28:543
215021539,126sleep32193-21ntp_kernel_pll_21:54:173
215021539,126sleep32193-21ntp_kernel_pll_21:54:173
11934215219,55sleep30-21swapper/320:44:053
752214528,17sleep30-21swapper/320:08:573
752214528,17sleep30-21swapper/320:08:573
5446214228,95sleep05465-21apt-get20:24:040
3232214212,115sleep03360-21sed23:39:250
10180214215,61sleep30-21swapper/300:03:543
28100214116,81sleep00-21swapper/021:34:110
6181214015,68sleep36245-21mailstats23:49:213
25828213843,56sleep325873-21ntpq23:09:173
20800213711,114sleep320889-21cat21:09:243
20285213617,61sleep30-21swapper/300:34:233
7569213114,54sleep30-21swapper/322:11:573
335421318,33sleep30-21swapper/321:59:093
7640212910,106sleep27641-21date22:12:362
385321299,102sleep33879-21awk23:43:523
1810921299,105sleep318267-21sed22:44:263
1810921299,105sleep318267-21sed22:44:263
1036721297,21sleep30-21swapper/300:04:023
8591212856,20sleep30-21swapper/323:58:543
13986212816,49sleep30-21swapper/320:49:223
13986212816,49sleep30-21swapper/320:49:223
9732212213,95sleep09792-21fschecks_time22:19:120
22915212215,18sleep10-21swapper/122:59:221
25225211840,63sleep325290-21date19:43:543
13905211543,32sleep20-21swapper/219:08:542
13905211543,32sleep20-21swapper/219:08:542
23895211311,22sleep20-21swapper/219:39:052
19159211010,81sleep118697-21apt-get22:48:561
17141211012,60sleep30-21swapper/300:24:223
7615210912,73sleep27220-21/usr/sbin/munin23:54:162
27541210912,22sleep127605-21sensors23:14:231
685210812,80sleep2694-21awk21:49:192
685210812,80sleep2694-21awk21:49:192
1397321079,18sleep30-21swapper/319:08:553
1397321079,18sleep30-21swapper/319:08:553
29928210613,52sleep330080-21latency_hist19:58:563
28636210530,16sleep30-21swapper/321:34:233
22317210412,50sleep30-21swapper/322:59:033
20501210444,16sleep30-21swapper/319:28:533
18210210427,62sleep318361-21date21:03:553
1022121048,16sleep30-21swapper/322:19:233
590210332,58sleep3629-21ntpq21:49:163
590210332,58sleep3629-21ntpq21:49:163
13944210324,28sleep313955-21ntpq00:14:203
27691210113,71sleep00-21swapper/019:49:240
19681210112,19sleep30-21swapper/321:08:523
19186210118,57sleep00-21swapper/022:49:030
494210045,18sleep31-21systemd20:08:533
312562995,13sleep20-21swapper/220:01:582
719229817,65sleep07236-21sort20:29:120
442629825,17sleep30-21swapper/320:19:223
2404729841,18sleep30-21swapper/323:04:123
87329720,62sleep0906-21grep20:09:100
87329720,62sleep0906-21grep20:09:100
78829722,60sleep1855-21ls20:09:071
78829722,60sleep1855-21ls20:09:071
733329726,48sleep17373-21sort23:54:111
998829616,61sleep110121-21sed20:38:541
81362969,44sleep30-21swapper/322:14:113
296729614,65sleep12990-21/usr/sbin/munin23:39:181
1105529524,59sleep311143-21cat20:40:093
57422946,34sleep30-21swapper/320:24:153
2432229411,64sleep024364-21/usr/sbin/munin19:39:190
2654429318,54sleep026623-21fschecks_time21:29:110
81662928,72sleep00-21swapper/023:54:310
2177329214,66sleep321777-21grep21:14:083
414329125,38sleep14140-21apt-get23:44:061
1266229127,49sleep312679-21sort19:04:113
1266229127,49sleep312679-21sort19:04:113
2858529015,55sleep00-21swapper/023:19:070
2771529025,42sleep127740-21munin-run21:33:521
2233628823,49sleep30-21swapper/319:34:053
2458328718,47sleep224746-21grep23:04:232
173652879,60sleep017445-21/usr/sbin/munin20:59:240
1525728713,54sleep014772-21apt-get20:54:030
842628614,54sleep18451-21awk22:14:171
2912228619,25sleep20-21swapper/219:54:232
940528515,55sleep29346-21latency_hist22:18:542
3002328513,16sleep20-21swapper/221:39:182
3002328513,16sleep20-21swapper/221:39:182
2555228514,58sleep125579-21fschecks_count19:44:131
159342850,8sleep00-21swapper/022:39:090
1049228518,46sleep110584-21fschecks_time00:04:131
1662628417,33sleep20-21swapper/220:58:532
1508028431,36sleep30-21swapper/300:19:053
752728319,31sleep10-21swapper/120:29:211
279592839,64sleep00-21swapper/023:15:030
233528320,31sleep00-21swapper/020:14:070
1325228217,45sleep113339-21latency_hist00:13:541
2344428117,46sleep123523-21fschecks_time21:19:111
2199628116,48sleep122041-21irqstats21:14:151
1779728114,19sleep00-21swapper/022:44:190
1779728114,19sleep00-21swapper/022:44:190
1714328011,51sleep217237-21/usr/sbin/munin20:59:182
545027911,22sleep00-21swapper/023:48:540
2776527915,41sleep30-21swapper/321:33:543
695327829,30sleep20-21swapper/223:53:522
1586727816,42sleep215942-21ntp_kernel_pll_19:13:582
111662786,12sleep30-21swapper/322:24:043
2502727718,28sleep20-21swapper/221:24:122
1453727718,40sleep114568-21/usr/sbin/munin19:09:171
1453727718,40sleep114568-21/usr/sbin/munin19:09:171
1297827734,23sleep10-21swapper/119:04:171
1297827734,23sleep10-21swapper/119:04:171
54062769,14sleep30-21swapper/323:48:533
3079827629,12sleep30-21swapper/321:43:193
3079827629,12sleep30-21swapper/321:43:193
2110727618,26sleep20-21swapper/200:38:542
2085427615,40sleep020896-21fschecks_count22:54:120
1869627616,16sleep00-21swapper/021:04:160
1829927622,23sleep10-21swapper/119:19:221
1014627616,46sleep210179-21latency_hist00:03:532
2820327529,12sleep30-21swapper/319:52:583
1924327518,29sleep10-21swapper/119:24:061
3233127415,43sleep131905-21/usr/sbin/munin23:29:451
285572740,8sleep20-21swapper/221:34:212
2658527430,11sleep30-21swapper/319:47:053
2366027427,29sleep223786-21cat23:03:552
1283727414,36sleep212916-21fschecks_time22:29:152
1634527316,43sleep116360-21open_inodes22:39:241
1627127329,10sleep30-21swapper/320:55:193
1294627316,16sleep00-21swapper/019:04:160
1294627316,16sleep00-21swapper/019:04:160
1292327335,10sleep30-21swapper/300:10:163
119042738,52sleep00-21swapper/020:43:580
1036527312,43sleep29849-21apt-get20:39:032
1952927212,25sleep20-21swapper/219:24:162
1911027214,25sleep20-21swapper/221:04:272
1547227221,30sleep215486-21munin-node19:13:542
1547227221,30sleep215486-21munin-node19:13:542
1438727218,21sleep20-21swapper/222:34:122
678327120,23sleep20-21swapper/220:28:542
2826627118,34sleep2302-21systemd-logind23:18:522
2592927114,44sleep225966-21ps19:44:222
2488127121,26sleep10-21swapper/121:24:031
2117027118,37sleep221219-21ntp_states22:54:182
1789127115,38sleep117920-21munin-run00:28:521
1918227012,43sleep218697-21apt-get22:49:012
500626915,23sleep20-21swapper/222:04:132
1508726920,26sleep10-21swapper/100:19:071
2965426814,36sleep129663-21/usr/sbin/munin21:39:091
2965426814,36sleep129663-21/usr/sbin/munin21:39:091
2870526823,24sleep10-21swapper/123:19:121
1980126819,22sleep10-21swapper/121:08:541
1182126822,23sleep20-21swapper/222:24:232
3177826715,22sleep20-21swapper/221:44:272
3177826715,22sleep20-21swapper/221:44:272
2204226714,28sleep20-21swapper/221:14:162
836426617,29sleep28311-21latency_hist20:33:552
3189826611,29sleep031411-21apt-get23:29:040
299622669,12sleep00-21swapper/023:23:550
299622669,12sleep00-21swapper/023:23:550
1364126619,29sleep113703-21/usr/sbin/munin00:14:141
693626515,28sleep10-21swapper/122:09:201
3206126518,23sleep20-21swapper/220:04:162
1690526514,20sleep00-21swapper/000:24:150
1193826522,23sleep20-21swapper/220:44:082
578326414,32sleep15788-21/usr/sbin/munin23:49:111
311226421,20sleep00-21swapper/021:58:540
311226421,20sleep00-21swapper/021:58:540
2548326419,24sleep00-21swapper/021:24:230
249022646,49sleep00-21swapper/023:05:340
1879326412,39sleep018886-21cat00:29:230
1809726419,27sleep00-21swapper/019:19:170
179226421,23sleep00-21swapper/023:34:250
1370426420,20sleep20-21swapper/200:14:152
883926313,32sleep18889-21/usr/sbin/munin23:59:081
624926314,35sleep06326-21switchtime23:49:220
545926317,28sleep15527-21/usr/sbin/munin22:04:241
1797826316,29sleep218062-21/usr/sbin/munin19:19:152
1763726318,26sleep10-21swapper/122:44:161
1763726318,26sleep10-21swapper/122:44:161
3034926220,20sleep10-21swapper/119:59:141
251426221,18sleep20-21swapper/221:54:242
251426221,18sleep20-21swapper/221:54:242
2125226224,21sleep20-21swapper/219:29:202
1630626220,18sleep10-21swapper/119:14:141
546826112,26sleep10-21swapper/120:24:051
299512616,12sleep00-21swapper/021:39:160
299512616,12sleep00-21swapper/021:39:160
2908026116,28sleep229158-21/usr/sbin/munin23:19:212
2390026115,27sleep123894-21apt-get23:04:061
23872618,12sleep10-21swapper/121:54:231
23872618,12sleep10-21swapper/121:54:231
2252726113,34sleep022590-21if_eth022:59:130
2089926119,23sleep10-21swapper/122:54:131
145712619,12sleep10-21swapper/122:34:151
1218226114,29sleep212216-21/usr/sbin/munin00:09:142
681126014,30sleep06829-21/usr/sbin/munin22:09:150
236326019,16sleep20-21swapper/223:38:552
2153226013,31sleep121651-21latency_hist21:13:541
1817426013,29sleep10-21swapper/121:03:531
1639626016,21sleep00-21swapper/000:23:540
1472126013,30sleep014754-21ntp_states22:34:180
2572825916,20sleep00-21swapper/019:44:160
242325913,24sleep20-21swapper/220:14:132
226062596,11sleep00-21swapper/019:34:150
1973125916,26sleep019725-21apt-get00:34:080
1620625919,19sleep00-21swapper/019:14:130
1348925914,27sleep013483-21apt-get20:49:070
1348925914,27sleep013483-21apt-get20:49:070
342525817,20sleep10-21swapper/121:59:111
2372725813,30sleep223774-21/usr/sbin/munin21:19:162
2079125813,26sleep120785-21apt-get19:29:071
1599325816,19sleep10-21swapper/120:55:091
152902585,11sleep00-21swapper/019:12:070
152902585,11sleep00-21swapper/019:12:070
1508825812,31sleep215082-21apt-get00:19:082
1349425814,18sleep00-21swapper/000:14:090
864425715,24sleep08651-21df_inode20:34:090
528225713,28sleep05427-21sed20:23:560
3169325716,25sleep031688-21apt-get20:04:060
128225711,32sleep21362-21sendmail_mailtr20:09:202
128225711,32sleep21362-21sendmail_mailtr20:09:202
902525619,20sleep20-21swapper/220:34:192
581025612,22sleep00-21swapper/022:04:270
382225619,19sleep00-21swapper/021:59:220
289642568,35sleep10-21swapper/121:35:011
1839025613,26sleep218430-21if_eth000:29:132
153325614,21sleep00-21swapper/021:53:530
153325614,21sleep00-21swapper/021:53:530
928825515,18sleep00-21swapper/023:59:190
3070825516,18sleep20-21swapper/223:24:212
3070825516,18sleep20-21swapper/223:24:212
2751825515,22sleep20-21swapper/219:49:222
217362557,36sleep00-21swapper/021:13:570
1298125514,23sleep013017-21irqstats22:29:180
1174425517,20sleep10-21swapper/122:24:221
1039525513,25sleep010388-21apt-get00:04:080
964125413,23sleep19639-21apt-get22:19:051
360225413,27sleep13625-21munin-run20:18:521
3214925412,18sleep10-21swapper/120:04:191
3171125417,21sleep131777-21sed21:44:261
3171125417,21sleep131777-21sed21:44:261
2747825419,19sleep10-21swapper/119:49:211
2031225420,20sleep00-21swapper/021:09:130
1977325416,17sleep00-21swapper/019:24:220
1394825414,23sleep113954-21ntp_states20:49:201
1394825414,23sleep113954-21ntp_states20:49:201
392425319,17sleep20-21swapper/220:19:092
2856025313,24sleep128557-21apt-get19:54:061
2079025316,21sleep00-21swapper/019:29:060
1769525316,17sleep20-21swapper/222:44:172
1769525316,17sleep20-21swapper/222:44:172
1098625313,18sleep20-21swapper/222:23:542
1043925313,19sleep00-21swapper/020:39:460
2687325213,24sleep226909-21ntp_states21:29:172
2543725213,22sleep225445-21apt-get23:09:032
234825214,23sleep12362-21latency_hist23:38:531
139622526,33sleep10-21swapper/122:33:181
3139425113,21sleep10-21swapper/123:28:521
3139425113,21sleep10-21swapper/123:28:521
2497525113,22sleep10-21swapper/119:41:181
233112515,33sleep10-21swapper/119:35:131
2234025110,17sleep20-21swapper/219:34:092
1395525117,19sleep20-21swapper/220:49:212
1395525117,19sleep20-21swapper/220:49:212
518125011,15sleep10-21swapper/120:23:521
1118925012,22sleep011185-21apt-get22:24:050
1817524910,22sleep118177-21apt-get00:29:041
3268024812,17sleep10-21swapper/121:49:061
3268024812,17sleep10-21swapper/121:49:061
258762489,16sleep00-21swapper/023:09:180
1596524810,20sleep215967-21diskstats22:39:122
1193724811,22sleep111931-21apt-get20:44:071
1129824813,16sleep20-21swapper/200:04:262
457224511,19sleep24604-21ntp_states23:44:192
301332458,13sleep00-21swapper/019:59:050
379724111,16sleep20-21swapper/221:59:202
308324010,17sleep20-21swapper/223:39:232
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional