You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-16 - 10:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Mon Mar 16, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29041283213,802sleep331062-1kworker/3:1H22:38:153
19226276711,741sleep219373-21cat22:08:062
19226276711,741sleep219373-21cat22:08:062
11450247419,405sleep211495-21latency_hist21:43:062
5184243139,291sleep25270-21latency_hist21:23:072
3810229718,221sleep23957-21unixbench_multi19:33:402
5693228221,145sleep25787-21latency_hist23:08:052
331223750,61sleep20-21swapper/222:48:362
13917222437,71sleep214065-21unixbench_multi21:48:352
13917222437,71sleep214065-21unixbench_multi21:48:352
8530221813,24sleep08546-21ntp_states19:49:130
21971221531,62sleep20-21swapper/220:33:072
18376221413,187sleep018408-21/usr/sbin/munin22:03:320
18376221413,187sleep018408-21/usr/sbin/munin22:03:320
27251218813,48sleep20-21swapper/200:13:352
177621830,12sleep00-21swapper/019:28:240
23809217935,107sleep223883-21cat00:03:262
28009217716,143sleep028157-21ls00:18:060
32756217423,132sleep3387-21/usr/sbin/munin19:23:273
10146217432,122sleep29515-21/usr/sbin/munin23:18:352
10146217432,122sleep29515-21/usr/sbin/munin23:18:352
19596217032,76sleep219747-21unixbench_singl23:48:362
30146216915,138sleep030173-21awk00:23:270
29221216725,62sleep129368-21unixbench_singl20:53:341
25666216210,139sleep325678-21kernelversion20:43:303
410216110,39sleep20-21swapper/200:33:182
1510721617,14sleep20-21swapper/221:53:322
1510721617,14sleep20-21swapper/221:53:322
29435215717,118sleep029469-21fschecks_count19:13:230
11207215612,105sleep211295-21fschecks_time23:23:242
16368215116,78sleep216476-21wc23:38:342
8554215015,114sleep20-21swapper/219:49:152
5965215030,108sleep21-21systemd19:43:052
22354215014,119sleep122297-21latency_hist22:18:061
30079214821,106sleep030132-21ls20:58:170
9554214753,42sleep29561-21acpi19:53:152
29192214314,102sleep20-21swapper/222:39:032
17390214312,53sleep20-21swapper/220:18:182
24603214267,59sleep124643-21/usr/sbin/munin22:23:321
17119214216,79sleep20-21swapper/220:18:052
1560721410,10sleep30-21swapper/321:53:403
1560721410,10sleep30-21swapper/321:53:403
2738721408,18sleep20-21swapper/220:48:302
13003214013,111sleep213156-21latency_hist21:48:072
31602213931,95sleep231726-21cat19:18:342
25086213737,85sleep225151-21ls00:08:182
27970213547,69sleep228115-21unixbench_singl22:33:342
32297213410,108sleep332441-21cat19:23:073
18409213413,18sleep10-21swapper/122:03:331
18409213413,18sleep10-21swapper/122:03:331
8610213336,46sleep28739-21ls23:13:592
11424213344,75sleep211435-21kernelversion19:58:272
18606213215,102sleep01-21systemd23:48:050
7238212712,96sleep20-21swapper/221:28:282
296121268,102sleep20-21swapper/221:13:382
25884212624,42sleep20-21swapper/222:28:252
30370212516,94sleep130433-21cut22:43:061
28714212514,16sleep20-21swapper/200:18:322
20066212513,18sleep20-21swapper/222:08:342
20066212513,18sleep20-21swapper/222:08:342
31409212324,16sleep20-21swapper/222:43:352
20403212358,25sleep20-21swapper/223:53:202
160221218,19sleep00-21swapper/000:37:080
31610212015,87sleep031612-21/usr/sbin/munin21:03:140
1337212020,48sleep20-21swapper/219:28:072
23535211912,94sleep223599-21sendmail-msp20:38:042
13764211953,18sleep20-21swapper/223:31:232
25788211743,38sleep20-21swapper/219:03:162
25788211743,38sleep20-21swapper/219:03:162
21658211624,68sleep121721-21modprobe23:58:041
5955211519,74sleep1222-21systemd-journal19:43:041
32135211556,46sleep232213-21mailstats21:03:322
28677211560,37sleep20-21swapper/220:53:232
17381211419,37sleep20-21swapper/223:43:232
15520211311,90sleep215563-21sh20:13:052
3286211213,77sleep10-21swapper/122:58:321
27446211221,74sleep127529-21grep20:48:331
7958211112,84sleep17943-21apt-get23:13:211
159321106,51sleep20-21swapper/221:08:372
12399211019,23sleep00-21swapper/023:28:060
13178210948,20sleep213243-21awk20:03:322
32228210710,81sleep032236-21df22:48:220
1658210710,56sleep20-21swapper/219:28:232
7941210626,62sleep37450-21apt-get23:13:153
22245210622,65sleep121761-21apt-get20:33:151
10851210614,77sleep110970-21latency_hist23:23:071
10851210614,77sleep110970-21latency_hist23:23:071
30082210518,61sleep130224-21head19:13:361
26924210328,47sleep026943-21apt-get20:48:160
6422210119,42sleep20-21swapper/221:24:452
512721019,16sleep30-21swapper/319:38:303
22678210138,18sleep20-21swapper/222:18:242
17840210116,60sleep30-21swapper/320:18:313
1767029945,18sleep217706-21sed22:03:052
1767029945,18sleep217706-21sed22:03:052
72472987,10sleep20-21swapper/219:44:472
661429819,56sleep16219-21/usr/sbin/munin19:43:281
322372965,11sleep20-21swapper/200:28:382
251862966,10sleep20-21swapper/222:25:222
190122967,45sleep20-21swapper/220:23:222
1820629511,70sleep318281-21memory22:03:263
1820629511,70sleep318281-21memory22:03:263
1298929513,68sleep113001-21latency_hist21:48:061
12412957,40sleep21243-21apt-get22:53:162
817729422,49sleep18003-21/usr/sbin/munin19:49:061
305742946,11sleep20-21swapper/219:17:292
1477129418,57sleep014862-21mailstats20:08:350
142762949,32sleep20-21swapper/223:33:252
282032937,31sleep20-21swapper/219:08:362
2607829322,32sleep00-21swapper/019:03:260
2607829322,32sleep00-21swapper/019:03:260
2227229315,33sleep222274-21cron22:18:032
841629113,59sleep08430-21/usr/sbin/munin23:13:530
3108429115,39sleep20-21swapper/220:59:372
132729117,59sleep11-21systemd19:28:061
757428914,57sleep27749-21sed23:13:062
613528919,47sleep36144-21/usr/sbin/munin23:08:153
2976928914,55sleep329805-21ntp_states19:13:303
49728814,56sleep3576-21sshd00:33:233
3058028719,24sleep10-21swapper/120:58:311
170642877,66sleep117108-21latency_hist23:43:051
903928511,23sleep20-21swapper/221:33:332
1762728528,40sleep1281-21dbus-daemon22:03:041
1762728528,40sleep1281-21dbus-daemon22:03:041
266028413,53sleep32689-21/usr/sbin/munin21:13:313
2529228418,46sleep325059-21/usr/sbin/munin00:08:233
1331628414,54sleep013402-21timerandwakeup20:03:340
2285528315,50sleep122868-21uname22:18:271
110382836,60sleep111049-21vmstat21:38:371
2858328219,46sleep128600-21systemd-run22:38:051
1370328212,49sleep313210-21/usr/sbin/munin21:48:323
1370328212,49sleep313210-21/usr/sbin/munin21:48:323
1177128120,41sleep311783-21sshd21:43:233
2977428016,34sleep10-21swapper/100:23:161
2426828020,35sleep324411-21sed00:03:353
1894228011,48sleep10-21swapper/120:23:211
1325228011,43sleep012611-21/usr/sbin/munin23:28:350
131828014,48sleep11330-21vmstat00:33:381
758827823,27sleep10-21swapper/121:28:391
445627814,30sleep00-21swapper/023:03:230
2748527818,37sleep127501-21fschecks_count19:08:231
2694427813,21sleep00-21swapper/022:33:030
2516927820,37sleep125203-21fschecks_count00:08:211
2422227810,54sleep024249-21fschecks_count22:23:220
217152786,12sleep00-21swapper/020:32:370
1707627821,40sleep117118-21idleruntime-cro20:18:041
2634927716,44sleep326425-21/usr/sbin/munin19:03:333
2634927716,44sleep326425-21/usr/sbin/munin19:03:333
535227619,36sleep05477-21ls19:38:350
254022769,55sleep225394-21apt-get20:43:212
303152755,59sleep00-21swapper/022:43:010
258727519,32sleep00-21swapper/022:58:060
2118527512,47sleep321243-21if_err_eth022:13:243
1856427510,16sleep30-21swapper/323:48:043
569227415,26sleep10-21swapper/121:23:261
2619227411,50sleep013441-21snmpd00:11:070
2088927413,48sleep320970-21mailstats23:53:333
856327317,38sleep08557-21apt-get21:33:190
304427321,12sleep20-21swapper/222:58:252
2174827312,44sleep321791-21gpgv20:33:043
624527211,38sleep00-21swapper/019:43:190
597427213,41sleep06049-21switchtime21:23:330
2617427216,29sleep30-21swapper/322:28:343
2392727219,13sleep20-21swapper/220:38:202
2211127237,18sleep20-21swapper/223:58:252
1730527218,27sleep00-21swapper/021:59:030
1730527218,27sleep00-21swapper/021:59:030
1083927222,31sleep310863-21munin-run19:58:043
95082716,11sleep00-21swapper/023:18:090
95082716,11sleep00-21swapper/023:18:090
414127119,24sleep30-21swapper/323:03:063
2770927115,44sleep327710-21sshd00:15:303
1594827112,14sleep00-21swapper/023:38:240
2266327018,32sleep022568-21/usr/sbin/munin22:18:220
1992227024,26sleep319941-21open_inodes22:08:313
1992227024,26sleep319941-21open_inodes22:08:313
1894027013,26sleep00-21swapper/020:23:200
804626922,30sleep38021-21apt-get19:48:273
764726912,38sleep37861-21apt-config19:48:073
339526914,32sleep10-21swapper/119:33:311
2748126915,32sleep027510-21irqstats22:33:240
204842696,11sleep00-21swapper/022:10:020
204842696,11sleep00-21swapper/022:10:020
48526820,20sleep00-21swapper/019:23:320
325126820,24sleep30-21swapper/319:33:293
2655626821,22sleep10-21swapper/119:03:351
2655626821,22sleep10-21swapper/119:03:351
2483826814,38sleep124848-21munin-run00:08:051
2015426820,25sleep30-21swapper/323:53:053
1585226815,38sleep315946-21sshd23:38:233
1172726819,27sleep10-21swapper/123:23:351
742726721,29sleep07445-21/usr/sbin/munin23:13:040
578926719,24sleep30-21swapper/323:08:073
3070526715,40sleep20-21swapper/200:23:492
286826714,23sleep10-21swapper/121:13:371
194592679,39sleep10-21swapper/122:08:171
194592679,39sleep10-21swapper/122:08:171
1776226713,35sleep117793-21/usr/sbin/munin20:18:291
1519826716,20sleep00-21swapper/021:53:340
1519826716,20sleep00-21swapper/021:53:340
2878226621,27sleep028905-21if_err_eth019:13:070
203726622,19sleep30-21swapper/319:28:323
1544026618,30sleep015494-21sendmail23:38:040
1149926610,38sleep311644-21sed21:43:073
2540626512,22sleep10-21swapper/120:43:241
1485026522,20sleep10-21swapper/123:33:371
1248326520,30sleep112635-21latency_hist20:03:071
1124126520,23sleep30-21swapper/319:58:243
990126417,18sleep30-21swapper/321:38:063
928526417,24sleep10-21swapper/121:33:361
321562647,14sleep30-21swapper/322:48:213
2976426414,26sleep30-21swapper/320:58:043
2780026419,26sleep30-21swapper/319:08:293
2017326413,31sleep020317-21latency_hist23:53:060
1091926416,20sleep10-21swapper/119:58:061
579226318,29sleep35839-21sed21:23:283
3221626315,25sleep30-21swapper/321:03:333
317572635,11sleep10-21swapper/121:03:231
2811926318,24sleep30-21swapper/322:33:353
2256726319,24sleep00-21swapper/020:33:420
2401626219,22sleep024080-21find00:03:320
324272618,41sleep30-21swapper/300:31:053
2953926118,22sleep30-21swapper/300:23:053
2741226116,26sleep327419-21ntp_states20:48:313
229772616,17sleep00-21swapper/023:59:320
201726116,27sleep12029-21ntp_states19:28:301
2188126017,21sleep10-21swapper/120:33:061
199262605,45sleep30-21swapper/320:24:333
1777626011,14sleep10-21swapper/123:43:341
1423526023,21sleep30-21swapper/323:33:223
1389326019,25sleep113917-21munin-run23:33:041
1279526019,22sleep10-21swapper/120:03:231
123112607,11sleep00-21swapper/020:01:360
757725915,24sleep00-21swapper/021:28:380
2922825911,36sleep013441-21snmpd00:19:430
279325914,28sleep02867-21/usr/sbin/munin21:13:360
1626925916,28sleep016275-21ntp_states20:13:350
101325917,19sleep30-21swapper/321:08:263
996025813,28sleep010009-21ntpq19:53:290
445925814,27sleep04574-21sed19:38:060
3151225820,17sleep10-21swapper/119:18:331
1506025819,15sleep10-21swapper/121:53:311
1506025819,15sleep10-21swapper/121:53:311
145322586,13sleep10-21swapper/120:08:281
1229025817,22sleep00-21swapper/021:43:370
563825720,20sleep10-21swapper/123:08:041
3068325713,28sleep330687-21diskstats22:43:203
2831525714,24sleep128410-21perl00:18:231
2549125714,19sleep00-21swapper/020:43:270
2511025716,26sleep125114-21modprobe20:43:041
2386425718,26sleep323899-21munin-run22:23:053
1454625720,25sleep014508-21latency_hist21:53:060
1454625720,25sleep014508-21latency_hist21:53:060
1428825711,33sleep114289-21sshd21:50:241
1428825711,33sleep114289-21sshd21:50:241
3213025614,24sleep032228-21users00:28:360
3182725616,19sleep10-21swapper/100:28:331
307262565,15sleep30-21swapper/300:24:063
2174725613,20sleep00-21swapper/023:58:070
184582566,38sleep00-21swapper/020:20:340
1715525615,20sleep30-21swapper/320:18:063
1291825615,27sleep312975-21interrupts20:03:253
74325514,25sleep0737-21apt-get21:08:190
697125513,25sleep36963-21apt-get21:28:213
2877325514,17sleep00-21swapper/020:53:250
2393425518,20sleep30-21swapper/320:38:213
2198825512,25sleep121982-21apt-get23:58:201
1507325512,24sleep315165-21users20:08:383
125125514,24sleep31243-21apt-get22:53:213
3093125414,24sleep030927-21apt-get19:18:150
2250825414,25sleep322565-21irqstats20:33:413
660825313,22sleep16631-21ntp_states23:08:371
3215425311,25sleep132148-21apt-get22:48:191
2567925313,23sleep125680-21apt-get22:28:161
2544325320,16sleep30-21swapper/322:28:063
1139925314,24sleep011427-21munin-run21:43:040
963025212,16sleep30-21swapper/323:18:243
963025212,16sleep30-21swapper/323:18:243
3121925212,23sleep331220-21sshd21:01:233
2671525214,22sleep126719-21diskstats00:13:221
172025214,17sleep10-21swapper/122:53:351
1656925213,24sleep316622-21/usr/sbin/munin21:58:263
1656925213,24sleep316622-21/usr/sbin/munin21:58:263
135592529,33sleep30-21swapper/323:28:463
305962515,13sleep10-21swapper/122:43:141
2219825115,16sleep30-21swapper/323:58:263
1157425113,24sleep311636-21mailstats23:23:333
88425013,16sleep10-21swapper/121:08:241
448025015,18sleep10-21swapper/121:18:341
411625012,20sleep04145-21irqstats21:18:250
2141524911,14sleep00-21swapper/022:13:290
198772495,32sleep10-21swapper/123:50:041
3087824816,17sleep00-21swapper/022:43:250
2382824513,16sleep00-21swapper/020:38:140
279724411,17sleep00-21swapper/022:58:150
48252435,29sleep30-21swapper/321:19:083
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional