You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-17 - 05:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Tue Mar 17, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
173852602714,5994sleep02570-21kworker/0:100:13:220
2905427567,739sleep28005-1kworker/2:2H23:08:202
915825588,534sleep07255-1kworker/0:0H20:23:140
3226025067,486sleep030561-21kworker/0:121:35:370
12510238210,357sleep312618-21fschecks_time23:58:233
19605238112,354sleep219679-21find22:38:322
19605238112,354sleep219679-21find22:38:322
24716233812,311sleep124723-21ntpq19:28:291
31702302104,174sleep33256-21latency_hist21:48:033
31702302104,174sleep33256-21latency_hist21:48:033
26728228982,102sleep326749-21apt23:03:003
6978224014,210sleep16989-21ntpq21:58:271
26446222717,193sleep126573-21timerwakeupswit19:33:331
1273922279,203sleep312804-21mailstats20:33:303
18947221215,181sleep119001-21sort00:18:151
22657220744,96sleep30-21swapper/300:28:313
2819122000,6sleep20-21swapper/221:23:182
8630219430,150sleep38658-21awk20:23:013
22046218614,159sleep022048-21grep00:28:150
32519218596,73sleep30-21swapper/321:38:023
18699218389,68sleep30-21swapper/300:18:013
21613217811,145sleep30-21swapper/321:03:023
223412173128,31sleep322404-21mailstats21:03:303
9303216158,89sleep39336-21fschecks_count20:23:453
24272215111,40sleep324420-21grep00:33:313
13013215115,119sleep113091-21find23:58:331
1033214819,26sleep31094-21latency_hist23:23:023
18876214654,37sleep318895-21modprobe19:13:003
7799214011,112sleep37847-21systemd22:03:013
12880213816,107sleep113009-21sort20:33:321
512721379,66sleep35170-21ls21:53:203
512721379,66sleep35170-21ls21:53:203
12532213619,20sleep30-21swapper/322:18:033
10967213597,21sleep30-21swapper/320:28:223
27246213450,64sleep30-21swapper/323:03:123
23246213432,25sleep30-21swapper/322:48:433
23246213432,25sleep30-21swapper/322:48:433
29354213177,40sleep30-21swapper/321:28:013
14869213115,50sleep314943-21sort22:23:323
17947213036,33sleep318076-21awk20:48:323
12808213023,20sleep00-21swapper/020:33:310
16596212913,97sleep316742-21head22:28:323
24780212815,20sleep30-21swapper/319:28:313
17999212614,95sleep10-21swapper/100:13:391
9854212414,86sleep30-21swapper/323:48:303
4315212413,94sleep34327-21cron23:33:003
30987212431,22sleep30-21swapper/323:13:263
21805212435,21sleep30-21swapper/300:28:023
16116212441,62sleep216133-21uniq19:03:192
26549212249,61sleep326554-21cpu21:18:163
2450211981,16sleep30-21swapper/320:03:023
23311211858,17sleep30-21swapper/319:23:303
10777211714,86sleep210786-21df_inode20:28:172
3491211410,41sleep33579-21tune2fs21:48:193
3491211410,41sleep33579-21tune2fs21:48:193
26232211426,64sleep026256-21apt21:18:000
22308211418,77sleep021824-21apt-get22:48:110
22308211418,77sleep021824-21apt-get22:48:110
2780211317,80sleep32872-21fschecks_time20:03:193
2967121128,92sleep329748-21xargs23:08:453
24389211218,76sleep20-21swapper/222:53:302
24389211218,76sleep20-21swapper/222:53:302
4328211117,73sleep04386-21fschecks_count20:08:200
21922211113,28sleep30-21swapper/322:48:023
12171211111,86sleep312303-21latency_hist23:58:033
21836211013,75sleep10-21swapper/119:18:311
17887210946,18sleep30-21swapper/322:33:293
27711210714,76sleep027753-21ntp_offset23:04:060
13836210710,40sleep313827-21apt-get20:38:193
13836210710,40sleep313827-21apt-get20:38:193
30693210611,81sleep230695-21sshd21:30:222
14235210611,78sleep014329-21/usr/sbin/munin00:03:210
13038210614,74sleep213132-21memory22:18:222
29727210423,62sleep029791-21fschecks_time21:28:180
5219210312,76sleep15278-21latency21:53:221
5219210312,76sleep15278-21latency21:53:221
6919210212,74sleep37074-21awk23:38:333
1952521026,15sleep30-21swapper/300:18:303
20312210111,74sleep020430-21date22:43:020
20312210111,74sleep020430-21date22:43:020
19086210110,75sleep30-21swapper/322:38:213
19086210110,75sleep30-21swapper/322:38:213
1189629914,70sleep112038-21/usr/sbin/munin22:13:321
3276629815,62sleep132741-21/usr/sbin/munin21:38:161
1712529812,71sleep017175-21unin-run22:33:010
40029616,57sleep0542-21unixbench_multi23:18:310
3069029617,62sleep230789-21cut23:13:212
1208229648,30sleep0281-21dbus-daemon23:58:000
74562955,11sleep20-21swapper/220:18:192
198162956,55sleep30-21swapper/320:55:513
45792949,72sleep30-21swapper/323:33:123
254029416,56sleep02690-21unixbench_multi21:43:320
254029416,56sleep02690-21unixbench_multi21:43:320
2721429313,65sleep227343-21sed21:18:312
1613729322,29sleep20-21swapper/222:28:232
205222926,73sleep30-21swapper/319:14:013
164432925,75sleep30-21swapper/300:08:453
1225629113,61sleep212326-21/usr/sbin/munin20:33:172
49762906,66sleep00-21swapper/023:33:250
2899929010,65sleep129006-21sh19:43:181
2063529017,59sleep020642-21forks22:43:190
1563129019,52sleep215605-21apt-get20:43:272
1031429014,62sleep010444-21unixbench_multi22:08:310
1698228911,62sleep217176-21latency_hist00:13:032
90428814,54sleep0991-21grep19:58:030
904428715,55sleep09142-21latency_hist23:48:020
82428623,42sleep2855-21munin-plugin-st19:58:012
2776828619,45sleep227771-21ntp_states23:04:082
2628928620,47sleep30-21swapper/319:33:313
2317928614,54sleep123239-21/usr/sbin/munin19:23:281
132292867,67sleep313263-21ps22:18:273
500428510,61sleep15042-21ntp_kernel_pll_23:33:261
1453728510,56sleep30-21swapper/300:03:283
102328513,55sleep21032-21unin-run23:23:012
936728420,28sleep10-21swapper/123:48:201
85112848,16sleep30-21swapper/322:03:253
432228411,57sleep30-21swapper/320:08:193
800428319,46sleep28021-21kernelversion23:43:242
62902838,64sleep35765-21/usr/sbin/munin20:13:323
319328319,44sleep13199-21cut23:28:291
3095328327,28sleep10-21swapper/121:33:031
2472728311,27sleep30-21swapper/321:13:033
2059228311,58sleep120687-21grep00:23:201
3035828210,46sleep030459-21sed23:13:020
101002829,59sleep110137-21ps22:08:281
27672810,7sleep10-21swapper/120:03:151
1440328117,43sleep114466-21cut22:23:211
65302807,52sleep20-21swapper/221:58:152
277542800,9sleep10-21swapper/123:04:061
2349728016,51sleep223533-21expr21:08:172
781927943,18sleep30-21swapper/323:43:213
2155227911,11sleep30-21swapper/319:18:253
625327812,52sleep26287-21ps20:13:312
2184827816,44sleep221921-21cmp22:48:012
1557927852,15sleep315103-21apt-get20:43:073
328527717,28sleep00-21swapper/023:28:330
266227717,39sleep22854-21latency_hist23:28:022
2165627716,19sleep00-21swapper/021:03:030
1657627726,26sleep00-21swapper/019:03:290
136742776,11sleep00-21swapper/000:02:560
1295027714,42sleep00-21swapper/022:18:200
8542756,11sleep30-21swapper/321:38:303
180902759,18sleep10-21swapper/120:48:341
1249827514,48sleep212509-21awk23:58:222
993727412,47sleep010062-21cat23:48:310
793127415,33sleep17348-21/usr/sbin/munin20:18:311
314972746,10sleep30-21swapper/319:50:023
2964027443,15sleep30-21swapper/321:28:133
2783327444,16sleep30-21swapper/319:38:283
257232745,11sleep30-21swapper/321:13:313
2281327416,37sleep222833-21/usr/sbin/munin22:48:382
2281327416,37sleep222833-21/usr/sbin/munin22:48:382
1788027410,30sleep20-21swapper/219:08:232
1768527414,41sleep10-21swapper/119:08:181
98832736,50sleep30-21swapper/322:08:223
235902737,40sleep10-21swapper/100:33:131
2025727310,18sleep00-21swapper/000:23:030
1768427322,31sleep10-21swapper/122:33:231
1513827316,34sleep215165-21munin-run20:43:022
1513827316,34sleep215165-21munin-run20:43:022
122627316,40sleep01118-21/usr/sbin/munin19:58:190
2680127216,34sleep126942-21date23:03:021
2554027211,39sleep125633-21cat21:13:291
248512726,10sleep30-21swapper/322:54:163
248512726,10sleep30-21swapper/322:54:163
24132727,10sleep30-21swapper/323:26:303
2335127213,36sleep223399-21latency_hist00:33:022
2316727217,33sleep2294-21rs:main2
2180927217,37sleep121824-21apt-get22:48:001
3239027142,14sleep30-21swapper/319:53:253
3178327111,30sleep10-21swapper/119:53:031
301042716,11sleep30-21swapper/319:46:593
2563527117,24sleep20-21swapper/221:13:302
1430527014,17sleep00-21swapper/020:38:320
1430527014,17sleep00-21swapper/020:38:320
464426924,25sleep20-21swapper/220:08:262
402026912,27sleep20-21swapper/221:48:302
402026912,27sleep20-21swapper/221:48:302
2941226920,19sleep00-21swapper/019:43:290
269126913,37sleep12761-21users21:43:331
269126913,37sleep12761-21users21:43:331
2405526910,42sleep024095-21ps00:33:280
2068826921,24sleep20-21swapper/200:23:212
1944526913,40sleep219524-21awk00:18:292
3261726816,36sleep132699-21find19:53:321
269626818,34sleep02669-21/usr/sbin/munin20:03:150
253026814,35sleep00-21swapper/023:28:000
2125526817,22sleep20-21swapper/219:18:202
1997426812,41sleep020008-21ntp_kernel_pll_19:13:260
940126721,23sleep10-21swapper/122:08:031
1664726725,19sleep10-21swapper/119:03:301
587526622,21sleep10-21swapper/120:13:221
3119126617,26sleep20-21swapper/221:33:162
276472666,15sleep00-21swapper/021:20:550
2462526613,36sleep224656-21/usr/sbin/munin19:28:262
1558626610,40sleep015605-21apt-get20:43:120
934826517,30sleep29372-21modprobe22:08:002
2589726516,28sleep20-21swapper/219:33:222
161342658,12sleep30-21swapper/319:03:203
1480226520,24sleep20-21swapper/222:23:312
2058126412,23sleep00-21swapper/000:23:190
1933226412,35sleep10-21swapper/119:13:071
1725026418,25sleep20-21swapper/220:48:192
873126316,30sleep08857-21/usr/sbin/munin22:03:300
3125926311,38sleep131377-21timerwakeupswit23:13:321
2817826312,36sleep128190-21fschecks_count21:23:181
1763026315,20sleep00-21swapper/022:33:220
1744526312,28sleep217520-21fschecks_time00:13:282
110132637,12sleep10-21swapper/123:53:201
633226215,29sleep00-21swapper/023:38:220
2397026214,23sleep00-21swapper/022:53:200
2397026214,23sleep00-21swapper/022:53:200
2241526219,23sleep10-21swapper/100:28:251
1918626212,21sleep20-21swapper/220:53:282
2500926116,26sleep20-21swapper/200:38:022
140472615,11sleep10-21swapper/120:38:251
140472615,11sleep10-21swapper/120:38:251
2907226017,28sleep129139-21/usr/sbin/munin23:08:311
2401626022,19sleep00-21swapper/019:28:030
912925913,30sleep18645-21apt-get20:23:111
211602599,36sleep00-21swapper/019:18:140
1742325916,26sleep217417-21apt-get22:33:152
1612325913,30sleep116205-21grep00:08:301
3057125814,23sleep00-21swapper/019:48:220
263772587,30sleep00-21swapper/019:33:320
2556425814,26sleep025697-21sed19:33:040
2465025810,37sleep10-21swapper/121:12:401
227712586,41sleep10-21swapper/121:03:361
151612586,41sleep20-21swapper/200:07:102
91462576,23sleep20-21swapper/220:23:122
679825717,21sleep20-21swapper/223:38:322
60932577,41sleep00-21swapper/021:56:280
60932577,41sleep00-21swapper/021:56:280
2892125713,25sleep228915-21apt-get19:43:152
257625722,21sleep12602-21munin-run23:28:011
137422578,39sleep10-21swapper/122:19:221
1140025713,28sleep211496-21sed23:53:292
785925613,24sleep00-21swapper/020:18:300
296525614,23sleep23055-21load20:03:222
240925611,26sleep20-21swapper/220:03:012
1927925614,27sleep019360-21cat20:53:300
1843525613,18sleep00-21swapper/019:08:330
1393825614,25sleep113957-21apt-get00:03:091
78725517,22sleep20-21swapper/221:38:292
388925515,21sleep00-21swapper/021:48:280
388925515,21sleep00-21swapper/021:48:280
2809325515,24sleep028088-21apt-get21:23:130
1957625512,25sleep219570-21apt-get19:13:162
1717625517,20sleep10-21swapper/122:33:021
114325517,20sleep10-21swapper/119:58:161
1117125518,20sleep10-21swapper/120:28:281
87772546,36sleep10-21swapper/123:45:171
370525418,19sleep20-21swapper/223:28:432
362525414,22sleep13673-21irqstats21:48:211
362525414,22sleep13673-21irqstats21:48:211
2788325417,21sleep127975-21mailstats19:38:301
2311525414,16sleep20-21swapper/219:23:252
1936125417,18sleep10-21swapper/120:53:311
3186025316,23sleep10-21swapper/121:33:321
2868525313,17sleep20-21swapper/219:43:032
2679125312,23sleep126818-21irqstats21:18:221
2675825313,24sleep026796-21munin-run23:03:010
2555025315,17sleep20-21swapper/222:58:232
2048025316,17sleep20-21swapper/220:58:202
1900025313,23sleep118997-21apt-get22:38:151
1900025313,23sleep118997-21apt-get22:38:151
185525313,23sleep21852-21apt-get21:43:142
185525313,23sleep21852-21apt-get21:43:142
3102725217,18sleep00-21swapper/023:13:270
1201825215,19sleep00-21swapper/020:33:030
928325112,23sleep29353-21df23:48:172
69192517,33sleep00-21swapper/020:16:150
468225113,23sleep24688-21diskstats23:33:192
2425425113,22sleep124282-21nfs4_client22:53:251
2425425113,22sleep124282-21nfs4_client22:53:251
2038425113,17sleep10-21swapper/120:58:191
1014025115,19sleep20-21swapper/222:08:292
61272505,34sleep20-21swapper/221:56:542
61272505,34sleep20-21swapper/221:56:542
3028725016,15sleep20-21swapper/219:48:032
1611325017,17sleep10-21swapper/120:43:501
1145425012,21sleep00-21swapper/022:13:230
7652499,16sleep00-21swapper/021:38:270
618424911,22sleep16179-21apt-get23:38:141
3218124913,20sleep132248-21df23:18:161
471124813,17sleep10-21swapper/120:08:291
2193224812,20sleep021938-21diskstats21:03:190
1439824811,22sleep014402-21diskstats22:23:200
1203924811,20sleep212085-21users22:13:332
703624714,14sleep00-21swapper/021:58:300
2296624713,15sleep10-21swapper/122:48:411
2296624713,15sleep10-21swapper/122:48:411
1755324711,21sleep00-21swapper/020:48:250
2735224614,16sleep00-21swapper/019:38:120
1572724410,15sleep00-21swapper/000:08:210
234822419,16sleep00-21swapper/021:08:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional