You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 14:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Wed Dec 03, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
573215818,1559sleep119120-21kworker/1:021:24:201
573215818,1559sleep119120-21kworker/1:021:24:201
20995210346,1013sleep221030-21ntpq22:28:352
31330291412,888sleep331408-21users21:18:373
311892511117,368sleep231327-21unixbench_multi21:18:362
23123242023,382sleep123130-21/usr/sbin/munin19:13:231
833623876,366sleep28380-21ntpq20:08:282
29637234912,200sleep229767-21unixbench_multi21:13:352
27689233723,294sleep327684-21apt-get19:28:193
214562314103,194sleep221522-21cpuspeed19:08:192
14713230887,197sleep214818-21timerandwakeup22:08:362
20224228836,92sleep220352-21timerwakeupswit20:43:342
216102287140,41sleep20-21swapper/220:48:312
16253227115,19sleep20-21swapper/223:56:192
23594224515,212sleep323720-21wc00:18:343
12086223724,193sleep212178-21apt-key22:03:022
3279221931,20sleep20-21swapper/219:53:182
10594221118,165sleep20-21swapper/223:38:262
10594221118,165sleep20-21swapper/223:38:262
19765220222,41sleep00-21swapper/000:08:130
12536219532,123sleep211962-21/usr/sbin/munin23:43:432
5054219113,123sleep20-21swapper/221:38:312
5054219113,123sleep20-21swapper/221:38:312
17232218911,163sleep117238-21grep22:18:161
29948218711,158sleep230097-21tail19:33:352
24992218213,127sleep225054-21cut00:23:302
739321779,153sleep27540-21cat21:48:042
739321779,153sleep27540-21cat21:48:042
9548217528,84sleep29678-21apt-key20:13:052
1927821737,129sleep25666-21snmpd20:43:022
7956217211,114sleep20-21swapper/223:31:192
8639216930,46sleep38636-21apt-get23:33:143
1083216411,140sleep21156-21mailstats23:08:292
1083216411,140sleep21156-21mailstats23:08:292
12728215826,105sleep00-21swapper/020:23:040
25724215615,41sleep20-21swapper/222:43:292
29326215214,124sleep329365-21ntpq21:13:283
26621215216,19sleep10-21swapper/122:48:051
28254214615,70sleep20-21swapper/221:08:342
13696214317,83sleep20-21swapper/220:23:552
22757213927,75sleep222910-21sed00:18:042
1861213514,100sleep20-21swapper/221:28:462
1861213514,100sleep20-21swapper/221:28:462
12566213539,18sleep20-21swapper/220:23:022
1644021347,46sleep20-21swapper/220:33:122
791213317,105sleep00-21swapper/021:27:060
791213317,105sleep00-21swapper/021:27:060
31417213212,52sleep231505-21timerandwakeup19:38:322
2438213010,102sleep32439-21sshd21:30:203
2438213010,102sleep32439-21sshd21:30:203
23529212911,98sleep20-21swapper/200:18:332
16093212911,35sleep20-21swapper/222:13:292
2755821288,57sleep227679-21awk22:48:342
8611212721,39sleep20-21swapper/223:33:092
3707212737,41sleep30-21swapper/323:18:123
9178212654,32sleep20-21swapper/221:53:132
9178212654,32sleep20-21swapper/221:53:132
593421218,80sleep25666-21snmpd20:00:572
760321160,11sleep312-21rcu_sched21:48:123
11282211613,84sleep311284-21/usr/sbin/munin20:18:153
30361211523,39sleep20-21swapper/222:58:282
27398211410,68sleep227431-21awk19:28:022
22738211410,63sleep20-21swapper/222:33:322
2582021138,41sleep20-21swapper/221:03:172
653621129,68sleep26363-21/usr/sbin/munin20:03:242
544521129,65sleep20-21swapper/223:23:242
437521128,66sleep24493-21sed23:18:312
26873211116,53sleep20-21swapper/200:28:342
1554721118,61sleep20-21swapper/220:28:522
1554721118,61sleep20-21swapper/220:28:522
13859211116,56sleep20-21swapper/222:08:032
2373021107,70sleep25666-21snmpd20:55:012
22677210920,37sleep10-21swapper/120:53:141
22532210812,75sleep022560-21/usr/sbin/munin22:33:260
2438021079,40sleep224522-21latency_hist19:18:042
20895210717,73sleep021043-21latency_hist19:08:040
18492210635,57sleep218511-21kernelversion00:03:222
14434210612,69sleep214449-21awk23:48:362
1987210516,76sleep32051-21cat19:48:383
1719210510,80sleep11721-21sshd23:11:111
1719210510,80sleep11721-21sshd23:11:111
1572821059,80sleep015755-21df22:13:180
28947210447,14sleep20-21swapper/219:33:022
32224210324,62sleep032363-21sed23:03:350
166821037,36sleep20-21swapper/219:48:132
7411210121,64sleep17485-21switchtime23:28:361
3473210019,68sleep13493-21unin-run23:18:021
17137210029,61sleep217143-21ntp_states23:58:302
1846729916,62sleep117900-21/usr/sbin/munin18:58:331
2076829814,64sleep120781-21sh19:08:011
250532967,68sleep00-21swapper/019:18:310
226329615,59sleep32157-21/usr/sbin/munin23:13:193
1409129612,70sleep314167-21sed23:48:333
759429439,36sleep20-21swapper/221:48:122
305729312,39sleep20-21swapper/221:33:222
305729312,39sleep20-21swapper/221:33:222
2336829315,56sleep122950-21/usr/sbin/munin00:18:281
1711329313,18sleep10-21swapper/123:58:291
528029116,60sleep35666-21snmpd23:23:173
1509629114,63sleep015150-21if_err_eth020:28:420
1509629114,63sleep015150-21if_err_eth020:28:420
1159729124,42sleep311695-21latency_hist23:43:043
1159729124,42sleep311695-21latency_hist23:43:043
936029016,55sleep19544-21latency_hist20:13:041
1429629010,61sleep114433-21/usr/sbin/munin23:48:351
115729011,65sleep31235-21switchtime23:08:303
115729011,65sleep31235-21switchtime23:08:303
304128912,54sleep00-21swapper/019:53:040
2493728916,63sleep20-21swapper/222:42:322
2024128915,60sleep320260-21wc19:03:353
117128917,56sleep31210-21http19:48:023
3157028820,30sleep20-21swapper/223:03:202
297112888,37sleep20-21swapper/222:58:032
195962889,37sleep20-21swapper/219:03:222
145828810,15sleep10-21swapper/121:28:201
145828810,15sleep10-21swapper/121:28:201
502128717,22sleep00-21swapper/019:58:250
193572874,11sleep00-21swapper/000:06:150
1622628719,50sleep10-21swapper/120:33:031
1622628719,50sleep10-21swapper/120:33:031
867328613,58sleep18636-21apt-get23:33:371
2590228610,61sleep126025-21latency_hist19:23:041
166728619,49sleep11184-21apt-get19:48:121
211732859,63sleep30-21swapper/322:28:403
1051928521,49sleep010613-21sed21:58:040
1051928521,49sleep010613-21sed21:58:040
2658628414,55sleep126594-21ntp_states19:23:281
1899228411,36sleep20-21swapper/222:23:222
343128320,30sleep20-21swapper/223:17:502
325132836,61sleep20-21swapper/219:43:242
297328315,49sleep33026-21cpuspeed21:33:193
297328315,49sleep33026-21cpuspeed21:33:193
273028318,46sleep02669-21latency_hist21:33:040
273028318,46sleep02669-21latency_hist21:33:040
2447128317,33sleep20-21swapper/220:58:242
176822839,49sleep00-21swapper/022:18:270
172402836,14sleep20-21swapper/222:18:172
2729928217,48sleep327367-21systemctl22:48:313
124028213,43sleep00-21swapper/023:08:310
124028213,43sleep00-21swapper/023:08:310
1237028221,32sleep10-21swapper/123:43:381
1019528218,30sleep00-21swapper/023:38:160
1019528218,30sleep00-21swapper/023:38:160
2674428112,51sleep126872-21sed00:28:331
2628428122,39sleep326322-21ps21:03:303
1271528122,35sleep10-21swapper/122:03:381
89582808,58sleep00-21swapper/021:53:030
320542809,54sleep20-21swapper/221:23:162
320542809,54sleep20-21swapper/221:23:162
284922809,15sleep00-21swapper/019:28:360
2707728018,46sleep127097-21munin-plugin-st21:08:011
1306528010,56sleep012580-21apt-get20:23:130
234442798,58sleep023485-21ntpq19:13:300
1410127923,25sleep10-21swapper/122:08:191
738327818,43sleep17390-21cat21:48:031
738327818,43sleep17390-21cat21:48:031
344427821,40sleep03471-21basename23:18:000
2741027815,29sleep30-21swapper/300:33:043
256972785,62sleep20-21swapper/219:21:032
2320827816,50sleep323295-21basename20:53:333
200202787,32sleep20-21swapper/200:08:232
1677827825,31sleep316871-21uniq23:58:223
1422427811,52sleep314301-21fschecks_time22:08:253
1340127810,17sleep30-21swapper/320:23:483
52572779,51sleep10-21swapper/123:23:141
3267027713,49sleep332767-21timerandwakeup21:23:343
3267027713,49sleep332767-21timerandwakeup21:23:343
2936827710,52sleep029410-21ntp_states21:13:290
2626327711,31sleep30-21swapper/319:23:223
702127618,40sleep07084-21/usr/sbin/munin23:28:270
1402627615,41sleep114037-21vmstat20:23:581
344827514,42sleep03507-21/usr/sbin/munin21:33:320
344827514,42sleep03507-21/usr/sbin/munin21:33:320
2863527525,32sleep128653-21apt21:13:011
2072127523,28sleep10-21swapper/122:28:301
1648127512,46sleep016620-21cat23:58:030
2251627412,51sleep00-21swapper/019:10:230
1994427417,28sleep30-21swapper/320:43:273
2188027328,26sleep121099-21/usr/sbin/munin20:48:341
2169327316,17sleep30-21swapper/320:48:323
1838427323,21sleep30-21swapper/318:58:313
1537327314,26sleep30-21swapper/320:28:493
1537327314,26sleep30-21swapper/320:28:493
1209127315,28sleep00-21swapper/023:43:330
343827219,25sleep30-21swapper/319:53:233
3032027218,30sleep10-21swapper/122:58:271
270942726,14sleep10-21swapper/122:48:251
2025827221,39sleep120323-21awk00:08:301
539127120,21sleep10-21swapper/119:58:341
2205127116,39sleep322174-21sed00:13:333
194622710,7sleep00-21swapper/022:23:310
1937127112,44sleep019519-21latency_hist20:43:040
2931027017,32sleep329338-21sshd19:33:203
2334427012,24sleep00-21swapper/000:18:270
1976227013,41sleep019825-21/usr/sbin/munin20:43:240
1106927018,27sleep00-21swapper/020:18:040
488926915,22sleep10-21swapper/121:38:261
488926915,22sleep10-21swapper/121:38:261
285212699,18sleep00-21swapper/021:11:310
2656326920,35sleep326572-21systemd-run22:48:033
2281526915,32sleep30-21swapper/322:33:333
1102526914,37sleep311103-21sed21:58:233
2836326717,36sleep128381-21apt22:53:141
2347526715,24sleep123564-21ls22:38:031
695326618,24sleep30-21swapper/323:28:263
463126614,35sleep04642-21/usr/sbin/munin23:18:330
321026612,24sleep10-21swapper/121:33:251
321026612,24sleep10-21swapper/121:33:251
2316526617,32sleep323251-21perl19:13:253
2039526624,26sleep30-21swapper/300:08:323
2018226615,37sleep320267-21apt-config22:28:043
520226513,23sleep30-21swapper/319:58:283
2452426515,27sleep10-21swapper/100:23:161
984226412,36sleep19969-21/usr/sbin/munin21:53:321
984226412,36sleep19969-21/usr/sbin/munin21:53:321
861626419,24sleep00-21swapper/020:08:350
665826414,25sleep06708-21sed20:03:260
3197926417,23sleep10-21swapper/123:03:321
3088126412,29sleep10-21swapper/119:38:181
2707726419,29sleep027093-21kernelversion22:48:240
2238426417,31sleep022433-21modprobe20:53:010
589026313,36sleep05966-21/usr/sbin/munin23:23:340
585526322,18sleep30-21swapper/321:43:043
585526322,18sleep30-21swapper/321:43:043
3175826311,39sleep00-21swapper/019:39:020
2791326316,30sleep327979-21/usr/sbin/munin21:08:313
1878526317,19sleep10-21swapper/122:23:171
1629126321,22sleep30-21swapper/322:13:343
62852620,7sleep30-21swapper/321:43:243
62852620,7sleep30-21swapper/321:43:243
3048126215,29sleep130500-21apt-get21:18:141
2507726222,21sleep10-21swapper/122:43:041
2423226219,28sleep324251-21apt-get20:58:133
2315026218,29sleep023157-21ntp_states20:53:300
190602627,16sleep00-21swapper/019:01:260
1777726211,26sleep00-21swapper/020:38:030
1692026211,32sleep10-21swapper/120:33:281
1644026216,22sleep30-21swapper/323:58:023
1314126213,31sleep313211-21/usr/sbin/munin22:03:483
422326124,23sleep10-21swapper/119:54:021
3099526112,36sleep01-21systemd23:00:200
88482604,10sleep30-21swapper/320:08:383
78882609,12sleep10-21swapper/121:48:231
42932608,42sleep00-21swapper/019:54:490
2505526011,23sleep30-21swapper/300:23:313
1703426016,25sleep10-21swapper/122:18:041
1251726022,22sleep00-21swapper/022:03:130
2987025912,31sleep10-21swapper/119:33:341
2071725916,19sleep00-21swapper/022:28:290
1430525910,14sleep00-21swapper/022:08:260
108122596,11sleep10-21swapper/120:15:261
607025814,27sleep06064-21apt-get21:43:170
607025814,27sleep06064-21apt-get21:43:170
3276025820,22sleep30-21swapper/323:08:023
2664525813,21sleep00-21swapper/019:23:310
2408225818,21sleep30-21swapper/322:38:263
1092725816,21sleep10-21swapper/123:38:331
1092725816,21sleep10-21swapper/123:38:331
2657425719,17sleep00-21swapper/022:48:040
253962576,42sleep10-21swapper/121:01:261
2526025715,26sleep325389-21sed19:18:343
1957525715,25sleep119570-21apt-get20:43:171
1687425718,22sleep00-21swapper/023:58:230
1487225722,19sleep00-21swapper/023:53:020
1092025720,20sleep10-21swapper/121:58:211
452025615,25sleep04512-21apt-get21:38:170
452025615,25sleep04512-21apt-get21:38:170
3091625616,16sleep00-21swapper/021:18:300
3081125614,25sleep330806-21apt-get19:38:163
2994925613,26sleep329944-21apt-get22:58:173
2950425618,19sleep00-21swapper/019:33:250
129425613,26sleep11379-21munin-run19:48:041
2149925512,29sleep021585-21fw_packets00:13:220
73972546,36sleep30-21swapper/320:04:543
2502325418,20sleep00-21swapper/022:43:030
1536125420,22sleep115097-21/usr/sbin/munin23:53:241
2838825314,23sleep028383-21apt-get22:53:170
2607125315,22sleep026066-21apt-get00:28:160
2583525314,16sleep00-21swapper/000:28:040
2185625318,19sleep30-21swapper/319:08:283
2154225319,20sleep10-21swapper/119:08:221
1792225313,25sleep217918-21apt-get18:58:182
1260725321,18sleep30-21swapper/323:43:443
1201625315,24sleep012020-21modprobe22:03:000
373125211,25sleep13726-21apt-get23:18:141
1884525218,21sleep318969-21timerandwakeup00:03:313
1822725214,23sleep118222-21apt-get00:03:131
1130625212,24sleep111303-21apt-get20:18:171
1050825212,24sleep310517-21munin-run21:58:033
1050825212,24sleep310517-21munin-run21:58:033
684825113,16sleep10-21swapper/120:03:311
2955425112,21sleep10-21swapper/121:13:341
1932925119,17sleep30-21swapper/320:43:033
1645825113,22sleep316460-21apt-get20:33:133
1058125110,16sleep30-21swapper/320:13:473
876025013,21sleep08764-21diskstats23:33:440
2808725015,18sleep10-21swapper/119:28:291
2713124914,14sleep00-21swapper/021:08:040
3236524713,16sleep00-21swapper/019:43:180
2876024710,21sleep328805-21ntp_kernel_pll_22:53:283
1802024711,21sleep118089-21df20:38:201
2188124311,17sleep121957-21systemctl00:13:311
1722924310,18sleep317225-21apt-get22:18:143
242602417,16sleep00-21swapper/020:58:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional