You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-18 - 02:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Fri Oct 17, 2025 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7230218576,1836sleep13838-21kworker/1:107:36:061
7230218576,1836sleep13838-21kworker/1:107:36:061
801210866,1068sleep29228-1kworker/2:1H12:26:122
63527296,710sleep1671-21ntpq10:41:271
2866524356,393sleep128704-21ntpq12:11:081
10982242917,397sleep311020-21ntpq09:31:063
1348237919,342sleep31361-21/usr/sbin/munin12:27:263
6967234815,316sleep37005-21unin-run09:20:443
175942314118,170sleep10-21swapper/111:36:031
175942314118,170sleep10-21swapper/111:36:031
2158322679,222sleep121597-21fschecks_count10:06:011
31681226280,58sleep10-21swapper/108:55:461
2134522476,208sleep121381-21ntpq08:21:051
2769722415,225sleep126166-21kworker/1:212:06:481
2769722415,225sleep126166-21kworker/1:212:06:481
1257223913,209sleep01303-21/usr/sbin/munin12:27:230
4652223532,144sleep14738-21latency_hist10:55:451
702223357,122sleep1706-21acpi07:15:541
9040222022,179sleep29033-21apt-get09:26:002
1666221839,115sleep10-21swapper/109:01:271
1666221839,115sleep10-21swapper/109:01:271
28842221539,163sleep128895-21fschecks_count08:46:051
3670220666,73sleep13721-21latency_hist12:35:441
3670220666,73sleep13721-21latency_hist12:35:441
13282219714,18sleep00-21swapper/007:56:040
13282219714,18sleep00-21swapper/007:56:040
30663219427,53sleep10-21swapper/112:16:211
30663219427,53sleep10-21swapper/112:16:211
18271219466,113sleep118333-21date09:55:471
12220219310,141sleep112300-21timerandwakeup07:51:161
2146218417,148sleep22191-21ntp_kernel_pll_10:46:092
4596217711,62sleep10-21swapper/109:11:131
1024821737,12sleep00-21swapper/007:46:060
3539217244,83sleep1284-21dbus-daemon07:25:441
3539217244,83sleep1284-21dbus-daemon07:25:441
9311216952,104sleep19314-21cron11:10:431
856216611,143sleep1894-21grep12:26:511
6719216030,108sleep129045-21cyclictest07:35:451
10138216020,78sleep10-21swapper/107:46:031
17094215811,125sleep21-21systemd11:35:442
17094215811,125sleep21-21systemd11:35:442
894521579,15sleep10-21swapper/107:41:111
894521579,15sleep10-21swapper/107:41:111
7127215728,92sleep17298-21latency_hist09:20:461
14533215739,59sleep10-21swapper/108:00:451
14533215739,59sleep10-21swapper/108:00:451
15267215322,57sleep10-21swapper/109:45:581
7008215231,99sleep029469-1kworker/0:0H09:20:450
24820215123,102sleep224959-21unixbench_singl08:31:162
28565214925,104sleep328629-21cat12:11:063
25195214829,64sleep125226-21grep12:00:591
1055821486,86sleep1362-21snmpd11:11:481
5264214721,43sleep10-21swapper/110:56:091
27472214670,60sleep10-21swapper/108:41:041
21061214632,75sleep121133-21cat11:46:171
8498214527,105sleep10-21swapper/109:22:271
25246214419,80sleep10-21swapper/110:16:151
24185214417,47sleep10-21swapper/111:56:141
31418213811,90sleep131421-21acpi12:20:541
30052213777,18sleep10-21swapper/107:06:541
30052213777,18sleep10-21swapper/107:06:541
28824213724,91sleep028850-21munin-plugin-st07:05:440
1477721376,13sleep10-21swapper/109:43:071
1477721376,13sleep10-21swapper/109:43:071
10947213320,98sleep310990-21latency_hist11:15:463
4408213011,74sleep10-21swapper/107:26:161
4858212914,96sleep05007-21grep09:11:160
2339821299,106sleep123531-21latency_hist11:55:461
2820121269,70sleep10-21swapper/110:26:121
12075212613,99sleep212082-21open_files07:51:132
13341212511,53sleep10-21swapper/111:21:131
25407212415,83sleep30-21swapper/308:35:453
14270212314,86sleep014338-21switchtime09:41:160
14270212314,86sleep014338-21switchtime09:41:160
10561212111,69sleep110565-21acpi09:30:511
9764211738,19sleep10-21swapper/109:26:231
2618921157,20sleep00-21swapper/010:20:580
2366021147,49sleep123775-21sed10:11:141
18492211316,81sleep00-21swapper/008:11:270
32683211216,76sleep332733-21apt-key07:15:443
32683211216,76sleep332733-21apt-key07:15:443
271921116,72sleep10-21swapper/110:46:441
12865210922,69sleep012882-21cron07:55:440
2929421089,42sleep10-21swapper/110:30:571
29366210517,73sleep028846-21apt-get07:05:560
29366210517,73sleep028846-21apt-get07:05:560
1908921055,48sleep10-21swapper/108:15:271
18875210132,50sleep10-21swapper/109:56:161
15014210022,28sleep20-21swapper/208:01:072
15014210022,28sleep20-21swapper/208:01:072
386329950,16sleep13871-21munin-run09:10:441
254729912,22sleep00-21swapper/012:31:060
254729912,22sleep00-21swapper/012:31:060
76129813,71sleep0863-21cut09:00:460
2691729847,40sleep126935-21munin-run08:40:431
2691729847,40sleep126935-21munin-run08:40:431
628729711,70sleep26344-21latency_hist11:00:462
3257329737,46sleep10-21swapper/107:15:061
3257329737,46sleep10-21swapper/107:15:061
214742977,16sleep20-21swapper/210:05:542
292262969,75sleep0362-21snmpd12:11:240
3014029541,24sleep230291-21grep07:06:552
3014029541,24sleep230291-21grep07:06:552
312772947,75sleep131321-21ntpq10:36:101
198022947,71sleep10-21swapper/108:16:081
695429319,57sleep26966-21cron09:20:432
2899829326,46sleep329019-21apt10:30:433
1262929312,63sleep00-21swapper/009:36:120
596229218,56sleep06032-21switchtime07:31:160
3262229222,46sleep232641-21apt-get07:15:432
3262229222,46sleep232641-21apt-get07:15:432
2059829211,63sleep220658-21/usr/sbin/munin11:46:062
1755129232,24sleep117576-21munin-run08:10:431
419529117,50sleep34259-21sort09:11:023
717129015,54sleep06937-21/usr/sbin/munin07:36:050
717129015,54sleep06937-21/usr/sbin/munin07:36:050
2099829011,61sleep021060-21/usr/sbin/munin11:46:160
3267628918,26sleep00-21swapper/010:41:060
3082628936,35sleep330966-21cpuspeed07:10:483
1982428924,47sleep10-21swapper/111:41:161
1233128911,66sleep112390-21cat09:36:041
2053728830,34sleep30-21swapper/310:01:233
2279028720,11sleep10-21swapper/108:26:061
1986928719,11sleep10-21swapper/110:00:481
105628712,56sleep21123-21memory07:16:242
3028828616,53sleep230306-21/usr/sbin/munin08:50:542
2030528624,30sleep30-21swapper/311:45:463
1760028613,59sleep317657-21latency_hist08:10:453
155292865,62sleep30-21swapper/309:46:083
1481628634,16sleep10-21swapper/108:01:021
1481628634,16sleep10-21swapper/108:01:021
1436028620,14sleep10-21swapper/111:26:011
111892869,16sleep10-21swapper/111:16:001
981728511,60sleep39926-21latency_hist07:45:453
981728511,60sleep39926-21latency_hist07:45:453
2121228520,46sleep0225-21systemd-journal10:05:440
168732859,64sleep00-21swapper/011:32:510
168732859,64sleep00-21swapper/011:32:510
64592847,56sleep30-21swapper/311:01:043
64592847,56sleep30-21swapper/311:01:043
2369428411,60sleep0362-21snmpd08:29:320
752328319,11sleep10-21swapper/111:02:391
752328319,11sleep10-21swapper/111:02:391
2260828313,54sleep222628-21fschecks_count08:26:032
1809928314,52sleep21-21systemd09:55:442
340428220,40sleep23444-21fschecks_count10:51:052
2892828223,37sleep20-21swapper/207:05:462
261902827,19sleep126185-21apt-get10:20:591
2537728219,44sleep225170-21/usr/sbin/munin12:01:042
51528118,10sleep10-21swapper/108:58:121
2407528120,15sleep10-21swapper/108:30:581
158272816,14sleep10-21swapper/111:30:581
158272816,14sleep10-21swapper/111:30:581
2437828016,48sleep324525-21sed11:56:163
1169428011,51sleep211789-21mailstats11:16:132
1104928019,40sleep211081-21proc_pri09:31:092
2885827923,33sleep11-21systemd07:05:451
270427916,43sleep32312-21/usr/sbin/munin12:31:093
270427916,43sleep32312-21/usr/sbin/munin12:31:093
177327917,43sleep2974-21/usr/sbin/munin09:01:282
177327917,43sleep2974-21/usr/sbin/munin09:01:282
1543727921,29sleep30-21swapper/308:01:163
1543727921,29sleep30-21swapper/308:01:163
393527821,38sleep34029-21fschecks_time07:26:063
3161427813,50sleep031759-21awk10:36:170
2644827816,44sleep226563-21users08:36:162
2644827816,44sleep226563-21users08:36:162
222012785,62sleep00-21swapper/008:25:160
204027819,34sleep30-21swapper/307:20:453
387227720,28sleep20-21swapper/209:10:452
264727726,30sleep00-21swapper/009:06:030
2377627721,22sleep20-21swapper/210:11:152
2223527618,36sleep322243-21modprobe08:25:433
2749927523,30sleep30-21swapper/310:25:463
2711127512,16sleep00-21swapper/012:06:070
2711127512,16sleep00-21swapper/012:06:070
1773727522,22sleep00-21swapper/009:51:220
1676027518,29sleep20-21swapper/208:06:122
372327420,38sleep23865-21latency_hist12:35:452
2252927411,18sleep00-21swapper/011:51:170
1684527418,30sleep00-21swapper/008:06:140
872627317,17sleep20-21swapper/211:06:282
518527313,39sleep25104-21latency_hist07:30:462
1996927323,29sleep30-21swapper/308:16:143
1809427322,32sleep018244-21sed11:36:140
1809427322,32sleep018244-21sed11:36:140
859827219,38sleep28629-21fschecks_count07:41:042
859827219,38sleep28629-21fschecks_count07:41:042
3133327221,30sleep30-21swapper/310:36:123
1243427221,35sleep312513-21latency09:36:063
12062727,12sleep30-21swapper/309:01:053
12062727,12sleep30-21swapper/309:01:053
70882719,12sleep30-21swapper/307:36:043
70882719,12sleep30-21swapper/307:36:043
1179027121,26sleep30-21swapper/311:16:143
900127017,38sleep09087-21mailstats07:41:140
900127017,38sleep09087-21mailstats07:41:140
73822706,13sleep30-21swapper/309:20:483
395727018,34sleep34079-21sed10:51:183
2782327023,21sleep00-21swapper/010:26:030
152527013,41sleep21567-21latency_hist10:45:442
2987126922,30sleep329879-21/usr/sbin/munin12:15:593
2987126922,30sleep329879-21/usr/sbin/munin12:15:593
1497626926,20sleep30-21swapper/311:26:163
1320326924,24sleep30-21swapper/311:21:113
305826714,27sleep20-21swapper/207:21:172
305826714,27sleep20-21swapper/207:21:172
2975826713,28sleep20-21swapper/210:31:102
2813726714,34sleep228181-21/usr/sbin/munin10:26:092
277426717,31sleep30-21swapper/307:21:143
277426717,31sleep30-21swapper/307:21:143
545626613,38sleep35465-21df_inode07:31:033
3218426618,29sleep332323-21grep12:21:163
3068026612,39sleep330714-21nfs_client08:51:073
2926526610,11sleep30-21swapper/308:46:153
2606026620,27sleep30-21swapper/308:36:093
2606026620,27sleep30-21swapper/308:36:093
250012666,15sleep00-21swapper/008:31:220
2623726513,37sleep026321-21switchtime08:36:140
2623726513,37sleep026321-21switchtime08:36:140
242526518,26sleep20-21swapper/212:31:042
185226516,31sleep00-21swapper/010:46:030
1351426519,21sleep20-21swapper/209:40:462
704026414,38sleep27133-21cat11:01:232
704026414,38sleep27133-21cat11:01:232
2509526421,21sleep00-21swapper/010:16:100
1035226419,20sleep30-21swapper/309:30:453
2945226316,31sleep329520-21df07:06:403
2823726316,26sleep30-21swapper/307:01:113
2439226317,22sleep30-21swapper/310:15:443
1605426315,23sleep00-21swapper/008:05:450
1605426315,23sleep00-21swapper/008:05:450
1177126319,23sleep30-21swapper/307:51:063
813926212,19sleep00-21swapper/011:06:060
60942625,45sleep00-21swapper/010:59:520
322642625,12sleep20-21swapper/208:56:112
3082026213,26sleep00-21swapper/008:51:120
2852626214,26sleep20-21swapper/208:45:452
2508326220,19sleep30-21swapper/310:16:093
2170526212,36sleep321755-21if_eth010:06:033
2073726212,24sleep20-21swapper/208:20:452
1904826219,19sleep00-21swapper/011:41:030
3142326112,35sleep230851-21/usr/sbin/munin10:36:152
2459426121,21sleep00-21swapper/011:56:460
1834726124,20sleep218409-21mailstats08:11:252
1566726113,31sleep215704-21sed09:46:112
742426014,29sleep27432-21awk07:36:112
742426014,29sleep27432-21awk07:36:112
3273726013,22sleep20-21swapper/212:25:432
3254926012,20sleep30-21swapper/310:40:483
279672604,11sleep20-21swapper/212:10:122
279672604,11sleep20-21swapper/212:10:122
2331826017,28sleep20-21swapper/211:55:432
1127326016,18sleep00-21swapper/011:16:030
338925915,19sleep00-21swapper/010:51:030
208822599,12sleep30-21swapper/311:46:113
1420725918,23sleep30-21swapper/309:41:153
1420725918,23sleep30-21swapper/309:41:153
1418425916,25sleep20-21swapper/209:41:142
1418425916,25sleep20-21swapper/209:41:142
553925821,19sleep20-21swapper/207:31:062
2999125816,18sleep00-21swapper/012:16:080
2999125816,18sleep00-21swapper/012:16:080
2546625812,19sleep00-21swapper/012:01:060
195842585,12sleep20-21swapper/209:59:442
1839425813,25sleep018461-21/usr/sbin/munin09:55:580
1185025817,25sleep00-21swapper/009:35:430
550225719,17sleep30-21swapper/310:56:153
3063225718,21sleep00-21swapper/010:35:450
2768425714,22sleep00-21swapper/008:41:110
1658025721,20sleep00-21swapper/009:50:450
1433525717,19sleep00-21swapper/011:26:000
282925614,23sleep30-21swapper/309:06:063
1716525617,21sleep30-21swapper/311:35:453
1716525617,21sleep30-21swapper/311:35:453
1099825615,18sleep00-21swapper/011:15:470
1035625612,23sleep00-21swapper/009:30:460
91082555,37sleep30-21swapper/311:08:063
729925511,24sleep20-21swapper/209:20:472
2526725517,23sleep325273-21df_inode12:01:013
2442525515,24sleep024574-21latency_hist10:15:450
2363625513,26sleep223630-21apt-get11:55:592
227825514,22sleep30-21swapper/310:46:143
1794125510,31sleep218015-21sensors_temp11:36:122
1794125510,31sleep218015-21sensors_temp11:36:122
437825418,19sleep20-21swapper/209:11:052
432625420,16sleep00-21swapper/007:26:150
2701625413,18sleep30-21swapper/312:06:063
2701625413,18sleep30-21swapper/312:06:063
2126325410,26sleep20-21swapper/210:05:462
176822545,13sleep30-21swapper/311:36:053
176822545,13sleep30-21swapper/311:36:053
1282125419,16sleep20-21swapper/211:21:022
1033825410,28sleep210432-21memory07:46:082
99825312,24sleep0992-21apt-get09:00:580
99825312,24sleep0992-21apt-get09:00:580
783525314,19sleep00-21swapper/009:22:020
3115025314,23sleep231189-21modprobe12:20:422
3115025314,23sleep231189-21modprobe12:20:422
2883725313,16sleep00-21swapper/008:46:040
3222725213,23sleep032265-21users07:11:170
3222725213,23sleep032265-21users07:11:170
3190625211,24sleep331969-21df08:56:043
2331325210,18sleep30-21swapper/308:26:163
160625214,21sleep01659-21users07:16:340
997125114,21sleep29972-21memory11:11:082
2628225115,19sleep20-21swapper/210:21:092
206872518,16sleep00-21swapper/008:20:430
1681625113,22sleep316813-21apt-get09:50:573
1445125113,23sleep314495-21systemd-run08:00:433
1445125113,23sleep314495-21systemd-run08:00:433
112892515,11sleep30-21swapper/307:50:213
72612509,15sleep00-21swapper/011:01:250
72612509,15sleep00-21swapper/011:01:250
3144425011,23sleep031439-21apt-get12:20:570
2470525014,21sleep224712-21df_inode10:16:002
213832509,17sleep20-21swapper/208:21:062
2027125014,17sleep20-21swapper/211:45:452
1360725012,21sleep213643-21ntp_states07:56:102
1360725012,21sleep213643-21ntp_states07:56:102
2929124912,21sleep329288-21apt-get10:30:553
2267224912,16sleep30-21swapper/311:51:203
1876424915,17sleep30-21swapper/309:56:103
1327024911,21sleep013340-21sensors11:21:120
2889724813,19sleep228967-21fw_packets08:46:062
255824812,15sleep00-21swapper/007:21:060
255824812,15sleep00-21swapper/007:21:060
2870624710,17sleep20-21swapper/212:11:092
1783824713,16sleep30-21swapper/308:10:573
152462465,32sleep20-21swapper/211:26:512
2737524512,13sleep30-21swapper/308:41:023
219422439,17sleep00-21swapper/010:06:080
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional