You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 16:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Wed Nov 19, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
127232775995,7647sleep012221-21apt-get10:33:590
7834237318,330sleep17856-21awk08:38:481
7834237318,330sleep17856-21awk08:38:481
10509234411,319sleep210645-21awk07:04:202
6140232118,280sleep26299-21grep11:54:212
3420227915,248sleep13437-21apt08:24:021
5988227811,253sleep06058-21sensors_temp11:54:190
12198224715,217sleep112275-21unin-run10:33:481
3063822458,221sleep017462-1kworker/0:2H09:49:010
23715224014,210sleep31-21systemd09:28:493
9142223522,194sleep19293-21latency_hist10:23:511
3226222420,100sleep29610-21cyclictest10:04:092
12862221325,110sleep212875-21fschecks_count10:34:262
31300221177,112sleep20-21swapper/208:09:132
31300221177,112sleep20-21swapper/208:09:132
9910218928,85sleep20-21swapper/212:08:502
21969218818,149sleep222062-21fschecks_time07:39:092
5400218448,120sleep25542-21sshd10:09:222
449221809,156sleep24591-21timerandwakeup11:49:242
794217715,147sleep2921-21cat09:55:212
11029217469,81sleep20-21swapper/208:48:502
2496821648,96sleep20-21swapper/207:49:022
13312215549,91sleep213328-21munin-run07:13:492
13312215549,91sleep213328-21munin-run07:13:492
28744215310,20sleep028767-21ntp_states11:24:140
3725215258,15sleep20-21swapper/211:48:522
3725215258,15sleep20-21swapper/211:48:522
17847215010,126sleep217455-21/usr/sbin/munin09:09:142
2131214837,93sleep20-21swapper/208:19:112
2039421488,13sleep00-21swapper/007:34:060
26815214713,15sleep00-21swapper/011:19:060
1907121476,131sleep219119-21awk09:14:062
1907121476,131sleep219119-21awk09:14:062
12969214350,78sleep212995-21idleruntime-cro12:18:482
25783214136,83sleep20-21swapper/209:34:092
16974213955,65sleep20-21swapper/212:29:222
16974213955,65sleep20-21swapper/212:29:222
1567213864,18sleep20-21swapper/209:59:002
15170213655,62sleep20-21swapper/207:18:502
517213241,18sleep29610-21cyclictest11:39:022
25418212913,100sleep3369-21sshd09:33:213
25418212913,100sleep3369-21sshd09:33:213
9434212856,20sleep29522-21wc08:43:502
24167212510,20sleep00-21swapper/009:28:580
24167212510,20sleep00-21swapper/009:28:580
17481212521,84sleep317494-21fschecks_count10:49:083
15230212120,79sleep3225-21systemd-journal07:18:513
2297211919,81sleep11810-21apt-get11:43:581
2297211919,81sleep11810-21apt-get11:43:581
8332211528,69sleep18357-21fschecks_count06:59:101
19183211517,21sleep20-21swapper/210:54:122
14489211519,23sleep20-21swapper/210:39:102
2114211418,75sleep02209-21cut09:59:310
18654211414,81sleep20-21swapper/210:53:482
20620211317,73sleep020650-21fschecks_count10:59:110
28626211118,38sleep228644-21latency_hist09:43:502
18751211120,63sleep018765-21modprobe09:13:480
12403211115,73sleep312604-21gpgv10:33:503
1560721106,12sleep20-21swapper/207:18:542
11389211014,82sleep311453-21sendmail-msp12:13:483
23505210920,71sleep023542-21fschecks_count07:44:080
26672210814,72sleep026721-21fschecks_time07:54:130
4400210723,59sleep04425-21awk10:08:470
18725210616,75sleep118744-21apt07:29:021
28316210511,78sleep20-21swapper/211:24:042
20996210510,81sleep220510-21/usr/sbin/munin10:59:212
1463921058,78sleep20-21swapper/208:59:122
1130821057,83sleep211344-21ntpq10:29:172
3004621049,71sleep20-21swapper/211:29:092
17619210417,69sleep017649-21idleruntime-cro12:33:480
17619210417,69sleep017649-21idleruntime-cro12:33:480
16016210318,19sleep20-21swapper/209:04:092
3039921026,44sleep230546-21awk08:04:212
2029721017,32sleep20-21swapper/207:34:002
13021210114,69sleep013108-21latency_hist12:18:500
22915210034,15sleep20-21swapper/209:24:202
305462999,72sleep330150-21apt-get09:48:523
169429910,18sleep20-21swapper/208:18:502
975629810,37sleep29794-21ntp_kernel_err10:24:162
2381929820,59sleep323877-21/usr/sbin/munin11:09:113
199172969,10sleep20-21swapper/207:32:432
83222959,73sleep28346-21cut12:03:502
83222959,73sleep28346-21cut12:03:502
74522955,47sleep20-21swapper/210:17:462
2147229510,23sleep211224-21munin-node09:19:522
1125329518,52sleep111437-21sort07:08:511
76522949,64sleep20-21swapper/210:18:512
46922946,13sleep10-21swapper/110:09:051
3264829411,67sleep032677-21fschecks_count09:55:070
292442946,74sleep229280-21ntpq09:44:142
2894829415,58sleep328955-21/usr/sbin/munin09:44:073
2855329418,49sleep10-21swapper/107:59:151
137652948,75sleep213840-21mailstats12:19:202
3244329311,68sleep332445-21sshd08:13:223
3244329311,68sleep332445-21sshd08:13:223
2732929316,60sleep327330-21cpu09:39:073
34382928,71sleep23440-21apt-get08:24:032
3112129222,46sleep031127-21modprobe11:33:470
2787029223,46sleep127998-21latency_hist07:58:501
220552929,23sleep222074-21apt-get11:04:012
220552929,23sleep222074-21apt-get11:04:012
132362927,26sleep20-21swapper/208:54:202
1169229223,14sleep20-21swapper/208:49:192
52632919,19sleep20-21swapper/208:29:122
1361629120,47sleep113743-21grep07:13:521
1361629120,47sleep113743-21grep07:13:521
1209029116,60sleep30-21swapper/312:14:163
2696829010,13sleep20-21swapper/207:54:192
244662905,50sleep20-21swapper/207:46:072
1394029016,49sleep00-21swapper/012:19:220
406728912,58sleep04120-21/usr/sbin/munin11:49:140
371628913,61sleep13783-21switchtime10:04:201
2858428915,59sleep028618-21awk09:43:480
267642897,15sleep20-21swapper/211:19:042
65732888,15sleep20-21swapper/208:34:052
65732888,15sleep20-21swapper/208:34:052
2838928812,20sleep20-21swapper/207:59:122
1739828810,56sleep00-21swapper/010:49:050
236052879,16sleep20-21swapper/211:09:022
327592869,57sleep20-21swapper/211:38:502
1934928622,45sleep019427-21find07:29:220
2313128524,38sleep123159-21modprobe07:43:481
2226528518,53sleep10-21swapper/111:04:131
2226528518,53sleep10-21swapper/111:04:131
242002849,57sleep20-21swapper/209:29:042
242002849,57sleep20-21swapper/209:29:042
1850928414,49sleep118596-21wc07:28:501
824928330,33sleep20-21swapper/206:59:072
2933028312,55sleep029336-21ntp_states09:44:160
2476628314,51sleep124834-21/usr/sbin/munin09:29:361
2476628314,51sleep124834-21/usr/sbin/munin09:29:361
508628214,50sleep35174-21/usr/sbin/munin08:29:093
1318928214,49sleep113234-21ntpq08:54:191
2098728119,42sleep121051-21sort09:19:381
1715828114,50sleep117302-21cat10:48:501
758928019,29sleep10-21swapper/110:18:501
38128018,39sleep1388-21kernelversion09:55:121
171732808,56sleep20-21swapper/207:24:032
168942806,63sleep20-21swapper/210:46:092
303512795,60sleep230517-21gpgv09:48:512
255852789,52sleep20-21swapper/211:14:152
2345127813,21sleep00-21swapper/011:08:520
977727718,35sleep09885-21grep08:44:080
2891427711,47sleep30-21swapper/307:59:213
1787627722,30sleep30-21swapper/309:09:153
1748127721,29sleep10-21swapper/109:09:041
119822774,12sleep30-21swapper/308:49:243
2235227615,22sleep30-21swapper/311:04:153
2235227615,22sleep30-21swapper/311:04:153
636327519,33sleep00-21swapper/010:14:090
3148527517,25sleep10-21swapper/108:09:161
3148527517,25sleep10-21swapper/108:09:161
2992827523,27sleep10-21swapper/108:04:111
2226827513,45sleep322305-21sed07:39:143
823227418,25sleep10-21swapper/108:39:081
633527427,26sleep10-21swapper/108:33:511
2839927410,48sleep128424-21fschecks_count11:24:071
2569927415,40sleep125840-21sed07:49:221
2083727420,23sleep00-21swapper/009:19:010
156702747,48sleep20-21swapper/212:24:252
1219327417,39sleep012195-21cron10:33:470
388527321,26sleep30-21swapper/311:49:093
323052737,52sleep00-21swapper/008:11:370
323052737,52sleep00-21swapper/008:11:370
1343227315,16sleep30-21swapper/308:54:253
111062735,12sleep20-21swapper/212:10:202
61827221,23sleep00-21swapper/008:14:120
61827221,23sleep00-21swapper/008:14:120
2653027221,30sleep10-21swapper/111:18:501
1561727212,41sleep015138-21apt-get07:19:000
1486127214,35sleep014940-21/usr/sbin/munin08:59:180
1281727210,12sleep00-21swapper/008:54:090
2524327121,33sleep025305-21cat07:49:120
1682227120,22sleep30-21swapper/312:29:193
1682227120,22sleep30-21swapper/312:29:193
1455927113,42sleep014696-21unixbench_multi07:14:360
1114127122,30sleep00-21swapper/007:08:500
935127023,24sleep30-21swapper/310:24:013
442027019,21sleep10-21swapper/111:49:231
2861227018,25sleep00-21swapper/007:59:180
121527037,16sleep20-21swapper/208:14:222
3180926912,16sleep10-21swapper/111:34:131
2333826920,26sleep00-21swapper/011:08:480
2333826920,26sleep00-21swapper/011:08:480
1573426919,25sleep10-21swapper/107:19:091
1485326918,32sleep30-21swapper/310:39:193
1478426918,21sleep30-21swapper/312:24:023
1439326923,21sleep10-21swapper/110:39:091
2626926813,38sleep026336-21mailstats09:34:230
2208026810,41sleep30-21swapper/309:23:203
2102126813,38sleep321100-21grep10:59:223
1617326812,39sleep116202-21irqstats09:04:121
1610826818,24sleep30-21swapper/310:44:123
1257226814,38sleep112705-21sed08:53:501
3013926720,30sleep330201-21/usr/sbin/munin08:04:183
1445926716,29sleep30-21swapper/308:59:093
106826720,21sleep10-21swapper/108:14:211
106826720,21sleep10-21swapper/108:14:211
815326615,36sleep38154-21sshd12:02:213
815326615,36sleep38154-21sshd12:02:213
43032667,48sleep30-21swapper/310:07:373
2037026613,36sleep120453-21sed09:18:501
2037026613,36sleep120453-21sed09:18:501
1767426614,34sleep30-21swapper/307:24:203
1156826617,27sleep311750-21fschecks_count07:08:533
1156826617,27sleep311750-21fschecks_count07:08:533
856426522,23sleep10-21swapper/112:04:051
582426510,41sleep35868-21ntpq11:54:143
278926516,32sleep12802-21vmstat08:19:221
2349926513,34sleep323503-21diskstats07:44:073
1373026522,25sleep10-21swapper/112:19:191
111172656,18sleep10-21swapper/112:10:271
3019726424,24sleep10-21swapper/109:48:501
2682126416,34sleep326909-21latency07:54:163
2570526415,24sleep025791-21cut11:14:210
169552647,17sleep00-21swapper/009:05:480
995226317,24sleep30-21swapper/307:04:093
372026318,25sleep00-21swapper/008:24:170
2986526312,38sleep129867-21cpu11:29:041
2388926312,18sleep10-21swapper/109:28:511
2277126314,27sleep00-21swapper/009:24:140
2032626315,29sleep00-21swapper/010:58:510
1915226318,31sleep319197-21/usr/sbin/munin07:29:173
93842628,14sleep00-21swapper/008:43:480
308026217,20sleep30-21swapper/311:44:413
308026217,20sleep30-21swapper/311:44:413
266962629,12sleep10-21swapper/109:34:281
1457226211,35sleep014666-21memory10:39:120
1314926213,34sleep313188-21ntpq10:34:353
984126110,36sleep09847-21ntp_states10:24:180
950226120,22sleep39534-21munin-run07:03:483
56702616,10sleep30-21swapper/310:10:163
47022619,38sleep34750-21unin-run08:28:493
2070226122,21sleep10-21swapper/107:34:151
111552614,13sleep00-21swapper/012:10:500
3254026016,27sleep332474-21apt-get09:54:453
2575326010,23sleep30-21swapper/309:34:023
2090126018,23sleep10-21swapper/110:59:161
1931126012,35sleep019340-21irqstats09:14:120
1931126012,35sleep019340-21irqstats09:14:120
307442596,41sleep00-21swapper/008:06:370
2713025918,25sleep327153-21/usr/sbin/munin11:19:133
23562595,35sleep00-21swapper/008:19:170
2095825911,22sleep30-21swapper/307:34:213
186325917,26sleep10-21swapper/109:59:161
1448025918,28sleep314553-21threads07:14:353
1305225917,21sleep10-21swapper/110:34:331
545525814,26sleep15449-21apt-get11:54:031
326222585,12sleep00-21swapper/011:37:510
2809425815,19sleep10-21swapper/111:23:511
1184525813,16sleep00-21swapper/007:09:060
1184525813,16sleep00-21swapper/007:09:060
318752575,15sleep10-21swapper/109:53:141
275552570,6sleep10-21swapper/109:39:121
1034925719,20sleep30-21swapper/308:44:193
78225615,26sleep0811-21irqstats11:39:120
1171425614,25sleep111708-21apt-get12:14:061
10222560,9sleep30-21swapper/311:39:203
754925511,24sleep10-21swapper/111:59:201
754925511,24sleep10-21swapper/111:59:201
1714825513,24sleep00-21swapper/010:48:490
1007725522,17sleep10-21swapper/108:44:131
863925414,24sleep38645-21diskstats12:04:073
76625415,24sleep3798-21ntp_states08:14:153
76625415,24sleep3798-21ntp_states08:14:153
595525411,31sleep0362-21snmpd10:13:440
312602547,13sleep30-21swapper/311:33:503
2363125413,24sleep123624-21apt-get11:09:051
2342125414,17sleep10-21swapper/107:44:051
1943725412,27sleep119469-21/usr/sbin/munin10:54:191
159425414,23sleep30-21swapper/309:59:053
216525314,18sleep30-21swapper/308:19:123
701425211,24sleep07008-21apt-get11:59:030
701425211,24sleep07008-21apt-get11:59:030
1719725213,23sleep017192-21apt-get07:24:050
163242526,35sleep10-21swapper/112:28:521
163242526,35sleep10-21swapper/112:28:521
1436725213,23sleep114362-21apt-get08:59:031
1334725214,18sleep30-21swapper/312:19:093
1064625213,25sleep110675-21munin-run10:28:471
885125111,24sleep08927-21/usr/sbin/munin06:59:220
3009325111,21sleep330131-21irqstats11:29:103
25652519,17sleep00-21swapper/011:44:290
25652519,17sleep00-21swapper/011:44:290
2230725112,24sleep022326-21ntp_states07:39:150
1905325114,23sleep319059-21diskstats10:54:093
1639225111,26sleep316407-21open_inodes09:04:183
1469925113,23sleep114778-21users07:14:371
923025013,18sleep00-21swapper/012:04:200
1906725012,22sleep319064-21apt-get09:14:033
1906725012,22sleep319064-21apt-get09:14:033
1552525015,16sleep10-21swapper/112:24:241
1486225014,17sleep00-21swapper/012:24:070
791624917,14sleep30-21swapper/310:19:123
74912499,18sleep30-21swapper/308:34:263
74912499,18sleep30-21swapper/308:34:263
2880724913,19sleep30-21swapper/311:24:173
790124810,15sleep00-21swapper/008:38:510
790124810,15sleep00-21swapper/008:38:510
2504724816,16sleep30-21swapper/307:49:073
753724711,17sleep30-21swapper/310:18:483
499024711,21sleep04986-21apt-get08:29:020
1256224710,21sleep012571-21munin-run08:53:490
912824611,16sleep00-21swapper/010:23:500
99524513,16sleep10-21swapper/111:39:181
855924510,13sleep30-21swapper/308:39:143
2647624410,19sleep31-21systemd11:18:483
1824824211,16sleep10-21swapper/110:49:231
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional