You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-14 - 07:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Fri Aug 14, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26131296210,937sleep015311-21kworker/0:121:12:120
525026079,579sleep35307-21/usr/sbin/munin23:30:133
11507254911,524sleep111538-21awk22:09:511
23015239512,370sleep123036-21dpkg19:20:031
13342227129,30sleep10-21swapper/122:15:041
1686224787,33sleep11689-21modprobe19:55:041
23284224059,164sleep123404-21latency_hist21:04:541
27013221212,49sleep10-21swapper/122:55:221
29531220711,178sleep129618-21latency19:40:141
32444219815,49sleep132511-21fschecks_time19:50:111
15378219717,47sleep115479-21timerandwakeup00:01:571
21695219612,168sleep221682-21apt-get22:40:092
5189219145,38sleep10-21swapper/120:05:141
5189219145,38sleep10-21swapper/120:05:141
14067219153,52sleep114104-21cron23:59:511
23987218311,161sleep124001-21awk21:05:221
29022218117,145sleep129150-21latency_hist19:39:541
29022218117,145sleep129150-21latency_hist19:39:541
3575218010,149sleep13607-21fschecks_count23:25:101
23337217912,57sleep123434-21fschecks_time22:45:121
22943217514,16sleep10-21swapper/100:25:151
22943217514,16sleep10-21swapper/100:25:151
11266216914,102sleep10-21swapper/123:50:101
5126216311,113sleep15177-21fschecks_count23:30:111
8195216115,131sleep18230-21latency20:15:151
31448216029,24sleep10-21swapper/119:45:231
26097216015,118sleep016272-1kworker/0:0H22:54:530
26097216015,118sleep016272-1kworker/0:0H22:54:530
2599215915,120sleep10-21swapper/121:40:231
2599215915,120sleep10-21swapper/121:40:231
21133215914,129sleep121137-21cron22:39:511
21133215914,129sleep121137-21cron22:39:511
12986215813,131sleep113030-21ntp_kernel_pll_20:30:171
15731215514,127sleep115778-21grep20:40:071
6956215416,44sleep10-21swapper/123:35:161
6956215416,44sleep10-21swapper/123:35:161
3009921539,13sleep20-21swapper/221:25:182
3009921539,13sleep20-21swapper/221:25:182
3271214844,65sleep13413-21latency_hist23:24:541
4118214412,120sleep14205-21switchtime21:45:241
4799214217,102sleep04813-21apt-get21:49:510
25299214212,92sleep125381-21tr22:50:211
25299214212,92sleep125381-21tr22:50:211
18161213910,52sleep118170-21forks22:30:101
856321389,51sleep18607-21ntpq23:40:171
19228213812,51sleep119264-21ntpq20:50:161
21722213613,73sleep121768-21date20:59:531
20046213414,104sleep220119-21awk00:15:232
11666213411,111sleep311706-21tr23:50:203
24959213210,84sleep125092-21sed00:30:241
6639213111,83sleep16726-21dpkg21:54:531
1703421318,108sleep10-21swapper/122:25:221
21210212913,25sleep10-21swapper/100:20:101
30833212818,42sleep10-21swapper/121:27:231
30833212818,42sleep10-21swapper/121:27:231
11565212816,97sleep111658-21grep20:25:211
3126121279,13sleep10-21swapper/123:10:081
23506212721,26sleep20-21swapper/219:20:202
20808212774,35sleep120850-21latency_hist19:14:541
27739212511,38sleep10-21swapper/122:59:541
20663212515,96sleep220694-21latency20:55:182
24550212321,87sleep224672-21latency_hist22:49:542
779212110,98sleep3790-21netstat21:35:163
779212110,98sleep3790-21netstat21:35:163
24309212121,80sleep024350-21uniq00:30:110
950821188,71sleep10-21swapper/120:20:041
16280211813,90sleep316340-21fschecks_time00:05:143
16280211813,90sleep316340-21fschecks_time00:05:143
1305521187,79sleep113087-21munin-run22:14:521
24779211619,77sleep2231-21systemd-journal21:09:512
1617211611,93sleep00-21swapper/021:39:090
1617211611,93sleep00-21swapper/021:39:090
28511211521,76sleep328591-21awk23:00:213
11020211411,86sleep011065-21latency_hist23:49:530
5241211114,78sleep35277-21ntp_states20:05:163
5241211114,78sleep35277-21ntp_states20:05:163
12891211114,77sleep312935-21fschecks_count23:55:093
11317211113,83sleep211341-21df_inode23:50:112
9985210919,68sleep210020-21grep23:45:142
20006210912,84sleep120044-21grep00:15:221
19842210815,70sleep00-21swapper/022:35:020
19842210815,70sleep00-21swapper/022:35:020
4803210611,80sleep04913-21cat23:29:540
32054210621,69sleep332192-21sed21:30:283
32054210621,69sleep332192-21sed21:30:283
17983210615,71sleep217967-21apt-get00:10:102
21514210514,78sleep121650-21sed22:39:551
8455210414,79sleep18416-21latency_hist21:59:541
26139210410,39sleep10-21swapper/119:30:081
26139210410,39sleep10-21swapper/119:30:081
1967210314,34sleep20-21swapper/221:40:042
1967210314,34sleep20-21swapper/221:40:042
250822999,36sleep10-21swapper/121:10:071
2867429810,69sleep00-21swapper/021:20:190
251882987,15sleep30-21swapper/321:10:133
223202977,32sleep10-21swapper/119:15:271
1802529719,61sleep318170-21unixbench_singl20:45:253
1770129710,31sleep10-21swapper/120:45:191
82129617,22sleep00-21swapper/023:15:200
281442966,11sleep30-21swapper/319:35:193
281442966,11sleep30-21swapper/319:35:193
2798029632,49sleep328079-21latency_hist21:19:543
1737429619,44sleep10-21swapper/100:08:521
1737429619,44sleep10-21swapper/100:08:521
1652829610,15sleep30-21swapper/322:25:073
1463729620,54sleep314622-21apt-get00:00:093
1228229612,72sleep212343-21sendmail_mailst22:10:212
981329518,43sleep10-21swapper/122:02:571
981329518,43sleep10-21swapper/122:02:571
732729518,45sleep10-21swapper/120:10:551
2776829520,43sleep10-21swapper/121:18:101
1947629517,58sleep019648-21latency_hist22:34:540
1906629520,42sleep10-21swapper/100:12:391
1498329510,45sleep10-21swapper/122:20:091
118442950,11sleep00-21swapper/023:50:240
282202946,48sleep10-21swapper/121:20:081
1958829416,51sleep00-21swapper/019:10:130
1312329410,68sleep013265-21date22:14:550
1067429416,43sleep10-21swapper/123:46:021
32782937,46sleep10-21swapper/120:00:091
32782937,46sleep10-21swapper/120:00:091
147029318,43sleep10-21swapper/123:17:511
1470129355,21sleep10-21swapper/120:35:221
323832928,10sleep10-21swapper/121:32:241
323832928,10sleep10-21swapper/121:32:241
297082927,48sleep10-21swapper/123:05:071
362629121,49sleep23644-21fschecks_count21:45:132
708229020,49sleep26600-21apt-get21:55:022
1078529011,63sleep010855-21sensors22:05:220
6712899,43sleep10-21swapper/121:35:141
6712899,43sleep10-21swapper/121:35:141
1941128911,17sleep30-21swapper/319:10:083
194072896,44sleep10-21swapper/119:10:061
550428810,63sleep25525-21awk23:30:202
2979428818,54sleep029826-21fschecks_count23:05:100
245902886,46sleep10-21swapper/119:25:081
245902886,46sleep10-21swapper/119:25:081
257452878,15sleep00-21swapper/000:35:030
2505728616,57sleep225063-21acpi21:10:042
1461628614,18sleep30-21swapper/320:35:203
2788428520,41sleep027903-21apt-get23:00:010
1839228529,27sleep20-21swapper/219:05:192
230092849,63sleep0367-21snmpd19:19:580
2040428422,32sleep20-21swapper/222:36:172
2040428422,32sleep20-21swapper/222:36:172
1114328419,34sleep30-21swapper/320:25:103
527928315,17sleep00-21swapper/020:05:170
527928315,17sleep00-21swapper/020:05:170
2944428313,18sleep30-21swapper/319:40:123
2829728322,41sleep228420-21timerandwakeup19:35:222
2829728322,41sleep228420-21timerandwakeup19:35:222
1410528318,45sleep214106-40mandb23:59:522
2842628214,50sleep228478-21/usr/sbin/munin21:20:132
2173028212,55sleep321759-21latency19:15:163
816228117,48sleep28156-21apt-get23:40:052
269692815,28sleep10-21swapper/100:38:561
1324328119,43sleep313349-21/usr/sbin/munin20:30:233
323927919,45sleep33241-21systemd-run23:24:523
181972797,12sleep20-21swapper/220:45:292
1594927914,50sleep216007-21cut00:04:532
3257227823,37sleep232589-21awk21:34:512
3257227823,37sleep232589-21awk21:34:512
1953827821,35sleep319531-21apt-get00:15:083
212042770,8sleep00-21swapper/020:55:270
664827521,32sleep26716-21latency20:10:152
558827515,34sleep00-21swapper/023:30:220
1541327524,33sleep215437-21munin-run20:39:512
3017127417,32sleep00-21swapper/021:25:200
3017127417,32sleep00-21swapper/021:25:200
502527319,26sleep30-21swapper/321:49:543
3259727317,29sleep20-21swapper/223:14:542
2322627315,41sleep323284-21/usr/sbin/munin19:20:133
2194327317,18sleep00-21swapper/022:41:040
1917027321,32sleep019205-21munin-run19:09:520
162327317,34sleep31677-21tr23:19:513
137927320,32sleep01180-21apt-get19:54:550
706827222,32sleep27119-21/usr/sbin/munin23:35:212
706827222,32sleep27119-21/usr/sbin/munin23:35:212
2476227212,46sleep324781-21apt-get22:50:033
2476227212,46sleep324781-21apt-get22:50:033
2267427213,45sleep222649-21/usr/sbin/munin00:25:082
2267427213,45sleep222649-21/usr/sbin/munin00:25:082
1132827213,44sleep211346-21sed20:25:132
2709327127,30sleep227127-21ps21:15:212
884427016,29sleep38667-21/usr/sbin/munin22:00:123
884427016,29sleep38667-21/usr/sbin/munin22:00:123
837427020,31sleep38393-21apt21:59:523
1938527020,29sleep319357-21apt-get22:34:533
3187426815,20sleep00-21swapper/023:10:230
2967426818,31sleep329207-21apt-get23:05:013
245626820,25sleep20-21swapper/219:55:462
1929526813,40sleep019346-21latency_hist00:14:520
1460326811,29sleep00-21swapper/000:00:020
3101826714,21sleep20-21swapper/223:09:532
2186626723,24sleep20-21swapper/219:15:202
1930026717,31sleep00-21swapper/020:50:190
1056126715,35sleep210616-21memory22:05:162
2321526620,27sleep023225-21find22:45:030
1476226622,21sleep30-21swapper/322:19:553
996626517,23sleep30-21swapper/320:20:183
2921926510,30sleep229307-21gpgv23:04:522
256792656,11sleep00-21swapper/019:27:410
256792656,11sleep00-21swapper/019:27:410
2357826514,33sleep00-21swapper/000:25:260
2357826514,33sleep00-21swapper/000:25:260
171626518,26sleep20-21swapper/223:19:542
1703926517,31sleep317082-21date20:44:533
1152426516,24sleep00-21swapper/020:25:200
1019326514,25sleep010329-21awk20:20:230
552426415,36sleep25573-21idleruntime21:50:222
3051726416,27sleep20-21swapper/219:44:512
2553126412,39sleep225609-21latency_hist00:34:532
216552646,38sleep30-21swapper/322:39:573
91622638,44sleep00-21swapper/023:41:290
85252638,14sleep30-21swapper/323:40:153
3078126316,28sleep230800-21apt-get19:45:042
2155126319,22sleep00-21swapper/019:15:130
208862639,39sleep020995-21sed00:19:530
1776126317,24sleep30-21swapper/300:09:553
166226316,30sleep31663-21/usr/sbin/munin19:55:023
866626211,35sleep28685-21apt-get22:00:032
866626211,35sleep28685-21apt-get22:00:032
683926219,18sleep30-21swapper/320:10:203
453926210,18sleep00-21swapper/020:04:520
453926210,18sleep00-21swapper/020:04:520
2015026220,19sleep30-21swapper/320:54:543
1466326218,20sleep00-21swapper/022:19:510
569926115,24sleep20-21swapper/220:05:232
569926115,24sleep20-21swapper/220:05:232
430126112,31sleep34393-21users23:25:253
3129226113,35sleep31-21systemd19:45:213
2712426121,20sleep20-21swapper/200:39:542
2413826115,18sleep00-21swapper/021:05:250
1758526116,20sleep20-21swapper/200:09:542
1758526116,20sleep20-21swapper/200:09:542
971326011,34sleep39781-21df23:45:073
2048026019,18sleep30-21swapper/320:55:153
167226014,28sleep01180-21apt-get19:55:030
1610426019,25sleep216152-21ntp_kernel_pll_20:40:162
708625912,21sleep00-21swapper/021:55:040
324325915,23sleep00-21swapper/023:24:530
3057225911,25sleep30-21swapper/321:25:273
3057225911,25sleep30-21swapper/321:25:273
1698225913,31sleep216985-21ntpq22:25:192
1637225916,23sleep00-21swapper/020:40:220
667125816,25sleep30-21swapper/323:35:103
667125816,25sleep30-21swapper/323:35:103
658525818,25sleep01-21systemd21:54:520
3133925816,26sleep331344-21diskstats23:10:103
2754925811,27sleep00-21swapper/022:59:170
1783825811,18sleep00-21swapper/022:29:530
2759325714,16sleep20-21swapper/222:59:512
2243625711,26sleep00-21swapper/000:24:540
1179525716,25sleep311791-21apt-get22:10:053
830425611,27sleep08329-21/usr/sbin/munin20:15:180
475025615,26sleep24791-21modprobe23:29:522
3172225610,27sleep030782-21/usr/sbin/munin19:45:260
2422125614,25sleep324216-21apt-get00:30:063
1335125612,25sleep013503-21/usr/sbin/munin20:30:240
458925520,18sleep20-21swapper/220:04:542
458925520,18sleep20-21swapper/220:04:542
2352225512,26sleep323517-21apt-get21:05:083
2331725516,24sleep323324-21diskstats22:45:103
1435825517,18sleep00-21swapper/020:35:130
1374225515,25sleep213777-21/usr/sbin/munin22:15:172
120525517,21sleep20-21swapper/219:54:532
1001125514,19sleep00-21swapper/022:04:530
1001125514,19sleep00-21swapper/022:04:530
2927225415,24sleep329274-21sshd21:22:163
278942545,37sleep00-21swapper/021:19:470
2616425413,23sleep226211-21df19:30:092
2616425413,23sleep226211-21df19:30:092
2498625419,18sleep30-21swapper/319:25:183
2498625419,18sleep30-21swapper/319:25:183
2215925416,17sleep20-21swapper/221:00:102
1989925416,23sleep219937-21ntpq19:10:192
1391625419,17sleep20-21swapper/220:34:552
2736725319,21sleep3367-21snmpd22:56:583
270812535,39sleep30-21swapper/319:30:513
270812535,39sleep30-21swapper/319:30:513
257642535,12sleep20-21swapper/219:28:482
257642535,12sleep20-21swapper/219:28:482
2284325321,17sleep30-21swapper/300:25:133
2284325321,17sleep30-21swapper/300:25:133
217525316,20sleep30-21swapper/321:40:133
217525316,20sleep30-21swapper/321:40:133
206725315,24sleep02076-21diskstats21:40:100
206725315,24sleep02076-21diskstats21:40:100
1619025314,22sleep016186-21apt-get00:05:070
1619025314,22sleep016186-21apt-get00:05:070
1542725313,24sleep015438-21ntp_states22:20:200
1155825315,20sleep30-21swapper/322:09:533
976725212,21sleep29783-21irqstats20:20:132
719525212,23sleep07272-21grep23:35:230
719525212,23sleep07272-21grep23:35:230
3243025215,22sleep332437-21df_inode19:50:093
1984025213,23sleep319357-21apt-get22:35:013
1984025213,23sleep319357-21apt-get22:35:013
919325111,22sleep29222-21munin-run20:19:522
557425115,22sleep35609-21irqstats21:50:233
2980725113,21sleep00-21swapper/019:40:210
2767925114,22sleep027675-21apt-get19:35:040
2767925114,22sleep027675-21apt-get19:35:040
2240025113,23sleep322419-21ntp_states21:00:153
221125115,17sleep20-21swapper/223:20:152
838625013,15sleep30-21swapper/320:15:213
638025011,20sleep06450-21df20:10:090
2613825012,22sleep026130-21apt-get19:30:070
2613825012,22sleep026130-21apt-get19:30:070
2122325012,19sleep20-21swapper/200:20:112
2112225012,21sleep321119-21apt-get00:20:043
1776825016,20sleep00-21swapper/020:45:220
1498525011,25sleep215054-21df22:20:102
335624914,18sleep30-21swapper/320:00:113
335624914,18sleep30-21swapper/320:00:113
248692499,15sleep20-21swapper/222:50:102
248692499,15sleep20-21swapper/222:50:102
167224912,20sleep21745-21systemd-run21:39:522
167224912,20sleep21745-21systemd-run21:39:522
2204924812,17sleep00-21swapper/021:00:080
1519124816,20sleep30-21swapper/322:20:153
3150224717,17sleep00-21swapper/021:30:170
3150224717,17sleep00-21swapper/021:30:170
136102479,14sleep00-21swapper/023:55:220
1926624610,20sleep219286-21ntp_states20:50:172
187842435,14sleep20-21swapper/222:30:232
1993924211,15sleep30-21swapper/319:10:203
256102397,14sleep30-21swapper/300:34:543
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional