You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-05 - 21:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Sun Apr 05, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29152284516,812sleep329175-21/usr/sbin/munin08:32:173
6913232313,294sleep06971-21/usr/sbin/munin07:22:130
1837223090,16sleep21439-21cyclictest07:57:232
2116722779,253sleep021203-21ntpq08:07:190
8202225058,177sleep28244-21cut12:36:532
21532246124,23sleep20-21swapper/207:07:212
29428224049,14sleep20-21swapper/208:32:242
10234222310,196sleep210259-21netstat07:32:162
182822090,29sleep20-21swapper/210:32:092
17424220413,152sleep217568-21latency_hist07:56:552
1132121978,175sleep06487-1kworker/0:0H11:02:080
26849219422,118sleep227058-21latency_hist08:26:552
3207219120,97sleep20-21swapper/207:11:572
29995219010,135sleep229969-21/usr/sbin/munin12:02:092
12198219021,147sleep312235-21latency09:22:153
23664218312,156sleep323806-21latency_hist08:16:553
25545218121,105sleep20-21swapper/210:02:312
25589217611,94sleep20-21swapper/211:47:152
27348217417,91sleep227410-21mailstats11:52:202
11195217416,141sleep311205-21systemd07:36:543
6769217035,117sleep16797-21/usr/sbin/munin10:47:151
2673321617,15sleep00-21swapper/010:07:180
19192216115,131sleep019286-21fschecks_time11:27:110
19192216115,131sleep019286-21fschecks_time11:27:110
24366216022,19sleep00-21swapper/008:17:200
4322215917,85sleep06487-1kworker/0:0H12:22:220
19406215917,129sleep219447-21http09:46:532
19406215917,129sleep219447-21http09:46:532
8931215640,89sleep20-21swapper/210:52:492
2606215620,17sleep10-21swapper/108:52:091
2606215620,17sleep10-21swapper/108:52:091
3191221548,126sleep218729-1kworker/2:1H08:42:092
3191221548,126sleep218729-1kworker/2:1H08:42:092
20678215437,96sleep220826-21grep09:47:292
7790215327,101sleep06487-1kworker/0:0H10:51:540
25209215210,127sleep225296-21grep08:21:542
7908215167,71sleep28029-21timerwakeupswit09:07:242
7908215167,71sleep28029-21timerwakeupswit09:07:242
21612214932,19sleep20-21swapper/211:33:112
6484214614,77sleep20-21swapper/207:21:552
14211214554,22sleep20-21swapper/211:11:552
3003721449,120sleep030118-21systemctl10:17:230
12001214125,21sleep20-21swapper/209:22:102
5819213913,103sleep20-21swapper/212:27:202
30391213722,101sleep330430-21ntpq12:02:183
31810213559,62sleep231894-21cat12:07:142
31810213559,62sleep231894-21cat12:07:142
3043421354,10sleep30-21swapper/308:37:093
16575213552,13sleep20-21swapper/211:17:202
10375213530,88sleep210482-21sort10:57:222
3914213473,45sleep20-21swapper/210:37:232
14786213453,67sleep214841-21latency_hist09:31:552
12175213423,31sleep212320-21unixbench_singl07:37:252
29216213391,24sleep229257-21munin-run10:16:522
2312213172,41sleep20-21swapper/212:17:112
6947213013,97sleep26950-21cron09:06:522
6947213013,97sleep26950-21cron09:06:522
11688212633,23sleep211748-21latency_hist09:21:542
19492212460,19sleep20-21swapper/208:02:162
19924212312,96sleep320072-21unixbench_singl08:02:253
15685212258,26sleep215747-21grep11:16:512
15685212258,26sleep215747-21grep11:16:512
24858212130,57sleep324840-21apt-get10:02:123
15110212062,20sleep215186-21wc07:47:222
569211913,81sleep20-21swapper/210:27:182
7387211511,78sleep20-21swapper/207:22:232
22819211510,21sleep20-21swapper/209:56:542
17209211518,80sleep217317-21timerandwakeup09:37:242
10573211527,71sleep110630-21if_eth009:17:121
790211429,25sleep2842-21ls12:12:102
20734211415,83sleep320798-21sort11:32:153
5417211317,73sleep15509-21fschecks_time12:27:111
2317210711,80sleep22436-21latency_hist08:51:542
2317210711,80sleep22436-21latency_hist08:51:542
24382210616,76sleep124445-21sendmail_mailqu08:17:211
8294210317,22sleep21439-21cyclictest07:27:032
8594210213,73sleep08750-21date09:11:540
21454210214,71sleep321455-21munin-node09:52:033
30841210116,66sleep330955-21latency_hist10:21:543
25695210014,69sleep025739-21ntpq11:47:180
1738210019,64sleep01770-21tune2fs07:07:110
13368210013,72sleep113412-21ntp_kernel_pll_07:42:171
937229916,69sleep28770-21/usr/sbin/munin09:12:212
2758029913,69sleep027594-21/usr/sbin/munin08:28:180
2724129920,58sleep127225-21apt-get08:27:101
237662995,80sleep20-21swapper/211:42:072
1876629917,67sleep018899-21timerwakeupswit09:42:230
1876629917,67sleep018899-21timerwakeupswit09:42:230
124932997,51sleep20-21swapper/211:05:492
107512998,14sleep00-21swapper/009:17:150
413429811,17sleep20-21swapper/208:57:072
413429811,17sleep20-21swapper/208:57:072
734929614,68sleep37453-21cut09:07:133
734929614,68sleep37453-21cut09:07:133
312572969,14sleep20-21swapper/208:37:252
61229525,55sleep3683-21wc07:02:223
423029512,64sleep10-21swapper/112:22:191
2221829515,25sleep20-21swapper/211:37:062
402729420,58sleep34174-21sed10:37:243
199229322,23sleep10-21swapper/112:16:541
55422927,13sleep00-21swapper/007:17:160
3238829216,53sleep20-21swapper/210:26:542
2554529216,54sleep325653-21fschecks_time08:22:113
135442918,64sleep20-21swapper/207:42:222
2681429011,12sleep20-21swapper/210:07:202
1849929011,63sleep118120-21/usr/sbin/munin09:42:151
1849929011,63sleep118120-21/usr/sbin/munin09:42:151
295342896,17sleep229525-21apt-get10:17:092
2314628913,57sleep023222-21fschecks_time09:57:120
2314628913,57sleep023222-21fschecks_time09:57:120
1754428821,39sleep017540-21apt-get11:22:020
176302877,65sleep217644-21grep11:22:082
2425828641,35sleep20-21swapper/210:01:062
2425828641,35sleep20-21swapper/210:01:062
1794428627,44sleep06487-1kworker/0:0H09:41:540
1249028613,56sleep30-21swapper/307:39:123
729628411,57sleep17385-21cat07:22:221
37762849,62sleep21439-21cyclictest12:22:082
2975528413,48sleep329718-21latency_hist12:01:543
236532846,12sleep20-21swapper/208:16:542
1323828416,50sleep012843-21/usr/sbin/munin11:07:170
1152628417,34sleep10-21swapper/107:37:121
48472837,14sleep20-21swapper/210:42:032
272002836,13sleep20-21swapper/208:26:592
227402838,13sleep00-21swapper/009:56:520
193552838,12sleep20-21swapper/211:27:132
193552838,12sleep20-21swapper/211:27:132
1731328317,43sleep017437-21cat11:21:540
297442829,12sleep20-21swapper/212:01:532
1812328217,21sleep10-21swapper/111:22:201
312228011,52sleep20-21swapper/208:52:222
312228011,52sleep20-21swapper/208:52:222
1803128013,43sleep317637-21/usr/sbin/munin07:57:163
1375828016,29sleep00-21swapper/007:42:240
944627919,41sleep19460-21modprobe10:56:521
889027915,21sleep00-21swapper/009:12:110
422027921,36sleep34302-21tune2fs08:57:123
422027921,36sleep34302-21tune2fs08:57:123
2302627927,28sleep00-21swapper/008:12:230
210312795,13sleep30-21swapper/309:49:483
73242785,10sleep20-21swapper/210:47:322
73232787,20sleep00-21swapper/009:07:100
2677327817,41sleep126813-21grep10:07:191
1649527819,41sleep116574-21/usr/sbin/munin11:17:191
1205327815,43sleep112139-21/usr/sbin/munin09:22:131
1171827813,42sleep311293-21/usr/sbin/munin11:02:193
1022427816,15sleep30-21swapper/310:57:183
202227713,18sleep10-21swapper/107:07:161
1509227722,30sleep00-21swapper/009:32:090
3260727612,39sleep132649-21sort07:02:121
3110227626,29sleep10-21swapper/108:37:241
2903327611,13sleep10-21swapper/108:32:151
2506027615,47sleep025121-21if_eth010:02:210
1606327628,25sleep00-21swapper/007:52:050
1006427618,38sleep110125-21sendmail-msp09:16:521
994627511,43sleep010004-21if_err_eth010:57:110
3162527525,25sleep30-21swapper/312:07:113
3162527525,25sleep30-21swapper/312:07:113
2054527513,45sleep020697-21ls08:06:540
1790327515,46sleep317939-21cat09:41:533
709027419,36sleep37113-21netstat07:22:163
2968127414,19sleep00-21swapper/008:32:260
218312745,11sleep20-21swapper/208:09:232
1826927422,27sleep10-21swapper/107:57:231
170027413,46sleep11718-21apt10:32:011
1548027445,13sleep20-21swapper/209:32:192
1531527416,26sleep00-21swapper/007:47:240
2135327312,36sleep121445-21threads08:07:241
2086627318,36sleep320961-21fschecks_time08:07:143
108827319,21sleep00-21swapper/008:47:120
108827319,21sleep00-21swapper/008:47:120
617727227,22sleep10-21swapper/109:02:231
617727227,22sleep10-21swapper/109:02:231
53727214,27sleep10-21swapper/110:27:171
296227219,35sleep3222-21systemd-journal07:11:543
2752827218,26sleep30-21swapper/308:28:173
906927116,23sleep30-21swapper/309:12:143
427627121,23sleep10-21swapper/110:38:111
1014027110,13sleep10-21swapper/107:32:151
684427023,25sleep36863-21apt-get12:32:023
145722709,15sleep30-21swapper/311:12:123
145722709,15sleep30-21swapper/311:12:123
701326920,30sleep00-21swapper/009:06:540
701326920,30sleep00-21swapper/009:06:540
3154226919,25sleep00-21swapper/012:07:080
3154226919,25sleep00-21swapper/012:07:080
1029826911,41sleep00-21swapper/007:32:180
3131926822,26sleep00-21swapper/010:22:150
149952686,12sleep30-21swapper/307:47:193
1201626813,35sleep011902-21/usr/sbin/munin09:22:120
116642689,43sleep011713-21irqstats07:37:150
503426716,32sleep05046-21vmstat08:57:250
503426716,32sleep05046-21vmstat08:57:250
3271426722,27sleep00-21swapper/010:27:120
3166926720,25sleep00-21swapper/008:41:550
1432926720,22sleep30-21swapper/307:46:553
118526720,21sleep10-21swapper/108:47:131
118526720,21sleep10-21swapper/108:47:131
3207026611,41sleep10-21swapper/110:23:251
2701226515,35sleep327114-21timerandwakeup10:07:253
885526412,25sleep00-21swapper/007:27:220
782726419,29sleep17906-21ls09:07:231
782726419,29sleep17906-21ls09:07:231
521326419,28sleep15222-21/usr/sbin/munin10:42:151
493926418,21sleep30-21swapper/307:16:553
2676026416,19sleep10-21swapper/108:26:541
1734326421,26sleep317344-21cron07:56:523
68926315,14sleep00-21swapper/007:02:230
649826317,21sleep30-21swapper/307:21:563
540926311,22sleep00-21swapper/012:27:100
298826318,25sleep30-21swapper/310:36:523
201262635,13sleep10-21swapper/108:02:271
1869326320,20sleep30-21swapper/309:42:223
1869326320,20sleep30-21swapper/309:42:223
952826212,33sleep09604-21sed10:56:550
545126217,27sleep15591-21sed09:01:541
545126217,27sleep15591-21sed09:01:541
528926221,22sleep30-21swapper/312:26:573
367726216,21sleep00-21swapper/010:37:170
264652629,14sleep10-21swapper/111:50:291
1949326215,23sleep10-21swapper/111:27:161
1949326215,23sleep10-21swapper/111:27:161
1579526221,23sleep30-21swapper/309:32:243
1140826222,20sleep10-21swapper/111:02:131
108326211,21sleep30-21swapper/308:47:113
108326211,21sleep30-21swapper/308:47:113
859026125,18sleep30-21swapper/310:52:423
569126120,25sleep30-21swapper/309:02:083
569126120,25sleep30-21swapper/309:02:083
2577826113,32sleep325801-21/usr/sbin/munin11:47:213
1371526113,33sleep113747-21irqstats09:27:141
1936926014,26sleep319463-21memory11:27:143
1936926014,26sleep319463-21memory11:27:143
153292605,13sleep10-21swapper/111:12:261
153292605,13sleep10-21swapper/111:12:261
132626021,25sleep21354-21munin-run07:06:532
3812599,39sleep00-21swapper/008:42:430
3812599,39sleep00-21swapper/008:42:430
2852425917,23sleep00-21swapper/011:57:110
2403625917,22sleep10-21swapper/111:42:141
178302596,10sleep10-21swapper/109:41:311
123625911,29sleep00-21swapper/012:12:210
76225818,23sleep3853-21awk10:27:233
508425816,25sleep35120-21irqstats10:42:133
3067125817,19sleep10-21swapper/112:02:241
253125816,25sleep02550-21apt-get08:52:040
253125816,25sleep02550-21apt-get08:52:040
248032589,38sleep00-21swapper/011:44:030
222925815,17sleep30-21swapper/312:17:083
2101725816,27sleep00-21swapper/011:32:200
2961125712,21sleep10-21swapper/110:17:121
2834725715,19sleep10-21swapper/110:12:151
2249825717,22sleep10-21swapper/108:12:121
1972925719,18sleep10-21swapper/109:46:561
106922576,16sleep10-21swapper/110:57:371
103325715,24sleep30-21swapper/312:12:163
892725618,23sleep19022-21cat07:27:231
75525617,16sleep30-21swapper/308:46:543
75525617,16sleep30-21swapper/308:46:543
289032566,41sleep00-21swapper/010:12:590
2803025615,18sleep30-21swapper/310:12:093
22252567,32sleep00-21swapper/012:17:050
1557925618,21sleep10-21swapper/109:32:221
1347625623,18sleep30-21swapper/309:27:073
2722925519,16sleep10-21swapper/111:52:151
2710425514,26sleep327133-21irqstats11:52:133
2376125515,23sleep323759-21apt-get11:42:043
2305725513,25sleep323050-21apt-get09:57:073
2305725513,25sleep323050-21apt-get09:57:073
530625412,25sleep05312-21ntp_states10:42:180
3236425414,24sleep132370-21ntp_states08:42:201
3236425414,24sleep132370-21ntp_states08:42:201
28922545,16sleep00-21swapper/010:35:400
2616325418,19sleep10-21swapper/110:06:541
2417425417,21sleep30-21swapper/308:17:153
2351725414,24sleep023526-21unin-run11:41:530
2221925417,22sleep322286-21df11:37:073
2189525415,17sleep10-21swapper/111:36:491
1448625416,16sleep00-21swapper/011:12:090
1448625416,16sleep00-21swapper/011:12:090
1295525412,23sleep313021-21fschecks_time11:07:123
744625318,17sleep10-21swapper/112:32:201
331125318,17sleep00-21swapper/007:11:590
2924825313,17sleep10-21swapper/111:57:241
2686225317,19sleep00-21swapper/011:52:060
1472525315,20sleep10-21swapper/107:47:131
1417625314,24sleep114210-21cat11:11:541
517425212,23sleep15171-21apt-get07:17:051
3153825212,24sleep131533-21apt-get12:07:051
3153825212,24sleep131533-21apt-get12:07:051
2999325214,22sleep029988-21apt-get12:02:070
1897825212,18sleep00-21swapper/008:01:540
1645525214,22sleep316461-21ntp_states11:17:173
2274125113,18sleep10-21swapper/109:56:531
2248725115,18sleep10-21swapper/111:37:131
1791425111,23sleep317522-21/usr/sbin/munin11:22:133
2544725012,22sleep025443-21apt-get08:22:040
832224914,19sleep38389-21df07:27:083
282924917,16sleep30-21swapper/308:52:153
282924917,16sleep30-21swapper/308:52:153
2436324913,18sleep10-21swapper/110:01:541
2436324913,18sleep10-21swapper/110:01:541
2016024911,22sleep020212-21irqstats09:47:190
377224811,21sleep33769-21apt-get12:22:053
35672489,19sleep10-21swapper/107:12:051
1395924811,18sleep00-21swapper/009:27:210
70642469,21sleep07117-21if_eth012:32:110
3109924513,15sleep30-21swapper/310:22:103
230724511,20sleep12311-21diskstats12:17:101
1608524512,19sleep116082-21apt-get07:52:061
1038124411,16sleep30-21swapper/309:17:063
69352439,15sleep30-21swapper/310:47:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional