You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-08 - 21:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Jul 08, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13183285868,8561sleep07908-21kworker/0:112:28:450
13183285868,8561sleep07908-21kworker/0:112:28:450
130432766310,7637sleep211060-21kworker/2:112:28:432
130432766310,7637sleep211060-21kworker/2:112:28:432
21378218349,1800sleep230915-21kworker/u8:209:29:252
13256215835,1558sleep13149-21kworker/1:212:28:461
13256215835,1558sleep13149-21kworker/1:212:28:461
16632288316,849sleep116671-21/usr/sbin/munin09:14:511
16632288316,849sleep116671-21/usr/sbin/munin09:14:511
2021026757,652sleep020259-21ntpq07:44:320
15913266422,624sleep215919-21/usr/sbin/munin10:54:302
576624839,337sleep029279-21kworker/u8:012:04:350
15920247326,429sleep315965-21/usr/sbin/munin10:54:313
14037242417,390sleep314076-21ntpq12:29:293
9234229033,199sleep00-21swapper/007:10:170
15152223851,82sleep00-21swapper/007:29:220
32266223612,209sleep232268-21sshd08:21:192
687922199,168sleep06969-21grep08:44:260
687922199,168sleep06969-21grep08:44:260
298212214132,19sleep00-21swapper/008:14:270
298212214132,19sleep00-21swapper/008:14:270
10535221215,91sleep010701-21unixbench_multi12:19:370
10535221215,91sleep010701-21unixbench_multi12:19:370
4311220328,147sleep00-21swapper/008:34:370
259022026,181sleep00-21swapper/008:29:330
14363218212,151sleep014515-21latency_hist09:09:090
10324217310,100sleep013215-1kworker/0:0H08:54:350
1876921707,14sleep00-21swapper/011:04:250
4917216810,128sleep00-21swapper/012:04:070
31364216710,61sleep00-21swapper/009:59:350
2714121587,94sleep00-21swapper/009:44:540
2714121587,94sleep00-21swapper/009:44:540
2476821558,84sleep024776-21forks09:39:410
1181621539,18sleep00-21swapper/007:19:210
20822215214,123sleep020930-21latency_hist09:29:070
1934421435,126sleep00-21swapper/007:41:370
1934421435,126sleep00-21swapper/007:41:370
6782214219,109sleep26784-21entropy07:04:302
6782214219,109sleep26784-21entropy07:04:302
2848214217,102sleep00-21swapper/010:14:080
2738213990,33sleep02888-21unixbench_singl11:54:380
22012213818,105sleep222142-21sed07:49:382
22012213818,105sleep222142-21sed07:49:382
1655321319,37sleep00-21swapper/012:39:070
5235212752,59sleep00-21swapper/008:39:220
24662212732,64sleep124649-21apt-get09:39:211
16810212712,54sleep016908-21grep07:34:270
12722212416,40sleep00-21swapper/010:44:270
7321212252,54sleep07449-21cat12:09:360
128421208,51sleep01298-21munin-run10:09:070
15080211549,55sleep08957-21kworker/0:010:51:500
8419211415,81sleep28546-21/usr/sbin/munin10:29:362
3691211422,66sleep33780-21fschecks_time11:59:263
60521138,84sleep00-21swapper/008:24:560
28161211311,80sleep00-21swapper/011:34:240
23891211226,65sleep224040-21head07:54:472
15832211219,74sleep115946-21timerandwakeup12:34:361
10663211220,75sleep210800-21latency_hist10:39:082
7530211017,76sleep37499-21latency_hist10:29:083
7530211017,76sleep37499-21latency_hist10:29:083
29878211010,83sleep129566-21/usr/sbin/munin11:39:291
518421090,6sleep10-21swapper/108:39:091
21876210973,18sleep00-21swapper/007:49:360
21876210973,18sleep00-21swapper/007:49:360
27801210832,63sleep027827-21munin-run08:09:060
27801210832,63sleep027827-21munin-run08:09:060
22482210717,72sleep30-21swapper/309:33:143
1736210729,66sleep01763-21munin-run11:54:060
1081210714,75sleep21173-21timerandwakeup11:49:362
7300210627,61sleep17378-21timerandwakeup07:04:411
7300210627,61sleep17378-21timerandwakeup07:04:411
26565210618,73sleep226671-21fschecks_time11:29:252
1367021067,12sleep20-21swapper/212:29:202
8345210530,14sleep00-21swapper/008:49:230
30585210318,72sleep00-21swapper/011:40:420
5233210220,41sleep30-21swapper/308:39:203
2106210222,47sleep22100-21apt-get08:29:182
17333210215,72sleep117384-21/usr/sbin/munin10:59:261
14842210213,68sleep00-21swapper/009:09:250
14842210213,68sleep00-21swapper/009:09:250
13681210213,69sleep00-21swapper/007:24:250
3087210146,15sleep00-21swapper/010:14:200
23248210125,59sleep00-21swapper/007:54:250
6699210016,62sleep36673-21/usr/sbin/munin07:04:273
6699210016,62sleep36673-21/usr/sbin/munin07:04:273
32350210028,62sleep0364-21snmpd08:22:220
25432210019,64sleep325551-21unixbench_multi07:59:373
24221210045,28sleep00-21swapper/009:39:080
237412996,51sleep30-21swapper/311:19:293
2646629823,58sleep00-21swapper/011:29:190
163402989,69sleep00-21swapper/009:14:170
163402989,69sleep00-21swapper/009:14:170
1587529716,65sleep215941-21apt-key09:14:072
1587529716,65sleep215941-21apt-key09:14:072
1400229720,56sleep10-21swapper/110:49:201
1420429612,64sleep00-21swapper/012:29:340
1360129615,61sleep313695-21/usr/sbin/munin09:04:363
1247729618,59sleep00-21swapper/009:00:490
872329420,13sleep00-21swapper/012:14:320
3088629420,45sleep230878-21apt-get09:59:202
2336829411,16sleep00-21swapper/011:19:180
1273129421,56sleep112732-21cron09:04:051
959529311,18sleep00-21swapper/010:34:260
2916129329,9sleep00-21swapper/008:11:400
2916129329,9sleep00-21swapper/008:11:400
2125629323,13sleep00-21swapper/009:29:110
1236529315,58sleep312434-21grep07:19:363
3134629213,53sleep30-21swapper/311:44:163
1827129222,13sleep00-21swapper/011:02:310
1138229220,38sleep00-21swapper/007:17:390
3244429113,39sleep20-21swapper/210:04:212
2901929117,57sleep00-21swapper/009:54:060
2843329127,44sleep327732-21/usr/sbin/munin09:49:373
2123229122,12sleep00-21swapper/011:10:430
1795329134,41sleep00-21swapper/009:19:180
1795329134,41sleep00-21swapper/009:19:180
1341829115,58sleep213458-21ntp_kernel_pll_09:04:312
747729036,39sleep07480-21cron10:29:050
747729036,39sleep07480-21cron10:29:050
1609929019,15sleep00-21swapper/010:54:360
464528910,18sleep00-21swapper/010:19:160
2664928918,51sleep126747-21fschecks_time08:04:351
2664928918,51sleep126747-21fschecks_time08:04:351
258512885,17sleep00-21swapper/008:02:040
1176828818,52sleep311795-21netstat12:24:283
1176828818,52sleep311795-21netstat12:24:283
313982877,15sleep00-21swapper/011:44:330
3043928723,11sleep00-21swapper/009:57:290
2312328715,58sleep123156-21cut11:19:071
873028618,11sleep00-21swapper/010:30:100
1999128615,20sleep20-21swapper/211:09:072
1209728619,10sleep00-21swapper/010:43:180
1179628613,56sleep211835-21memory08:59:302
3149928520,46sleep131593-21/usr/sbin/munin08:19:281
2282428519,12sleep00-21swapper/011:16:110
2869428421,38sleep328842-21grep11:34:353
1497928430,25sleep10-21swapper/112:34:071
675528320,43sleep36759-21find08:44:173
675528320,43sleep36759-21find08:44:173
2612628319,11sleep00-21swapper/011:28:260
668428219,41sleep16719-21grep10:24:341
668428219,41sleep16719-21grep10:24:341
598228224,26sleep30-21swapper/310:24:083
2972928224,30sleep30-21swapper/308:14:263
2972928224,30sleep30-21swapper/308:14:263
2674928219,35sleep20-21swapper/208:04:362
2674928219,35sleep20-21swapper/208:04:362
2629728215,17sleep30-21swapper/308:04:093
2629728215,17sleep30-21swapper/308:04:093
1719728216,46sleep116674-21/usr/sbin/munin07:34:361
563028111,50sleep15206-21/usr/sbin/munin08:39:311
2410128120,42sleep224154-21modprobe09:39:062
121628123,30sleep10-21swapper/108:25:141
3161528023,30sleep30-21swapper/309:59:383
2806528017,19sleep30-21swapper/308:09:173
2806528017,19sleep30-21swapper/308:09:173
2467928014,20sleep10-21swapper/111:24:081
1772028016,49sleep217875-21cat09:19:082
1772028016,49sleep217875-21cat09:19:082
3251727915,45sleep332553-21munin-run08:24:063
1422027921,33sleep314328-21grep07:24:453
1163427930,22sleep30-21swapper/310:39:383
3063627817,14sleep10-21swapper/109:59:071
392627725,31sleep30-21swapper/308:34:283
1526127721,28sleep10-21swapper/109:09:341
1526127721,28sleep10-21swapper/109:09:341
3252227625,25sleep10-21swapper/110:04:241
2330127614,22sleep10-21swapper/107:54:341
1278127614,17sleep30-21swapper/309:04:073
2239027522,23sleep20-21swapper/211:14:362
801827412,20sleep00-21swapper/007:09:080
801827412,20sleep00-21swapper/007:09:080
2169727423,26sleep10-21swapper/109:29:321
178772740,7sleep30-21swapper/309:19:083
178772740,7sleep30-21swapper/309:19:083
3085127316,31sleep20-21swapper/211:44:062
1804227314,37sleep20-21swapper/209:19:242
1804227314,37sleep20-21swapper/209:19:242
178552736,12sleep20-21swapper/207:37:482
1186627312,31sleep20-21swapper/207:19:232
336727221,33sleep1225-21systemd-journal08:34:061
317027226,25sleep30-21swapper/310:14:233
3024227219,38sleep230320-21find08:14:372
3024227219,38sleep230320-21find08:14:372
2906927214,42sleep229171-21latency_hist09:54:082
2633427215,22sleep10-21swapper/109:44:261
2633427215,22sleep10-21swapper/109:44:261
328327119,23sleep30-21swapper/311:59:053
3158727115,28sleep20-21swapper/211:44:392
1617027119,22sleep10-21swapper/110:54:371
969227022,24sleep10-21swapper/108:54:081
814327019,40sleep38150-21needreboot10:29:283
62412705,10sleep10-21swapper/112:06:501
341827015,39sleep33490-21date08:34:083
209812707,16sleep20-21swapper/207:47:452
1025527015,26sleep20-21swapper/212:19:312
1025527015,26sleep20-21swapper/212:19:312
3256626916,38sleep332652-21fw_packets10:04:263
2480526915,40sleep324860-21fschecks_time09:39:443
2342026924,26sleep10-21swapper/109:34:371
207652699,46sleep320818-21dpkg09:29:063
96002688,44sleep39644-21systemd-run08:54:063
836426821,20sleep20-21swapper/207:09:262
1953826815,39sleep319582-21systemd-run07:44:063
1953826815,39sleep319582-21systemd-run07:44:063
1418426824,23sleep20-21swapper/210:49:262
1010726821,21sleep20-21swapper/207:14:142
941226727,17sleep10-21swapper/110:34:231
416126719,34sleep24240-21run-parts08:34:352
2534826721,22sleep20-21swapper/207:59:362
2415626723,21sleep30-21swapper/309:39:073
229626727,20sleep20-21swapper/211:54:282
1032026718,23sleep10-21swapper/112:19:341
1032026718,23sleep10-21swapper/112:19:341
2571526614,35sleep225819-21users11:24:372
1778326623,26sleep30-21swapper/310:59:363
1646026619,33sleep216564-21latency_hist07:34:082
125162667,15sleep20-21swapper/210:44:212
61482659,20sleep20-21swapper/208:39:402
2940126523,25sleep10-21swapper/108:14:081
2940126523,25sleep10-21swapper/108:14:081
2828326524,19sleep20-21swapper/211:34:262
2187226516,21sleep30-21swapper/311:14:253
2154626517,21sleep20-21swapper/211:14:082
21432657,15sleep10-21swapper/110:09:371
2081126520,20sleep10-21swapper/111:09:351
1805326519,23sleep30-21swapper/309:19:253
1805326519,23sleep30-21swapper/309:19:253
1736726518,31sleep317483-21timerwakeupswit07:34:383
1594726517,23sleep20-21swapper/212:34:372
1338926521,21sleep10-21swapper/109:04:291
73026417,30sleep2802-21fschecks_time08:25:022
71732647,11sleep10-21swapper/108:44:311
71732647,11sleep10-21swapper/108:44:311
597126410,39sleep25981-21munin-run10:24:072
2835926422,18sleep20-21swapper/208:09:282
2835926422,18sleep20-21swapper/208:09:282
2605026421,26sleep126081-21munin-run08:04:071
2541826418,21sleep30-21swapper/311:24:343
103562647,12sleep20-21swapper/208:54:372
640626322,21sleep20-21swapper/210:24:272
640626322,21sleep20-21swapper/210:24:272
2060326320,28sleep320626-21/usr/sbin/munin11:09:293
1335126315,30sleep113087-21apt-get07:24:101
1151726317,30sleep211633-21sed10:39:372
891226220,21sleep10-21swapper/107:09:371
725126214,29sleep37319-21/usr/sbin/munin12:09:353
3091926224,20sleep30-21swapper/311:44:073
2815826220,22sleep10-21swapper/109:49:311
1528226216,28sleep115362-21fw_packets07:29:271
1455426213,27sleep30-21swapper/310:49:353
1926726119,19sleep10-21swapper/109:24:071
1926726119,19sleep10-21swapper/109:24:071
1369126114,27sleep10-21swapper/107:24:261
894626018,20sleep20-21swapper/208:49:372
61892605,11sleep20-21swapper/212:06:112
2841726013,30sleep128466-21/usr/sbin/munin11:34:291
197202606,11sleep30-21swapper/311:06:143
181026015,29sleep31966-21sed11:54:093
1100526020,23sleep10-21swapper/108:57:191
745025924,18sleep10-21swapper/112:09:371
47462596,10sleep20-21swapper/212:02:312
3098925917,19sleep10-21swapper/108:19:081
3098925917,19sleep10-21swapper/108:19:081
2559425922,17sleep10-21swapper/111:24:361
1958525913,27sleep219591-21diskstats09:24:252
134142597,38sleep30-21swapper/310:45:193
1868825812,20sleep10-21swapper/111:04:221
108725812,12sleep30-21swapper/308:25:123
810325716,26sleep28198-21latency_hist08:49:082
810325716,26sleep28198-21latency_hist08:49:082
776425715,26sleep17758-21apt-get10:29:181
266732576,11sleep30-21swapper/311:29:263
1594225722,18sleep30-21swapper/309:14:083
1594225722,18sleep30-21swapper/309:14:083
871125619,21sleep38718-21ntp_states12:14:313
482425614,22sleep10-21swapper/112:03:311
3149525617,26sleep131581-21fschecks_time11:44:381
2671625616,25sleep226803-21mailstats09:44:352
2671625616,25sleep226803-21mailstats09:44:352
2337225615,23sleep223374-21cpu11:19:202
2137925616,23sleep121371-21apt-get07:49:211
2137925616,23sleep121371-21apt-get07:49:211
1712125613,26sleep217117-21apt-get10:59:182
162525617,18sleep20-21swapper/210:09:252
1402125614,24sleep214066-21ntp_states07:24:402
974225512,16sleep30-21swapper/310:34:293
826625514,26sleep18334-21/usr/sbin/munin12:14:211
1990825512,18sleep10-21swapper/107:44:251
192822557,34sleep10-21swapper/107:40:491
192822557,34sleep10-21swapper/107:40:491
1559825521,19sleep30-21swapper/307:29:333
1116325517,25sleep311268-21cat12:24:083
1116325517,25sleep311268-21cat12:24:083
645725413,24sleep36467-21munin-run12:09:063
3220425415,23sleep132235-21sed10:04:081
2976325415,16sleep30-21swapper/309:54:333
2655425419,19sleep10-21swapper/111:29:241
1242525414,23sleep112443-21apt-get10:44:161
466725313,23sleep14664-21apt-get10:19:171
34822539,30sleep13520-21ntpq10:14:291
336325315,19sleep10-21swapper/111:59:071
2680425320,18sleep30-21swapper/309:44:363
2680425320,18sleep30-21swapper/309:44:363
242902536,11sleep30-21swapper/307:57:223
2269925312,26sleep322781-21munin-run07:54:073
2269925312,26sleep322781-21munin-run07:54:073
118642536,35sleep10-21swapper/110:40:161
1038025319,17sleep30-21swapper/308:54:383
876025214,18sleep30-21swapper/307:09:353
2093125214,19sleep10-21swapper/109:29:081
1561225211,17sleep30-21swapper/312:34:303
152925211,25sleep31531-21apt-get10:09:183
1068425212,16sleep10-21swapper/107:14:341
69125110,23sleep1744-21irqstats11:49:261
3066625114,22sleep230823-21latency_hist09:59:082
2495525119,15sleep20-21swapper/209:39:472
826025012,21sleep28257-21apt-get12:14:182
191992495,33sleep20-21swapper/207:39:462
191992495,33sleep20-21swapper/207:39:462
670224810,22sleep20-21swapper/212:09:182
308224812,21sleep23078-21apt-get10:14:182
102682479,20sleep310303-21df07:14:233
1983224511,15sleep30-21swapper/307:44:233
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional