You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-01 - 12:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Apr 01, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
226302520912,5185sleep312388-21kworker/3:019:17:283
226302520912,5185sleep312388-21kworker/3:019:17:283
13812247198,4656sleep127023-21kworker/u8:123:57:251
19869212016,1147sleep116937-21kworker/1:120:49:291
1235628836,865sleep112399-21ntpq23:52:331
1235628836,865sleep112399-21ntpq23:52:331
13967234011,313sleep314053-21perl23:57:563
2540230010,273sleep02624-21find21:37:370
19251223745,176sleep119289-21modprobe22:32:081
701922238,199sleep17034-21vmstat20:07:411
701922238,199sleep17034-21vmstat20:07:411
11505222356,107sleep111541-21ntp_kernel_pll_20:23:281
31867221510,122sleep131950-21latency_hist19:47:101
16728221434,159sleep116874-21unixbench_singl00:07:351
4042221048,106sleep14073-21idleruntime-cro20:02:081
12606219524,150sleep312734-21timerwakeupswit23:52:393
12606219524,150sleep312734-21timerwakeupswit23:52:393
262721899,159sleep12758-21grep21:37:381
15767218930,71sleep10-21swapper/100:07:101
15767218930,71sleep10-21swapper/100:07:101
29672218417,75sleep10-21swapper/123:02:401
30086217728,28sleep20-21swapper/219:42:102
30086217728,28sleep20-21swapper/219:42:102
12155217775,22sleep10-21swapper/122:07:341
21384217511,54sleep10-21swapper/122:37:291
18238217349,38sleep118396-21unixbench_multi00:12:401
583521728,76sleep10-21swapper/120:07:081
583521728,76sleep10-21swapper/120:07:081
2422217014,78sleep12507-21latency_hist23:22:091
2422217014,78sleep12507-21latency_hist23:22:091
21739216447,90sleep10-21swapper/120:57:121
20786216150,93sleep120670-21/usr/sbin/munin00:22:291
14343216110,134sleep114402-21irqstats20:32:281
11255216121,119sleep011295-21grep20:23:230
19631215916,124sleep019728-21timerandwakeup19:07:400
19631215916,124sleep019728-21timerandwakeup19:07:400
24100215133,102sleep10-21swapper/119:22:221
256521489,68sleep10-21swapper/119:57:111
3932214272,51sleep10-21swapper/121:42:341
6598214011,110sleep16705-21latency_hist21:52:111
6598214011,110sleep16705-21latency_hist21:52:111
26243214069,37sleep10-21swapper/122:52:391
26243214069,37sleep10-21swapper/122:52:391
2982721369,112sleep129929-21fschecks_time21:22:281
26354213443,79sleep126378-21sed21:12:101
24122213160,16sleep10-21swapper/121:02:371
21433212917,32sleep00-21swapper/019:12:360
20947212916,98sleep021040-21/usr/sbin/munin20:52:390
4155212856,53sleep24277-21timerwakeupswit21:42:402
12588212848,23sleep10-21swapper/120:27:241
10578212857,16sleep10-21swapper/122:02:331
10578212857,16sleep10-21swapper/122:02:331
16436212419,89sleep216471-21cpu22:22:242
8152212395,14sleep10-21swapper/120:12:381
22814212311,99sleep122893-21sort00:27:381
14650212311,90sleep10-21swapper/122:17:101
8675212235,18sleep10-21swapper/121:57:411
21996212212,94sleep2222-21systemd-journal19:14:082
19351212214,23sleep30-21swapper/320:47:373
15925212229,11sleep10-21swapper/122:19:161
5385212127,12sleep10-21swapper/123:31:011
17831212111,19sleep10-21swapper/120:42:381
32075212019,84sleep332182-21grep23:12:263
32075212019,84sleep332182-21grep23:12:263
25127212025,80sleep125239-21date00:37:111
9262211942,43sleep19308-21ntp_kernel_pll_23:42:321
22307211931,37sleep11-21systemd19:17:101
8047211716,78sleep38196-21unixbench_singl23:37:413
2358121177,13sleep10-21swapper/122:42:461
2660211536,49sleep22656-21apt-get23:22:222
13123211311,83sleep313195-21/usr/sbin/munin20:27:383
17351211222,71sleep317448-21fschecks_time20:42:283
17011211131,13sleep10-21swapper/120:42:101
32715211017,74sleep10-21swapper/119:47:381
21950211022,29sleep321979-21awk00:27:073
13240211021,70sleep00-21swapper/023:57:080
13240211021,70sleep00-21swapper/023:57:080
28454210710,82sleep128505-21basename19:37:081
31749210610,22sleep10-21swapper/121:27:361
29526210630,12sleep10-21swapper/119:37:411
29526210630,12sleep10-21swapper/119:37:411
28567210618,71sleep328613-21ntp_kernel_pll_21:17:333
28567210618,71sleep328613-21ntp_kernel_pll_21:17:333
10668210618,21sleep10-21swapper/120:22:111
809321059,25sleep18171-21apt-config21:57:081
27407210510,76sleep327465-21irqstats19:32:303
2790821030,13sleep3222-21systemd-journal21:17:083
2790821030,13sleep3222-21systemd-journal21:17:083
16198210320,67sleep015969-1kworker/0:0H22:22:100
14523210125,58sleep213779-21/usr/sbin/munin23:58:272
3064210016,34sleep13111-21ntpq23:22:341
18687210034,40sleep118702-21modprobe19:07:081
424329711,65sleep04454-21latency_hist20:02:110
369529617,24sleep00-21swapper/021:42:290
859429510,71sleep18600-21modprobe23:42:071
597229515,64sleep06029-21awk23:32:300
2392029528,43sleep323935-21idleruntime-cro22:47:083
991729421,51sleep09919-21cron22:02:080
255282937,18sleep10-21swapper/122:52:101
962629226,44sleep09777-21head23:42:390
281882927,16sleep10-21swapper/121:17:231
281882927,16sleep10-21swapper/121:17:231
2715529212,18sleep10-21swapper/121:12:381
2715529212,18sleep10-21swapper/121:12:381
171229210,56sleep20-21swapper/221:37:102
15792928,18sleep10-21swapper/121:36:291
596929115,57sleep36105-21awk21:47:413
596929115,57sleep36105-21awk21:47:413
555629017,54sleep25259-21/usr/sbin/munin21:47:322
555629017,54sleep25259-21/usr/sbin/munin21:47:322
2650429021,31sleep00-21swapper/022:52:420
2650429021,31sleep00-21swapper/022:52:420
2546129011,61sleep10-21swapper/119:27:111
2208128915,58sleep022154-21/usr/sbin/munin20:57:310
930028811,63sleep09326-21sh20:17:250
2284828818,18sleep00-21swapper/022:42:280
200352878,21sleep120113-21mailstats22:32:361
1704428711,61sleep117148-21timerandwakeup22:22:401
117762870,7sleep00-21swapper/022:07:240
1175528715,19sleep30-21swapper/322:07:243
3068028622,43sleep230701-21fschecks_count19:42:552
2701628519,41sleep227055-21basename22:57:082
2701628519,41sleep227055-21basename22:57:082
1890428513,60sleep318947-21latency_hist00:17:103
196262847,15sleep20-21swapper/220:47:402
1547128416,23sleep20-21swapper/220:37:102
2428728318,25sleep00-21swapper/019:22:300
237128322,36sleep32391-21modprobe23:22:073
237128322,36sleep32391-21modprobe23:22:073
2368228315,43sleep323731-21/usr/sbin/munin21:02:273
2358728324,37sleep20-21swapper/221:02:262
1962428318,43sleep319637-21fschecks_count22:32:263
106882838,17sleep110700-21irqstats23:47:281
48492820,6sleep20-21swapper/220:02:402
48492820,6sleep20-21swapper/220:02:402
36028214,46sleep2507-21unixbench_singl19:47:392
1754328214,51sleep217545-21apt-get00:12:202
2786928120,39sleep227895-21munin-run21:17:072
2786928120,39sleep227895-21munin-run21:17:072
1338128115,50sleep313388-21/usr/sbin/munin22:12:243
128828112,50sleep21355-21fschecks_time19:52:322
687228016,29sleep00-21swapper/020:07:400
687228016,29sleep00-21swapper/020:07:400
3257928022,44sleep232598-21latency_hist21:32:102
2795028020,46sleep228048-21timerandwakeup22:57:402
277228013,47sleep30-21swapper/319:57:213
2301928018,45sleep323140-21users00:27:403
2242628019,45sleep222575-21latency_hist22:42:102
2160528022,27sleep30-21swapper/319:12:393
925827917,36sleep29356-21threads21:58:022
692027922,40sleep27004-21fschecks_time21:52:292
57602797,34sleep10-21swapper/123:32:221
312452798,15sleep10-21swapper/123:07:401
3052227917,32sleep00-21swapper/023:07:270
104827916,44sleep31180-21/usr/sbin/munin21:32:393
2767127817,47sleep227730-21sendmail_mailqu19:32:372
209727822,34sleep30-21swapper/319:52:453
620527723,31sleep30-21swapper/320:07:273
620527723,31sleep30-21swapper/320:07:273
3012927715,43sleep230166-21sed21:22:332
2422127611,15sleep00-21swapper/000:32:330
2203027620,33sleep30-21swapper/320:57:303
1082327618,40sleep010860-21/usr/sbin/munin23:47:310
2855427515,44sleep228566-21/usr/sbin/munin21:17:322
2855427515,44sleep228566-21/usr/sbin/munin21:17:322
2742927511,27sleep30-21swapper/322:57:293
223472758,13sleep10-21swapper/120:57:361
1387427517,19sleep20-21swapper/222:12:352
1051027510,51sleep310565-21systemd-run20:22:093
2750127413,46sleep027512-21vmstat21:12:410
2750127413,46sleep027512-21vmstat21:12:410
204452748,50sleep10-21swapper/100:22:081
738127310,48sleep07449-21awk21:52:390
3103527318,33sleep30-21swapper/323:07:383
274122736,13sleep10-21swapper/122:57:271
262602738,53sleep00-21swapper/021:11:280
1951527318,26sleep00-21swapper/000:17:310
1770127327,23sleep30-21swapper/319:02:303
14222736,14sleep00-21swapper/019:52:340
775027213,38sleep27643-21/usr/sbin/munin20:12:262
1930227211,15sleep30-21swapper/322:32:103
1178527214,42sleep311910-21cat20:23:343
1151827219,31sleep20-21swapper/222:07:102
1151827219,31sleep20-21swapper/222:07:102
2602727124,21sleep30-21swapper/319:27:333
2385327117,40sleep023866-21latency_hist19:22:090
2385327117,40sleep023866-21latency_hist19:22:090
2158427118,36sleep321648-21ps22:37:353
1421927119,28sleep30-21swapper/320:32:263
5632706,10sleep10-21swapper/123:14:141
5632706,10sleep10-21swapper/123:14:141
462027021,31sleep34663-21grep23:27:323
2885627016,35sleep028932-21fschecks_time19:37:280
2885627016,35sleep028932-21fschecks_time19:37:280
2696027013,42sleep027107-21cat19:32:110
235702706,10sleep10-21swapper/119:18:591
235702706,10sleep10-21swapper/119:18:591
79452696,48sleep27311-21/usr/sbin/munin23:37:402
190182696,47sleep10-21swapper/119:07:251
190182696,47sleep10-21swapper/119:07:251
1730926924,27sleep30-21swapper/300:12:093
1614626916,36sleep216172-21munin-run22:22:082
766826815,38sleep37669-21cpu20:12:233
552026718,27sleep00-21swapper/023:32:090
546926720,23sleep20-21swapper/223:32:072
2489226719,27sleep225034-21grep22:47:392
2152026717,32sleep021565-21/usr/sbin/munin22:37:320
2118926716,33sleep20-21swapper/200:22:382
2045526713,36sleep220429-21latency_hist00:22:092
1802726720,22sleep20-21swapper/219:02:382
1339126721,26sleep00-21swapper/022:12:250
840226616,33sleep08547-21unixbench_singl20:12:410
42782664,34sleep30-21swapper/321:42:413
1950426613,24sleep30-21swapper/300:17:303
1617726616,33sleep216201-21/usr/sbin/munin20:37:382
731626522,22sleep30-21swapper/321:52:383
584626515,27sleep20-21swapper/223:32:282
3264226512,23sleep00-21swapper/019:47:370
2221326511,27sleep20-21swapper/200:27:182
1735026517,24sleep00-21swapper/000:12:100
1636726519,27sleep216407-21grep00:07:282
3181526420,26sleep331839-21modprobe19:47:083
2077826417,20sleep00-21swapper/000:22:280
1844626416,26sleep00-21swapper/022:27:350
897126312,25sleep30-21swapper/321:57:553
1840326318,19sleep00-21swapper/019:03:280
1791626318,26sleep20-21swapper/220:42:392
45642628,21sleep30-21swapper/320:02:223
45642628,21sleep30-21swapper/320:02:223
3260126215,27sleep30-21swapper/321:32:113
298026217,19sleep20-21swapper/219:57:282
2474226221,25sleep20-21swapper/219:22:402
2274726217,20sleep20-21swapper/222:42:262
2033926215,27sleep220358-21apt-get20:52:202
2012426217,27sleep00-21swapper/020:52:100
1949526223,23sleep20-21swapper/219:07:382
1949526223,23sleep20-21swapper/219:07:382
1598226221,22sleep00-21swapper/020:37:320
15912618,39sleep20-21swapper/223:17:382
15912618,39sleep20-21swapper/223:17:382
1081326117,20sleep310899-21ls22:02:393
1081326117,20sleep310899-21ls22:02:393
1051426123,21sleep20-21swapper/223:47:252
588326018,18sleep30-21swapper/323:32:293
307102607,42sleep00-21swapper/021:23:320
2467826014,22sleep30-21swapper/322:47:363
2410226021,19sleep30-21swapper/319:22:243
157126020,23sleep00-21swapper/023:17:360
157126020,23sleep00-21swapper/023:17:360
1335126015,26sleep213455-21apt-config23:57:102
1335126015,26sleep213455-21apt-config23:57:102
1018326015,18sleep00-21swapper/022:02:200
1018326015,18sleep00-21swapper/022:02:200
669525918,25sleep20-21swapper/220:07:382
669525918,25sleep20-21swapper/220:07:382
541325925,21sleep05259-21/usr/sbin/munin21:47:300
541325925,21sleep05259-21/usr/sbin/munin21:47:300
231525917,24sleep20-21swapper/221:37:312
227282598,28sleep00-21swapper/000:27:370
1961725914,23sleep219623-21diskstats22:32:252
1040425916,26sleep310423-21apt-get23:47:183
2441925810,29sleep024473-21/usr/sbin/munin22:47:290
1815825819,20sleep00-21swapper/000:12:390
1773825811,34sleep317766-21latency_hist22:27:103
1643425817,24sleep016429-21apt-get22:22:220
752125713,26sleep00-21swapper/023:37:300
459125721,19sleep00-21swapper/020:02:350
459125721,19sleep00-21swapper/020:02:350
334525717,17sleep30-21swapper/323:22:403
3157525718,19sleep00-21swapper/021:27:310
297462576,36sleep30-21swapper/319:38:323
297462576,36sleep30-21swapper/319:38:323
2057025720,17sleep30-21swapper/320:52:293
1611625716,23sleep30-21swapper/320:37:353
3197725616,23sleep031973-21apt-get23:12:190
3197725616,23sleep031973-21apt-get23:12:190
2980825616,24sleep329815-21diskstats21:22:253
2560925620,20sleep20-21swapper/221:07:392
933625519,18sleep20-21swapper/220:17:272
396325518,16sleep00-21swapper/023:27:090
2888725513,25sleep028881-21apt-get23:02:220
2568625514,24sleep325797-21/usr/sbin/munin21:07:403
2255425514,24sleep222551-21apt-get19:17:232
2255425514,24sleep222551-21apt-get19:17:232
1497725514,18sleep00-21swapper/020:32:400
1380225514,25sleep013798-21apt-get23:57:200
1344425514,22sleep213491-21users20:27:412
1154325514,23sleep211562-21ntp_states20:23:292
3094225417,17sleep20-21swapper/223:07:372
2415425416,16sleep20-21swapper/200:32:312
1943225410,16sleep30-21swapper/319:07:353
1943225410,16sleep30-21swapper/319:07:353
1571425418,20sleep30-21swapper/300:07:083
1571425418,20sleep30-21swapper/300:07:083
1304325410,18sleep00-21swapper/020:27:350
1183825414,24sleep211839-21diskstats22:07:272
3243425315,22sleep232440-21ntp_states23:12:342
3243425315,22sleep232440-21ntp_states23:12:342
2895525315,22sleep228962-21df_inode23:02:242
205842534,15sleep00-21swapper/022:34:210
3247625216,17sleep30-21swapper/319:47:323
231292527,16sleep20-21swapper/221:01:082
1725525213,22sleep017251-21apt-get20:42:210
933825112,23sleep39344-21ntp_states23:42:343
3066025112,23sleep030672-21df_inode19:42:530
3019825115,17sleep30-21swapper/323:07:103
2621425113,23sleep326220-21ntp_states22:52:373
2621425113,23sleep326220-21ntp_states22:52:373
163225010,21sleep01690-21tr21:37:080
268022496,10sleep20-21swapper/219:30:472
194424912,21sleep31941-21apt-get21:37:193
1567124912,17sleep20-21swapper/200:06:352
1567124912,17sleep20-21swapper/200:06:352
1929024811,22sleep219301-21munin-run22:32:092
2739124714,18sleep00-21swapper/022:57:260
2727624710,15sleep00-21swapper/019:32:270
238552478,14sleep30-21swapper/300:32:243
1509324710,22sleep315143-21if_eth022:17:293
2818724610,17sleep00-21swapper/021:17:220
2818724610,17sleep00-21swapper/021:17:220
2404524614,18sleep00-21swapper/021:02:360
1576524511,17sleep00-21swapper/000:07:090
1576524511,17sleep00-21swapper/000:07:090
888224412,15sleep20-21swapper/223:42:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional