You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-20 - 09:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Wed May 20, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
41322405413,4028sleep31647-21kworker/3:021:03:033
243462265810,2636sleep222591-21kworker/2:023:49:122
23439267319,616sleep123520-21cat23:44:391
23439267319,616sleep123520-21cat23:44:391
11400262618,585sleep211420-21cron23:09:062
3443259319,553sleep13478-21cron19:19:061
3408257712,550sleep03415-21ntp_states20:59:330
14827247212,445sleep014820-21apt-get23:19:220
20388239887,213sleep120532-21tail23:34:381
18556238633,297sleep118597-21ntpq23:29:451
1013238611,358sleep21139-21/usr/sbin/munin19:09:422
27341233033,282sleep127432-21sed22:14:371
29352307107,92sleep12973-21fschecks_count00:24:251
20718229698,184sleep120756-21ls21:54:261
32678225965,157sleep132714-21ntpq20:49:331
32678225965,157sleep132714-21ntpq20:49:331
36212241109,116sleep13632-21modprobe22:44:051
36212241109,116sleep13632-21modprobe22:44:051
12525224013,214sleep212534-21unin-run21:29:072
20246223638,179sleep120256-21modprobe21:54:061
17702223138,171sleep117729-21munin-run23:29:081
12513222791,30sleep10-21swapper/119:44:381
10867222545,162sleep111057-21apt-config21:24:091
26008222413,199sleep326014-21cat23:54:223
30715222359,121sleep130823-21fschecks_time00:09:271
31944221981,27sleep10-21swapper/122:29:281
17204221618,22sleep10-21swapper/121:44:081
2645220650,114sleep12795-21latency_hist00:24:091
32431220390,21sleep10-21swapper/119:09:091
10718219831,24sleep10-21swapper/119:39:581
10718219831,24sleep10-21swapper/119:39:581
1358217924,105sleep11432-21switchtime22:34:361
1358217924,105sleep11432-21switchtime22:34:361
12072217742,89sleep112114-21ntpq23:09:301
25825217411,149sleep125925-21timerandwakeup22:09:371
25825217411,149sleep125925-21timerandwakeup22:09:371
345217113,114sleep10-21swapper/100:14:361
29606217134,124sleep129646-21http22:24:061
883217012,19sleep10-21swapper/119:09:411
14506216811,103sleep10-21swapper/123:19:071
25702216612,141sleep125846-21latency_hist23:54:081
1816421656,13sleep00-21swapper/023:29:240
15195216542,109sleep115243-21ntp_kernel_pll_23:19:311
2629021636,121sleep00-21swapper/023:54:280
32033215312,107sleep132087-21cat20:49:081
15454214915,86sleep10-21swapper/119:54:361
21491214714,16sleep10-21swapper/120:14:471
21491214714,16sleep10-21swapper/120:14:471
15983214723,100sleep316023-21fschecks_count21:39:263
693021457,12sleep00-21swapper/019:29:230
5658213036,24sleep10-21swapper/100:34:061
1642213011,17sleep10-21swapper/120:54:291
28776212920,92sleep128786-21modprobe00:04:061
8541212816,90sleep38674-21sed21:14:423
5804212812,73sleep1284-21dbus-daemon21:09:071
5804212812,73sleep1284-21dbus-daemon21:09:071
24272212811,102sleep124384-21latency_hist20:24:081
23661212521,43sleep10-21swapper/122:04:201
29363212414,62sleep10-21swapper/120:39:271
7832212325,57sleep17851-21apt-get21:14:201
28338212054,52sleep128481-21sed20:34:381
21370212019,82sleep020875-21apt-get20:14:220
21370212019,82sleep020875-21apt-get20:14:220
1754121209,19sleep10-21swapper/121:44:291
12597212013,37sleep0307-21in:imuxsock21:29:090
26425211759,42sleep10-21swapper/123:54:331
30654211555,44sleep130714-21find22:25:071
30654211555,44sleep130714-21find22:25:071
2367211512,90sleep22341-21/usr/sbin/munin22:39:222
2367211512,90sleep22341-21/usr/sbin/munin22:39:222
13517211513,63sleep113651-21grep21:29:391
9323211414,64sleep19444-21cat19:34:451
9323211414,64sleep19444-21cat19:34:451
1676921148,90sleep016347-21/usr/sbin/munin23:24:310
375521138,61sleep10-21swapper/119:19:181
2306921138,72sleep123071-21sshd22:00:081
709521129,67sleep17098-21/usr/sbin/munin22:54:231
6148211222,67sleep06288-21sed22:49:380
343321124,15sleep00-21swapper/022:41:380
343321124,15sleep00-21swapper/022:41:380
22009211213,14sleep10-21swapper/123:39:381
22009211213,14sleep10-21swapper/123:39:381
2266211119,74sleep22350-21fschecks_time19:14:272
3416210913,59sleep10-21swapper/120:59:341
1485421099,65sleep114923-21sensors_temp21:34:351
751321089,86sleep17523-21munin-run00:39:071
1619621089,22sleep00-21swapper/019:59:070
18011210715,76sleep318071-21sort21:44:393
6715210517,75sleep36732-21latency_hist19:29:103
8042210311,77sleep08103-21cat21:14:310
1052321030,6sleep20-21swapper/219:39:502
1052321030,6sleep20-21swapper/219:39:502
32140210216,71sleep232148-21ntpq22:29:332
19662210215,74sleep019669-21df_inode20:09:250
6193210113,70sleep36216-21cpu00:34:253
1845429913,72sleep118480-21ntp_kernel_pll_20:04:331
284829816,66sleep32913-21sendmail_mailst22:39:353
284829816,66sleep32913-21sendmail_mailst22:39:353
3235829622,51sleep332400-21cut20:49:263
3235829622,51sleep332400-21cut20:49:263
250442968,28sleep10-21swapper/120:24:371
2253929617,64sleep022552-21idleruntime-cro23:44:060
2253929617,64sleep022552-21idleruntime-cro23:44:060
1131729617,65sleep311342-21/usr/sbin/munin21:24:563
69392958,47sleep10-21swapper/119:29:241
2738929515,56sleep027530-21latency_hist20:34:090
952429424,49sleep29584-21fschecks_time21:19:262
2352329418,53sleep223680-21tail23:44:402
2352329418,53sleep223680-21tail23:44:402
569229325,39sleep20-21swapper/200:34:072
2663329322,32sleep10-21swapper/120:29:371
2089529316,61sleep020896-21sshd23:38:120
1394229325,45sleep013912-21apt-get19:49:380
2917929211,66sleep029229-21cpuspeed20:39:220
200332927,46sleep10-21swapper/120:09:341
570529114,19sleep15740-21ntpq19:24:311
1403829017,59sleep114155-21unixbench_multi19:49:391
694228917,54sleep27013-21fschecks_time19:29:252
627928916,57sleep35798-21apt-get21:09:173
3199128915,53sleep032005-21/usr/sbin/munin19:04:400
3078528919,40sleep30-21swapper/322:25:093
3078528919,40sleep30-21swapper/322:25:093
1306928815,55sleep213097-21latency_hist19:49:082
3198228716,53sleep331983-21cron20:49:063
2926228719,46sleep329277-21cut20:39:253
2569528722,50sleep325754-21sendmail_mailqu22:09:353
2569528722,50sleep325754-21sendmail_mailqu22:09:353
576828613,56sleep05800-21awk21:09:060
576828613,56sleep05800-21awk21:09:060
228072867,14sleep00-21swapper/023:44:180
228072867,14sleep00-21swapper/023:44:180
2266728518,47sleep022736-21grep21:59:380
281612840,7sleep00-21swapper/023:59:400
1967628421,45sleep219708-21fschecks_count20:09:272
1669728413,30sleep10-21swapper/119:59:291
1457628417,18sleep20-21swapper/223:19:082
1076328418,54sleep010866-21apt-key21:24:080
81328315,33sleep00-21swapper/019:09:400
39152839,32sleep10-21swapper/122:44:221
2276428317,41sleep322889-21wc20:19:093
2276428317,41sleep322889-21wc20:19:093
2907528214,50sleep20-21swapper/200:04:272
2053728120,47sleep020543-21grep21:54:210
1045728114,52sleep010507-21if_eth019:39:490
1045728114,52sleep010507-21if_eth019:39:490
101452817,18sleep110138-21apt-get23:04:211
587028014,50sleep25919-21ntp_states22:49:322
1845028018,45sleep318540-21load23:29:433
1685528017,31sleep00-21swapper/019:59:320
1325828014,28sleep20-21swapper/221:29:362
116412807,14sleep10-21swapper/121:25:061
861427818,37sleep28132-21apt-get19:34:182
861427818,37sleep28132-21apt-get19:34:182
2421427811,51sleep024215-21sshd20:24:030
19692780,5sleep00-21swapper/000:19:390
1616727718,38sleep316186-21latency_hist19:59:063
944027616,17sleep30-21swapper/321:19:233
676727510,30sleep10-21swapper/121:09:561
67552758,29sleep10-21swapper/122:54:071
1841327519,39sleep018452-21ntpq20:04:320
446327418,25sleep00-21swapper/019:19:380
296872745,12sleep10-21swapper/100:04:421
2809027416,23sleep30-21swapper/323:59:403
2718327421,33sleep227214-21munin-run23:59:062
184327425,22sleep30-21swapper/320:54:353
1762927421,32sleep30-21swapper/323:29:063
7722734,56sleep00-21swapper/020:50:040
7722734,56sleep00-21swapper/020:50:040
2894127320,34sleep328949-21latency_hist20:39:093
2025727318,39sleep21-21systemd21:54:072
2106327214,38sleep220512-21/usr/sbin/munin21:54:352
1011427222,30sleep310086-21apt-get19:39:403
1011427222,30sleep310086-21apt-get19:39:403
99052719,48sleep110016-21cat23:04:081
3156827110,43sleep231569-21sshd00:10:132
3059427114,41sleep230630-21ntp_states22:25:042
3059427114,41sleep230630-21ntp_states22:25:042
1349727118,25sleep20-21swapper/219:49:282
1331427113,44sleep313323-21df_inode23:14:283
521927024,26sleep20-21swapper/222:49:072
2821727019,25sleep00-21swapper/020:34:370
2731327022,28sleep20-21swapper/220:34:072
2408827020,33sleep324099-21modprobe23:49:053
2408827020,33sleep324099-21modprobe23:49:053
16922700,9sleep212-21rcu_sched20:54:292
127992709,17sleep30-21swapper/323:12:103
2563626921,30sleep30-21swapper/323:54:063
2177026924,20sleep20-21swapper/223:39:352
2177026924,20sleep20-21swapper/223:39:352
1871126919,30sleep30-21swapper/320:04:403
4422688,11sleep30-21swapper/319:09:313
420926813,35sleep34216-21/usr/sbin/munin19:19:333
3214926816,24sleep30-21swapper/322:29:343
2973626819,21sleep30-21swapper/322:24:083
1385926821,25sleep30-21swapper/319:49:373
586726713,36sleep06029-21latency_hist00:34:090
3029826714,35sleep030357-21tune2fs22:24:580
3029826714,35sleep030357-21tune2fs22:24:580
259282679,14sleep20-21swapper/222:09:382
259282679,14sleep20-21swapper/222:09:382
98326624,24sleep30-21swapper/300:19:063
595626620,20sleep30-21swapper/321:09:093
595626620,20sleep30-21swapper/321:09:093
1014426614,29sleep00-21swapper/023:04:200
3235826520,29sleep332387-21munin-run19:09:073
2459426512,35sleep224633-21fschecks_count20:24:262
2139426516,22sleep20-21swapper/220:14:362
2139426516,22sleep20-21swapper/220:14:362
2029626516,35sleep320344-21sshd21:54:083
1300126511,31sleep00-21swapper/023:14:090
88026418,19sleep30-21swapper/322:34:263
88026418,19sleep30-21swapper/322:34:263
752426422,22sleep20-21swapper/200:39:082
518726416,30sleep25310-21sed21:04:402
518726416,30sleep25310-21sed21:04:402
2187926420,20sleep30-21swapper/321:59:093
1025126412,33sleep310318-21df19:39:443
1025126412,33sleep310318-21df19:39:443
2759826317,22sleep20-21swapper/223:59:262
26202636,12sleep00-21swapper/020:58:560
1807326316,21sleep00-21swapper/021:44:400
1626626312,21sleep00-21swapper/021:39:310
1238526318,25sleep00-21swapper/019:44:370
1193726315,23sleep00-21swapper/021:25:130
687926222,19sleep00-21swapper/000:34:460
53112625,13sleep00-21swapper/019:24:220
1782326216,21sleep20-21swapper/223:29:092
1459026215,25sleep20-21swapper/221:34:282
733626116,19sleep30-21swapper/319:29:343
506426120,19sleep20-21swapper/219:24:082
1930026115,32sleep219301-21cron20:09:072
1428226116,31sleep00-21swapper/023:16:110
913226018,22sleep00-21swapper/022:59:350
877126020,21sleep30-21swapper/322:59:263
645726019,20sleep20-21swapper/221:09:492
39042605,11sleep30-21swapper/300:26:293
3272426014,28sleep00-21swapper/000:14:350
254526015,24sleep30-21swapper/319:14:323
2285726017,27sleep20-21swapper/221:59:402
606925910,33sleep36146-21grep22:49:373
40632595,10sleep00-21swapper/000:28:320
2735725922,21sleep30-21swapper/320:34:083
2458925915,25sleep00-21swapper/023:49:270
738825816,27sleep37430-21sed22:54:293
47425815,23sleep20-21swapper/200:14:372
458625815,20sleep30-21swapper/300:29:283
654625712,24sleep00-21swapper/021:09:510
456925719,24sleep34689-21/usr/sbin/munin22:44:403
39625721,20sleep20-21swapper/220:49:382
39625721,20sleep20-21swapper/220:49:382
3168125716,25sleep031677-21apt-get22:29:180
2814625718,24sleep00-21swapper/022:19:090
2676725715,26sleep226761-21apt-get22:14:222
2304925714,24sleep323141-21perl20:19:263
223762575,12sleep30-21swapper/323:41:593
223762575,12sleep30-21swapper/323:41:593
2104525716,18sleep30-21swapper/323:39:083
181462579,22sleep20-21swapper/223:29:172
1524525714,24sleep215277-21ntp_states23:19:322
904925623,17sleep20-21swapper/222:59:332
785925613,25sleep27851-21apt-get21:14:252
3194425618,18sleep30-21swapper/300:14:093
3039725617,21sleep30-21swapper/300:09:083
2964725613,26sleep229676-21munin-run22:24:072
2713325616,20sleep30-21swapper/322:14:313
2463525619,21sleep324718-21/usr/sbin/munin20:24:273
116325613,26sleep01212-21ntpq22:34:310
116325613,26sleep01212-21ntpq22:34:310
838325514,19sleep20-21swapper/222:59:092
3208825512,18sleep20-21swapper/220:49:092
2500625522,17sleep20-21swapper/222:09:092
2492925522,18sleep00-21swapper/022:09:070
143525515,24sleep21558-21wc22:34:372
143525515,24sleep21558-21wc22:34:372
1142225516,24sleep30-21swapper/323:09:073
3126425415,18sleep00-21swapper/000:09:380
3109725412,25sleep231093-21apt-get19:04:222
293392548,15sleep00-21swapper/022:20:380
2392725412,20sleep30-21swapper/322:04:303
201012546,37sleep00-21swapper/021:52:110
1648525413,25sleep216478-21apt-get19:59:212
1569625410,18sleep00-21swapper/021:39:110
1500125417,19sleep30-21swapper/321:34:373
127282546,11sleep30-21swapper/319:45:173
388825314,24sleep03905-21apt22:44:170
217525315,22sleep02170-21apt-get19:14:210
2090625315,22sleep321023-21grep20:14:093
1744425312,24sleep217441-21apt-get21:44:222
1319125310,28sleep313227-21ntp_states21:29:333
32162526,14sleep20-21swapper/200:24:302
3072025213,22sleep030717-21apt-get20:44:180
2848225219,18sleep20-21swapper/220:34:392
1722325213,16sleep20-21swapper/221:44:092
64182515,11sleep20-21swapper/219:27:112
2914825116,22sleep029154-21df_inode00:04:290
1985525112,25sleep219919-21idleruntime23:34:262
1536225114,23sleep215368-21ntp_states19:54:332
137125111,17sleep20-21swapper/200:19:252
1679425012,22sleep316853-21memory19:59:313
1168725012,22sleep211683-21apt-get23:09:182
2942924913,23sleep229448-21irqstats20:39:282
2358324912,21sleep20-21swapper/220:19:372
187924911,23sleep31964-21cat00:19:393
701124811,21sleep07006-21apt-get22:54:180
2582024814,21sleep025889-21latency_hist20:29:080
2211324811,22sleep322109-21apt-get21:59:213
892024510,20sleep08923-21irqstats19:34:360
892024510,20sleep08923-21irqstats19:34:360
2652724513,17sleep00-21swapper/022:14:080
2652724513,17sleep00-21swapper/022:14:080
149062447,14sleep30-21swapper/323:19:253
302024111,15sleep20-21swapper/220:59:232
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional