You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-06 - 05:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Jun 06, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15212243345,4296sleep010808-21kworker/0:221:59:080
26429276817,735sleep126463-21/usr/sbin/munin20:53:281
6057243052,39sleep00-21swapper/023:13:370
966223239,300sleep09676-21fschecks_count21:43:250
26895229246,100sleep00-21swapper/019:13:140
27741225132,205sleep027751-21find20:58:150
4563224933,190sleep04709-21unixbench_singl23:08:360
3096922497,205sleep031052-21fschecks_time21:08:250
19491224936,194sleep019585-21threads22:13:370
1162022496,206sleep00-21swapper/023:33:070
24340222722,187sleep324486-21wc00:13:073
7793222131,153sleep07915-21latency_hist21:38:080
17723221411,81sleep017330-21/usr/sbin/munin22:08:280
12991221416,186sleep00-21swapper/023:36:260
19993221213,182sleep120032-21grep23:58:231
1238222128,191sleep312404-21awk21:53:053
14006220939,125sleep00-21swapper/021:58:070
29179220213,119sleep028576-21/usr/sbin/munin19:18:350
15304220210,27sleep00-21swapper/020:18:180
25842220176,87sleep025875-21awk00:18:050
1221122000,37sleep00-21swapper/020:08:280
3063721949,151sleep030693-21interrupts19:23:560
29296219210,146sleep029306-21find21:03:160
18173219223,79sleep00-21swapper/020:28:070
25952218815,157sleep125976-21latency_hist20:53:071
25952218815,157sleep125976-21latency_hist20:53:071
1990621828,42sleep00-21swapper/023:58:180
23194218117,150sleep123230-21idleruntime00:08:251
32448218035,106sleep032526-21mailstats19:28:360
27032218010,157sleep327003-21apt-get19:13:333
3548217922,140sleep13566-21modprobe23:08:051
2717921799,124sleep00-21swapper/022:38:320
2564521707,30sleep00-21swapper/020:49:430
2564521707,30sleep00-21swapper/020:49:430
26238216721,106sleep00-21swapper/020:53:230
2173521677,38sleep00-21swapper/000:03:270
27419216623,16sleep00-21swapper/000:23:050
2411321665,11sleep00-21swapper/020:45:070
28204216417,104sleep00-21swapper/000:23:340
70421609,139sleep0724-21apt-get22:58:150
15373215613,125sleep00-21swapper/023:43:250
16366215133,44sleep11-21systemd23:46:321
706021489,124sleep07122-21grep23:18:250
3042621465,28sleep00-21swapper/000:31:580
2374214610,124sleep02431-21grep23:03:230
18796214620,50sleep00-21swapper/023:53:340
18796214620,50sleep00-21swapper/023:53:340
11336214018,111sleep00-21swapper/020:04:050
378221395,12sleep00-21swapper/019:42:000
32693213832,92sleep024596-21cyclictest21:13:370
32693213832,92sleep024596-21cyclictest21:13:370
2165221356,12sleep00-21swapper/022:21:490
29989213421,94sleep130004-21apt-key19:23:081
11309213216,81sleep00-21swapper/021:48:260
3916213021,30sleep20-21swapper/219:43:072
22724213022,87sleep222869-21sed22:23:412
335521298,85sleep00-21swapper/021:23:230
27792212917,98sleep227820-21cut00:23:242
7226212818,80sleep00-21swapper/019:53:150
1930921279,102sleep219319-21vmstat20:28:382
3118021265,11sleep00-21swapper/022:51:160
13670212422,84sleep113161-21apt-get23:38:171
671121215,11sleep00-21swapper/019:49:480
30393212011,93sleep219577-21kworker/u8:019:23:252
19583212023,76sleep319602-21apt23:58:053
19583212023,76sleep319602-21apt23:58:053
18659211915,84sleep318682-21/usr/sbin/munin20:28:273
1492211946,35sleep00-21swapper/019:33:330
28272211824,25sleep00-21swapper/022:43:210
23103211842,38sleep023193-21perl00:08:240
20429211729,24sleep020470-21grep20:33:360
23672211620,74sleep223727-21fschecks_count22:28:252
3076121140,9sleep327-21ksoftirqd/300:33:093
8212211217,66sleep00-21swapper/021:38:250
582021128,68sleep04881-21/usr/sbin/munin21:28:400
2785121128,14sleep20-21swapper/219:13:572
15421211122,67sleep215461-21sort20:18:412
10996211118,78sleep111083-21timerandwakeup20:03:381
32671211015,79sleep132659-21apt-get21:13:221
32671211015,79sleep132659-21apt-get21:13:221
27324211010,85sleep327402-21find22:38:353
10148210727,42sleep010185-21latency_hist20:03:080
4616210616,24sleep34644-21munin-run21:28:063
8739210449,39sleep38878-21/usr/sbin/munin21:38:363
18497210425,37sleep00-21swapper/020:28:240
27983210319,30sleep128011-21munin-run22:43:051
22835210318,60sleep222848-21latency_hist20:43:072
479210126,40sleep0489-21munin-run22:58:060
24550210135,40sleep224574-21latency_hist19:08:072
249072999,77sleep025042-21sed22:33:070
51882985,83sleep00-21swapper/019:45:070
1733029813,69sleep217393-21ps20:23:332
1754529724,51sleep117695-21unixbench_multi20:23:361
150112970,10sleep312-21rcu_sched20:18:083
127582967,12sleep10-21swapper/121:53:231
1032129621,57sleep310299-21apt-get23:28:303
2494929519,57sleep124541-21/usr/sbin/munin00:13:291
138572957,53sleep013894-21sed20:13:300
2896729423,51sleep329000-21awk00:28:053
251452949,43sleep00-21swapper/022:33:190
2491829417,56sleep124798-21/usr/sbin/munin19:08:261
2434129427,49sleep324370-21idleruntime-cro20:48:053
3202429328,39sleep232064-21fschecks_count19:28:262
1589329334,33sleep115949-21tune2fs22:03:251
1410229318,35sleep00-21swapper/023:38:440
628529213,61sleep16377-21cat19:48:361
305629212,60sleep12577-21/usr/sbin/munin19:38:301
2394829129,33sleep30-21swapper/319:03:363
148629119,55sleep31500-21modprobe21:18:053
148629119,55sleep31500-21modprobe21:18:053
130729112,65sleep3363-21snmpd22:58:343
1300129020,56sleep20-21swapper/220:10:332
248232899,36sleep00-21swapper/019:08:210
179932899,45sleep023758-21sshd23:52:330
1488428926,43sleep215007-21find20:18:082
218302889,36sleep00-21swapper/020:38:290
3094828714,60sleep231018-21basename00:33:252
3094828714,60sleep231018-21basename00:33:252
2352928714,58sleep123540-21ntpq20:43:311
17802867,34sleep00-21swapper/021:18:180
851428525,39sleep28540-21munin-run19:58:052
2629128527,40sleep326450-21irqstats19:13:093
2628428515,55sleep326370-21fschecks_time20:53:263
1285728524,40sleep312649-21/usr/sbin/munin21:53:253
891828420,44sleep28982-21sort19:58:262
87282847,34sleep00-21swapper/023:23:260
3105328418,19sleep10-21swapper/121:08:261
2488928419,49sleep224899-21df_inode19:08:232
233452847,41sleep023347-21munin-run22:28:070
1958728417,32sleep10-21swapper/122:13:381
1712128430,41sleep017251-21latency_hist22:08:080
2404828311,55sleep324085-21grep22:28:343
2341728327,30sleep10-21swapper/119:03:261
197252830,6sleep20-21swapper/223:58:072
197252830,6sleep20-21swapper/223:58:072
746128212,57sleep37499-21cat19:53:263
515028223,36sleep30-21swapper/323:13:073
308002827,34sleep00-21swapper/000:33:190
308002827,34sleep00-21swapper/000:33:190
1320928223,30sleep30-21swapper/323:38:063
1018628224,26sleep10-21swapper/120:03:091
585028129,28sleep10-21swapper/123:13:341
2659828111,18sleep10-21swapper/119:13:111
1915728119,41sleep119307-21unixbench_singl20:28:371
541628021,45sleep3225-21systemd-journal19:48:053
375027915,48sleep23786-21ntp_kernel_pll_21:23:332
2853127917,47sleep228543-21/usr/sbin/munin22:43:272
200027929,30sleep20-21swapper/223:03:052
724227814,32sleep30-21swapper/323:18:283
3061327819,42sleep330682-21sensors_temp22:48:493
1874027811,50sleep118774-21/usr/sbin/munin23:53:311
1874027811,50sleep118774-21/usr/sbin/munin23:53:311
1062827823,24sleep20-21swapper/223:29:052
1803127719,28sleep30-21swapper/322:08:363
16072779,52sleep21676-21/usr/sbin/munin19:33:352
771227617,42sleep17716-21cron21:38:051
1686727611,29sleep30-21swapper/323:48:283
1267427614,25sleep20-21swapper/221:53:202
588127515,43sleep25962-21/usr/sbin/munin23:13:352
34327430,29sleep2385-21tune2fs21:13:482
34327430,29sleep2385-21tune2fs21:13:482
2958327428,27sleep10-21swapper/122:48:061
2661727418,22sleep20-21swapper/200:18:312
345227319,22sleep10-21swapper/121:23:281
2219127322,31sleep10-21swapper/100:03:361
920027220,32sleep39245-21awk19:58:313
637827222,25sleep20-21swapper/219:48:372
580427219,37sleep35901-21fschecks_time19:48:253
1378627215,26sleep20-21swapper/223:38:382
2004127113,42sleep120048-21/usr/sbin/munin20:33:251
1557227118,24sleep10-21swapper/122:03:081
1480627114,18sleep10-21swapper/120:18:071
1462227119,36sleep214667-21ntp_kernel_pll_21:58:292
584827020,30sleep20-21swapper/221:29:022
3054827017,38sleep230611-21ps22:48:482
2484227017,21sleep30-21swapper/300:13:273
2049027012,29sleep30-21swapper/322:18:213
107992707,13sleep10-21swapper/121:47:381
2745126922,30sleep227475-21munin-run20:58:052
1996526921,31sleep320035-21/usr/sbin/munin20:33:233
183862697,10sleep10-21swapper/122:10:071
1243426918,30sleep10-21swapper/121:53:071
3061726820,24sleep20-21swapper/200:33:092
3054026815,23sleep10-21swapper/122:48:471
289126815,26sleep30-21swapper/323:03:353
2376426826,20sleep20-21swapper/220:43:362
2251026827,18sleep30-21swapper/322:23:383
1311926817,30sleep20-21swapper/223:38:052
828126722,22sleep20-21swapper/223:23:082
3097426720,27sleep10-21swapper/119:24:051
2037426719,26sleep20-21swapper/223:58:332
1085826720,25sleep30-21swapper/320:03:363
3088726613,22sleep10-21swapper/100:33:241
3088726613,22sleep10-21swapper/100:33:241
3039426625,26sleep30-21swapper/319:23:263
2906526619,23sleep20-21swapper/219:18:332
282442668,12sleep30-21swapper/322:43:163
243462667,12sleep00-21swapper/019:06:110
116622657,12sleep10-21swapper/123:33:081
898326412,34sleep30-21swapper/323:23:333
50212647,12sleep10-21swapper/123:12:041
311426417,32sleep13191-21latency_hist21:23:071
2523126427,17sleep10-21swapper/122:33:241
997726317,22sleep20-21swapper/223:28:082
851626314,30sleep18510-21apt-get23:23:191
565526321,16sleep30-21swapper/323:13:283
2612926313,20sleep10-21swapper/100:18:141
2200526319,25sleep10-21swapper/120:38:341
2056726313,21sleep10-21swapper/122:18:231
1501126319,25sleep315204-21apt-config20:18:093
1449726317,31sleep314520-21/usr/sbin/munin21:58:263
111226321,23sleep30-21swapper/319:33:233
2180126219,21sleep20-21swapper/222:23:082
1203626214,31sleep112144-21perl20:08:261
6702610,6sleep30-21swapper/322:58:083
456026116,30sleep14606-21/usr/sbin/munin19:43:311
3227026119,20sleep30-21swapper/321:13:083
3056426122,24sleep10-21swapper/100:33:081
2828426117,27sleep128370-21sed00:23:351
2693326113,14sleep20-21swapper/222:38:262
1996326116,27sleep219955-21apt-get20:33:222
73826021,24sleep1769-21munin-run19:33:051
64172608,12sleep20-21swapper/221:33:102
2788926020,25sleep127975-21fw_packets20:58:241
2664226017,22sleep326727-21mailstats00:18:323
2648526016,21sleep30-21swapper/322:38:073
1043326016,18sleep20-21swapper/221:43:392
47832598,15sleep20-21swapper/223:09:032
3218225918,25sleep232269-21gpgv21:13:072
2985925916,26sleep329887-21munin-run19:23:063
2511025917,20sleep30-21swapper/320:48:333
2511025917,20sleep30-21swapper/320:48:333
2277425921,17sleep30-21swapper/300:08:063
672225813,29sleep36814-21/usr/sbin/munin21:33:273
3064225820,18sleep30-21swapper/321:08:073
2814425818,25sleep228183-21ntp_kernel_pll_20:58:292
2514725819,20sleep20-21swapper/222:33:212
231325815,20sleep30-21swapper/321:18:373
1793425819,18sleep20-21swapper/222:08:352
49025714,31sleep1517-21cat22:58:071
3005925716,19sleep10-21swapper/121:03:371
2498925716,24sleep225025-21ntp_states00:13:302
1345225715,18sleep10-21swapper/123:38:091
85325614,26sleep1937-21perl22:58:241
3275025612,17sleep30-21swapper/319:28:393
2532625616,22sleep20-21swapper/220:48:362
2532625616,22sleep20-21swapper/220:48:362
249825615,24sleep22555-21irqstats23:03:252
161872565,11sleep30-21swapper/323:44:163
1498825617,24sleep314961-21latency_hist23:43:073
933725515,18sleep20-21swapper/221:43:072
3169625515,17sleep20-21swapper/222:53:242
3101925515,24sleep331068-21irqstats00:33:263
3101925515,24sleep331068-21irqstats00:33:263
1890325513,26sleep318898-21apt-get22:13:203
1721325517,23sleep217237-21ntp_states23:48:352
91012545,37sleep10-21swapper/121:40:401
58752546,36sleep10-21swapper/121:29:211
539225412,15sleep30-21swapper/321:28:343
1653825414,19sleep20-21swapper/223:48:072
103362545,12sleep20-21swapper/220:03:102
2907425313,24sleep329083-21munin-run21:03:073
1726525311,18sleep30-21swapper/320:23:303
782425214,21sleep17877-21users23:18:391
2431625217,19sleep20-21swapper/200:13:062
2264625217,20sleep122719-21threads22:23:401
2165225215,18sleep20-21swapper/200:03:252
1580425213,23sleep315801-21apt-get22:03:193
983725114,20sleep30-21swapper/321:43:283
460725112,23sleep24612-21ntp_states19:43:322
245862516,34sleep10-21swapper/122:29:411
1685325114,14sleep10-21swapper/123:48:261
1532625113,22sleep115299-21apt-get20:18:311
1329425112,18sleep30-21swapper/320:13:083
2939225013,15sleep20-21swapper/221:03:222
2198425014,22sleep321993-21uname20:38:323
193142499,17sleep20-21swapper/222:13:312
185624913,16sleep20-21swapper/221:18:242
115792499,26sleep211585-21ntp_states21:48:322
786824713,18sleep10-21swapper/119:53:361
152624712,14sleep10-21swapper/119:33:341
1627324512,13sleep20-21swapper/222:03:342
1990424310,16sleep30-21swapper/323:58:173
27692426,14sleep30-21swapper/319:38:243
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional