You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-14 - 11:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Nov 14, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
226922710712,7080sleep215115-21kworker/u8:322:19:462
27325257845,5768sleep127259-21kworker/1:000:15:491
13377238015,3763sleep113271-21kworker/1:020:06:571
10295230325,3016sleep05370-21kworker/0:123:22:310
2275327865,766sleep18967-1kworker/1:0H22:19:531
1773925356,512sleep1195-21jbd2/mmcblk2p2-22:04:231
1421023717,313sleep114246-21ntpq20:09:351
1127234621,308sleep21190-21/usr/sbin/munin22:54:232
21433234412,192sleep121582-21sort20:34:101
7931233718,303sleep27985-21/usr/sbin/munin23:14:382
26949232316,292sleep026970-21nin-node19:09:090
20673230812,280sleep120770-21timerandwakeup23:54:381
11943230690,199sleep112092-21unixbench_multi21:44:401
244612279135,101sleep124470-21modprobe20:44:071
6247226695,23sleep10-21swapper/123:09:361
27284226232,210sleep127440-21unixbench_singl20:49:491
27284226232,210sleep127440-21unixbench_singl20:49:491
29432224619,206sleep029426-21apt-get00:24:240
24958223649,24sleep10-21swapper/122:24:421
10829222021,185sleep110843-21fschecks_count19:59:301
30821221748,18sleep10-21swapper/119:19:281
30825220919,169sleep130947-21dpkg00:29:101
26971220715,172sleep126958-21/usr/sbin/munin19:09:101
14155220057,21sleep10-21swapper/121:54:091
13520220018,157sleep113578-21sendmail23:34:071
15474219125,151sleep115553-21fschecks_time23:39:281
6585219030,116sleep16559-21/usr/sbin/munin21:29:221
6585219030,116sleep16559-21/usr/sbin/munin21:29:221
31760218812,159sleep131835-21latency_hist22:49:091
31624218510,132sleep131663-21ntp_kernel_err21:04:351
8534218216,16sleep10-21swapper/121:34:351
724217116,135sleep10-21swapper/119:24:411
5124217010,147sleep15144-21fschecks_count21:24:291
3188521697,20sleep10-21swapper/100:30:191
158121676,47sleep10-21swapper/122:54:341
29904216516,128sleep10-21swapper/120:59:341
29904216516,128sleep10-21swapper/120:59:341
7443216217,126sleep10-21swapper/123:14:241
10805215532,26sleep10-21swapper/123:24:301
906215415,93sleep11055-21latency_hist22:54:091
1044921548,111sleep1362-21snmpd19:59:051
26375215319,113sleep326519-21grep19:04:403
621215212,88sleep10-21swapper/121:09:311
3260215211,105sleep13343-21sed22:59:391
1404921529,18sleep10-21swapper/123:34:311
24218215112,125sleep024249-21grep22:24:250
2825621506,100sleep00-21swapper/000:19:400
2825621506,100sleep00-21swapper/000:19:400
5016214826,71sleep15091-21run-parts19:39:391
28818214618,82sleep10-21swapper/120:54:491
28818214618,82sleep10-21swapper/120:54:491
473621456,44sleep10-21swapper/123:04:391
1824621449,100sleep118345-21latency_hist23:49:091
25393214017,88sleep125500-21latency_hist19:04:101
16975214014,21sleep00-21swapper/020:19:230
14254214017,97sleep214455-21apt-config21:54:102
20624213614,106sleep220652-21latency_hist22:14:092
8669213314,82sleep20-21swapper/221:34:402
3009921316,41sleep10-21swapper/122:43:191
626821266,108sleep16320-21if_eth019:44:321
2732621248,15sleep10-21swapper/122:34:191
2732621248,15sleep10-21swapper/122:34:191
2919121227,80sleep129194-21latency_hist00:24:091
2919121227,80sleep129194-21latency_hist00:24:091
29266212118,83sleep329320-21latency_hist00:24:113
29266212118,83sleep329320-21latency_hist00:24:113
10474212112,94sleep110441-21latency_hist23:24:091
873921195,103sleep00-21swapper/019:52:300
21715211930,14sleep10-21swapper/120:34:261
15129211912,93sleep115193-21sendmail_mailst21:54:501
3045211819,76sleep23060-21fschecks_count19:34:272
2209921165,10sleep00-21swapper/022:18:430
2209921165,10sleep00-21swapper/022:18:430
21901211317,75sleep021969-21ls23:59:330
29761211210,89sleep129857-21sed19:14:401
29761211210,89sleep129857-21sed19:14:401
2160721129,17sleep10-21swapper/123:59:221
19521211230,71sleep10-21swapper/123:51:191
4798211168,19sleep10-21swapper/121:24:091
2049211118,83sleep00-21swapper/022:54:490
20166211015,71sleep220045-21/usr/sbin/munin23:54:272
31338210514,77sleep231229-21/usr/sbin/munin21:04:272
2541221057,15sleep10-21swapper/120:44:521
2541221057,15sleep10-21swapper/120:44:521
24530210323,57sleep224592-21sendmail_mailqu18:59:352
17067210316,65sleep117100-21fschecks_count20:19:281
194221028,16sleep10-21swapper/121:14:241
14601210127,39sleep00-21swapper/021:54:200
29281210026,47sleep329372-21cut19:14:293
29281210026,47sleep329372-21cut19:14:293
16546210012,72sleep016547-21sshd23:42:500
2601129913,36sleep10-21swapper/122:29:301
2392529917,53sleep024073-21unixbench_singl20:39:420
1018029946,41sleep110244-21ps21:39:351
740129817,64sleep37554-21latency_hist19:49:103
17992987,14sleep10-21swapper/119:29:341
17992987,14sleep10-21swapper/119:29:341
277029716,67sleep22801-21fschecks_count22:59:272
251272978,76sleep125168-21ntpq00:09:321
79752969,14sleep10-21swapper/119:49:321
275972969,54sleep10-21swapper/119:09:231
1638629613,68sleep016441-21if_eth021:59:290
3134029517,22sleep00-21swapper/019:19:390
154182957,13sleep10-21swapper/120:14:191
208682948,45sleep10-21swapper/122:14:201
208682948,45sleep10-21swapper/122:14:201
3113429324,44sleep231299-21sed22:44:422
3119129218,55sleep230706-21apt-get00:29:192
1970929128,52sleep10-21swapper/120:28:081
232492897,44sleep10-21swapper/100:04:271
232492897,44sleep10-21swapper/100:04:271
37328811,63sleep2446-21/usr/sbin/munin19:24:382
409328716,51sleep24202-21timerandwakeup21:19:382
3228228716,53sleep332366-21/usr/sbin/munin22:49:313
3082128616,51sleep330428-21/usr/sbin/munin22:44:343
2702628617,43sleep227203-21sed19:09:112
913728516,44sleep29156-21apt-get23:19:222
3172428514,52sleep031726-21ntp_states00:30:140
2532028529,31sleep325354-21munin-run19:04:083
2146428521,48sleep321549-21threads22:14:383
2146428521,48sleep321549-21threads22:14:383
193542856,45sleep10-21swapper/122:09:231
1782528523,27sleep20-21swapper/222:04:282
173722857,11sleep10-21swapper/122:03:101
3263828317,30sleep00-21swapper/019:24:320
2532928312,27sleep225330-21sshd22:27:172
1982128316,51sleep019846-21sed23:54:080
1076428315,52sleep20-21swapper/223:24:272
1828728213,54sleep218260-21latency_hist20:24:102
40932817,42sleep10-21swapper/119:37:291
2859128011,54sleep028602-21ntpq20:54:440
2859128011,54sleep028602-21ntpq20:54:440
243027919,42sleep32440-21true22:59:083
1978427916,47sleep319786-21cron20:29:073
1616327916,50sleep216170-21acpi21:59:192
1476327922,41sleep314778-21fschecks_count21:54:403
3025027816,15sleep20-21swapper/220:59:432
3025027816,15sleep20-21swapper/220:59:432
2030627818,46sleep320311-21sh22:11:073
1737627818,41sleep317422-21ntp_offset20:19:343
1039427820,35sleep010550-21grep21:39:390
642127712,53sleep06542-21sed23:09:390
2851127713,30sleep20-21swapper/220:54:422
2851127713,30sleep20-21swapper/220:54:422
2375127710,47sleep223820-21switchtime00:04:402
2375127710,47sleep223820-21switchtime00:04:402
2947827623,22sleep20-21swapper/222:39:412
2900227622,35sleep029090-21grep22:39:310
2383527625,32sleep323954-21timerwakeupswit00:04:413
2383527625,32sleep323954-21timerwakeupswit00:04:413
2140627620,28sleep00-21swapper/020:34:090
1628327614,47sleep316380-21fschecks_time21:59:283
1050427623,29sleep20-21swapper/219:59:102
3204527520,25sleep20-21swapper/219:24:092
3006427522,25sleep00-21swapper/020:59:370
3006427522,25sleep00-21swapper/020:59:370
273527524,31sleep30-21swapper/322:59:243
2519027512,24sleep00-21swapper/000:09:350
2447427514,46sleep224526-21apt-config20:44:082
2381427519,31sleep30-21swapper/320:39:413
618027415,40sleep26214-21/usr/sbin/munin23:09:332
4962745,11sleep00-21swapper/022:50:090
241722746,11sleep00-21swapper/000:05:470
241722746,11sleep00-21swapper/000:05:470
1850827412,27sleep30-21swapper/322:04:413
3100527317,41sleep331014-21munin-run21:04:083
3100527317,41sleep331014-21munin-run21:04:083
2053527320,26sleep30-21swapper/323:54:363
2781027215,27sleep30-21swapper/320:54:073
2781027215,27sleep30-21swapper/320:54:073
1869227217,34sleep20-21swapper/223:49:302
474827111,45sleep04797-21unin-run21:24:080
2958227117,36sleep029616-21/usr/sbin/munin19:14:350
2958227117,36sleep029616-21/usr/sbin/munin19:14:350
2396527112,43sleep023974-21munin-run22:24:080
181632715,57sleep00-21swapper/023:48:430
459527020,32sleep30-21swapper/319:39:293
2034727015,38sleep220393-21sed20:29:352
60342696,15sleep00-21swapper/021:25:500
42032697,14sleep30-21swapper/321:19:393
2757326918,27sleep30-21swapper/319:09:213
247292698,43sleep30-21swapper/300:09:213
227022687,44sleep30-21swapper/322:19:513
1819526823,28sleep318226-21modprobe23:49:073
1410826814,26sleep20-21swapper/223:34:322
134872685,19sleep00-21swapper/023:33:410
1122826813,23sleep30-21swapper/319:59:403
73732676,13sleep00-21swapper/023:14:230
652826716,24sleep00-21swapper/019:44:390
632126730,21sleep26035-21/usr/sbin/munin19:44:332
278132667,13sleep20-21swapper/222:34:362
278132667,13sleep20-21swapper/222:34:362
228026615,28sleep00-21swapper/019:29:410
228026615,28sleep00-21swapper/019:29:410
2226026618,23sleep20-21swapper/222:19:102
2226026618,23sleep20-21swapper/222:19:102
130192668,13sleep00-21swapper/021:49:280
89626519,21sleep00-21swapper/021:09:380
453726512,35sleep34604-21memory23:04:333
2179526521,22sleep20-21swapper/223:59:312
1867326515,32sleep218737-21/usr/sbin/munin20:24:302
399726419,28sleep04122-21grep23:04:100
2608826420,21sleep20-21swapper/200:14:092
192862649,39sleep019353-21df22:09:220
1290826419,23sleep30-21swapper/321:49:273
964326319,26sleep30-21swapper/319:54:353
782526320,26sleep37831-21ntp_states23:14:353
53426324,24sleep30-21swapper/321:09:293
2862126315,26sleep00-21swapper/019:11:070
2537126316,27sleep025391-21munin-run19:04:090
137062635,12sleep00-21swapper/023:34:100
1282626316,25sleep00-21swapper/020:04:410
925326211,13sleep00-21swapper/019:54:240
50342629,38sleep05137-21users23:04:420
1430226215,31sleep014308-21/usr/sbin/munin20:09:380
1229526215,33sleep212310-21cpu20:04:272
1024626213,28sleep210315-21/usr/sbin/munin21:39:372
979426113,33sleep39798-21diskstats21:39:253
938826121,22sleep30-21swapper/323:19:433
3170826114,29sleep331714-21/usr/sbin/munin21:04:373
2655226117,26sleep00-21swapper/020:49:230
2655226117,26sleep00-21swapper/020:49:230
1674926117,18sleep30-21swapper/320:19:103
553626023,20sleep20-21swapper/223:09:082
3236826015,27sleep332373-21/usr/sbin/munin19:24:273
2985826014,25sleep20-21swapper/219:14:412
2985826014,25sleep20-21swapper/219:14:412
226252607,12sleep00-21swapper/020:35:300
1693925919,19sleep20-21swapper/223:44:252
1129325919,21sleep00-21swapper/019:59:410
894525819,20sleep20-21swapper/219:54:102
477825818,20sleep20-21swapper/219:39:322
464625815,25sleep04719-21fw_packets19:39:300
3171525817,19sleep00-21swapper/022:49:080
255225812,17sleep00-21swapper/021:14:390
233325812,31sleep22368-21ntp_kernel_pll_21:14:332
228742584,10sleep30-21swapper/300:03:563
1539925815,28sleep015417-21/usr/sbin/munin20:14:180
2948225719,23sleep229504-21df00:24:262
2786825716,18sleep20-21swapper/219:09:322
1411025719,19sleep30-21swapper/320:09:333
1392625716,26sleep313958-21grep23:34:293
760625615,26sleep37601-21apt-get19:49:223
550625617,21sleep30-21swapper/321:24:393
2230825613,26sleep322436-21cat20:34:413
200025611,27sleep20-21swapper/219:29:392
200025611,27sleep20-21swapper/219:29:392
1813925616,20sleep00-21swapper/022:04:340
2954725520,21sleep329670-21timerwakeupswit22:39:423
2783725515,22sleep30-21swapper/322:34:373
2783725515,22sleep30-21swapper/322:34:373
2771425520,21sleep30-21swapper/300:19:103
2496925513,23sleep324987-21apt-get20:44:223
2496925513,23sleep324987-21apt-get20:44:223
2014625515,17sleep00-21swapper/023:54:240
1294325517,23sleep313089-21sed23:29:413
1229125515,24sleep312288-21apt-get20:04:243
658225413,24sleep36578-21apt-get21:29:203
658225413,24sleep36578-21apt-get21:29:203
4612545,12sleep20-21swapper/222:49:422
3121925412,24sleep331211-21apt-get00:29:243
2838925414,24sleep328412-21irqstats20:54:393
2838925414,24sleep328412-21irqstats20:54:393
260552544,12sleep00-21swapper/020:46:330
260552544,12sleep00-21swapper/020:46:330
2577225416,23sleep025768-21apt-get22:29:210
2552325419,22sleep325532-21systemd-run22:29:083
2499325412,24sleep224987-21apt-get20:44:252
2499325412,24sleep224987-21apt-get20:44:252
229062545,11sleep20-21swapper/220:39:042
1591925413,25sleep315990-21/usr/sbin/munin23:39:383
1549425414,25sleep215500-21df_inode20:14:252
1135625413,16sleep00-21swapper/023:24:410
824425313,23sleep38251-21diskstats21:34:293
665325313,24sleep26657-21df21:29:232
665325313,24sleep26657-21df21:29:232
633925316,16sleep30-21swapper/323:09:383
296225317,16sleep30-21swapper/319:34:243
2788325312,24sleep227877-21apt-get00:19:262
2788325312,24sleep227877-21apt-get00:19:262
275092536,36sleep00-21swapper/000:18:060
2622825310,27sleep226234-21ntp_states22:29:352
213125318,20sleep32276-21uptime19:29:403
213125318,20sleep32276-21uptime19:29:403
1126525312,26sleep211335-21df21:44:252
1078825315,21sleep310796-21diskstats23:24:283
427125215,24sleep04349-21latency_hist19:39:100
2735125211,24sleep027346-21apt-get22:34:220
2735125211,24sleep027346-21apt-get22:34:220
201925213,24sleep32023-21diskstats21:14:263
1555625214,16sleep20-21swapper/223:39:292
1126325211,27sleep011257-21apt-get21:44:230
1084725211,22sleep210950-21perl19:59:312
2586425110,15sleep30-21swapper/322:29:283
2007925115,23sleep320054-21/usr/sbin/munin20:29:283
197432515,36sleep00-21swapper/020:28:360
792425015,15sleep00-21swapper/021:34:090
792425015,15sleep00-21swapper/021:34:090
1462825011,23sleep214624-21apt-get21:54:222
1226225011,18sleep20-21swapper/223:29:232
269592497,27sleep2360-21ntpd20:49:422
269592497,27sleep2360-21ntpd20:49:422
1892624911,20sleep318937-21ntp_states23:49:353
3137524817,16sleep00-21swapper/021:04:300
3080524810,21sleep330814-21df_inode19:19:263
2987924711,22sleep329889-21ntp_states00:24:393
2707224712,15sleep20-21swapper/200:14:452
149252464,11sleep20-21swapper/220:11:502
44124510,18sleep2533-21perl21:09:282
814224411,18sleep00-21swapper/021:34:220
514524414,16sleep20-21swapper/221:24:302
1166224411,16sleep30-21swapper/321:44:343
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional