You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-02 - 07:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Thu Apr 02, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
274982688125,6844sleep224343-21kworker/2:200:19:512
3832245646,4518sleep224082-21kworker/u8:223:07:212
23278286924,826sleep123321-21/usr/sbin/munin19:02:321
25020271418,676sleep025086-21/usr/sbin/munin20:47:360
5031243933,81sleep25007-21latency_hist19:47:082
6524237717,311sleep26527-21cron19:52:042
31962234923,307sleep332078-21/usr/sbin/munin19:28:023
2445232829,286sleep22459-21grep19:37:262
2711523248,279sleep227267-21date20:57:072
42762316114,187sleep24318-21fschecks_count21:27:252
2209523008,254sleep222109-21grep00:02:312
325582287193,75sleep232575-21modprobe00:37:042
1304922578,195sleep213193-21latency_hist20:12:082
24877225555,177sleep225057-21latency_hist22:32:072
15951225443,190sleep216103-21unixbench_singl23:42:352
1424322426,201sleep213441-21snmpd20:13:172
228872239121,76sleep222889-21munin-run22:27:062
14378223935,81sleep20-21swapper/223:37:382
14378223935,81sleep20-21swapper/223:37:382
412122360,7sleep10-21swapper/119:42:291
13885222317,190sleep013919-21fschecks_count21:57:440
13885222317,190sleep013919-21fschecks_count21:57:440
14286221985,118sleep214346-21sendmail_mailqu21:57:582
14286221985,118sleep214346-21sendmail_mailqu21:57:582
21491221310,162sleep224197-21cyclictest00:02:062
16964221240,57sleep216981-21fschecks_count22:07:232
30943220917,165sleep131136-21apt-config19:27:071
18958220727,109sleep218971-21netstat23:52:292
18227220311,41sleep20-21swapper/220:27:272
7518220115,69sleep20-21swapper/221:37:262
8368220049,63sleep28430-21apt-key23:22:072
5446219117,150sleep15461-21ls21:32:041
934621909,142sleep29431-21switchtime23:22:522
5772217743,95sleep25808-21ntp_kernel_pll_23:12:302
1121721779,37sleep20-21swapper/223:27:532
30737217448,57sleep230765-21idleruntime-cro19:27:042
23351217419,111sleep20-21swapper/200:07:192
28948216422,18sleep10-21swapper/121:02:241
300621638,142sleep23026-21netstat21:22:282
7577216019,126sleep37615-21latency21:37:273
31970215812,127sleep232021-21sort21:12:222
16643215110,121sleep20-21swapper/222:07:072
16643215110,121sleep20-21swapper/222:07:072
31117214815,118sleep131101-21latency_hist22:52:051
5735214615,116sleep05736-21cpu21:32:200
29729214614,115sleep029748-21apt-get00:27:160
10767214541,57sleep20-21swapper/221:47:282
32580214320,51sleep20-21swapper/222:56:312
25552214129,75sleep225699-21sed00:12:362
4083213812,112sleep33799-21/usr/sbin/munin23:07:383
28619213510,111sleep228780-21unixbench_multi19:17:352
28619213510,111sleep228780-21unixbench_multi19:17:352
12294213218,36sleep20-21swapper/223:32:252
12294213218,36sleep20-21swapper/223:32:252
2868321318,51sleep20-21swapper/222:42:292
193421309,87sleep21970-21date19:37:062
6284212929,54sleep20-21swapper/221:32:342
18512212710,47sleep20-21swapper/222:12:242
10590212613,42sleep310653-21/usr/sbin/munin21:47:253
1184212420,88sleep31269-21tune2fs21:17:243
10663212410,96sleep010674-21ntp_states20:02:570
1635221239,101sleep113441-21snmpd22:03:591
1635221239,101sleep113441-21snmpd22:03:591
8750211921,80sleep08735-21apt-get23:22:220
2866221199,72sleep228603-21latency_hist21:02:082
11814211821,78sleep111855-21fschecks_count20:07:221
23212211616,85sleep223256-21ntpq19:02:292
8363211529,58sleep18382-21apt-get19:57:171
8363211529,58sleep18382-21apt-get19:57:171
26929211419,76sleep326979-21if_eth022:37:253
26929211419,76sleep326979-21if_eth022:37:253
12113211416,72sleep20-21swapper/221:52:252
26617211313,37sleep20-21swapper/219:12:262
26617211313,37sleep20-21swapper/219:12:262
1218211328,69sleep21358-21head22:57:362
26003211121,72sleep30-21swapper/320:52:283
13215211116,76sleep113253-21dpkg21:57:041
731121109,16sleep10-21swapper/121:37:211
28204211027,72sleep228188-21apt-get00:22:222
28204211027,72sleep228188-21apt-get00:22:222
25115211032,32sleep225134-21apt-get22:32:152
3222210929,68sleep20-21swapper/223:06:042
2549521088,14sleep20-21swapper/220:52:072
13671210713,79sleep313704-21ntp_kernel_pll_20:12:293
22819210614,77sleep022833-21apt22:27:040
24860210412,59sleep10-21swapper/119:07:321
26723210315,18sleep10-21swapper/122:37:191
26723210315,18sleep10-21swapper/122:37:191
12059210314,72sleep012076-21apt23:32:150
12059210314,72sleep012076-21apt23:32:150
27567210111,73sleep127580-21uname20:57:271
28303210011,76sleep00-21swapper/020:58:350
23517210013,68sleep023595-21/usr/sbin/munin22:27:530
2054921006,12sleep20-21swapper/220:34:172
10825210014,30sleep10-21swapper/123:27:441
1640829921,38sleep20-21swapper/220:22:182
1877529814,69sleep318837-21if_err_eth023:52:263
3195829719,62sleep132074-21ls00:32:351
209602975,36sleep20-21swapper/222:18:062
109529718,38sleep20-21swapper/221:17:192
93952966,36sleep20-21swapper/219:58:432
93952966,36sleep20-21swapper/219:58:432
660329616,68sleep00-21swapper/023:16:320
2319629610,18sleep20-21swapper/220:42:332
1616029640,14sleep20-21swapper/220:22:062
58929514,63sleep132544-21apt-get19:32:161
56332958,17sleep30-21swapper/323:12:273
3172729511,41sleep20-21swapper/221:12:062
310622948,49sleep20-21swapper/200:32:072
1171829412,65sleep011860-21cat21:52:070
2194429320,33sleep20-21swapper/220:37:372
943129219,35sleep20-21swapper/221:42:352
2669929220,30sleep20-21swapper/222:37:162
2669929220,30sleep20-21swapper/222:37:162
113442925,11sleep20-21swapper/220:05:502
3042229110,17sleep10-21swapper/121:07:201
289792919,66sleep029123-21unixbench_singl22:42:350
1700629115,15sleep20-21swapper/223:47:242
60929010,14sleep30-21swapper/319:32:273
2846529011,62sleep028537-21sensors19:17:330
2846529011,62sleep028537-21sensors19:17:330
1494929013,62sleep114992-21fschecks_count20:17:241
2902028913,61sleep229082-21idleruntime21:02:252
246028916,53sleep30-21swapper/319:37:273
2341928934,30sleep323515-21sed19:02:343
2331728914,55sleep322833-21apt-get22:27:163
1365628716,54sleep113733-21apt-get21:57:081
1365628716,54sleep113733-21apt-get21:57:081
980028620,21sleep30-21swapper/320:02:073
980028620,21sleep30-21swapper/320:02:073
2498128611,61sleep025072-21sed19:07:360
681428527,46sleep26860-21grep19:52:212
549928519,37sleep00-21swapper/021:32:070
287022859,55sleep10-21swapper/100:23:051
287022859,55sleep10-21swapper/100:23:051
1696728519,49sleep116494-21apt-get23:47:151
1015528422,43sleep110174-21apt-get20:02:171
2853828316,53sleep128614-21timerandwakeup19:17:341
2853828316,53sleep128614-21timerandwakeup19:17:341
2632228331,31sleep30-21swapper/319:12:153
2632228331,31sleep30-21swapper/319:12:153
2058928217,49sleep020683-21switchtime22:17:350
1580028215,48sleep015863-21awk22:02:330
1580028215,48sleep015863-21awk22:02:330
905228117,48sleep19108-21if_eth021:42:261
3269228117,46sleep032703-21unin-run22:57:050
2671728115,51sleep0367-21sshd19:12:280
2671728115,51sleep0367-21sshd19:12:280
2019028128,20sleep30-21swapper/320:32:373
2014928120,46sleep320151-21acpi23:57:143
1015828125,28sleep10-21swapper/123:27:071
2620328011,55sleep126267-21ps20:52:341
2311428027,25sleep10-21swapper/120:42:321
1951028011,15sleep00-21swapper/020:32:220
1689328012,54sleep116386-21/usr/sbin/munin20:22:331
1996427916,17sleep00-21swapper/023:57:070
114922799,13sleep10-21swapper/121:49:491
1000827917,38sleep30-21swapper/323:27:053
739527814,43sleep07442-21fschecks_count21:37:240
34742788,21sleep10-21swapper/119:42:061
330027824,35sleep13326-21munin-run23:07:041
3224127812,15sleep10-21swapper/121:12:291
247712789,30sleep20-21swapper/219:07:292
3042127722,33sleep030416-21apt-get21:07:190
2953827721,35sleep11-21systemd22:47:051
2564627712,50sleep125654-21ntpq22:33:031
1882427728,30sleep00-21swapper/022:12:300
1795927716,20sleep30-21swapper/320:27:203
1686227721,33sleep316868-21ntp_states20:22:313
712027614,46sleep37266-21latency_hist21:37:073
282327616,43sleep32881-21if_err_eth021:22:253
312682756,32sleep20-21swapper/219:27:212
2459427420,27sleep10-21swapper/100:12:071
2084427422,24sleep10-21swapper/120:37:071
2939627316,36sleep329465-21sensors_temp21:02:343
828027220,24sleep10-21swapper/123:22:061
2978627228,33sleep20-21swapper/222:47:102
2204527218,29sleep10-21swapper/122:22:331
1541027218,34sleep315184-21/usr/sbin/munin23:42:243
1042527214,29sleep30-21swapper/320:02:523
103127219,22sleep00-21swapper/022:57:340
39952717,13sleep30-21swapper/319:42:283
2641427111,30sleep30-21swapper/300:17:183
2618527115,40sleep026325-21latency_hist00:17:070
2474627113,41sleep124876-21latency_hist22:32:061
2017727122,22sleep00-21swapper/023:57:200
1619327114,38sleep18127-1kworker/1:0H23:43:311
79802705,56sleep00-21swapper/023:18:580
1579027014,43sleep315797-21open_files22:02:323
1579027014,43sleep315797-21open_files22:02:323
2458526915,35sleep124627-21ntpq20:47:291
396726818,29sleep10-21swapper/123:07:361
337726822,22sleep30-21swapper/323:07:063
3076726818,25sleep30-21swapper/319:27:053
2338726811,37sleep00-21swapper/020:42:350
227142685,52sleep00-21swapper/000:03:340
2211026817,37sleep322150-21ntp_kernel_pll_00:02:323
657726719,27sleep10-21swapper/119:52:081
3100926717,32sleep331010-21cron00:32:043
3009026714,26sleep330183-21memory22:47:273
2853726715,24sleep30-21swapper/322:42:263
180626713,25sleep112-21rcu_sched23:02:071
1767026715,34sleep017684-21modprobe20:27:040
1434926719,29sleep314416-21/usr/sbin/munin21:57:593
1434926719,29sleep314416-21/usr/sbin/munin21:57:593
1004026716,28sleep00-21swapper/021:46:050
2687526616,28sleep00-21swapper/000:17:310
2066326611,42sleep120737-21mailstats23:57:331
1832926619,26sleep00-21swapper/023:52:060
1332126620,29sleep313442-21dpkg21:57:063
1254726615,20sleep10-21swapper/123:32:331
1254726615,20sleep10-21swapper/123:32:331
117126610,23sleep10-21swapper/121:17:221
1064326618,23sleep00-21swapper/023:27:390
61912655,12sleep00-21swapper/019:47:460
547226521,28sleep15568-21grep23:12:251
3178126518,21sleep30-21swapper/321:12:073
3172626513,36sleep00-21swapper/000:32:290
2975626512,24sleep10-21swapper/100:27:221
1299726516,31sleep312060-21/usr/sbin/munin23:32:393
1299726516,31sleep312060-21/usr/sbin/munin23:32:393
736626422,21sleep00-21swapper/019:52:350
3262426420,19sleep30-21swapper/322:57:043
325912645,46sleep10-21swapper/122:56:381
304682645,11sleep00-21swapper/019:23:350
254326415,33sleep02576-21/usr/sbin/munin19:37:280
898926317,29sleep09067-21find19:57:360
898926317,29sleep09067-21find19:57:360
1617226319,20sleep30-21swapper/320:22:073
350726219,18sleep00-21swapper/023:07:070
3177026214,18sleep00-21swapper/019:27:590
3095426222,22sleep00-21swapper/022:50:310
2040726217,24sleep30-21swapper/322:17:303
1513226214,17sleep00-21swapper/020:17:270
1384726219,22sleep00-21swapper/020:12:340
708226115,30sleep37165-21perl23:17:243
2570826115,30sleep025727-21apt-get20:52:170
214126117,22sleep00-21swapper/023:02:260
2068626114,32sleep120802-21sed22:17:361
311712605,12sleep30-21swapper/300:32:083
295482600,6sleep30-21swapper/300:27:063
295482600,6sleep30-21swapper/300:27:063
1879426017,23sleep30-21swapper/322:12:293
1222126014,32sleep012253-21/usr/sbin/munin21:52:270
62925913,27sleep00-21swapper/019:32:400
621825913,28sleep16283-21awk21:32:331
465925917,18sleep30-21swapper/321:27:343
38562596,11sleep10-21swapper/121:26:311
1836525917,25sleep118408-21ntpq20:27:301
1734625918,18sleep30-21swapper/322:07:333
156425913,28sleep01628-21ps21:17:330
1092725915,19sleep30-21swapper/323:27:463
2840025814,20sleep30-21swapper/319:17:323
2840025814,20sleep30-21swapper/319:17:323
282302587,39sleep00-21swapper/000:22:400
282302587,39sleep00-21swapper/000:22:400
2571125816,24sleep325789-21munin-node19:12:073
2352125818,19sleep00-21swapper/019:02:350
1810525815,23sleep00-21swapper/020:27:250
1695825813,30sleep116963-21diskstats22:07:221
912825713,28sleep39222-21memory21:42:283
2592925718,23sleep10-21swapper/119:12:091
2592925718,23sleep10-21swapper/119:12:091
1985125715,23sleep119863-21modprobe23:57:041
2886225613,27sleep028856-21apt-get21:02:180
2716425613,26sleep027175-21ntp_states22:37:310
2716425613,26sleep027175-21ntp_states22:37:310
2486525616,16sleep30-21swapper/320:47:353
2372725618,17sleep30-21swapper/300:07:283
1689125615,23sleep316494-21apt-get23:47:093
148562565,12sleep10-21swapper/123:41:131
148562565,12sleep10-21swapper/123:41:131
1336025615,25sleep113370-21forks20:12:231
47625515,24sleep3473-21apt-get22:57:163
3236625514,17sleep30-21swapper/321:12:313
2520825516,18sleep30-21swapper/300:12:293
78025414,25sleep0810-21munin-run21:17:050
306725413,24sleep03093-21ntp_states21:22:300
2545525412,27sleep025473-21grep22:32:580
2511125417,21sleep324628-21apt-get22:32:123
2393825414,25sleep324048-21cat20:47:073
2162425416,17sleep30-21swapper/320:37:343
2108025413,25sleep121073-21apt-get20:37:191
1545125414,16sleep00-21swapper/023:42:250
3126725316,22sleep131256-21apt-get19:27:201
2337825312,24sleep023480-21perl00:07:210
2169325314,21sleep00-21swapper/020:37:350
1900225319,17sleep10-21swapper/122:12:351
1371225316,21sleep313738-21df23:37:233
1371225316,21sleep313738-21df23:37:233
681025213,23sleep36807-21apt-get19:52:183
652825213,24sleep36573-21systemd-run19:52:053
418625219,18sleep00-21swapper/019:42:330
2767525211,19sleep10-21swapper/100:22:041
2544425212,25sleep125541-21timerandwakeup00:12:351
2234825210,29sleep122474-21cat00:02:381
2201225216,21sleep322018-21ntp_states22:22:313
1708625214,21sleep00-21swapper/022:07:250
815325012,18sleep10-21swapper/119:57:081
137402478,16sleep00-21swapper/023:37:240
137402478,16sleep00-21swapper/023:37:240
1219924716,15sleep30-21swapper/320:07:333
2767324212,16sleep30-21swapper/320:57:293
117282428,17sleep00-21swapper/020:07:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional