You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-09 - 11:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sun Nov 09, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11286222028,2124sleep09114-21kworker/u8:021:09:440
73252194212,1913sleep27326-21ntp_states22:39:512
29905212699,1243sleep030056-21/usr/sbin/munin23:50:000
29905212699,1243sleep030056-21/usr/sbin/munin23:50:000
12974269217,660sleep113037-21/usr/sbin/munin21:14:461
3295265317,622sleep014035-21kworker/u8:100:09:460
3179253614,509sleep13192-21irqstats20:44:511
18457249414,463sleep218491-21/usr/sbin/munin19:49:472
19790232221,282sleep119934-21/usr/sbin/munin21:35:121
19790232221,282sleep119934-21/usr/sbin/munin21:35:121
308992320119,61sleep030904-21fschecks_count23:55:040
2538123126,12sleep00-21swapper/021:54:440
2538123126,12sleep00-21swapper/021:54:440
677322989,256sleep06819-21cat20:55:480
31950229792,185sleep031981-21cat20:34:240
31950229792,185sleep031981-21cat20:34:240
2891822488,226sleep029068-21latency_hist23:49:270
2891822488,226sleep029068-21latency_hist23:49:270
1085223290,68sleep01125-21systemd-run20:39:250
1085223290,68sleep01125-21systemd-run20:39:250
26055221412,188sleep326096-21grep23:39:413
9260220014,141sleep03360-21cyclictest19:19:580
22995219929,140sleep00-21swapper/020:04:460
8388219628,58sleep00-21swapper/000:24:560
15632219416,160sleep015723-21mailstats19:39:560
173521938,173sleep01774-21ntpq20:39:490
6092218427,62sleep00-21swapper/022:34:560
18138218320,131sleep318293-21grep21:30:093
18138218320,131sleep318293-21grep21:30:093
91121795,163sleep00-21swapper/000:01:460
26131217812,24sleep00-21swapper/020:14:470
2482421788,157sleep00-21swapper/020:09:500
2482421788,157sleep00-21swapper/020:09:500
647321775,161sleep00-21swapper/019:10:290
22035217715,143sleep222147-21timerwakeupswit23:25:112
13109217210,19sleep00-21swapper/022:59:380
1378216810,143sleep01524-21unixbench_singl22:19:590
311721678,19sleep00-21swapper/020:44:500
1922021667,127sleep00-21swapper/021:34:590
1922021667,127sleep00-21swapper/021:34:590
17933216611,139sleep018000-21sendmail_mailqu21:30:060
17933216611,139sleep018000-21sendmail_mailqu21:30:060
1717921638,16sleep00-21swapper/021:29:260
1717921638,16sleep00-21swapper/021:29:260
24983216114,107sleep025044-21sendmail_mailqu23:34:540
21150215912,130sleep221095-21latency_hist19:59:272
10475215884,22sleep00-21swapper/021:06:540
15125215615,104sleep014639-21/usr/sbin/munin23:04:550
13078214812,118sleep313164-21rm19:34:263
13078214812,118sleep313164-21rm19:34:263
17744214510,96sleep017890-21cut23:14:280
17744214510,96sleep017890-21cut23:14:280
5822214239,63sleep05920-21latency_hist20:54:260
10938214223,92sleep211084-21head19:25:002
8823214154,38sleep08401-21/usr/sbin/munin22:44:490
9185214011,92sleep00-21swapper/021:04:270
2882021386,13sleep00-21swapper/020:23:590
342621346,93sleep00-21swapper/022:28:590
11556213419,87sleep10-21swapper/119:29:271
13015213120,95sleep213039-21apt-get19:34:252
13015213120,95sleep213039-21apt-get19:34:252
11760212910,34sleep00-21swapper/022:54:460
1291421275,55sleep00-21swapper/019:33:060
1291421275,55sleep00-21swapper/019:33:060
2412621265,94sleep00-21swapper/023:33:370
14369212621,88sleep114375-21kworker/u8:223:04:241
3747212516,44sleep00-21swapper/022:29:400
2808021235,53sleep00-21swapper/022:03:570
2808021235,53sleep00-21swapper/022:03:570
2083221228,87sleep019278-21kworker/0:023:24:120
1773321189,38sleep00-21swapper/019:46:540
25793211519,15sleep30-21swapper/320:14:253
25793211519,15sleep30-21swapper/320:14:253
2497921159,56sleep025037-21idleruntime-cro21:54:240
2497921159,56sleep025037-21idleruntime-cro21:54:240
4743211211,86sleep24797-21sed20:49:482
2186121120,7sleep20-21swapper/221:44:242
16760211218,78sleep216772-21fschecks_count19:44:462
15863211228,53sleep315882-21apt-get21:24:363
15863211228,53sleep315882-21apt-get21:24:363
9164211123,67sleep39181-21latency_hist21:04:263
11233211118,77sleep211238-21modprobe22:54:242
3457211018,76sleep13469-21cron22:29:241
2091021086,57sleep00-21swapper/019:57:230
7341210735,59sleep07405-21processes22:39:530
3013621076,59sleep00-21swapper/020:26:010
27679210715,17sleep30-21swapper/320:19:463
7660210618,70sleep37786-21wc00:24:263
14893210620,71sleep314966-21ls21:19:563
3259221055,55sleep00-21swapper/022:16:480
2906121050,6sleep10-21swapper/120:24:281
424421048,74sleep00-21swapper/019:04:560
2964521049,81sleep129708-21sendmail_mailst23:49:561
2964521049,81sleep129708-21sendmail_mailst23:49:561
19482210410,79sleep019548-21modprobe19:54:250
1811821045,12sleep00-21swapper/023:14:440
13251210454,32sleep013285-21proc_pri21:14:530
828621035,55sleep00-21swapper/019:18:200
271721038,18sleep00-21swapper/000:09:260
2151921036,53sleep00-21swapper/021:40:280
507021029,52sleep03360-21cyclictest00:14:470
19935210217,67sleep219948-21/usr/sbin/munin21:35:132
19935210217,67sleep219948-21/usr/sbin/munin21:35:132
1118621025,58sleep00-21swapper/019:25:510
2329721015,61sleep00-21swapper/021:48:190
2329721015,61sleep00-21swapper/021:48:190
1529621015,54sleep00-21swapper/021:20:340
12021210124,60sleep112053-21ps22:54:551
7206210014,71sleep37275-21/usr/sbin/munin20:55:553
29710210019,61sleep229781-21grep23:49:582
29710210019,61sleep229781-21grep23:49:582
2145121009,52sleep021545-21perl19:59:500
633529817,60sleep26279-21apt-get20:55:132
540729810,64sleep30-21swapper/322:34:433
147152985,20sleep00-21swapper/019:37:540
71232977,17sleep00-21swapper/000:19:590
3038829719,38sleep00-21swapper/022:09:540
294352965,63sleep00-21swapper/022:06:310
2620229518,11sleep00-21swapper/023:39:460
2967829413,63sleep329749-21sensors_temp20:24:553
162592948,11sleep10-21swapper/121:24:511
162592948,11sleep10-21swapper/121:24:511
1075229420,48sleep310877-21wc21:09:273
2731329319,58sleep127319-21ntp_states21:59:541
2731329319,58sleep127319-21ntp_states21:59:541
500929241,28sleep1284-21dbus-daemon22:34:251
99722917,45sleep00-21swapper/022:49:370
35529122,23sleep30-21swapper/320:34:563
35529122,23sleep30-21swapper/320:34:563
94492908,43sleep00-21swapper/000:29:400
2769529019,55sleep127726-21fschecks_count23:44:481
2769529019,55sleep127726-21fschecks_count23:44:481
2338129021,50sleep123395-21idleruntime-cro21:49:251
2338129021,50sleep123395-21idleruntime-cro21:49:251
2250729018,60sleep00-21swapper/023:27:480
2144628912,55sleep321337-21/usr/sbin/munin19:59:493
2929028821,50sleep229354-21/usr/sbin/munin20:24:462
2575428814,51sleep225755-21ntp_states21:54:552
2575428814,51sleep225755-21ntp_states21:54:552
2362828818,21sleep30-21swapper/323:29:573
1471028812,63sleep214757-21ntp_kernel_pll_21:19:512
118628717,48sleep21254-21/usr/sbin/munin22:19:572
483228517,53sleep14877-21sh19:09:251
3115328512,54sleep30-21swapper/323:55:193
244528517,31sleep20-21swapper/218:59:542
3130228423,31sleep10-21swapper/122:14:271
2730128422,40sleep127315-21apt20:19:241
2426728412,58sleep124379-21latency_hist23:34:271
2110828411,56sleep121179-21sensors21:39:561
1519828413,55sleep215269-21sensors23:04:572
320428319,38sleep23195-21apt-get00:09:402
845528220,26sleep30-21swapper/319:19:283
2440428223,36sleep224547-21sed21:49:582
2440428223,36sleep224547-21sed21:49:582
3266428121,46sleep232701-21sed20:34:512
3266428121,46sleep232701-21sed20:34:512
3236528118,44sleep132471-21fschecks_time20:34:461
3236528118,44sleep132471-21fschecks_time20:34:461
2807428125,34sleep20-21swapper/223:44:572
2807428125,34sleep20-21swapper/223:44:572
626028015,47sleep15758-21apt-get20:54:361
2972728021,33sleep10-21swapper/122:09:271
253227930,26sleep00-21swapper/018:59:560
2496827916,48sleep324981-21/usr/sbin/munin23:34:533
1193127917,44sleep212035-21sed21:10:112
921027823,27sleep30-21swapper/300:29:273
3078927818,44sleep130897-21perl20:29:451
3078927818,44sleep130897-21perl20:29:451
2133627814,48sleep221337-21/usr/sbin/munin19:59:392
3190527712,41sleep30-21swapper/322:14:493
280627718,36sleep12963-21gpgv00:09:271
2644527715,16sleep20-21swapper/220:14:532
1431127713,16sleep30-21swapper/319:35:433
2660627623,36sleep126621-21latency_hist21:59:261
2660627623,36sleep126621-21latency_hist21:59:261
1395927611,46sleep30-21swapper/322:59:583
324327526,31sleep13279-21munin-run19:04:251
3126527510,46sleep331351-21switchtime20:29:553
3126527510,46sleep331351-21switchtime20:29:553
3070127537,18sleep20-21swapper/222:10:002
2943127514,46sleep329520-21load23:49:503
2943127514,46sleep329520-21load23:49:503
2703527514,46sleep127036-21sshd23:40:521
2284127522,26sleep30-21swapper/321:44:593
2284127522,26sleep30-21swapper/321:44:593
1869827518,34sleep218732-21awk21:34:242
1869827518,34sleep218732-21awk21:34:242
492027421,34sleep34970-21munin-node19:09:273
2811327422,26sleep30-21swapper/322:04:243
2811327422,26sleep30-21swapper/322:04:243
2664127416,26sleep30-21swapper/320:14:583
692627314,41sleep26995-21/usr/sbin/munin00:19:572
487927314,44sleep24851-21latency_hist19:09:262
375027314,42sleep23813-21cpuspeed22:29:422
2093827318,41sleep320961-21netstat21:39:503
1689027315,25sleep10-21swapper/123:09:571
1689027315,25sleep10-21swapper/123:09:571
1422127323,28sleep20-21swapper/219:35:422
1067527325,32sleep110698-21munin-run21:09:251
663727214,44sleep26779-21latency_hist22:39:262
206442727,12sleep20-21swapper/223:21:522
66742718,17sleep30-21swapper/319:12:543
247292710,8sleep10-21swapper/123:34:461
207627118,23sleep30-21swapper/320:39:563
1580627115,24sleep20-21swapper/219:39:582
908527011,43sleep29217-21timerwakeupswit22:44:552
2428627014,24sleep10-21swapper/121:49:571
2428627014,24sleep10-21swapper/121:49:571
1070427023,33sleep2284-21dbus-daemon21:09:262
1043627026,24sleep30-21swapper/322:49:533
45602690,9sleep30-21swapper/320:49:453
1859826919,31sleep118643-21grep19:49:501
1591826919,32sleep115919-21cron23:09:241
2111226814,32sleep30-21swapper/323:24:283
96942679,12sleep10-21swapper/119:21:361
3040226713,33sleep1284-21dbus-daemon20:29:251
2383026713,32sleep323882-21/usr/sbin/munin21:49:473
2383026713,32sleep323882-21/usr/sbin/munin21:49:473
2268326723,24sleep10-21swapper/123:29:261
157242670,7sleep10-21swapper/119:39:561
1136926714,26sleep30-21swapper/321:09:593
2007026615,25sleep30-21swapper/319:54:483
1080326622,22sleep20-21swapper/200:34:272
592426519,20sleep10-21swapper/120:54:271
235326528,20sleep30-21swapper/318:59:513
1637626518,27sleep20-21swapper/221:24:562
1637626518,27sleep20-21swapper/221:24:562
1231726517,34sleep112401-21cat19:30:001
1231726517,34sleep112401-21cat19:30:001
973726421,16sleep10-21swapper/122:49:261
2485326417,23sleep10-21swapper/120:09:521
2485326417,23sleep10-21swapper/120:09:521
1998226414,31sleep120040-21/usr/sbin/munin19:54:461
1756026423,23sleep20-21swapper/223:14:262
1756026423,23sleep20-21swapper/223:14:262
1321226425,22sleep20-21swapper/222:59:452
1357726312,24sleep10-21swapper/119:34:441
854526217,29sleep27874-21/usr/sbin/munin00:24:582
2885226221,19sleep20-21swapper/220:24:252
1939326213,37sleep319490-21latency_hist23:19:273
878026110,26sleep28866-21perl19:19:482
267726121,25sleep10-21swapper/120:44:261
2459026121,21sleep20-21swapper/223:34:442
3165826016,25sleep131705-21users23:55:301
1149226019,28sleep111537-21latency21:10:011
846625918,19sleep10-21swapper/100:24:571
675425910,13sleep10-21swapper/100:19:521
289725912,19sleep30-21swapper/320:44:413
224942596,17sleep10-21swapper/120:02:491
64225811,30sleep10-21swapper/122:19:441
607925823,19sleep10-21swapper/119:09:581
393625817,23sleep14066-21sed00:10:071
3239425817,26sleep332399-21entropy23:59:433
3108925820,21sleep20-21swapper/220:29:502
3108925820,21sleep20-21swapper/220:29:502
1878825824,17sleep10-21swapper/123:15:131
1077925822,17sleep10-21swapper/100:34:261
755825718,18sleep30-21swapper/322:39:563
727525717,18sleep10-21swapper/122:39:501
686925719,22sleep26926-21sed19:14:272
2339125717,21sleep10-21swapper/120:04:551
2151725725,19sleep30-21swapper/323:25:003
1769425716,26sleep117728-21/usr/sbin/munin21:29:591
1769425716,26sleep117728-21/usr/sbin/munin21:29:591
1765425716,24sleep317739-21apt-config23:14:273
1765425716,24sleep317739-21apt-config23:14:273
1410125713,29sleep214204-21cat21:19:272
1167225713,19sleep30-21swapper/322:54:453
1152525714,26sleep211544-21apt-get22:54:362
852625617,16sleep10-21swapper/120:59:561
3243125617,17sleep10-21swapper/123:59:461
708625515,26sleep27105-21apt-get19:14:392
611125520,17sleep30-21swapper/300:19:263
605925511,28sleep16098-21modprobe00:19:241
265625513,20sleep30-21swapper/322:24:543
2317825515,19sleep30-21swapper/320:04:493
1793325517,20sleep10-21swapper/119:49:251
1667725513,25sleep316744-21df19:44:433
1287225518,21sleep30-21swapper/321:14:443
840025413,24sleep38419-21apt-get22:44:363
2837825414,24sleep328397-21apt-get22:04:363
2731625415,21sleep227346-21munin-run23:44:242
273025416,16sleep20-21swapper/220:44:272
2489025413,25sleep224909-21ntp_states20:09:532
2489025413,25sleep224909-21ntp_states20:09:532
1822125413,24sleep318216-21apt-get19:49:393
17512548,15sleep30-21swapper/322:22:363
961925313,22sleep30-21swapper/300:29:483
738025319,16sleep10-21swapper/119:14:501
3199225311,26sleep231998-21ntp_states22:14:522
2541825316,23sleep325444-21fschecks_count21:54:483
2541825316,23sleep325444-21fschecks_count21:54:483
181925313,24sleep21825-21ntp_states20:39:512
1146725314,19sleep30-21swapper/319:29:253
2858825217,18sleep20-21swapper/222:04:472
2363125215,22sleep20-21swapper/221:49:342
2090125215,18sleep10-21swapper/123:24:261
1902025210,26sleep319015-21apt-get21:34:423
1902025210,26sleep319015-21apt-get21:34:423
178425214,19sleep30-21swapper/300:04:523
1519925210,17sleep30-21swapper/319:39:433
1004925211,17sleep10-21swapper/122:49:431
865825112,24sleep38655-21apt-get19:19:403
2332425110,23sleep223360-21ntp_states20:04:522
1443225113,17sleep10-21swapper/121:19:461
2353125017,18sleep20-21swapper/223:29:562
79472499,16sleep20-21swapper/220:59:452
472624920,17sleep30-21swapper/322:30:523
376024916,19sleep30-21swapper/300:10:053
3170624912,21sleep230487-40mandb23:55:312
2890724912,21sleep328916-21/usr/sbin/munin23:49:263
2890724912,21sleep328916-21/usr/sbin/munin23:49:263
2697624912,21sleep227038-21perl21:59:472
2697624912,21sleep227038-21perl21:59:472
227424918,18sleep20-21swapper/222:24:452
2242024911,22sleep222487-21memory21:44:502
1842624912,19sleep20-21swapper/223:15:062
3230924810,22sleep232305-21apt-get23:59:382
3044424714,14sleep30-21swapper/322:09:573
147432459,15sleep30-21swapper/323:04:463
662724310,17sleep10-21swapper/122:39:251
315382438,18sleep10-21swapper/122:14:391
150224214,14sleep20-21swapper/200:04:472
528324012,15sleep20-21swapper/200:14:532
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional