You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-20 - 12:25
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot4.osadl.org (updated Sat Apr 20, 2024 00:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22093342325283,27sleep00-21swapper/019:09:220
22093202324281,28sleep10-21swapper/119:09:131
220951699225219,4cyclictest2243295-21kworker/u8:220:40:240
159199550,8rtkit-daemon2229051-21ntp_states19:55:310
159199550,3rtkit-daemon1590-21rtkit-daemon22:58:230
23986962500,4sleep1251rcuc/100:30:211
159199500,4rtkit-daemon1590-21rtkit-daemon00:08:491
22263082490,1sleep12226306-21tail19:50:221
159199490,3rtkit-daemon1590-21rtkit-daemon20:49:241
22750882480,3sleep10-21swapper/121:35:161
220952099473,11cyclictest2318008-21perf22:40:001
220952099453,13cyclictest2288132-40apt-check21:55:111
220952099444,8cyclictest1-21systemd22:50:001
220952099432,11cyclictest2277751-21cron21:40:001
159199430,4rtkit-daemon0-21swapper/020:38:420
159199430,3rtkit-daemon1590-21rtkit-daemon22:31:020
220952099422,15cyclictest2268161-21gpg-connect-age21:25:011
220952099422,11cyclictest2390768-21latency_hist00:20:001
220952099421,11cyclictest2285586-21mailstats21:50:251
220952099403,12cyclictest2219839-21ls19:35:001
220952099402,12cyclictest2319369-21grep22:40:261
220952099401,11cyclictest2282661-21unixbench-2d21:45:301
220952099397,13cyclictest2242769-21cut20:30:231
220952099392,10cyclictest2331256-21date23:00:011
220952099391,11cyclictest2215923-21sh19:25:001
220952099390,11cyclictest2314639-21sh22:35:001
220952099384,8cyclictest2243902-21apt-get20:35:121
220952099384,8cyclictest2228275-21apt-get19:55:211
220952099383,9cyclictest2381640-21date00:05:011
220952099383,9cyclictest2378242-40mandb00:00:011
220952099382,13cyclictest2361711-21ntpq23:45:211
220952099380,9cyclictest2387520-21latency_hist00:15:001
220952099380,9cyclictest2387520-21latency_hist00:15:001
159199380,9rtkit-daemon2337311-21ssh23:09:110
220952099374,12cyclictest2307869-21python322:25:021
220952099373,13cyclictest2345586-21/usr/sbin/munin23:20:301
220952099372,11cyclictest2341898-21ntpq23:15:251
220952099372,10cyclictest2350949-21date23:30:011
220952099371,10cyclictest2264280-21latency_hist21:20:011
2209516993723,6cyclictest2268288-21apt-get21:25:120
220952099361,7cyclictest2397268-21turbostat00:30:001
220952099361,12cyclictest2235421-21who20:10:241
159199360,3rtkit-daemon0-21swapper/023:11:120
220952099356,10cyclictest2402339-21unixbench-2d00:35:311
220952099354,9cyclictest2307393-21sendmail-mta22:23:521
220952099352,9cyclictest2217880-21systemd-run19:30:011
220952099352,11cyclictest2325621-21cut22:50:231
220952099352,11cyclictest2241888-21date20:30:001
220952099352,11cyclictest2237007-21gzip20:15:231
220952099352,10cyclictest2210830-21sed19:10:281
220952099351,9cyclictest2274481-21sh21:35:001
220952099351,9cyclictest2221881-21date19:40:001
220952099351,13cyclictest2338660-21/usr/sbin/munin23:10:201
220952099351,11cyclictest2391718-21ntpq00:20:201
159199350,3rtkit-daemon0-21swapper/022:12:010
220952099345,8cyclictest2303608-21ssh22:17:441
220952099344,12cyclictest2357605-21ssh23:40:091
220952099343,11cyclictest2254158-21basename21:00:171
220952099342,15cyclictest2259909-21rm21:12:341
220952099341,13cyclictest2294859-21apt-get22:05:111
220952099341,12cyclictest2256237-21grep21:05:211
159199340,3rtkit-daemon0-21swapper/120:23:471
159199340,3rtkit-daemon0-21swapper/022:45:010
220952099335,7cyclictest2249791-21taskset20:50:111
220952099332,13cyclictest2212423-21fschecks_count19:15:181
220952099332,12cyclictest2222781-21grep19:40:251
220952099332,11cyclictest2224730-21grep19:45:241
220952099331,9cyclictest2332400-21mailstats23:00:241
220952099331,12cyclictest2230998-21sh20:00:211
220952099331,11cyclictest2232347-21cstates20:05:271
159199330,4rtkit-daemon1590-21rtkit-daemon23:06:581
159199330,4rtkit-daemon1590-21rtkit-daemon22:36:000
159199330,3rtkit-daemon1590-21rtkit-daemon21:50:190
159199330,3rtkit-daemon0-21swapper/123:50:081
159199330,3rtkit-daemon0-21swapper/122:11:191
220952099323,27cyclictest22930492sleep122:00:371
220952099322,14cyclictest2352468-21ssh23:30:391
220952099322,11cyclictest2356730-21taskset23:38:321
220951699320,10cyclictest22263062tail19:50:220
159199320,6rtkit-daemon1590-21rtkit-daemon00:10:480
159199320,6rtkit-daemon1590-21rtkit-daemon00:10:480
159199320,5rtkit-daemon2365634-21mkinitramfs23:48:080
159199320,3rtkit-daemon1590-21rtkit-daemon21:24:121
159199320,3rtkit-daemon1590-21rtkit-daemon19:40:250
159199320,3rtkit-daemon0-21swapper/100:28:050
220952099312,11cyclictest2245880-21grep20:40:111
220952099301,11cyclictest2251748-21cpu20:55:121
2209516993013,11cyclictest2318251-21apt-get22:40:140
159199300,3rtkit-daemon0-21swapper/122:16:360
159199300,3rtkit-daemon0-21swapper/121:10:170
220951699290,7cyclictest2308452-21apt-get22:25:100
159199290,3rtkit-daemon1590-21rtkit-daemon21:48:470
159199290,3rtkit-daemon1590-21rtkit-daemon20:33:080
2209516992817,2cyclictest0-21swapper/021:20:000
159199280,3rtkit-daemon0-21swapper/121:57:470
159199280,3rtkit-daemon0-21swapper/100:01:200
220951699272,7cyclictest2378251-40systemctl00:00:000
2209516992711,10cyclictest2371972-21apt-get23:50:120
220951699270,5cyclictest0-21swapper/023:36:200
159199270,3rtkit-daemon0-21swapper/123:18:150
159199270,3rtkit-daemon0-21swapper/120:54:360
159199270,22rtkit-daemon3381468-21ntpd21:00:380
2209516992611,9cyclictest2262989-1kworker/u9:123:25:130
2209516992611,9cyclictest2257716-1kworker/u9:223:00:110
2209516992510,3cyclictest2237836-21turbostat.cron20:20:000
220951699250,6cyclictest0-21swapper/022:52:040
159199250,1rtkit-daemon1590-21rtkit-daemon23:23:240
220951699242,15cyclictest2357601-21apt-get23:40:130
2209516992410,8cyclictest2251736-21apt-get20:55:120
220951699240,6cyclictest2305062-21fschecks_time22:20:160
220951699240,6cyclictest0-21swapper/023:34:130
220951699240,6cyclictest0-21swapper/019:37:140
159199240,19rtkit-daemon1590-21rtkit-daemon19:11:030
220951699238,9cyclictest2262989-1kworker/u9:121:35:110
220951699231,11cyclictest2292924-21grep22:00:280
159199230,3rtkit-daemon1590-21rtkit-daemon00:06:210
159199230,1rtkit-daemon1590-21rtkit-daemon20:01:160
159199230,1rtkit-daemon1590-21rtkit-daemon19:21:430
159199230,15rtkit-daemon0-21swapper/000:39:490
220951699223,17cyclictest0-21swapper/020:25:200
220951699213,12cyclictest2262989-1kworker/u9:121:20:150
220951699211,14cyclictest2278032-21apt-get21:40:130
220951699211,14cyclictest2271407-21apt-get21:30:100
220951699211,14cyclictest2223927-21taskset19:45:140
220951699210,5cyclictest0-21swapper/019:30:210
220951699210,19cyclictest2397732-21apt-key00:30:020
159199210,17rtkit-daemon0-21swapper/000:17:190
159199210,16rtkit-daemon1590-21rtkit-daemon00:21:160
159199210,14rtkit-daemon0-21swapper/019:15:130
220951699202,13cyclictest2294839-21/usr/sbin/munin22:05:300
2209516992014,4cyclictest2239781-21cat20:24:550
220951699196,11cyclictest2249616-21munin-run20:49:590
220951699191,16cyclictest2217934-21cat19:30:010
2209516991811,4cyclictest2441029-21snapd20:05:130
220951699180,16cyclictest2257409-21latency_hist21:10:010
220951699180,16cyclictest2235883-21ls20:15:000
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional