You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-06-02 - 10:49

x86 Intel Celeron N3350 @1100 MHz, Linux 5.10.90-rt60 (Profile)

Latency plot of system in rack #c, slot #4
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot4.osadl.org (updated Fri Jun 02, 2023 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23085512310189,27sleep00-21swapper/019:07:530
23085202272195,27sleep10-21swapper/119:07:341
1591991060,4rtkit-daemon0-21swapper/123:50:401
159199990,3rtkit-daemon0-21swapper/121:12:251
159199970,3rtkit-daemon0-21swapper/119:51:201
159199970,3rtkit-daemon0-21swapper/119:51:191
159199960,3rtkit-daemon0-21swapper/123:44:551
159199960,3rtkit-daemon0-21swapper/123:05:271
159199940,3rtkit-daemon0-21swapper/120:29:501
159199890,3rtkit-daemon1590-21rtkit-daemon19:12:521
159199840,3rtkit-daemon1590-21rtkit-daemon19:20:101
159199790,4rtkit-daemon1590-21rtkit-daemon00:31:160
159199770,4rtkit-daemon1590-21rtkit-daemon00:25:570
159199740,4rtkit-daemon1590-21rtkit-daemon20:55:290
159199740,3rtkit-daemon1590-21rtkit-daemon19:55:170
159199720,4rtkit-daemon1590-21rtkit-daemon20:06:491
24003102700,4sleep0161rcuc/022:50:240
230884499700,4cyclictest2387815-21head22:20:280
159199700,4rtkit-daemon1590-21rtkit-daemon19:19:191
159199700,3rtkit-daemon1590-21rtkit-daemon19:37:391
159199700,3rtkit-daemon0-21swapper/021:03:521
2308844996918,30cyclictest2343102-21python320:35:010
159199690,16rtkit-daemon0-21swapper/020:58:561
159199680,2rtkit-daemon1590-21rtkit-daemon20:40:441
230884499670,2cyclictest2393172-21chrt22:33:430
230884499670,2cyclictest2324023-21ntpq19:45:280
159199670,4rtkit-daemon1590-21rtkit-daemon20:24:361
159199670,16rtkit-daemon0-21swapper/000:32:261
230884499660,2cyclictest2433324-21taskset00:14:370
230884499660,2cyclictest2412391-21sed23:20:290
230884499660,2cyclictest2402201-21grep22:55:250
159199660,4rtkit-daemon1590-21rtkit-daemon22:36:211
159199650,16rtkit-daemon0-21swapper/000:23:221
23619372640,4sleep10-21swapper/121:15:351
230884499640,9cyclictest2390883-21python322:30:000
230884499640,4cyclictest2429598-21chrt00:05:050
230884499640,2cyclictest2365056-21systemd-run21:25:010
159199640,3rtkit-daemon0-21swapper/120:37:021
230884499631,5cyclictest2333830-21cut20:10:240
2308844996314,39cyclictest2427430-21perf00:00:000
159199630,3rtkit-daemon0-21swapper/120:24:420
159199630,3rtkit-daemon0-21swapper/100:02:220
230884499626,5cyclictest2379836-21uname22:00:280
230884499620,6cyclictest2434892-21who00:15:380
230884499620,6cyclictest2434892-21who00:15:370
159199610,16rtkit-daemon0-21swapper/021:43:091
230884499601,2cyclictest0-21swapper/023:25:210
2308844996012,23cyclictest2420061-21head23:40:290
230884499592,5cyclictest2360942-21grep21:15:200
230884499590,7cyclictest2356584-21apt-get21:05:200
230884499585,5cyclictest2378107-21gzip21:55:300
230884499581,2cyclictest0-21swapper/022:06:500
2308844995812,24cyclictest2336214-21ntp_kernel_pll_20:15:270
159199580,3rtkit-daemon0-21swapper/123:10:280
159199580,3rtkit-daemon0-21swapper/123:05:200
230884499572,4cyclictest2425466-21cat23:55:000
2308844995720,16cyclictest2419487-21awk23:39:590
230884499571,6cyclictest2396003-21ls22:40:210
2308844995714,18cyclictest2333352-21date20:10:010
2308844995712,2cyclictest0-21swapper/022:48:500
230884499570,2cyclictest2375018-21systemd-run21:50:000
230884499570,2cyclictest2326965-21munin-run19:55:010
230884499570,2cyclictest2326965-21munin-run19:55:000
230884499560,1cyclictest2310839-21latency_hist19:15:000
159199560,4rtkit-daemon1590-21rtkit-daemon23:55:291
159199560,48rtkit-daemon3437640-21snmpd20:50:051
230884499559,5cyclictest2331406-21sed20:05:000
230884499559,2cyclictest2313650-21cat19:20:200
230884499558,6cyclictest2358548-21grep21:10:250
230884499556,2cyclictest2387003-21date22:20:010
230884499553,2cyclictest2349641-21taskset20:49:130
230884499552,4cyclictest2339541-21expr20:25:210
230884499550,3cyclictest2316813-21taskset19:26:040
159199550,3rtkit-daemon1590-21rtkit-daemon00:15:201
159199550,3rtkit-daemon1590-21rtkit-daemon00:15:191
2308849995419,9cyclictest2422445-21ntp_offset23:45:311
2308849995419,9cyclictest2422445-21ntp_offset23:45:301
230884499545,2cyclictest23209182sleep019:38:180
230884499543,6cyclictest2393602-21cat22:35:200
230884499543,5cyclictest2346386-21cut20:40:290
230884499541,6cyclictest2311692-21ls19:15:160
2308844995411,2cyclictest2345278-21taskset20:38:260
230884499540,3cyclictest0-21swapper/000:40:000
230884499538,5cyclictest2365883-21cat21:25:260
230884499536,2cyclictest2371824-21cat21:40:290
230884499533,3cyclictest2368708-21taskset21:32:170
230884499533,3cyclictest2368708-21taskset21:32:170
230884499530,9cyclictest2437465-21taskset00:21:260
230884499530,4cyclictest2354635-21apt-get21:00:210
230884499530,2cyclictest2323102-21cat19:45:010
159199530,3rtkit-daemon1590-21rtkit-daemon22:13:401
2308849995223,9cyclictest2389498-21gltestperf22:25:251
2308849995218,10cyclictest2369595-21gltestperf21:35:241
2308849995217,13cyclictest2428011-21gltestperf00:00:221
230884499527,2cyclictest2375974-21ntpq21:50:300
230884499526,2cyclictest2405558-21seq23:04:430
2308844995226,9cyclictest2350771-21head20:50:270
2308844995213,13cyclictest2423458-21munin-run23:50:010
2308844995213,13cyclictest2423458-21munin-run23:50:000
2308844995211,3cyclictest2409860-21cat23:15:230
230884499520,7cyclictest2383188-21grep22:10:210
159199520,4rtkit-daemon1590-21rtkit-daemon23:18:391
2308849995128,11cyclictest2334361-21pmu-power20:10:321
2308849995125,9cyclictest2376345-21timerwakeupswit21:50:341
2308849995125,8cyclictest2342573-21unixbench_multi20:30:391
2308849995123,12cyclictest2321437-21cpuspeed_turbos19:40:221
230884499517,2cyclictest2318428-21switchtime19:30:340
230884499515,2cyclictest2416245-21awk23:30:280
230884499510,2cyclictest2370577-21taskset21:36:120
2308849995026,7cyclictest2328643-21mailstats19:55:391
2308849995025,8cyclictest2430143-21if_err_enp2s000:05:341
2308849995025,7cyclictest2396751-21pmu-power22:40:321
2308849995023,9cyclictest2324488-21unixbench_multi19:45:341
2308849995021,12cyclictest2387783-21needreboot22:20:281
2308849995016,9cyclictest2348927-21rt-features20:45:331
159199500,3rtkit-daemon0-21swapper/000:38:201
2308849994927,5cyclictest2336443-21sendmail_mailtr20:15:301
2308849994925,8cyclictest2418956-21unixbench_singl23:35:351
2308849994925,8cyclictest2368180-21mailstats21:30:311
2308849994925,8cyclictest2368180-21mailstats21:30:301
2308849994925,3cyclictest2317810-21tail19:30:251
2308849994924,14cyclictest2366274-21switchtime21:25:321
159199490,4rtkit-daemon1590-21rtkit-daemon22:18:351
159199490,3rtkit-daemon0-21swapper/123:00:441
159199490,3rtkit-daemon0-21swapper/122:33:231
2308849994826,6cyclictest2363570-21fschecks_time21:20:231
2308849994824,6cyclictest2316516-21switchtime19:25:301
2308849994821,10cyclictest2413776-21cpu23:25:201
159199480,3rtkit-daemon1590-21rtkit-daemon23:11:381
159199480,3rtkit-daemon0-21swapper/100:11:331
2308849994725,10cyclictest2438916-21proc_pri00:25:301
2308849994723,9cyclictest2398385-21if_err_enp2s022:45:271
2308849994722,8cyclictest2330302-21needreboot20:00:341
2308849994720,9cyclictest2378181-21mailstats21:55:311
2308849994716,9cyclictest2357685-21timerandwakeup21:05:451
159199470,3rtkit-daemon1590-21rtkit-daemon23:30:151
2308849994621,7cyclictest2413029-21unixbench_singl23:20:361
2308849994618,15cyclictest2379186-21apt-get22:00:191
2308849994616,9cyclictest2374112-21rt-features21:45:321
2308849994613,10cyclictest2400392-21kernelversion22:50:251
159199460,3rtkit-daemon1590-21rtkit-daemon22:06:231
2308849994514,9cyclictest2402294-21idleruntime22:55:251
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional