You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-25 - 12:12
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot6 (updated Mon Sep 25, 2023 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13192210219,38sleep00-21swapper/019:07:200
116382920,8sleep014-21ksoftirqd/020:30:370
13628998567,11cyclictest26-21ksoftirqd/121:50:221
13628998553,16cyclictest16898-21apt-get23:50:191
13628998451,16cyclictest26467-21apt-get19:45:221
13628998450,23cyclictest14550irq/26-ahci[00021:00:191
13628998431,21cyclictest23272-21chrt19:35:301
124002826,29sleep112445-21cat19:05:261
13628998156,11cyclictest26-21ksoftirqd/120:10:201
13628998047,20cyclictest26-21ksoftirqd/123:30:181
13628997851,14cyclictest6426-21apt-get23:20:211
13628997762,9cyclictest26-21ksoftirqd/120:00:221
13628997759,9cyclictest26-21ksoftirqd/119:50:211
13628997759,9cyclictest26-21ksoftirqd/119:50:201
13628997752,12cyclictest25256-21apt-get21:10:201
13628997648,18cyclictest26-21ksoftirqd/121:35:211
13628997640,32cyclictest26158-21awk22:45:001
13628997550,15cyclictest26-21ksoftirqd/121:15:211
13628997549,20cyclictest1157-21runrttasks23:05:231
13628997549,20cyclictest1157-21runrttasks23:05:221
13628997549,12cyclictest28748-21apt-get21:20:221
13628997549,12cyclictest28748-21apt-get21:20:211
13628997454,11cyclictest26-21ksoftirqd/100:05:211
13628997447,17cyclictest26-21ksoftirqd/119:55:181
13628997446,19cyclictest26-21ksoftirqd/123:40:191
13628997445,18cyclictest26-21ksoftirqd/119:25:221
13628997435,34cyclictest10446-21cron20:30:011
13628997430,37cyclictest13231-21latency_hist23:40:021
13628997341,24cyclictest26-21ksoftirqd/120:30:221
13628997313,53cyclictest0-21swapper/122:25:201
13623997332,29cyclictest14-21ksoftirqd/020:50:220
13628997243,22cyclictest14145-21if_err_enp2s019:10:281
13628997243,17cyclictest0-21swapper/100:35:231
13628997242,18cyclictest0-21swapper/100:10:341
13628997225,43cyclictest8431-21awk20:25:011
13628997141,19cyclictest0-21swapper/122:00:201
13628997138,24cyclictest1943-21strings20:05:361
13628997044,15cyclictest26-21ksoftirqd/122:50:221
13628997040,25cyclictest27259-21sendmail-msp00:20:001
13628997038,27cyclictest4623-21timerwakeupswit21:40:451
13628997031,33cyclictest27876-21munin-run22:49:591
13628997017,20cyclictest0-21swapper/122:35:401
13628996943,15cyclictest26-21ksoftirqd/123:25:221
13628996940,16cyclictest0-21swapper/120:50:231
13628996938,25cyclictest22707-21ls22:35:021
13628996938,18cyclictest0-21swapper/123:15:221
13628996936,10cyclictest31517-21apt-config23:00:031
13623996937,18cyclictest14-21ksoftirqd/019:20:220
13628996842,18cyclictest26-21ksoftirqd/121:45:181
13628996840,22cyclictest18894-21cpuspeed_turbos23:55:261
13628996840,22cyclictest18894-21cpuspeed_turbos23:55:261
13628996839,15cyclictest0-21swapper/100:20:201
13628996838,19cyclictest0-21swapper/100:25:221
13628996837,22cyclictest14435-21df_inode22:10:281
13628996837,19cyclictest0-21swapper/119:30:201
13628996828,19cyclictest31870-21unixbench_singl21:25:451
13628996825,38cyclictest14279-21sh20:44:551
201252670,8chrt0-21swapper/120:55:311
13628996741,16cyclictest0-21swapper/121:05:231
13628996740,17cyclictest13780-21unixbench_singl20:35:441
13628996738,25cyclictest6369-21idleruntime-cro20:20:001
13628996738,25cyclictest16033-21telnet19:15:231
13628996738,24cyclictest31893-21basename23:00:221
13628996737,23cyclictest13245-21timerandwakeup22:05:441
13628996736,17cyclictest0-21swapper/122:20:211
1362399675,12cyclictest28757-21memory19:50:350
1362399675,12cyclictest28757-21memory19:50:340
13623996728,29cyclictest14-21ksoftirqd/021:00:210
13628996636,22cyclictest255-21systemd-journal00:31:251
13628996636,21cyclictest9836-21grep21:55:451
13628996635,21cyclictest16393-21cat22:15:361
13628996634,22cyclictest15951-21proc_pri23:45:411
13628996634,22cyclictest15951-21proc_pri23:45:411
13628996538,22cyclictest3908-21timerwakeupswit23:10:431
13628996535,22cyclictest19283-21apt19:25:011
13628996535,19cyclictest16210-21apt-get20:45:221
1362399657,49cyclictest0-21swapper/021:25:210
13628996440,19cyclictest467-21dbus-daemon21:34:591
13628996434,20cyclictest24515-21cron19:40:021
13628996433,23cyclictest20424-21apt-get00:00:211
13623996223,25cyclictest12087-21irqstats23:35:300
1362399614,52cyclictest0-21swapper/000:13:460
1362399613,19cyclictest14440-21runrttasks19:10:340
1362399595,49cyclictest0-21swapper/023:25:440
1362399595,48cyclictest0-21swapper/022:54:190
1362399585,47cyclictest0-21swapper/022:50:010
1362399579,43cyclictest0-21swapper/000:18:010
1362399578,10cyclictest0-21swapper/021:15:370
1362399576,12cyclictest14-21ksoftirqd/022:43:260
13623995741,10cyclictest14-21ksoftirqd/020:05:310
1362399572,42cyclictest25259-21runrttasks21:10:210
1362399572,12cyclictest30392-21aten2.4-expect19:55:220
1362399565,22cyclictest14-21ksoftirqd/000:34:210
1362399564,47cyclictest0-21swapper/021:40:380
1362399564,46cyclictest0-21swapper/000:35:350
1362399563,47cyclictest22155-21apt-get00:05:190
13623995615,36cyclictest14-21ksoftirqd/021:55:410
13623995615,23cyclictest19659-21sed23:55:440
13623995615,23cyclictest19659-21sed23:55:430
1362399557,42cyclictest0-21swapper/019:15:370
1362399556,43cyclictest0-21swapper/021:30:040
1362399555,45cyclictest0-21swapper/022:55:390
1362399555,44cyclictest0-21swapper/023:50:380
1362399555,10cyclictest0-21swapper/020:15:380
1362399554,45cyclictest10736-21python323:30:420
1362399554,10cyclictest0-21swapper/021:35:260
13623995538,12cyclictest12668-21apt-get20:35:230
13623995515,11cyclictest171rcuc/023:05:390
13623995515,11cyclictest171rcuc/023:05:390
13623995514,34cyclictest0-21swapper/019:41:290
1362399548,41cyclictest0-21swapper/020:25:190
1362399548,12cyclictest14-21ksoftirqd/022:35:090
1362399545,44cyclictest0-21swapper/022:30:360
1362399545,44cyclictest0-21swapper/019:45:300
1362399545,44cyclictest0-21swapper/019:30:310
13623995420,16cyclictest18311-21cat22:20:390
13623995418,13cyclictest151rcu_preempt20:20:250
13623995414,7cyclictest151rcu_preempt23:20:250
13623995414,26cyclictest20892-21chrt20:56:300
162552530,7chrt16252-21aten2.4_r3power20:45:230
1362399538,14cyclictest14-21ksoftirqd/022:25:170
1362399538,13cyclictest14-21ksoftirqd/023:17:250
1362399537,9cyclictest0-21swapper/022:16:160
1362399537,13cyclictest14-21ksoftirqd/021:51:390
1362399535,21cyclictest14-21ksoftirqd/020:13:050
13623995329,18cyclictest14-21ksoftirqd/023:40:440
13623995327,13cyclictest14-21ksoftirqd/000:02:560
1362399532,44cyclictest9292sleep023:04:410
13623995314,15cyclictest3032-21aten2.4-expect23:10:220
1362399529,14cyclictest14-21ksoftirqd/021:07:000
1362399526,12cyclictest14-21ksoftirqd/019:35:510
13623995236,9cyclictest14-21ksoftirqd/021:46:200
1362399522,15cyclictest15337-21grep23:45:260
1362399522,15cyclictest15337-21grep23:45:260
13623995211,16cyclictest151rcu_preempt00:23:150
1362399515,10cyclictest0-21swapper/022:10:270
13623995136,9cyclictest14-21ksoftirqd/022:00:420
13623995113,14cyclictest30063-21cat00:25:460
1362399503,41cyclictest14476-21/usr/sbin/munin20:40:410
13623995018,13cyclictest14-21ksoftirqd/021:20:440
13623995018,13cyclictest14-21ksoftirqd/021:20:440
13623995014,17cyclictest12452-21aten2.4_r3power22:05:250
13623995013,15cyclictest151rcu_preempt19:26:110
13623995012,14cyclictest151rcu_preempt20:00:240
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:263
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
none15:32:262
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional