You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-09-28 - 16:10

Intel(R) Celeron(R) CPU G3900 @ 2.80GHz, Linux 5.10.35-rt39 (Profile)

Latency plot of system in rack #c, slot #5
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rackcslot5.osadl.org (updated Tue Sep 28, 2021 12:43:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
133932320,1sleep013387-21grep11:10:240
1596399300,29cyclictest1288-21meminfo09:15:201
1596399290,28cyclictest9784-21meminfo11:00:191
1596399280,27cyclictest2684-21meminfo12:05:201
1596299280,27cyclictest20107-21meminfo08:40:220
1596299260,25cyclictest20776-21meminfo10:05:210
1596399250,24cyclictest23387-21meminfo11:35:191
1596299240,23cyclictest19511-21meminfo11:25:210
1596399230,22cyclictest9094-21meminfo09:35:191
1596399230,22cyclictest27866-21meminfo09:00:161
1596299230,22cyclictest31763-21meminfo09:10:270
1596399220,21cyclictest27011-21meminfo07:35:181
1596299220,22cyclictest23092-21meminfo07:25:180
1596299210,20cyclictest32438-21meminfo10:35:260
1596299210,20cyclictest12315-21meminfo08:20:170
1596399200,19cyclictest19209-21meminfo07:15:251
15962992016,2cyclictest11-21ksoftirqd/010:05:010
1596299200,19cyclictest25346-21meminfo11:40:200
1596399190,18cyclictest3276-21meminfo09:20:211
1596299190,19cyclictest4546-21meminfo08:00:210
1596299190,19cyclictest29804-21meminfo09:05:170
1596299190,18cyclictest16204-21meminfo08:30:180
1596399180,17cyclictest14343-21meminfo12:35:211
1596299180,18cyclictest11053-21meminfo09:40:190
1596299180,17cyclictest27288-21meminfo11:45:230
1596299180,17cyclictest2411-21meminfo07:55:200
1596399170,17cyclictest26611-21meminfo10:20:191
1596299172,3cyclictest131rcu_preempt08:45:130
15962991716,1cyclictest11-21ksoftirqd/011:30:010
1596399163,11cyclictest21385-21apt-get08:45:141
1596399160,16cyclictest3952-21meminfo10:45:201
1596299160,15cyclictest2019-21meminfo10:40:260
1596299160,15cyclictest10462-21meminfo12:25:200
1596399157,4cyclictest31182-21missed_timers11:55:241
1596399154,2cyclictest0-21swapper/108:25:151
1596399153,10cyclictest13444-21munin-run08:24:591
1596399150,15cyclictest8422-21meminfo08:10:211
1596399150,14cyclictest31636-21taskset07:45:501
1596299158,3cyclictest11087-21runrttasks08:15:530
1596299156,4cyclictest11-21ksoftirqd/008:15:000
1596299153,11cyclictest27899-21apt-get10:25:160
1596299152,9cyclictest121rcuc/009:54:230
1596299152,9cyclictest121rcuc/009:54:230
15962991513,1cyclictest10250-21sendmail-msp09:40:000
15962991510,4cyclictest5831-21/usr/sbin/munin08:05:230
1596299150,9cyclictest0-21swapper/010:31:270
1596299150,14cyclictest5220-21meminfo09:25:220
1596399143,8cyclictest24692-21apt-get11:40:161
1596399142,7cyclictest13017-21apt-get11:10:201
1596299141,3cyclictest131rcu_preempt11:55:300
15962991411,2cyclictest24930-21seq08:53:220
1596299140,9cyclictest30130-21runrttasks11:52:540
1596299140,5cyclictest0-21swapper/011:35:140
1596299140,13cyclictest7184-21meminfo09:30:380
1596399132,6cyclictest10457-21/usr/sbin/munin09:40:151
1596399131,10cyclictest28302-21taskset11:49:041
1596399130,13cyclictest22710-21meminfo10:10:201
1596299134,4cyclictest11-21ksoftirqd/009:00:210
1596299132,6cyclictest0-21swapper/012:35:150
1596299131,4cyclictest20502-21apt-get07:20:130
15962991311,2cyclictest2321-21runrttasks07:52:390
15962991311,2cyclictest11-21ksoftirqd/012:11:540
15962991311,1cyclictest20324-21cron07:20:000
15962991311,1cyclictest121rcuc/010:51:560
1596299130,13cyclictest25042-21meminfo07:30:200
1596299130,13cyclictest16637-21meminfo07:10:160
1596299130,12cyclictest25909-21meminfo08:55:180
1596399124,6cyclictest8415-21grep10:55:291
1596399124,5cyclictest32428-21ls10:35:251
1596399124,5cyclictest0-21swapper/107:10:111
1596399123,8cyclictest382-21memory07:50:171
1596399123,2cyclictest0-21swapper/112:10:311
1596399122,6cyclictest4864-21kworker/u4:011:54:121
1596399121,5cyclictest11752-21apt-get12:30:151
1596299128,3cyclictest0-21swapper/011:05:170
1596299124,5cyclictest2321-21runrttasks07:43:290
1596299120,11cyclictest7831-21meminfo10:55:240
1596299120,0cyclictest131rcu_preempt07:45:170
1596399118,3cyclictest21-21ksoftirqd/110:01:031
1596399116,4cyclictest28951-21chrt10:25:291
1596399114,2cyclictest29855-21ls09:05:181
1596399113,5cyclictest0-21swapper/112:25:141
1596399113,4cyclictest30926-21diskmemload10:15:151
1596399112,8cyclictest25411-21grep07:30:241
1596399111,6cyclictest18498-21runrttasks11:23:041
1596399110,11cyclictest28973-21meminfo07:40:181
1596299112,8cyclictest11-21ksoftirqd/012:01:020
1596299111,5cyclictest13608-21apt-get08:25:150
1596299111,3cyclictest0-21swapper/012:30:150
157462112,2sleep10-21swapper/107:09:401
1596399108,1cyclictest20125-21apt-get10:05:171
1596399105,3cyclictest24226-21turbostat07:30:001
1596399105,3cyclictest0-21swapper/108:00:191
1596399104,5cyclictest18858-21apt-get11:25:161
1596399104,2cyclictest0-21swapper/111:05:171
1596399102,8cyclictest7958-21taskset09:31:421
15963991010,0cyclictest0-21swapper/110:52:121
1596399100,10cyclictest21453-21meminfo11:30:221
1596399100,0cyclictest0-21swapper/107:57:031
1596299109,1cyclictest11-21ksoftirqd/009:20:220
1596299109,1cyclictest0-21swapper/010:18:550
1596299101,3cyclictest0-21swapper/011:00:140
15962991010,0cyclictest2321-21runrttasks10:11:320
1596299100,7cyclictest30926-21diskmemload09:16:350
1596299100,7cyclictest1662-21snmpd12:23:570
151172102,4sleep015265-21grep07:05:270
159639996,3cyclictest18017-21munin-run10:00:011
159639991,7cyclictest4561-21/usr/sbin/munin09:25:271
159639990,9cyclictest15612-21meminfo11:15:231
159639990,6cyclictest31484-21taskset10:33:481
159629996,2cyclictest0-21swapper/008:37:060
159629993,1cyclictest13584-21kworker/u4:111:18:470
159629992,4cyclictest30926-21diskmemload11:21:480
159629991,3cyclictest0-21swapper/012:08:120
159639986,2cyclictest18997-21seq08:37:171
159639986,1cyclictest6263-21grep12:15:161
159639986,1cyclictest2502-21cat10:40:311
159639986,1cyclictest22290-21ls07:25:011
159639986,1cyclictest12700-21cut09:45:151
159639985,2cyclictest25586-21gltestperf08:55:141
159639983,2cyclictest9500-21seq12:23:361
159639983,2cyclictest0-21swapper/108:50:141
159639982,4cyclictest0-21swapper/108:30:141
159639981,7cyclictest7535-21taskset08:09:251
159639980,8cyclictest14946-21meminfo09:50:271
159639980,8cyclictest14946-21meminfo09:50:271
159639980,3cyclictest1865-21grep12:05:001
159629986,1cyclictest6215-21tail12:15:150
159629986,1cyclictest27362-21sendmail_mailtr07:35:220
159629984,2cyclictest11-21ksoftirqd/010:45:150
159629983,2cyclictest0-21swapper/009:55:170
159629982,1cyclictest0-21swapper/010:20:160
159629981,5cyclictest0-21swapper/009:47:070
159639976,1cyclictest32261-21cat09:10:311
159639976,1cyclictest19460-21apt-esm-hook08:40:171
159639971,3cyclictest0-21swapper/108:15:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional