You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-09-17 - 22:23

Intel(R) Pentium(R) CPU 2020M @ 2.40GHz, Linux 4.16.15-rt7 (Profile)

Latency plot of system in rack #6, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack6slot2.osadl.org (updated Thu Jan 02, 2020 12:43:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1987425115,8sleep00-21swapper/007:05:580
1964124410,8sleep10-21swapper/107:03:351
2016199117,1cyclictest7882-22perl10:03:181
2016199117,1cyclictest0-21swapper/110:37:101
2016199116,2cyclictest9199-21latency_hist11:58:141
2016199110,9cyclictest6378-22fschecks_time09:58:191
2016199110,9cyclictest21396-22iostat10:48:201
2016199110,8cyclictest30729-22/usr/sbin/munin07:43:281
2016199110,8cyclictest14996-22/usr/sbin/munin10:28:211
2016199110,2cyclictest19448-22/usr/sbin/munin10:43:201
2016199110,2cyclictest11553-21rm08:28:141
2016199110,10cyclictest0-21swapper/110:28:141
2016199110,1cyclictest15340-22/usr/sbin/munin12:18:171
2016199108,1cyclictest29403-22df_abs07:38:171
2016199108,1cyclictest28206-22iostat_ios07:33:221
2016199106,2cyclictest5776-22/usr/sbin/munin08:08:231
2016199106,0cyclictest0-21swapper/108:38:251
2016199105,2cyclictest1983-22if_enp0s29u1u209:43:201
2016199102,1cyclictest0-21swapper/111:03:181
2016199100,9cyclictest9095-22grep10:08:181
2016199100,9cyclictest3563-22uname09:48:201
2016199100,9cyclictest29132-22python2.711:13:261
2016199100,9cyclictest22724-22cut10:53:191
2016199100,9cyclictest18718-22python2.710:38:241
2016199100,9cyclictest18718-22python2.710:38:241
2016199100,8cyclictest7251-22/usr/sbin/munin08:13:211
2016199100,8cyclictest3437-22/usr/sbin/munin11:38:231
2016199100,8cyclictest28046-22/usr/sbin/munin09:23:221
2016199100,8cyclictest26922-22pmu-power07:28:261
2016199100,8cyclictest25077-22/usr/sbin/munin09:13:191
2016199100,8cyclictest20440-21latency_hist08:58:151
2016199100,8cyclictest10572-22grep10:13:181
2016199100,8cyclictest0-21swapper/111:48:201
2016199100,7cyclictest0-21swapper/111:23:161
2016199100,6cyclictest18122-21latency_hist12:28:141
2016199100,2cyclictest5369-22iostat_ios11:43:201
2016199100,2cyclictest20655-22cut07:08:201
2016199100,1cyclictest29586-22grep09:28:181
2016199100,1cyclictest23411-21turbostat.cron09:08:141
2016199100,1cyclictest17271-22irqstats12:23:211
201619997,1cyclictest18119-22uname08:48:221
201619997,1cyclictest13841-22/usr/sbin/munin12:13:241
201619997,1cyclictest10864-22/usr/sbin/munin12:03:211
201619997,1cyclictest0-21swapper/109:33:201
201619997,1cyclictest0-21swapper/108:28:201
201619997,1cyclictest0-21swapper/107:18:221
201619997,0cyclictest0-21swapper/107:15:411
201619995,1cyclictest0-21swapper/107:48:141
201619990,8cyclictest9836-22awk11:58:221
201619990,8cyclictest32527-22/usr/sbin/munin09:38:241
201619990,8cyclictest2971-22df_abs07:58:171
201619990,8cyclictest25231-22irqstats07:23:201
201619990,8cyclictest23603-22/usr/sbin/munin09:08:191
201619990,8cyclictest21930-21latency_hist09:03:141
201619990,8cyclictest0-21swapper/111:33:201
201619990,8cyclictest0-21swapper/110:18:221
201619990,8cyclictest0-21swapper/108:33:261
201619990,8cyclictest0-21swapper/108:03:201
201619990,7cyclictest8743-22/usr/sbin/munin08:18:271
201619990,2cyclictest19786-22/usr/sbin/munin12:33:191
201619990,1cyclictest30048-22df_abs11:18:191
201619990,1cyclictest26904-22/usr/sbin/munin11:08:191
201619990,1cyclictest0-21swapper/109:18:271
201619990,1cyclictest0-21swapper/107:53:251
201619986,1cyclictest0-21swapper/111:28:221
201619980,7cyclictest0-21swapper/110:58:551
201619980,1cyclictest16604-22interrupts08:43:211
201619980,1cyclictest12359-22/usr/sbin/munin12:08:171
201619980,0cyclictest4579-22/usr/sbin/munin09:53:201
201609984,1cyclictest88450irq/23-ehci_hcd10:58:290
201609981,6cyclictest11844-21run-crons10:18:140
201609981,2cyclictest32122-21chrt11:23:270
201609980,7cyclictest0-21swapper/011:48:250
201609980,7cyclictest0-21swapper/010:33:310
201609980,7cyclictest0-21swapper/008:40:590
201609980,0cyclictest0-21swapper/012:12:260
201609980,0cyclictest0-21swapper/011:21:570
201609980,0cyclictest0-21swapper/010:53:230
201609980,0cyclictest0-21swapper/008:38:130
201609976,0cyclictest0-21swapper/009:08:260
201609976,0cyclictest0-21swapper/008:59:490
201609975,1cyclictest3147-21runrttasks11:57:290
201609973,1cyclictest88450irq/23-ehci_hcd08:28:250
201609970,6cyclictest0-21swapper/011:44:100
201609970,6cyclictest0-21swapper/010:48:220
201609970,6cyclictest0-21swapper/010:44:410
201609970,6cyclictest0-21swapper/010:24:300
201609970,6cyclictest0-21swapper/009:43:190
201609970,6cyclictest0-21swapper/009:23:200
201609970,6cyclictest0-21swapper/009:06:590
201609970,6cyclictest0-21swapper/008:47:500
201609970,6cyclictest0-21swapper/007:56:110
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
none12:07:567
201609966,0cyclictest0-21swapper/007:29:110
201609965,0cyclictest0-21swapper/012:29:010
201609965,0cyclictest0-21swapper/010:19:290
201609965,0cyclictest0-21swapper/010:03:260
201609965,0cyclictest0-21swapper/007:36:340
201609964,1cyclictest0-21swapper/011:39:340
201609963,2cyclictest7389-21munin-run10:03:140
201609963,2cyclictest3846-21chrt08:00:380
201609963,2cyclictest25785-22cat11:03:200
201609963,2cyclictest23436-22expr07:18:170
201609963,2cyclictest20610-22awk12:33:290
201609963,2cyclictest17306-21taskset12:23:210
201609963,2cyclictest16604-21chrt12:22:560
201609963,2cyclictest0-21swapper/009:38:260
201609962,1cyclictest88450irq/23-ehci_hcd11:13:300
201609962,1cyclictest88450irq/23-ehci_hcd10:13:140
201609962,1cyclictest88450irq/23-ehci_hcd09:28:210
201609962,1cyclictest88450irq/23-ehci_hcd08:18:170
201609962,1cyclictest88450irq/23-ehci_hcd07:43:170
201609962,1cyclictest88450irq/23-ehci_hcd07:23:180
201609961,2cyclictest32471-22/usr/sbin/munin07:48:200
201609961,2cyclictest3147-21runrttasks11:58:430
201609961,2cyclictest11091-22df_inode12:03:180
201609960,3cyclictest29832-22ntp_states07:38:230
201609960,2cyclictest5597-21munin-run08:08:140
201609960,0cyclictest0-21swapper/011:08:500
201609960,0cyclictest0-21swapper/009:17:000
201609960,0cyclictest0-21swapper/007:15:110
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
none12:07:566
201609953,1cyclictest27124-22ntpq09:18:220
201609953,1cyclictest20621-22fschecks_time07:08:200
201609953,1cyclictest19695-22ntpq08:53:250
201609953,1cyclictest0-21swapper/008:23:180
201609952,2cyclictest794-22idleruntime11:28:190
201609952,2cyclictest1994-22cat11:33:180
201609952,2cyclictest18357-22idleruntime10:38:190
201609952,2cyclictest18357-22idleruntime10:38:190
201609952,2cyclictest15061-22expr10:28:170
201609952,2cyclictest14006-22df_abs12:13:180
201609951,2cyclictest4808-22df_inode09:53:190
201609951,1cyclictest88450irq/23-ehci_hcd09:48:170
201609951,1cyclictest88450irq/23-ehci_hcd09:33:170
201609950,4cyclictest6110-22fschecks_time08:08:200
201609950,4cyclictest17945-22grep08:48:200
201609950,2cyclictest7251-22/usr/sbin/munin08:13:190
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:565
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:564
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:563
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
none12:07:562
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional