You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2019-06-27 - 10:22

Intel(R) Pentium(R) CPU 2020M @ 2.40GHz, Linux 4.16.15-rt7 (Profile)

Latency plot of system in rack #6, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack6slot2.osadl.org (updated Thu Jun 27, 2019 00:43:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
309552488,8sleep00-21swapper/019:05:480
3114024520,18sleep10-21swapper/119:07:451
3129899144,5cyclictest2065-21munin-run21:08:440
3129999130,9cyclictest29072-21run-crons00:28:441
3129999130,9cyclictest14836-22irqstats23:38:521
3129999128,1cyclictest0-21swapper/123:24:011
3129999120,8cyclictest8200-22/usr/sbin/munin21:28:471
3129999120,8cyclictest1810-21latency_hist19:18:441
3129999120,8cyclictest15197-21latency_hist20:03:451
3129999120,2cyclictest7159-22iostat21:23:511
3129999117,1cyclictest28972-22/usr/sbin/munin22:38:541
3129999110,9cyclictest16856-22/usr/sbin/munin20:08:551
3129999110,8cyclictest9484-22cpuspeed19:43:471
3129999110,8cyclictest5930-22iostat_ios23:08:511
3129999110,8cyclictest31519-21latency_hist20:58:441
3129999110,8cyclictest26844-22ntp_states00:18:541
3129999110,8cyclictest16682-21munin-run20:08:451
3129999110,8cyclictest16060-22iostat21:53:511
3129999110,8cyclictest1079-22fschecks_time21:03:491
3129999110,8cyclictest0-21swapper/122:43:591
3129999110,8cyclictest0-21swapper/119:18:521
3129999110,7cyclictest0-21swapper/123:53:521
3129999110,7cyclictest0-21swapper/121:58:581
3129999110,2cyclictest7405-22iostat_ios23:13:511
3129999108,1cyclictest31316-22ntp_states00:33:531
3129999108,1cyclictest24803-22grep00:13:461
3129999108,1cyclictest24267-22/usr/sbin/munin20:33:471
3129999108,1cyclictest19978-22df20:18:491
3129999106,1cyclictest23283-22/usr/sbin/munin00:08:481
3129999106,1cyclictest0-21swapper/122:11:591
3129999104,2cyclictest22614-21bash20:28:451
3129999101,8cyclictest0-21swapper/121:43:501
3129999100,9cyclictest25362-21kworker/u16:222:53:571
3129999100,9cyclictest20132-21latency_hist23:58:441
3129999100,8cyclictest26090-22cstates22:28:471
3129999100,8cyclictest0-21swapper/122:03:491
3129999100,7cyclictest0-21swapper/123:03:581
3129999100,7cyclictest0-21swapper/121:38:571
3129999100,7cyclictest0-21swapper/120:28:501
3129999100,7cyclictest0-21swapper/100:03:471
3129999100,6cyclictest18330-22/usr/sbin/munin20:13:481
3129999100,2cyclictest31702-22/usr/sbin/munin20:58:591
3129999100,1cyclictest8287-22fschecks_time19:38:491
3129999100,1cyclictest5583-22open_files19:28:531
3129999100,1cyclictest3997-22memory19:23:521
3129999100,1cyclictest3736-22/usr/sbin/munin21:13:491
3129999100,1cyclictest31746-22tune2fs19:08:481
3129999100,1cyclictest2243-22/usr/sbin/munin21:08:531
3129999100,1cyclictest0-21swapper/123:18:531
3129999100,1cyclictest0-21swapper/122:23:571
312999998,0cyclictest27482-22/usr/sbin/munin22:33:541
312999997,1cyclictest15851-22/usr/sbin/munin23:43:481
312999997,1cyclictest0-21swapper/123:29:011
312999991,7cyclictest29600-22tune2fs00:28:511
312999990,8cyclictest6461-22/usr/sbin/munin19:33:551
312999990,8cyclictest5232-22/usr/sbin/munin21:18:521
312999990,8cyclictest22290-22smartctl22:13:541
312999990,8cyclictest0-21swapper/122:21:411
312999990,8cyclictest0-21swapper/121:33:561
312999990,6cyclictest11889-21sshd19:50:231
312999990,6cyclictest0-21swapper/123:48:491
312999990,6cyclictest0-21swapper/121:48:441
312999990,1cyclictest27232-22/usr/sbin/munin20:43:501
312999990,1cyclictest21-21rcuc/120:38:491
312989996,1cyclictest27920-22interrupts22:33:510
312989990,8cyclictest8406-22iostat_ios19:38:510
312989990,8cyclictest0-21swapper/021:13:550
312989990,8cyclictest0-21swapper/019:18:470
312989990,5cyclictest6551-21munin-run21:23:440
312989990,5cyclictest3821-21munin-run23:03:440
312989990,5cyclictest29073-21munin-run00:28:440
312989990,0cyclictest0-21swapper/021:48:590
312999980,7cyclictest0-21swapper/123:36:221
312999980,5cyclictest0-21swapper/120:48:521
312989987,0cyclictest0-21swapper/023:32:000
312989982,1cyclictest156950irq/19-ahci[00022:18:500
312989980,7cyclictest0-21swapper/020:01:350
312989980,4cyclictest18421-21munin-run22:03:440
312989980,0cyclictest0-21swapper/020:49:180
312989980,0cyclictest0-21swapper/019:58:060
312999976,0cyclictest12903-22needreboot19:53:521
312999970,6cyclictest2788-21kworker/1:223:00:271
312999970,1cyclictest0-21swapper/122:48:441
312989977,0cyclictest0-21swapper/020:29:310
312989976,0cyclictest0-21swapper/019:10:110
312989974,1cyclictest19073-22grep23:53:490
312989973,1cyclictest88350irq/23-ehci_hcd23:18:500
312989973,1cyclictest88350irq/23-ehci_hcd23:08:510
312989973,1cyclictest88350irq/23-ehci_hcd20:18:500
312989973,1cyclictest88350irq/23-ehci_hcd20:03:500
312989973,1cyclictest88350irq/23-ehci_hcd00:18:530
312989973,1cyclictest88350irq/23-ehci_hcd00:03:550
312989973,0cyclictest0-21swapper/020:33:570
312989971,2cyclictest4804-21latency_hist19:28:440
312989970,6cyclictest0-21swapper/023:50:100
312989970,6cyclictest0-21swapper/022:54:080
312989970,6cyclictest0-21swapper/022:13:570
312989970,6cyclictest0-21swapper/021:41:380
312989970,6cyclictest0-21swapper/019:38:390
312989970,6cyclictest0-21swapper/019:32:310
312989970,6cyclictest0-21swapper/019:18:040
312989970,6cyclictest0-21swapper/000:35:430
312989970,6cyclictest0-21swapper/000:28:570
312989970,0cyclictest0-21swapper/022:45:200
312989970,0cyclictest0-21swapper/000:10:000
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
none18:17:437
312989965,0cyclictest0-21swapper/021:02:130
312989963,2cyclictest8478-22uniq21:28:500
312989963,2cyclictest4889-21taskset23:04:430
312989963,2cyclictest24661-22expr22:23:470
312989963,2cyclictest18580-22grep20:13:490
312989963,2cyclictest13507-22open_inodes23:33:530
312989963,2cyclictest10414-22cut23:23:540
312989962,2cyclictest8133-21chrt23:17:310
312989962,1cyclictest88350irq/23-ehci_hcd23:43:480
312989962,1cyclictest88350irq/23-ehci_hcd22:38:550
312989962,1cyclictest88350irq/23-ehci_hcd22:03:480
312989962,1cyclictest88350irq/23-ehci_hcd21:43:490
312989962,1cyclictest88350irq/23-ehci_hcd21:23:490
312989962,1cyclictest25400-22ls00:13:530
312989962,1cyclictest17210-22tune2fs20:08:500
312989962,1cyclictest0-21swapper/022:28:480
312989961,3cyclictest21757-22iostat_ios20:23:510
312989961,1cyclictest16068-22irqstats21:53:520
312989960,6cyclictest0-21swapper/021:34:510
312989960,5cyclictest0-21swapper/022:08:570
312989960,5cyclictest0-21swapper/021:08:580
312989960,4cyclictest2875-21lldpd22:50:210
312989960,0cyclictest0-21swapper/020:56:310
312989960,0cyclictest0-21swapper/019:44:110
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
none18:17:436
312989953,1cyclictest11071-22df19:48:470
312989953,1cyclictest0-21swapper/020:38:520
312989952,2cyclictest27545-22cut20:43:480
312989950,2cyclictest20316-22/usr/sbin/munin23:59:000
312989950,2cyclictest14378-22/usr/sbin/munin23:38:520
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:435
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:434
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:433
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
none18:17:432
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional