You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2019-06-26 - 20:29

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #1, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack1slot0.osadl.org (updated Wed Jun 26, 2019 12:44:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1200826643,7sleep10-21swapper/107:07:551
1200226138,7sleep00-21swapper/007:07:500
6392590,0sleep30-21swapper/312:00:083
1188825936,7sleep30-21swapper/307:06:293
1221425533,7sleep20-21swapper/207:09:522
12412991515,0cyclictest2624-40kipmi007:45:051
12412991212,0cyclictest2624-40kipmi009:45:041
1240299110,10cyclictest0-21swapper/011:30:030
1242499109,0cyclictest0-21swapper/310:25:003
1241899109,0cyclictest0-21swapper/208:20:032
1241299100,9cyclictest0-21swapper/111:14:131
1241299100,9cyclictest0-21swapper/110:53:521
124249998,0cyclictest0-21swapper/311:30:443
124249997,1cyclictest11442-21vmstat12:25:183
124189998,0cyclictest0-21swapper/208:50:052
124129998,0cyclictest0-21swapper/112:29:461
124129998,0cyclictest0-21swapper/110:03:291
124129998,0cyclictest0-21swapper/109:08:261
124129998,0cyclictest0-21swapper/108:55:101
124129998,0cyclictest0-21swapper/108:52:071
124129990,0cyclictest0-21swapper/108:33:241
124129990,0cyclictest0-21swapper/107:31:431
124029993,3cyclictest18592-21perf08:45:000
124249987,0cyclictest0-21swapper/309:03:373
124249987,0cyclictest0-21swapper/308:43:073
124249980,0cyclictest0-21swapper/310:03:313
124189987,0cyclictest0-21swapper/211:13:372
124129987,0cyclictest0-21swapper/112:39:471
124129987,0cyclictest0-21swapper/111:56:111
124129987,0cyclictest0-21swapper/110:32:441
124129987,0cyclictest0-21swapper/108:48:541
124129986,1cyclictest25-21ksoftirqd/108:15:001
124129980,7cyclictest0-21swapper/111:33:451
124029987,0cyclictest11549-21kworker/0:012:33:410
124029987,0cyclictest0-21swapper/011:11:100
124249976,0cyclictest0-21swapper/312:08:503
124249976,0cyclictest0-21swapper/312:08:493
124189970,6cyclictest31784-21sshd11:55:412
124129977,0cyclictest2624-40kipmi010:55:051
124129977,0cyclictest2624-40kipmi010:10:061
124129976,0cyclictest0-21swapper/112:19:371
124129976,0cyclictest0-21swapper/111:51:301
124129976,0cyclictest0-21swapper/111:20:591
124129976,0cyclictest0-21swapper/110:44:591
124129976,0cyclictest0-21swapper/109:17:091
124129976,0cyclictest0-21swapper/108:20:381
124129976,0cyclictest0-21swapper/108:09:491
124129975,1cyclictest4422-21snmpd11:00:041
124129975,1cyclictest25-21ksoftirqd/110:10:001
124129970,6cyclictest0-21swapper/107:38:201
124129970,0cyclictest0-21swapper/108:16:161
124029976,0cyclictest0-21swapper/012:00:050
124029976,0cyclictest0-21swapper/008:10:090
124029970,6cyclictest24009-21sshd11:38:320
124029970,6cyclictest2213-21sshd09:24:560
124029970,6cyclictest14327-21sshd08:32:420
19471260,4sleep219689-21awk11:25:122
14724262,0sleep12624-40kipmi011:15:061
124249964,1cyclictest9767-21smicount08:20:113
124249964,1cyclictest829-21irqstats09:20:063
124249964,1cyclictest4423-21snmpd09:37:533
124249964,1cyclictest18628-21df_inode11:25:033
124249964,1cyclictest12342-21crond09:50:003
124249963,2cyclictest5821-21awk12:14:093
124249963,2cyclictest4423-21snmpd12:15:033
124249963,2cyclictest4423-21snmpd11:19:153
124249963,2cyclictest4423-21snmpd10:50:533
124249963,2cyclictest26002-21sshd11:43:263
124249960,5cyclictest0-21swapper/308:03:243
124249960,5cyclictest0-21swapper/307:30:273
124189964,1cyclictest4422-21snmpd11:30:052
124189963,2cyclictest4423-21snmpd07:47:052
124189963,2cyclictest4422-21snmpd12:00:332
124189963,2cyclictest3784-21dbus-daemon08:29:552
124189960,5cyclictest26487-21sshd09:03:362
124129966,0cyclictest2624-40kipmi007:20:051
124129965,0cyclictest0-21swapper/108:41:381
124129965,0cyclictest0-21swapper/107:19:361
124129965,0cyclictest0-21swapper/107:19:361
124129964,1cyclictest25-21ksoftirqd/111:50:001
124129964,1cyclictest25-21ksoftirqd/109:15:001
124129964,1cyclictest25-21ksoftirqd/108:30:001
124129963,2cyclictest673-21wc08:00:011
124129963,2cyclictest4423-21snmpd09:03:311
124129962,1cyclictest25-21ksoftirqd/111:40:011
124129961,2cyclictest25-21ksoftirqd/107:50:001
124129961,2cyclictest131rcu_sched12:20:001
124129960,5cyclictest24416-21kworker/1:010:23:381
124129960,0cyclictest0-21swapper/109:54:271
124029964,1cyclictest4978-21irqstats09:30:070
124029964,1cyclictest4422-21snmpd08:03:510
124029964,1cyclictest19948-21cat08:45:110
124029963,2cyclictest4423-21snmpd11:58:120
124029963,2cyclictest4423-21snmpd11:26:140
124029963,2cyclictest4423-21snmpd10:40:020
124029963,2cyclictest4423-21snmpd10:27:330
124029963,2cyclictest4423-21snmpd08:59:400
124029963,2cyclictest4422-21snmpd12:18:140
124029960,5cyclictest28480-21sshd09:08:340
124029960,5cyclictest22412-21sshd07:30:410
124029960,5cyclictest0-21swapper/008:16:270
124029960,5cyclictest0-21swapper/007:36:250
4437250,1sleep2311rcuc/212:10:042
124249953,1cyclictest8976-21gawk09:40:063
124249953,1cyclictest8976-21gawk09:40:063
124249953,1cyclictest8714-21cat11:00:043
124249953,1cyclictest7151-21cat08:15:053
124249953,1cyclictest7138-21open_inodes10:55:083
124249953,1cyclictest4541-21/usr/sbin/munin09:30:093
124249953,1cyclictest4423-21snmpd12:31:103
124249953,1cyclictest4423-21snmpd08:50:223
124249953,1cyclictest4423-21snmpd07:35:023
124249953,1cyclictest4423-21snmpd07:11:013
124249953,1cyclictest4422-21snmpd12:20:353
124249953,1cyclictest4422-21snmpd11:35:263
124249953,1cyclictest4422-21snmpd10:41:043
124249953,1cyclictest4422-21snmpd10:15:063
124249953,1cyclictest4422-21snmpd08:35:153
124249953,1cyclictest4422-21snmpd08:10:033
124249953,1cyclictest4422-21snmpd08:06:053
124249953,1cyclictest4422-21snmpd07:42:383
124249953,1cyclictest4422-21snmpd07:25:023
124249953,1cyclictest3157-21cut09:25:083
124249953,1cyclictest30929-21cpuspeed_turbos09:15:023
124249953,1cyclictest30767-21cat11:55:043
124249953,1cyclictest29317-21nscd12:35:083
124249953,1cyclictest29253-21cat09:10:053
124249953,1cyclictest28624-21/usr/sbin/munin10:30:053
124249953,1cyclictest27449-21cat07:45:053
124249953,1cyclictest27236-21cat09:05:053
124249953,1cyclictest26776-21cat11:45:073
124249953,1cyclictest2471-21df_inode10:45:023
124249953,1cyclictest22808-21/usr/sbin/munin08:55:043
124249953,1cyclictest20640-21cpuspeed_turbos10:10:023
124249953,1cyclictest18992-21cat10:05:073
124249953,1cyclictest18786-21/usr/sbin/munin08:45:093
124249953,1cyclictest17007-21/usr/sbin/munin07:20:123
124249953,1cyclictest14968-21cat09:55:053
124249953,1cyclictest14945-21cstates07:15:023
124249953,1cyclictest14945-21cstates07:15:013
124249953,1cyclictest13399-21needreboot08:30:093
124249953,1cyclictest12560-21/usr/sbin/munin09:50:053
124249953,1cyclictest12510-21cstates11:10:023
124249953,1cyclictest11171-21cat08:25:053
124249953,1cyclictest10336-21/usr/sbin/munin11:05:103
124249952,2cyclictest29364-21hostname07:50:043
124249952,2cyclictest28695-21sh11:50:053
124249952,2cyclictest17354-21which11:20:113
124249950,4cyclictest0-21swapper/310:24:513
124249950,1cyclictest0-21swapper/307:55:083
124249950,0cyclictest0-21swapper/310:37:353
124189954,0cyclictest0-21swapper/211:20:042
124189954,0cyclictest0-21swapper/207:35:082
124189953,1cyclictest8698-21cat11:00:052
124189953,1cyclictest8514-21cat12:20:052
124189953,1cyclictest6678-21cat10:55:042
124189953,1cyclictest6494-21cat12:15:042
124189953,1cyclictest4423-21snmpd12:36:502
124189953,1cyclictest4423-21snmpd12:05:192
124189953,1cyclictest4423-21snmpd12:05:182
124189953,1cyclictest4423-21snmpd10:30:032
124189953,1cyclictest4423-21snmpd10:05:242
124189953,1cyclictest4423-21snmpd09:46:052
124189953,1cyclictest4423-21snmpd09:40:082
124189953,1cyclictest4423-21snmpd09:40:082
124189953,1cyclictest4423-21snmpd09:36:322
124189953,1cyclictest4423-21snmpd09:25:442
124189953,1cyclictest4423-21snmpd09:15:012
124189953,1cyclictest4423-21snmpd08:35:062
124189953,1cyclictest4423-21snmpd08:31:382
124189953,1cyclictest4423-21snmpd08:10:022
124189953,1cyclictest4423-21snmpd07:55:202
124189953,1cyclictest4423-21snmpd07:51:132
124189953,1cyclictest4423-21snmpd07:25:072
124189953,1cyclictest4423-21snmpd07:20:002
124189953,1cyclictest4423-21snmpd07:15:072
124189953,1cyclictest4423-21snmpd07:15:072
124189953,1cyclictest4422-21snmpd11:45:182
124189953,1cyclictest4422-21snmpd11:40:072
124189953,1cyclictest4422-21snmpd11:15:032
124189953,1cyclictest4422-21snmpd11:05:352
124189953,1cyclictest4422-21snmpd10:50:162
124189953,1cyclictest4422-21snmpd10:45:282
124189953,1cyclictest4422-21snmpd10:40:582
124189953,1cyclictest4422-21snmpd10:35:012
124189953,1cyclictest4422-21snmpd10:15:302
124189953,1cyclictest4422-21snmpd09:50:062
124189953,1cyclictest4422-21snmpd09:30:082
124189953,1cyclictest4422-21snmpd08:55:582
124189953,1cyclictest4422-21snmpd08:40:452
124189953,1cyclictest4422-21snmpd08:05:272
124189953,1cyclictest4251-21runrttasks10:25:272
124189953,1cyclictest29252-21cat09:10:052
124189953,1cyclictest28805-21cat11:50:062
124189953,1cyclictest27616-21ntp_kernel_pll_09:05:082
124189953,1cyclictest25413-21cat07:40:052
124189953,1cyclictest24881-21df_inode10:20:032
124189953,1cyclictest22781-21cat11:35:052
124189953,1cyclictest21639-21missed_timers07:30:072
124189953,1cyclictest18878-21grep08:45:032
124189953,1cyclictest1493-21xargs09:20:112
124189953,1cyclictest14921-21cat09:55:042
124189953,1cyclictest12857-21cat07:10:062
124189953,1cyclictest12117-21cpu12:30:052
124189952,2cyclictest8058-21sed08:15:112
124189952,2cyclictest21858-21awk10:10:122
124189952,2cyclictest2021-21taskset08:00:512
124189952,2cyclictest16778-21cat10:00:022
124189950,4cyclictest11166-21cat12:25:162
124129953,1cyclictest9801-21grep09:40:121
124129953,1cyclictest9801-21grep09:40:121
124129953,1cyclictest5457-21processes09:30:101
124129953,1cyclictest4423-21snmpd12:11:361
124129953,1cyclictest4423-21snmpd10:47:231
124129953,1cyclictest4423-21snmpd10:25:031
124129953,1cyclictest4423-21snmpd07:58:141
124129953,1cyclictest4422-21snmpd12:30:021
124129953,1cyclictest4422-21snmpd11:05:231
124129953,1cyclictest4422-21snmpd09:25:051
124129953,1cyclictest4422-21snmpd07:40:531
124129953,1cyclictest332-21cat12:00:051
124129953,1cyclictest30635-21/usr/sbin/munin10:35:021
124129953,1cyclictest2379-21cat12:05:051
124129953,1cyclictest2379-21cat12:05:041
124129953,1cyclictest22768-21cat11:35:051
124129953,1cyclictest22606-21/usr/sbin/munin10:15:051
124129953,1cyclictest20315-21grep07:25:121
124129953,1cyclictest15182-21cat08:35:051
124129953,1cyclictest14965-21cat09:55:051
124129953,1cyclictest12903-21cat07:10:051
124129952,2cyclictest19112-21cut11:25:081
124129951,1cyclictest121rcu_preempt09:35:021
124129950,1cyclictest396-21/usr/sbin/munin09:20:031
124029954,0cyclictest0-21swapper/010:47:060
124029953,1cyclictest9347-21missed_timers08:20:070
124029953,1cyclictest8662-21tr09:40:030
124029953,1cyclictest8662-21tr09:40:030
124029953,1cyclictest8474-21if_eno212:20:040
124029953,1cyclictest4423-21snmpd12:25:330
124029953,1cyclictest4423-21snmpd11:00:130
124029953,1cyclictest4423-21snmpd10:56:140
124029953,1cyclictest4423-21snmpd10:00:300
124029953,1cyclictest4423-21snmpd09:55:080
124029953,1cyclictest4423-21snmpd09:50:020
124029953,1cyclictest4423-21snmpd09:25:050
124029953,1cyclictest4423-21snmpd09:10:040
124029953,1cyclictest4423-21snmpd07:47:220
124029953,1cyclictest4422-21snmpd12:35:550
124029953,1cyclictest4422-21snmpd12:10:030
124029953,1cyclictest4422-21snmpd12:05:100
124029953,1cyclictest4422-21snmpd12:05:090
124029953,1cyclictest4422-21snmpd11:20:350
124029953,1cyclictest4422-21snmpd11:16:170
124029953,1cyclictest4422-21snmpd10:50:460
124029953,1cyclictest4422-21snmpd10:12:150
124029953,1cyclictest4422-21snmpd10:05:180
124029953,1cyclictest4422-21snmpd09:35:020
124029953,1cyclictest4422-21snmpd09:15:110
124029953,1cyclictest4422-21snmpd07:50:200
124029953,1cyclictest4422-21snmpd07:40:050
124029953,1cyclictest4422-21snmpd07:25:230
124029953,1cyclictest4422-21snmpd07:20:370
124029953,1cyclictest4422-21snmpd07:10:090
124029953,1cyclictest31358-21hddtemp_smartct07:55:050
124029953,1cyclictest30677-21cpu10:35:020
124029953,1cyclictest3040-21cat08:05:040
124029953,1cyclictest29366-21sendmail_mailqu11:50:130
124029953,1cyclictest28624-21/usr/sbin/munin10:30:110
124029953,1cyclictest26746-21cat11:45:070
124029953,1cyclictest25170-21ntp_offset11:40:090
124029953,1cyclictest24973-21if_eno210:20:040
124029953,1cyclictest24824-21cpu09:00:020
124029953,1cyclictest22982-21cat10:15:040
124029953,1cyclictest21216-21cat08:50:050
124029953,1cyclictest15000-21/usr/sbin/munin07:15:040
124029953,1cyclictest15000-21/usr/sbin/munin07:15:040
124029953,1cyclictest11057-21hddtemp_smartct08:25:030
124029953,1cyclictest10776-21kernelversion11:05:060
124029952,2cyclictest16142-21chrt08:35:310
124029952,2cyclictest11626-21threads09:45:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional