You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-05-12 - 13:45

Intel(R) Xeon(R) CPU E3-1270 V2 @ 3.50GHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Wed May 12, 2021 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1010925344,0sleep20-21swapper/207:15:282
291450460,0irq/31-eth10-21swapper/107:14:451
1003224634,0sleep30-21swapper/307:14:333
1011723524,0sleep00-21swapper/007:15:340
104999980,0cyclictest0-21swapper/112:34:131
104999973,0cyclictest6262-21sadc09:18:541
104999973,0cyclictest4522-21awk08:09:091
104999973,0cyclictest32373-21awk12:09:091
104999972,0cyclictest16616-21awk07:29:181
104999970,0cyclictest6275-21diskmemload11:53:041
104999970,0cyclictest4753-21if_eth112:14:081
104999970,0cyclictest25953-21turbostat.cron08:53:541
104999970,0cyclictest0-21swapper/110:43:311
104939977,0cyclictest0-21swapper/009:19:130
149960,0migration/018466-21latency_hist08:38:540
149960,0migration/014614-21timerwakeupswit11:49:190
149960,0migration/01137-21df_abs08:04:040
105119965,0cyclictest3385-21snmpd09:54:163
105059966,0cyclictest6275-21diskmemload10:51:562
105059960,0cyclictest0-21swapper/211:52:372
104999965,0cyclictest0-21swapper/109:41:331
104999962,0cyclictest3385-21snmpd11:56:181
104999962,0cyclictest3385-21snmpd08:00:321
104999962,0cyclictest28573-21ps12:39:241
104999962,0cyclictest23199-21awk10:14:091
104999962,0cyclictest14923-21ntpdc08:29:161
104999961,0cyclictest9052-21rt-features10:34:131
104999961,0cyclictest27517-21awk10:19:081
104999961,0cyclictest27161-21awk10:54:081
104999960,0cyclictest8946-21rm09:20:301
104999960,0cyclictest554-21ssh12:44:121
104999960,0cyclictest5462-21ssh11:04:371
104999960,0cyclictest4339-21chrt11:39:131
104999960,0cyclictest3349-21fschecks_count10:29:101
104999960,0cyclictest32730-21sensors_volt09:04:131
104999960,0cyclictest32730-21sensors_volt09:04:131
104999960,0cyclictest31957-21awk10:59:181
104999960,0cyclictest30549-21ssh09:45:421
104999960,0cyclictest301792sleep107:54:291
104999960,0cyclictest29765-21sh12:06:211
104999960,0cyclictest28712-21ssh11:29:521
104999960,0cyclictest27683-21munin-node08:54:141
104999960,0cyclictest26024-21df_inode07:49:041
104999960,0cyclictest24567-21ntp_states07:44:111
104999960,0cyclictest23161-21netstat11:24:121
104999960,0cyclictest21378-21sh09:34:511
104999960,0cyclictest20867-21ssh10:11:521
104999960,0cyclictest19591-21ssh11:20:121
104999960,0cyclictest1904-21ssh11:36:451
104999960,0cyclictest17500-21ssh09:30:461
104999960,0cyclictest15425-21taskset07:28:211
104999960,0cyclictest15010-21ssh11:15:151
104999960,0cyclictest1447-21ntp_states09:49:191
104999960,0cyclictest13838-21sh09:27:111
104999960,0cyclictest13133-21ssh12:24:101
104999960,0cyclictest11826-21ssh12:22:091
104999960,0cyclictest11735-21irqstats07:19:101
104999960,0cyclictest10090-21iostat09:59:081
104939966,0cyclictest23314-40yum-updatesd-he08:47:390
35150,0ktimersoftd/30-21swapper/311:53:043
35150,0ktimersoftd/30-21swapper/310:21:473
28425050,0irq/33-eth0-tx-0-21swapper/212:24:352
28425050,0irq/33-eth0-tx-0-21swapper/008:49:110
28425050,0irq/33-eth0-tx-0-21swapper/008:29:230
28425050,0irq/33-eth0-tx-0-21swapper/008:14:100
27150,0ktimersoftd/20-21swapper/212:34:132
249950,0watchdog/24924-21smartctl11:04:162
249950,0watchdog/20-21swapper/210:47:402
149950,0migration/07037-21sh12:16:540
149950,0migration/04552-21missed_timers11:04:110
149950,0migration/031920-21hwlatdetect11:34:070
149950,0migration/027740-21needreboot11:29:110
149950,0migration/027401-21fschecks_time12:04:060
149950,0migration/026208-21taskset10:54:040
149950,0migration/025469-21taskset07:48:140
149950,0migration/024614-21sh10:51:230
149950,0migration/024286-21taskset10:14:390
149950,0migration/021598-21latency_hist11:23:540
149950,0migration/021280-21fschecks_time07:39:050
149950,0migration/020484-21taskset07:38:450
149950,0migration/018806-21munin-node11:54:200
149950,0migration/015971-21df_abs07:29:070
105119955,0cyclictest0-21swapper/312:45:153
105119950,0cyclictest0-21swapper/312:22:523
105119950,0cyclictest0-21swapper/311:57:203
105119950,0cyclictest0-21swapper/310:55:123
105059955,0cyclictest0-21swapper/212:45:452
105059955,0cyclictest0-21swapper/211:24:242
105059955,0cyclictest0-21swapper/211:10:412
105059955,0cyclictest0-21swapper/210:19:462
105059955,0cyclictest0-21swapper/210:06:502
105059955,0cyclictest0-21swapper/209:55:012
105059955,0cyclictest0-21swapper/209:25:132
105059954,0cyclictest7436-21awk09:19:082
105059954,0cyclictest4524-21awk11:39:172
105059954,0cyclictest31862-21awk10:24:142
105059950,0cyclictest0-21swapper/211:54:032
105059950,0cyclictest0-21swapper/211:34:112
104999951,0cyclictest0-21swapper/109:09:061
104999950,0cyclictest8685-21tr08:19:041
104999950,0cyclictest8071-21sh11:43:561
104999950,0cyclictest8041-21sh11:09:051
104999950,0cyclictest618-21latency_hist08:03:541
104999950,0cyclictest6155-21munin-node08:14:041
104999950,0cyclictest4590-21taskset09:54:011
104999950,0cyclictest30861-21taskset10:23:561
104999950,0cyclictest28645-21wc08:58:541
104999950,0cyclictest22058-21sh11:59:001
104999950,0cyclictest21473-21sh10:48:551
104999950,0cyclictest21263-21cpuspeed_turbos08:44:091
104999950,0cyclictest20680-21cat07:38:541
104999950,0cyclictest18740-21cpuspeed_turbos08:39:061
104999950,0cyclictest18335-21cut07:34:051
104999950,0cyclictest17509-21sed12:28:541
104999950,0cyclictest16816-21sh10:43:571
104999950,0cyclictest16244-21cut08:34:051
104999950,0cyclictest13272-21rm10:04:001
104999950,0cyclictest11240-21tr08:24:051
104939950,0cyclictest0-21swapper/012:29:000
104939950,0cyclictest0-21swapper/012:19:230
104939950,0cyclictest0-21swapper/011:46:450
104939950,0cyclictest0-21swapper/009:26:170
8305040,0irq/26-0000:00:0-21swapper/309:58:543
35140,0ktimersoftd/30-21swapper/312:29:583
35140,0ktimersoftd/30-21swapper/312:27:073
329940,0watchdog/3284150irq/32-eth0-rx-11:44:283
329940,0watchdog/320495-21munin-node09:34:123
329940,0watchdog/319380-21perl08:39:083
329940,0watchdog/3182252sleep311:19:083
329940,0watchdog/30-21swapper/311:04:203
329940,0watchdog/30-21swapper/310:12:323
329940,0watchdog/30-21swapper/309:39:203
329940,0watchdog/30-21swapper/307:34:273
28425040,0irq/33-eth0-tx-32415-21ssh09:49:012
28425040,0irq/33-eth0-tx-30824-21ssh09:46:132
28425040,0irq/33-eth0-tx-26211-21munin-node10:54:182
28425040,0irq/33-eth0-tx-22156-21munin-node10:14:132
28425040,0irq/33-eth0-tx-20409-21missed_timers09:34:112
28425040,0irq/33-eth0-tx-17147-21munin-node11:19:152
28425040,0irq/33-eth0-tx-12569-21cpu11:14:052
28425040,0irq/33-eth0-tx-0-21swapper/212:14:102
28425040,0irq/33-eth0-tx-0-21swapper/211:44:292
28425040,0irq/33-eth0-tx-0-21swapper/009:09:200
28425040,0irq/33-eth0-tx-0-21swapper/008:24:130
28425040,0irq/33-eth0-tx-0-21swapper/008:22:440
28415040,0irq/32-eth0-rx-9571-21swap11:09:173
28415040,0irq/32-eth0-rx-13400-21ssh10:04:093
28415040,0irq/32-eth0-rx-0-21swapper/312:42:073
28415040,0irq/32-eth0-rx-0-21swapper/312:04:173
28415040,0irq/32-eth0-rx-0-21swapper/311:33:593
28415040,0irq/32-eth0-rx-0-21swapper/310:59:123
28415040,0irq/32-eth0-rx-0-21swapper/310:35:333
28415040,0irq/32-eth0-rx-0-21swapper/310:17:453
28415040,0irq/32-eth0-rx-0-21swapper/309:28:093
28415040,0irq/32-eth0-rx-0-21swapper/309:21:373
259940,0migration/29081-21munin-node10:34:142
259940,0migration/226548-21gltestperf07:49:052
259940,0migration/224479-21needreboot07:44:102
249940,0watchdog/26275-21diskmemload11:01:482
249940,0watchdog/23385-21snmpd08:00:322
249940,0watchdog/217950-21diskstats12:29:082
249940,0watchdog/211150-21ssh09:59:362
249940,0watchdog/20-21swapper/212:00:042
249940,0watchdog/20-21swapper/208:04:122
249940,0watchdog/20-21swapper/207:24:112
149940,0migration/09961-21runrttasks11:09:230
149940,0migration/028996-21kernelversion09:44:090
149940,0migration/028813-21timerwakeupswit12:39:280
149940,0migration/023103-21gltestperf12:34:060
149940,0migration/020018-21runrttasks10:46:340
149940,0migration/019220-21fschecks_time08:39:070
105119944,0cyclictest0-21swapper/311:41:053
105119944,0cyclictest0-21swapper/310:38:583
105119944,0cyclictest0-21swapper/309:30:333
105119943,0cyclictest982-21awk09:49:163
105119943,0cyclictest3385-21snmpd12:09:443
105119943,0cyclictest3385-21snmpd10:44:133
105119943,0cyclictest3385-21snmpd10:28:573
105119943,0cyclictest3385-21snmpd09:14:573
105119943,0cyclictest3385-21snmpd09:06:573
105119943,0cyclictest3385-21snmpd09:06:573
105119943,0cyclictest3385-21snmpd08:57:573
105119943,0cyclictest3385-21snmpd08:35:263
105119943,0cyclictest3385-21snmpd08:19:263
105119943,0cyclictest3385-21snmpd07:30:403
105119943,0cyclictest3385-21snmpd07:28:103
105119943,0cyclictest28821-21awk09:44:073
105119943,0cyclictest27489-21awk11:29:103
105119943,0cyclictest2296-21awk09:09:153
105119943,0cyclictest22088-21awk08:44:163
105119943,0cyclictest14135-21perl08:29:093
105119943,0cyclictest13503-21awk11:14:083
105119943,0cyclictest11981-21awk08:24:083
105119940,0cyclictest22271-21grep11:24:063
105059944,0cyclictest0-21swapper/212:09:202
105059944,0cyclictest0-21swapper/210:29:422
105059944,0cyclictest0-21swapper/209:39:242
105059943,0cyclictest9389-21awk12:19:092
105059943,0cyclictest7067-21awk08:14:092
105059943,0cyclictest4982-21awk09:14:092
105059943,0cyclictest3385-21snmpd10:41:422
105059943,0cyclictest3385-21snmpd09:28:572
105059943,0cyclictest3385-21snmpd09:10:272
105059943,0cyclictest3385-21snmpd08:24:112
105059943,0cyclictest29597-21awk08:59:082
105059943,0cyclictest29165-21awk07:54:082
105059943,0cyclictest27959-21awk12:39:192
105059943,0cyclictest27682-21awk12:04:082
105059943,0cyclictest18677-21awk10:09:092
105059943,0cyclictest16720-21awk07:29:182
105059940,0cyclictest26462-21munin-node11:29:062
104939944,0cyclictest0-21swapper/010:30:080
104939943,0cyclictest5579-21awk09:54:110
104939943,0cyclictest3385-21snmpd12:44:140
104939943,0cyclictest3385-21snmpd10:25:420
104939943,0cyclictest3385-21snmpd10:06:420
104939943,0cyclictest3385-21snmpd10:02:420
104939943,0cyclictest3385-21snmpd09:39:420
104939943,0cyclictest3385-21snmpd07:52:110
104939943,0cyclictest3385-21snmpd07:27:100
104939943,0cyclictest29816-21kworker/0:007:56:060
104939943,0cyclictest27566-21awk10:19:080
104939943,0cyclictest23760-21kworker/0:211:39:460
104939943,0cyclictest23760-21kworker/0:211:03:220
104939943,0cyclictest20088-21awk09:34:090
104939943,0cyclictest18651-21awk10:09:090
104939943,0cyclictest14112-21awk12:24:160
104939943,0cyclictest13471-21awk11:14:080
104939943,0cyclictest11564-21awk07:19:080
104939940,0cyclictest0-21swapper/010:34:020
104939940,0cyclictest0-21swapper/007:59:140
8305030,0irq/26-0000:00:0-21swapper/307:54:203
339930,0migration/325551-21runrttasks08:49:223
329930,0watchdog/30-21swapper/307:50:123
329930,0watchdog/30-21swapper/307:43:553
28415030,0irq/32-eth0-rx-0-21swapper/312:33:593
28415030,0irq/32-eth0-rx-0-21swapper/312:14:063
28415030,0irq/32-eth0-rx-0-21swapper/310:48:593
249930,0watchdog/20-21swapper/208:41:122
249930,0watchdog/20-21swapper/207:36:312
105119932,0cyclictest3470-21cut08:08:543
105119932,0cyclictest3385-21snmpd07:19:103
105119932,0cyclictest31745-21awk07:59:093
105119932,0cyclictest23000-21awk11:59:093
105119932,0cyclictest0-21swapper/310:24:073
105119930,0cyclictest6807-21perl08:14:063
105119930,0cyclictest21268-21tune2fs07:39:053
105119930,0cyclictest2062-21memory08:04:103
105059932,0cyclictest8545-21ls08:18:542
105059932,0cyclictest27058-21awk08:54:082
105059932,0cyclictest22071-21awk08:44:162
105059931,0cyclictest12201-21sensors07:19:142
105059930,0cyclictest16733-21perl08:34:072
105059930,0cyclictest14669-21irqstats08:29:132
105059930,0cyclictest0-21swapper/208:12:322
105059930,0cyclictest0-21swapper/207:39:102
104939933,0cyclictest29816-21kworker/0:008:09:120
104939933,0cyclictest23760-21kworker/0:212:09:060
104939933,0cyclictest23760-21kworker/0:211:58:560
104939933,0cyclictest23760-21kworker/0:211:23:540
104939933,0cyclictest23760-21kworker/0:210:39:000
104939933,0cyclictest20177-21kworker/0:109:28:540
104939933,0cyclictest20177-21kworker/0:109:14:000
104939933,0cyclictest20177-21kworker/0:109:04:000
104939933,0cyclictest20177-21kworker/0:109:04:000
104939933,0cyclictest20177-21kworker/0:108:58:580
104939933,0cyclictest20177-21kworker/0:108:54:040
104939930,0cyclictest0-21swapper/009:49:090
249920,0watchdog/20-21swapper/209:03:562
249920,0watchdog/20-21swapper/209:03:562
249920,0watchdog/20-21swapper/208:48:562
105119921,0cyclictest3071-13audispd08:58:543
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional