You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-22 - 02:39

Intel(R) Xeon(R) CPU E3-1270 V2 @ 3.50GHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Fri Oct 22, 2021 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2378924031,0sleep30-21swapper/319:08:093
2355423728,0sleep10-21swapper/119:05:191
2376723627,0sleep20-21swapper/219:07:522
2382123127,0sleep00-21swapper/019:08:260
23917991313,0cyclictest3596-21snmpd20:05:131
23917991212,0cyclictest2654-21nvme23:19:091
23916991212,0cyclictest3596-21snmpd23:02:210
239179999,0cyclictest2654-21nvme23:58:591
239179999,0cyclictest2654-21nvme00:31:371
239179988,0cyclictest2654-21nvme21:38:241
4160,0ktimersoftd/022453-21diskmemload21:50:450
35160,0ktimersoftd/30-21swapper/323:56:483
339960,0migration/320899-21gltestperf22:49:053
239199966,0cyclictest0-21swapper/322:43:203
239199960,0cyclictest0-21swapper/320:59:173
239189965,0cyclictest3347-21kworker/2:022:52:042
239189965,0cyclictest3347-21kworker/2:021:23:442
239189960,0cyclictest0-21swapper/223:57:192
239189960,0cyclictest0-21swapper/221:11:562
239179966,0cyclictest2654-21nvme22:16:221
239169960,0cyclictest0-21swapper/023:06:350
149960,0migration/022735-21sh00:00:460
8305050,0irq/26-0000:00:17839-21idleruntime-cro19:58:462
8305050,0irq/26-0000:00:17109-21crond20:58:472
339950,0migration/39518-21aten2_r1power_e20:44:013
339950,0migration/323527-21runrttasks23:25:193
339950,0migration/322098-21taskset21:04:553
339950,0migration/321792-21latency00:34:163
339950,0migration/313463-21diskmemload23:51:143
339950,0migration/31261-21sh23:03:203
329950,0watchdog/38912-21sh00:20:143
329950,0watchdog/35373-21tune2fs19:34:023
30535050,0irq/33-eth0-tx-981-21munin-node22:29:152
30535050,0irq/33-eth0-tx-6259-21tune2fs23:44:112
30535050,0irq/33-eth0-tx-5769-21ssh22:33:562
30535050,0irq/33-eth0-tx-20898-21munin-node23:24:032
30535050,0irq/33-eth0-tx-0-21swapper/019:55:530
30535050,0irq/33-eth0-tx-0-21swapper/019:29:050
30525050,0irq/32-eth0-rx-7116-21uname23:44:183
30525050,0irq/32-eth0-rx-3624-21lldpd22:47:163
30525050,0irq/32-eth0-rx-26187-21cpu20:14:023
30525050,0irq/32-eth0-rx-1955-21python21:54:253
30525050,0irq/32-eth0-rx-12036-21sh22:04:413
30525050,0irq/32-eth0-rx-0-21swapper/321:26:243
30525050,0irq/32-eth0-rx-0-21swapper/300:31:513
27150,0ktimersoftd/20-21swapper/222:55:312
259950,0migration/218580-21sh23:20:002
249950,0watchdog/26820-21ssh23:09:062
239199955,0cyclictest0-21swapper/323:06:023
239199955,0cyclictest0-21swapper/322:37:213
239199955,0cyclictest0-21swapper/300:00:543
239199950,0cyclictest0-21swapper/323:20:233
239189955,0cyclictest3347-21kworker/2:023:49:042
239189955,0cyclictest3347-21kworker/2:023:39:282
239189955,0cyclictest3347-21kworker/2:023:34:242
239189955,0cyclictest3347-21kworker/2:023:28:542
239189955,0cyclictest3347-21kworker/2:023:13:542
239189955,0cyclictest3347-21kworker/2:023:04:442
239189955,0cyclictest3347-21kworker/2:022:58:542
239189955,0cyclictest3347-21kworker/2:022:44:142
239189955,0cyclictest3347-21kworker/2:022:14:222
239189955,0cyclictest3347-21kworker/2:022:08:542
239189955,0cyclictest3347-21kworker/2:022:03:562
239189955,0cyclictest3347-21kworker/2:021:58:542
239189955,0cyclictest3347-21kworker/2:021:54:042
239189955,0cyclictest3347-21kworker/2:021:48:562
239189955,0cyclictest3347-21kworker/2:021:38:542
239189955,0cyclictest3347-21kworker/2:021:34:262
239189955,0cyclictest3347-21kworker/2:021:29:082
239189955,0cyclictest3347-21kworker/2:021:24:582
239189955,0cyclictest3347-21kworker/2:021:14:042
239189955,0cyclictest3347-21kworker/2:021:04:082
239189955,0cyclictest3347-21kworker/2:020:54:042
239189955,0cyclictest3347-21kworker/2:020:44:042
239189955,0cyclictest3347-21kworker/2:020:33:502
239189955,0cyclictest3347-21kworker/2:020:29:142
239189955,0cyclictest3347-21kworker/2:020:20:302
239189955,0cyclictest3347-21kworker/2:020:14:042
239189955,0cyclictest3347-21kworker/2:020:08:542
239189955,0cyclictest3347-21kworker/2:020:04:142
239189955,0cyclictest3347-21kworker/2:019:58:542
239189955,0cyclictest3347-21kworker/2:019:52:502
239189955,0cyclictest3347-21kworker/2:019:44:022
239189955,0cyclictest3347-21kworker/2:019:24:122
239189955,0cyclictest3347-21kworker/2:019:19:082
239189955,0cyclictest3347-21kworker/2:019:09:062
239189955,0cyclictest3347-21kworker/2:000:34:242
239189955,0cyclictest3347-21kworker/2:000:30:042
239189955,0cyclictest3347-21kworker/2:000:23:542
239189955,0cyclictest3347-21kworker/2:000:19:242
239189955,0cyclictest3347-21kworker/2:000:14:542
239189955,0cyclictest3347-21kworker/2:000:08:522
239189955,0cyclictest3347-21kworker/2:000:05:242
239189955,0cyclictest3347-21kworker/2:000:05:242
239189955,0cyclictest3347-21kworker/2:000:00:162
239189955,0cyclictest0-21swapper/222:38:482
239189955,0cyclictest0-21swapper/222:18:562
239189954,0cyclictest3347-21kworker/2:022:24:142
239189954,0cyclictest3347-21kworker/2:021:43:482
239179950,0cyclictest22453-21diskmemload21:44:221
239169955,0cyclictest3596-21snmpd21:42:230
239169955,0cyclictest2654-21nvme22:31:020
239169955,0cyclictest2654-21nvme22:24:580
239169955,0cyclictest2654-21nvme21:16:240
239169955,0cyclictest0-21swapper/022:55:230
239169955,0cyclictest0-21swapper/020:19:190
239169954,0cyclictest3596-21snmpd19:11:570
239169950,0cyclictest0-21swapper/022:37:440
179950,0migration/130538-21munin-node20:19:171
179950,0migration/126373-21sh21:12:431
179950,0migration/125387-21mdstat19:09:061
179950,0migration/113309-21taskset00:24:381
169950,0watchdog/130930-21sensors_fan00:09:101
169950,0watchdog/10-21swapper/119:53:461
149950,0migration/09046-21munin-node19:39:100
149950,0migration/06802-21df_abs00:19:130
149950,0migration/06646-21sh21:25:100
149950,0migration/04126-21latency_hist20:33:460
149950,0migration/04029-21sh00:14:530
149950,0migration/031809-21fschecks_count20:24:040
149950,0migration/031307-21taskset23:35:420
149950,0migration/029823-21fschecks_count19:19:040
149950,0migration/028060-21runrttasks00:07:040
149950,0migration/028060-21runrttasks00:07:040
149950,0migration/026767-21sh21:47:240
149950,0migration/022495-21latency_hist21:08:460
149950,0migration/021173-21fschecks_time00:34:110
149950,0migration/016009-21sh22:09:110
149950,0migration/013998-21taskset22:08:350
149950,0migration/011756-21taskset21:30:550
149950,0migration/010831-21munin-node22:39:030
149950,0migration/010533-21cpuspeed_turbos23:48:580
339940,0migration/39023-21taskset20:41:133
339940,0migration/37997-21fschecks_count19:39:033
339940,0migration/37919-21if_eth1.323:09:143
339940,0migration/330906-21munin-node19:19:113
339940,0migration/330330-21if_eth000:09:043
339940,0migration/325980-21missed_timers00:04:113
339940,0migration/325980-21missed_timers00:04:113
339940,0migration/314600-21if_err_eth1.221:34:043
329940,0watchdog/329675-21kernelversion23:34:063
329940,0watchdog/325425-21sed19:09:063
329940,0watchdog/316680-21latency19:54:063
329940,0watchdog/30-21swapper/321:09:103
329940,0watchdog/30-21swapper/320:09:103
329940,0watchdog/30-21swapper/300:13:503
30535040,0irq/33-eth0-tx-0-21swapper/019:49:030
30535040,0irq/33-eth0-tx-0-21swapper/019:34:050
30525040,0irq/32-eth0-rx-25107-21munin-node21:44:113
30525040,0irq/32-eth0-rx-15295-21perl22:09:033
30525040,0irq/32-eth0-rx-0-21swapper/323:39:003
30525040,0irq/32-eth0-rx-0-21swapper/323:29:033
30525040,0irq/32-eth0-rx-0-21swapper/322:24:113
30525040,0irq/32-eth0-rx-0-21swapper/322:13:473
30525040,0irq/32-eth0-rx-0-21swapper/321:38:503
30525040,0irq/32-eth0-rx-0-21swapper/321:13:573
249940,0watchdog/20-21swapper/219:34:022
239199944,0cyclictest0-21swapper/319:28:193
239199943,0cyclictest3596-21snmpd23:13:593
239199943,0cyclictest3596-21snmpd20:53:583
239199943,0cyclictest3596-21snmpd20:34:123
239199943,0cyclictest3596-21snmpd20:25:123
239199943,0cyclictest3596-21snmpd20:21:273
239199943,0cyclictest3596-21snmpd20:04:273
239199943,0cyclictest3596-21snmpd20:00:573
239199943,0cyclictest3322-21irqbalance22:54:043
239199943,0cyclictest3322-21irqbalance22:19:043
239199943,0cyclictest3322-21irqbalance21:58:543
239199943,0cyclictest3322-21irqbalance21:49:443
239199943,0cyclictest3322-21irqbalance21:29:043
239199943,0cyclictest3322-21irqbalance21:19:543
239199943,0cyclictest3322-21irqbalance00:24:543
239199943,0cyclictest2483-21awk20:29:053
239199943,0cyclictest13235-21awk20:49:063
239199942,0cyclictest2024-21awk22:29:093
239199940,0cyclictest0-21swapper/319:31:093
239189944,0cyclictest3347-21kworker/2:020:48:462
239189944,0cyclictest3347-21kworker/2:020:39:022
239189943,0cyclictest3322-21irqbalance20:23:542
239189943,0cyclictest3322-21irqbalance19:38:542
239189943,0cyclictest3322-21irqbalance19:28:542
239189943,0cyclictest3322-21irqbalance19:13:542
239179944,0cyclictest2654-21nvme22:09:191
239179943,0cyclictest8639-21awk19:39:061
239179943,0cyclictest3640-21lldpd20:33:401
239179943,0cyclictest3640-21lldpd00:22:161
239179943,0cyclictest3640-21lldpd00:05:161
239179943,0cyclictest3640-21lldpd00:05:161
239179943,0cyclictest3596-21snmpd23:56:141
239179943,0cyclictest3596-21snmpd23:25:441
239179943,0cyclictest3596-21snmpd23:04:141
239179943,0cyclictest3596-21snmpd22:24:291
239179943,0cyclictest3596-21snmpd22:05:281
239179943,0cyclictest3596-21snmpd20:55:581
239179943,0cyclictest3596-21snmpd20:51:131
239179943,0cyclictest3596-21snmpd20:45:431
239179943,0cyclictest3596-21snmpd20:25:191
239179943,0cyclictest3596-21snmpd20:16:271
239179943,0cyclictest3596-21snmpd20:02:271
239179943,0cyclictest3596-21snmpd19:30:421
239179943,0cyclictest3596-21snmpd19:14:271
239179943,0cyclictest25828-21awk22:54:091
239179943,0cyclictest2555-21awk00:14:051
239179943,0cyclictest11219-21awk19:44:061
239169944,0cyclictest22453-21diskmemload23:10:130
239169944,0cyclictest0-21swapper/023:26:180
239169944,0cyclictest0-21swapper/023:13:500
239169944,0cyclictest0-21swapper/022:21:200
239169944,0cyclictest0-21swapper/022:18:020
239169944,0cyclictest0-21swapper/000:29:020
239169943,0cyclictest3640-21lldpd23:33:160
239169943,0cyclictest3596-21snmpd21:22:130
239169943,0cyclictest3596-21snmpd20:52:280
239169943,0cyclictest3596-21snmpd20:32:270
239169943,0cyclictest3596-21snmpd19:44:270
239169943,0cyclictest10468-21awk20:44:050
239169940,0cyclictest3838-21httpd23:45:160
179940,0migration/17492-21diskmemload22:34:081
179940,0migration/15776-21aten2_r1power_v23:44:061
179940,0migration/15559-21fschecks_count21:59:011
179940,0migration/13325-21taskset23:40:021
179940,0migration/131723-21runrttasks21:18:511
179940,0migration/128678-21fschecks_time21:49:171
179940,0migration/128235-21idleruntime21:14:091
179940,0migration/119009-21rt-features20:59:131
179940,0migration/118471-21cpuspeed_turbos21:39:041
179940,0migration/116217-21munin-node22:44:031
179940,0migration/112720-21latency23:14:141
169940,0watchdog/19618-21sh23:10:541
169940,0watchdog/18640-21sed20:39:141
169940,0watchdog/128850-21ssh23:34:021
169940,0watchdog/126174-21ssh23:29:541
169940,0watchdog/125200-21timerwakeupswit20:09:101
169940,0watchdog/113664-21sh22:41:181
169940,0watchdog/112804-21latency_hist19:48:461
169940,0watchdog/11174-21munin-node19:24:101
169940,0watchdog/111333-21ssh21:30:061
169940,0watchdog/10-21swapper/122:59:541
169940,0watchdog/10-21swapper/122:19:101
169940,0watchdog/10-21swapper/121:25:501
169940,0watchdog/10-21swapper/120:34:261
169940,0watchdog/10-21swapper/119:34:021
149940,0migration/03686-21sh23:40:410
149940,0migration/030547-21load00:09:050
149940,0migration/026651-21sh21:13:180
149940,0migration/019231-21if_eth1.219:59:040
149940,0migration/016332-21diskmemload23:54:110
329930,0watchdog/30-21swapper/319:48:463
329930,0watchdog/30-21swapper/319:43:463
239199933,0cyclictest0-21swapper/319:14:053
239179932,0cyclictest3596-21snmpd21:54:431
239179932,0cyclictest3596-21snmpd21:04:431
239179932,0cyclictest3596-21snmpd19:20:121
239169932,0cyclictest6013-21awk21:59:030
239169932,0cyclictest3640-21lldpd19:17:400
239169932,0cyclictest3596-21snmpd22:44:440
239169932,0cyclictest3596-21snmpd20:40:430
239169932,0cyclictest3596-21snmpd20:08:530
239169932,0cyclictest16358-21sensors20:54:130
239169930,0cyclictest3773-21mysqld23:19:100
239169930,0cyclictest32004-21munin-node19:24:020
239169930,0cyclictest26414-21forks20:14:030
239169930,0cyclictest21810-21munin-node20:04:110
239169930,0cyclictest18698-21missed_timers20:59:090
239169930,0cyclictest0-21swapper/022:48:480
239169930,0cyclictest0-21swapper/021:54:270
239169930,0cyclictest0-21swapper/021:36:300
169930,0watchdog/122453-21diskmemload23:49:021
169930,0watchdog/122453-21diskmemload22:49:341
169930,0watchdog/10-21swapper/122:28:541
169930,0watchdog/10-21swapper/100:34:141
149930,0migration/013434-21runrttasks00:24:570
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional