You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-12-06 - 21:54

Intel(R) Xeon(R) CPU E3-1270 V2 @ 3.50GHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack1slot2.osadl.org (updated Mon Dec 06, 2021 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1085723824,0sleep00-21swapper/007:05:130
1082823828,0sleep30-21swapper/307:04:523
1096423425,0sleep10-21swapper/107:06:311
1078323424,0sleep20-21swapper/207:04:232
30535060,0irq/33-eth0-tx-19290-21munin-node07:23:010
30535060,0irq/33-eth0-tx-12746-21snmpd08:03:130
30535060,0irq/33-eth0-tx-0-21swapper/007:13:060
30525060,0irq/32-eth0-rx-0-21swapper/310:53:453
19160,0ktimersoftd/10-21swapper/109:31:161
169960,0watchdog/13595-21python08:53:041
149960,0migration/032293-21sh09:32:160
149960,0migration/029015-21sh11:08:010
149960,0migration/022915-21sh09:55:040
149960,0migration/011612-21sh12:32:090
8305050,0irq/26-0000:00:28307-21munin-node08:38:082
339950,0migration/39698-21sh11:56:143
339950,0migration/39352-21sh11:22:483
339950,0migration/331753-21sh11:12:383
339950,0migration/322569-21runrttasks09:21:233
329950,0watchdog/33645-21sh09:35:083
329950,0watchdog/33518-21munin-node10:08:083
329950,0watchdog/319044-21sh09:51:523
329950,0watchdog/318528-21gltestperf10:57:563
329950,0watchdog/316440-21sh11:28:283
329950,0watchdog/313755-21dump-pmu-power07:12:403
329950,0watchdog/312318-21ntp_kernel_err11:58:083
30535050,0irq/33-eth0-tx-4628-21munin-node08:58:020
30535050,0irq/33-eth0-tx-3128-21ssh11:15:372
30535050,0irq/33-eth0-tx-28109-21ssh12:14:372
30535050,0irq/33-eth0-tx-26432-21munin-node08:37:520
30535050,0irq/33-eth0-tx-25514-21sh09:57:552
30535050,0irq/33-eth0-tx-25332-21ssh10:31:592
30535050,0irq/33-eth0-tx-13256-21ssh11:26:312
30535050,0irq/33-eth0-tx-12257-21perl10:17:532
30535050,0irq/33-eth0-tx-0-21swapper/211:58:532
30535050,0irq/33-eth0-tx-0-21swapper/211:47:452
30535050,0irq/33-eth0-tx-0-21swapper/210:05:472
30535050,0irq/33-eth0-tx-0-21swapper/209:48:012
30535050,0irq/33-eth0-tx-0-21swapper/209:32:512
30535050,0irq/33-eth0-tx-0-21swapper/209:21:332
30525050,0irq/32-eth0-rx-30959-21pmu-power10:03:033
30525050,0irq/32-eth0-rx-23565-21if_err_eth1.311:02:573
30525050,0irq/32-eth0-rx-11405-21ssh10:50:293
30525050,0irq/32-eth0-rx-11405-21ssh10:50:293
30525050,0irq/32-eth0-rx-0-21swapper/310:32:443
30525050,0irq/32-eth0-rx-0-21swapper/307:49:423
27150,0ktimersoftd/20-21swapper/207:14:442
259950,0migration/24577-21unixbench-2d07:53:042
259950,0migration/232468-21taskset12:18:452
249950,0watchdog/218506-21gltestperf10:57:562
179950,0migration/1672-21sh10:06:141
179950,0migration/14963-21taskset07:56:031
179950,0migration/129453-21sh10:35:391
179950,0migration/129229-21latency_hist10:02:401
179950,0migration/119636-21latency_hist09:52:401
179950,0migration/111645-21sh12:32:121
169950,0watchdog/110456-21latency_hist11:57:401
149950,0migration/09601-21taskset11:56:040
149950,0migration/07243-21timerwakeupswit09:38:050
149950,0migration/04798-21taskset07:54:060
149950,0migration/032173-21aten2_r1power_e11:12:570
149950,0migration/030938-21gltestperf12:17:530
149950,0migration/030502-21taskset10:37:370
149950,0migration/026070-21fschecks_time12:13:000
149950,0migration/025286-21sh10:31:530
149950,0migration/023946-21munin-node11:02:590
149950,0migration/023909-21sh09:22:530
149950,0migration/021219-21aten2_r1power_e08:27:510
149950,0migration/018537-21sh09:51:010
149950,0migration/017389-21switchtime12:03:050
149950,0migration/017217-21fschecks_time07:17:540
149950,0migration/017044-21sh11:29:370
149950,0migration/016151-21latency_hist10:22:400
149950,0migration/013827-21fschecks_count08:12:540
149950,0migration/012700-21sh11:25:270
111579955,0cyclictest0-21swapper/312:28:113
111579955,0cyclictest0-21swapper/312:24:433
111579955,0cyclictest0-21swapper/310:43:113
111579955,0cyclictest0-21swapper/310:39:333
111579955,0cyclictest0-21swapper/310:16:513
111579955,0cyclictest0-21swapper/309:39:583
111579955,0cyclictest0-21swapper/307:38:063
111579950,0cyclictest0-21swapper/311:19:173
111579950,0cyclictest0-21swapper/309:58:153
111519953,0cyclictest5053-21kworker/2:011:11:262
111519953,0cyclictest5053-21kworker/2:008:18:522
111409955,0cyclictest0-21swapper/111:21:381
111409955,0cyclictest0-21swapper/111:10:451
111409955,0cyclictest0-21swapper/109:23:511
111409955,0cyclictest0-21swapper/109:18:581
111409954,0cyclictest3640-21lldpd10:12:171
111409950,0cyclictest0-21swapper/112:06:051
111409950,0cyclictest0-21swapper/110:15:341
111409950,0cyclictest0-21swapper/108:10:521
111399955,0cyclictest2654-21nvme07:43:480
8305040,0irq/26-0000:00:0-21swapper/208:02:522
339940,0migration/34064-21taskset08:54:123
339940,0migration/325845-21taskset08:33:433
339940,0migration/323421-21sh09:56:033
339940,0migration/321938-21aten2_r1power_c07:27:523
339940,0migration/317955-21taskset08:19:103
339940,0migration/316908-21runrttasks09:14:303
339940,0migration/315573-21sh12:02:473
339940,0migration/310562-21latency_hist08:07:403
329940,0watchdog/3845-21latency_hist11:47:403
329940,0watchdog/331218-21iostat12:17:563
329940,0watchdog/328537-21wc11:42:403
329940,0watchdog/30-21swapper/312:09:563
329940,0watchdog/30-21swapper/311:38:003
30535040,0irq/33-eth0-tx-9675-21diskmemload11:17:452
30535040,0irq/33-eth0-tx-9675-21diskmemload10:12:552
30535040,0irq/33-eth0-tx-3796-21ssh10:42:572
30535040,0irq/33-eth0-tx-31733-21perl10:38:002
30535040,0irq/33-eth0-tx-19268-21ssh11:32:582
30535040,0irq/33-eth0-tx-0-21swapper/212:32:452
30535040,0irq/33-eth0-tx-0-21swapper/212:08:312
30535040,0irq/33-eth0-tx-0-21swapper/211:37:562
30535040,0irq/33-eth0-tx-0-21swapper/210:33:012
30535040,0irq/33-eth0-tx-0-21swapper/210:22:572
30535040,0irq/33-eth0-tx-0-21swapper/209:52:572
30535040,0irq/33-eth0-tx-0-21swapper/209:42:522
30535040,0irq/33-eth0-tx-0-21swapper/209:37:482
30535040,0irq/33-eth0-tx-0-21swapper/209:22:532
30535040,0irq/33-eth0-tx-0-21swapper/009:07:530
30535040,0irq/33-eth0-tx-0-21swapper/009:02:540
30535040,0irq/33-eth0-tx-0-21swapper/008:52:530
30535040,0irq/33-eth0-tx-0-21swapper/008:47:540
30535040,0irq/33-eth0-tx-0-21swapper/008:32:540
30525040,0irq/32-eth0-rx-22060-21gltestperf08:27:533
30525040,0irq/32-eth0-rx-0-21swapper/311:32:563
30525040,0irq/32-eth0-rx-0-21swapper/310:28:083
30525040,0irq/32-eth0-rx-0-21swapper/309:43:363
30525040,0irq/32-eth0-rx-0-21swapper/309:30:023
30525040,0irq/32-eth0-rx-0-21swapper/309:10:273
259940,0migration/28350-21cpuspeed_turbos10:47:522
259940,0migration/28350-21cpuspeed_turbos10:47:522
259940,0migration/227345-21cpuspeed_turbos07:37:532
259940,0migration/222426-21runrttasks11:02:432
259940,0migration/21976-21runrttasks07:48:092
259940,0migration/215621-21aten2_r1power_c12:02:522
259940,0migration/213983-21latency_hist11:27:402
249940,0watchdog/25804-21latency_hist11:52:402
249940,0watchdog/20-21swapper/212:27:522
249940,0watchdog/20-21swapper/211:42:402
179940,0migration/18605-21missed_timers09:02:581
179940,0migration/16561-21irqstats09:37:571
179940,0migration/130678-21fschecks_time12:17:531
179940,0migration/12381-21sh12:22:431
179940,0migration/114898-21taskset09:47:481
179940,0migration/114819-21fschecks_count11:27:591
179940,0migration/114137-21diskmemload10:53:051
179940,0migration/112208-21sh11:24:321
169940,0watchdog/13252-1kworker/1:2H11:45:121
169940,0watchdog/123864-21munin-node08:32:521
169940,0watchdog/1152572sleep112:34:561
169940,0watchdog/113824-21ssh11:59:521
169940,0watchdog/113752-21munin-run07:12:401
169940,0watchdog/10-21swapper/110:27:401
169940,0watchdog/10-21swapper/109:08:401
149940,0migration/06388-21sh10:12:380
149940,0migration/031959-21sh10:03:590
149940,0migration/030370-21missed_timers08:43:100
149940,0migration/024875-21df_abs07:32:550
149940,0migration/019307-21fschecks_count09:17:540
111579944,0cyclictest0-21swapper/307:18:043
111579943,0cyclictest9672-21sadc09:07:403
111579943,0cyclictest9003-21awk08:02:553
111579943,0cyclictest3640-21lldpd08:59:433
111579943,0cyclictest3640-21lldpd08:23:433
111579943,0cyclictest3640-21lldpd07:58:133
111579943,0cyclictest3640-21lldpd07:55:433
111579943,0cyclictest3640-21lldpd07:42:423
111579943,0cyclictest3640-21lldpd07:34:423
111579943,0cyclictest3640-21lldpd07:23:423
111579943,0cyclictest3322-21irqbalance10:17:473
111579943,0cyclictest32710-21awk08:47:573
111579943,0cyclictest26349-21awk12:13:023
111579943,0cyclictest24671-21awk09:22:573
111579943,0cyclictest17530-21awk10:22:583
111579943,0cyclictest12746-21snmpd12:33:353
111579943,0cyclictest12746-21snmpd08:38:033
111579940,0cyclictest32457-21df_abs11:12:583
111519943,0cyclictest5053-21kworker/2:007:10:102
111519943,0cyclictest3640-21lldpd10:07:472
111519943,0cyclictest3322-21irqbalance08:53:182
111519943,0cyclictest3322-21irqbalance08:48:182
111519943,0cyclictest3322-21irqbalance08:43:082
111519943,0cyclictest3322-21irqbalance08:34:582
111519943,0cyclictest3322-21irqbalance08:24:182
111519943,0cyclictest30834-21awk07:42:562
111519943,0cyclictest20416-21awk07:22:562
111519943,0cyclictest15272-21awk09:13:032
111519943,0cyclictest12746-21snmpd09:28:042
111519943,0cyclictest12746-21snmpd09:08:332
111519943,0cyclictest12746-21snmpd09:03:032
111519943,0cyclictest12746-21snmpd08:59:032
111519943,0cyclictest12746-21snmpd08:27:482
111519943,0cyclictest12746-21snmpd08:13:182
111519943,0cyclictest12746-21snmpd08:11:482
111519943,0cyclictest12746-21snmpd07:58:032
111519943,0cyclictest12746-21snmpd07:32:472
111519943,0cyclictest12746-21snmpd07:22:022
111519940,0cyclictest3201-21fschecks_time12:22:542
111519940,0cyclictest0-21swapper/210:53:022
111409944,0cyclictest0-21swapper/110:22:211
111409944,0cyclictest0-21swapper/109:43:341
111409943,0cyclictest972-21awk07:47:561
111409943,0cyclictest5636-21awk08:58:011
111409943,0cyclictest3640-21lldpd10:43:471
111409943,0cyclictest3640-21lldpd10:38:171
111409943,0cyclictest3640-21lldpd10:27:471
111409943,0cyclictest3640-21lldpd08:49:431
111409943,0cyclictest3640-21lldpd08:44:431
111409943,0cyclictest3640-21lldpd07:43:121
111409943,0cyclictest3640-21lldpd07:17:421
111409943,0cyclictest3322-21irqbalance11:49:171
111409943,0cyclictest3322-21irqbalance09:16:181
111409943,0cyclictest3322-21irqbalance08:13:281
111409943,0cyclictest3322-21irqbalance08:02:481
111409943,0cyclictest3322-21irqbalance07:57:481
111409943,0cyclictest3322-21irqbalance07:37:581
111409943,0cyclictest26374-21awk12:13:031
111409943,0cyclictest24919-21awk11:37:571
111409943,0cyclictest23566-21awk11:02:571
111409943,0cyclictest21483-21awk12:08:001
111409943,0cyclictest20319-21awk07:22:561
111409943,0cyclictest19595-21awk08:22:561
111409943,0cyclictest12746-21snmpd11:33:051
111409943,0cyclictest12746-21snmpd11:15:041
111409943,0cyclictest12746-21snmpd10:57:491
111409943,0cyclictest12746-21snmpd10:48:341
111409943,0cyclictest12746-21snmpd10:48:341
111409943,0cyclictest12746-21snmpd07:33:021
111399944,0cyclictest2654-21nvme12:32:590
111399944,0cyclictest2654-21nvme10:41:350
111399943,0cyclictest3640-21lldpd10:43:170
111399943,0cyclictest3640-21lldpd07:47:430
111399943,0cyclictest28216-21awk07:37:570
111399943,0cyclictest12746-21snmpd12:26:350
111399943,0cyclictest12746-21snmpd12:10:050
111399943,0cyclictest12746-21snmpd11:58:050
111399943,0cyclictest12746-21snmpd11:48:200
111399943,0cyclictest12746-21snmpd11:44:050
111399943,0cyclictest12746-21snmpd11:41:500
111399943,0cyclictest12746-21snmpd11:33:500
111399943,0cyclictest12746-21snmpd10:58:340
111399943,0cyclictest12480-21awk07:07:590
111399943,0cyclictest11736-21awk08:07:580
111399940,0cyclictest0-21swapper/011:21:490
111399940,0cyclictest0-21swapper/010:24:250
30535030,0irq/33-eth0-tx-0-21swapper/008:21:470
169930,0watchdog/10-21swapper/108:37:561
149930,0migration/014280-21sh09:47:040
111579932,0cyclictest29969-21grep08:43:063
111579930,0cyclictest14659-21perl07:12:553
111579930,0cyclictest0-21swapper/308:12:543
111519933,0cyclictest5053-21kworker/2:007:28:042
111409933,0cyclictest3640-21lldpd07:27:421
111409933,0cyclictest0-21swapper/108:29:281
111409932,0cyclictest17505-21sensors08:18:031
111409932,0cyclictest1457-21awk09:32:561
111409932,0cyclictest14092-21egrep07:12:531
111399933,0cyclictest0-21swapper/010:52:580
111399933,0cyclictest0-21swapper/009:59:120
111399932,0cyclictest15288-21awk09:13:030
111399932,0cyclictest12746-21snmpd10:48:040
111399932,0cyclictest12746-21snmpd10:48:040
111399930,0cyclictest5368-21munin-node07:58:080
111399930,0cyclictest32760-21ssh09:32:440
111399930,0cyclictest21914-21munin-node07:28:050
111399930,0cyclictest0-21swapper/010:14:380
111399920,0cyclictest18503-21munin-node08:22:520
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional