You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2022-01-23 - 19:03

Intel(R) Xeon(R) CPU E3-1270 V2 @ 3.50GHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rack1slot2.osadl.org (updated Sun Jan 23, 2022 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
496724746,0sleep30-21swapper/307:01:363
704123725,0sleep20-21swapper/207:02:392
701523424,0sleep10-21swapper/107:02:181
719123026,0sleep00-21swapper/007:04:210
7445991717,0cyclictest6650-21kworker/2:211:38:512
7445991515,0cyclictest6650-21kworker/2:211:47:252
7445991515,0cyclictest6650-21kworker/2:210:39:152
7445991515,0cyclictest6650-21kworker/2:209:15:062
7445991515,0cyclictest2654-21nvme08:32:212
7445991515,0cyclictest2654-21nvme08:32:212
7445991313,0cyclictest2654-21nvme08:08:592
7445991212,0cyclictest6650-21kworker/2:212:28:312
7445991212,0cyclictest6650-21kworker/2:209:20:562
7445991111,0cyclictest6650-21kworker/2:210:13:352
7445991111,0cyclictest6650-21kworker/2:209:53:282
7445991111,0cyclictest6650-21kworker/2:208:13:302
7445991010,0cyclictest6650-21kworker/2:208:46:282
7445991010,0cyclictest6650-21kworker/2:208:26:302
7445991010,0cyclictest6650-21kworker/2:207:49:082
7445991010,0cyclictest2654-21nvme11:21:092
7445991010,0cyclictest2654-21nvme09:27:032
74459999,0cyclictest6650-21kworker/2:211:22:572
74459999,0cyclictest6650-21kworker/2:210:43:412
74459999,0cyclictest6650-21kworker/2:210:23:212
74459999,0cyclictest6650-21kworker/2:210:07:352
74459999,0cyclictest2654-21nvme11:15:142
74459988,0cyclictest6650-21kworker/2:207:57:122
74459988,0cyclictest2654-21nvme08:56:572
74459988,0cyclictest2654-21nvme08:05:212
74459977,0cyclictest2654-21nvme11:11:122
74459977,0cyclictest2654-21nvme10:17:222
74459977,0cyclictest2654-21nvme10:01:332
74459977,0cyclictest2654-21nvme09:38:012
74489966,0cyclictest0-21swapper/312:30:353
74489960,0cyclictest0-21swapper/312:07:153
74489960,0cyclictest0-21swapper/312:05:403
74489960,0cyclictest0-21swapper/310:58:403
74489960,0cyclictest0-21swapper/309:06:343
74459966,0cyclictest6650-21kworker/2:211:02:312
74459966,0cyclictest6650-21kworker/2:210:54:312
74459966,0cyclictest6650-21kworker/2:209:33:582
74459966,0cyclictest0-21swapper/210:57:162
74459960,0cyclictest0-21swapper/211:44:062
74449960,0cyclictest0-21swapper/110:54:281
4160,0ktimersoftd/023058-21ssh09:24:490
149960,0migration/019401-21unixbench_singl08:27:060
74489955,0cyclictest0-21swapper/312:22:513
74489955,0cyclictest0-21swapper/311:37:553
74489955,0cyclictest0-21swapper/310:03:483
74489950,0cyclictest0-21swapper/311:57:473
74489950,0cyclictest0-21swapper/311:51:383
74489950,0cyclictest0-21swapper/311:10:103
74489950,0cyclictest0-21swapper/310:12:093
74489950,0cyclictest0-21swapper/309:51:213
74489950,0cyclictest0-21swapper/308:26:313
74459955,0cyclictest6650-21kworker/2:212:16:252
74459955,0cyclictest6650-21kworker/2:211:27:112
74459955,0cyclictest6650-21kworker/2:209:23:402
74459955,0cyclictest6650-21kworker/2:208:26:502
74459955,0cyclictest2654-21nvme07:51:532
74459955,0cyclictest0-21swapper/210:32:592
74449955,0cyclictest5913-21diskmemload11:10:231
74449955,0cyclictest0-21swapper/112:06:211
74449955,0cyclictest0-21swapper/109:23:201
74449950,0cyclictest0-21swapper/111:52:101
74449950,0cyclictest0-21swapper/110:21:401
74449950,0cyclictest0-21swapper/109:41:091
74449950,0cyclictest0-21swapper/109:27:261
74439955,0cyclictest0-21swapper/010:18:160
74439955,0cyclictest0-21swapper/009:46:160
74439950,0cyclictest0-21swapper/010:37:590
74439950,0cyclictest0-21swapper/010:07:050
4150,0ktimersoftd/00-21swapper/011:22:030
4150,0ktimersoftd/00-21swapper/009:35:420
35150,0ktimersoftd/30-21swapper/311:22:033
339950,0migration/328734-21timerandwakeup11:42:053
339950,0migration/323675-21sh11:05:483
339950,0migration/32011-21munin-node11:16:483
339950,0migration/316187-21sh10:24:043
339950,0migration/313579-21sh10:55:083
329950,0watchdog/35857-21gltestperf10:46:503
329950,0watchdog/327988-21sh09:30:103
329950,0watchdog/320852-21sort09:56:303
329950,0watchdog/312815-21smartctl11:26:503
329950,0watchdog/30-21swapper/307:56:583
30535050,0irq/33-eth0-tx-19827-21entropy12:06:492
30535050,0irq/33-eth0-tx-17607-21perl11:31:502
30535050,0irq/33-eth0-tx-14695-21ssh09:49:292
30535050,0irq/33-eth0-tx-0-21swapper/212:23:222
30535050,0irq/33-eth0-tx-0-21swapper/008:53:520
30535050,0irq/33-eth0-tx-0-21swapper/007:41:590
30535050,0irq/33-eth0-tx-0-21swapper/007:21:570
30525050,0irq/32-eth0-rx-8542-21df_inode10:16:493
30525050,0irq/32-eth0-rx-20611-21idleruntime09:21:523
30525050,0irq/32-eth0-rx-11815-21memory09:11:573
259950,0migration/232188-21munin-node08:51:592
259950,0migration/220866-21latency_hist09:56:302
259950,0migration/219320-21munin-node10:26:542
259950,0migration/212220-21munin-node08:16:492
249950,0watchdog/25023-21grep11:51:502
19150,0ktimersoftd/10-21swapper/111:22:031
19150,0ktimersoftd/10-21swapper/111:05:491
179950,0migration/18617-21switchtime08:07:061
179950,0migration/127945-21munin-node10:02:031
179950,0migration/12782-21munin-node08:57:071
179950,0migration/127699-21smart_sda07:41:531
179950,0migration/123263-21fschecks_count08:36:521
179950,0migration/122800-21munin-node10:31:481
179950,0migration/11461-21munin-node11:47:041
179950,0migration/111247-21munin-node11:57:031
179950,0migration/111178-21fschecks_time09:11:501
179950,0migration/110574-21sh09:45:511
179950,0migration/110251-21latency_hist07:11:311
169950,0watchdog/112631-21crond10:21:301
149950,0migration/09295-21munin-node07:06:580
149950,0migration/08296-21sh11:55:080
149950,0migration/06597-21latency_hist11:21:300
149950,0migration/05634-21fschecks_time10:46:500
149950,0migration/05282-21timerandwakeup09:02:020
149950,0migration/03234-21sh10:44:060
149950,0migration/027750-21sh11:09:200
149950,0migration/027151-21timerwakeupswit08:42:070
149950,0migration/025381-21munin-node12:11:520
149950,0migration/023913-21sh11:06:150
149950,0migration/021253-21fschecks_count07:31:460
149950,0migration/018505-21sh11:00:180
149950,0migration/014372-21sh09:48:520
149950,0migration/012117-21latency_hist08:16:310
74489944,0cyclictest0-21swapper/307:16:313
74489943,0cyclictest3640-21lldpd07:47:293
74489943,0cyclictest26381-21awk08:41:573
74489943,0cyclictest23723-21awk08:36:553
74489943,0cyclictest17877-21awk11:31:513
74489943,0cyclictest16636-21awk07:21:533
74489943,0cyclictest16008-21awk09:16:513
74489943,0cyclictest12746-21snmpd09:03:103
74489943,0cyclictest12746-21snmpd08:32:403
74489943,0cyclictest12746-21snmpd08:32:403
74489943,0cyclictest12746-21snmpd08:22:253
74489943,0cyclictest12746-21snmpd08:21:103
74489943,0cyclictest12746-21snmpd08:16:253
74489943,0cyclictest12746-21snmpd08:08:253
74489943,0cyclictest12746-21snmpd08:01:403
74489943,0cyclictest12746-21snmpd07:06:393
74489943,0cyclictest11420-21awk07:11:563
74459943,0cyclictest4524-21awk09:01:522
74459943,0cyclictest3640-21lldpd08:36:372
74459943,0cyclictest12746-21snmpd07:41:542
74459943,0cyclictest12746-21snmpd07:21:392
74459943,0cyclictest12746-21snmpd07:16:392
74449944,0cyclictest2654-21nvme10:41:201
74449944,0cyclictest0-21swapper/110:41:321
74449944,0cyclictest0-21swapper/109:46:401
74449944,0cyclictest0-21swapper/109:31:321
74449943,0cyclictest7327-21awk09:06:511
74449943,0cyclictest5290-21awk08:01:541
74449943,0cyclictest4645-21awk09:01:531
74449943,0cyclictest3640-21lldpd12:09:381
74449943,0cyclictest3640-21lldpd11:19:071
74449943,0cyclictest3640-21lldpd11:13:071
74449943,0cyclictest3640-21lldpd09:54:061
74449943,0cyclictest3640-21lldpd07:40:591
74449943,0cyclictest3322-21irqbalance08:41:581
74449943,0cyclictest12746-21snmpd10:46:561
74449943,0cyclictest12746-21snmpd07:18:091
74439944,0cyclictest0-21swapper/012:24:050
74439944,0cyclictest0-21swapper/011:31:350
74439944,0cyclictest0-21swapper/009:16:190
74439943,0cyclictest4441-21awk10:11:550
74439943,0cyclictest3640-21lldpd11:56:370
74439943,0cyclictest3640-21lldpd10:22:360
74439943,0cyclictest3640-21lldpd09:07:060
74439943,0cyclictest23055-21awk11:36:540
74439943,0cyclictest19233-21awk07:26:520
74439943,0cyclictest1800-21awk08:56:540
74439943,0cyclictest12901-21awk11:26:510
74439943,0cyclictest12746-21snmpd12:31:560
74439943,0cyclictest12746-21snmpd12:29:110
74439943,0cyclictest12746-21snmpd11:46:560
74439943,0cyclictest12746-21snmpd11:42:560
74439943,0cyclictest12746-21snmpd11:12:560
74439943,0cyclictest12746-21snmpd10:53:110
74439943,0cyclictest12746-21snmpd10:33:560
74439943,0cyclictest12746-21snmpd09:54:100
74439943,0cyclictest12746-21snmpd09:28:100
74439943,0cyclictest12746-21snmpd08:50:100
74439943,0cyclictest12746-21snmpd08:39:250
74439943,0cyclictest12746-21snmpd08:31:550
74439943,0cyclictest12746-21snmpd08:31:550
74439943,0cyclictest12746-21snmpd08:21:550
74439943,0cyclictest12746-21snmpd08:09:550
74439943,0cyclictest12746-21snmpd08:02:250
74439943,0cyclictest12746-21snmpd08:00:240
74439943,0cyclictest12746-21snmpd07:52:240
74439943,0cyclictest12746-21snmpd07:37:240
74439943,0cyclictest11371-21awk07:11:560
339940,0migration/331730-21kernelversion08:51:543
339940,0migration/329159-21sh09:31:493
339940,0migration/324457-21munin-node07:36:463
339940,0migration/319646-21runrttasks07:26:583
339940,0migration/31284-21fschecks_count08:56:513
329940,0watchdog/321617-21fschecks_time07:31:463
329940,0watchdog/314565-21ssh12:32:463
329940,0watchdog/30-21swapper/310:41:583
329940,0watchdog/30-21swapper/309:41:303
30535040,0irq/33-eth0-tx-0-21swapper/212:31:582
30535040,0irq/33-eth0-tx-0-21swapper/212:16:582
30535040,0irq/33-eth0-tx-0-21swapper/210:46:532
30535040,0irq/33-eth0-tx-0-21swapper/209:41:572
30535040,0irq/33-eth0-tx-0-21swapper/209:06:542
30525040,0irq/32-eth0-rx-0-21swapper/312:18:193
30525040,0irq/32-eth0-rx-0-21swapper/312:12:313
30525040,0irq/32-eth0-rx-0-21swapper/311:46:493
30525040,0irq/32-eth0-rx-0-21swapper/311:14:553
30525040,0irq/32-eth0-rx-0-21swapper/310:38:023
30525040,0irq/32-eth0-rx-0-21swapper/310:32:173
30525040,0irq/32-eth0-rx-0-21swapper/310:27:033
30525040,0irq/32-eth0-rx-0-21swapper/310:11:063
30525040,0irq/32-eth0-rx-0-21swapper/309:51:403
30525040,0irq/32-eth0-rx-0-21swapper/309:38:123
30525040,0irq/32-eth0-rx-0-21swapper/307:52:023
259940,0migration/27733-21munin-node07:06:472
259940,0migration/224340-21gltestperf07:36:442
259940,0migration/214180-21latency_hist12:01:312
259940,0migration/210907-21munin-node11:56:572
249940,0watchdog/221616-21fschecks_time07:31:462
179940,0migration/13638-21taskset07:57:351
179940,0migration/130496-21timerwakeupswit07:46:561
179940,0migration/124705-21munin-node12:11:501
179940,0migration/117788-21taskset07:25:131
179940,0migration/115752-21fschecks_time09:16:501
179940,0migration/115395-21fschecks_time08:21:511
179940,0migration/114451-21latency_hist10:56:301
179940,0migration/111628-21taskset08:13:201
169940,0watchdog/14356-21rm12:22:221
169940,0watchdog/131621-21perl10:06:501
169940,0watchdog/126196-21ssh11:41:021
169940,0watchdog/119288-21missed_timers10:26:541
169940,0watchdog/119103-21python08:27:021
169940,0watchdog/113967-21users11:27:021
169940,0watchdog/10-21swapper/112:34:541
169940,0watchdog/10-21swapper/112:26:381
169940,0watchdog/10-21swapper/109:56:301
149940,0migration/029657-21taskset10:05:070
149940,0migration/029323-21diskmemload12:16:400
149940,0migration/023197-21runrttasks09:57:060
74489930,0cyclictest29098-21iostat_ios08:46:573
74459933,0cyclictest6650-21kworker/2:208:46:522
74459932,0cyclictest12746-21snmpd07:26:392
74459932,0cyclictest10409-21egrep07:11:502
74449932,0cyclictest3640-21lldpd11:41:381
74449932,0cyclictest3640-21lldpd11:31:381
74449932,0cyclictest3640-21lldpd08:53:061
74449932,0cyclictest20132-21tr08:31:471
74449932,0cyclictest20132-21tr08:31:471
74439933,0cyclictest0-21swapper/007:46:460
74439932,0cyclictest20591-21awk12:06:530
74439932,0cyclictest15324-21gltestperf12:01:530
74439930,0cyclictest2684-21interrupts09:36:560
74439930,0cyclictest18504-21perl10:26:500
74439930,0cyclictest15948-21basename09:16:510
74439930,0cyclictest10362-21perl08:11:520
74439930,0cyclictest0-21swapper/007:17:030
339930,0migration/328019-21unixbench_singl07:41:563
179930,0migration/17648-21runrttasks07:06:361
179930,0migration/128287-21df_abs08:46:511
179930,0migration/120328-21runrttasks07:28:341
169930,0watchdog/10-21swapper/112:16:341
169930,0watchdog/10-21swapper/110:11:541
169930,0watchdog/10-21swapper/108:16:341
74449920,0cyclictest0-21swapper/107:31:311
169920,0watchdog/10-21swapper/107:51:341
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional