You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-06-06 - 14:36

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2.osadl.org (updated Tue Jun 06, 2023 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8277995353,0cyclictest27563-21md0_resync08:13:522
8274993434,0cyclictest27123-21kworker/0:208:13:520
8274991919,0cyclictest14208-21sensors09:16:580
83150120,0irq/26-0000:00:0-21swapper/306:07:173
81721211,0sleep20-21swapper/206:06:302
28015090,0irq/29-eth0-tx-6111-21avahi-daemon06:06:510
179990,0migration/11667-21fschecks_count06:06:471
82779976,0cyclictest24987-21kworker/2:011:18:342
8315060,0irq/26-0000:00:27410-21kthreadcore11:11:533
8315060,0irq/26-0000:00:23778-21kthreadcore11:26:523
8315060,0irq/26-0000:00:22009-21load08:27:173
8315060,0irq/26-0000:00:2069-21munin-node06:52:143
8315060,0irq/26-0000:00:14425-21ssh11:03:053
8315060,0irq/26-0000:00:14380-21crond08:06:293
8315060,0irq/26-0000:00:0-21swapper/307:57:173
82779960,0cyclictest0-21swapper/208:19:512
82759966,0cyclictest0-21swapper/110:53:131
82759966,0cyclictest0-21swapper/109:13:251
82759960,0cyclictest6111-21avahi-daemon08:11:181
82759960,0cyclictest0-21swapper/107:01:271
82749965,0cyclictest28352-21ps06:46:490
82749965,0cyclictest24591-21taskset07:49:250
82749960,0cyclictest0-21swapper/009:55:320
82749960,0cyclictest0-21swapper/007:59:020
82749960,0cyclictest0-21swapper/007:02:450
4160,0ktimersoftd/027563-21md0_resync09:15:390
339960,0migration/35873-21sh09:13:033
339960,0migration/35214-21sh09:48:203
339960,0migration/328004-21kthreadcore08:31:543
339960,0migration/32323-21sh08:54:193
339960,0migration/319889-21sh10:15:563
339960,0migration/319889-21sh10:15:563
339960,0migration/316543-21sh11:40:363
339960,0migration/314683-21sh09:54:063
339960,0migration/313500-21gltestperf06:36:513
339960,0migration/310563-21sh11:18:573
28005060,0irq/28-eth0-rx-24057-21kthreadcore10:16:563
149960,0migration/06486-21sh09:51:110
149960,0migration/025954-21munin-node08:31:520
149960,0migration/020024-21timerandwakeup07:22:110
149960,0migration/017842-21kthreadcore06:16:580
149960,0migration/015103-21unixbench_singl06:12:050
149960,0migration/011649-21cpuspeed_turbos10:46:470
8315050,0irq/26-0000:00:9606-21if_eth1.306:11:513
8315050,0irq/26-0000:00:8951-21kthreadcore08:57:073
8315050,0irq/26-0000:00:31662-21munin-node07:52:053
8315050,0irq/26-0000:00:27277-21aten2_r1power_v06:46:443
8315050,0irq/26-0000:00:26339-21diskstats09:26:493
8315050,0irq/26-0000:00:24430-21ssh11:09:293
8315050,0irq/26-0000:00:21249-21kthreadcore07:46:523
8315050,0irq/26-0000:00:19434-21kthreadcore08:06:553
8315050,0irq/26-0000:00:1918-21sh10:06:323
8315050,0irq/26-0000:00:1739-21grep08:36:523
8315050,0irq/26-0000:00:17204-21df_inode09:21:483
8315050,0irq/26-0000:00:16633-21grep08:42:053
8315050,0irq/26-0000:00:12665-21ssh09:01:353
8315050,0irq/26-0000:00:10856-21diskstats07:41:543
8315050,0irq/26-0000:00:0-21swapper/311:31:463
8315050,0irq/26-0000:00:0-21swapper/310:46:483
8315050,0irq/26-0000:00:0-21swapper/310:21:523
8315050,0irq/26-0000:00:0-21swapper/310:01:473
8315050,0irq/26-0000:00:0-21swapper/307:36:483
8315050,0irq/26-0000:00:0-21swapper/307:21:523
8315050,0irq/26-0000:00:0-21swapper/307:11:573
8315050,0irq/26-0000:00:0-21swapper/306:56:463
82779955,0cyclictest6111-21avahi-daemon09:45:372
82779955,0cyclictest6111-21avahi-daemon08:22:322
82779955,0cyclictest24987-21kworker/2:009:10:202
82759955,0cyclictest6111-21avahi-daemon11:11:021
82759955,0cyclictest6111-21avahi-daemon10:30:401
82759955,0cyclictest6111-21avahi-daemon09:58:111
82759955,0cyclictest6111-21avahi-daemon08:47:431
82759955,0cyclictest6111-21avahi-daemon08:15:331
82759955,0cyclictest6111-21avahi-daemon07:03:021
82759955,0cyclictest0-21swapper/110:43:451
82759950,0cyclictest6111-21avahi-daemon08:19:491
82749955,0cyclictest0-21swapper/009:26:510
82749955,0cyclictest0-21swapper/009:10:460
82749955,0cyclictest0-21swapper/007:30:480
82749955,0cyclictest0-21swapper/006:22:040
82749952,0cyclictest3070-21irqbalance11:39:240
82749950,0cyclictest0-21swapper/010:01:320
82749950,0cyclictest0-21swapper/007:10:090
82749950,0cyclictest0-21swapper/006:29:380
339950,0migration/37879-21kthreadcore06:31:583
339950,0migration/36634-21df_abs07:16:463
339950,0migration/36194-21taskset10:26:313
339950,0migration/33768-21idleruntime09:31:533
339950,0migration/330058-21runrttasks10:56:473
339950,0migration/326031-21kthreadcore07:07:063
339950,0migration/325449-21kthreadcore08:11:593
339950,0migration/324892-21kthreadcore06:21:573
339950,0migration/321946-21kthreadcore06:42:103
339950,0migration/321731-21munin-node10:51:483
339950,0migration/318508-21kthreadcore06:16:583
339950,0migration/317793-21munin-node07:01:523
339950,0migration/313865-21munin-node08:22:053
339950,0migration/312552-21munin-node09:36:513
339950,0migration/311943-21latency_hist11:21:273
329950,0watchdog/330400-21cut06:26:493
329950,0watchdog/329253-21awk07:32:083
329950,0watchdog/31857-21ssh10:41:393
329950,0watchdog/30-21swapper/307:31:283
28015050,0irq/29-eth0-tx-923-21kthreadcore10:21:572
28015050,0irq/29-eth0-tx-685-21tr09:11:542
28015050,0irq/29-eth0-tx-6428-21grep08:57:062
28015050,0irq/29-eth0-tx-22096-21grep08:46:552
28015050,0irq/29-eth0-tx-19296-21chrt10:49:202
28015050,0irq/29-eth0-tx-1188-21munin-node08:36:592
28015050,0irq/29-eth0-tx-0-21swapper/211:11:542
28015050,0irq/29-eth0-tx-0-21swapper/209:50:542
28015050,0irq/29-eth0-tx-0-21swapper/209:34:462
28015050,0irq/29-eth0-tx-0-21swapper/208:31:372
28015050,0irq/29-eth0-tx-0-21swapper/008:01:580
28015050,0irq/29-eth0-tx-0-21swapper/007:45:170
28005050,0irq/28-eth0-rx-25729-21kthreadcore09:06:583
28005050,0irq/28-eth0-rx-23226-21cut09:42:053
28005050,0irq/28-eth0-rx-0-21swapper/310:36:353
28005050,0irq/28-eth0-rx-0-21swapper/310:31:323
28005050,0irq/28-eth0-rx-0-21swapper/309:57:403
28005050,0irq/28-eth0-rx-0-21swapper/309:19:583
28005050,0irq/28-eth0-rx-0-21swapper/308:46:543
259950,0migration/22040-21latency_hist06:51:292
259950,0migration/217676-21hddtemp_smartct07:01:502
259950,0migration/210543-21kthreadcore09:51:562
249950,0watchdog/231200-21perl06:26:532
249950,0watchdog/222072-21kthreadcore10:16:562
19150,0ktimersoftd/16111-21avahi-daemon06:45:321
179950,0migration/19162-21taskset06:55:581
179950,0migration/13893-21sh10:06:551
179950,0migration/131829-21kthreadcore10:56:551
179950,0migration/131756-21taskset07:52:471
179950,0migration/129534-21sh11:29:301
179950,0migration/114756-21sendmail_mailst06:12:001
179950,0migration/114187-21taskset11:02:351
179950,0migration/111419-21sh11:20:411
169950,0watchdog/119477-21sed09:02:041
169950,0watchdog/10-21swapper/107:31:281
149950,0migration/06185-21kthreadcore07:36:520
149950,0migration/0578-21kthreadcore07:11:580
149950,0migration/05769-21kthreadcore08:57:060
149950,0migration/0321-21sh10:39:060
149950,0migration/029720-21runrttasks11:29:520
149950,0migration/029146-21kthreadcore08:51:530
149950,0migration/027270-21kthreadcore07:51:550
149950,0migration/025269-21sh11:11:110
149950,0migration/024334-21runrttasks09:23:390
149950,0migration/023211-21kthreadcore09:42:050
149950,0migration/021604-21kthreadcore08:46:550
149950,0migration/01412-21taskset11:14:380
149950,0migration/013518-21irqcore11:21:530
149950,0migration/012838-21cpuspeed_turbos06:36:500
149950,0migration/011573-21cpuspeed_turbos10:11:490
149950,0migration/011573-21cpuspeed_turbos10:11:490
149950,0migration/010999-21gltestperf08:41:520
82779944,0cyclictest28000-21kworker/2:106:16:292
82779944,0cyclictest24987-21kworker/2:010:01:522
82779944,0cyclictest0-21swapper/209:18:492
82779943,0cyclictest3346-21snmpd09:28:272
82779943,0cyclictest3346-21snmpd08:02:272
82779943,0cyclictest21658-21sadc08:11:292
82779943,0cyclictest14661-21kthreadcore10:11:572
82779943,0cyclictest14661-21kthreadcore10:11:572
82779943,0cyclictest14023-21lldpd11:31:292
82779943,0cyclictest14023-21lldpd07:38:032
82779943,0cyclictest14023-21lldpd06:45:592
82779943,0cyclictest14023-21lldpd06:40:372
82779943,0cyclictest14023-21lldpd06:26:202
82759944,0cyclictest6111-21avahi-daemon09:51:061
82759944,0cyclictest6111-21avahi-daemon08:59:131
82759944,0cyclictest1917-21md0_raid110:17:161
82759944,0cyclictest0-21swapper/108:29:541
82759943,0cyclictest3346-21snmpd08:05:121
82759943,0cyclictest3346-21snmpd06:16:561
82759943,0cyclictest32417-21awk07:11:561
82759943,0cyclictest3070-21irqbalance11:40:041
82759943,0cyclictest3070-21irqbalance10:14:541
82759943,0cyclictest3070-21irqbalance10:14:541
82759943,0cyclictest3070-21irqbalance10:05:541
82759943,0cyclictest3070-21irqbalance08:55:151
82759943,0cyclictest3070-21irqbalance07:08:151
82759943,0cyclictest25993-21awk10:36:531
82759943,0cyclictest13225-21awk11:21:511
82749944,0cyclictest0-21swapper/010:33:510
82749944,0cyclictest0-21swapper/008:22:310
82749944,0cyclictest0-21swapper/008:18:160
82749943,0cyclictest3070-21irqbalance11:34:340
82749943,0cyclictest3070-21irqbalance11:04:040
82749943,0cyclictest3070-21irqbalance10:24:040
82749943,0cyclictest3070-21irqbalance10:17:040
82749943,0cyclictest3070-21irqbalance09:01:550
82749943,0cyclictest3070-21irqbalance08:30:450
82749943,0cyclictest3070-21irqbalance07:19:250
82749943,0cyclictest3070-21irqbalance06:57:350
82749943,0cyclictest2986-21awk10:41:510
82749943,0cyclictest2386-21grep08:36:560
82749943,0cyclictest14023-21lldpd09:33:540
329940,0watchdog/30-21swapper/308:16:323
28015040,0irq/29-eth0-tx-6111-21avahi-daemon10:06:552
28015040,0irq/29-eth0-tx-27563-21md0_resync10:51:302
28015040,0irq/29-eth0-tx-21465-21ssh11:26:342
28015040,0irq/29-eth0-tx-1917-21md0_raid110:36:352
28015040,0irq/29-eth0-tx-16781-21ssh09:21:402
28015040,0irq/29-eth0-tx-148852sleep211:04:022
28015040,0irq/29-eth0-tx-12807-21cat10:27:022
28015040,0irq/29-eth0-tx-11916-21ssh09:36:442
28015040,0irq/29-eth0-tx-0-21swapper/211:37:382
28015040,0irq/29-eth0-tx-0-21swapper/211:24:092
28015040,0irq/29-eth0-tx-0-21swapper/211:07:282
28015040,0irq/29-eth0-tx-0-21swapper/210:56:472
28015040,0irq/29-eth0-tx-0-21swapper/210:41:272
28015040,0irq/29-eth0-tx-0-21swapper/210:32:222
28015040,0irq/29-eth0-tx-0-21swapper/209:56:292
28015040,0irq/29-eth0-tx-0-21swapper/209:01:342
28015040,0irq/29-eth0-tx-0-21swapper/208:51:312
28015040,0irq/29-eth0-tx-0-21swapper/208:41:572
28015040,0irq/29-eth0-tx-0-21swapper/208:26:592
28015040,0irq/29-eth0-tx-0-21swapper/008:06:410
28015040,0irq/29-eth0-tx-0-21swapper/006:55:050
259940,0migration/27558-21kthreadcore06:31:582
259940,0migration/227567-21munin-node06:46:462
249940,0watchdog/29988-21irqstats06:11:532
249940,0watchdog/217830-21aten2_r1power_p07:46:442
249940,0watchdog/212198-21latency07:16:532
249940,0watchdog/211632-21iostat07:42:002
249940,0watchdog/20-21swapper/207:57:082
179940,0migration/19089-21sh08:39:411
179940,0migration/16919-21runrttasks08:00:061
179940,0migration/125074-21kthreadcore09:06:571
179940,0migration/123374-21timerandwakeup09:22:021
179940,0migration/118897-21munin-node07:46:501
179940,0migration/116890-21timerandwakeup07:42:121
179940,0migration/114044-21irqrtprio06:36:541
169940,0watchdog/16363-21runrttasks09:51:321
169940,0watchdog/16159-21ssh11:33:351
169940,0watchdog/13721-21hddtemp_smartct09:31:521
169940,0watchdog/125905-21sh09:26:401
169940,0watchdog/116851-21cut07:22:001
169940,0watchdog/10-21swapper/110:21:561
169940,0watchdog/10-21swapper/108:41:561
149940,0migration/04475-21kthreadcore11:16:520
149940,0migration/030030-21kthreadcore07:32:100
82779932,0cyclictest14762-21awk07:21:592
82759933,0cyclictest6111-21avahi-daemon06:46:381
82759933,0cyclictest0-21swapper/106:23:091
82759932,0cyclictest3346-21snmpd07:36:421
82759932,0cyclictest30827-21cut11:11:541
82759932,0cyclictest3070-21irqbalance10:32:141
82759932,0cyclictest3070-21irqbalance06:26:451
82759932,0cyclictest29214-21awk07:32:081
82759932,0cyclictest22161-21awk09:42:021
82759930,0cyclictest11416-21munin-node10:46:461
82749932,0cyclictest6235-21ssh10:26:350
82749932,0cyclictest4044-21pmu-power10:57:000
82749932,0cyclictest3140-21awk10:06:530
82749932,0cyclictest3070-21irqbalance09:36:540
82749932,0cyclictest3070-21irqbalance06:31:450
82749932,0cyclictest25162-21cut10:51:520
82749930,0cyclictest22389-21port_sshpriv09:56:550
249930,0watchdog/26111-21avahi-daemon07:31:282
249930,0watchdog/26111-21avahi-daemon07:06:532
249930,0watchdog/20-21swapper/207:33:562
249930,0watchdog/20-21swapper/206:56:492
169930,0watchdog/16111-21avahi-daemon09:17:001
169930,0watchdog/16111-21avahi-daemon08:36:081
169930,0watchdog/16111-21avahi-daemon08:21:521
169930,0watchdog/10-21swapper/107:16:291
169930,0watchdog/10-21swapper/106:31:531
82749920,0cyclictest0-21swapper/006:41:290
249920,0watchdog/26111-21avahi-daemon07:51:362
249920,0watchdog/20-21swapper/207:11:292
169920,0watchdog/16111-21avahi-daemon09:36:441
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional