You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-05-28 - 21:54

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Sun May 28, 2023 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1593499130,0cyclictest0-21swapper/008:13:000
1699100,0watchdog/16111-21avahi-daemon06:08:261
28015090,0irq/29-eth0-tx-0-21swapper/006:08:080
28005090,0irq/28-eth0-rx-0-21swapper/306:07:483
259990,0migration/214199-21kthreadcore06:07:562
159359976,0cyclictest0-21swapper/108:19:361
159349974,0cyclictest8305-21turbostat07:37:340
8315060,0irq/26-0000:00:3033-13audispd06:57:343
8315060,0irq/26-0000:00:26574-21perl10:32:573
8315060,0irq/26-0000:00:11579-21latency_hist10:07:343
339960,0migration/3936-21sh09:25:123
339960,0migration/35764-21sh09:45:193
339960,0migration/328507-21sh09:04:523
339960,0migration/319819-21sh10:45:173
339960,0migration/311135-21sh09:31:523
339960,0migration/310891-21sh08:21:403
339960,0migration/310725-21runrttasks10:40:333
28005060,0irq/28-eth0-rx-22178-21perl11:22:543
28005060,0irq/28-eth0-rx-18115-21grep08:43:003
28005060,0irq/28-eth0-rx-12179-21df_abs09:32:583
28005060,0irq/28-eth0-rx-0-21swapper/309:42:013
159409960,0cyclictest26059-21diskmemload10:39:422
159409960,0cyclictest0-21swapper/208:48:522
159359960,0cyclictest0-21swapper/111:28:001
159349965,0cyclictest16783-21head06:57:500
149960,0migration/026350-21fschecks_time08:47:530
149960,0migration/024429-21taskset10:30:220
8315050,0irq/26-0000:00:18929-21grep08:07:343
8315050,0irq/26-0000:00:0-21swapper/306:43:033
339950,0migration/39523-21kthreadcore11:33:013
339950,0migration/39398-21munin-node10:03:143
339950,0migration/38067-21munin-node09:12:553
339950,0migration/37586-21kthreadcore06:27:573
339950,0migration/37005-21sh08:37:423
339950,0migration/332720-21kthreadcore10:53:023
339950,0migration/329428-21sh10:16:193
339950,0migration/329203-21kthreadcore09:57:573
339950,0migration/325963-21kthreadcore07:02:573
339950,0migration/324836-21kthreadcore07:48:013
339950,0migration/323934-21fschecks_time06:17:543
339950,0migration/321534-21kthreadcore07:22:583
339950,0migration/321085-21latency_hist10:47:333
339950,0migration/320835-21runrttasks08:27:453
339950,0migration/319310-21kthreadcore11:37:593
339950,0migration/318222-21kthreadcore06:12:523
339950,0migration/317611-21missed_timers08:57:583
339950,0migration/310364-21kthreadcore07:38:173
329950,0watchdog/32655-21if_eth1.206:47:583
329950,0watchdog/319667-21kthreadcore11:03:053
329950,0watchdog/31169-21kthreadcore07:08:063
329950,0watchdog/311006-21ntpq07:13:023
329950,0watchdog/30-21swapper/311:07:493
329950,0watchdog/30-21swapper/310:28:213
28015050,0irq/29-eth0-tx-7110-21munin-node08:38:112
28015050,0irq/29-eth0-tx-5255-21munin-node06:27:520
28015050,0irq/29-eth0-tx-494-21cat09:08:002
28015050,0irq/29-eth0-tx-30922-21fschecks_count09:42:542
28015050,0irq/29-eth0-tx-27768-21cat11:07:562
28015050,0irq/29-eth0-tx-25447-21ssh11:07:152
28015050,0irq/29-eth0-tx-22624-21ssh09:18:082
28015050,0irq/29-eth0-tx-21782-21perl09:37:542
28015050,0irq/29-eth0-tx-19435-21wget07:22:510
28015050,0irq/29-eth0-tx-11672-21munin-node08:23:012
28015050,0irq/29-eth0-tx-0-21swapper/210:56:292
28015050,0irq/29-eth0-tx-0-21swapper/209:54:112
28015050,0irq/29-eth0-tx-0-21swapper/209:33:082
28015050,0irq/29-eth0-tx-0-21swapper/208:45:192
28015050,0irq/29-eth0-tx-0-21swapper/008:08:040
28015050,0irq/29-eth0-tx-0-21swapper/007:27:500
28015050,0irq/29-eth0-tx-0-21swapper/007:08:010
28015050,0irq/29-eth0-tx-0-21swapper/006:13:000
28005050,0irq/28-eth0-rx-5234-21cat11:12:573
28005050,0irq/28-eth0-rx-31802-21kthreadcore09:08:003
28005050,0irq/28-eth0-rx-17755-21ntp_kernel_err08:23:053
28005050,0irq/28-eth0-rx-17461-21munin-node09:17:583
28005050,0irq/28-eth0-rx-0-21swapper/310:58:013
28005050,0irq/28-eth0-rx-0-21swapper/310:22:593
28005050,0irq/28-eth0-rx-0-21swapper/310:17:533
28005050,0irq/28-eth0-rx-0-21swapper/310:17:533
28005050,0irq/28-eth0-rx-0-21swapper/309:52:403
28005050,0irq/28-eth0-rx-0-21swapper/309:48:303
28005050,0irq/28-eth0-rx-0-21swapper/308:52:563
28005050,0irq/28-eth0-rx-0-21swapper/308:51:293
28005050,0irq/28-eth0-rx-0-21swapper/308:32:413
28005050,0irq/28-eth0-rx-0-21swapper/308:12:583
259950,0migration/27740-21munin-node10:22:532
259950,0migration/229496-21switchtime06:18:062
259950,0migration/225957-21munin-node09:57:512
259950,0migration/223663-21fschecks_time07:47:562
259950,0migration/220554-21taskset11:21:262
259950,0migration/219726-21sh10:45:062
259950,0migration/213892-21kthreadcore08:03:042
259950,0migration/213430-21irqrtprio10:07:592
259950,0migration/210253-21sh09:30:032
259950,0migration/210156-21sh08:55:102
259950,0migration/210067-21munin-node07:38:162
249950,0watchdog/26225-21irqcore07:12:582
179950,0migration/18715-21idleruntime11:32:581
179950,0migration/13991-21munin-node08:33:011
179950,0migration/130929-21fschecks_count06:22:561
179950,0migration/126544-21aten2_r1power_c07:27:501
179950,0migration/123146-21latency_hist06:17:341
179950,0migration/122226-21sh10:47:531
179950,0migration/118072-21switchtime08:58:041
169950,0watchdog/10-21swapper/107:52:341
159409955,0cyclictest6111-21avahi-daemon10:58:572
159409955,0cyclictest6111-21avahi-daemon10:29:472
159409955,0cyclictest6111-21avahi-daemon10:03:272
159409955,0cyclictest6111-21avahi-daemon09:13:282
159409955,0cyclictest0-21swapper/210:33:052
159409955,0cyclictest0-21swapper/208:14:022
159409950,0cyclictest0-21swapper/211:14:522
159359955,0cyclictest6111-21avahi-daemon09:57:501
159359955,0cyclictest6111-21avahi-daemon09:08:571
159359955,0cyclictest6111-21avahi-daemon08:51:181
159359955,0cyclictest6111-21avahi-daemon06:18:031
159359955,0cyclictest0-21swapper/111:23:361
159359955,0cyclictest0-21swapper/111:05:281
159359955,0cyclictest0-21swapper/109:06:161
159359954,0cyclictest20554-21tr07:22:571
149950,0migration/09623-21taskset08:54:050
149950,0migration/09343-21kthreadcore09:12:580
149950,0migration/09196-21kthreadcore10:58:010
149950,0migration/07242-21sh11:32:220
149950,0migration/03473-21fschecks_time10:37:550
149950,0migration/028730-21kthreadcore06:43:080
149950,0migration/027537-21kthreadcore10:33:020
149950,0migration/024711-21kthreadcore07:02:560
149950,0migration/023417-21runrttasks08:43:590
149950,0migration/019217-21taskset11:03:050
149950,0migration/018106-21taskset09:33:160
149950,0migration/017038-21irqcore07:43:000
149950,0migration/016335-21sh09:17:380
149950,0migration/015465-21sh11:35:180
149950,0migration/012959-21irqcore08:57:550
149950,0migration/012514-21fschecks_count08:02:590
149950,0migration/010575-21sh11:15:040
8315040,0irq/26-0000:00:0-21swapper/308:07:573
339940,0migration/313928-21kthreadcore07:18:003
339940,0migration/311149-21timerandwakeup07:58:053
329940,0watchdog/3903-21ksmtuned07:31:063
329940,0watchdog/30-21swapper/307:52:343
329940,0watchdog/30-21swapper/307:32:583
28015040,0irq/29-eth0-tx-0-21swapper/211:37:392
28015040,0irq/29-eth0-tx-0-21swapper/211:33:342
28015040,0irq/29-eth0-tx-0-21swapper/211:28:132
28015040,0irq/29-eth0-tx-0-21swapper/211:22:472
28015040,0irq/29-eth0-tx-0-21swapper/209:47:392
28015040,0irq/29-eth0-tx-0-21swapper/209:02:382
28015040,0irq/29-eth0-tx-0-21swapper/208:27:472
28015040,0irq/29-eth0-tx-0-21swapper/006:47:550
28005040,0irq/28-eth0-rx-0-21swapper/311:27:373
28005040,0irq/28-eth0-rx-0-21swapper/311:17:393
259940,0migration/227566-21munin-node07:27:552
259940,0migration/227395-21fschecks_time06:43:002
259940,0migration/214626-21missed_timers06:52:582
249940,0watchdog/221160-21ssh10:47:332
249940,0watchdog/218420-21ntp_states06:33:022
249940,0watchdog/216674-21fschecks_time06:57:502
249940,0watchdog/210470-21kthreadcore07:57:582
179940,0migration/18991-21cpuspeed_turbos06:52:501
179940,0migration/18742-21kthreadcore09:12:581
179940,0migration/12924-21fschecks_count08:52:521
179940,0migration/124773-21runrttasks10:31:041
179940,0migration/123907-21fschecks_time07:02:521
179940,0migration/123115-21kthreadcore10:13:011
179940,0migration/123073-21kthreadcore08:28:001
179940,0migration/122331-21irqcore09:37:571
179940,0migration/11990-21fschecks_count06:47:551
179940,0migration/116877-21fschecks_count08:42:551
169940,0watchdog/16111-21avahi-daemon10:02:571
169940,0watchdog/120608-21kthreadcore07:43:021
169940,0watchdog/120229-21ssh09:36:371
169940,0watchdog/119000-21date08:07:341
169940,0watchdog/114407-21ssh08:39:261
169940,0watchdog/10-21swapper/110:18:011
169940,0watchdog/10-21swapper/110:18:011
169940,0watchdog/10-21swapper/109:47:531
169940,0watchdog/10-21swapper/108:24:341
159409943,0cyclictest3346-21snmpd10:12:452
159409943,0cyclictest3346-21snmpd08:58:142
159409943,0cyclictest3346-21snmpd07:24:292
159409943,0cyclictest31019-21grep10:17:552
159409943,0cyclictest31019-21grep10:17:552
159409943,0cyclictest3070-21irqbalance08:32:402
159409943,0cyclictest3070-21irqbalance07:42:412
159409943,0cyclictest3070-21irqbalance07:32:512
159409943,0cyclictest3070-21irqbalance06:37:512
159409943,0cyclictest2631-21awk06:47:582
159409943,0cyclictest19652-21grep08:07:532
159409943,0cyclictest11443-21kworker/2:208:17:382
159409940,0cyclictest0-21swapper/209:22:502
159359944,0cyclictest6111-21avahi-daemon11:22:291
159359943,0cyclictest3346-21snmpd07:51:141
159359943,0cyclictest3346-21snmpd07:14:431
159359943,0cyclictest3346-21snmpd07:09:131
159359943,0cyclictest3070-21irqbalance11:40:191
159359943,0cyclictest3070-21irqbalance10:23:001
159359943,0cyclictest3070-21irqbalance09:44:401
159359943,0cyclictest3070-21irqbalance08:12:501
159359943,0cyclictest3070-21irqbalance06:43:111
159359943,0cyclictest3070-21irqbalance06:32:511
159359943,0cyclictest3070-21irqbalance06:29:211
159359943,0cyclictest20739-21awk06:37:541
159359943,0cyclictest14023-21lldpd09:52:371
159359943,0cyclictest14023-21lldpd09:24:371
159359943,0cyclictest14023-21lldpd08:07:381
159359943,0cyclictest13198-21awk07:17:581
159359943,0cyclictest13135-21awk10:42:571
159349943,0cyclictest9813-21awk06:52:540
159349943,0cyclictest8390-21awk09:48:070
159349943,0cyclictest3346-21snmpd11:18:460
159349943,0cyclictest3346-21snmpd08:21:290
159349943,0cyclictest3346-21snmpd06:33:280
159349943,0cyclictest3070-21irqbalance11:38:590
159349943,0cyclictest3070-21irqbalance11:27:290
159349943,0cyclictest3070-21irqbalance11:10:390
159349943,0cyclictest3070-21irqbalance10:49:300
159349943,0cyclictest3070-21irqbalance10:05:200
159349943,0cyclictest3070-21irqbalance09:52:500
159349943,0cyclictest3070-21irqbalance09:43:300
159349943,0cyclictest3070-21irqbalance09:38:000
159349943,0cyclictest3070-21irqbalance09:32:300
159349943,0cyclictest3070-21irqbalance09:23:100
159349943,0cyclictest3070-21irqbalance08:38:000
159349943,0cyclictest3070-21irqbalance08:29:300
159349943,0cyclictest3070-21irqbalance07:57:400
159349943,0cyclictest3070-21irqbalance07:53:000
159349943,0cyclictest3070-21irqbalance07:50:500
159349943,0cyclictest3070-21irqbalance07:17:510
159349943,0cyclictest3070-21irqbalance07:12:510
159349943,0cyclictest3070-21irqbalance06:24:010
159349943,0cyclictest30201-21sadc06:22:340
159349943,0cyclictest14023-21lldpd09:05:070
159349943,0cyclictest14023-21lldpd08:34:080
149940,0migration/05303-21sh09:09:400
149940,0migration/032063-21sh10:53:010
149940,0migration/030425-21aten2_r1power_c10:17:520
149940,0migration/030425-21aten2_r1power_c10:17:520
149940,0migration/025639-21sh09:57:340
329930,0watchdog/30-21swapper/306:40:583
329930,0watchdog/30-21swapper/306:25:303
28005030,0irq/28-eth0-rx-0-21swapper/306:57:543
28005030,0irq/28-eth0-rx-0-21swapper/306:32:523
249930,0watchdog/20-21swapper/207:17:502
169930,0watchdog/16111-21avahi-daemon10:57:491
169930,0watchdog/10-21swapper/111:12:531
169930,0watchdog/10-21swapper/110:32:371
169930,0watchdog/10-21swapper/109:27:411
169930,0watchdog/10-21swapper/107:57:341
159409933,0cyclictest7953-21kworker/2:006:27:432
159409933,0cyclictest7953-21kworker/2:006:22:352
159409933,0cyclictest11443-21kworker/2:207:52:362
159409933,0cyclictest11443-21kworker/2:207:07:342
159409932,0cyclictest23034-21wc07:02:342
159409930,0cyclictest16909-21perl06:12:492
159359933,0cyclictest6111-21avahi-daemon09:21:531
159359933,0cyclictest2431-21nvme10:53:291
159359932,0cyclictest3346-21snmpd06:59:581
159359932,0cyclictest32295-21sensors11:08:011
159359932,0cyclictest3070-21irqbalance10:07:501
159349932,0cyclictest8547-21munin-node07:38:070
159349932,0cyclictest21229-21wget10:12:520
159349932,0cyclictest13641-21sensors10:23:030
159349932,0cyclictest13213-21awk10:42:580
159349932,0cyclictest13084-21awk10:07:580
159349932,0cyclictest11554-21seq08:22:370
159349931,0cyclictest0-21swapper/006:37:440
329920,0watchdog/30-21swapper/307:42:343
169920,0watchdog/16111-21avahi-daemon10:37:371
169920,0watchdog/16111-21avahi-daemon07:37:381
169920,0watchdog/16111-21avahi-daemon07:32:421
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional