You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-24 - 05:41
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Sun Sep 24, 2023 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1755323727,0sleep30-21swapper/319:05:553
1739223625,0sleep00-21swapper/019:04:030
1737823626,0sleep10-21swapper/119:03:521
1756223525,0sleep20-21swapper/219:06:022
339960,0migration/38740-21sh23:04:083
339960,0migration/332445-21fschecks_count22:42:573
339960,0migration/330286-21sh21:24:583
339960,0migration/326975-21munin-node23:33:063
339960,0migration/32639-21sh00:24:383
339960,0migration/324521-21kthreadcore22:18:043
339960,0migration/312017-21idleruntime-cro00:32:353
29935060,0irq/33-eth0-tx-7292-21munin-node19:23:120
29935060,0irq/33-eth0-tx-29743-21diskstats21:08:000
29925060,0irq/32-eth0-rx-8139-21unixbench_multi21:48:133
29925060,0irq/32-eth0-rx-17604-21perl22:52:593
29925060,0irq/32-eth0-rx-12831-21df_inode23:28:003
29925060,0irq/32-eth0-rx-11137-21kthreadcore20:53:053
29925060,0irq/32-eth0-rx-0-21swapper/320:33:093
177749960,0cyclictest0-21swapper/022:33:040
177749960,0cyclictest0-21swapper/000:33:310
8315050,0irq/26-0000:00:31208-21sh21:27:362
8315050,0irq/26-0000:00:13456-21sed20:53:062
4150,0ktimersoftd/00-21swapper/021:26:230
339950,0migration/38975-21kthreadcore22:08:043
339950,0migration/37217-21kthreadcore21:13:043
339950,0migration/328443-21taskset20:18:213
339950,0migration/328135-21fschecks_time22:02:593
339950,0migration/323326-21munin-node23:52:533
339950,0migration/323246-21kthreadcore19:33:033
339950,0migration/322012-21latency_hist21:02:363
339950,0migration/319435-21kthreadcore21:38:073
339950,0migration/319217-21cpuspeed_turbos23:12:573
339950,0migration/318209-21munin-node19:07:553
339950,0migration/317381-21unixbench-2d19:48:123
339950,0migration/316762-21kthreadcore20:58:053
339950,0migration/315141-21fschecks_time21:17:573
339950,0migration/310505-21kthreadcore19:23:043
329950,0watchdog/39985-21python19:43:093
329950,0watchdog/327095-21cat19:13:053
329950,0watchdog/323879-21grep22:38:003
329950,0watchdog/310801-21sh23:23:483
329950,0watchdog/30-21swapper/320:27:573
29935050,0irq/33-eth0-tx-3724-21munin-node19:43:060
29935050,0irq/33-eth0-tx-32410-21munin-node19:18:060
29935050,0irq/33-eth0-tx-32110-21ntp_kernel_pll_00:18:022
29935050,0irq/33-eth0-tx-29463-21munin-node23:38:102
29935050,0irq/33-eth0-tx-18816-21munin-node21:58:012
29935050,0irq/33-eth0-tx-18146-21munin-node19:53:100
29935050,0irq/33-eth0-tx-15312-21ssh21:35:082
29935050,0irq/33-eth0-tx-0-21swapper/223:05:042
29935050,0irq/33-eth0-tx-0-21swapper/222:39:582
29935050,0irq/33-eth0-tx-0-21swapper/020:48:050
29925050,0irq/32-eth0-rx-7406-21irqrtprio22:28:033
29925050,0irq/32-eth0-rx-6580-21sh21:29:473
29925050,0irq/32-eth0-rx-6580-21sh21:29:473
29925050,0irq/32-eth0-rx-608-21kthreadcore23:58:073
29925050,0irq/32-eth0-rx-25445-21sort20:42:363
29925050,0irq/32-eth0-rx-21158-21grep21:58:043
29925050,0irq/32-eth0-rx-15869-21kthreadcore00:33:073
29925050,0irq/32-eth0-rx-15503-21sed22:32:573
29925050,0irq/32-eth0-rx-13874-21memory21:33:033
29925050,0irq/32-eth0-rx-13486-21ssh23:46:053
29925050,0irq/32-eth0-rx-13149-21fschecks_time22:12:573
29925050,0irq/32-eth0-rx-13016-21kthreadcore22:48:053
29925050,0irq/32-eth0-rx-11590-21irqcore21:53:043
29925050,0irq/32-eth0-rx-0-21swapper/323:49:403
29925050,0irq/32-eth0-rx-0-21swapper/323:38:033
29925050,0irq/32-eth0-rx-0-21swapper/322:59:263
29925050,0irq/32-eth0-rx-0-21swapper/322:24:443
29925050,0irq/32-eth0-rx-0-21swapper/300:12:333
259950,0migration/224967-21sh22:57:252
259950,0migration/224124-21kthreadcore21:03:052
259950,0migration/222824-21latency_hist23:52:352
259950,0migration/218064-21kthreadcore20:13:032
259950,0migration/212334-21sh21:13:242
249950,0watchdog/231728-21aten_r1power_vo21:27:562
249950,0watchdog/231728-21aten_r1power_vo21:27:562
177889955,0cyclictest0-21swapper/222:31:242
177889954,0cyclictest26188-21kthreadcore22:18:052
177889950,0cyclictest0-21swapper/222:25:152
177829955,0cyclictest29019-21diskmemload23:21:311
177829955,0cyclictest0-21swapper/122:35:191
177829955,0cyclictest0-21swapper/121:55:261
177829955,0cyclictest0-21swapper/100:08:201
177829950,0cyclictest0-21swapper/123:05:541
177829950,0cyclictest0-21swapper/122:19:141
177829950,0cyclictest0-21swapper/121:58:441
177829950,0cyclictest0-21swapper/121:14:361
177829950,0cyclictest0-21swapper/100:16:411
177749954,0cyclictest3262-21irqbalance23:21:400
177749954,0cyclictest3262-21irqbalance23:03:500
177749954,0cyclictest3262-21irqbalance21:37:400
177749954,0cyclictest3262-21irqbalance00:02:400
177749950,0cyclictest0-21swapper/022:55:460
177749950,0cyclictest0-21swapper/021:56:010
179950,0migration/127949-21kthreadcore20:43:041
179950,0migration/127199-21kthreadcore19:13:051
179950,0migration/126950-21kthreadcore22:38:051
179950,0migration/124324-21munin-node20:38:091
149950,0migration/09694-21kthreadcore23:23:030
149950,0migration/08085-21kthreadcore23:43:040
149950,0migration/04147-21cpu20:27:570
149950,0migration/0327-21sh21:28:010
149950,0migration/0327-21sh21:28:010
149950,0migration/032606-21latency_hist20:47:360
149950,0migration/031887-21aten_r1power_fr22:42:550
149950,0migration/029519-21latency_hist22:22:350
149950,0migration/021600-21latency_hist19:32:360
149950,0migration/020900-21latency_hist22:17:350
149950,0migration/018698-21sendmail_mailqu23:28:090
149950,0migration/011986-21kthreadcore21:33:020
8315040,0irq/26-0000:00:0-21swapper/223:48:042
339940,0migration/325952-21taskset00:17:483
339940,0migration/32435-21kthreadcore20:03:023
339940,0migration/310482-21kthreadcore20:08:053
329940,0watchdog/30-21swapper/321:08:053
329940,0watchdog/30-21swapper/320:49:013
329940,0watchdog/30-21swapper/319:57:573
329940,0watchdog/30-21swapper/319:28:133
329940,0watchdog/30-21swapper/300:05:323
29935040,0irq/33-eth0-tx-14233-21ssh21:17:512
29935040,0irq/33-eth0-tx-0-21swapper/223:45:132
29935040,0irq/33-eth0-tx-0-21swapper/223:28:162
29935040,0irq/33-eth0-tx-0-21swapper/223:22:592
29935040,0irq/33-eth0-tx-0-21swapper/222:57:442
29935040,0irq/33-eth0-tx-0-21swapper/222:44:112
29935040,0irq/33-eth0-tx-0-21swapper/222:03:042
29935040,0irq/33-eth0-tx-0-21swapper/221:39:562
29935040,0irq/33-eth0-tx-0-21swapper/200:32:452
29935040,0irq/33-eth0-tx-0-21swapper/200:23:492
29935040,0irq/33-eth0-tx-0-21swapper/200:14:052
29935040,0irq/33-eth0-tx-0-21swapper/020:22:550
29925040,0irq/32-eth0-rx-25097-21ssh21:42:423
29925040,0irq/32-eth0-rx-16686-21smartctl23:08:103
29925040,0irq/32-eth0-rx-0-21swapper/323:17:453
29925040,0irq/32-eth0-rx-0-21swapper/319:38:073
29925040,0irq/32-eth0-rx-0-21swapper/300:13:253
259940,0migration/27367-21aten_r1power_fr19:22:562
259940,0migration/25286-21kernelversion22:08:032
259940,0migration/24815-21fschecks_time00:27:532
259940,0migration/2331-21taskset20:02:552
259940,0migration/231447-21kthreadcore19:38:072
259940,0migration/230438-21munin-node21:08:022
259940,0migration/223994-21kthreadcore23:33:032
259940,0migration/221158-21kthreadcore19:53:022
259940,0migration/217337-21sh23:09:032
259940,0migration/212785-21kthreadcore21:53:052
259940,0migration/212133-21kthreadcore00:03:042
249940,0watchdog/27836-21rt-features21:48:082
249940,0watchdog/232015-21fschecks_count23:58:002
249940,0watchdog/230165-21tr20:43:052
249940,0watchdog/225971-21fschecks_count19:57:572
249940,0watchdog/225596-21df21:42:562
249940,0watchdog/219662-21cut00:08:042
249940,0watchdog/218510-21tune2fs19:07:572
249940,0watchdog/215991-21irqrtprio22:33:002
249940,0watchdog/210948-21kthreadcore20:08:052
249940,0watchdog/20-21swapper/223:13:042
249940,0watchdog/20-21swapper/220:27:572
249940,0watchdog/20-21swapper/219:13:052
177889944,0cyclictest0-21swapper/223:18:322
177889943,0cyclictest9224-21awk22:48:012
177889943,0cyclictest4799-21awk19:42:592
177889943,0cyclictest3536-21snmpd20:38:202
177889943,0cyclictest22799-21awk19:33:002
177889943,0cyclictest16051-21sort20:58:042
177889943,0cyclictest15570-21awk19:28:012
177889943,0cyclictest13579-21awk22:12:592
177889943,0cyclictest11913-21gltestperf19:47:592
177889943,0cyclictest1174-21awk19:18:012
177829944,0cyclictest0-21swapper/122:59:451
177829944,0cyclictest0-21swapper/121:23:251
177829944,0cyclictest0-21swapper/121:17:541
177829944,0cyclictest0-21swapper/100:17:481
177829943,0cyclictest959-21cat22:03:071
177829943,0cyclictest9208-21awk22:48:011
177829943,0cyclictest7017-21awk22:28:011
177829943,0cyclictest3599-21lldpd22:42:431
177829943,0cyclictest3599-21lldpd19:26:301
177829943,0cyclictest3536-21snmpd23:47:201
177829943,0cyclictest3536-21snmpd23:27:051
177829943,0cyclictest3536-21snmpd20:53:351
177829943,0cyclictest3536-21snmpd20:37:051
177829943,0cyclictest3536-21snmpd20:05:051
177829943,0cyclictest3536-21snmpd19:58:051
177829943,0cyclictest3536-21snmpd19:55:051
177829943,0cyclictest3536-21snmpd19:51:201
177829943,0cyclictest3536-21snmpd19:20:051
177829943,0cyclictest3536-21snmpd00:27:501
177829943,0cyclictest30050-21awk19:38:051
177829943,0cyclictest24023-21awk23:52:571
177829943,0cyclictest22775-21awk19:33:001
177829943,0cyclictest15643-21awk19:28:011
177829943,0cyclictest13577-21awk23:28:031
177829942,0cyclictest3599-21lldpd20:08:001
177749944,0cyclictest0-21swapper/023:54:440
177749944,0cyclictest0-21swapper/022:03:450
177749943,0cyclictest8743-21awk20:53:010
177749943,0cyclictest3599-21lldpd22:07:410
177749943,0cyclictest3599-21lldpd19:13:010
177749943,0cyclictest3536-21snmpd22:57:500
177749943,0cyclictest3262-21irqbalance23:57:400
177749943,0cyclictest3262-21irqbalance22:00:000
177749943,0cyclictest30026-21awk19:38:050
177749943,0cyclictest26373-21awk21:43:000
177749943,0cyclictest24314-21awk22:38:020
177749943,0cyclictest19411-21awk20:38:040
177749943,0cyclictest16866-21awk00:08:010
177749943,0cyclictest12311-21awk20:33:010
177749943,0cyclictest0-21swapper/019:07:570
179940,0migration/15395-21runrttasks22:27:031
179940,0migration/13040-21sh00:25:491
179940,0migration/122404-21fschecks_count20:17:591
179940,0migration/120561-21kernelversion23:13:041
179940,0migration/120393-21kthreadcore20:58:061
179940,0migration/117877-21kthreadcore22:13:041
179940,0migration/116771-21kthreadcore20:13:021
179940,0migration/115800-21irqrtprio23:48:021
179940,0migration/115576-21latency_hist00:07:351
179940,0migration/114242-21kthreadcore23:08:051
179940,0migration/11373-21munin-node20:48:011
179940,0migration/11372-21kthreadcore21:08:051
169940,0watchdog/19151-21iostat_ios21:33:001
169940,0watchdog/14043-21cut20:27:571
169940,0watchdog/131727-21basename21:27:561
169940,0watchdog/131727-21basename21:27:561
169940,0watchdog/129603-21fschecks_count20:22:571
169940,0watchdog/129463-21munin-node23:38:121
169940,0watchdog/118511-21grep19:07:571
169940,0watchdog/116479-21ssh21:37:521
169940,0watchdog/10-21swapper/123:58:001
149940,0migration/02420-21hddtemp_smartct21:47:590
149940,0migration/022521-21sh23:52:230
149940,0migration/02228-21kthreadcore23:38:040
149940,0migration/020002-21idleruntime23:13:000
149940,0migration/015737-21kthreadcore19:48:040
149940,0migration/013040-21sh21:15:130
149940,0migration/01288-21diskmemload00:21:130
149940,0migration/011070-21runrttasks00:29:210
329930,0watchdog/30-21swapper/320:13:053
29935030,0irq/33-eth0-tx-0-21swapper/021:02:570
29935030,0irq/33-eth0-tx-0-21swapper/019:58:010
29925030,0irq/32-eth0-rx-0-21swapper/320:37:593
29925030,0irq/32-eth0-rx-0-21swapper/320:22:533
29925030,0irq/32-eth0-rx-0-21swapper/319:53:183
29925030,0irq/32-eth0-rx-0-21swapper/319:18:553
259930,0migration/22886-21sensors_volt20:23:072
177889932,0cyclictest6386-21runrttasks20:51:522
177889932,0cyclictest12213-21awk20:33:002
177829932,0cyclictest3536-21snmpd19:43:201
177829932,0cyclictest22747-21kthreadcore23:33:031
177829931,0cyclictest6023-21kthreadcore22:08:031
177829930,0cyclictest17642-21perl22:52:591
177749932,0cyclictest7182-21irqcore22:28:020
177749932,0cyclictest3536-21snmpd22:13:430
177749932,0cyclictest3536-21snmpd00:24:100
177749932,0cyclictest16530-21pmu-power23:08:080
177749932,0cyclictest15283-21cut20:57:590
177749932,0cyclictest14657-21munin-node20:12:580
177749931,0cyclictest13562-21proc_pri20:08:080
177749930,0cyclictest8863-21tune2fs22:47:590
177749930,0cyclictest25582-21ssh00:17:200
177749930,0cyclictest25445-21cut23:33:040
177749930,0cyclictest20358-21ntp_kernel_pll_21:18:040
177749930,0cyclictest0-21swapper/020:17:580
169930,0watchdog/13536-21snmpd21:03:331
169930,0watchdog/12571-21ssh21:48:001
169930,0watchdog/10-21swapper/121:42:561
169930,0watchdog/10-21swapper/100:32:401
177889920,0cyclictest0-21swapper/220:17:362
177749921,0cyclictest32502-21wc20:02:360
177749921,0cyclictest14452-21latency_hist19:27:360
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional