You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-26 - 13:18
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Fri Apr 26, 2024 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2892324839,0sleep20-21swapper/218:54:292
2872823324,0sleep10-21swapper/118:52:121
2880723026,0sleep30-21swapper/318:53:103
2876322824,0sleep00-21swapper/018:52:350
290109970,0cyclictest20391-21telnet21:39:520
339960,0migration/39932-21sh22:49:093
339960,0migration/33059-21irqrtprio00:20:003
339960,0migration/330386-21munin-node21:44:583
339960,0migration/328514-21kthreadcore19:15:053
339960,0migration/321951-21sh23:32:483
339960,0migration/319477-21hwlatdetect21:19:583
339960,0migration/317677-21fschecks_count22:34:543
29925060,0irq/32-eth0-rx-837-21ssh21:09:183
29925060,0irq/32-eth0-rx-26201-21cut22:19:593
290129965,0cyclictest0-21swapper/220:40:012
290119965,0cyclictest0-21swapper/123:02:591
290109966,0cyclictest0-21swapper/022:59:480
290109960,0cyclictest0-21swapper/022:00:220
290109960,0cyclictest0-21swapper/020:45:100
149960,0migration/022086-21sh22:15:280
8315050,0irq/26-0000:00:24886-21crond21:04:392
8315050,0irq/26-0000:00:19861-21sensors_volt21:55:042
339950,0migration/39030-21kthreadcore19:45:013
339950,0migration/35981-21cpuspeed_turbos23:24:503
339950,0migration/34459-21latency_hist20:29:393
339950,0migration/34395-21kthreadcore20:49:593
339950,0migration/331343-21kthreadcore18:55:013
339950,0migration/329681-21runrttasks20:24:493
339950,0migration/327470-21munin-node20:44:583
339950,0migration/32609-21kthreadcore23:40:023
339950,0migration/324012-21kthreadcore21:40:013
339950,0migration/322747-21latency_hist23:34:403
339950,0migration/321938-21munin-node23:14:563
339950,0migration/321235-21kthreadcore19:54:593
339950,0migration/320190-21kthreadcore23:49:593
339950,0migration/31740-21taskset21:26:273
339950,0migration/312763-21kthreadcore19:25:003
339950,0migration/310174-21cpuspeed_turbos21:14:523
329950,0watchdog/38550-21munin-node22:30:003
329950,0watchdog/328720-21cut23:55:003
329950,0watchdog/324084-21vmstat00:10:123
329950,0watchdog/312290-21kthreadcore23:45:003
329950,0watchdog/30-21swapper/320:35:003
29935050,0irq/33-eth0-tx-7838-21ssh00:01:422
29935050,0irq/33-eth0-tx-7803-21munin-node22:45:022
29935050,0irq/33-eth0-tx-28933-21sshd19:36:320
29935050,0irq/33-eth0-tx-28105-21nfs4_client22:00:012
29935050,0irq/33-eth0-tx-25929-21munin-node20:00:040
29935050,0irq/33-eth0-tx-18542-21munin-node19:09:530
29935050,0irq/33-eth0-tx-18471-21munin-node21:20:052
29935050,0irq/33-eth0-tx-12621-21kthreadcore22:30:022
29935050,0irq/33-eth0-tx-0-21swapper/223:15:362
29935050,0irq/33-eth0-tx-0-21swapper/223:01:132
29935050,0irq/33-eth0-tx-0-21swapper/222:15:142
29935050,0irq/33-eth0-tx-0-21swapper/200:19:532
29935050,0irq/33-eth0-tx-0-21swapper/020:55:040
29935050,0irq/33-eth0-tx-0-21swapper/019:04:520
29935050,0irq/33-eth0-tx-0-21swapper/019:03:300
29925050,0irq/32-eth0-rx-9354-21df_abs00:04:503
29925050,0irq/32-eth0-rx-8268-21sed22:09:573
29925050,0irq/32-eth0-rx-7930-21cut23:05:003
29925050,0irq/32-eth0-rx-6386-21runrttasks21:04:193
29925050,0irq/32-eth0-rx-5666-21ssh22:07:183
29925050,0irq/32-eth0-rx-31947-21tr22:59:593
29925050,0irq/32-eth0-rx-22283-21ssh22:16:033
29925050,0irq/32-eth0-rx-20058-21ksmtuned23:13:123
29925050,0irq/32-eth0-rx-18123-21smartctl21:35:073
29925050,0irq/32-eth0-rx-0-21swapper/323:59:573
29925050,0irq/32-eth0-rx-0-21swapper/322:58:353
29925050,0irq/32-eth0-rx-0-21swapper/322:49:423
29925050,0irq/32-eth0-rx-0-21swapper/321:50:253
29925050,0irq/32-eth0-rx-0-21swapper/321:29:573
29925050,0irq/32-eth0-rx-0-21swapper/321:09:553
29925050,0irq/32-eth0-rx-0-21swapper/320:20:003
29925050,0irq/32-eth0-rx-0-21swapper/300:15:133
290129955,0cyclictest0-21swapper/222:05:282
290129955,0cyclictest0-21swapper/221:54:352
290129954,0cyclictest31839-21strings20:00:012
290129950,0cyclictest0-21swapper/223:49:212
290129950,0cyclictest0-21swapper/223:25:332
290129950,0cyclictest0-21swapper/220:55:562
290129950,0cyclictest0-21swapper/200:09:422
290119955,0cyclictest0-21swapper/122:39:391
290119955,0cyclictest0-21swapper/122:31:081
290119954,0cyclictest30157-21grep18:54:571
290119950,0cyclictest0-21swapper/121:24:081
290109955,0cyclictest0-21swapper/023:45:030
290109955,0cyclictest0-21swapper/023:09:250
290109955,0cyclictest0-21swapper/021:52:570
290109955,0cyclictest0-21swapper/021:27:530
290109950,0cyclictest0-21swapper/022:28:450
27150,0ktimersoftd/20-21swapper/221:19:252
27150,0ktimersoftd/20-21swapper/219:22:142
259950,0migration/29120-21aten_r1power_po00:04:492
259950,0migration/232123-21pmu-power20:45:042
259950,0migration/229617-21munin-node20:24:482
259950,0migration/219235-21kthreadcore23:49:592
259950,0migration/214653-21df_abs23:29:542
259950,0migration/210718-21cpuspeed_turbos22:49:512
249950,0watchdog/231462-21kthreadcore23:20:002
249950,0watchdog/227314-21kthreadcore00:15:042
19150,0ktimersoftd/10-21swapper/119:22:141
179950,0migration/13472-21kthreadcore22:45:001
179950,0migration/126942-21fschecks_time20:44:541
179950,0migration/123723-21taskset21:01:101
179950,0migration/117729-21kthreadcore20:15:011
179950,0migration/117520-21aten_r1power_po00:09:521
179950,0migration/113690-21taskset23:28:421
169950,0watchdog/127207-21irqrtprio19:15:041
169950,0watchdog/115186-21cut21:35:001
149950,0migration/09155-21sh22:46:540
149950,0migration/08559-21fschecks_time19:24:540
149950,0migration/08556-21kthreadcore22:09:570
149950,0migration/07133-21kthreadcore20:30:010
149950,0migration/06868-21missed_timers20:50:010
149950,0migration/05645-21irqcore19:45:000
149950,0migration/04062-21sh23:20:500
149950,0migration/032464-21kthreadcore22:05:010
149950,0migration/031875-21ksmtuned21:05:110
149950,0migration/030375-21if_eth1.218:54:590
149950,0migration/028730-21latency_hist21:44:390
149950,0migration/024935-21kthreadcore23:14:590
149950,0migration/023652-21kthreadcore20:40:010
149950,0migration/019430-21kthreadcore20:15:020
149950,0migration/016435-21cpuspeed_turbos23:49:510
149950,0migration/015570-21kthreadcore21:35:000
149950,0migration/01456-21kthreadcore20:25:050
149950,0migration/011217-21kthreadcore22:30:010
149950,0migration/010886-21sh21:32:540
35140,0ktimersoftd/30-21swapper/319:22:143
339940,0migration/330957-21irqrtprio19:39:583
339940,0migration/315472-21cpuspeed_turbos20:14:533
339940,0migration/31118-21irqcore22:24:583
329940,0watchdog/32772-21kthreadcore20:05:003
329940,0watchdog/30-21swapper/323:20:003
329940,0watchdog/30-21swapper/322:40:043
329940,0watchdog/30-21swapper/319:35:003
329940,0watchdog/30-21swapper/319:29:563
329940,0watchdog/30-21swapper/319:10:123
329940,0watchdog/30-21swapper/319:08:163
329940,0watchdog/30-21swapper/319:00:003
29935040,0irq/33-eth0-tx-8097-21telnet20:09:490
29935040,0irq/33-eth0-tx-0-21swapper/223:39:442
29935040,0irq/33-eth0-tx-0-21swapper/223:34:422
29935040,0irq/33-eth0-tx-0-21swapper/222:54:442
29935040,0irq/33-eth0-tx-0-21swapper/222:09:472
29935040,0irq/33-eth0-tx-0-21swapper/221:45:122
29935040,0irq/33-eth0-tx-0-21swapper/221:34:502
29935040,0irq/33-eth0-tx-0-21swapper/221:29:452
29935040,0irq/33-eth0-tx-0-21swapper/020:19:530
29935040,0irq/33-eth0-tx-0-21swapper/019:39:510
29935040,0irq/33-eth0-tx-0-21swapper/019:15:040
29925040,0irq/32-eth0-rx-0-21swapper/322:01:053
29925040,0irq/32-eth0-rx-0-21swapper/321:54:483
29925040,0irq/32-eth0-rx-0-21swapper/320:54:553
29925040,0irq/32-eth0-rx-0-21swapper/320:39:543
29925040,0irq/32-eth0-rx-0-21swapper/319:59:493
290129943,0cyclictest3599-21lldpd23:10:342
290129943,0cyclictest3599-21lldpd23:05:332
290129943,0cyclictest3599-21lldpd22:19:532
290129943,0cyclictest3599-21lldpd21:40:452
290129943,0cyclictest3599-21lldpd19:49:022
290129943,0cyclictest30526-21awk19:39:562
290129943,0cyclictest28185-21awk21:24:552
290129943,0cyclictest26875-21awk19:15:032
290129943,0cyclictest23871-21snmpd22:25:212
290129943,0cyclictest1981-21awk20:49:572
290129943,0cyclictest18121-21awk22:34:562
290129943,0cyclictest15004-21sadc20:14:392
290119944,0cyclictest0-21swapper/123:36:341
290119943,0cyclictest9294-21awk20:54:561
290119943,0cyclictest8832-21awk23:44:571
290119943,0cyclictest3599-21lldpd23:39:441
290119943,0cyclictest3599-21lldpd23:30:381
290119943,0cyclictest3599-21lldpd22:40:011
290119943,0cyclictest3599-21lldpd21:51:201
290119943,0cyclictest3599-21lldpd00:20:581
290119943,0cyclictest29980-21awk21:44:571
290119943,0cyclictest23871-21snmpd20:32:041
290119943,0cyclictest23172-21awk21:59:561
290119943,0cyclictest19483-21grep21:55:001
290119943,0cyclictest10647-21ssh23:05:491
290109944,0cyclictest0-21swapper/000:09:480
290109944,0cyclictest0-21swapper/000:05:520
290109943,0cyclictest3599-21lldpd21:18:110
290109943,0cyclictest3262-21irqbalance21:11:440
290109943,0cyclictest3262-21irqbalance21:02:440
290109943,0cyclictest3262-21irqbalance00:15:240
290109943,0cyclictest2771-21awk00:19:580
290109943,0cyclictest24873-21awk22:19:560
290109943,0cyclictest23871-21snmpd19:54:340
290109943,0cyclictest20069-21awk22:54:550
290109943,0cyclictest1565-21awk19:19:550
259940,0migration/229737-21fschecks_count18:54:562
259940,0migration/228233-21kthreadcore22:40:012
259940,0migration/228054-21kthreadcore23:54:592
259940,0migration/227498-21kthreadcore21:05:002
259940,0migration/223022-21kthreadcore19:10:022
259940,0migration/218747-21munin-node19:54:502
259940,0migration/216800-21kernelversion20:15:012
259940,0migration/215683-21fschecks_count19:29:562
259940,0migration/213373-21kthreadcore20:34:582
259940,0migration/21214-21sh21:09:442
249940,0watchdog/212883-21tr19:25:002
249940,0watchdog/212765-21iostat_ios19:50:002
249940,0watchdog/20-21swapper/220:05:002
249940,0watchdog/20-21swapper/219:00:002
179940,0migration/19855-21kthreadcore20:09:581
179940,0migration/17635-21proc_pri21:10:061
179940,0migration/16410-21kthreadcore19:45:011
179940,0migration/15469-21kthreadcore22:25:011
179940,0migration/131160-21kthreadcore21:24:581
179940,0migration/129745-21cpuspeed_turbos19:39:511
179940,0migration/127805-21kthreadcore21:05:001
179940,0migration/127213-21irqcore19:59:581
179940,0migration/126654-21smart_sda21:40:081
179940,0migration/125690-21aten_r1power_cu00:14:511
179940,0migration/125454-21kthreadcore22:19:591
179940,0migration/123440-21kthreadcore19:55:001
179940,0migration/122550-21aten_r1power_po20:19:501
179940,0migration/120820-21kthreadcore20:40:001
179940,0migration/11903-21munin-node20:04:591
179940,0migration/117156-21kthreadcore19:30:021
179940,0migration/115772-21kthreadcore20:34:591
179940,0migration/115054-21kthreadcore21:15:031
179940,0migration/11383-21fschecks_time23:59:521
179940,0migration/112770-21munin-node22:10:001
179940,0migration/112411-21aten_r1power_vo23:09:511
179940,0migration/110503-21taskset21:31:461
169940,0watchdog/19660-21grep00:04:521
169940,0watchdog/131781-21sh22:05:001
169940,0watchdog/125051-21diskstats23:54:521
169940,0watchdog/121259-21df_abs23:14:521
169940,0watchdog/118382-21ssh22:53:561
169940,0watchdog/116863-21kernelversion22:15:001
169940,0watchdog/116157-21ssh23:49:441
169940,0watchdog/10-21swapper/120:24:481
149940,0migration/0843-21aten_r1power_vo20:04:520
149940,0migration/021649-21sh21:59:230
149940,0migration/020936-21missed_timers19:30:040
149940,0migration/017274-21kthreadcore23:10:010
149940,0migration/012054-21sendmail_mailqu23:25:020
329930,0watchdog/30-21swapper/319:50:043
29935030,0irq/33-eth0-tx-0-21swapper/019:54:500
29925030,0irq/32-eth0-rx-0-21swapper/320:09:493
290129932,0cyclictest23542-21awk20:19:572
290129932,0cyclictest23346-21awk19:34:562
290129932,0cyclictest11507-21basename20:34:392
290119933,0cyclictest8952-21awk19:24:571
290119932,0cyclictest5266-21awk18:59:581
290119932,0cyclictest30503-21awk23:19:561
290119932,0cyclictest12598-21awk19:49:581
290119930,0cyclictest24448-21grep19:35:001
290109932,0cyclictest32582-21awk23:39:580
290109932,0cyclictest27811-21tr23:34:580
290109932,0cyclictest26799-21awk22:40:000
290109932,0cyclictest24648-21basename23:54:480
290109932,0cyclictest20877-21grep22:35:000
290109932,0cyclictest1635-21awk23:59:550
290109932,0cyclictest12755-21awk20:34:560
290109930,0cyclictest19162-21perl21:19:560
290109930,0cyclictest15093-21perl23:29:550
290109930,0cyclictest0-21swapper/022:49:510
249930,0watchdog/20-21swapper/219:04:522
179930,0migration/12463-21kthreadcore20:49:581
179930,0migration/121098-21kthreadcore19:10:011
179930,0migration/117157-21latency19:05:031
169930,0watchdog/118908-21ssh22:54:481
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional