You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-27 - 03:27
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2.osadl.org (updated Sat Apr 27, 2024 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3057624936,0sleep00-21swapper/018:52:250
3062323927,0sleep20-21swapper/218:53:032
3056323725,0sleep10-21swapper/118:52:221
3038323626,0sleep30-21swapper/318:50:163
308299977,0cyclictest2546-21nvme23:11:100
308299977,0cyclictest2546-21nvme20:03:230
308299976,0cyclictest8158-21grep00:19:560
339960,0migration/39691-21sh21:34:403
339960,0migration/36869-21unixbench_multi22:25:073
339960,0migration/3392-21sh21:29:143
339960,0migration/328698-21irqcore19:14:543
339960,0migration/326579-21sh21:43:433
339960,0migration/315166-21sh21:18:423
339960,0migration/314693-21kthreadcore19:04:563
308319960,0cyclictest9847-21diskmemload23:38:042
308319960,0cyclictest0-21swapper/219:05:002
308309960,0cyclictest0-21swapper/123:20:411
308309960,0cyclictest0-21swapper/122:50:151
308299966,0cyclictest9847-21diskmemload21:02:040
308299966,0cyclictest2546-21nvme23:02:050
308299966,0cyclictest2546-21nvme22:00:590
308299966,0cyclictest2546-21nvme21:58:120
308299966,0cyclictest2546-21nvme21:35:560
308299966,0cyclictest2546-21nvme21:23:410
308299966,0cyclictest2546-21nvme20:27:550
308299960,0cyclictest0-21swapper/022:05:080
29935060,0irq/33-eth0-tx-13355-21gltestperf21:54:512
29925060,0irq/32-eth0-rx-27876-21fschecks_time21:44:503
149960,0migration/06338-21munin-node18:59:480
149960,0migration/011528-21kthreadcore20:29:560
149960,0migration/011160-21unixbench_singl00:00:010
8315050,0irq/26-0000:00:17235-21python23:45:032
339950,0migration/38928-21kthreadcore22:09:543
339950,0migration/36463-21fschecks_count18:59:483
339950,0migration/35633-21taskset19:42:443
339950,0migration/35072-21kthreadcore19:19:553
339950,0migration/331143-21kthreadcore19:59:543
339950,0migration/330934-21kthreadcore22:39:593
339950,0migration/330547-21diskmemload22:20:103
339950,0migration/329606-21kthreadcore19:34:553
339950,0migration/326250-21kthreadcore23:14:533
339950,0migration/325785-21munin-node23:34:463
339950,0migration/31995-21kthreadcore21:09:553
339950,0migration/31445-21kthreadcore22:59:573
339950,0migration/313398-21kthreadcore20:54:573
339950,0migration/313037-21taskset20:31:073
339950,0migration/311047-21munin-node00:00:013
329950,0watchdog/39182-21ssh21:33:563
329950,0watchdog/324252-21sh23:31:323
329950,0watchdog/321274-21port_sshpriv22:15:003
329950,0watchdog/319518-21aten_r1power_cu23:49:443
329950,0watchdog/317682-21fschecks_count20:14:493
329950,0watchdog/315629-21cut23:09:563
329950,0watchdog/30-21swapper/319:54:533
308319954,0cyclictest23871-21snmpd22:01:262
308319950,0cyclictest0-21swapper/220:57:062
308309955,0cyclictest0-21swapper/122:34:481
308309955,0cyclictest0-21swapper/122:05:101
308309955,0cyclictest0-21swapper/121:13:371
308309950,0cyclictest0-21swapper/123:37:521
308309950,0cyclictest0-21swapper/122:44:441
308309950,0cyclictest0-21swapper/122:43:381
308309950,0cyclictest0-21swapper/122:14:281
308309950,0cyclictest0-21swapper/121:39:191
308309950,0cyclictest0-21swapper/119:10:011
308299955,0cyclictest2546-21nvme20:41:090
308299955,0cyclictest2546-21nvme19:20:340
308299955,0cyclictest0-21swapper/021:14:120
308299954,0cyclictest4048-21awk22:44:530
308299950,0cyclictest0-21swapper/023:17:110
308299950,0cyclictest0-21swapper/021:47:000
29935050,0irq/33-eth0-tx-878-21munin-node21:29:502
29935050,0irq/33-eth0-tx-6201-21ssh21:13:102
29935050,0irq/33-eth0-tx-3948-21ssh22:05:032
29935050,0irq/33-eth0-tx-21855-21munin-node21:04:502
29935050,0irq/33-eth0-tx-168512sleep221:36:072
29935050,0irq/33-eth0-tx-16238-21ssh22:31:252
29935050,0irq/33-eth0-tx-0-21swapper/223:52:212
29935050,0irq/33-eth0-tx-0-21swapper/222:47:242
29935050,0irq/33-eth0-tx-0-21swapper/222:23:152
29935050,0irq/33-eth0-tx-0-21swapper/221:40:312
29935050,0irq/33-eth0-tx-0-21swapper/221:27:052
29935050,0irq/33-eth0-tx-0-21swapper/200:02:202
29935050,0irq/33-eth0-tx-0-21swapper/020:54:410
29935050,0irq/33-eth0-tx-0-21swapper/020:38:250
29935050,0irq/33-eth0-tx-0-21swapper/019:39:030
29925050,0irq/32-eth0-rx-7369-21if_eth1.300:19:533
29925050,0irq/32-eth0-rx-5154-21kthreadcore20:49:583
29925050,0irq/32-eth0-rx-32016-21smartctl18:54:533
29925050,0irq/32-eth0-rx-29445-21irqcore23:54:523
29925050,0irq/32-eth0-rx-22958-21python19:30:013
29925050,0irq/32-eth0-rx-21207-21entropy20:39:503
29925050,0irq/32-eth0-rx-18980-21iostat_ios22:34:553
29925050,0irq/32-eth0-rx-15877-21ssh22:30:283
29925050,0irq/32-eth0-rx-13226-21sort21:54:513
29925050,0irq/32-eth0-rx-0-21swapper/323:45:253
29925050,0irq/32-eth0-rx-0-21swapper/323:42:003
29925050,0irq/32-eth0-rx-0-21swapper/323:24:573
29925050,0irq/32-eth0-rx-0-21swapper/323:20:493
29925050,0irq/32-eth0-rx-0-21swapper/323:06:073
29925050,0irq/32-eth0-rx-0-21swapper/322:51:103
29925050,0irq/32-eth0-rx-0-21swapper/322:00:123
29925050,0irq/32-eth0-rx-0-21swapper/321:49:503
29925050,0irq/32-eth0-rx-0-21swapper/320:59:473
29925050,0irq/32-eth0-rx-0-21swapper/300:15:133
29925050,0irq/32-eth0-rx-0-21swapper/300:09:583
259950,0migration/26548-21sh23:21:052
259950,0migration/25597-21aten_r1power_en23:04:482
259950,0migration/230583-21timerandwakeup20:20:092
259950,0migration/228590-21runrttasks00:11:142
259950,0migration/228384-21munin-node21:44:532
259950,0migration/228066-21latency_hist23:54:352
259950,0migration/22769-21fschecks_time23:39:482
259950,0migration/227607-21munin-node22:39:552
259950,0migration/220292-21taskset00:06:442
249950,0watchdog/224611-21ssh22:35:242
179950,0migration/17193-21idleruntime00:19:511
179950,0migration/12490-21munin-node19:19:461
179950,0migration/119724-21sh21:56:341
169950,0watchdog/12949-21diskstats20:04:491
169950,0watchdog/115628-21ssh23:09:561
149950,0migration/09966-21aten_r1power_vo20:09:480
149950,0migration/0943-21missed_timers19:14:570
149950,0migration/09350-21sh23:41:140
149950,0migration/08669-21kthreadcore19:44:540
149950,0migration/07500-21timerandwakeup21:30:100
149950,0migration/06836-21fschecks_count22:09:490
149950,0migration/06351-21ntp_kernel_pll_22:25:010
149950,0migration/05518-21sh23:04:440
149950,0migration/03241-21fschecks_time20:04:490
149950,0migration/03172-21taskset23:56:540
149950,0migration/030574-21sh22:20:140
149950,0migration/029786-21kthreadcore20:19:590
149950,0migration/02975-21kthreadcore18:54:590
149950,0migration/028894-21kthreadcore22:39:580
149950,0migration/028096-21aten_r1power_vo20:44:460
149950,0migration/027541-21kthreadcore21:24:560
149950,0migration/027328-21latency_hist19:14:360
149950,0migration/026183-21kthreadcore00:09:580
149950,0migration/019858-21taskset00:05:270
149950,0migration/015298-21kthreadcore19:24:590
149950,0migration/015168-21sh23:25:400
149950,0migration/014012-21fschecks_time19:49:490
149950,0migration/011081-21cpu23:44:480
8315040,0irq/26-0000:00:0-21swapper/319:24:363
8315040,0irq/26-0000:00:0-21swapper/221:49:362
339940,0migration/39270-21kthreadcore19:44:543
339940,0migration/332432-21kthreadcore20:44:553
339940,0migration/332124-21kthreadcore22:04:543
339940,0migration/32681-21kthreadcore20:24:543
339940,0migration/326568-21sh22:55:043
339940,0migration/313186-21aten_r1power_vo00:04:513
329940,0watchdog/30-21swapper/321:19:563
329940,0watchdog/30-21swapper/321:04:413
329940,0watchdog/30-21swapper/319:09:573
308319944,0cyclictest0-21swapper/223:24:402
308319943,0cyclictest7250-21awk00:19:522
308319943,0cyclictest3924-21awk20:49:562
308319943,0cyclictest3599-21lldpd22:12:252
308319943,0cyclictest3599-21lldpd19:19:552
308319943,0cyclictest31089-21awk00:14:512
308319943,0cyclictest28523-21awk19:14:512
308319943,0cyclictest24163-21awk23:14:512
308319943,0cyclictest23871-21snmpd20:05:412
308319943,0cyclictest23871-21snmpd19:59:112
308319943,0cyclictest20247-21sadc19:54:362
308319943,0cyclictest17870-21awk19:29:502
308319943,0cyclictest12826-21awk22:49:522
308319940,0cyclictest13418-21ls20:34:362
308309944,0cyclictest0-21swapper/122:56:531
308309943,0cyclictest7103-21awk18:59:521
308309943,0cyclictest5637-21awk23:59:491
308309943,0cyclictest3599-21lldpd19:50:261
308309943,0cyclictest3599-21lldpd19:46:551
308309943,0cyclictest3599-21lldpd00:15:351
308309943,0cyclictest3262-21irqbalance23:27:501
308309943,0cyclictest3262-21irqbalance21:40:201
308309943,0cyclictest3262-21irqbalance21:29:501
308309943,0cyclictest3262-21irqbalance21:14:401
308309943,0cyclictest3262-21irqbalance21:05:201
308309943,0cyclictest3262-21irqbalance20:59:401
308309943,0cyclictest3262-21irqbalance20:54:501
308309943,0cyclictest3262-21irqbalance20:44:501
308309943,0cyclictest3262-21irqbalance20:41:101
308309943,0cyclictest3262-21irqbalance20:34:501
308309943,0cyclictest3262-21irqbalance20:29:501
308309943,0cyclictest3262-21irqbalance20:20:011
308309943,0cyclictest3262-21irqbalance20:09:511
308309943,0cyclictest3262-21irqbalance18:55:411
308309943,0cyclictest3262-21irqbalance00:11:201
308309943,0cyclictest30269-21awk22:59:531
308309943,0cyclictest29347-21awk23:54:511
308309943,0cyclictest24256-21awk23:14:521
308299943,0cyclictest23871-21snmpd21:40:560
308299943,0cyclictest22937-21awk21:04:560
308299943,0cyclictest21509-21awk22:54:550
308299943,0cyclictest16164-21awk22:14:520
29935040,0irq/33-eth0-tx-9847-21diskmemload21:03:262
29935040,0irq/33-eth0-tx-21677-21ssh22:54:572
29935040,0irq/33-eth0-tx-13112-21munin-node19:04:450
29935040,0irq/33-eth0-tx-0-21swapper/223:29:502
29935040,0irq/33-eth0-tx-0-21swapper/222:59:542
29935040,0irq/33-eth0-tx-0-21swapper/222:15:232
29935040,0irq/33-eth0-tx-0-21swapper/221:15:192
29935040,0irq/33-eth0-tx-0-21swapper/020:50:090
29935040,0irq/33-eth0-tx-0-21swapper/020:15:060
29925040,0irq/32-eth0-rx-9954-21ssh22:45:413
259940,0migration/26510-21fschecks_count19:44:482
259940,0migration/24822-21kthreadcore22:24:582
259940,0migration/232627-21irqrtprio19:39:572
259940,0migration/225934-21kthreadcore19:34:542
259940,0migration/218896-21kthreadcore21:19:572
259940,0migration/21701-21timerwakeupswit20:00:042
259940,0migration/215402-21munin-node23:09:542
259940,0migration/21353-21kthreadcore18:54:592
249940,0watchdog/20-21swapper/220:14:492
249940,0watchdog/20-21swapper/219:09:572
179940,0migration/19675-21missed_timers21:50:011
179940,0migration/19655-21aten_r1power_fr19:24:481
179940,0migration/14927-21latency20:24:551
179940,0migration/130383-21unixbench-2d19:35:051
179940,0migration/12971-21aten_r1power_vo20:49:501
179940,0migration/127472-21munin-node21:59:571
179940,0migration/125015-21munin-node21:24:501
179940,0migration/124641-21fschecks_time22:19:521
179940,0migration/121931-21kthreadcore23:29:541
179940,0migration/116670-21kthreadcore19:04:561
179940,0migration/111727-21munin-node23:05:041
169940,0watchdog/1549-21kthreadcore21:44:561
169940,0watchdog/120433-21perl23:49:481
169940,0watchdog/117794-21cut21:19:561
169940,0watchdog/117670-21fschecks_count20:14:491
169940,0watchdog/112982-21ssh00:04:401
169940,0watchdog/10-21swapper/123:39:481
169940,0watchdog/10-21swapper/122:29:481
169940,0watchdog/10-21swapper/122:14:561
169940,0watchdog/10-21swapper/119:54:531
149940,0migration/031020-21if_err_eth000:14:500
149940,0migration/021080-21kthreadcore23:29:540
149940,0migration/01395-21sh23:38:380
8315030,0irq/26-0000:00:0-21swapper/220:34:362
329930,0watchdog/30-21swapper/320:09:573
329930,0watchdog/30-21swapper/320:04:493
329930,0watchdog/30-21swapper/319:50:053
308319932,0cyclictest8409-21grep18:59:532
308319932,0cyclictest3599-21lldpd20:39:552
308319932,0cyclictest3599-21lldpd20:28:252
308319932,0cyclictest3599-21lldpd19:24:552
308319932,0cyclictest10881-21awk20:09:522
308309932,0cyclictest3599-21lldpd20:00:251
308309932,0cyclictest3262-21irqbalance22:24:401
308309932,0cyclictest31629-21tune2fs19:39:521
308309932,0cyclictest28641-21awk19:14:521
308309932,0cyclictest23871-21snmpd23:44:421
308309931,0cyclictest16729-21latency_hist19:29:361
308299933,0cyclictest5370-21kworker/0:121:14:470
308299933,0cyclictest3080-21kworker/0:023:49:360
308299933,0cyclictest30030-21kworker/0:019:54:370
308299933,0cyclictest30030-21kworker/0:019:39:510
308299933,0cyclictest30030-21kworker/0:019:29:450
308299933,0cyclictest10744-21kworker/0:123:19:360
308299933,0cyclictest10744-21kworker/0:122:49:380
308299933,0cyclictest10744-21kworker/0:122:34:370
308299933,0cyclictest10744-21kworker/0:122:29:490
308299933,0cyclictest10744-21kworker/0:121:49:490
29925030,0irq/32-eth0-rx-0-21swapper/320:19:543
249930,0watchdog/229172-21ps20:44:532
329920,0watchdog/30-21swapper/320:34:413
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional