You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-28 - 19:36
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 100 highest latencies:
System rack1slot2.osadl.org (updated Thu Sep 28, 2023 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1406525039,0sleep20-21swapper/207:04:582
1411824939,0sleep30-21swapper/307:05:353
1425723627,0sleep10-21swapper/107:07:131
1420123625,0sleep00-21swapper/007:06:320
35160,0ktimersoftd/30-21swapper/308:32:423
339960,0migration/332514-21kthreadcore08:02:463
339960,0migration/320518-21missed_timers10:42:473
339960,0migration/315505-21sh11:53:493
339960,0migration/310218-21fschecks_time09:42:413
29925060,0irq/32-eth0-rx-21579-21iostat_ios12:17:423
29925060,0irq/32-eth0-rx-17307-21aten_r1power_en11:57:393
143389960,0cyclictest0-21swapper/210:17:492
149960,0migration/07518-21sh11:51:020
149960,0migration/03798-21latency_hist07:22:200
149960,0migration/031401-21sh09:33:040
149960,0migration/02568-21kthreadcore09:17:490
149960,0migration/02085-21aten_r1power_fr11:12:370
149960,0migration/012482-21sh11:34:120
8315050,0irq/26-0000:00:28298-21crond11:27:192
339950,0migration/39835-21kthreadcore08:07:493
339950,0migration/3676-21munin-node10:52:443
339950,0migration/332297-21taskset09:35:383
339950,0migration/332174-21missed_timers12:02:493
339950,0migration/330018-21hddtemp_smartct07:17:483
339950,0migration/328268-21mdstat09:12:443
339950,0migration/328192-21munin-node07:57:483
339950,0migration/327770-21kthreadcore11:42:463
339950,0migration/323097-21kthreadcore10:27:443
339950,0migration/319714-21aten_r1power_en10:07:403
339950,0migration/316261-21if_eth1.307:52:443
339950,0migration/314989-21munin-node07:07:433
339950,0migration/312078-21kthreadcore10:57:423
339950,0migration/311554-21kthreadcore09:22:443
339950,0migration/310938-21latency_hist10:02:203
339950,0migration/310938-21latency_hist10:02:203
339950,0migration/310801-21rt-features10:17:523
329950,0watchdog/35148-21interrupts07:22:443
329950,0watchdog/33410-21ssh10:15:283
329950,0watchdog/332288-21fschecks_count11:47:403
329950,0watchdog/32927-21ntpq12:22:483
329950,0watchdog/322400-21grep08:17:483
329950,0watchdog/319632-21iostat07:32:443
329950,0watchdog/31801-21irqcore07:42:443
329950,0watchdog/31500-21fschecks_time09:37:443
329950,0watchdog/31472-21ssh09:54:203
329950,0watchdog/30-21swapper/309:07:443
329950,0watchdog/30-21swapper/308:12:523
29935050,0irq/33-eth0-tx-5089-21ssh10:35:252
29935050,0irq/33-eth0-tx-3969-21munin-node07:22:420
29935050,0irq/33-eth0-tx-26107-21munin-node12:02:552
29935050,0irq/33-eth0-tx-22319-21munin-node07:57:410
29935050,0irq/33-eth0-tx-18469-21ps12:13:262
29935050,0irq/33-eth0-tx-12953-21munin-node10:22:432
29935050,0irq/33-eth0-tx-0-21swapper/209:59:082
29935050,0irq/33-eth0-tx-0-21swapper/209:12:512
29925050,0irq/32-eth0-rx-21968-21sh11:40:423
29925050,0irq/32-eth0-rx-19689-21ssh11:22:213
29925050,0irq/32-eth0-rx-0-21swapper/312:29:393
29925050,0irq/32-eth0-rx-0-21swapper/312:13:593
29925050,0irq/32-eth0-rx-0-21swapper/311:32:393
29925050,0irq/32-eth0-rx-0-21swapper/311:10:133
29925050,0irq/32-eth0-rx-0-21swapper/311:04:343
29925050,0irq/32-eth0-rx-0-21swapper/310:47:463
29925050,0irq/32-eth0-rx-0-21swapper/310:37:473
29925050,0irq/32-eth0-rx-0-21swapper/309:29:293
29925050,0irq/32-eth0-rx-0-21swapper/309:17:313
27150,0ktimersoftd/20-21swapper/212:27:082
27150,0ktimersoftd/20-21swapper/208:32:422
259950,0migration/29651-21sh11:14:532
259950,0migration/29322-21latency_hist09:42:202
259950,0migration/29118-21latency_hist08:37:202
259950,0migration/2588-21kthreadcore08:52:472
259950,0migration/232704-21fschecks_time10:52:412
259950,0migration/231230-21sh11:46:432
259950,0migration/224517-21sh11:59:112
259950,0migration/222927-21kthreadcore10:07:482
259950,0migration/222355-21kthreadcore08:17:482
259950,0migration/221202-21timerwakeupswit12:32:522
259950,0migration/218140-21kthreadcore10:42:452
259950,0migration/216436-21fschecks_count09:27:392
259950,0migration/213830-21sh09:22:522
259950,0migration/211053-21latency_hist07:27:202
249950,0watchdog/223347-21cut12:17:442
19150,0ktimersoftd/10-21swapper/112:19:111
19150,0ktimersoftd/10-21swapper/108:32:421
179950,0migration/13565-21kthreadcore08:52:481
179950,0migration/128573-21munin-node10:12:421
179950,0migration/127103-21irqrtprio07:37:451
179950,0migration/122784-21taskset12:36:301
179950,0migration/121537-21kthreadcore11:22:451
179950,0migration/117116-21latency_hist11:57:191
179950,0migration/116681-21kthreadcore11:37:471
143389954,0cyclictest83150irq/26-0000:00:07:31:332
143389954,0cyclictest440-21crond09:37:202
143389954,0cyclictest0-21swapper/209:52:452
143379955,0cyclictest0-21swapper/111:43:241
143379954,0cyclictest3536-21snmpd08:22:201
143379950,0cyclictest0-21swapper/112:12:241
143379950,0cyclictest0-21swapper/112:03:251
143379950,0cyclictest0-21swapper/111:21:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional