You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-02-24 - 00:03
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2.osadl.org (updated Fri Feb 23, 2024 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2733624533,0sleep10-21swapper/106:56:361
2730024030,0sleep30-21swapper/306:56:103
2055124027,0sleep20-21swapper/206:53:292
2726523524,0sleep00-21swapper/006:55:470
275649983,0cyclictest9300-21awk11:33:530
275649983,0cyclictest29210-21awk09:03:530
275649983,0cyclictest19484-21awk10:03:520
275649980,0cyclictest0-21swapper/009:48:260
275649975,0cyclictest0-21swapper/009:34:070
275649973,0cyclictest23828-21grep08:38:460
275649972,0cyclictest31534-21sadc11:28:250
275649972,0cyclictest13518-21awk09:13:520
275649970,0cyclictest26831-21munin-node08:19:030
275649970,0cyclictest0-21swapper/012:25:480
275649970,0cyclictest0-21swapper/012:11:550
8315060,0irq/26-0000:00:18110-21awk10:03:252
339960,0migration/39055-21sh11:51:053
339960,0migration/33551-21kthreadcore09:33:533
339960,0migration/317199-21sh11:54:493
339960,0migration/316097-21aten_r1power_cu08:33:483
29925060,0irq/32-eth0-rx-31307-21memory11:43:543
29925060,0irq/32-eth0-rx-0-21swapper/312:03:583
29925060,0irq/32-eth0-rx-0-21swapper/309:44:513
275659960,0cyclictest0-21swapper/110:09:001
275649961,0cyclictest0-21swapper/007:03:260
275649960,0cyclictest0-21swapper/007:46:070
8315050,0irq/26-0000:00:30264-21crond08:43:262
8315050,0irq/26-0000:00:21169-21latency_hist10:43:252
339950,0migration/39689-21munin-node12:08:563
339950,0migration/36892-21taskset07:33:173
339950,0migration/35116-21taskset08:48:153
339950,0migration/34005-21kthreadcore09:53:553
339950,0migration/324120-21kthreadcore10:03:563
339950,0migration/322308-21latency_hist11:03:253
339950,0migration/318708-21kthreadcore10:58:563
339950,0migration/315848-21fschecks_count11:18:583
339950,0migration/314191-21kthreadcore09:58:543
339950,0migration/311595-21diskmemload10:35:203
339950,0migration/310714-21kthreadcore11:33:563
339950,0migration/310674-21kthreadcore10:53:553
329950,0watchdog/329588-21cut08:18:543
329950,0watchdog/32675-21users10:29:023
329950,0watchdog/323818-21cut07:53:513
329950,0watchdog/31829-21cut11:28:503
329950,0watchdog/312040-21cut09:38:543
329950,0watchdog/30-21swapper/310:48:543
29935050,0irq/33-eth0-tx-4884-21kthreadcore11:48:532
29935050,0irq/33-eth0-tx-15873-21kthreadcore10:58:552
29935050,0irq/33-eth0-tx-0-21swapper/212:13:562
29935050,0irq/33-eth0-tx-0-21swapper/211:33:472
29935050,0irq/33-eth0-tx-0-21swapper/210:49:022
29935050,0irq/33-eth0-tx-0-21swapper/209:36:532
29935050,0irq/33-eth0-tx-0-21swapper/209:31:432
29935050,0irq/33-eth0-tx-0-21swapper/209:31:432
29925050,0irq/32-eth0-rx-5301-21taskset12:28:103
29925050,0irq/32-eth0-rx-32748-21cat10:08:583
29925050,0irq/32-eth0-rx-28958-21ssh10:47:003
29925050,0irq/32-eth0-rx-13204-21munin-node12:13:503
29925050,0irq/32-eth0-rx-0-21swapper/311:59:523
29925050,0irq/32-eth0-rx-0-21swapper/311:38:513
29925050,0irq/32-eth0-rx-0-21swapper/311:25:023
29925050,0irq/32-eth0-rx-0-21swapper/311:13:343
29925050,0irq/32-eth0-rx-0-21swapper/311:09:023
29925050,0irq/32-eth0-rx-0-21swapper/310:18:563
29925050,0irq/32-eth0-rx-0-21swapper/309:25:493
29925050,0irq/32-eth0-rx-0-21swapper/309:13:413
29925050,0irq/32-eth0-rx-0-21swapper/309:08:523
29925050,0irq/32-eth0-rx-0-21swapper/307:36:133
275669955,0cyclictest0-21swapper/209:53:532
275669950,0cyclictest0-21swapper/210:11:322
275659955,0cyclictest0-21swapper/111:22:191
275659955,0cyclictest0-21swapper/111:03:241
275659955,0cyclictest0-21swapper/107:31:491
275659950,0cyclictest0-21swapper/112:27:161
275659950,0cyclictest0-21swapper/111:08:171
275649955,0cyclictest0-21swapper/012:02:470
275649955,0cyclictest0-21swapper/010:58:080
275649950,0cyclictest22895-21munin-run07:53:260
275649950,0cyclictest0-21swapper/011:18:000
275649950,0cyclictest0-21swapper/010:51:410
259950,0migration/23648-21unixbench-2d12:23:592
259950,0migration/228198-21kthreadcore09:48:502
259950,0migration/223409-21aten_r1power_cu11:23:432
259950,0migration/217814-21kthreadcore07:48:542
259950,0migration/21670-21aten_r1power_fr08:23:452
179950,0migration/18089-21timerwakeupswit08:24:011
179950,0migration/15926-21munin-node08:48:471
179950,0migration/132174-21taskset09:49:041
179950,0migration/128717-21kthreadcore08:18:541
179950,0migration/126755-21missed_timers12:18:561
179950,0migration/122531-21kthreadcore08:58:561
179950,0migration/120307-21taskset09:17:571
179950,0migration/118194-21df_abs07:18:481
14471250,0sleep216422-21grep10:38:522
8315040,0irq/26-0000:00:6386-21runrttasks08:28:323
8315040,0irq/26-0000:00:0-21swapper/307:03:523
339940,0migration/36863-21kthreadcore08:03:593
339940,0migration/35028-21kthreadcore08:23:523
339940,0migration/330325-21taskset07:58:403
339940,0migration/320681-21sh12:18:373
339940,0migration/320152-21cpuspeed_turbos08:58:493
339940,0migration/315268-21taskset07:44:023
339940,0migration/312245-21taskset08:51:393
329940,0watchdog/328453-21kthreadcore09:48:503
329940,0watchdog/319522-21kthreadcore07:48:553
329940,0watchdog/311720-21if_err_eth1.307:13:513
329940,0watchdog/30-21swapper/308:38:503
329940,0watchdog/30-21swapper/308:08:503
329940,0watchdog/30-21swapper/307:42:113
29935040,0irq/33-eth0-tx-9784-21ssh10:14:152
29935040,0irq/33-eth0-tx-23899-21kthreadcore11:58:552
29935040,0irq/33-eth0-tx-19610-21diskmemload10:30:332
29935040,0irq/33-eth0-tx-0-21swapper/212:18:432
29935040,0irq/33-eth0-tx-0-21swapper/212:03:302
29935040,0irq/33-eth0-tx-0-21swapper/211:53:502
29935040,0irq/33-eth0-tx-0-21swapper/211:43:272
29935040,0irq/33-eth0-tx-0-21swapper/211:29:012
29935040,0irq/33-eth0-tx-0-21swapper/211:19:052
29935040,0irq/33-eth0-tx-0-21swapper/211:08:332
29935040,0irq/33-eth0-tx-0-21swapper/211:03:502
29935040,0irq/33-eth0-tx-0-21swapper/210:54:162
29935040,0irq/33-eth0-tx-0-21swapper/210:03:282
29935040,0irq/33-eth0-tx-0-21swapper/209:43:452
29935040,0irq/33-eth0-tx-0-21swapper/209:39:332
29935040,0irq/33-eth0-tx-0-21swapper/209:24:142
29925040,0irq/32-eth0-rx-0-21swapper/310:38:283
29925040,0irq/32-eth0-rx-0-21swapper/310:23:533
29925040,0irq/32-eth0-rx-0-21swapper/310:13:413
29925040,0irq/32-eth0-rx-0-21swapper/309:28:293
29925040,0irq/32-eth0-rx-0-21swapper/309:28:293
29925040,0irq/32-eth0-rx-0-21swapper/309:20:103
29925040,0irq/32-eth0-rx-0-21swapper/309:03:323
275669944,0cyclictest0-21swapper/210:23:312
275669943,0cyclictest6501-21awk08:48:492
275669943,0cyclictest4711-21awk12:08:512
275669943,0cyclictest3599-21lldpd10:18:482
275669943,0cyclictest3599-21lldpd09:19:112
275669943,0cyclictest3599-21lldpd09:08:362
275669943,0cyclictest3599-21lldpd08:36:572
275669943,0cyclictest3599-21lldpd07:45:442
275669943,0cyclictest3599-21lldpd07:42:432
275669943,0cyclictest3599-21lldpd07:35:432
275669943,0cyclictest3599-21lldpd07:28:402
275669943,0cyclictest3599-21lldpd07:03:472
275669943,0cyclictest3536-21snmpd08:20:222
275669943,0cyclictest3536-21snmpd08:03:372
275669943,0cyclictest3536-21snmpd07:22:222
275669943,0cyclictest3536-21snmpd07:12:372
275669943,0cyclictest29662-21ps09:03:552
275669943,0cyclictest20958-21awk08:58:522
275669943,0cyclictest13547-21awk08:08:532
275669943,0cyclictest12417-21munin-node09:13:492
275669940,0cyclictest8615-21cut11:13:512
275659944,0cyclictest0-21swapper/110:42:451
275659944,0cyclictest0-21swapper/109:57:581
275659944,0cyclictest0-21swapper/109:26:261
275659943,0cyclictest3599-21lldpd10:53:491
275659943,0cyclictest3599-21lldpd10:14:481
275659943,0cyclictest3599-21lldpd09:45:131
275659943,0cyclictest3599-21lldpd09:34:111
275659943,0cyclictest3599-21lldpd08:03:571
275659943,0cyclictest32016-21sadc07:28:261
275659943,0cyclictest24469-21awk11:23:511
275659943,0cyclictest20816-21awk08:13:521
275659943,0cyclictest19531-21awk10:03:531
275659943,0cyclictest16692-21sed09:59:031
275659943,0cyclictest13701-21awk08:53:501
275649943,0cyclictest543-21awk11:28:480
275649943,0cyclictest3536-21snmpd11:09:080
275649943,0cyclictest3536-21snmpd10:26:380
275649943,0cyclictest3536-21snmpd10:20:080
275649943,0cyclictest3536-21snmpd09:57:370
275649943,0cyclictest3536-21snmpd09:00:370
275649943,0cyclictest3536-21snmpd08:53:370
275649943,0cyclictest3262-21irqbalance10:14:100
275649943,0cyclictest3262-21irqbalance09:39:000
275649943,0cyclictest3262-21irqbalance09:12:200
275649943,0cyclictest3262-21irqbalance07:38:300
275649943,0cyclictest3262-21irqbalance07:33:400
275649943,0cyclictest3262-21irqbalance07:31:200
275649943,0cyclictest31567-21awk08:43:510
275649943,0cyclictest29385-21sadc10:48:250
275649943,0cyclictest28588-21awk12:03:520
275649940,0cyclictest5209-21munin-node08:03:540
275649940,0cyclictest30333-21munin-node07:58:540
275649940,0cyclictest1592-21munin-node08:23:520
275649940,0cyclictest1161-21munin-run11:48:250
259940,0migration/29119-21cpuspeed_turbos08:28:472
259940,0migration/26151-21kthreadcore10:33:582
259940,0migration/231221-21kthreadcore06:58:542
259940,0migration/222565-21kthreadcore08:13:542
259940,0migration/214121-21kthreadcore07:13:532
249940,0watchdog/231512-21idleruntime08:43:502
249940,0watchdog/225418-21kthreadcore07:53:552
249940,0watchdog/22176-21kthreadcore07:58:552
249940,0watchdog/20-21swapper/211:38:462
179940,0migration/16494-21kthreadcore10:33:581
179940,0migration/15886-21kthreadcore11:48:541
179940,0migration/14572-21idleruntime12:08:501
179940,0migration/13244-21fschecks_count07:03:511
179940,0migration/132276-21df_abs11:28:451
179940,0migration/13109-21kthreadcore11:08:531
179940,0migration/125087-21fschecks_time09:28:481
179940,0migration/125087-21fschecks_time09:28:481
179940,0migration/119848-21sh10:23:261
179940,0migration/118927-21kthreadcore07:48:541
179940,0migration/114921-21kthreadcore08:08:551
179940,0migration/11489-21kthreadcore08:43:541
179940,0migration/114533-21switchtime11:34:041
179940,0migration/113187-21kthreadcore10:18:551
179940,0migration/112803-21missed_timers11:13:531
179940,0migration/111685-21irqrtprio11:53:521
169940,0watchdog/132448-21ssh11:45:141
169940,0watchdog/130721-21cut12:03:541
169940,0watchdog/125016-21open_inodes11:58:581
169940,0watchdog/122563-21iostat_ios10:43:501
169940,0watchdog/115693-21kthreadcore12:13:541
169940,0watchdog/114784-21chrt07:42:111
169940,0watchdog/10-21swapper/109:38:541
169940,0watchdog/10-21swapper/107:58:551
169940,0watchdog/10-21swapper/107:53:511
8315030,0irq/26-0000:00:0-21swapper/308:13:363
329930,0watchdog/30-21swapper/308:53:463
329930,0watchdog/30-21swapper/306:58:353
29925030,0irq/32-eth0-rx-0-21swapper/307:28:203
29925030,0irq/32-eth0-rx-0-21swapper/307:18:463
275669933,0cyclictest3599-21lldpd07:23:362
275669932,0cyclictest3599-21lldpd08:53:272
275659932,0cyclictest5185-21awk09:08:501
275659932,0cyclictest3599-21lldpd07:35:131
275659932,0cyclictest20429-21cut08:33:591
275659932,0cyclictest15655-21sadc07:48:261
275659932,0cyclictest12602-21kthreadcore08:28:531
275659930,0cyclictest28528-21perl06:58:501
275649933,0cyclictest0-21swapper/010:32:450
275649932,0cyclictest7618-21kthreadcore07:03:560
275649932,0cyclictest3599-21lldpd12:13:480
275649932,0cyclictest3599-21lldpd08:28:570
275649932,0cyclictest3536-21snmpd11:18:530
275649932,0cyclictest3536-21snmpd10:36:080
275649932,0cyclictest3536-21snmpd09:28:220
275649932,0cyclictest3536-21snmpd09:18:370
275649932,0cyclictest3536-21snmpd08:50:370
275649932,0cyclictest29956-21grep09:48:510
275649932,0cyclictest27676-21fschecks_time10:08:520
275649932,0cyclictest26075-21awk07:23:500
275649932,0cyclictest2572-21awk11:48:500
275649932,0cyclictest25247-21awk09:28:490
275649932,0cyclictest25247-21awk09:28:490
275649932,0cyclictest24242-21awk07:53:530
275649932,0cyclictest23049-21sensors11:38:570
275649932,0cyclictest20889-21basename12:18:480
275649932,0cyclictest20407-21sensors10:59:010
275649932,0cyclictest19610-21diskmemload11:04:300
275649932,0cyclictest19416-21uname07:18:530
275649932,0cyclictest18602-21cut08:33:580
275649932,0cyclictest14364-21ps10:38:510
275649932,0cyclictest12410-21munin-node08:08:500
275649932,0cyclictest11241-21awk11:53:500
275649930,0cyclictest0-21swapper/007:13:520
275649930,0cyclictest0-21swapper/007:08:500
169930,0watchdog/12688-21ssh09:05:061
169930,0watchdog/10-21swapper/111:38:461
169930,0watchdog/10-21swapper/110:48:541
169930,0watchdog/10-21swapper/110:29:101
169930,0watchdog/10-21swapper/109:22:501
169930,0watchdog/10-21swapper/108:38:501
169930,0watchdog/10-21swapper/107:13:511
329920,0watchdog/30-21swapper/307:08:273
275649921,0cyclictest9780-21sadc09:58:260
275649920,0cyclictest0-21swapper/008:13:260
169920,0watchdog/10-21swapper/107:08:271
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional