You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-25 - 15:41
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Thu Apr 25, 2024 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2678923929,0sleep10-21swapper/106:54:391
2645023826,0sleep20-21swapper/206:50:442
2644023728,0sleep30-21swapper/306:50:373
1990223625,0sleep00-21swapper/006:49:460
4160,0ktimersoftd/00-21swapper/010:09:520
339960,0migration/327462-21sh09:45:253
339960,0migration/322878-21switchtime11:35:143
339960,0migration/318401-21sh11:54:413
339960,0migration/315961-21sh09:03:593
339960,0migration/315961-21sh09:03:593
339960,0migration/310417-21sh10:52:293
329960,0watchdog/332203-21ps11:41:063
329960,0watchdog/318906-21ssh10:56:543
29925060,0irq/32-eth0-rx-31956-21perl10:09:553
268649960,0cyclictest0-21swapper/207:24:482
268639960,0cyclictest0-21swapper/111:50:091
268629963,0cyclictest7673-21basename08:59:420
8315050,0irq/26-0000:00:24261-21turbostat.cron08:49:422
4150,0ktimersoftd/00-21swapper/011:28:320
4150,0ktimersoftd/00-21swapper/009:12:130
35150,0ktimersoftd/30-21swapper/311:28:323
339950,0migration/38609-21unixbench_multi07:45:143
339950,0migration/37976-21kthreadcore08:10:003
339950,0migration/35598-21kthreadcore12:05:083
339950,0migration/3406-21munin-node10:45:083
339950,0migration/328681-21kthreadcore06:55:003
339950,0migration/327463-21kthreadcore07:14:593
339950,0migration/324913-21kthreadcore08:20:033
339950,0migration/323332-21fschecks_time10:04:543
339950,0migration/322529-21irqrtprio12:15:023
339950,0migration/321404-21kthreadcore11:55:023
339950,0migration/32041-21taskset09:31:343
339950,0migration/317577-21kthreadcore08:14:593
339950,0migration/316919-21cpuspeed_turbos08:39:523
339950,0migration/315777-21kthreadcore07:30:023
339950,0migration/314535-21fschecks_count09:59:553
339950,0migration/313177-21munin-node09:39:593
339950,0migration/311135-21kthreadcore07:50:003
329950,0watchdog/39866-21diskstats08:34:583
329950,0watchdog/34042-21yum12:20:133
329950,0watchdog/330480-21kthreadcore12:00:063
329950,0watchdog/329915-21python08:00:143
329950,0watchdog/323903-21ssh11:20:023
329950,0watchdog/320069-21grep10:39:583
329950,0watchdog/318488-21kthreadcore10:20:023
329950,0watchdog/315516-21ls07:05:103
329950,0watchdog/314204-21aten_r1power_vo11:14:543
329950,0watchdog/31246-21cut08:05:023
329950,0watchdog/310643-21fschecks_count09:19:583
29935050,0irq/33-eth0-tx-1880-21munin-node07:00:120
29935050,0irq/33-eth0-tx-14052-21munin-node11:15:092
29935050,0irq/33-eth0-tx-0-21swapper/209:41:102
29925050,0irq/32-eth0-rx-5632-21kthreadcore09:35:013
29925050,0irq/32-eth0-rx-31873-21grep11:05:013
29925050,0irq/32-eth0-rx-27748-21ssh11:02:153
29925050,0irq/32-eth0-rx-11882-21python09:55:083
29925050,0irq/32-eth0-rx-10084-21fschecks_count10:34:553
29925050,0irq/32-eth0-rx-0-21swapper/310:33:483
29925050,0irq/32-eth0-rx-0-21swapper/310:26:423
29925050,0irq/32-eth0-rx-0-21swapper/310:15:183
29925050,0irq/32-eth0-rx-0-21swapper/309:49:473
29925050,0irq/32-eth0-rx-0-21swapper/309:24:533
29925050,0irq/32-eth0-rx-0-21swapper/309:09:533
268649955,0cyclictest0-21swapper/212:10:192
268649955,0cyclictest0-21swapper/211:11:492
268649955,0cyclictest0-21swapper/210:35:242
268649955,0cyclictest0-21swapper/210:33:402
268649955,0cyclictest0-21swapper/210:01:492
268649954,0cyclictest23871-21snmpd10:58:472
268649950,0cyclictest0-21swapper/212:23:142
268649950,0cyclictest0-21swapper/212:01:192
268649950,0cyclictest0-21swapper/210:15:182
268649950,0cyclictest0-21swapper/208:55:362
268639955,0cyclictest0-21swapper/111:38:481
268639955,0cyclictest0-21swapper/109:05:151
268639950,0cyclictest0-21swapper/112:20:451
268629955,0cyclictest0-21swapper/010:28:120
268629954,0cyclictest23871-21snmpd10:53:320
268629950,0cyclictest0-21swapper/011:06:020
249950,0watchdog/20-21swapper/211:47:382
19150,0ktimersoftd/10-21swapper/111:39:551
179950,0migration/18068-21kthreadcore10:50:061
179950,0migration/132525-21fschecks_time08:54:561
179950,0migration/13190-21irqcore07:00:001
179950,0migration/124945-21kthreadcore11:00:041
179950,0migration/124922-21kthreadcore11:20:051
179950,0migration/114910-21fschecks_time11:14:561
179950,0migration/113462-21missed_timers10:15:031
169950,0watchdog/130479-21kthreadcore12:00:061
169950,0watchdog/11875-21kthreadcore07:20:021
169950,0watchdog/115931-21kthreadcore08:14:581
149950,0migration/0986-21taskset11:44:260
149950,0migration/09168-21sh09:17:460
149950,0migration/06277-21fschecks_count08:09:550
149950,0migration/02745-21kthreadcore09:50:030
149950,0migration/024577-21kthreadcore11:20:050
149950,0migration/023986-21kthreadcore12:15:030
149950,0migration/022532-21switchtime10:20:110
149950,0migration/021906-21kthreadcore07:35:000
149950,0migration/020261-21aten_r1power_cu10:59:520
149950,0migration/015819-21kthreadcore08:14:580
149950,0migration/011886-21sendmail_mailtr07:25:080
8315040,0irq/26-0000:00:22885-21unixbench_multi07:10:152
339940,0migration/326799-21taskset07:37:283
339940,0migration/321187-21kthreadcore09:05:003
339940,0migration/31724-21taskset07:44:063
339940,0migration/316318-21latency_hist07:09:423
339940,0migration/31323-21kthreadcore08:55:023
329940,0watchdog/39204-21kthreadcore11:10:023
329940,0watchdog/30-21swapper/311:44:423
329940,0watchdog/30-21swapper/307:25:023
29935040,0irq/33-eth0-tx-8716-21fschecks_time08:59:542
29935040,0irq/33-eth0-tx-8716-21fschecks_time08:59:542
29935040,0irq/33-eth0-tx-31451-21diskmemload11:51:202
29935040,0irq/33-eth0-tx-31451-21diskmemload11:25:242
29935040,0irq/33-eth0-tx-18540-21ssh10:39:512
29935040,0irq/33-eth0-tx-10456-21ssh12:05:232
29935040,0irq/33-eth0-tx-0-21swapper/210:49:592
29935040,0irq/33-eth0-tx-0-21swapper/210:48:092
29935040,0irq/33-eth0-tx-0-21swapper/210:21:382
29935040,0irq/33-eth0-tx-0-21swapper/210:05:042
29935040,0irq/33-eth0-tx-0-21swapper/209:54:482
29935040,0irq/33-eth0-tx-0-21swapper/209:26:032
29935040,0irq/33-eth0-tx-0-21swapper/209:19:582
29935040,0irq/33-eth0-tx-0-21swapper/209:05:452
29935040,0irq/33-eth0-tx-0-21swapper/008:40:020
29935040,0irq/33-eth0-tx-0-21swapper/008:29:560
29935040,0irq/33-eth0-tx-0-21swapper/008:24:500
29925040,0irq/32-eth0-rx-0-21swapper/312:09:423
29925040,0irq/32-eth0-rx-0-21swapper/311:31:553
29925040,0irq/32-eth0-rx-0-21swapper/309:15:113
29925040,0irq/32-eth0-rx-0-21swapper/307:20:063
268649944,0cyclictest0-21swapper/207:55:132
268649943,0cyclictest8898-21awk11:29:592
268649943,0cyclictest4686-21awk09:34:592
268649943,0cyclictest3599-21lldpd08:30:502
268649943,0cyclictest3599-21lldpd08:24:502
268649943,0cyclictest3599-21lldpd08:22:492
268649943,0cyclictest3599-21lldpd07:52:262
268649943,0cyclictest3599-21lldpd07:40:102
268649943,0cyclictest3599-21lldpd07:15:182
268649943,0cyclictest31915-21awk08:04:582
268649943,0cyclictest31908-21awk07:19:592
268649943,0cyclictest26388-21awk09:10:002
268649943,0cyclictest25893-21awk10:24:572
268649943,0cyclictest2566-21awk09:14:582
268649943,0cyclictest23871-21snmpd11:20:472
268649943,0cyclictest23871-21snmpd09:30:012
268649943,0cyclictest23871-21snmpd07:32:002
268649943,0cyclictest22023-21awk12:14:592
268649943,0cyclictest21753-21awk09:44:572
268649943,0cyclictest14085-21awk08:14:562
268639944,0cyclictest0-21swapper/111:05:131
268639944,0cyclictest0-21swapper/110:41:531
268639944,0cyclictest0-21swapper/109:53:291
268639944,0cyclictest0-21swapper/109:34:151
268639944,0cyclictest0-21swapper/108:20:091
268639943,0cyclictest8999-21awk08:59:561
268639943,0cyclictest8999-21awk08:59:561
268639943,0cyclictest32537-21awk11:25:011
268639943,0cyclictest3210-21awk08:29:561
268639943,0cyclictest28308-21awk07:39:561
268639943,0cyclictest25792-21awk10:24:571
268639943,0cyclictest23871-21snmpd11:32:321
268639943,0cyclictest23871-21snmpd10:09:321
268639943,0cyclictest23871-21snmpd09:15:011
268639943,0cyclictest23871-21snmpd08:53:311
268639943,0cyclictest21117-21awk07:34:571
268639943,0cyclictest19803-21awk11:55:001
268639943,0cyclictest13946-21awk07:29:591
268639943,0cyclictest10819-21awk10:35:001
268629944,0cyclictest0-21swapper/012:13:590
268629944,0cyclictest0-21swapper/010:34:020
268629944,0cyclictest0-21swapper/009:41:420
268629944,0cyclictest0-21swapper/009:24:490
268629943,0cyclictest8947-21awk11:29:590
268629943,0cyclictest3599-21lldpd11:50:230
268629943,0cyclictest3599-21lldpd08:50:390
268629943,0cyclictest28356-21awk07:39:560
268629943,0cyclictest23871-21snmpd12:21:320
268629943,0cyclictest23871-21snmpd12:07:320
268629943,0cyclictest23871-21snmpd07:20:450
268629943,0cyclictest23871-21snmpd07:10:000
268629943,0cyclictest17651-21awk11:35:020
268629943,0cyclictest15279-21awk11:15:000
268629943,0cyclictest15132-21awk09:59:580
268629941,0cyclictest0-21swapper/008:34:550
268629940,0cyclictest12879-21munin-node07:29:570
268629940,0cyclictest0-21swapper/009:04:390
268629940,0cyclictest0-21swapper/009:04:390
259940,0migration/25169-21switchtime10:10:092
259940,0migration/230231-21kthreadcore08:50:022
259940,0migration/229470-21latency08:00:072
259940,0migration/221999-21kthreadcore11:00:032
259940,0migration/217075-21fschecks_count11:34:572
249940,0watchdog/226972-21grep11:39:582
249940,0watchdog/2262582sleep207:35:102
249940,0watchdog/218825-21aten_r1power_vo11:54:542
249940,0watchdog/20-21swapper/211:05:022
249940,0watchdog/20-21swapper/209:50:022
249940,0watchdog/20-21swapper/208:39:542
19140,0ktimersoftd/10-21swapper/107:54:071
179940,0migration/17363-21rt-features10:30:081
179940,0migration/130505-21kthreadcore06:55:011
179940,0migration/130490-21kthreadcore08:24:591
179940,0migration/12587-21runrttasks11:44:581
179940,0migration/123838-21ksmtuned08:45:041
179940,0migration/123719-21kthreadcore12:15:031
179940,0migration/123641-21aten_r1power_en07:14:511
179940,0migration/120384-21kthreadcore07:55:031
179940,0migration/118764-21kthreadcore07:10:031
179940,0migration/116730-21aten_r1power_po08:39:521
179940,0migration/116555-21kthreadcore12:10:021
179940,0migration/11562-21kthreadcore10:10:001
179940,0migration/113645-21kthreadcore07:05:011
169940,0watchdog/16386-21runrttasks07:49:301
169940,0watchdog/14652-21sh12:05:021
169940,0watchdog/131451-21diskmemload09:55:101
169940,0watchdog/126117-21cut08:00:061
169940,0watchdog/125509-21ssh09:25:141
169940,0watchdog/118489-21kthreadcore10:20:021
169940,0watchdog/114966-21kthreadcore09:40:021
169940,0watchdog/111862-21munin-node08:10:061
169940,0watchdog/110638-21ssh09:36:181
169940,0watchdog/10-21swapper/110:59:261
169940,0watchdog/10-21swapper/109:19:581
169940,0watchdog/10-21swapper/109:10:021
169940,0watchdog/10-21swapper/108:05:021
169940,0watchdog/10-21swapper/107:25:021
149940,0migration/08020-21kthreadcore09:35:020
149940,0migration/06802-21kthreadcore07:45:060
149940,0migration/04507-21kthreadcore11:45:000
149940,0migration/030297-21kthreadcore06:55:010
149940,0migration/029829-21kthreadcore09:30:000
149940,0migration/027670-21cpuspeed_turbos11:59:560
149940,0migration/022383-21latency_hist10:04:410
149940,0migration/021908-21ksmtuned07:55:040
149940,0migration/020878-21fschecks_time08:19:580
149940,0migration/020461-21kthreadcore11:55:020
149940,0migration/018419-21runrttasks10:55:360
149940,0migration/017743-21sh09:22:360
149940,0migration/011233-21latency11:10:020
149940,0migration/010005-21kthreadcore10:15:010
329930,0watchdog/30-21swapper/308:46:143
329930,0watchdog/30-21swapper/308:29:583
329930,0watchdog/30-21swapper/308:24:423
329930,0watchdog/30-21swapper/306:59:543
29935030,0irq/33-eth0-tx-0-21swapper/008:04:530
29925030,0irq/32-eth0-rx-0-21swapper/307:54:543
268649932,0cyclictest7325-21kthreadcore07:00:022
268649932,0cyclictest31116-21cut06:55:012
268649931,0cyclictest0-21swapper/207:05:012
268639932,0cyclictest23871-21snmpd10:45:021
268639932,0cyclictest21711-21awk09:44:571
268629933,0cyclictest23481-21kworker/0:010:37:340
268629933,0cyclictest14277-21kworker/0:109:56:380
268629932,0cyclictest27744-21awk10:44:590
268629932,0cyclictest21829-21awk09:44:570
268629932,0cyclictest18610-21kthreadcore09:04:590
268629932,0cyclictest10281-21awk07:04:580
268629930,0cyclictest24909-21irqcore08:00:030
268629930,0cyclictest24487-21perl07:14:540
268629930,0cyclictest22343-21grep10:39:580
268629930,0cyclictest0-21swapper/007:49:500
259930,0migration/22280-21cpuspeed_turbos07:44:582
259930,0migration/211706-21kthreadcore08:35:042
179930,0migration/120810-21sh10:00:341
169930,0watchdog/10-21swapper/111:10:021
329920,0watchdog/30-21swapper/308:49:423
268639921,0cyclictest3599-21lldpd08:34:491
268629920,0cyclictest23733-21latency_hist08:44:420
249920,0watchdog/20-21swapper/208:09:462
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional