You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-26 - 16:08
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Fri Apr 26, 2024 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1372023928,0sleep30-21swapper/306:52:353
1355723624,0sleep00-21swapper/006:50:400
1389723523,0sleep10-21swapper/106:54:371
1372423425,0sleep20-21swapper/206:52:382
13976991010,0cyclictest25754-21sensors10:50:051
339960,0migration/39587-21cpuspeed_turbos10:24:493
339960,0migration/38813-21sh11:55:403
339960,0migration/36750-21sh10:01:583
339960,0migration/330599-21kthreadcore10:35:023
339960,0migration/326085-21latency_hist10:34:373
339960,0migration/320879-21sh09:50:043
339960,0migration/320879-21sh09:50:043
329960,0watchdog/39129-21awk10:40:063
29925060,0irq/32-eth0-rx-27241-21sleep08:59:383
29925060,0irq/32-eth0-rx-25765-21sensors10:50:053
149960,0migration/032370-21kthreadcore08:40:010
149960,0migration/021165-21sh12:23:090
149960,0migration/017834-21kthreadcore11:24:580
139769960,0cyclictest0-21swapper/111:44:051
339950,0migration/36561-21fschecks_time11:19:533
339950,0migration/36260-21kthreadcore12:14:593
339950,0migration/3571-21taskset07:50:013
339950,0migration/35519-21kthreadcore08:19:573
339950,0migration/35108-21munin-node08:44:503
339950,0migration/34584-21kthreadcore07:30:043
339950,0migration/332381-21kthreadcore09:00:013
339950,0migration/331125-21kthreadcore08:14:543
339950,0migration/330142-21taskset11:14:543
339950,0migration/329806-21munin-node09:19:563
339950,0migration/328482-21diskmemload11:12:393
339950,0migration/323836-21kthreadcore07:00:003
339950,0migration/320495-21sh09:31:013
339950,0migration/318091-21latency_hist08:04:383
339950,0migration/317579-21sh12:00:463
339950,0migration/314047-21kthreadcore11:04:553
339950,0migration/311756-21if_eth107:14:563
329950,0watchdog/330204-21basename09:39:463
329950,0watchdog/325874-21perl07:24:513
329950,0watchdog/323694-21interrupts08:34:543
329950,0watchdog/322792-21iostat_ios07:44:553
329950,0watchdog/311032-21ssh09:07:503
329950,0watchdog/310002-21ntp_states11:00:023
329950,0watchdog/30-21swapper/312:09:503
29935050,0irq/33-eth0-tx-23156-21munin-node09:54:562
29935050,0irq/33-eth0-tx-11463-21fschecks_time11:59:492
29935050,0irq/33-eth0-tx-0-21swapper/209:13:442
29935050,0irq/33-eth0-tx-0-21swapper/008:54:560
29935050,0irq/33-eth0-tx-0-21swapper/008:01:480
29935050,0irq/33-eth0-tx-0-21swapper/007:56:470
29935050,0irq/33-eth0-tx-0-21swapper/007:10:110
29925050,0irq/32-eth0-rx-9025-21if_err_eth011:39:523
29925050,0irq/32-eth0-rx-877-21ssh10:19:483
29925050,0irq/32-eth0-rx-8683-21fschecks_time10:04:543
29925050,0irq/32-eth0-rx-29419-21kthreadcore11:49:563
29925050,0irq/32-eth0-rx-28478-21kthreadcore10:14:583
29925050,0irq/32-eth0-rx-23259-21cat11:45:073
29925050,0irq/32-eth0-rx-21412-21ssh12:23:493
29925050,0irq/32-eth0-rx-20882-21kthreadcore10:09:573
29925050,0irq/32-eth0-rx-0-21swapper/312:08:243
29925050,0irq/32-eth0-rx-0-21swapper/310:58:103
29925050,0irq/32-eth0-rx-0-21swapper/309:55:023
29925050,0irq/32-eth0-rx-0-21swapper/309:39:293
29925050,0irq/32-eth0-rx-0-21swapper/309:25:153
27150,0ktimersoftd/20-21swapper/211:55:072
259950,0migration/27072-21sh11:38:012
259950,0migration/231143-21kthreadcore11:14:592
249950,0watchdog/20-21swapper/212:09:502
19150,0ktimersoftd/10-21swapper/111:55:071
179950,0migration/18528-21kthreadcore09:24:581
179950,0migration/16632-21kthreadcore10:59:581
179950,0migration/130842-21kthreadcore12:09:551
179950,0migration/129962-21kthreadcore07:24:561
179950,0migration/128450-21taskset07:48:181
179950,0migration/123120-21sh09:54:391
179950,0migration/118521-21sh10:47:451
179950,0migration/11623-21kthreadcore08:40:011
179950,0migration/11614-21munin-node10:55:031
179950,0migration/114481-21aten_r1power_po11:24:491
179950,0migration/114123-21kthreadcore11:59:561
169950,0watchdog/18216-21df_abs10:04:501
169950,0watchdog/14283-21fschecks_time07:09:511
149950,0migration/07788-21kthreadcore12:14:590
149950,0migration/06934-21latency10:20:000
149950,0migration/05432-21kthreadcore11:54:590
149950,0migration/0523-21taskset11:52:120
149950,0migration/03846-21sh10:59:440
149950,0migration/03281-21latency_hist07:09:380
149950,0migration/031755-21kthreadcore08:14:540
149950,0migration/029443-21sh09:38:220
149950,0migration/028345-21kthreadcore10:35:010
149950,0migration/027441-21sh12:08:460
149950,0migration/025095-21kthreadcore07:00:010
149950,0migration/024355-21sendmail_mailqu08:05:050
149950,0migration/023698-21kthreadcore11:09:590
149950,0migration/021671-21kthreadcore07:20:040
149950,0migration/020580-21munin-node10:49:550
149950,0migration/020161-21munin-node07:40:010
149950,0migration/019676-21kthreadcore08:29:560
149950,0migration/017883-21irqrtprio10:09:560
149950,0migration/014999-21taskset08:28:560
149950,0migration/014943-21fschecks_time06:54:550
149950,0migration/014211-21kthreadcore11:04:550
149950,0migration/012538-21fschecks_time08:49:510
149950,0migration/011715-21kthreadcore10:24:550
149950,0migration/011505-21sh09:09:170
149950,0migration/010031-21kthreadcore09:24:580
139829950,0cyclictest0-21swapper/210:56:192
139829950,0cyclictest0-21swapper/210:50:292
139829950,0cyclictest0-21swapper/209:00:072
139769955,0cyclictest0-21swapper/112:15:361
139769955,0cyclictest0-21swapper/110:18:191
139769955,0cyclictest0-21swapper/109:05:001
139769950,0cyclictest0-21swapper/110:25:531
139769950,0cyclictest0-21swapper/110:04:361
139689955,0cyclictest0-21swapper/009:22:180
139689955,0cyclictest0-21swapper/007:15:040
139689950,0cyclictest0-21swapper/009:48:470
8315040,0irq/26-0000:00:0-21swapper/210:39:372
339940,0migration/321173-21kthreadcore07:20:043
339940,0migration/311172-21kthreadcore08:24:583
329940,0watchdog/326519-21kthreadcore11:29:543
329940,0watchdog/30-21swapper/309:44:503
29935040,0irq/33-eth0-tx-6315-21iceccd10:05:132
29935040,0irq/33-eth0-tx-6212-21ssh09:44:462
29935040,0irq/33-eth0-tx-3599-21lldpd10:10:572
29935040,0irq/33-eth0-tx-0-21swapper/211:39:422
29935040,0irq/33-eth0-tx-0-21swapper/211:24:462
29935040,0irq/33-eth0-tx-0-21swapper/210:44:542
29935040,0irq/33-eth0-tx-0-21swapper/210:34:572
29935040,0irq/33-eth0-tx-0-21swapper/210:19:452
29935040,0irq/33-eth0-tx-0-21swapper/210:14:412
29935040,0irq/33-eth0-tx-0-21swapper/210:00:002
29935040,0irq/33-eth0-tx-0-21swapper/209:29:492
29935040,0irq/33-eth0-tx-0-21swapper/209:24:412
29935040,0irq/33-eth0-tx-0-21swapper/209:19:492
29925040,0irq/32-eth0-rx-0-21swapper/311:35:023
29925040,0irq/32-eth0-rx-0-21swapper/311:25:163
29925040,0irq/32-eth0-rx-0-21swapper/310:45:543
29925040,0irq/32-eth0-rx-0-21swapper/309:15:013
29925040,0irq/32-eth0-rx-0-21swapper/309:10:063
29925040,0irq/32-eth0-rx-0-21swapper/307:09:573
259940,0migration/27123-21latency_hist07:34:382
259940,0migration/25592-21kthreadcore09:04:572
259940,0migration/227444-21irqrtprio08:14:522
259940,0migration/223970-21hwlatdetect11:29:512
259940,0migration/223388-21kthreadcore09:15:002
259940,0migration/220836-21kthreadcore10:30:002
259940,0migration/212683-21kthreadcore10:24:552
249940,0watchdog/24907-21diskstats12:14:502
249940,0watchdog/231949-21tr09:39:542
249940,0watchdog/221982-21fschecks_count06:59:552
249940,0watchdog/216317-21kthreadcore06:54:592
249940,0watchdog/210521-21sshd07:57:272
249940,0watchdog/20-21swapper/212:19:542
249940,0watchdog/20-21swapper/212:04:582
249940,0watchdog/20-21swapper/210:59:582
179940,0migration/18165-21kthreadcore11:20:011
179940,0migration/16402-21kthreadcore08:19:571
179940,0migration/129993-21kthreadcore10:35:021
179940,0migration/129348-21fschecks_count07:49:531
179940,0migration/123618-21kthreadcore11:09:591
179940,0migration/122072-21kthreadcore12:04:571
179940,0migration/120585-21irqrtprio08:54:551
179940,0migration/120032-21kthreadcore07:20:031
179940,0migration/118793-21kthreadcore10:09:561
179940,0migration/11757-21kthreadcore11:15:001
179940,0migration/115146-21fschecks_time07:39:551
179940,0migration/113674-21kthreadcore07:14:591
169940,0watchdog/126738-21iostat11:49:541
169940,0watchdog/126520-21kthreadcore11:29:541
169940,0watchdog/122552-21iostat06:59:591
169940,0watchdog/113069-21kthreadcore08:24:581
169940,0watchdog/112326-21ssh11:04:421
169940,0watchdog/11193-21df_abs10:19:501
169940,0watchdog/111277-21df07:59:511
149940,0migration/03159-21kthreadcore07:30:040
149940,0migration/029466-21fschecks_time07:49:540
149940,0migration/025806-21kthreadcore11:29:540
139829943,0cyclictest6859-21awk11:19:562
139829943,0cyclictest3599-21lldpd08:35:162
139829943,0cyclictest3599-21lldpd08:26:152
139829943,0cyclictest3599-21lldpd08:13:022
139829943,0cyclictest3599-21lldpd07:49:532
139829943,0cyclictest3599-21lldpd07:17:462
139829943,0cyclictest29681-21awk07:04:552
139829943,0cyclictest23871-21snmpd11:49:392
139829943,0cyclictest23871-21snmpd08:51:082
139829943,0cyclictest23871-21snmpd08:41:232
139829943,0cyclictest23871-21snmpd08:30:232
139829943,0cyclictest23871-21snmpd07:30:082
139829943,0cyclictest23871-21snmpd07:20:532
139829943,0cyclictest22711-21awk09:34:562
139829943,0cyclictest22010-21awk11:09:572
139829943,0cyclictest20146-21awk08:54:542
139829943,0cyclictest17944-21sadc08:04:382
139829943,0cyclictest17790-21awk11:44:542
139829943,0cyclictest15547-21awk07:39:582
139829943,0cyclictest13477-21awk11:04:532
139829940,0cyclictest0-21swapper/209:50:442
139829940,0cyclictest0-21swapper/209:50:442
139769943,0cyclictest7316-21awk09:44:561
139769943,0cyclictest410-21awk11:34:581
139769943,0cyclictest3599-21lldpd11:44:521
139769943,0cyclictest3599-21lldpd10:40:071
139769943,0cyclictest3599-21lldpd09:41:001
139769943,0cyclictest3599-21lldpd09:33:301
139769943,0cyclictest3599-21lldpd08:34:461
139769943,0cyclictest3599-21lldpd07:35:221
139769943,0cyclictest3599-21lldpd07:04:411
139769943,0cyclictest27176-21awk08:14:521
139769943,0cyclictest23871-21snmpd09:04:081
139769943,0cyclictest23871-21snmpd07:32:531
139769943,0cyclictest22664-21awk09:34:551
139769943,0cyclictest16349-21awk08:29:531
139769943,0cyclictest12997-21awk09:09:571
139689944,0cyclictest0-21swapper/009:51:370
139689944,0cyclictest0-21swapper/009:51:370
139689944,0cyclictest0-21swapper/009:31:170
139689943,0cyclictest9087-21awk11:39:520
139689943,0cyclictest4147-21kworker/0:111:46:300
139689943,0cyclictest3262-21irqbalance11:21:020
139689943,0cyclictest3262-21irqbalance11:15:420
139689943,0cyclictest3262-21irqbalance10:58:120
139689943,0cyclictest3262-21irqbalance10:31:120
139689943,0cyclictest3262-21irqbalance10:00:420
139689943,0cyclictest3262-21irqbalance09:43:520
139689943,0cyclictest3262-21irqbalance09:10:020
139689943,0cyclictest3262-21irqbalance07:24:520
139689943,0cyclictest29260-21awk12:09:540
139689943,0cyclictest23871-21snmpd09:55:080
139689943,0cyclictest23871-21snmpd09:01:080
139689943,0cyclictest23582-21awk08:34:540
139689943,0cyclictest1844-21awk08:19:520
139689943,0cyclictest12050-21awk10:44:530
339930,0migration/316499-21iostat_ios08:29:543
329930,0watchdog/30-21swapper/308:49:503
329930,0watchdog/30-21swapper/308:00:313
329930,0watchdog/30-21swapper/307:05:033
29925030,0irq/32-eth0-rx-0-21swapper/308:39:523
29925030,0irq/32-eth0-rx-0-21swapper/307:56:013
29925030,0irq/32-eth0-rx-0-21swapper/307:34:563
29925030,0irq/32-eth0-rx-0-21swapper/306:54:523
259930,0migration/28160-21kthreadcore07:09:572
249930,0watchdog/20-21swapper/208:04:582
249930,0watchdog/20-21swapper/207:44:512
139829932,0cyclictest777-21munin-node08:19:542
139829932,0cyclictest3599-21lldpd08:44:502
139829932,0cyclictest3599-21lldpd07:24:522
139769933,0cyclictest21642-21awk09:14:581
139769932,0cyclictest3599-21lldpd12:19:521
139769932,0cyclictest3599-21lldpd08:45:191
139769932,0cyclictest30008-21awk09:19:591
139769932,0cyclictest18905-21awk10:29:571
139769932,0cyclictest15948-21awk09:49:571
139769932,0cyclictest15948-21awk09:49:571
139769932,0cyclictest14303-21head06:54:521
139769932,0cyclictest12872-21awk08:49:541
139769930,0cyclictest5576-21grep07:55:051
139769930,0cyclictest19232-21if_err_eth1.208:04:551
139769930,0cyclictest0-21swapper/108:12:201
139689933,0cyclictest4147-21kworker/0:111:59:380
139689933,0cyclictest4147-21kworker/0:111:34:420
139689933,0cyclictest4147-21kworker/0:110:39:400
139689933,0cyclictest4147-21kworker/0:110:14:380
139689933,0cyclictest4147-21kworker/0:110:04:400
139689933,0cyclictest31139-21kworker/0:007:44:390
139689933,0cyclictest31139-21kworker/0:007:34:410
139689933,0cyclictest24394-21kworker/0:209:14:410
139689933,0cyclictest24394-21kworker/0:208:44:470
139689933,0cyclictest24394-21kworker/0:208:09:450
329920,0watchdog/30-21swapper/308:09:383
329920,0watchdog/30-21swapper/307:39:433
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional