You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 12:05
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2.osadl.org (updated Thu Sep 18, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2329323828,0sleep30-21swapper/318:20:413
2354823727,0sleep20-21swapper/218:23:402
2338623725,0sleep10-21swapper/118:21:471
2321323725,0sleep00-21swapper/018:19:460
236319984,0cyclictest28649-21awk18:49:061
236319983,0cyclictest21689-21awk19:29:071
329970,0watchdog/30-21swapper/322:02:063
329970,0watchdog/30-21swapper/319:44:073
29925070,0irq/32-eth0-rx-5228-21grep23:39:073
29925070,0irq/32-eth0-rx-0-21swapper/323:45:333
29925070,0irq/32-eth0-rx-0-21swapper/322:43:563
29925070,0irq/32-eth0-rx-0-21swapper/322:39:093
29925070,0irq/32-eth0-rx-0-21swapper/321:34:123
29925070,0irq/32-eth0-rx-0-21swapper/320:59:043
236319973,0cyclictest8177-21awk22:44:031
236319973,0cyclictest26339-21needreboot18:44:111
236319972,0cyclictest17947-21awk19:04:071
236319972,0cyclictest1265-21awk21:44:071
236319970,0cyclictest32382-21fschecks_time20:44:061
8315060,0irq/26-0000:00:0-21swapper/318:53:513
339960,0migration/33941-21sh22:22:133
339960,0migration/317365-21sh23:07:483
329960,0watchdog/3299250irq/32-eth0-rx-21:04:423
329960,0watchdog/30-21swapper/323:30:383
329960,0watchdog/30-21swapper/323:16:383
329960,0watchdog/30-21swapper/323:10:063
329960,0watchdog/30-21swapper/322:53:543
329960,0watchdog/30-21swapper/322:48:503
329960,0watchdog/30-21swapper/322:34:463
329960,0watchdog/30-21swapper/322:24:103
329960,0watchdog/30-21swapper/322:14:503
329960,0watchdog/30-21swapper/321:49:223
329960,0watchdog/30-21swapper/321:30:583
329960,0watchdog/30-21swapper/321:24:423
329960,0watchdog/30-21swapper/321:10:263
329960,0watchdog/30-21swapper/320:57:103
329960,0watchdog/30-21swapper/320:49:143
329960,0watchdog/30-21swapper/320:23:543
329960,0watchdog/30-21swapper/320:19:103
329960,0watchdog/30-21swapper/320:12:463
329960,0watchdog/30-21swapper/319:50:273
329960,0watchdog/30-21swapper/319:30:113
329960,0watchdog/30-21swapper/319:14:233
329960,0watchdog/30-21swapper/318:50:153
329960,0watchdog/30-21swapper/318:46:313
329960,0watchdog/30-21swapper/318:43:113
29925060,0irq/32-eth0-rx-32575-21hostname20:44:073
29925060,0irq/32-eth0-rx-32203-21munin-run21:43:493
29925060,0irq/32-eth0-rx-23871-21snmpd20:38:513
29925060,0irq/32-eth0-rx-17327-21yum23:24:203
29925060,0irq/32-eth0-rx-17015-21grep20:34:083
29925060,0irq/32-eth0-rx-0-21swapper/323:50:543
29925060,0irq/32-eth0-rx-0-21swapper/323:21:033
29925060,0irq/32-eth0-rx-0-21swapper/322:58:583
29925060,0irq/32-eth0-rx-0-21swapper/322:29:393
29925060,0irq/32-eth0-rx-0-21swapper/322:05:403
29925060,0irq/32-eth0-rx-0-21swapper/321:54:503
29925060,0irq/32-eth0-rx-0-21swapper/321:45:273
29925060,0irq/32-eth0-rx-0-21swapper/321:19:103
29925060,0irq/32-eth0-rx-0-21swapper/320:28:533
236319964,0cyclictest0-21swapper/122:18:031
236319962,0cyclictest5190-21awk21:09:031
236319962,0cyclictest31870-21awk22:39:061
236319962,0cyclictest23871-21snmpd20:02:441
236319962,0cyclictest13925-21tr20:09:021
236319962,0cyclictest10888-21awk18:59:071
236319961,0cyclictest6145-21awk20:29:071
236319961,0cyclictest32436-21sendmail_mailqu21:59:111
236319961,0cyclictest27275-21fschecks_time23:14:011
236319961,0cyclictest27246-21uname20:59:051
236319961,0cyclictest23434-21cut20:39:041
236319961,0cyclictest20369-21cut20:54:091
236319961,0cyclictest20109-21awk23:29:101
236319961,0cyclictest16381-21awk21:34:091
236319961,0cyclictest13972-21awk21:14:061
236319961,0cyclictest11355-21sadc23:43:491
236319960,0cyclictest825-21kthreadcore21:24:091
236319960,0cyclictest7374-21awk19:19:071
236319960,0cyclictest30472-21ntp_states22:54:121
236319960,0cyclictest25554-21ssh21:58:071
236319960,0cyclictest19944-21aten2.4-expect23:49:011
236319960,0cyclictest1481-21fschecks_time22:59:051
236319960,0cyclictest14709-21kthreadcore23:24:111
236319960,0cyclictest11984-21kthreadcore20:04:131
236319960,0cyclictest11217-21kthreadcore21:49:091
236319960,0cyclictest0-21swapper/123:04:511
236319960,0cyclictest0-21swapper/122:29:231
236319960,0cyclictest0-21swapper/122:20:531
236319960,0cyclictest0-21swapper/122:04:031
236319960,0cyclictest0-21swapper/121:33:181
236319960,0cyclictest0-21swapper/120:24:011
149960,0migration/026702-21kthreadcore18:24:080
8315050,0irq/26-0000:00:11571-21ssh23:43:502
339950,0migration/36396-21latency_hist20:03:503
339950,0migration/316140-21unixbench-2d18:59:163
339950,0migration/313573-21aten_r1power_po19:24:033
339950,0migration/312736-21taskset21:13:563
329950,0watchdog/312205-21python18:34:153
329950,0watchdog/30-21swapper/322:09:103
329950,0watchdog/30-21swapper/320:14:303
329950,0watchdog/30-21swapper/319:59:023
329950,0watchdog/30-21swapper/319:55:073
329950,0watchdog/30-21swapper/319:39:313
329950,0watchdog/30-21swapper/319:36:073
329950,0watchdog/30-21swapper/319:21:433
329950,0watchdog/30-21swapper/319:08:553
329950,0watchdog/30-21swapper/319:05:153
329950,0watchdog/30-21swapper/318:29:393
329950,0watchdog/30-21swapper/318:25:273
29935050,0irq/33-eth0-tx-6315-21iceccd19:38:530
29935050,0irq/33-eth0-tx-5931-21munin-node18:34:140
29935050,0irq/33-eth0-tx-4095-21kthreadcore23:19:112
29935050,0irq/33-eth0-tx-28494-21diskmemload22:42:182
29935050,0irq/33-eth0-tx-24114-21munin-node19:09:030
29935050,0irq/33-eth0-tx-20639-21munin-node19:29:060
29935050,0irq/33-eth0-tx-16910-21munin-node19:04:180
29935050,0irq/33-eth0-tx-13648-21munin-node20:09:030
29935050,0irq/33-eth0-tx-10655-21unixbench-2d21:09:122
29935050,0irq/33-eth0-tx-10612-21munin-node23:24:132
29935050,0irq/33-eth0-tx-0-21swapper/223:13:162
29935050,0irq/33-eth0-tx-0-21swapper/222:46:132
29935050,0irq/33-eth0-tx-0-21swapper/222:37:302
29935050,0irq/33-eth0-tx-0-21swapper/222:32:522
29935050,0irq/33-eth0-tx-0-21swapper/222:24:102
29935050,0irq/33-eth0-tx-0-21swapper/221:49:192
29935050,0irq/33-eth0-tx-0-21swapper/220:30:092
29935050,0irq/33-eth0-tx-0-21swapper/020:24:030
29925050,0irq/32-eth0-rx-0-21swapper/323:34:043
259950,0migration/232087-21sh22:57:082
259950,0migration/230592-21kthreadcore23:14:052
259950,0migration/22739-21cpuspeed_turbos18:54:032
259950,0migration/224253-21latency_hist19:53:492
259950,0migration/220709-21aten_r1power_fr19:29:032
259950,0migration/219926-21ntp_offset20:34:102
259950,0migration/217858-21kthreadcore22:49:102
259950,0migration/215559-21sh20:51:042
259950,0migration/213726-21aten_r1power_fr20:09:012
236329955,0cyclictest0-21swapper/220:54:172
236329955,0cyclictest0-21swapper/218:58:522
236329954,0cyclictest23871-21snmpd22:03:212
236329950,0cyclictest0-21swapper/219:39:142
236319951,0cyclictest6386-21runrttasks20:19:261
236319951,0cyclictest13646-21expr18:39:011
236319951,0cyclictest0-21swapper/123:33:501
236319951,0cyclictest0-21swapper/122:48:561
236319951,0cyclictest0-21swapper/122:09:001
236319951,0cyclictest0-21swapper/121:18:501
236319951,0cyclictest0-21swapper/120:34:011
236319950,0cyclictest8176-21munin-node20:49:041
236319950,0cyclictest5931-21munin-node18:34:021
236319950,0cyclictest4936-21aten2.4-expect22:24:041
236319950,0cyclictest3461-21idleruntime18:54:061
236319950,0cyclictest31499-21forks18:29:061
236319950,0cyclictest2826-21cpu19:39:031
236319950,0cyclictest2781-21munin-node23:19:061
236319950,0cyclictest24655-21df_abs19:54:041
236319950,0cyclictest24170-21aten2.4-expect19:09:011
236319950,0cyclictest23962-21aten2.4-expect21:39:021
236319950,0cyclictest2310-21ssh21:05:071
236319950,0cyclictest22146-21aten2.4-expect22:34:021
236319950,0cyclictest21894-21if_err_eth1.220:14:061
236319950,0cyclictest17939-21ssh23:08:561
236319950,0cyclictest13874-21entropy19:24:041
236319950,0cyclictest12750-21if_eth023:44:091
236319950,0cyclictest10249-21df_abs19:44:021
236319950,0cyclictest0-21swapper/119:48:501
236319950,0cyclictest0-21swapper/119:34:021
236319950,0cyclictest0-21swapper/119:14:011
236319950,0cyclictest0-21swapper/118:24:051
149950,0migration/09263-21kthreadcore19:19:090
149950,0migration/07110-21kthreadcore21:09:050
149950,0migration/05546-21kthreadcore22:04:080
149950,0migration/05331-21kthreadcore18:54:080
149950,0migration/04870-21sh21:25:140
149950,0migration/04548-21kthreadcore21:44:090
149950,0migration/031946-21fschecks_time18:29:070
149950,0migration/031762-21idleruntime22:39:060
149950,0migration/031118-21kthreadcore23:34:100
149950,0migration/031052-21taskset20:43:180
149950,0migration/029977-21kthreadcore22:54:100
149950,0migration/026613-21gltestperf20:59:020
149950,0migration/026573-21aten_r1power_po23:13:590
149950,0migration/026549-21kthreadcore19:54:080
149950,0migration/026358-21timerwakeupswit22:14:180
149950,0migration/025205-21kthreadcore23:49:090
149950,0migration/02417-21taskset21:05:240
149950,0migration/023846-21kthreadcore18:44:090
149950,0migration/022222-21timerwakeupswit22:49:170
149950,0migration/022084-21sh21:34:540
149950,0migration/020234-21kthreadcore19:49:080
149950,0migration/019973-21gltestperf23:29:090
149950,0migration/018458-21kthreadcore18:39:060
149950,0migration/018122-21kthreadcore19:24:090
149950,0migration/018085-21timerandwakeup23:44:180
149950,0migration/017413-21proc_pri22:09:130
149950,0migration/016976-21aten_r1power_cu20:54:020
149950,0migration/016364-21taskset20:53:060
149950,0migration/01557-21taskset22:02:180
149950,0migration/010507-21kthreadcore22:24:100
29935040,0irq/33-eth0-tx-9461-21aten_r1power_fr23:04:022
29935040,0irq/33-eth0-tx-29485-21kthreadcore21:39:082
29935040,0irq/33-eth0-tx-2199-40yum-updatesd-he21:04:512
29935040,0irq/33-eth0-tx-0-21swapper/223:49:372
29935040,0irq/33-eth0-tx-0-21swapper/223:29:242
29935040,0irq/33-eth0-tx-0-21swapper/222:19:072
29935040,0irq/33-eth0-tx-0-21swapper/222:14:012
29935040,0irq/33-eth0-tx-0-21swapper/222:08:592
29935040,0irq/33-eth0-tx-0-21swapper/222:05:172
29935040,0irq/33-eth0-tx-0-21swapper/221:53:502
29935040,0irq/33-eth0-tx-0-21swapper/221:29:082
29935040,0irq/33-eth0-tx-0-21swapper/221:14:462
29935040,0irq/33-eth0-tx-0-21swapper/220:58:552
29935040,0irq/33-eth0-tx-0-21swapper/220:38:502
29935040,0irq/33-eth0-tx-0-21swapper/220:24:192
29935040,0irq/33-eth0-tx-0-21swapper/020:20:040
29935040,0irq/33-eth0-tx-0-21swapper/020:04:110
29935040,0irq/33-eth0-tx-0-21swapper/019:59:060
29935040,0irq/33-eth0-tx-0-21swapper/019:47:410
29935040,0irq/33-eth0-tx-0-21swapper/019:14:140
29935040,0irq/33-eth0-tx-0-21swapper/018:52:300
259940,0migration/27397-21gltestperf20:04:092
259940,0migration/25762-21kthreadcore23:39:072
259940,0migration/24021-21kthreadcore21:44:092
259940,0migration/232530-21irqrtprio18:29:102
259940,0migration/23126-21kthreadcore21:24:092
259940,0migration/227388-21runrttasks23:33:582
259940,0migration/227266-21taskset18:47:352
259940,0migration/213580-21kthreadcore19:44:082
259940,0migration/21216-21fschecks_count22:59:052
249940,0watchdog/231521-21sh20:43:542
249940,0watchdog/228064-21munin-node19:34:032
249940,0watchdog/213194-21aten_r1power_cu18:38:592
249940,0watchdog/20-21swapper/221:34:102
236329943,0cyclictest25151-21awk19:09:052
236329943,0cyclictest24091-21sadc19:53:492
236329943,0cyclictest23871-21snmpd19:15:212
236329943,0cyclictest22703-21awk21:19:082
236309943,0cyclictest28951-21awk19:34:060
236309943,0cyclictest24821-21kworker/0:123:05:060
236309943,0cyclictest23871-21snmpd22:32:060
236309943,0cyclictest23871-21snmpd20:48:210
236309943,0cyclictest16675-21sadc19:03:500
149940,0migration/08750-21kthreadcore23:39:080
149940,0migration/03842-21irqcore23:19:100
149940,0migration/022507-21kthreadcore21:54:100
149940,0migration/021417-21fschecks_count20:14:040
149940,0migration/019784-21sh21:14:540
149940,0migration/019263-21irqcore23:09:090
149940,0migration/018250-21sh23:27:200
27130,0ktimersoftd/20-21swapper/220:23:132
259930,0migration/22389-21kthreadcore19:59:082
236329932,0cyclictest26883-21pluginstate20:14:112
236329932,0cyclictest18088-21awk19:04:082
236329932,0cyclictest1664-21tail18:49:162
236329932,0cyclictest12789-21vmstat19:19:172
236309933,0cyclictest24821-21kworker/0:122:58:520
236309933,0cyclictest24821-21kworker/0:122:43:560
236309933,0cyclictest24821-21kworker/0:122:18:540
236309933,0cyclictest24821-21kworker/0:121:49:020
236309933,0cyclictest24821-21kworker/0:121:38:540
236309933,0cyclictest24821-21kworker/0:121:28:500
236309933,0cyclictest24821-21kworker/0:121:19:010
236309933,0cyclictest24821-21kworker/0:120:33:570
236309933,0cyclictest24821-21kworker/0:120:29:450
236309932,0cyclictest23871-21snmpd22:33:510
236329921,0cyclictest13364-21latency_hist19:23:502
236329920,0cyclictest5907-21cat18:33:512
236329920,0cyclictest23758-21ls18:23:502
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional