You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 10:03
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2.osadl.org (updated Fri Dec 05, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2385725442,0sleep20-21swapper/218:16:552
2382023828,0sleep30-21swapper/318:16:283
2373423625,0sleep00-21swapper/018:15:290
2367323523,0sleep10-21swapper/118:14:461
329970,0watchdog/311362-21kthreadcore20:39:313
29925070,0irq/32-eth0-rx-5992-21perf20:39:063
29925070,0irq/32-eth0-rx-0-21swapper/321:12:153
339960,0migration/329885-21munin-node23:04:263
339960,0migration/322471-21munin-node21:44:243
339960,0migration/312180-21sh21:54:293
329960,0watchdog/323281-21ntp_states18:59:283
329960,0watchdog/315622-21kthreadcore22:54:273
329960,0watchdog/30-21swapper/323:47:313
329960,0watchdog/30-21swapper/323:34:073
329960,0watchdog/30-21swapper/323:21:073
329960,0watchdog/30-21swapper/323:00:393
329960,0watchdog/30-21swapper/322:44:593
329960,0watchdog/30-21swapper/322:31:033
329960,0watchdog/30-21swapper/322:01:353
329960,0watchdog/30-21swapper/321:50:473
329960,0watchdog/30-21swapper/321:28:153
329960,0watchdog/30-21swapper/321:20:153
329960,0watchdog/30-21swapper/321:08:153
329960,0watchdog/30-21swapper/320:49:193
329960,0watchdog/30-21swapper/320:48:233
329960,0watchdog/30-21swapper/319:56:363
329960,0watchdog/30-21swapper/319:47:043
329960,0watchdog/30-21swapper/318:49:243
329960,0watchdog/30-21swapper/318:35:003
29925060,0irq/32-eth0-rx-8607-21port_sshpriv21:14:303
29925060,0irq/32-eth0-rx-3040-21crond20:19:063
29925060,0irq/32-eth0-rx-14317-21munin-node22:14:313
29925060,0irq/32-eth0-rx-122052sleep323:10:343
29925060,0irq/32-eth0-rx-0-21swapper/323:24:263
29925060,0irq/32-eth0-rx-0-21swapper/323:16:003
29925060,0irq/32-eth0-rx-0-21swapper/322:35:213
29925060,0irq/32-eth0-rx-0-21swapper/322:09:123
29925060,0irq/32-eth0-rx-0-21swapper/321:33:293
29925060,0irq/32-eth0-rx-0-21swapper/321:03:523
29925060,0irq/32-eth0-rx-0-21swapper/320:54:473
29925060,0irq/32-eth0-rx-0-21swapper/320:28:363
241239966,0cyclictest3045-21diskmemload21:34:290
241239960,0cyclictest0-21swapper/019:01:050
149960,0migration/08125-21sh22:30:130
149960,0migration/03607-21sh23:05:340
149960,0migration/030960-21kthreadcore19:29:270
149960,0migration/030960-21kthreadcore19:29:270
8315050,0irq/26-0000:00:3171-21cat20:19:072
8315050,0irq/26-0000:00:0-21swapper/319:19:263
8315050,0irq/26-0000:00:0-21swapper/219:54:072
4150,0ktimersoftd/00-21swapper/022:17:030
339950,0migration/323743-21kthreadcore22:39:303
339950,0migration/323010-21kthreadcore20:09:303
339950,0migration/312400-21kthreadcore23:29:293
339950,0migration/310553-21munin-node18:54:223
329950,0watchdog/319801-21latency20:04:283
329950,0watchdog/30-21swapper/322:24:553
329950,0watchdog/30-21swapper/322:19:073
329950,0watchdog/30-21swapper/322:04:073
329950,0watchdog/30-21swapper/321:39:433
329950,0watchdog/30-21swapper/321:34:233
329950,0watchdog/30-21swapper/320:29:083
329950,0watchdog/30-21swapper/320:19:083
329950,0watchdog/30-21swapper/320:01:043
329950,0watchdog/30-21swapper/319:49:203
329950,0watchdog/30-21swapper/319:34:243
329950,0watchdog/30-21swapper/319:29:243
329950,0watchdog/30-21swapper/319:29:243
329950,0watchdog/30-21swapper/319:24:203
329950,0watchdog/30-21swapper/319:14:563
329950,0watchdog/30-21swapper/319:10:323
329950,0watchdog/30-21swapper/319:04:083
329950,0watchdog/30-21swapper/318:39:363
329950,0watchdog/30-21swapper/318:29:363
329950,0watchdog/30-21swapper/318:24:323
329950,0watchdog/30-21swapper/318:19:123
29935050,0irq/33-eth0-tx-3704-21kthreadcore20:54:292
29935050,0irq/33-eth0-tx-26132-21grep20:29:272
29935050,0irq/33-eth0-tx-19985-21ssh22:59:072
29935050,0irq/33-eth0-tx-17505-21munin-node19:44:250
29935050,0irq/33-eth0-tx-13526-21cat23:14:072
29935050,0irq/33-eth0-tx-11836-21latency_hist20:24:062
29935050,0irq/33-eth0-tx-0-21swapper/223:22:272
29935050,0irq/33-eth0-tx-0-21swapper/222:36:142
29935050,0irq/33-eth0-tx-0-21swapper/222:12:392
29935050,0irq/33-eth0-tx-0-21swapper/222:04:352
29935050,0irq/33-eth0-tx-0-21swapper/220:59:232
29935050,0irq/33-eth0-tx-0-21swapper/220:52:062
29935050,0irq/33-eth0-tx-0-21swapper/019:29:050
29935050,0irq/33-eth0-tx-0-21swapper/019:04:550
29925050,0irq/32-eth0-rx-0-21swapper/323:39:293
29925050,0irq/32-eth0-rx-0-21swapper/322:49:223
29925050,0irq/32-eth0-rx-0-21swapper/319:39:263
29925050,0irq/32-eth0-rx-0-21swapper/318:44:253
259950,0migration/28785-21munin-node21:14:342
259950,0migration/27967-21sh22:29:452
259950,0migration/25946-21taskset21:34:282
259950,0migration/25382-21kthreadcore18:49:242
259950,0migration/229325-21sh21:48:082
259950,0migration/229179-21idleruntime21:29:242
259950,0migration/223678-21sh22:20:482
259950,0migration/223398-21munin-node23:39:192
259950,0migration/218939-21kthreadcore23:34:342
259950,0migration/217534-21kthreadcore19:19:272
259950,0migration/216093-21sh22:18:332
241259955,0cyclictest0-21swapper/223:04:242
241259955,0cyclictest0-21swapper/221:54:242
241259954,0cyclictest0-21swapper/222:39:292
241259950,0cyclictest0-21swapper/220:36:122
241249954,0cyclictest22392-21grep20:09:291
241249950,0cyclictest0-21swapper/123:22:191
241249950,0cyclictest0-21swapper/123:10:211
241239955,0cyclictest0-21swapper/023:13:200
241239955,0cyclictest0-21swapper/022:49:130
241239950,0cyclictest0-21swapper/021:08:000
19150,0ktimersoftd/10-21swapper/123:26:071
19150,0ktimersoftd/10-21swapper/121:35:551
19150,0ktimersoftd/10-21swapper/121:11:011
19150,0ktimersoftd/10-21swapper/120:32:041
179950,0migration/14970-21kthreadcore18:49:241
179950,0migration/1374-21df_abs20:54:241
179950,0migration/126544-21munin-node22:59:291
179950,0migration/124212-21taskset19:46:251
179950,0migration/117844-21kthreadcore20:44:241
179950,0migration/110580-21fschecks_count18:54:221
179950,0migration/110034-21sh21:17:071
169950,0watchdog/18798-21kthreadcore18:29:241
169950,0watchdog/110303-21fschecks_count22:34:231
149950,0migration/09480-21kthreadcore21:54:270
149950,0migration/08690-21kthreadcore23:29:280
149950,0migration/06872-21kthreadcore21:14:270
149950,0migration/04719-21sh21:50:440
149950,0migration/030873-21taskset23:42:000
149950,0migration/030024-21kthreadcore19:49:270
149950,0migration/029263-21kthreadcore18:19:260
149950,0migration/027457-21switchtime20:09:380
149950,0migration/025229-21sh22:40:130
149950,0migration/024635-21kthreadcore20:29:260
149950,0migration/023543-21cpuspeed_turbos22:04:220
149950,0migration/02353-21taskset18:46:400
149950,0migration/023015-21kthreadcore21:24:240
149950,0migration/022005-21fschecks_count21:44:220
149950,0migration/020395-21taskset19:20:100
149950,0migration/01949-21kthreadcore20:54:290
149950,0migration/019077-21kthreadcore21:59:270
149950,0migration/01903-21missed_timers20:14:300
149950,0migration/018819-21kthreadcore22:19:300
149950,0migration/01851-21ntp_kernel_pll_21:29:290
149950,0migration/016817-21kthreadcore23:34:330
149950,0migration/016019-21kthreadcore18:34:240
149950,0migration/015169-21aten_r1power_cu20:44:170
149950,0migration/014157-21kthreadcore22:54:270
149950,0migration/013545-21irqrtprio20:24:260
149950,0migration/013130-21kthreadcore21:19:260
149950,0migration/01144-21kthreadcore23:24:260
149950,0migration/011001-21kthreadcore20:59:240
8315040,0irq/26-0000:00:0-21swapper/219:54:082
29935040,0irq/33-eth0-tx-26836-21ssh22:44:132
29935040,0irq/33-eth0-tx-11974-21ssh20:24:152
29935040,0irq/33-eth0-tx-0-21swapper/223:29:272
29935040,0irq/33-eth0-tx-0-21swapper/223:24:132
29935040,0irq/33-eth0-tx-0-21swapper/222:50:302
29935040,0irq/33-eth0-tx-0-21swapper/221:39:182
29935040,0irq/33-eth0-tx-0-21swapper/221:19:312
29935040,0irq/33-eth0-tx-0-21swapper/221:09:442
29935040,0irq/33-eth0-tx-0-21swapper/221:04:132
29935040,0irq/33-eth0-tx-0-21swapper/019:59:220
259940,0migration/2954-21kernelversion23:44:292
259940,0migration/25922-21kthreadcore19:34:262
259940,0migration/230522-21fschecks_count21:49:232
259940,0migration/229462-21irqcore19:29:252
259940,0migration/229462-21irqcore19:29:252
259940,0migration/228484-21fschecks_count18:44:212
259940,0migration/228258-21kthreadcore19:04:272
259940,0migration/221012-21irqrtprio21:24:232
259940,0migration/219096-21kthreadcore23:14:272
259940,0migration/218212-21kthreadcore20:44:242
259940,0migration/217857-21fschecks_count18:59:212
259940,0migration/214686-21fschecks_time20:04:222
259940,0migration/211192-21munin-node18:54:242
259940,0migration/210298-21kthreadcore19:14:262
241259943,0cyclictest7373-21awk20:39:282
241259943,0cyclictest353-21awk19:09:252
241259943,0cyclictest26133-21awk22:24:232
241259943,0cyclictest25511-21awk18:19:242
241259943,0cyclictest23871-21snmpd19:27:392
241259943,0cyclictest15797-21awk21:59:242
241259943,0cyclictest14694-21awk18:34:222
241259943,0cyclictest12796-21awk22:54:252
241249944,0cyclictest0-21swapper/121:20:191
241249943,0cyclictest7997-21awk23:29:261
241249943,0cyclictest31024-21awk21:49:251
241249943,0cyclictest30024-21awk23:04:271
241249943,0cyclictest29673-21sadc21:49:061
241249943,0cyclictest27963-21awk22:44:241
241249943,0cyclictest25574-21awk19:04:251
241249943,0cyclictest25463-21awk18:19:231
241249943,0cyclictest23871-21snmpd18:26:231
241249943,0cyclictest21867-21awk18:39:221
241249943,0cyclictest16228-21awk23:34:321
241249943,0cyclictest15863-21awk21:59:251
241249943,0cyclictest15562-21sed20:04:261
241239943,0cyclictest405-21awk23:44:270
241239943,0cyclictest3599-21lldpd22:59:160
241239943,0cyclictest3599-21lldpd20:52:560
241239943,0cyclictest3599-21lldpd20:34:560
241239943,0cyclictest32595-21awk18:24:220
241239943,0cyclictest23871-21snmpd22:24:090
241239943,0cyclictest20544-21kworker/0:222:10:010
241239943,0cyclictest10910-21awk22:34:260
241239940,0cyclictest0-21swapper/019:54:200
241239940,0cyclictest0-21swapper/018:43:540
249940,0watchdog/28799-21kthreadcore18:29:242
249940,0watchdog/27224-21diskstats19:59:242
249940,0watchdog/213376-21kthreadcore19:39:322
19140,0ktimersoftd/10-21swapper/122:51:071
179940,0migration/19490-21irqrtprio22:14:271
179940,0migration/18782-21kthreadcore19:59:281
179940,0migration/17135-21fschecks_count19:14:221
179940,0migration/131427-21taskset19:51:171
179940,0migration/130434-21kthreadcore20:14:281
179940,0migration/129823-21kthreadcore19:29:261
179940,0migration/129823-21kthreadcore19:29:261
179940,0migration/126695-21sh21:25:401
179940,0migration/123912-21fschecks_time22:04:231
179940,0migration/123909-21timerwakeupswit21:04:361
179940,0migration/121757-21sh23:18:411
179940,0migration/120522-21runrttasks20:29:051
179940,0migration/120421-21kthreadcore22:39:291
179940,0migration/11991-21kthreadcore23:44:291
179940,0migration/118397-21kthreadcore21:39:301
179940,0migration/118059-21kthreadcore19:19:281
179940,0migration/116935-21kthreadcore18:34:251
179940,0migration/11571-21kthreadcore22:09:251
179940,0migration/11386-21kthreadcore20:34:281
169940,0watchdog/19725-21grep21:54:271
169940,0watchdog/17543-21python22:29:311
169940,0watchdog/131967-21ssh22:25:111
169940,0watchdog/124171-21df_abs20:49:191
169940,0watchdog/111361-21kthreadcore20:39:311
169940,0watchdog/10-21swapper/123:43:071
169940,0watchdog/10-21swapper/122:54:271
169940,0watchdog/10-21swapper/121:29:271
169940,0watchdog/10-21swapper/120:59:271
169940,0watchdog/10-21swapper/120:19:201
169940,0watchdog/10-21swapper/119:39:281
149940,0migration/06877-21fschecks_count20:39:270
149940,0migration/03850-21sh18:49:210
149940,0migration/026736-21latency_hist22:44:070
149940,0migration/017212-21kthreadcore21:39:300
149940,0migration/015165-21kthreadcore18:54:270
149940,0migration/011728-21kthreadcore18:29:250
149940,0migration/010210-21sh20:20:060
241259932,0cyclictest32580-21awk18:24:222
241259932,0cyclictest23676-21grep20:09:302
241259930,0cyclictest0-21swapper/219:47:002
241259930,0cyclictest0-21swapper/218:39:282
241249932,0cyclictest481-21awk19:54:231
241249932,0cyclictest30385-21cut18:44:251
241249932,0cyclictest21031-21aten2.4-expect19:24:191
241249932,0cyclictest18483-21awk18:59:241
241249932,0cyclictest16802-21cpuspeed_turbos22:19:251
241249930,0cyclictest32450-21perl19:09:231
241239933,0cyclictest29853-21kworker/0:123:19:230
241239933,0cyclictest29853-21kworker/0:123:14:570
241239933,0cyclictest28978-21kworker/0:121:09:200
241239933,0cyclictest28978-21kworker/0:120:04:220
241239933,0cyclictest28978-21kworker/0:119:39:120
241239933,0cyclictest28978-21kworker/0:119:34:100
241239932,0cyclictest6667-21hostname19:14:200
241239930,0cyclictest0-21swapper/019:09:280
169920,0watchdog/10-21swapper/119:34:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional