You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 03:53
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2.osadl.org (updated Wed Dec 03, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
51726867,0sleep20-21swapper/218:18:402
3270525343,0sleep30-21swapper/318:15:323
3265724837,0sleep10-21swapper/118:14:541
3266623625,0sleep00-21swapper/018:15:010
329970,0watchdog/30-21swapper/322:34:383
329970,0watchdog/30-21swapper/322:28:583
329970,0watchdog/30-21swapper/322:15:223
329970,0watchdog/30-21swapper/321:51:113
329970,0watchdog/30-21swapper/321:40:193
329970,0watchdog/30-21swapper/321:28:273
329970,0watchdog/30-21swapper/321:07:553
329970,0watchdog/30-21swapper/321:01:353
329970,0watchdog/30-21swapper/320:58:513
29925070,0irq/32-eth0-rx-31201-21if_eth1.319:24:323
29925070,0irq/32-eth0-rx-28302sleep322:23:333
29925070,0irq/32-eth0-rx-16318-21sh23:08:593
6499966,0cyclictest29665-21kthreadcore18:59:351
6419964,0cyclictest10868-21awk22:09:310
6419961,0cyclictest0-21swapper/018:59:300
6419960,0cyclictest10-21rcuc/020:59:140
6419960,0cyclictest0-21swapper/022:19:100
339960,0migration/36879-21munin-node20:49:363
329960,0watchdog/3299250irq/32-eth0-rx-18:49:313
329960,0watchdog/31135-21aten2.4-expect22:04:263
329960,0watchdog/30-21swapper/323:44:503
329960,0watchdog/30-21swapper/323:39:463
329960,0watchdog/30-21swapper/323:34:503
329960,0watchdog/30-21swapper/323:31:463
329960,0watchdog/30-21swapper/323:25:023
329960,0watchdog/30-21swapper/323:20:503
329960,0watchdog/30-21swapper/323:14:463
329960,0watchdog/30-21swapper/323:09:143
329960,0watchdog/30-21swapper/322:49:343
329960,0watchdog/30-21swapper/322:42:103
329960,0watchdog/30-21swapper/322:31:223
329960,0watchdog/30-21swapper/322:11:143
329960,0watchdog/30-21swapper/322:02:463
329960,0watchdog/30-21swapper/321:57:103
329960,0watchdog/30-21swapper/321:34:193
329960,0watchdog/30-21swapper/321:30:033
329960,0watchdog/30-21swapper/321:19:313
329960,0watchdog/30-21swapper/321:14:473
329960,0watchdog/30-21swapper/320:44:273
329960,0watchdog/30-21swapper/320:39:153
329960,0watchdog/30-21swapper/320:34:153
329960,0watchdog/30-21swapper/320:25:033
329960,0watchdog/30-21swapper/320:19:313
329960,0watchdog/30-21swapper/320:10:513
329960,0watchdog/30-21swapper/319:54:513
329960,0watchdog/30-21swapper/319:44:193
329960,0watchdog/30-21swapper/319:33:433
329960,0watchdog/30-21swapper/319:16:113
329960,0watchdog/30-21swapper/319:07:153
329960,0watchdog/30-21swapper/318:41:353
329960,0watchdog/30-21swapper/318:34:273
329960,0watchdog/30-21swapper/318:31:353
329960,0watchdog/30-21swapper/318:25:113
329960,0watchdog/30-21swapper/318:20:073
29925060,0irq/32-eth0-rx-6134-21grep22:44:283
29925060,0irq/32-eth0-rx-3205-21cpuspeed_turbos21:09:273
29925060,0irq/32-eth0-rx-2016-21kthreadcore21:44:343
29925060,0irq/32-eth0-rx-0-21swapper/323:00:113
29925060,0irq/32-eth0-rx-0-21swapper/322:54:533
6499950,0cyclictest0-21swapper/120:54:311
6419955,0cyclictest41ktimersoftd/019:44:300
6419955,0cyclictest10-21rcuc/023:19:040
6419955,0cyclictest0-21swapper/023:07:240
6419955,0cyclictest0-21swapper/022:20:160
6419955,0cyclictest0-21swapper/021:06:290
6419955,0cyclictest0-21swapper/020:57:300
6419953,0cyclictest22012-21iostat_ios22:34:300
6419953,0cyclictest0-21swapper/021:58:400
6419952,0cyclictest9496-21grep21:09:400
6419952,0cyclictest891-21munin-node18:19:380
6419952,0cyclictest30079-21awk21:24:320
6419952,0cyclictest0-21swapper/021:44:340
6419952,0cyclictest0-21swapper/019:54:350
6419951,0cyclictest8396-21latency_hist19:54:150
6419951,0cyclictest23968-21perf20:44:140
6419951,0cyclictest20771-21sh20:24:140
6419951,0cyclictest20225-21grep19:39:290
6419951,0cyclictest11918-21sh19:34:150
6419951,0cyclictest1151-21cat23:34:410
6419951,0cyclictest0-21swapper/023:24:320
6419951,0cyclictest0-21swapper/021:34:330
6419951,0cyclictest0-21swapper/018:44:290
6419951,0cyclictest0-21swapper/018:24:320
6419950,0cyclictest22849-21munin-node19:19:330
6419950,0cyclictest0-21swapper/022:40:170
6419950,0cyclictest0-21swapper/021:20:590
339950,0migration/36114-21kthreadcore19:49:323
339950,0migration/326747-21munin-node18:59:303
329950,0watchdog/322040-21python19:59:393
329950,0watchdog/30-21swapper/320:14:473
329950,0watchdog/30-21swapper/320:04:193
329950,0watchdog/30-21swapper/319:39:513
329950,0watchdog/30-21swapper/319:34:273
329950,0watchdog/30-21swapper/319:34:273
329950,0watchdog/30-21swapper/319:19:273
329950,0watchdog/30-21swapper/319:09:153
29935050,0irq/33-eth0-tx-6660-21ssh23:01:072
29935050,0irq/33-eth0-tx-28013-21smartctl23:34:302
29935050,0irq/33-eth0-tx-27104-21munin-node22:19:282
29935050,0irq/33-eth0-tx-1847-21ssh21:07:042
29935050,0irq/33-eth0-tx-16198-21munin-node21:54:292
29935050,0irq/33-eth0-tx-11077-21ssh23:43:042
29935050,0irq/33-eth0-tx-0-21swapper/223:14:222
29935050,0irq/33-eth0-tx-0-21swapper/222:43:512
29935050,0irq/33-eth0-tx-0-21swapper/222:38:072
29935050,0irq/33-eth0-tx-0-21swapper/222:28:532
29935050,0irq/33-eth0-tx-0-21swapper/221:29:522
29935050,0irq/33-eth0-tx-0-21swapper/221:18:442
29925050,0irq/32-eth0-rx-9550-21kthreadcore18:44:353
29925050,0irq/32-eth0-rx-22839-21kthreadcore18:54:323
29925050,0irq/32-eth0-rx-0-21swapper/320:30:353
27150,0ktimersoftd/20-21swapper/221:52:032
259950,0migration/26898-21ntp_offset20:49:372
259950,0migration/25213-21kthreadcore18:19:342
259950,0migration/230763-21sh22:57:212
259950,0migration/22269-21kthreadcore20:29:342
259950,0migration/210807-21kthreadcore23:04:322
259950,0migration/210322-21munin-node22:09:292
19150,0ktimersoftd/10-21swapper/122:14:361
19150,0ktimersoftd/10-21swapper/121:52:031
179950,0migration/18419-21kthreadcore22:24:321
179950,0migration/17221-21latency19:49:331
179950,0migration/15690-21df_abs21:29:271
179950,0migration/14631-21taskset20:11:311
179950,0migration/129582-21taskset20:29:121
179950,0migration/125020-21kthreadcore22:54:321
6549943,0cyclictest5738-21awk18:44:322
6549943,0cyclictest3262-21irqbalance18:34:492
6549943,0cyclictest27604-21awk19:44:322
6549943,0cyclictest23871-21snmpd18:50:102
6549943,0cyclictest23871-21snmpd18:29:402
6549943,0cyclictest23871-21snmpd18:28:402
6499944,0cyclictest0-21swapper/121:39:271
6499943,0cyclictest8756-21uniq18:24:291
6499943,0cyclictest5748-21grep20:14:291
6499943,0cyclictest3262-21irqbalance20:39:291
6499943,0cyclictest3262-21irqbalance18:34:391
6499943,0cyclictest31067-21awk20:29:321
6499943,0cyclictest30860-21awk18:39:301
6499943,0cyclictest26321-21awk23:14:321
6499943,0cyclictest23871-21snmpd23:33:551
6499943,0cyclictest2252-21awk22:04:311
6499943,0cyclictest21450-21awk21:19:301
6499943,0cyclictest20388-21awk19:39:301
6499943,0cyclictest17282-21awk21:54:301
6499943,0cyclictest13451-21awk20:19:301
6499943,0cyclictest12959-21awk18:49:311
6419942,0cyclictest8597-21cut19:09:270
6419942,0cyclictest3680-21cpuspeed_turbos22:24:260
6419942,0cyclictest30166-21df_abs18:39:280
6419942,0cyclictest24679-21latency_hist21:59:140
6419942,0cyclictest16148-21tr19:59:290
6419942,0cyclictest15910-21expr19:14:280
6419941,0cyclictest6802-21sed22:44:300
6419941,0cyclictest30968-21grep19:24:300
6419941,0cyclictest22921-21grep18:34:280
6419941,0cyclictest21364-21tr20:24:270
6419941,0cyclictest1473-21cpu23:19:250
6419941,0cyclictest12745-21fschecks_count19:34:300
6419941,0cyclictest12745-21fschecks_count19:34:300
6419941,0cyclictest12559-21diskstats22:29:260
6419941,0cyclictest11825-21latency_hist18:49:150
6419941,0cyclictest1162-21cut22:04:270
6419941,0cyclictest0-21swapper/023:39:140
6419941,0cyclictest0-21swapper/023:09:140
6419941,0cyclictest0-21swapper/022:54:140
6419941,0cyclictest0-21swapper/022:49:140
6419941,0cyclictest0-21swapper/021:39:140
6419941,0cyclictest0-21swapper/021:29:270
6419941,0cyclictest0-21swapper/020:34:140
6419941,0cyclictest0-21swapper/020:29:140
6419941,0cyclictest0-21swapper/020:14:140
6419941,0cyclictest0-21swapper/020:09:140
6419941,0cyclictest0-21swapper/020:04:150
6419941,0cyclictest0-21swapper/019:04:280
6419941,0cyclictest0-21swapper/018:29:310
6419940,0cyclictest0-21swapper/023:44:290
6419940,0cyclictest0-21swapper/023:29:280
6419940,0cyclictest0-21swapper/022:59:280
6419940,0cyclictest0-21swapper/021:49:260
6419940,0cyclictest0-21swapper/021:14:250
6419940,0cyclictest0-21swapper/020:49:290
6419940,0cyclictest0-21swapper/020:44:270
6419940,0cyclictest0-21swapper/018:54:270
29935040,0irq/33-eth0-tx-29126-21ssh21:24:282
29935040,0irq/33-eth0-tx-2249-21gltestperf23:19:272
29935040,0irq/33-eth0-tx-21894-21fschecks_time20:24:282
29935040,0irq/33-eth0-tx-18122-21telnet20:59:272
29935040,0irq/33-eth0-tx-17058-21cut22:49:322
29935040,0irq/33-eth0-tx-0-21swapper/223:24:402
29935040,0irq/33-eth0-tx-0-21swapper/222:44:372
29935040,0irq/33-eth0-tx-0-21swapper/222:04:272
29935040,0irq/33-eth0-tx-0-21swapper/221:59:262
29935040,0irq/33-eth0-tx-0-21swapper/221:44:162
29935040,0irq/33-eth0-tx-0-21swapper/221:39:212
29935040,0irq/33-eth0-tx-0-21swapper/221:09:242
29935040,0irq/33-eth0-tx-0-21swapper/220:54:262
29935040,0irq/33-eth0-tx-0-21swapper/220:44:412
29935040,0irq/33-eth0-tx-0-21swapper/220:34:332
29935040,0irq/33-eth0-tx-0-21swapper/220:20:092
259940,0migration/24974-21aten_r1power_cu19:29:272
259940,0migration/230897-21fschecks_time20:09:292
259940,0migration/230563-21fschecks_count19:24:302
259940,0migration/230508-21fschecks_time18:39:282
259940,0migration/229668-21kthreadcore18:59:352
259940,0migration/226904-21kthreadcore19:19:342
259940,0migration/224825-21taskset22:14:402
259940,0migration/224531-21irqrtprio20:04:312
259940,0migration/224315-21kthreadcore21:19:322
259940,0migration/221274-21kthreadcore19:39:322
259940,0migration/219962-21kthreadcore23:09:322
259940,0migration/219511-21kthreadcore19:59:342
259940,0migration/218672-21aten_r1power_vo23:29:262
259940,0migration/217520-21kthreadcore20:39:322
259940,0migration/216176-21kthreadcore22:29:312
259940,0migration/215864-21kthreadcore21:34:322
259940,0migration/213753-21kthreadcore19:09:342
259940,0migration/211904-21aten_r1power_vo23:44:272
249940,0watchdog/20-21swapper/219:54:352
179940,0migration/16902-21fschecks_count20:34:291
179940,0migration/1427-21kthreadcore22:39:331
179940,0migration/14222-21kthreadcore23:19:311
179940,0migration/14114-21kthreadcore19:04:331
179940,0migration/13161-21taskset23:39:221
179940,0migration/125794-21irqrtprio20:44:321
179940,0migration/12421-21kthreadcore21:24:351
179940,0migration/123068-21aten_r1power_en20:04:251
179940,0migration/121857-21munin-node19:14:381
179940,0migration/118299-21kthreadcore23:09:321
179940,0migration/116779-21gltestperf19:59:301
179940,0migration/11569-21kthreadcore21:44:341
179940,0migration/114730-21kthreadcore22:29:311
179940,0migration/114719-21fschecks_time21:34:281
179940,0migration/114418-21kthreadcore23:44:331
179940,0migration/112528-21fschecks_time21:14:271
179940,0migration/112092-21latency_hist19:34:151
179940,0migration/112092-21latency_hist19:34:151
179940,0migration/111544-21munin-node23:24:321
179940,0migration/11107-21kthreadcore22:59:341
179940,0migration/110415-21kthreadcore23:04:321
169940,0watchdog/131957-21ssh22:00:541
169940,0watchdog/13192-21sort21:09:271
169940,0watchdog/126525-21munin-node19:44:391
169940,0watchdog/121221-21diskstats22:34:261
169940,0watchdog/117983-21ssh20:59:151
169940,0watchdog/112135-21sort19:54:351
169940,0watchdog/10-21swapper/123:36:221
169940,0watchdog/10-21swapper/122:52:501
169940,0watchdog/10-21swapper/122:20:461
169940,0watchdog/10-21swapper/118:44:351
8315030,0irq/26-0000:00:0-21swapper/219:04:152
6549932,0cyclictest6182-21awk20:14:302
6549932,0cyclictest23871-21snmpd18:55:402
6549932,0cyclictest16727-21awk19:14:312
6549932,0cyclictest15493-21kthreadcore19:34:342
6549932,0cyclictest15493-21kthreadcore19:34:342
6499933,0cyclictest23930-21awk19:19:321
6499932,0cyclictest6935-21awk22:44:311
6499932,0cyclictest31083-21awk19:24:311
6499932,0cyclictest28723-21kthreadcore21:04:321
6499932,0cyclictest2027-21awk18:19:311
6499932,0cyclictest16607-21awk18:29:331
6499931,0cyclictest8461-21munin-node19:09:381
249930,0watchdog/20-21swapper/219:49:272
179930,0migration/19700-21sh22:09:141
169930,0watchdog/10-21swapper/120:49:351
169930,0watchdog/10-21swapper/118:54:351
169920,0watchdog/10-21swapper/119:29:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional