You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 17:22
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Mon Nov 24, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3184424332,0sleep00-21swapper/006:16:360
3199923927,0sleep30-21swapper/306:18:233
3208823726,0sleep10-21swapper/106:19:251
3175023425,0sleep20-21swapper/206:15:302
29925070,0irq/32-eth0-rx-0-21swapper/311:20:053
4160,0ktimersoftd/00-21swapper/009:11:430
339960,0migration/37950-21munin-node09:34:563
339960,0migration/36369-21sh11:27:103
339960,0migration/34334-21sh08:34:563
339960,0migration/319520-21sh08:40:133
321869960,0cyclictest0-21swapper/208:24:522
321859960,0cyclictest8866-21ssh10:10:171
329960,0watchdog/330158-21grep11:05:013
329960,0watchdog/321962-21tr10:20:053
329960,0watchdog/30-21swapper/311:43:413
329960,0watchdog/30-21swapper/310:45:013
329960,0watchdog/30-21swapper/310:42:533
329960,0watchdog/30-21swapper/310:25:173
329960,0watchdog/30-21swapper/309:45:013
329960,0watchdog/30-21swapper/309:39:573
329960,0watchdog/30-21swapper/309:29:573
329960,0watchdog/30-21swapper/309:25:413
329960,0watchdog/30-21swapper/309:21:013
329960,0watchdog/30-21swapper/309:16:093
329960,0watchdog/30-21swapper/309:09:013
329960,0watchdog/30-21swapper/308:56:413
329960,0watchdog/30-21swapper/307:35:023
329960,0watchdog/30-21swapper/306:34:583
29925060,0irq/32-eth0-rx-6631-21cut10:10:043
29925060,0irq/32-eth0-rx-30098-21sed08:49:593
29925060,0irq/32-eth0-rx-23782-21grep08:45:043
29925060,0irq/32-eth0-rx-18927-21ssh10:37:093
29925060,0irq/32-eth0-rx-12936-21grep11:30:033
29925060,0irq/32-eth0-rx-0-21swapper/311:19:043
29925060,0irq/32-eth0-rx-0-21swapper/311:00:023
29925060,0irq/32-eth0-rx-0-21swapper/310:49:483
29925060,0irq/32-eth0-rx-0-21swapper/310:08:203
29925060,0irq/32-eth0-rx-0-21swapper/308:32:443
29925060,0irq/32-eth0-rx-0-21swapper/308:27:353
339950,0migration/37405-21aten_r1power_fr06:24:573
339950,0migration/36455-21latency11:45:053
339950,0migration/36153-21kthreadcore07:30:043
339950,0migration/34355-21aten_r1power_cu08:14:583
339950,0migration/330663-21latency_hist09:09:453
339950,0migration/330512-21kthreadcore06:40:043
339950,0migration/325185-21kthreadcore08:05:043
339950,0migration/314649-21latency_hist07:14:453
339950,0migration/313566-21fschecks_time10:55:003
321869955,0cyclictest0-21swapper/210:59:122
321869955,0cyclictest0-21swapper/210:27:402
321869955,0cyclictest0-21swapper/210:01:592
321869955,0cyclictest0-21swapper/209:50:002
321869955,0cyclictest0-21swapper/209:45:252
321869955,0cyclictest0-21swapper/209:24:562
321869950,0cyclictest0-21swapper/209:17:312
321859955,0cyclictest0-21swapper/111:37:091
321859955,0cyclictest0-21swapper/110:20:511
321859955,0cyclictest0-21swapper/108:54:461
321859955,0cyclictest0-21swapper/106:40:171
321859955,0cyclictest0-21swapper/106:34:461
321859950,0cyclictest0-21swapper/110:52:321
321859950,0cyclictest0-21swapper/109:21:531
321859950,0cyclictest0-21swapper/108:53:021
321849955,0cyclictest0-21swapper/009:20:130
321849950,0cyclictest0-21swapper/011:21:370
321849950,0cyclictest0-21swapper/010:47:250
329950,0watchdog/30-21swapper/311:34:573
329950,0watchdog/30-21swapper/311:09:573
329950,0watchdog/30-21swapper/310:29:493
329950,0watchdog/30-21swapper/310:14:453
329950,0watchdog/30-21swapper/309:55:293
329950,0watchdog/30-21swapper/308:10:413
329950,0watchdog/30-21swapper/307:59:453
329950,0watchdog/30-21swapper/307:45:333
329950,0watchdog/30-21swapper/307:29:063
329950,0watchdog/30-21swapper/307:21:143
329950,0watchdog/30-21swapper/307:10:023
329950,0watchdog/30-21swapper/307:05:383
329950,0watchdog/30-21swapper/307:00:023
329950,0watchdog/30-21swapper/306:57:343
329950,0watchdog/30-21swapper/306:50:023
329950,0watchdog/30-21swapper/306:44:583
329950,0watchdog/30-21swapper/306:30:383
329950,0watchdog/30-21swapper/306:20:023
29935050,0irq/33-eth0-tx-15011-21taskset09:36:192
29935050,0irq/33-eth0-tx-0-21swapper/008:19:450
29925050,0irq/32-eth0-rx-20438-21python09:00:103
29925050,0irq/32-eth0-rx-0-21swapper/309:59:533
29925050,0irq/32-eth0-rx-0-21swapper/309:49:573
29925050,0irq/32-eth0-rx-0-21swapper/308:19:563
29925050,0irq/32-eth0-rx-0-21swapper/307:54:593
29925050,0irq/32-eth0-rx-0-21swapper/307:50:003
29925050,0irq/32-eth0-rx-0-21swapper/307:39:563
259950,0migration/219807-21dump-pmu-power10:39:442
179950,0migration/13175-21kthreadcore11:45:041
179950,0migration/131744-21munin-node09:10:011
179950,0migration/129198-21kthreadcore10:25:041
179950,0migration/127772-21mdstat07:20:041
179950,0migration/125522-21kthreadcore08:25:041
179950,0migration/118482-21taskset08:20:571
179950,0migration/117761-21munin-node10:35:071
169950,0watchdog/16766-21sh09:14:451
169950,0watchdog/127074-21kthreadcore08:45:051
149950,0migration/08894-21kthreadcore06:45:040
149950,0migration/05420-21munin-node08:35:030
149950,0migration/031250-21kthreadcore10:05:030
149950,0migration/02386-21kthreadcore07:25:040
149950,0migration/023504-21kthreadcore08:45:040
149950,0migration/022495-21kthreadcore10:20:050
149950,0migration/020874-21sh10:59:440
149950,0migration/020695-21munin-node06:30:060
149950,0migration/017497-21kthreadcore09:00:040
149950,0migration/016394-21taskset09:56:510
149950,0migration/015215-21cpuspeed_turbos07:59:550
149950,0migration/013432-21kthreadcore10:15:040
321869944,0cyclictest0-21swapper/211:39:532
321869944,0cyclictest0-21swapper/209:07:452
321869943,0cyclictest8835-21awk07:55:022
321869943,0cyclictest3599-21lldpd11:25:052
321869943,0cyclictest3599-21lldpd07:47:522
321869943,0cyclictest3599-21lldpd06:59:342
321869943,0cyclictest30306-21awk07:25:022
321869943,0cyclictest23871-21snmpd10:41:152
321869943,0cyclictest23871-21snmpd08:06:142
321869943,0cyclictest23871-21snmpd07:06:282
321869943,0cyclictest23871-21snmpd06:36:132
321869943,0cyclictest23871-21snmpd06:21:132
321869943,0cyclictest13871-21awk08:40:002
321869940,0cyclictest0-21swapper/210:49:592
321859944,0cyclictest0-21swapper/109:57:491
321859943,0cyclictest6918-21awk11:09:591
321859943,0cyclictest4152-21awk10:30:011
321859943,0cyclictest27040-21awk10:05:001
321859943,0cyclictest25876-21awk09:45:011
321859943,0cyclictest25522-21awk11:40:001
321859943,0cyclictest23871-21snmpd11:29:451
321859943,0cyclictest23871-21snmpd10:04:301
321859943,0cyclictest23871-21snmpd09:49:451
321859943,0cyclictest23871-21snmpd07:39:441
321859943,0cyclictest23564-21awk09:05:031
321859943,0cyclictest15272-21awk11:15:001
321859943,0cyclictest13941-21awk08:40:001
321859943,0cyclictest1382-21awk07:05:021
321849944,0cyclictest0-21swapper/011:43:420
321849944,0cyclictest0-21swapper/009:26:320
321849944,0cyclictest0-21swapper/008:30:530
321849943,0cyclictest9008-21awk09:35:010
321849943,0cyclictest8962-21awk07:55:030
321849943,0cyclictest7237-21turbostat.cron11:29:440
321849943,0cyclictest3599-21lldpd09:33:080
321849943,0cyclictest3262-21irqbalance11:09:290
321849943,0cyclictest24710-21tail11:00:040
321849943,0cyclictest23871-21snmpd10:39:450
321849943,0cyclictest23871-21snmpd06:35:580
321849943,0cyclictest21098-21awk08:25:020
321849943,0cyclictest17025-21awk11:35:010
321849943,0cyclictest15369-21awk11:15:000
321849943,0cyclictest13901-21awk08:40:000
321849940,0cyclictest11366-21munin-run08:19:450
321849940,0cyclictest11014-21munin-node06:49:590
321849940,0cyclictest0-21swapper/009:48:170
321849940,0cyclictest0-21swapper/007:19:580
29935040,0irq/33-eth0-tx-3034-21munin-node10:30:032
29935040,0irq/33-eth0-tx-2912-21ssh08:31:292
29935040,0irq/33-eth0-tx-22169-21ssh11:00:012
29935040,0irq/33-eth0-tx-1987-21munin-node10:10:012
29935040,0irq/33-eth0-tx-15967-21ssh11:34:542
29935040,0irq/33-eth0-tx-14735-21munin-node07:15:080
29935040,0irq/33-eth0-tx-11371-21diskmemload11:45:252
29935040,0irq/33-eth0-tx-11371-21diskmemload09:10:292
29935040,0irq/33-eth0-tx-0-21swapper/211:20:032
29935040,0irq/33-eth0-tx-0-21swapper/211:15:242
29935040,0irq/33-eth0-tx-0-21swapper/211:04:542
29935040,0irq/33-eth0-tx-0-21swapper/210:15:472
29935040,0irq/33-eth0-tx-0-21swapper/209:55:342
29935040,0irq/33-eth0-tx-0-21swapper/209:29:542
29935040,0irq/33-eth0-tx-0-21swapper/209:00:042
29935040,0irq/33-eth0-tx-0-21swapper/208:56:272
29935040,0irq/33-eth0-tx-0-21swapper/208:20:102
29935040,0irq/33-eth0-tx-0-21swapper/007:40:110
29935040,0irq/33-eth0-tx-0-21swapper/007:34:580
29935040,0irq/33-eth0-tx-0-21swapper/006:55:020
259940,0migration/27478-21sh11:29:472
259940,0migration/25336-21idleruntime08:15:022
259940,0migration/231629-21munin-node07:00:102
259940,0migration/228528-21munin-node10:44:552
249940,0watchdog/23289-21grep08:50:052
249940,0watchdog/23195-21cut07:50:052
249940,0watchdog/227075-21kthreadcore08:45:052
249940,0watchdog/225938-21munin-node10:04:572
249940,0watchdog/221444-21kthreadcore07:40:092
249940,0watchdog/217949-21kthreadcore09:20:052
249940,0watchdog/211298-21cut07:34:582
249940,0watchdog/210190-21grep11:10:012
249940,0watchdog/20-21swapper/210:20:012
249940,0watchdog/20-21swapper/209:39:572
179940,0migration/19147-21irqcore07:55:031
179940,0migration/16293-21vmstat09:30:121
179940,0migration/13082-21kthreadcore06:20:031
179940,0migration/128061-21kthreadcore09:25:041
179940,0migration/12493-21kthreadcore08:10:151
179940,0migration/122965-21cpuspeed_turbos11:20:001
179940,0migration/118619-21df_abs06:55:031
179940,0migration/118462-21kthreadcore08:00:011
179940,0migration/118404-21kthreadcore06:30:021
179940,0migration/116060-21kthreadcore10:55:041
179940,0migration/115349-21fschecks_count07:15:001
179940,0migration/113587-21pmu-power07:10:061
179940,0migration/111863-21fschecks_time06:49:591
169940,0watchdog/19155-21kthreadcore08:15:051
169940,0watchdog/15327-21smartctl08:35:011
169940,0watchdog/128510-21ssh10:44:531
169940,0watchdog/128028-21memory08:05:051
169940,0watchdog/121299-21iostat10:40:011
169940,0watchdog/11947-21ssh08:30:051
169940,0watchdog/116478-21aten_r1power_en09:39:571
169940,0watchdog/110029-21python06:45:101
169940,0watchdog/10-21swapper/107:40:051
149940,0migration/07506-21latency_hist07:09:450
149940,0migration/06922-21runrttasks07:05:300
149940,0migration/06216-21fschecks_count08:55:010
149940,0migration/05462-21kthreadcore07:50:060
149940,0migration/05099-21kthreadcore10:30:040
149940,0migration/05024-21taskset08:53:440
149940,0migration/031350-21kthreadcore08:10:140
149940,0migration/028535-21irqrtprio10:25:030
149940,0migration/02542-21kthreadcore06:20:030
149940,0migration/025378-21kthreadcore08:05:040
149940,0migration/023843-21taskset09:42:210
149940,0migration/014347-21sh09:17:520
149940,0migration/010593-21kthreadcore11:30:020
8315030,0irq/26-0000:00:0-21swapper/208:09:452
8315030,0irq/26-0000:00:0-21swapper/207:59:512
8315030,0irq/26-0000:00:0-21swapper/207:15:132
4130,0ktimersoftd/07723-21sh11:46:280
321869933,0cyclictest3599-21lldpd07:29:502
321869933,0cyclictest3599-21lldpd07:22:402
321869932,0cyclictest23871-21snmpd06:26:432
321869930,0cyclictest0-21swapper/208:35:012
321859933,0cyclictest0-21swapper/107:00:281
321859932,0cyclictest8687-21grep09:34:591
321859932,0cyclictest8395-21awk06:25:011
321859932,0cyclictest27490-21sensors11:00:081
321859932,0cyclictest1706-21irqcore07:50:051
321859932,0cyclictest14544-21diskstats08:59:591
321859930,0cyclictest9040-21sshd07:30:051
321859930,0cyclictest32656-21grep11:05:061
321859930,0cyclictest0-21swapper/107:48:141
321849933,0cyclictest434-21kworker/0:211:11:370
321849933,0cyclictest434-21kworker/0:210:50:030
321849933,0cyclictest434-21kworker/0:210:09:550
321849933,0cyclictest434-21kworker/0:209:50:150
321849932,0cyclictest32669-21sadc07:49:450
321849932,0cyclictest31563-21kthreadcore06:40:040
321849932,0cyclictest26538-21awk07:00:010
321849932,0cyclictest19444-21tr10:00:020
321849932,0cyclictest12748-21awk10:35:020
321849931,0cyclictest4854-21cut07:30:010
321849930,0cyclictest23074-21grep09:05:000
29935030,0irq/33-eth0-tx-0-21swapper/006:24:570
259930,0migration/25501-21kthreadcore06:45:032
259930,0migration/214323-21kthreadcore06:50:032
249930,0watchdog/20-21swapper/207:10:022
249930,0watchdog/20-21swapper/206:44:262
249930,0watchdog/20-21swapper/206:29:462
169930,0watchdog/1388-21kthreadcore11:25:011
169930,0watchdog/10-21swapper/110:15:491
169930,0watchdog/10-21swapper/107:25:061
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional