You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-21 - 22:13
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2.osadl.org (updated Thu Aug 21, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2081224026,0sleep00-21swapper/006:21:020
2091123627,0sleep30-21swapper/306:22:133
2105123523,0sleep10-21swapper/106:23:511
2113823127,0sleep20-21swapper/206:24:532
329970,0watchdog/30-21swapper/308:00:593
29925070,0irq/32-eth0-rx-28623-21python11:30:533
339960,0migration/332634-21switchtime08:40:553
339960,0migration/320405-21runrttasks11:26:253
339960,0migration/31597-21munin-node10:40:483
339960,0migration/313869-21sh10:06:343
329960,0watchdog/32312-21kthreadcore10:20:513
329960,0watchdog/30-21swapper/311:38:273
329960,0watchdog/30-21swapper/311:19:273
329960,0watchdog/30-21swapper/311:07:113
329960,0watchdog/30-21swapper/311:02:233
329960,0watchdog/30-21swapper/310:45:513
329960,0watchdog/30-21swapper/310:39:353
329960,0watchdog/30-21swapper/310:34:313
329960,0watchdog/30-21swapper/310:25:313
329960,0watchdog/30-21swapper/309:41:033
329960,0watchdog/30-21swapper/309:35:433
329960,0watchdog/30-21swapper/309:30:433
329960,0watchdog/30-21swapper/309:16:073
329960,0watchdog/30-21swapper/308:51:473
329960,0watchdog/30-21swapper/308:15:433
29925060,0irq/32-eth0-rx-3172-21users09:21:003
29925060,0irq/32-eth0-rx-3059-21grep08:45:463
29925060,0irq/32-eth0-rx-29670-21cut10:55:483
29925060,0irq/32-eth0-rx-25794-21ssh08:40:083
29925060,0irq/32-eth0-rx-19741-21diskstats08:55:463
29925060,0irq/32-eth0-rx-19106-21ps10:50:463
29925060,0irq/32-eth0-rx-17944-21tr09:10:523
29925060,0irq/32-eth0-rx-0-21swapper/311:47:573
29925060,0irq/32-eth0-rx-0-21swapper/309:57:313
29925060,0irq/32-eth0-rx-0-21swapper/309:54:033
29925060,0irq/32-eth0-rx-0-21swapper/309:02:253
212699960,0cyclictest0-21swapper/208:41:332
212699960,0cyclictest0-21swapper/208:29:532
149960,0migration/06098-21sh09:25:480
149960,0migration/028380-21kthreadcore06:50:510
8315050,0irq/26-0000:00:799-21awk11:55:302
339950,0migration/39459-21munin-node08:30:453
339950,0migration/39324-21kthreadcore11:40:493
339950,0migration/37236-21kthreadcore09:05:533
339950,0migration/35580-21kthreadcore07:40:523
339950,0migration/35266-21kthreadcore06:35:493
339950,0migration/33792-21kthreadcore08:25:523
339950,0migration/331241-21kthreadcore06:30:513
339950,0migration/327384-21kthreadcore11:50:493
339950,0migration/327252-21latency06:25:503
339950,0migration/32513-21kthreadcore10:00:493
339950,0migration/32398-21munin-node07:15:553
339950,0migration/318338-21munin-node06:45:483
339950,0migration/315572-21fschecks_count10:10:453
339950,0migration/311680-21munin-node06:40:473
339950,0migration/310626-21kthreadcore07:45:493
329950,0watchdog/319762-21ssh11:10:393
329950,0watchdog/30-21swapper/310:15:473
329950,0watchdog/30-21swapper/309:45:313
329950,0watchdog/30-21swapper/308:20:353
329950,0watchdog/30-21swapper/308:06:113
329950,0watchdog/30-21swapper/307:56:233
329950,0watchdog/30-21swapper/307:54:193
329950,0watchdog/30-21swapper/307:36:393
329950,0watchdog/30-21swapper/307:32:433
329950,0watchdog/30-21swapper/307:25:353
329950,0watchdog/30-21swapper/307:20:433
329950,0watchdog/30-21swapper/307:20:433
329950,0watchdog/30-21swapper/307:10:353
329950,0watchdog/30-21swapper/307:01:033
329950,0watchdog/30-21swapper/306:56:393
329950,0watchdog/30-21swapper/306:50:593
29935050,0irq/33-eth0-tx-16040-21munin-node11:45:502
29935050,0irq/33-eth0-tx-0-21swapper/209:36:292
29935050,0irq/33-eth0-tx-0-21swapper/209:16:412
29935050,0irq/33-eth0-tx-0-21swapper/008:25:460
29935050,0irq/33-eth0-tx-0-21swapper/008:14:430
29935050,0irq/33-eth0-tx-0-21swapper/006:44:270
29925050,0irq/32-eth0-rx-0-21swapper/311:20:353
29925050,0irq/32-eth0-rx-0-21swapper/309:25:463
29925050,0irq/32-eth0-rx-0-21swapper/308:10:553
27150,0ktimersoftd/20-21swapper/209:58:102
259950,0migration/229785-21latency_hist09:40:302
259950,0migration/229123-21irqcore09:00:492
259950,0migration/222562-21kthreadcore08:55:512
259950,0migration/221014-21sh09:34:542
259950,0migration/219114-21kthreadcore10:30:502
212699955,0cyclictest0-21swapper/211:39:232
212699955,0cyclictest0-21swapper/211:00:172
212699955,0cyclictest0-21swapper/209:05:402
212699955,0cyclictest0-21swapper/207:15:562
212699954,0cyclictest23871-21snmpd07:48:552
212699950,0cyclictest0-21swapper/207:44:292
212679955,0cyclictest0-21swapper/111:51:431
212679955,0cyclictest0-21swapper/109:00:301
212679950,0cyclictest0-21swapper/111:45:151
179950,0migration/14212-21kthreadcore11:00:471
169950,0watchdog/10-21swapper/107:55:431
149950,0migration/09874-21munin-node09:05:550
149950,0migration/09267-21kthreadcore07:00:500
149950,0migration/08959-21sshd09:45:510
149950,0migration/07766-21kthreadcore08:05:490
149950,0migration/07096-21sh10:23:460
149950,0migration/07032-21kthreadcore06:35:490
149950,0migration/05447-21taskset10:01:340
149950,0migration/04219-21kthreadcore06:55:510
149950,0migration/032744-21kthreadcore09:40:480
149950,0migration/029233-21irqrtprio09:00:490
149950,0migration/028996-21kthreadcore10:55:480
149950,0migration/02648-21unixbench-2d11:16:000
149950,0migration/025788-21timerandwakeup08:55:580
149950,0migration/025417-21taskset10:53:510
149950,0migration/024537-21kthreadcore09:35:490
149950,0migration/023673-21munin-node10:15:440
149950,0migration/022883-21if_eth1.207:10:470
149950,0migration/021775-21kthreadcore11:10:480
149950,0migration/021653-21kthreadcore07:30:500
149950,0migration/021407-21kthreadcore08:35:500
149950,0migration/020892-21kthreadcore06:45:530
149950,0migration/020367-21sh09:50:510
149950,0migration/019192-21fschecks_time08:15:450
149950,0migration/018019-21kthreadcore07:50:490
149950,0migration/016200-21sh08:32:190
149950,0migration/016071-21idleruntime-cro10:30:300
149950,0migration/015990-21kthreadcore09:30:480
149950,0migration/013192-21kthreadcore11:40:510
149950,0migration/012610-21kthreadcore08:50:500
149950,0migration/011720-21fschecks_time07:25:450
149950,0migration/011151-21munin-node11:20:490
8315040,0irq/26-0000:00:0-21swapper/210:35:302
329940,0watchdog/30-21swapper/307:05:473
29935040,0irq/33-eth0-tx-9152-21ssh10:45:392
29935040,0irq/33-eth0-tx-0-21swapper/211:30:342
29935040,0irq/33-eth0-tx-0-21swapper/211:05:352
29935040,0irq/33-eth0-tx-0-21swapper/210:53:492
29935040,0irq/33-eth0-tx-0-21swapper/210:40:542
29935040,0irq/33-eth0-tx-0-21swapper/210:26:002
29935040,0irq/33-eth0-tx-0-21swapper/210:21:302
29935040,0irq/33-eth0-tx-0-21swapper/210:05:442
29935040,0irq/33-eth0-tx-0-21swapper/209:46:542
29935040,0irq/33-eth0-tx-0-21swapper/209:21:332
29935040,0irq/33-eth0-tx-0-21swapper/208:31:082
259940,0migration/29862-21kthreadcore09:25:502
259940,0migration/27395-21kthreadcore11:00:482
259940,0migration/24980-21switchtime10:01:002
259940,0migration/225919-21ntp_offset11:10:512
259940,0migration/224029-21kthreadcore07:55:492
259940,0migration/219786-21kthreadcore09:50:502
259940,0migration/218273-21kthreadcore07:05:502
259940,0migration/212397-21aten_r1power_vo09:10:462
249940,0watchdog/25873-21if_err_eth111:20:432
249940,0watchdog/229908-21munin-node11:15:512
249940,0watchdog/229258-21aten_r1power_en08:00:432
249940,0watchdog/214936-21munin-node07:50:432
249940,0watchdog/214701-21cut08:10:552
249940,0watchdog/20-21swapper/211:26:352
212699944,0cyclictest26420-21irqcore06:50:502
212699944,0cyclictest0-21swapper/207:11:322
212699943,0cyclictest3599-21lldpd11:40:572
212699943,0cyclictest3599-21lldpd07:37:522
212699943,0cyclictest3262-21irqbalance10:15:352
212699943,0cyclictest3262-21irqbalance08:50:352
212699943,0cyclictest3262-21irqbalance08:45:452
212699943,0cyclictest23871-21snmpd08:18:262
212699943,0cyclictest23871-21snmpd07:21:402
212699943,0cyclictest23871-21snmpd07:21:402
212699943,0cyclictest23871-21snmpd06:38:102
212699943,0cyclictest23871-21snmpd06:30:402
212699943,0cyclictest22583-21awk06:25:472
212699943,0cyclictest18695-21diskstats07:30:462
212699943,0cyclictest16294-21grep10:10:482
212699943,0cyclictest12140-21awk07:25:462
212699943,0cyclictest11865-21awk06:40:492
212699940,0cyclictest0-21swapper/208:35:452
212679944,0cyclictest0-21swapper/109:52:321
212679943,0cyclictest6076-21awk09:25:481
212679943,0cyclictest5902-21awk11:20:431
212679943,0cyclictest5222-21awk08:05:481
212679943,0cyclictest3599-21lldpd10:41:131
212679943,0cyclictest3599-21lldpd10:35:431
212679943,0cyclictest3599-21lldpd10:26:381
212679943,0cyclictest3599-21lldpd09:31:291
212679943,0cyclictest3599-21lldpd08:39:571
212679943,0cyclictest3599-21lldpd08:33:271
212679943,0cyclictest3599-21lldpd07:21:331
212679943,0cyclictest3599-21lldpd07:21:331
212679943,0cyclictest3599-21lldpd07:01:451
212679943,0cyclictest3329-21awk08:45:481
212679943,0cyclictest29731-21awk06:30:481
212679943,0cyclictest29690-21awk11:15:501
212679943,0cyclictest26780-21awk08:20:461
212679943,0cyclictest23871-21snmpd11:39:121
212679943,0cyclictest23871-21snmpd10:00:561
212679943,0cyclictest23871-21snmpd09:36:261
212679943,0cyclictest23871-21snmpd06:37:101
212679943,0cyclictest23679-21awk09:55:461
212679943,0cyclictest23327-21awk11:30:461
212679943,0cyclictest21768-21sadc07:55:301
212679943,0cyclictest17456-21awk10:30:481
212679943,0cyclictest1349-21awk07:40:491
212679943,0cyclictest13428-21awk09:10:501
212679943,0cyclictest12125-21awk07:25:461
212679943,0cyclictest1201-21awk06:55:491
212679943,0cyclictest10336-21awk10:45:461
212679940,0cyclictest0-21swapper/108:41:541
212659943,0cyclictest3599-21lldpd11:49:570
212659943,0cyclictest3599-21lldpd10:49:230
212659943,0cyclictest3599-21lldpd08:00:550
212659943,0cyclictest3561-21awk11:00:450
212659943,0cyclictest3262-21irqbalance11:35:540
212659943,0cyclictest3262-21irqbalance11:30:340
212659943,0cyclictest3262-21irqbalance09:55:550
212659943,0cyclictest3262-21irqbalance09:11:450
212659943,0cyclictest3014-21kworker/0:108:41:310
212659943,0cyclictest29210-21grep09:20:470
212659943,0cyclictest23871-21snmpd08:46:260
212659943,0cyclictest14593-21awk11:25:450
212659943,0cyclictest12215-21awk11:05:480
212659943,0cyclictest11757-21kworker/0:007:21:250
212659943,0cyclictest11757-21kworker/0:007:21:250
212659940,0cyclictest14532-21munin-node07:05:540
179940,0migration/17263-21cpuspeed_turbos10:05:451
179940,0migration/13333-21kthreadcore08:25:521
179940,0migration/131335-21kthreadcore07:15:541
179940,0migration/125406-21latency_hist07:35:311
179940,0migration/120329-21kthreadcore07:30:501
179940,0migration/115331-21kthreadcore11:25:481
179940,0migration/114545-21kthreadcore08:50:501
179940,0migration/111543-21kthreadcore07:45:501
169940,0watchdog/16913-21hddtemp_smartct09:45:471
169940,0watchdog/16266-21ssh10:21:071
169940,0watchdog/129344-21kthreadcore10:15:511
169940,0watchdog/129259-21aten_r1power_en08:00:431
169940,0watchdog/121550-21sort10:10:591
169940,0watchdog/119995-21ssh11:10:431
169940,0watchdog/115758-21kthreadcore06:40:511
169940,0watchdog/10-21swapper/111:45:471
169940,0watchdog/10-21swapper/110:50:471
169940,0watchdog/10-21swapper/106:25:431
149940,0migration/0963-21kthreadcore07:15:550
149940,0migration/04917-21kthreadcore10:40:520
149940,0migration/029661-21kthreadcore08:20:490
149940,0migration/027116-21kthreadcore07:35:480
149940,0migration/023005-21kthreadcore09:15:520
8315030,0irq/26-0000:00:0-21swapper/208:21:062
29935030,0irq/33-eth0-tx-0-21swapper/006:25:430
212699932,0cyclictest6337-21sensors06:55:552
212699932,0cyclictest23871-21snmpd08:08:402
212699932,0cyclictest23871-21snmpd07:02:402
212679932,0cyclictest30913-21awk09:40:461
212679932,0cyclictest26392-21awk06:50:491
212679932,0cyclictest26269-21munin-node10:55:421
212679932,0cyclictest23871-21snmpd11:05:421
212679932,0cyclictest22885-21awk07:10:471
212679932,0cyclictest20959-21grep07:05:571
212679932,0cyclictest19191-21awk06:45:521
212679930,0cyclictest21695-21if_err_eth1.209:15:501
212679930,0cyclictest12679-21munin-node08:10:541
212659933,0cyclictest20294-21kworker/0:210:35:430
212659933,0cyclictest20294-21kworker/0:210:30:310
212659933,0cyclictest20294-21kworker/0:210:10:310
212659933,0cyclictest20294-21kworker/0:210:05:470
212659933,0cyclictest15726-21kworker/0:206:30:520
212659933,0cyclictest11757-21kworker/0:007:55:410
212659933,0cyclictest11757-21kworker/0:007:40:370
212659932,0cyclictest26945-21taskset11:50:480
212659930,0cyclictest7997-21diskstats07:45:450
169930,0watchdog/10-21swapper/109:05:551
169930,0watchdog/10-21swapper/108:55:471
169930,0watchdog/10-21swapper/108:15:431
249920,0watchdog/20-21swapper/206:45:352
169920,0watchdog/10-21swapper/109:20:311
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional