You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-03 - 11:34
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2.osadl.org (updated Thu Jul 03, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
199823726,0sleep20-21swapper/218:27:512
183123626,0sleep10-21swapper/118:25:541
172723626,0sleep30-21swapper/318:24:443
184023524,0sleep00-21swapper/018:26:010
8315080,0irq/26-0000:00:0-21swapper/320:33:323
329970,0watchdog/318393-21fschecks_count20:13:493
329970,0watchdog/315235-21cut20:48:523
329970,0watchdog/30-21swapper/321:21:283
29925070,0irq/32-eth0-rx-0-21swapper/322:33:203
21319970,0cyclictest31521-21grep19:58:500
339960,0migration/330894-21munin-node19:33:523
339960,0migration/324121-21munin-node19:28:583
339960,0migration/315473-21sh22:59:053
329960,0watchdog/329493-21kthreadcore23:28:523
329960,0watchdog/30-21swapper/323:52:243
329960,0watchdog/30-21swapper/322:54:283
329960,0watchdog/30-21swapper/322:43:523
329960,0watchdog/30-21swapper/322:43:203
329960,0watchdog/30-21swapper/322:33:483
329960,0watchdog/30-21swapper/322:11:283
329960,0watchdog/30-21swapper/321:49:563
329960,0watchdog/30-21swapper/321:42:203
329960,0watchdog/30-21swapper/321:33:443
329960,0watchdog/30-21swapper/321:26:283
329960,0watchdog/30-21swapper/321:14:323
329960,0watchdog/30-21swapper/321:11:363
329960,0watchdog/30-21swapper/321:03:403
329960,0watchdog/30-21swapper/320:53:483
329960,0watchdog/30-21swapper/320:30:053
329960,0watchdog/30-21swapper/319:04:013
329960,0watchdog/30-21swapper/318:49:573
329960,0watchdog/30-21swapper/318:43:533
29925060,0irq/32-eth0-rx-7412-21cut21:58:573
29925060,0irq/32-eth0-rx-32073-21kthreadcore22:13:543
29925060,0irq/32-eth0-rx-12217-21port_sshpriv18:53:543
29925060,0irq/32-eth0-rx-0-21swapper/323:47:033
29925060,0irq/32-eth0-rx-0-21swapper/323:38:443
29925060,0irq/32-eth0-rx-0-21swapper/323:23:043
29925060,0irq/32-eth0-rx-0-21swapper/323:13:233
29925060,0irq/32-eth0-rx-0-21swapper/323:04:343
29925060,0irq/32-eth0-rx-0-21swapper/322:50:583
29925060,0irq/32-eth0-rx-0-21swapper/322:26:133
29925060,0irq/32-eth0-rx-0-21swapper/322:18:463
29925060,0irq/32-eth0-rx-0-21swapper/322:04:543
29925060,0irq/32-eth0-rx-0-21swapper/321:45:263
29925060,0irq/32-eth0-rx-0-21swapper/320:45:293
29925060,0irq/32-eth0-rx-0-21swapper/320:45:293
29925060,0irq/32-eth0-rx-0-21swapper/320:38:453
21349966,0cyclictest0-21swapper/221:27:022
21349965,0cyclictest0-21swapper/218:43:512
339950,0migration/39254-21munin-node19:18:543
339950,0migration/314534-21kthreadcore18:33:493
339950,0migration/31315-21latency20:58:523
329950,0watchdog/329695-21cut19:13:453
329950,0watchdog/30-21swapper/323:54:163
329950,0watchdog/30-21swapper/323:35:403
329950,0watchdog/30-21swapper/323:26:123
329950,0watchdog/30-21swapper/323:13:563
329950,0watchdog/30-21swapper/321:53:363
329950,0watchdog/30-21swapper/320:24:253
329950,0watchdog/30-21swapper/320:20:333
329950,0watchdog/30-21swapper/320:11:333
329950,0watchdog/30-21swapper/320:03:373
329950,0watchdog/30-21swapper/319:58:533
329950,0watchdog/30-21swapper/319:53:453
329950,0watchdog/30-21swapper/319:48:493
329950,0watchdog/30-21swapper/319:45:373
329950,0watchdog/30-21swapper/319:39:173
329950,0watchdog/30-21swapper/319:24:333
329950,0watchdog/30-21swapper/319:08:453
329950,0watchdog/30-21swapper/318:58:373
329950,0watchdog/30-21swapper/318:39:333
29935050,0irq/33-eth0-tx-32357-21munin-node20:23:500
29935050,0irq/33-eth0-tx-1301-21memory23:28:542
29935050,0irq/33-eth0-tx-0-21swapper/019:28:540
29925050,0irq/32-eth0-rx-6671-21kthreadcore18:28:503
29925050,0irq/32-eth0-rx-0-21swapper/321:28:453
259950,0migration/215209-21aten_r1power_po22:43:442
21349955,0cyclictest0-21swapper/221:44:222
21349955,0cyclictest0-21swapper/221:35:422
21349950,0cyclictest0-21swapper/223:13:272
21349950,0cyclictest0-21swapper/222:57:132
21349950,0cyclictest0-21swapper/221:54:562
21349950,0cyclictest0-21swapper/221:13:092
21319955,0cyclictest0-21swapper/023:38:460
21319955,0cyclictest0-21swapper/022:53:410
21319950,0cyclictest0-21swapper/023:38:180
21319950,0cyclictest0-21swapper/022:24:500
21319950,0cyclictest0-21swapper/021:37:030
21319950,0cyclictest0-21swapper/021:30:170
21319950,0cyclictest0-21swapper/020:29:440
179950,0migration/19899-21kthreadcore19:43:501
179950,0migration/18264-21kthreadcore21:03:501
179950,0migration/16209-21latency_hist22:38:311
179950,0migration/129818-21irqrtprio19:13:451
179950,0migration/124770-21fschecks_time18:43:471
179950,0migration/123496-21kthreadcore22:28:531
179950,0migration/119281-21latency_hist22:08:311
179950,0migration/113034-21aten_r1power_po22:23:441
179950,0migration/112654-21kthreadcore20:28:531
169950,0watchdog/111989-21if_eth1.222:03:521
149950,0migration/09515-21switchtime19:18:590
149950,0migration/05153-21idleruntime21:03:470
149950,0migration/04748-21kthreadcore20:43:510
149950,0migration/04748-21kthreadcore20:43:510
149950,0migration/031843-21kthreadcore22:33:500
149950,0migration/031390-21kthreadcore20:58:510
149950,0migration/028918-21sh22:30:510
149950,0migration/027501-21pmu-power19:53:530
149950,0migration/024927-21aten2.4-expect19:33:430
149950,0migration/021942-21kthreadcore21:48:500
149950,0migration/019143-21kthreadcore22:43:510
8315040,0irq/26-0000:00:0-21swapper/223:53:562
4140,0ktimersoftd/00-21swapper/023:49:200
29935040,0irq/33-eth0-tx-3449-21kthreadcore21:18:532
29935040,0irq/33-eth0-tx-28362-21munin-node19:13:510
29935040,0irq/33-eth0-tx-21748-21ssh21:29:012
29935040,0irq/33-eth0-tx-2108-21munin-node21:58:532
29935040,0irq/33-eth0-tx-17354-21munin-node23:03:492
29935040,0irq/33-eth0-tx-1259-21munin-node20:43:542
29935040,0irq/33-eth0-tx-1259-21munin-node20:43:542
29935040,0irq/33-eth0-tx-0-21swapper/223:38:452
29935040,0irq/33-eth0-tx-0-21swapper/222:34:432
29935040,0irq/33-eth0-tx-0-21swapper/222:04:412
29935040,0irq/33-eth0-tx-0-21swapper/221:38:492
29935040,0irq/33-eth0-tx-0-21swapper/220:56:582
29935040,0irq/33-eth0-tx-0-21swapper/220:48:522
29935040,0irq/33-eth0-tx-0-21swapper/020:08:470
259940,0migration/23355-21cpu19:18:462
259940,0migration/227821-21kthreadcore20:38:502
259940,0migration/224880-21if_eth022:48:472
259940,0migration/218788-21kthreadcore20:33:522
259940,0migration/210917-21aten_r1power_en23:18:452
249940,0watchdog/25013-21kthreadcore23:13:522
249940,0watchdog/23823-21kthreadcore20:23:532
249940,0watchdog/229696-21irqcore19:13:452
249940,0watchdog/221518-21idleruntime19:03:492
249940,0watchdog/215257-21cut19:23:532
249940,0watchdog/20-21swapper/223:48:522
249940,0watchdog/20-21swapper/221:03:322
249940,0watchdog/20-21swapper/220:28:492
21349943,0cyclictest839-21awk19:38:482
21349943,0cyclictest4991-21awk23:33:482
21349943,0cyclictest4650-21awk20:03:502
21349943,0cyclictest3599-21lldpd22:58:482
21349943,0cyclictest3599-21lldpd20:13:452
21349943,0cyclictest3599-21lldpd20:12:152
21349943,0cyclictest3599-21lldpd20:02:452
21349943,0cyclictest3599-21lldpd18:29:432
21349943,0cyclictest29185-21awk22:13:522
21349943,0cyclictest23871-21snmpd22:40:222
21349943,0cyclictest23871-21snmpd22:28:372
21349943,0cyclictest23871-21snmpd22:18:522
21349943,0cyclictest23871-21snmpd22:08:522
21349943,0cyclictest23871-21snmpd21:13:522
21349943,0cyclictest23871-21snmpd20:59:222
21349943,0cyclictest23871-21snmpd19:35:072
21349943,0cyclictest23871-21snmpd18:37:072
21349943,0cyclictest22572-21awk19:53:482
21349943,0cyclictest22021-21awk23:43:482
21349943,0cyclictest20398-21awk23:23:482
21349943,0cyclictest18305-21awk21:48:482
21349943,0cyclictest15387-21awk19:48:532
21349943,0cyclictest14401-21awk18:58:492
21349943,0cyclictest14011-21awk22:23:482
21349943,0cyclictest13101-21sadc18:58:322
21329943,0cyclictest9801-21awk22:58:521
21329943,0cyclictest741-21awk19:38:481
21329943,0cyclictest7201-21awk18:53:481
21329943,0cyclictest5367-21awk22:18:511
21329943,0cyclictest3599-21lldpd20:58:451
21329943,0cyclictest29135-21awk22:13:511
21329943,0cyclictest28985-21awk23:28:511
21329943,0cyclictest27095-21awk21:53:551
21329943,0cyclictest24705-21awk21:33:491
21329943,0cyclictest16076-21awk21:28:491
21329943,0cyclictest15363-21awk19:48:531
21329943,0cyclictest13965-21awk21:08:501
21329940,0cyclictest6996-21diskmemload21:49:251
21319944,0cyclictest0-21swapper/022:17:350
21319944,0cyclictest0-21swapper/022:00:420
21319944,0cyclictest0-21swapper/021:14:210
21319944,0cyclictest0-21swapper/019:08:330
21319943,0cyclictest3599-21lldpd23:14:180
21319943,0cyclictest3599-21lldpd23:11:480
21319943,0cyclictest3599-21lldpd21:43:470
21319943,0cyclictest3599-21lldpd21:42:470
21319943,0cyclictest23871-21snmpd23:23:370
21319943,0cyclictest23871-21snmpd22:49:370
21319943,0cyclictest21997-21awk23:43:480
21319943,0cyclictest21685-21awk19:03:500
21319943,0cyclictest17914-21awk18:38:480
21319943,0cyclictest13992-21sadc19:48:320
21319943,0cyclictest11988-21awk22:03:520
21319940,0cyclictest0-21swapper/023:05:110
179940,0migration/19003-21kthreadcore22:38:511
179940,0migration/16565-21kthreadcore23:33:511
179940,0migration/16445-21aten_r1power_fr21:23:451
179940,0migration/16107-21kthreadcore18:28:501
179940,0migration/14916-21kthreadcore20:43:521
179940,0migration/14916-21kthreadcore20:43:521
179940,0migration/131832-21kthreadcore19:58:501
179940,0migration/131785-21diskmemload20:39:321
179940,0migration/127110-21kthreadcore20:18:511
179940,0migration/126350-21irqrtprio19:33:491
179940,0migration/125077-21kthreadcore19:53:501
179940,0migration/124552-21fschecks_time22:48:451
179940,0migration/122680-21kthreadcore20:53:531
179940,0migration/121268-21kthreadcore20:13:531
179940,0migration/118897-21irqrtprio23:03:531
179940,0migration/117888-21kthreadcore22:43:501
179940,0migration/114953-21kthreadcore23:38:501
179940,0migration/114429-21kthreadcore19:23:521
179940,0migration/110743-21aten_r1power_cu20:08:431
179940,0migration/110580-21latency_hist20:08:321
169940,0watchdog/17104-21ssh21:40:161
169940,0watchdog/122504-21ssh21:13:481
169940,0watchdog/120855-21ssh23:43:401
169940,0watchdog/116905-21cut20:33:491
169940,0watchdog/115014-21cut18:33:491
169940,0watchdog/10-21swapper/123:08:521
169940,0watchdog/10-21swapper/120:48:521
169940,0watchdog/10-21swapper/118:38:491
149940,0migration/08707-21kthreadcore22:38:510
149940,0migration/08526-21kthreadcore18:53:500
149940,0migration/03228-21kthreadcore18:48:520
149940,0migration/028544-21kthreadcore20:18:510
149940,0migration/018704-21runrttasks21:08:530
149940,0migration/016288-21kthreadcore18:58:520
149940,0migration/015696-21kthreadcore23:18:530
149940,0migration/015232-21kthreadcore19:23:530
149940,0migration/011971-21kthreadcore18:33:480
29935030,0irq/33-eth0-tx-3611-21aten2.4-expect20:03:460
29935030,0irq/33-eth0-tx-0-21swapper/018:43:500
21349933,0cyclictest3599-21lldpd19:46:442
21349933,0cyclictest3599-21lldpd19:10:132
21349932,0cyclictest3599-21lldpd19:28:442
21349932,0cyclictest3599-21lldpd18:48:442
21349932,0cyclictest26204-21awk20:18:492
21349932,0cyclictest17899-21awk18:38:482
21329933,0cyclictest0-21swapper/123:53:581
21329932,0cyclictest4423-21kthreadcore18:48:521
21329932,0cyclictest3257-21awk21:58:541
21329932,0cyclictest14416-21awk18:58:501
21329932,0cyclictest1422-21cut21:18:521
21329932,0cyclictest1046-21awk20:23:501
21329930,0cyclictest18117-21diskstats19:28:461
21319932,0cyclictest6996-21diskmemload20:34:430
21319932,0cyclictest695-21awk19:38:470
21319932,0cyclictest5408-21awk22:18:510
21319932,0cyclictest3599-21lldpd22:13:170
21319932,0cyclictest32564-21unixbench_singl21:54:050
21319932,0cyclictest25884-21ssh20:54:210
21319932,0cyclictest23871-21snmpd21:26:520
21319932,0cyclictest23871-21snmpd19:48:460
21319932,0cyclictest19042-21awk20:13:510
21319930,0cyclictest1692-21diskmemload23:28:590
21319930,0cyclictest0-21swapper/018:28:520
169930,0watchdog/10-21swapper/123:48:521
169930,0watchdog/10-21swapper/123:24:001
169930,0watchdog/10-21swapper/123:18:521
169930,0watchdog/10-21swapper/123:13:481
169930,0watchdog/10-21swapper/122:53:481
169930,0watchdog/10-21swapper/121:45:361
169930,0watchdog/10-21swapper/119:19:011
169930,0watchdog/10-21swapper/119:03:491
149930,0migration/031560-21taskset21:18:510
149930,0migration/017629-21runrttasks20:50:540
149930,0migration/013089-21kthreadcore22:58:540
21319921,0cyclictest24842-21ssh20:38:430
21319920,0cyclictest5442-21tr23:53:480
169920,0watchdog/10-21swapper/119:08:451
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional