You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-11-28 - 11:49
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Tue Nov 28, 2023 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
348424637,0sleep30-21swapper/319:03:363
312724030,0sleep20-21swapper/218:59:422
340423624,0sleep00-21swapper/019:02:450
338823526,0sleep10-21swapper/119:02:341
35979965,0cyclictest0-21swapper/219:49:092
35819960,0cyclictest0-21swapper/023:23:510
339960,0migration/34834-21sh22:53:063
339960,0migration/329305-21sh21:51:043
339960,0migration/327048-21sh23:42:563
339960,0migration/321667-21irqcore22:44:053
339960,0migration/317207-21munin-node22:04:093
29925060,0irq/32-eth0-rx-10087-21ssh22:35:013
15211260,0sleep015217-21ssh00:29:550
149960,0migration/026484-21ntp_offset22:44:090
149960,0migration/015870-21kthreadcore23:34:090
35909955,0cyclictest0-21swapper/121:20:001
35909955,0cyclictest0-21swapper/100:18:091
35909950,0cyclictest0-21swapper/123:48:501
35909950,0cyclictest0-21swapper/122:57:261
35909950,0cyclictest0-21swapper/121:28:591
35909950,0cyclictest0-21swapper/121:25:591
35819955,0cyclictest0-21swapper/022:12:350
35819955,0cyclictest0-21swapper/000:09:060
35819950,0cyclictest0-21swapper/021:23:340
35819950,0cyclictest0-21swapper/000:17:180
339950,0migration/39949-21missed_timers19:49:103
339950,0migration/39327-21kthreadcore22:54:063
339950,0migration/36988-21fschecks_time00:09:043
339950,0migration/34000-21munin-node00:04:203
339950,0migration/33294-21kthreadcore00:24:083
339950,0migration/332642-21taskset20:08:213
339950,0migration/332446-21munin-node23:09:043
339950,0migration/329097-21latency_hist19:43:413
339950,0migration/328846-21kthreadcore22:29:093
339950,0migration/328486-21kthreadcore20:49:103
339950,0migration/321083-21taskset22:59:563
339950,0migration/314972-21latency_hist21:03:413
339950,0migration/313586-21kthreadcore22:39:083
329950,0watchdog/36696-21cat20:54:093
329950,0watchdog/34376-21ntp_kernel_pll_21:54:133
329950,0watchdog/323512-21grep19:59:093
329950,0watchdog/321949-21cut20:44:093
329950,0watchdog/321194-21python20:19:133
329950,0watchdog/318375-21ssh22:24:053
329950,0watchdog/310727-21sed23:49:173
329950,0watchdog/30-21swapper/300:19:053
29935050,0irq/33-eth0-tx-6234-21munin-node00:09:132
29935050,0irq/33-eth0-tx-28516-21perl23:44:022
29935050,0irq/33-eth0-tx-20578-21ssh23:57:282
29935050,0irq/33-eth0-tx-19357-21open_inodes21:44:102
29935050,0irq/33-eth0-tx-12930-21missed_timers21:59:092
29935050,0irq/33-eth0-tx-11437-21munin-node20:39:040
29935050,0irq/33-eth0-tx-1082-21ssh21:14:082
29935050,0irq/33-eth0-tx-10235-21ssh23:33:432
29935050,0irq/33-eth0-tx-0-21swapper/020:48:390
29935050,0irq/33-eth0-tx-0-21swapper/019:38:270
29925050,0irq/32-eth0-rx-9287-21ssh22:18:533
29925050,0irq/32-eth0-rx-869-21grep19:24:023
29925050,0irq/32-eth0-rx-24786-21if_eth1.321:09:063
29925050,0irq/32-eth0-rx-2478-21ssh21:34:163
29925050,0irq/32-eth0-rx-23273-21fschecks_count23:04:063
29925050,0irq/32-eth0-rx-21833-21vmstat21:04:183
29925050,0irq/32-eth0-rx-19575-21python21:44:133
29925050,0irq/32-eth0-rx-17664-21sh23:19:083
29925050,0irq/32-eth0-rx-13984-21ntpq23:14:113
29925050,0irq/32-eth0-rx-1058-21kthreadcore19:44:103
29925050,0irq/32-eth0-rx-0-21swapper/323:54:083
29925050,0irq/32-eth0-rx-0-21swapper/323:45:213
29925050,0irq/32-eth0-rx-0-21swapper/323:33:183
29925050,0irq/32-eth0-rx-0-21swapper/322:14:093
29925050,0irq/32-eth0-rx-0-21swapper/322:00:533
29925050,0irq/32-eth0-rx-0-21swapper/321:39:073
29925050,0irq/32-eth0-rx-0-21swapper/321:30:263
29925050,0irq/32-eth0-rx-0-21swapper/321:14:103
29925050,0irq/32-eth0-rx-0-21swapper/300:13:513
27150,0ktimersoftd/20-21swapper/223:04:562
259950,0migration/27946-21df_abs19:29:042
259950,0migration/24838-21sh00:05:402
259950,0migration/23436-21kthreadcore23:29:032
259950,0migration/229694-21df_abs20:29:062
259950,0migration/226409-21latency_hist22:28:402
259950,0migration/220553-21kthreadcore23:19:112
259950,0migration/217301-21kthreadcore21:04:082
259950,0migration/214576-21kthreadcore19:09:072
259950,0migration/214556-21kthreadcore22:39:082
259950,0migration/213919-21kthreadcore00:29:112
19150,0ktimersoftd/10-21swapper/123:04:561
179950,0migration/19045-21if_eth123:14:061
179950,0migration/14804-21kthreadcore19:24:071
179950,0migration/128869-21kthreadcore22:09:111
179950,0migration/122559-21sh23:59:061
179950,0migration/121210-21sh23:00:171
179950,0migration/117497-21gltestperf23:19:061
179950,0migration/116359-21munin-node21:04:061
179950,0migration/113143-21irqcore22:39:071
169950,0watchdog/123674-21aten2.4-expect21:09:011
149950,0migration/08439-21sh23:29:280
149950,0migration/07499-21latency_hist19:28:410
149950,0migration/05829-21kthreadcore20:34:070
149950,0migration/04162-21kthreadcore00:24:080
149950,0migration/032716-21kthreadcore22:49:090
149950,0migration/031334-21fschecks_time21:54:050
149950,0migration/027733-21aten_r1power_fr23:44:000
149950,0migration/027007-21taskset21:31:320
149950,0migration/026141-21kthreadcore00:19:090
149950,0migration/025830-21kthreadcore21:09:070
149950,0migration/025623-21aten2.4-expect19:18:590
149950,0migration/025202-21taskset19:18:110
149950,0migration/023682-21irqrtprio20:24:060
149950,0migration/023227-21sh21:08:380
149950,0migration/020371-21kthreadcore22:04:110
149950,0migration/019006-21kthreadcore21:44:070
149950,0migration/018182-21aten_r1power_vo22:24:040
149950,0migration/017417-21kthreadcore22:39:090
149950,0migration/015604-21irqrtprio22:59:100
149950,0migration/014316-21kernelversion23:54:050
149950,0migration/011444-21aten2.4-expect19:54:020
35979944,0cyclictest0-21swapper/220:08:132
35979943,0cyclictest661-21awk00:24:062
35979943,0cyclictest3536-21snmpd20:56:492
35979943,0cyclictest3536-21snmpd20:35:192
35979943,0cyclictest3536-21snmpd19:35:492
35979943,0cyclictest3536-21snmpd19:19:342
35979943,0cyclictest12366-21awk19:54:062
35909944,0cyclictest0-21swapper/123:42:221
35909943,0cyclictest3536-21snmpd20:29:041
35909943,0cyclictest32630-21awk23:09:061
35909943,0cyclictest3262-21irqbalance23:32:251
35909943,0cyclictest3262-21irqbalance21:53:451
35909943,0cyclictest3262-21irqbalance20:49:051
35909943,0cyclictest3262-21irqbalance20:34:051
35909943,0cyclictest3262-21irqbalance20:23:451
35909943,0cyclictest3262-21irqbalance20:19:151
35909943,0cyclictest3262-21irqbalance19:31:151
35909943,0cyclictest3262-21irqbalance19:19:151
35909943,0cyclictest3262-21irqbalance19:13:451
35909943,0cyclictest3262-21irqbalance19:09:051
35909943,0cyclictest3262-21irqbalance00:28:541
35909943,0cyclictest28753-21awk23:44:041
35909943,0cyclictest26369-21awk23:24:071
35909943,0cyclictest15845-21awk19:34:051
35909943,0cyclictest11443-21awk23:34:051
35819944,0cyclictest0-21swapper/023:21:290
35819944,0cyclictest0-21swapper/020:21:170
35819943,0cyclictest9109-21awk20:59:070
35819943,0cyclictest8146-21awk21:59:040
35819943,0cyclictest3599-21lldpd21:36:500
35819943,0cyclictest3536-21snmpd23:16:500
35819943,0cyclictest3536-21snmpd00:03:210
35819943,0cyclictest2740-21kworker/0:219:52:150
35819943,0cyclictest26761-21awk20:04:040
35819943,0cyclictest23044-21awk21:49:060
35819943,0cyclictest10454-21kworker/0:221:28:190
35819943,0cyclictest0-21swapper/019:08:410
35819940,0cyclictest25827-21munin-node20:49:030
339940,0migration/33647-21taskset20:30:133
339940,0migration/326501-21idleruntime19:19:033
339940,0migration/312626-21kthreadcore20:14:133
339940,0migration/312395-21hddtemp_smartct20:39:053
339940,0migration/310399-21kthreadcore19:29:093
29935040,0irq/33-eth0-tx-9236-21ssh22:18:452
29935040,0irq/33-eth0-tx-31468-21munin-node23:09:082
29935040,0irq/33-eth0-tx-0-21swapper/223:48:572
29935040,0irq/33-eth0-tx-0-21swapper/223:24:162
29935040,0irq/33-eth0-tx-0-21swapper/222:53:572
29935040,0irq/33-eth0-tx-0-21swapper/222:49:042
29935040,0irq/33-eth0-tx-0-21swapper/222:43:462
29935040,0irq/33-eth0-tx-0-21swapper/222:33:402
29935040,0irq/33-eth0-tx-0-21swapper/222:28:562
29935040,0irq/33-eth0-tx-0-21swapper/222:09:132
29935040,0irq/33-eth0-tx-0-21swapper/221:54:022
29935040,0irq/33-eth0-tx-0-21swapper/221:38:562
29935040,0irq/33-eth0-tx-0-21swapper/221:33:472
29935040,0irq/33-eth0-tx-0-21swapper/221:29:332
29935040,0irq/33-eth0-tx-0-21swapper/221:23:422
29935040,0irq/33-eth0-tx-0-21swapper/221:18:542
29935040,0irq/33-eth0-tx-0-21swapper/221:08:462
29935040,0irq/33-eth0-tx-0-21swapper/200:19:392
29935040,0irq/33-eth0-tx-0-21swapper/200:15:402
29935040,0irq/33-eth0-tx-0-21swapper/020:29:060
29935040,0irq/33-eth0-tx-0-21swapper/020:18:350
29935040,0irq/33-eth0-tx-0-21swapper/019:39:130
29935040,0irq/33-eth0-tx-0-21swapper/019:09:020
29925040,0irq/32-eth0-rx-0-21swapper/323:34:143
29925040,0irq/32-eth0-rx-0-21swapper/323:24:063
29925040,0irq/32-eth0-rx-0-21swapper/322:09:333
29925040,0irq/32-eth0-rx-0-21swapper/321:23:433
29925040,0irq/32-eth0-rx-0-21swapper/321:20:263
29925040,0irq/32-eth0-rx-0-21swapper/319:09:113
29925040,0irq/32-eth0-rx-0-21swapper/300:28:433
29925040,0irq/32-eth0-rx-0-21swapper/300:02:173
259940,0migration/28622-21fschecks_time23:14:032
259940,0migration/27150-21runrttasks19:26:172
259940,0migration/25584-21kthreadcore19:04:082
259940,0migration/22807-21kthreadcore22:14:092
259940,0migration/225297-21kthreadcore19:39:092
259940,0migration/223749-21kthreadcore20:44:102
259940,0migration/221826-21runrttasks20:22:292
259940,0migration/220237-21ntp_kernel_err22:59:142
259940,0migration/219066-21aten2.4-expect23:39:022
259940,0migration/218631-21df19:14:012
259940,0migration/21491-21kthreadcore19:44:102
249940,0watchdog/228984-21ssh21:50:132
249940,0watchdog/22720-21kthreadcore20:09:092
249940,0watchdog/224896-21kthreadcore23:59:092
249940,0watchdog/215332-21kthreadcore20:39:092
249940,0watchdog/214004-21mailstats20:14:172
249940,0watchdog/20-21swapper/222:05:212
249940,0watchdog/20-21swapper/220:49:052
249940,0watchdog/20-21swapper/220:24:452
179940,0migration/17746-21irqrtprio00:09:081
179940,0migration/14274-21fschecks_count19:04:031
179940,0migration/13576-21kthreadcore21:14:101
179940,0migration/1335-21kthreadcore21:34:011
179940,0migration/125852-21kthreadcore22:44:071
179940,0migration/123293-21irqcore21:49:061
179940,0migration/12051-21taskset22:32:041
179940,0migration/119120-21fschecks_count20:44:031
179940,0migration/113303-21sensors_volt21:59:131
179940,0migration/112997-21kthreadcore19:54:081
179940,0migration/112974-21irqstats20:39:081
179940,0migration/111092-21kthreadcore22:19:071
169940,0watchdog/130699-21fschecks_time00:04:091
169940,0watchdog/127287-21kthreadcore19:39:091
169940,0watchdog/124303-21fschecks_time00:19:051
169940,0watchdog/115572-21ssh22:03:531
169940,0watchdog/114320-21unixbench_singl20:14:211
169940,0watchdog/11293-21fschecks_time20:54:011
169940,0watchdog/110713-21cut20:59:091
169940,0watchdog/110527-21ssh21:39:091
169940,0watchdog/10-21swapper/122:49:011
169940,0watchdog/10-21swapper/122:34:131
169940,0watchdog/10-21swapper/119:59:091
149940,0migration/05409-21kthreadcore22:14:100
149940,0migration/04173-21aten_r1power_po23:49:010
149940,0migration/030437-21irqcore19:44:090
35979932,0cyclictest3536-21snmpd20:58:412
35979932,0cyclictest19629-21awk19:59:062
35909932,0cyclictest7599-21kthreadcore19:49:091
35909932,0cyclictest26746-21awk20:04:041
35909932,0cyclictest14225-21expr21:44:021
35909932,0cyclictest13792-21awk23:54:031
35909932,0cyclictest1301-21fschecks_time20:09:051
35819933,0cyclictest5515-21kworker/0:122:18:470
35819933,0cyclictest4055-21kworker/0:123:38:470
35819933,0cyclictest4055-21kworker/0:123:08:410
35819933,0cyclictest4055-21kworker/0:123:03:490
35819933,0cyclictest4055-21kworker/0:122:53:410
35819933,0cyclictest4055-21kworker/0:100:03:410
35819933,0cyclictest2740-21kworker/0:219:58:430
35819933,0cyclictest2740-21kworker/0:219:28:410
35819933,0cyclictest26664-21kworker/0:020:08:410
35819933,0cyclictest19039-21kworker/0:022:33:470
35819933,0cyclictest19039-21kworker/0:022:28:410
35819933,0cyclictest10454-21kworker/0:221:38:410
35819933,0cyclictest10454-21kworker/0:221:13:430
35819933,0cyclictest10454-21kworker/0:220:53:410
329930,0watchdog/32721-21kthreadcore20:09:093
329930,0watchdog/30-21swapper/319:54:013
329930,0watchdog/30-21swapper/319:04:093
29925030,0irq/32-eth0-rx-0-21swapper/320:33:443
29925030,0irq/32-eth0-rx-0-21swapper/320:24:003
29925030,0irq/32-eth0-rx-0-21swapper/319:14:013
179930,0migration/117926-21sh22:23:441
169930,0watchdog/13094-21grep22:14:091
169930,0watchdog/10-21swapper/119:43:411
169930,0watchdog/10-21swapper/100:24:091
329920,0watchdog/30-21swapper/319:33:413
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional