You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-09-30 - 18:29
/usr/bin/Xorg /usr/bin/Xorg

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Fri Sep 30, 2022 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2396625141,0sleep30-21swapper/306:39:513
2390823627,0sleep20-21swapper/206:39:102
2374523624,0sleep00-21swapper/006:37:160
2392323525,0sleep10-21swapper/106:39:171
241979977,0cyclictest26691-21awk10:21:562
8315060,0irq/26-0000:00:30844-21sed06:42:123
339960,0migration/319142-21sh09:24:033
339960,0migration/317562-21munin-node09:56:593
339960,0migration/316604-21kthreadcore11:42:013
28005060,0irq/28-eth0-rx-6985-21cut08:42:003
241979960,0cyclictest0-21swapper/211:17:142
241969960,0cyclictest2014-21diskmemload12:01:361
19160,0ktimersoftd/10-21swapper/110:22:071
149960,0migration/09198-21kthreadcore07:36:570
149960,0migration/08417-21missed_timers09:52:020
149960,0migration/03834-21smart_sdb08:57:040
149960,0migration/025292-21sh12:05:510
149960,0migration/025272-21kthreadcore07:02:010
8315050,0irq/26-0000:00:26643-21iostat09:46:572
339950,0migration/39078-21taskset07:57:083
339950,0migration/38397-21gltestperf11:56:543
339950,0migration/35946-21kthreadcore11:01:583
339950,0migration/332662-21kthreadcore08:36:583
339950,0migration/330931-21gltestperf09:31:563
339950,0migration/330415-21fschecks_count08:56:533
339950,0migration/329360-21kthreadcore09:46:593
339950,0migration/328157-21kthreadcore11:31:563
339950,0migration/32610-21fschecks_count09:16:533
339950,0migration/323583-21taskset08:09:293
339950,0migration/322537-21munin-node09:42:043
339950,0migration/316254-21kthreadcore10:31:583
339950,0migration/316180-21kthreadcore08:27:003
339950,0migration/312874-21kthreadcore08:01:583
339950,0migration/311932-21kthreadcore10:11:593
329950,0watchdog/39189-21kthreadcore08:21:593
329950,0watchdog/324501-21df07:26:523
329950,0watchdog/322869-21memory07:22:003
329950,0watchdog/321575-21perl09:26:593
329950,0watchdog/319251-21sh11:08:193
329950,0watchdog/317111-21kthreadcore08:46:593
329950,0watchdog/310856-21ssh09:55:393
329950,0watchdog/30-21swapper/310:51:513
329950,0watchdog/30-21swapper/310:26:393
329950,0watchdog/30-21swapper/310:21:393
329950,0watchdog/30-21swapper/308:16:553
329950,0watchdog/30-21swapper/306:56:563
28015050,0irq/29-eth0-tx-9787-21wget08:01:520
28015050,0irq/29-eth0-tx-8801-21rt-features10:27:042
28015050,0irq/29-eth0-tx-8315-21sh09:17:012
28015050,0irq/29-eth0-tx-29825-21munin-node08:56:522
28015050,0irq/29-eth0-tx-27828-21wget07:51:530
28015050,0irq/29-eth0-tx-27828-21wget07:51:530
28015050,0irq/29-eth0-tx-2545-21munin-node11:01:562
28015050,0irq/29-eth0-tx-23083-21grep11:46:582
28015050,0irq/29-eth0-tx-1501-21ssh12:09:312
28015050,0irq/29-eth0-tx-14499-21ssh10:48:142
28015050,0irq/29-eth0-tx-0-21swapper/211:41:562
28015050,0irq/29-eth0-tx-0-21swapper/210:57:172
28015050,0irq/29-eth0-tx-0-21swapper/210:00:082
28015050,0irq/29-eth0-tx-0-21swapper/209:40:572
28015050,0irq/29-eth0-tx-0-21swapper/209:12:002
28015050,0irq/29-eth0-tx-0-21swapper/007:11:060
28005050,0irq/28-eth0-rx-29527-21kthreadcore12:06:563
28005050,0irq/28-eth0-rx-28266-21cut09:11:593
28005050,0irq/28-eth0-rx-26511-21load08:51:593
28005050,0irq/28-eth0-rx-25415-21seq11:31:193
28005050,0irq/28-eth0-rx-23522-21ssh12:02:153
28005050,0irq/28-eth0-rx-21340-21tune2fs10:01:533
28005050,0irq/28-eth0-rx-16965-21fschecks_count10:16:543
28005050,0irq/28-eth0-rx-0-21swapper/311:55:453
28005050,0irq/28-eth0-rx-0-21swapper/311:36:593
28005050,0irq/28-eth0-rx-0-21swapper/311:11:593
28005050,0irq/28-eth0-rx-0-21swapper/311:00:243
28005050,0irq/28-eth0-rx-0-21swapper/310:48:383
28005050,0irq/28-eth0-rx-0-21swapper/310:42:033
28005050,0irq/28-eth0-rx-0-21swapper/310:36:523
28005050,0irq/28-eth0-rx-0-21swapper/309:36:523
28005050,0irq/28-eth0-rx-0-21swapper/309:06:533
28005050,0irq/28-eth0-rx-0-21swapper/309:02:313
28005050,0irq/28-eth0-rx-0-21swapper/308:31:593
259950,0migration/26750-21latency_hist09:36:392
259950,0migration/232073-21gltestperf08:16:562
259950,0migration/227313-21latency_hist08:36:392
259950,0migration/22580-21latency_hist11:36:392
259950,0migration/224417-21sh10:54:362
259950,0migration/223381-21taskset11:27:092
259950,0migration/221402-21cpuspeed_turbos11:11:532
259950,0migration/220679-21aten2_r1power_p08:51:522
259950,0migration/213479-21munin-node11:22:002
259950,0migration/210632-21kthreadcore08:22:002
241979955,0cyclictest2431-21nvme10:31:582
241979955,0cyclictest0-21swapper/209:29:522
241979954,0cyclictest3346-21snmpd07:30:252
241979954,0cyclictest15872-21kworker/2:209:54:042
241979950,0cyclictest0-21swapper/211:09:492
241979950,0cyclictest0-21swapper/210:43:112
241979950,0cyclictest0-21swapper/209:04:102
241979950,0cyclictest0-21swapper/206:47:192
241969955,0cyclictest0-21swapper/111:23:081
241969955,0cyclictest0-21swapper/110:57:261
241969955,0cyclictest0-21swapper/110:50:021
241969955,0cyclictest0-21swapper/109:51:041
241969955,0cyclictest0-21swapper/108:12:031
241969950,0cyclictest0-21swapper/111:42:221
241969950,0cyclictest0-21swapper/111:19:121
241969950,0cyclictest0-21swapper/109:16:231
241969950,0cyclictest0-21swapper/108:42:201
249950,0watchdog/219085-21kthreadcore12:01:582
249950,0watchdog/216286-21date10:16:392
249950,0watchdog/213686-21cut11:57:022
249950,0watchdog/20-21swapper/209:41:512
179950,0migration/18616-21mdstat11:01:591
179950,0migration/15635-21taskset07:32:541
179950,0migration/15253-21pmu-power06:47:021
179950,0migration/14952-21kthreadcore09:52:001
179950,0migration/13372-21kthreadcore10:42:001
179950,0migration/130114-21aten2_r1power_v09:31:541
179950,0migration/128134-21fschecks_time08:36:531
179950,0migration/124649-21sh12:04:311
179950,0migration/123456-21sh11:27:181
179950,0migration/119587-21sh10:34:171
179950,0migration/110857-21sh09:21:021
169950,0watchdog/12263-21latency_hist10:26:391
169950,0watchdog/120801-21kthreadcore10:16:591
169950,0watchdog/114641-21sed08:26:591
149950,0migration/09879-21sh10:28:270
149950,0migration/09198-21taskset07:58:300
149950,0migration/09164-21sh09:17:370
149950,0migration/08631-21kernelversion10:11:580
149950,0migration/07794-21fschecks_count10:46:520
149950,0migration/07550-21kthreadcore08:21:590
149950,0migration/032440-21kthreadcore10:41:590
149950,0migration/031839-21kthreadcore08:36:580
149950,0migration/029091-21kthreadcore08:12:000
149950,0migration/027082-21munin-node09:27:080
149950,0migration/026674-21hddtemp_smartct10:56:540
149950,0migration/026203-21cpuspeed_turbos12:06:500
149950,0migration/024441-21kthreadcore10:36:570
149950,0migration/022787-21kthreadcore08:31:580
149950,0migration/02240-21sh11:36:170
149950,0migration/021192-21kthreadcore11:26:590
149950,0migration/018334-21kthreadcore06:57:020
149950,0migration/016198-21kthreadcore09:21:580
149950,0migration/015777-21kthreadcore11:06:570
149950,0migration/013975-21kthreadcore09:56:580
149950,0migration/011145-21kthreadcore09:36:590
149950,0migration/010461-21fschecks_count07:16:560
8315040,0irq/26-0000:00:0-21swapper/211:51:392
339940,0migration/35847-21taskset07:35:193
339940,0migration/332599-21kthreadcore07:07:003
339940,0migration/325599-21kthreadcore07:02:013
339940,0migration/323619-21kthreadcore07:46:583
339940,0migration/321317-21diskmemload11:46:473
329940,0watchdog/331872-21kthreadcore07:52:003
329940,0watchdog/331872-21kthreadcore07:52:003
329940,0watchdog/310618-21fschecks_time07:16:563
329940,0watchdog/30-21swapper/311:21:423
329940,0watchdog/30-21swapper/307:41:563
329940,0watchdog/30-21swapper/306:52:043
28015040,0irq/29-eth0-tx-11308-21ssh08:46:402
28015040,0irq/29-eth0-tx-0-21swapper/211:31:482
28015040,0irq/29-eth0-tx-0-21swapper/210:11:472
28015040,0irq/29-eth0-tx-0-21swapper/209:22:052
28015040,0irq/29-eth0-tx-0-21swapper/007:42:060
28015040,0irq/29-eth0-tx-0-21swapper/006:46:520
28015040,0irq/29-eth0-tx-0-21swapper/006:42:010
28005040,0irq/28-eth0-rx-0-21swapper/311:16:563
28005040,0irq/28-eth0-rx-0-21swapper/310:07:013
259940,0migration/230367-21kthreadcore07:51:592
259940,0migration/230367-21kthreadcore07:51:592
259940,0migration/222742-21kthreadcore10:01:572
259940,0migration/221416-21kthreadcore07:21:592
259940,0migration/221330-21fschecks_time07:01:552
259940,0migration/212515-21kthreadcore08:01:582
241979944,0cyclictest0-21swapper/207:16:582
241979944,0cyclictest0-21swapper/207:12:142
241979943,0cyclictest3346-21snmpd07:47:552
241979943,0cyclictest3346-21snmpd06:56:552
241979943,0cyclictest3070-21irqbalance10:06:452
241979943,0cyclictest3070-21irqbalance09:06:462
241979943,0cyclictest22074-21awk10:36:542
241979943,0cyclictest14023-21lldpd08:09:482
241979943,0cyclictest14023-21lldpd08:01:182
241979943,0cyclictest14023-21lldpd07:45:182
241969944,0cyclictest7356-21cut08:21:581
241969944,0cyclictest0-21swapper/110:07:561
241969944,0cyclictest0-21swapper/107:16:431
241969944,0cyclictest0-21swapper/107:15:091
241969943,0cyclictest3346-21snmpd09:04:551
241969943,0cyclictest25380-21irqcore07:26:561
241969943,0cyclictest21880-21awk09:27:011
241969943,0cyclictest14023-21lldpd09:21:471
241969943,0cyclictest14023-21lldpd08:06:471
241969943,0cyclictest14023-21lldpd08:01:481
241969943,0cyclictest14023-21lldpd07:58:181
241969943,0cyclictest14023-21lldpd07:54:181
241969943,0cyclictest14023-21lldpd07:54:181
241969943,0cyclictest14023-21lldpd06:54:181
241969943,0cyclictest13162-21sadc07:41:401
241969943,0cyclictest12563-21awk08:46:571
241959943,0cyclictest3660-21awk11:01:550
241959943,0cyclictest3070-21irqbalance11:58:050
241959943,0cyclictest3070-21irqbalance11:52:250
241959943,0cyclictest3070-21irqbalance11:49:050
241959943,0cyclictest3070-21irqbalance11:42:350
241959943,0cyclictest3070-21irqbalance11:38:150
241959943,0cyclictest3070-21irqbalance11:23:250
241959943,0cyclictest3070-21irqbalance09:31:560
241959943,0cyclictest3070-21irqbalance08:55:160
241959943,0cyclictest3070-21irqbalance07:26:560
241959943,0cyclictest3070-21irqbalance07:21:560
241959943,0cyclictest3070-21irqbalance07:12:060
241959943,0cyclictest3070-21irqbalance06:51:560
241959943,0cyclictest26506-21awk09:46:560
241959943,0cyclictest14023-21lldpd10:34:470
241959943,0cyclictest14023-21lldpd10:21:460
249940,0watchdog/230141-21ls06:42:002
249940,0watchdog/228514-21if_eth008:36:552
249940,0watchdog/224700-21perl08:11:552
249940,0watchdog/22229-21ssh08:41:392
249940,0watchdog/21148-21kthreadcore07:07:002
249940,0watchdog/20-21swapper/208:26:592
179940,0migration/18827-21kthreadcore09:36:581
179940,0migration/13036-21kthreadcore11:51:581
179940,0migration/130073-21cpuspeed_turbos08:56:521
179940,0migration/128678-21fschecks_time07:06:551
179940,0migration/127717-21kthreadcore06:41:591
179940,0migration/126878-21idleruntime11:31:531
179940,0migration/125933-21taskset12:06:421
179940,0migration/125222-21kthreadcore10:01:581
179940,0migration/123827-21kthreadcore11:46:581
179940,0migration/121646-21kthreadcore09:42:001
179940,0migration/121514-21idleruntime07:46:551
179940,0migration/117381-21df_abs07:21:521
179940,0migration/114724-21munin-node06:57:011
169940,0watchdog/124809-21kthreadcore11:11:591
169940,0watchdog/123806-21ssh09:09:151
169940,0watchdog/115169-21kthreadcore09:56:591
169940,0watchdog/111885-21ssh11:06:471
169940,0watchdog/10-21swapper/110:51:511
169940,0watchdog/10-21swapper/110:36:511
169940,0watchdog/10-21swapper/110:11:591
169940,0watchdog/10-21swapper/108:16:551
169940,0watchdog/10-21swapper/107:41:561
149940,0migration/04295-21kthreadcore08:41:590
149940,0migration/031907-21kthreadcore10:06:570
149940,0migration/0314-21irqrtprio07:31:570
149940,0migration/019355-21kthreadcore09:06:570
149940,0migration/016421-21sh10:51:400
149940,0migration/016338-21sh09:41:540
5021230,0sleep00-21swapper/011:17:300
329930,0watchdog/30-21swapper/308:11:593
329930,0watchdog/30-21swapper/307:11:523
329930,0watchdog/30-21swapper/306:48:203
28005030,0irq/28-eth0-rx-0-21swapper/307:36:593
241979933,0cyclictest11005-21kworker/2:007:36:572
241979933,0cyclictest11005-21kworker/2:007:31:512
241979933,0cyclictest11005-21kworker/2:006:51:412
241969933,0cyclictest14023-21lldpd08:51:471
241969933,0cyclictest14023-21lldpd07:01:481
241969932,0cyclictest14023-21lldpd08:33:471
241959933,0cyclictest14023-21lldpd10:16:460
241959932,0cyclictest8899-21cut09:01:570
241959932,0cyclictest21862-21fschecks_time11:11:540
241959932,0cyclictest16804-21cat08:06:400
241959932,0cyclictest14217-21awk08:26:570
241959930,0cyclictest27501-21munin-run07:51:400
241959930,0cyclictest26083-21perl09:11:550
241959930,0cyclictest23304-21grep10:01:580
241959930,0cyclictest0-21swapper/008:16:560
169930,0watchdog/10-21swapper/111:36:581
241959920,0cyclictest0-21swapper/008:46:460
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional