You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-03 - 07:15

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2.osadl.org (updated Sun Jul 03, 2022 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3254823727,0sleep20-21swapper/218:49:502
3251623329,0sleep00-21swapper/018:49:280
3270623228,0sleep30-21swapper/318:51:443
3250623021,0sleep10-21swapper/118:49:211
39799130,0cyclictest0-21swapper/020:52:530
4109970,0cyclictest10461-21chrt23:29:392
4109960,0cyclictest0-21swapper/223:43:222
4109960,0cyclictest0-21swapper/221:44:202
3979963,0cyclictest0-21swapper/020:12:370
3979960,0cyclictest10788-21diskmemload21:57:560
3979960,0cyclictest10788-21diskmemload21:57:560
35160,0ktimersoftd/30-21swapper/320:12:383
339960,0migration/37016-21sh22:34:593
339960,0migration/330415-21kthreadcore21:37:553
339960,0migration/328280-21sh00:17:413
339960,0migration/328265-21sh21:01:483
339960,0migration/324115-21sh23:18:533
339960,0migration/320894-21kthreadcore19:27:583
339960,0migration/313909-21sh21:11:173
28005060,0irq/28-eth0-rx-23145-21cut23:02:533
249960,0watchdog/214909-21latency_hist19:02:372
149960,0migration/031749-21sh21:20:260
149960,0migration/026362-21sh22:47:160
149960,0migration/02356-21unixbench_singl21:38:030
149960,0migration/010351-21sh23:29:240
4109955,0cyclictest0-21swapper/223:22:332
4109955,0cyclictest0-21swapper/221:29:312
4109953,0cyclictest1315-21perl21:22:502
4109950,0cyclictest0-21swapper/221:12:382
4150,0ktimersoftd/00-21swapper/023:03:590
3979955,0cyclictest0-21swapper/023:17:420
3979955,0cyclictest0-21swapper/022:05:110
3979954,0cyclictest25470-21sadc19:32:370
3979952,0cyclictest0-21swapper/019:57:370
3979950,0cyclictest0-21swapper/023:15:110
3979950,0cyclictest0-21swapper/023:00:590
3979950,0cyclictest0-21swapper/022:41:450
339950,0migration/36018-21missed_timers21:57:573
339950,0migration/36018-21missed_timers21:57:573
339950,0migration/35414-21kthreadcore20:47:543
339950,0migration/332140-21kthreadcore19:57:553
339950,0migration/330666-21sh21:54:363
339950,0migration/330307-21swap21:18:023
339950,0migration/325535-21kthreadcore19:07:563
339950,0migration/324070-21munin-node22:43:013
339950,0migration/32363-21kthreadcore21:03:113
339950,0migration/321894-21taskset21:13:463
339950,0migration/321706-21munin-node23:52:543
339950,0migration/320929-21taskset00:12:543
339950,0migration/320317-21taskset22:59:533
339950,0migration/319432-21kthreadcore20:32:563
339950,0migration/318392-21kthreadcore21:47:543
339950,0migration/316986-21kthreadcore22:22:553
339950,0migration/314138-21kthreadcore21:27:553
339950,0migration/313953-21munin-node23:12:563
339950,0migration/312687-21kthreadcore19:42:563
339950,0migration/310853-21sh22:54:103
339950,0migration/310299-21munin-node22:02:523
329950,0watchdog/38831-21ssh00:04:323
329950,0watchdog/37302-21df_inode23:47:483
329950,0watchdog/34750-21grep23:07:563
329950,0watchdog/322002-21munin-run19:07:373
329950,0watchdog/311536-21munin-node20:07:493
329950,0watchdog/30-21swapper/320:37:493
329950,0watchdog/30-21swapper/320:22:533
329950,0watchdog/30-21swapper/300:08:243
28015050,0irq/29-eth0-tx-9620-21diskstats22:02:482
28015050,0irq/29-eth0-tx-5563-21pmu-power22:32:572
28015050,0irq/29-eth0-tx-31672-21kthreadcore00:17:542
28015050,0irq/29-eth0-tx-28117-21munin-node21:38:022
28015050,0irq/29-eth0-tx-25215-21diskstats23:57:492
28015050,0irq/29-eth0-tx-2051-21ssh22:50:032
28015050,0irq/29-eth0-tx-17688-21munin-node22:43:022
28015050,0irq/29-eth0-tx-0-21swapper/223:47:492
28015050,0irq/29-eth0-tx-0-21swapper/223:32:432
28015050,0irq/29-eth0-tx-0-21swapper/223:03:302
28015050,0irq/29-eth0-tx-0-21swapper/222:52:562
28015050,0irq/29-eth0-tx-0-21swapper/222:25:272
28015050,0irq/29-eth0-tx-0-21swapper/222:17:562
28015050,0irq/29-eth0-tx-0-21swapper/222:09:052
28015050,0irq/29-eth0-tx-0-21swapper/221:20:592
28015050,0irq/29-eth0-tx-0-21swapper/020:39:220
28015050,0irq/29-eth0-tx-0-21swapper/019:07:500
28005050,0irq/28-eth0-rx-9229-21kthreadcore20:02:553
28005050,0irq/28-eth0-rx-8094-21tr22:17:543
28005050,0irq/28-eth0-rx-7957-21ssh21:24:053
28005050,0irq/28-eth0-rx-29382-21kthreadcore22:12:533
28005050,0irq/28-eth0-rx-27556-21kthreadcore19:52:563
28005050,0irq/28-eth0-rx-11740-21fschecks_time20:52:513
28005050,0irq/28-eth0-rx-0-21swapper/323:57:533
28005050,0irq/28-eth0-rx-0-21swapper/323:32:543
28005050,0irq/28-eth0-rx-0-21swapper/323:29:023
28005050,0irq/28-eth0-rx-0-21swapper/323:22:553
28005050,0irq/28-eth0-rx-0-21swapper/322:40:133
28005050,0irq/28-eth0-rx-0-21swapper/321:46:303
28005050,0irq/28-eth0-rx-0-21swapper/321:32:553
259950,0migration/28627-21taskset00:04:032
259950,0migration/28294-21fschecks_count23:12:492
259950,0migration/24425-21fschecks_time20:47:502
259950,0migration/228550-21missed_timers22:27:552
259950,0migration/228200-21munin-node19:53:022
259950,0migration/227246-21kthreadcore20:17:542
259950,0migration/224336-21kthreadcore00:12:562
259950,0migration/216061-21kthreadcore22:57:542
259950,0migration/212312-21kthreadcore00:07:542
259950,0migration/211595-21cpuspeed_turbos19:22:502
249950,0watchdog/23980-21turbostat.cron19:17:372
249950,0watchdog/21523-21sed20:22:532
249950,0watchdog/20-21swapper/218:52:372
179950,0migration/126155-21kthreadcore22:27:541
179950,0migration/124687-21unixbench-2d20:13:021
179950,0migration/117643-21kthreadcore19:47:551
179950,0migration/117367-21munin-node20:53:001
179950,0migration/115006-21latency_hist19:02:371
179950,0migration/114446-21taskset19:45:181
179950,0migration/113481-21fschecks_time22:57:491
179950,0migration/112341-21munin-node20:07:531
179950,0migration/11132-21kthreadcore19:12:551
179950,0migration/111113-21kthreadcore22:02:531
179950,0migration/110862-21kthreadcore21:07:561
149950,0migration/08672-21idleruntime19:42:520
149950,0migration/07568-21kthreadcore20:47:550
149950,0migration/06024-21kthreadcore20:02:540
149950,0migration/0552-21kthreadcore23:07:540
149950,0migration/03548-21taskset22:52:420
149950,0migration/032764-21cpuspeed_turbos20:22:480
149950,0migration/032274-21taskset19:33:360
149950,0migration/032105-21ksmtuned20:19:510
149950,0migration/03134-21kthreadcore22:32:540
149950,0migration/029899-21kthreadcore23:22:540
149950,0migration/02964-21kthreadcore18:52:540
149950,0migration/028592-21needreboot22:27:550
149950,0migration/027662-21sh00:16:460
149950,0migration/026815-21sh22:47:460
149950,0migration/026038-21irqcore23:57:530
149950,0migration/02594-21kthreadcore21:22:540
149950,0migration/021221-21kthreadcore20:12:550
149950,0migration/020895-21taskset23:37:190
149950,0migration/017396-21runrttasks21:29:430
149950,0migration/017320-21kthreadcore21:12:540
149950,0migration/017303-21kthreadcore19:47:550
149950,0migration/01658-21kthreadcore23:42:550
149950,0migration/01475-21kthreadcore00:17:550
149950,0migration/014569-21kthreadcore00:07:550
149950,0migration/013879-21taskset23:48:410
149950,0migration/012717-21kthreadcore18:57:550
149950,0migration/012466-21sh22:21:010
4109944,0cyclictest8372-21cut20:27:502
4109943,0cyclictest3346-21snmpd19:34:022
4109943,0cyclictest3346-21snmpd19:30:022
4109943,0cyclictest3346-21snmpd19:08:472
4109943,0cyclictest3070-21irqbalance20:42:432
4109943,0cyclictest23839-21kworker/2:119:19:092
4109943,0cyclictest16098-21awk19:02:532
4109940,0cyclictest0-21swapper/223:07:472
4039943,0cyclictest8036-21awk23:47:511
4039943,0cyclictest4014-21awk23:27:511
4039943,0cyclictest3346-21snmpd23:38:321
4039943,0cyclictest3346-21snmpd21:57:471
4039943,0cyclictest3346-21snmpd21:57:471
4039943,0cyclictest3070-21irqbalance22:37:031
4039943,0cyclictest30190-21awk19:57:521
4039943,0cyclictest27932-21awk22:47:521
4039943,0cyclictest26723-21awk19:32:531
4039943,0cyclictest18713-21awk22:42:511
4039943,0cyclictest15855-21awk21:12:521
4039943,0cyclictest14863-21sadc19:02:371
4039943,0cyclictest1462-21awk19:37:511
4039943,0cyclictest14023-21lldpd23:22:491
4039943,0cyclictest14023-21lldpd23:06:191
4039943,0cyclictest14023-21lldpd22:37:481
4039943,0cyclictest14023-21lldpd22:18:481
4039943,0cyclictest14023-21lldpd21:47:441
4039943,0cyclictest14023-21lldpd21:30:451
4039943,0cyclictest14023-21lldpd21:17:451
4039943,0cyclictest14023-21lldpd20:47:431
4039943,0cyclictest14023-21lldpd20:20:131
3979944,0cyclictest0-21swapper/022:24:440
3979944,0cyclictest0-21swapper/021:09:320
3979944,0cyclictest0-21swapper/021:02:350
3979943,0cyclictest3771-21sadc20:02:370
3979943,0cyclictest3070-21irqbalance21:43:530
3979943,0cyclictest14023-21lldpd23:38:490
3979942,0cyclictest23749-21sh23:55:470
339940,0migration/316259-21irqrtprio19:47:543
329940,0watchdog/38251-21df_abs18:57:493
329940,0watchdog/37783-21df_abs20:27:493
329940,0watchdog/37772-21cat18:57:373
329940,0watchdog/33982-21latency_hist19:17:373
329940,0watchdog/3323142chrt19:34:013
329940,0watchdog/329243-21df20:42:493
28015040,0irq/29-eth0-tx-9238-21kthreadcore21:07:562
28015040,0irq/29-eth0-tx-32356-21ssh21:57:472
28015040,0irq/29-eth0-tx-32356-21ssh21:57:472
28015040,0irq/29-eth0-tx-29065-21ssh22:12:532
28015040,0irq/29-eth0-tx-26291-21cat20:58:012
28015040,0irq/29-eth0-tx-0-21swapper/223:37:382
28015040,0irq/29-eth0-tx-0-21swapper/223:22:442
28015040,0irq/29-eth0-tx-0-21swapper/221:53:162
28015040,0irq/29-eth0-tx-0-21swapper/221:02:532
28015040,0irq/29-eth0-tx-0-21swapper/019:38:010
28015040,0irq/29-eth0-tx-0-21swapper/019:24:080
28015040,0irq/29-eth0-tx-0-21swapper/019:04:050
28005040,0irq/28-eth0-rx-6171-21cut19:17:543
28005040,0irq/28-eth0-rx-0-21swapper/323:42:463
28005040,0irq/28-eth0-rx-0-21swapper/323:37:563
28005040,0irq/28-eth0-rx-0-21swapper/322:47:523
28005040,0irq/28-eth0-rx-0-21swapper/322:30:573
28005040,0irq/28-eth0-rx-0-21swapper/322:07:563
259940,0migration/28673-21kthreadcore20:02:552
259940,0migration/2787-21munin-node19:37:492
259940,0migration/229875-21fschecks_time19:57:502
259940,0migration/222755-21kthreadcore20:12:552
259940,0migration/220303-21irqcore21:32:532
259940,0migration/216051-21kthreadcore21:47:532
259940,0migration/215375-21kthreadcore20:07:552
259940,0migration/212748-21kthreadcore22:37:552
249940,0watchdog/222005-21cat20:37:492
249940,0watchdog/215843-21ssh23:52:442
249940,0watchdog/214667-21aten2_r1power_e20:32:492
249940,0watchdog/211187-21cpu20:52:492
249940,0watchdog/20-21swapper/219:42:492
179940,0migration/17732-21proc_pri00:02:571
179940,0migration/16653-21kthreadcore21:42:541
179940,0migration/1613-21kthreadcore20:42:551
179940,0migration/15892-21kthreadcore21:22:551
179940,0migration/13441-21kthreadcore23:42:561
179940,0migration/131267-21kthreadcore22:12:541
179940,0migration/130183-21sh21:53:341
179940,0migration/129478-21munin-node00:17:531
179940,0migration/128904-21kthreadcore23:57:551
179940,0migration/125602-21sh22:09:061
179940,0migration/121375-21kthreadcore19:27:581
179940,0migration/115260-21kthreadcore19:22:571
179940,0migration/113551-21kthreadcore00:07:541
169940,0watchdog/15499-21irqcore19:17:531
169940,0watchdog/14272-21ssh21:41:571
169940,0watchdog/122499-21df19:07:491
169940,0watchdog/119890-21ssh20:57:411
169940,0watchdog/119021-21ssh23:33:161
169940,0watchdog/115348-21ssh23:14:441
169940,0watchdog/1105622sleep120:03:171
169940,0watchdog/10-21swapper/123:52:441
169940,0watchdog/10-21swapper/123:17:481
169940,0watchdog/10-21swapper/123:07:561
169940,0watchdog/10-21swapper/122:22:521
169940,0watchdog/10-21swapper/121:32:451
169940,0watchdog/10-21swapper/120:27:491
169940,0watchdog/10-21swapper/120:22:531
149940,0migration/05765-21kthreadcore00:02:540
149940,0migration/031412-21kthreadcore19:12:540
149940,0migration/024033-21fschecks_time21:52:510
149940,0migration/010194-21kthreadcore20:27:550
4109933,0cyclictest23839-21kworker/2:119:47:402
4039932,0cyclictest3346-21snmpd19:54:141
4039932,0cyclictest0-21swapper/100:12:491
3979932,0cyclictest4753-21ssh21:06:160
3979932,0cyclictest3070-21irqbalance21:33:130
3979932,0cyclictest28465-21hddtemp_smartct22:12:500
3979932,0cyclictest14639-21munin-node20:32:530
3979931,0cyclictest20938-21kthreadcore22:07:530
329930,0watchdog/30-21swapper/320:17:533
329930,0watchdog/30-21swapper/319:40:333
329930,0watchdog/30-21swapper/319:22:533
28015030,0irq/29-eth0-tx-0-21swapper/019:17:530
179930,0migration/121814-21runrttasks20:37:461
169930,0watchdog/10-21swapper/122:53:081
169930,0watchdog/10-21swapper/121:03:531
169930,0watchdog/10-21swapper/120:32:491
169930,0watchdog/10-21swapper/118:52:371
149930,0migration/028957-21runrttasks20:42:380
149930,0migration/014060-21sh21:47:440
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional