You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-01-29 - 04:33

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2.osadl.org (updated Sun Jan 29, 2023 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
191640,0ktimersoftd/120505-21perf18:26:511
2741224535,0sleep20-21swapper/218:24:522
2753423928,0sleep30-21swapper/318:26:213
2755523127,0sleep00-21swapper/018:26:340
276669988,0cyclictest24075-21awk23:32:290
28005070,0irq/28-eth0-rx-28178-21kthreadcore20:17:233
276669974,0cyclictest28353-21munin-run20:57:010
276669974,0cyclictest28353-21munin-run20:57:010
4160,0ktimersoftd/00-21swapper/021:15:100
35160,0ktimersoftd/30-21swapper/321:15:103
339960,0migration/39719-21munin-node23:42:203
339960,0migration/37487-21sh22:27:413
339960,0migration/323774-21sh22:22:083
339960,0migration/321440-21sh22:52:573
339960,0migration/321185-21gltestperf19:52:173
339960,0migration/317056-21sh21:22:583
339960,0migration/311592-21munin-node22:12:273
329960,0watchdog/39652-21perf18:37:013
329960,0watchdog/331401-21df19:37:173
329960,0watchdog/327406-21rm22:57:243
329960,0watchdog/316338-21ntp_kernel_pll_20:47:293
28005060,0irq/28-eth0-rx-25830-21awk23:12:363
28005060,0irq/28-eth0-rx-21656-21ssh22:18:103
28005060,0irq/28-eth0-rx-15974-21iostat_ios21:07:213
276689960,0cyclictest0-21swapper/220:36:232
149960,0migration/06452-21sh22:12:210
149960,0migration/03933-21sh22:09:430
149960,0migration/03349-21sh21:33:250
149960,0migration/032166-21sh23:01:090
149960,0migration/02898-21sh22:43:010
8315050,0irq/26-0000:00:20244-21grep19:07:012
8315050,0irq/26-0000:00:16928-21cat18:42:012
339950,0migration/37508-21kthreadcore21:37:243
339950,0migration/37465-21kernelversion20:27:213
339950,0migration/35661-21fschecks_time23:22:213
339950,0migration/35560-21taskset20:26:383
339950,0migration/34675-21kthreadcore23:37:313
339950,0migration/34489-21kthreadcore18:52:293
339950,0migration/33348-21gltestperf19:17:193
339950,0migration/330018-21irqrtprio22:07:243
339950,0migration/328626-21taskset21:32:183
339950,0migration/326777-21kthreadcore21:47:223
339950,0migration/325041-21gltestperf18:47:193
339950,0migration/32081-21proc_pri23:17:293
339950,0migration/319426-21munin-node21:27:173
339950,0migration/317125-21sh21:58:213
339950,0migration/316481-21kthreadcore23:27:253
339950,0migration/316129-21taskset20:08:203
339950,0migration/315230-21kthreadcore19:47:223
339950,0migration/310372-21fschecks_count18:37:193
329950,0watchdog/36330-21df_inode19:42:173
329950,0watchdog/35605-21diskstats22:47:163
329950,0watchdog/3327-21python23:52:323
329950,0watchdog/318719-21kthreadcore21:42:243
329950,0watchdog/318179-21iostat_ios18:42:213
28005050,0irq/28-eth0-rx-6509-21tr20:42:263
28005050,0irq/28-eth0-rx-6365-21kthreadcore23:02:273
28005050,0irq/28-eth0-rx-6363-21runrttasks20:54:273
28005050,0irq/28-eth0-rx-6363-21runrttasks20:54:273
28005050,0irq/28-eth0-rx-4839-21kthreadcore21:17:223
28005050,0irq/28-eth0-rx-26636-21ssh22:39:143
28005050,0irq/28-eth0-rx-23216-21grep22:02:273
28005050,0irq/28-eth0-rx-22225-21ssh20:33:423
28005050,0irq/28-eth0-rx-17646-21perl19:27:193
28005050,0irq/28-eth0-rx-0-21swapper/323:32:223
28005050,0irq/28-eth0-rx-0-21swapper/323:07:323
28005050,0irq/28-eth0-rx-0-21swapper/322:42:193
28005050,0irq/28-eth0-rx-0-21swapper/322:32:103
28005050,0irq/28-eth0-rx-0-21swapper/321:53:323
28005050,0irq/28-eth0-rx-0-21swapper/320:38:083
276689955,0cyclictest0-21swapper/223:27:272
276689955,0cyclictest0-21swapper/222:39:032
276689955,0cyclictest0-21swapper/221:19:162
276689955,0cyclictest0-21swapper/220:30:242
276689954,0cyclictest3346-21snmpd19:13:332
276689950,0cyclictest0-21swapper/223:14:312
276689950,0cyclictest0-21swapper/222:02:472
276689950,0cyclictest0-21swapper/221:24:382
276689950,0cyclictest0-21swapper/220:49:282
276679955,0cyclictest0-21swapper/123:48:101
276679955,0cyclictest0-21swapper/121:42:591
276679955,0cyclictest0-21swapper/120:42:081
276679954,0cyclictest10634-21awk20:07:211
276679950,0cyclictest0-21swapper/123:16:191
276679950,0cyclictest0-21swapper/121:26:161
27150,0ktimersoftd/20-21swapper/223:32:482
27150,0ktimersoftd/20-21swapper/222:48:022
27150,0ktimersoftd/20-21swapper/221:01:272
249950,0watchdog/29976-21kthreadcore21:37:242
179950,0migration/18985-21timerandwakeup18:32:341
179950,0migration/16713-21munin-node23:02:281
179950,0migration/13414-21idleruntime19:17:201
179950,0migration/122170-21kthreadcore20:52:221
179950,0migration/122170-21kthreadcore20:52:221
179950,0migration/112439-21sh22:48:271
179950,0migration/111568-21sh21:37:271
169950,0watchdog/17809-21ssh21:53:081
169950,0watchdog/129703-21cut23:17:241
169950,0watchdog/124264-21aten2_r1power_v18:47:171
169950,0watchdog/10-21swapper/123:52:241
169950,0watchdog/10-21swapper/122:57:241
149950,0migration/09068-21kthreadcore21:02:260
149950,0migration/07900-21taskset23:41:050
149950,0migration/07417-21timerwakeupswit21:17:300
149950,0migration/06778-21kthreadcore20:02:260
149950,0migration/05957-21aten2_r1power_c19:42:150
149950,0migration/0550-21aten2_r1power_e23:02:200
149950,0migration/05290-21kthreadcore20:42:250
149950,0migration/04493-21missed_timers20:22:230
149950,0migration/03700-21taskset18:32:240
149950,0migration/031931-21fschecks_time19:37:180
149950,0migration/031906-21taskset20:39:480
149950,0migration/03183-21taskset20:57:540
149950,0migration/030268-21kthreadcore18:27:200
149950,0migration/029760-21kthreadcore19:57:240
149950,0migration/028719-21fschecks_count23:17:200
149950,0migration/028106-21sh22:06:560
149950,0migration/027067-21kthreadcore20:17:220
149950,0migration/026757-21taskset21:29:020
149950,0migration/021658-21irqcore19:07:230
149950,0migration/018345-21kthreadcore23:27:250
149950,0migration/018293-21taskset22:00:440
149950,0migration/017018-21taskset20:47:410
149950,0migration/016366-21runrttasks23:44:560
149950,0migration/016296-21kthreadcore22:17:210
149950,0migration/012474-21sh22:48:320
8315040,0irq/26-0000:00:0-21swapper/218:27:282
4140,0ktimersoftd/00-21swapper/021:52:360
339940,0migration/36520-21df_abs18:57:213
339940,0migration/331846-21kthreadcore20:57:223
329940,0watchdog/30-21swapper/320:02:333
329940,0watchdog/30-21swapper/319:22:213
329940,0watchdog/30-21swapper/319:07:213
28015040,0irq/29-eth0-tx-8475-21kthreadcore23:22:262
28015040,0irq/29-eth0-tx-5951-21munin-node19:42:272
28015040,0irq/29-eth0-tx-5604-21diskmemload23:17:402
28015040,0irq/29-eth0-tx-5604-21diskmemload21:49:042
28015040,0irq/29-eth0-tx-5390-21munin-node22:12:192
28015040,0irq/29-eth0-tx-21641-21ssh21:07:492
28015040,0irq/29-eth0-tx-0-21swapper/223:47:092
28015040,0irq/29-eth0-tx-0-21swapper/223:09:302
28015040,0irq/29-eth0-tx-0-21swapper/222:42:232
28015040,0irq/29-eth0-tx-0-21swapper/222:07:522
28015040,0irq/29-eth0-tx-0-21swapper/221:53:022
28015040,0irq/29-eth0-tx-0-21swapper/221:42:062
28015040,0irq/29-eth0-tx-0-21swapper/221:12:142
28015040,0irq/29-eth0-tx-0-21swapper/221:02:572
28005040,0irq/28-eth0-rx-0-21swapper/323:47:083
28005040,0irq/28-eth0-rx-0-21swapper/321:02:223
28005040,0irq/28-eth0-rx-0-21swapper/318:29:533
276689944,0cyclictest0-21swapper/223:03:192
276689944,0cyclictest0-21swapper/222:52:082
276689943,0cyclictest7222-21awk18:57:242
276689943,0cyclictest5607-21sadc20:27:012
276689943,0cyclictest3346-21snmpd20:52:182
276689943,0cyclictest3346-21snmpd20:52:182
276689943,0cyclictest3346-21snmpd19:27:032
276689943,0cyclictest27610-21awk23:52:212
276689943,0cyclictest20366-21awk21:27:212
276689943,0cyclictest1937-21awk20:42:222
276689943,0cyclictest1733-21awk22:27:192
276689943,0cyclictest14023-21lldpd21:32:232
276689943,0cyclictest14023-21lldpd18:43:492
276689943,0cyclictest10492-21awk20:07:202
276689943,0cyclictest10428-21grep19:22:172
276679944,0cyclictest1391-21kworker/1:221:33:081
276679944,0cyclictest0-21swapper/120:27:091
276679944,0cyclictest0-21swapper/120:22:161
276679943,0cyclictest3462-21awk20:02:241
276679943,0cyclictest3346-21snmpd22:44:181
276679943,0cyclictest3070-21irqbalance23:42:261
276679943,0cyclictest3070-21irqbalance22:27:071
276679943,0cyclictest3070-21irqbalance21:18:071
276679943,0cyclictest3070-21irqbalance21:08:171
276679943,0cyclictest3070-21irqbalance19:42:071
276679943,0cyclictest3070-21irqbalance19:37:371
276679943,0cyclictest3070-21irqbalance19:34:181
276679943,0cyclictest28548-21awk19:57:221
276679943,0cyclictest18029-21awk18:42:201
276679943,0cyclictest14023-21lldpd18:55:491
276669943,0cyclictest5985-21awk23:22:230
276669943,0cyclictest3346-21snmpd19:51:330
276669943,0cyclictest3070-21irqbalance23:16:260
276669943,0cyclictest3070-21irqbalance22:22:570
276669943,0cyclictest3070-21irqbalance21:07:070
276669943,0cyclictest3070-21irqbalance20:32:170
276669943,0cyclictest3070-21irqbalance20:27:170
276669943,0cyclictest3070-21irqbalance20:12:170
276669943,0cyclictest3070-21irqbalance19:32:080
276669943,0cyclictest3070-21irqbalance19:29:280
276669943,0cyclictest3070-21irqbalance19:22:580
276669943,0cyclictest3070-21irqbalance19:17:480
276669943,0cyclictest3070-21irqbalance19:12:180
276669943,0cyclictest3070-21irqbalance19:02:380
276669943,0cyclictest3070-21irqbalance18:52:480
276669943,0cyclictest3070-21irqbalance18:42:580
276669943,0cyclictest25270-21awk18:47:210
276669943,0cyclictest14023-21lldpd22:54:540
276669943,0cyclictest14023-21lldpd18:37:190
276669943,0cyclictest10538-21awk20:07:210
259940,0migration/226898-21kthreadcore22:57:242
259940,0migration/22308-21kthreadcore23:37:302
259940,0migration/222501-21munin-node20:12:252
259940,0migration/215894-21mdstat22:32:272
249940,0watchdog/29643-21munin-run18:37:012
249940,0watchdog/232081-21cut19:57:252
249940,0watchdog/224778-21perl20:37:212
249940,0watchdog/20-21swapper/223:42:002
249940,0watchdog/20-21swapper/222:22:202
249940,0watchdog/20-21swapper/221:57:242
249940,0watchdog/20-21swapper/219:52:212
179940,0migration/18704-21kthreadcore21:02:261
179940,0migration/130953-21kthreadcore18:27:201
179940,0migration/129122-21netstat23:32:341
179940,0migration/128676-21kthreadcore22:22:221
179940,0migration/125845-21kthreadcore21:12:241
179940,0migration/125767-21kthreadcore21:47:221
179940,0migration/125225-21latency21:27:241
179940,0migration/123105-21kthreadcore19:07:231
179940,0migration/121137-21kthreadcore19:27:231
179940,0migration/117050-21kthreadcore19:47:231
179940,0migration/115742-21fschecks_time20:32:221
179940,0migration/115398-21kthreadcore19:02:261
179940,0migration/114946-21kthreadcore22:32:261
179940,0migration/113587-21kthreadcore19:22:221
179940,0migration/113228-21kthreadcore20:47:261
179940,0migration/111717-21kthreadcore18:57:261
179940,0migration/110913-21missed_timers23:22:281
179940,0migration/110007-21runrttasks21:57:111
169940,0watchdog/19992-21kthreadcore22:12:241
169940,0watchdog/16363-21runrttasks20:13:291
169940,0watchdog/131886-21cut23:37:001
169940,0watchdog/122081-21kthreadcore19:52:211
169940,0watchdog/119362-21cpuspeed_turbos22:37:201
169940,0watchdog/114476-21munin-node22:52:161
169940,0watchdog/10-21swapper/120:57:171
169940,0watchdog/10-21swapper/120:17:211
149940,0migration/07839-21kthreadcore18:57:250
149940,0migration/028879-21kthreadcore23:52:230
149940,0migration/023763-21kthreadcore22:37:280
149940,0migration/017315-21sh23:09:060
149940,0migration/014093-21kthreadcore22:32:260
339930,0migration/316028-21kthreadcore19:02:273
329930,0watchdog/30-21swapper/319:57:253
329930,0watchdog/30-21swapper/319:32:213
28005030,0irq/28-eth0-rx-0-21swapper/320:16:453
28005030,0irq/28-eth0-rx-0-21swapper/319:14:333
276689932,0cyclictest8284-21awk20:02:272
276689932,0cyclictest31338-21munin-node18:52:202
276689932,0cyclictest24542-21entropy18:47:182
276689932,0cyclictest2391-21munin-node19:17:272
276689932,0cyclictest23852-21munin-node20:17:172
276689932,0cyclictest14168-21awk19:47:202
276689932,0cyclictest0-21swapper/219:37:162
276689930,0cyclictest20830-21df_inode19:07:182
276679932,0cyclictest28652-21awk19:12:191
276679932,0cyclictest20153-21tail22:02:221
276679932,0cyclictest16039-21sensors18:37:281
276679932,0cyclictest14835-21tr22:17:161
276679932,0cyclictest10938-21awk23:07:201
276679932,0cyclictest0-21swapper/122:08:031
276679930,0cyclictest24492-21diskstats20:37:201
276669932,0cyclictest3346-21snmpd22:27:560
276669932,0cyclictest26482-21sensors19:52:260
276669932,0cyclictest19468-21kthreadcore21:42:250
276669930,0cyclictest11668-21ssh21:37:280
259930,0migration/227095-21kthreadcore19:32:262
249930,0watchdog/20-21swapper/222:17:002
169930,0watchdog/10-21swapper/123:27:001
4120,0ktimersoftd/017481-21diskmemload23:47:020
276669921,0cyclictest0-21swapper/021:47:080
276669920,0cyclictest10091-21diskmemload21:22:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional