You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-06-27 - 04:53

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Mon Jun 27, 2022 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1128023930,0sleep20-21swapper/218:53:092
1123823925,0sleep00-21swapper/018:52:430
441223526,0sleep10-21swapper/118:48:311
1104923329,0sleep30-21swapper/318:50:303
35160,0ktimersoftd/30-21swapper/321:16:573
339960,0migration/38453-21sh21:53:523
339960,0migration/33706-21sh22:09:183
339960,0migration/330909-21sh22:24:013
339960,0migration/318108-21sh22:36:493
149960,0migration/06652-21cpuspeed_turbos00:18:310
149960,0migration/021640-21sh23:32:030
113679966,0cyclictest0-21swapper/223:14:412
113679966,0cyclictest0-21swapper/221:05:432
113679966,0cyclictest0-21swapper/221:05:432
339950,0migration/39672-21kthreadcore20:43:393
339950,0migration/39220-21kthreadcore21:38:383
339950,0migration/38847-21kthreadcore23:23:393
339950,0migration/332316-21munin-node23:53:393
339950,0migration/328604-21taskset21:31:263
339950,0migration/327880-21munin-node20:08:423
339950,0migration/327252-21kthreadcore20:33:393
339950,0migration/322950-21timerandwakeup21:08:523
339950,0migration/321630-21kthreadcore23:13:383
339950,0migration/318055-21kthreadcore21:43:453
339950,0migration/314511-21taskset19:15:363
339950,0migration/312795-21kthreadcore20:23:373
339950,0migration/311673-21aten2_r1power_v18:53:333
339950,0migration/310571-21kthreadcore22:48:393
329950,0watchdog/37585-21sh23:42:123
329950,0watchdog/332756-21df20:38:333
329950,0watchdog/330987-21df_inode20:58:333
329950,0watchdog/329293-21if_eth022:43:363
329950,0watchdog/328252-21processes19:23:413
329950,0watchdog/326946-21if_eth1.219:03:373
329950,0watchdog/325989-21diskstats19:48:333
329950,0watchdog/323239-21irqstats20:53:373
329950,0watchdog/318777-21aten2_r1power_p18:58:333
329950,0watchdog/30-21swapper/323:43:443
329950,0watchdog/30-21swapper/323:23:203
28015050,0irq/29-eth0-tx-31636-21yum21:48:552
28015050,0irq/29-eth0-tx-25623-21munin-node19:48:430
28015050,0irq/29-eth0-tx-25025-21ssh23:50:592
28015050,0irq/29-eth0-tx-0-21swapper/020:51:330
28005050,0irq/28-eth0-rx-7608-21if_eth1.300:18:353
28005050,0irq/28-eth0-rx-4833-21sh21:35:133
28005050,0irq/28-eth0-rx-26049-21kthreadcore23:33:383
28005050,0irq/28-eth0-rx-14395-21kthreadcore21:58:523
28005050,0irq/28-eth0-rx-11825-21grep19:38:343
28005050,0irq/28-eth0-rx-10335-21if_err_eth1.323:08:423
28005050,0irq/28-eth0-rx-0-21swapper/323:58:373
28005050,0irq/28-eth0-rx-0-21swapper/323:48:343
28005050,0irq/28-eth0-rx-0-21swapper/323:03:453
28005050,0irq/28-eth0-rx-0-21swapper/322:53:383
28005050,0irq/28-eth0-rx-0-21swapper/322:28:543
28005050,0irq/28-eth0-rx-0-21swapper/322:20:013
28005050,0irq/28-eth0-rx-0-21swapper/322:03:373
28005050,0irq/28-eth0-rx-0-21swapper/321:48:463
28005050,0irq/28-eth0-rx-0-21swapper/321:23:393
28005050,0irq/28-eth0-rx-0-21swapper/321:18:343
28005050,0irq/28-eth0-rx-0-21swapper/321:03:383
28005050,0irq/28-eth0-rx-0-21swapper/321:03:383
28005050,0irq/28-eth0-rx-0-21swapper/319:08:393
28005050,0irq/28-eth0-rx-0-21swapper/300:08:383
28005050,0irq/28-eth0-rx-0-21swapper/300:04:273
27150,0ktimersoftd/20-21swapper/200:12:232
249950,0watchdog/222248-21crond23:33:202
179950,0migration/19567-21sh21:56:171
179950,0migration/17165-21taskset23:04:321
179950,0migration/16187-21idleruntime-cro00:18:201
179950,0migration/131948-21kthreadcore22:08:391
179950,0migration/126448-21kthreadcore22:23:391
179950,0migration/121077-21taskset20:49:001
179950,0migration/110212-21cpuspeed_turbos22:33:321
169950,0watchdog/17243-21ssh23:41:281
169950,0watchdog/125680-21df_abs20:33:331
169950,0watchdog/119978-21ssh21:27:291
169950,0watchdog/119606-21grep22:38:321
169950,0watchdog/119280-21kthreadcore21:08:451
169950,0watchdog/114888-21ssh21:05:331
169950,0watchdog/114888-21ssh21:05:331
149950,0migration/09397-21sh21:19:230
149950,0migration/07208-21fschecks_count21:38:340
149950,0migration/06254-21missed_timers19:53:400
149950,0migration/06076-21kthreadcore22:28:390
149950,0migration/04989-21sh22:48:280
149950,0migration/04343-21fschecks_count20:18:340
149950,0migration/04260-21kthreadcore19:08:390
149950,0migration/03931-21sh23:23:160
149950,0migration/031983-21taskset20:35:180
149950,0migration/030868-21kthreadcore22:08:390
149950,0migration/030006-21kthreadcore22:43:380
149950,0migration/028858-21kthreadcore22:58:390
149950,0migration/026966-21munin-node21:28:430
149950,0migration/021839-21hddtemp_smartct00:08:340
149950,0migration/019342-21gltestperf19:43:340
149950,0migration/019116-21aten2_r1power_c22:38:300
149950,0migration/018277-21sh00:03:440
149950,0migration/016757-21kthreadcore19:18:380
149950,0migration/012879-21kthreadcore20:23:370
149950,0migration/012405-21kthreadcore21:03:380
149950,0migration/012405-21kthreadcore21:03:380
149950,0migration/012218-21sh19:38:350
149950,0migration/011955-21if_err_eth021:58:480
113679955,0cyclictest14747-21kworker/2:122:50:142
113679955,0cyclictest0-21swapper/223:18:562
113679955,0cyclictest0-21swapper/222:42:402
113679955,0cyclictest0-21swapper/220:23:282
113679954,0cyclictest3346-21snmpd00:05:132
113679950,0cyclictest0-21swapper/222:56:332
113679950,0cyclictest0-21swapper/221:19:352
113669955,0cyclictest21459-21diskmemload23:25:451
113669955,0cyclictest0-21swapper/123:58:421
113669950,0cyclictest0-21swapper/123:29:321
113659954,0cyclictest5237-21awk19:33:370
8315040,0irq/26-0000:00:4158-21latency_hist23:23:202
339940,0migration/34761-21kthreadcore19:53:383
329940,0watchdog/30-21swapper/322:38:363
329940,0watchdog/30-21swapper/319:43:333
329940,0watchdog/30-21swapper/300:13:323
28015040,0irq/29-eth0-tx-27525-21ssh23:53:362
28015040,0irq/29-eth0-tx-23276-21kthreadcore20:53:382
28015040,0irq/29-eth0-tx-21459-21diskmemload21:58:402
28015040,0irq/29-eth0-tx-16708-21cat21:08:392
28015040,0irq/29-eth0-tx-11348-21grep21:38:392
28015040,0irq/29-eth0-tx-0-21swapper/223:38:382
28015040,0irq/29-eth0-tx-0-21swapper/223:33:412
28015040,0irq/29-eth0-tx-0-21swapper/222:34:052
28015040,0irq/29-eth0-tx-0-21swapper/222:05:542
28015040,0irq/29-eth0-tx-0-21swapper/221:53:242
28015040,0irq/29-eth0-tx-0-21swapper/221:44:002
28015040,0irq/29-eth0-tx-0-21swapper/221:33:442
28015040,0irq/29-eth0-tx-0-21swapper/200:14:572
28015040,0irq/29-eth0-tx-0-21swapper/019:13:340
28015040,0irq/29-eth0-tx-0-21swapper/019:03:330
28005040,0irq/28-eth0-rx-0-21swapper/323:28:323
28005040,0irq/28-eth0-rx-0-21swapper/322:58:473
28005040,0irq/28-eth0-rx-0-21swapper/322:13:463
28005040,0irq/28-eth0-rx-0-21swapper/319:33:383
259940,0migration/29912-21sh23:59:162
259940,0migration/27777-21kthreadcore22:13:382
259940,0migration/227111-21timerandwakeup21:28:462
259940,0migration/226944-21irqcore19:48:372
259940,0migration/224822-21kthreadcore19:23:372
259940,0migration/220569-21kthreadcore20:28:422
259940,0migration/215943-21runrttasks22:18:352
259940,0migration/21592-21kthreadcore22:08:402
249940,0watchdog/27861-21df19:58:332
249940,0watchdog/228448-21tr22:43:322
249940,0watchdog/225681-21df_abs20:33:332
249940,0watchdog/215359-21diskstats19:18:332
249940,0watchdog/20-21swapper/221:23:412
249940,0watchdog/20-21swapper/220:03:532
179940,0migration/19381-21kthreadcore21:38:381
179940,0migration/18187-21kthreadcore20:18:391
179940,0migration/16794-21sh21:02:431
179940,0migration/129768-21fschecks_count19:28:341
179940,0migration/127929-21sh22:06:291
179940,0migration/127526-21munin-node23:53:361
179940,0migration/125243-21kthreadcore22:58:381
179940,0migration/124480-21latency18:58:411
179940,0migration/122376-21runrttasks22:20:151
179940,0migration/114216-21munin-node22:53:321
179940,0migration/113570-21kthreadcore23:08:451
179940,0migration/111918-21sh00:03:241
179940,0migration/110407-21kthreadcore19:58:381
169940,0watchdog/18857-21munin-node19:13:371
169940,0watchdog/18088-21rm22:29:521
169940,0watchdog/17281-21cut21:18:411
169940,0watchdog/16814-21df_abs00:18:321
169940,0watchdog/16051-21cpu22:13:321
169940,0watchdog/128462-21sh20:54:011
169940,0watchdog/127636-21ssh00:08:521
169940,0watchdog/121028-21timerandwakeup20:03:531
169940,0watchdog/115161-21ssh23:43:481
169940,0watchdog/10-21swapper/123:18:201
169940,0watchdog/10-21swapper/120:08:331
169940,0watchdog/10-21swapper/119:23:331
149940,0migration/06730-21kthreadcore23:58:370
149940,0migration/022670-21cpu23:33:320
149940,0migration/022170-21kthreadcore21:08:460
149940,0migration/021341-21sh22:54:480
113679944,0cyclictest0-21swapper/219:01:002
113679943,0cyclictest8828-21awk19:13:372
113679943,0cyclictest3346-21snmpd23:03:432
113679943,0cyclictest3346-21snmpd22:24:282
113679943,0cyclictest3346-21snmpd20:59:432
113679943,0cyclictest3346-21snmpd20:49:132
113679943,0cyclictest3346-21snmpd20:20:132
113679943,0cyclictest3346-21snmpd19:09:432
113679943,0cyclictest3346-21snmpd18:56:432
113679943,0cyclictest30431-21awk19:28:382
113679943,0cyclictest26930-21awk19:03:372
113679943,0cyclictest26605-21awk21:13:362
113679943,0cyclictest1831-21awk22:28:362
113679943,0cyclictest1416-21awk19:53:362
113679943,0cyclictest14023-21lldpd22:58:212
113679943,0cyclictest14023-21lldpd00:18:222
113679943,0cyclictest12380-21awk19:38:362
113679940,0cyclictest9286-21munin-node23:08:402
113679940,0cyclictest8435-21munin-node23:43:342
113669943,0cyclictest6079-21awk22:48:361
113669943,0cyclictest3346-21snmpd21:58:281
113669943,0cyclictest3070-21irqbalance22:43:271
113669943,0cyclictest3070-21irqbalance21:48:371
113669943,0cyclictest3070-21irqbalance21:28:271
113669943,0cyclictest3070-21irqbalance19:48:581
113669943,0cyclictest23678-21awk23:33:361
113669943,0cyclictest19525-21awk19:43:361
113669943,0cyclictest18430-21awk23:48:361
113669943,0cyclictest16952-21awk21:43:431
113659943,0cyclictest791-21grep23:03:330
113659943,0cyclictest32755-21awk23:38:360
113659943,0cyclictest3070-21irqbalance23:48:370
113659943,0cyclictest3070-21irqbalance23:43:570
113659943,0cyclictest3070-21irqbalance23:13:570
113659943,0cyclictest3070-21irqbalance23:10:170
113659943,0cyclictest3070-21irqbalance22:33:270
113659943,0cyclictest3070-21irqbalance21:43:570
113659943,0cyclictest3070-21irqbalance21:33:570
113659943,0cyclictest3070-21irqbalance21:24:270
113659943,0cyclictest3070-21irqbalance21:17:570
113659943,0cyclictest3070-21irqbalance20:53:270
113659943,0cyclictest3070-21irqbalance20:44:470
113659943,0cyclictest3070-21irqbalance20:38:280
113659943,0cyclictest3070-21irqbalance20:32:580
113659943,0cyclictest3070-21irqbalance19:28:580
113659943,0cyclictest3070-21irqbalance19:23:380
113659943,0cyclictest3070-21irqbalance00:13:360
113659943,0cyclictest27474-21awk23:53:350
113659943,0cyclictest25624-21sadc19:03:220
113659943,0cyclictest21124-21awk22:03:360
113659943,0cyclictest16076-21awk22:18:360
113659943,0cyclictest12677-21awk18:53:370
113659940,0cyclictest13133-21idleruntime-cro23:28:200
113659940,0cyclictest0-21swapper/022:15:150
329930,0watchdog/30-21swapper/320:13:373
329930,0watchdog/30-21swapper/319:58:333
329930,0watchdog/30-21swapper/319:18:453
28005030,0irq/28-eth0-rx-0-21swapper/320:28:343
28005030,0irq/28-eth0-rx-0-21swapper/320:18:343
28005030,0irq/28-eth0-rx-0-21swapper/320:03:373
28005030,0irq/28-eth0-rx-0-21swapper/319:28:343
249930,0watchdog/20-21swapper/220:13:412
249930,0watchdog/20-21swapper/220:08:332
179930,0migration/14862-21fschecks_time19:33:341
179930,0migration/111187-21kthreadcore20:43:391
169930,0watchdog/10-21swapper/120:38:331
169930,0watchdog/10-21swapper/120:28:371
169930,0watchdog/10-21swapper/120:16:211
169930,0watchdog/10-21swapper/119:18:331
169930,0watchdog/10-21swapper/119:03:371
113679932,0cyclictest7520-21kthreadcore19:33:382
113679932,0cyclictest18450-21munin-node19:43:322
113679932,0cyclictest1095-21awk20:38:362
113679930,0cyclictest0-21swapper/220:43:442
113669933,0cyclictest0-21swapper/121:33:301
113669932,0cyclictest3346-21snmpd20:23:241
113669932,0cyclictest3070-21irqbalance19:53:281
113669932,0cyclictest25602-21munin-node21:13:321
113669932,0cyclictest19488-21awk23:13:361
113669932,0cyclictest12317-21awk19:38:361
113659932,0cyclictest6739-21sh21:02:360
113659932,0cyclictest3346-21snmpd20:00:580
113659932,0cyclictest3070-21irqbalance22:23:270
113659932,0cyclictest22931-21awk20:08:360
113659932,0cyclictest18498-21grep20:03:440
113659930,0cyclictest0-21swapper/020:13:340
329920,0watchdog/30-21swapper/320:48:293
113669921,0cyclictest11748-21cpuspeed_turbos18:53:331
113669920,0cyclictest451-21latency_hist19:08:221
113659921,0cyclictest1484-21latency_hist21:53:200
113659920,0cyclictest0-21swapper/021:48:360
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional