You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-01-27 - 14:58

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2.osadl.org (updated Fri Jan 27, 2023 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2659325349,0sleep10-21swapper/106:26:531
2582625251,0sleep00-21swapper/006:22:350
2634623927,0sleep20-21swapper/206:24:012
2634223627,0sleep30-21swapper/306:23:583
266889977,0cyclictest2431-21nvme09:47:320
339960,0migration/38120-21taskset10:31:063
339960,0migration/3539-21gltestperf09:52:313
339960,0migration/331701-21sh08:41:493
339960,0migration/331424-21sh11:36:443
339960,0migration/330874-21sh09:15:163
339960,0migration/330824-21kthreadcore11:52:293
339960,0migration/313022-21sh08:31:063
339960,0migration/311579-21unixbench-2d06:57:413
28005060,0irq/28-eth0-rx-4821-21cut09:17:343
28005060,0irq/28-eth0-rx-392-21perl08:42:283
28005060,0irq/28-eth0-rx-23529-21perl10:57:303
28005060,0irq/28-eth0-rx-14867-21cut10:17:393
28005060,0irq/28-eth0-rx-11027-21munin-node10:12:463
28005060,0irq/28-eth0-rx-11027-21munin-node10:12:463
267049960,0cyclictest0-21swapper/209:55:172
266889965,0cyclictest29805-21ssh10:23:200
266889960,0cyclictest0-21swapper/009:19:470
266889960,0cyclictest0-21swapper/008:30:450
149960,0migration/08820-21sh11:42:180
149960,0migration/026915-21sh10:41:560
149960,0migration/025108-21sh11:48:150
149960,0migration/014814-21sh09:42:440
149960,0migration/014329-21kthreadcore10:32:330
8315050,0irq/26-0000:00:29939-21kthreadcore07:57:412
35150,0ktimersoftd/30-21swapper/309:06:543
339950,0migration/36503-21sh11:02:533
339950,0migration/36125-21kthreadcore10:47:303
339950,0migration/331975-21kthreadcore07:57:423
339950,0migration/331636-21irqcore06:52:363
339950,0migration/331087-21kthreadcore10:07:383
339950,0migration/329536-21kthreadcore07:12:333
339950,0migration/329176-21munin-node06:47:363
339950,0migration/32839-21kthreadcore08:02:313
339950,0migration/32836-21kthreadcore07:37:493
339950,0migration/327362-21kthreadcore10:22:423
339950,0migration/32564-21kthreadcore08:22:363
339950,0migration/322876-21taskset06:46:053
339950,0migration/320624-21if_eth107:07:363
339950,0migration/320342-21kthreadcore10:02:363
339950,0migration/317676-21kthreadcore08:32:433
339950,0migration/316289-21fschecks_count07:27:303
339950,0migration/313928-21cpuspeed_turbos09:07:283
339950,0migration/31278-21diskmemload08:57:353
339950,0migration/312657-21fschecks_count07:47:283
339950,0migration/311226-21kthreadcore07:22:363
339950,0migration/311129-21kthreadcore09:22:353
339950,0migration/310853-21kthreadcore08:47:313
339950,0migration/310660-21kthreadcore10:32:323
339950,0migration/310047-21kthreadcore08:07:333
329950,0watchdog/327357-21ssh11:17:193
329950,0watchdog/324401-21cut08:17:363
329950,0watchdog/320859-21kthreadcore07:52:363
329950,0watchdog/318080-21kthreadcore07:02:363
329950,0watchdog/30-21swapper/311:22:273
28015050,0irq/29-eth0-tx-934-21iostat_ios09:17:322
28015050,0irq/29-eth0-tx-30719-21ssh08:39:482
28015050,0irq/29-eth0-tx-1918-21ssh10:08:062
28015050,0irq/29-eth0-tx-1224-21kthreadcore10:27:382
28015050,0irq/29-eth0-tx-0-21swapper/210:05:382
28015050,0irq/29-eth0-tx-0-21swapper/209:49:492
28015050,0irq/29-eth0-tx-0-21swapper/209:35:362
28015050,0irq/29-eth0-tx-0-21swapper/209:01:132
28005050,0irq/28-eth0-rx-8962-21aten2_r1power_e11:07:223
28005050,0irq/28-eth0-rx-21370-21kthreadcore09:27:333
28005050,0irq/28-eth0-rx-20542-21kthreadcore10:37:303
28005050,0irq/28-eth0-rx-16556-21kthreadcore09:42:473
28005050,0irq/28-eth0-rx-16230-21ssh11:43:583
28005050,0irq/28-eth0-rx-14012-21lldpd06:37:193
28005050,0irq/28-eth0-rx-1129-21munin-node11:37:333
28005050,0irq/28-eth0-rx-0-21swapper/311:48:513
28005050,0irq/28-eth0-rx-0-21swapper/311:30:053
28005050,0irq/28-eth0-rx-0-21swapper/311:14:083
28005050,0irq/28-eth0-rx-0-21swapper/310:52:513
28005050,0irq/28-eth0-rx-0-21swapper/310:43:173
28005050,0irq/28-eth0-rx-0-21swapper/309:57:283
28005050,0irq/28-eth0-rx-0-21swapper/309:48:453
28005050,0irq/28-eth0-rx-0-21swapper/309:37:183
28005050,0irq/28-eth0-rx-0-21swapper/309:32:533
28005050,0irq/28-eth0-rx-0-21swapper/308:53:193
267049955,0cyclictest0-21swapper/211:55:442
267049955,0cyclictest0-21swapper/210:35:102
267049950,0cyclictest0-21swapper/211:41:422
267049950,0cyclictest0-21swapper/211:24:122
267049950,0cyclictest0-21swapper/210:26:212
267049950,0cyclictest0-21swapper/208:57:072
266999950,0cyclictest0-21swapper/111:22:371
266999950,0cyclictest0-21swapper/111:04:391
266999950,0cyclictest0-21swapper/110:28:131
266999950,0cyclictest0-21swapper/109:23:451
266999950,0cyclictest0-21swapper/109:15:521
266889955,0cyclictest0-21swapper/010:17:580
266889950,0cyclictest0-21swapper/011:30:200
266889950,0cyclictest0-21swapper/009:13:100
259950,0migration/223545-21fschecks_time10:57:302
259950,0migration/219217-21munin-node10:37:272
259950,0migration/217163-21sh11:10:482
249950,0watchdog/22283-21perl07:17:282
249950,0watchdog/20-21swapper/206:57:122
19150,0ktimersoftd/10-21swapper/111:48:011
179950,0migration/12678-21kthreadcore07:37:491
179950,0migration/117631-21kthreadcore09:42:471
179950,0migration/11704-21sh11:17:411
179950,0migration/116606-21kthreadcore09:07:341
179950,0migration/115293-21taskset10:32:391
169950,0watchdog/16415-21irqcore06:57:321
169950,0watchdog/123323-21cut08:17:321
169950,0watchdog/122367-21ssh08:36:311
169950,0watchdog/120861-21tail07:52:361
149950,0migration/0780-21if_eth1.310:27:360
149950,0migration/04899-21kthreadcore06:32:430
149950,0migration/04345-21timerandwakeup06:52:450
149950,0migration/04043-21latency_hist11:22:100
149950,0migration/032631-21kthreadcore10:42:330
149950,0migration/032339-21df_abs11:37:260
149950,0migration/02851-21runrttasks11:55:100
149950,0migration/027834-21munin-node10:07:310
149950,0migration/025716-21kthreadcore08:37:350
149950,0migration/02350-21sh08:58:360
149950,0migration/022832-21sh11:32:240
149950,0migration/022066-21switchtime07:27:420
149950,0migration/016561-21kthreadcore09:07:340
149950,0migration/012848-21fschecks_time07:47:280
149950,0migration/011479-21sh10:48:460
149950,0migration/011323-21rt-features06:57:370
149950,0migration/011193-21kthreadcore09:22:350
339940,0migration/330639-21kthreadcore06:27:323
339940,0migration/329908-21taskset07:37:113
329940,0watchdog/39042-21kthreadcore07:42:363
329940,0watchdog/31600-21latency_hist06:32:123
329940,0watchdog/30-21swapper/307:17:323
28015040,0irq/29-eth0-tx-9699-21perl11:42:252
28015040,0irq/29-eth0-tx-9006-21ssh09:22:192
28015040,0irq/29-eth0-tx-8578-21wget08:07:262
28015040,0irq/29-eth0-tx-27385-21aten2_r1power_c10:42:232
28015040,0irq/29-eth0-tx-22711-21ssh11:32:132
28015040,0irq/29-eth0-tx-18159-21munin-node11:12:262
28015040,0irq/29-eth0-tx-13867-21ssh08:32:242
28015040,0irq/29-eth0-tx-12172-21munin-node07:47:302
28015040,0irq/29-eth0-tx-0-21swapper/211:02:192
28015040,0irq/29-eth0-tx-0-21swapper/210:17:122
28015040,0irq/29-eth0-tx-0-21swapper/210:12:342
28015040,0irq/29-eth0-tx-0-21swapper/210:12:342
28015040,0irq/29-eth0-tx-0-21swapper/209:42:182
28015040,0irq/29-eth0-tx-0-21swapper/209:37:132
28015040,0irq/29-eth0-tx-0-21swapper/209:27:162
28015040,0irq/29-eth0-tx-0-21swapper/209:07:282
28015040,0irq/29-eth0-tx-0-21swapper/208:43:172
28015040,0irq/29-eth0-tx-0-21swapper/208:27:142
267049943,0cyclictest6305-21awk06:57:312
267049943,0cyclictest3346-21snmpd11:17:292
267049943,0cyclictest3346-21snmpd09:12:282
267049943,0cyclictest3346-21snmpd09:03:582
267049943,0cyclictest3346-21snmpd07:42:282
267049943,0cyclictest3346-21snmpd07:40:432
267049943,0cyclictest3346-21snmpd07:30:282
267049943,0cyclictest2727-21awk06:32:382
267049943,0cyclictest23834-21awk08:17:342
267049943,0cyclictest20369-21awk07:52:342
267049943,0cyclictest19238-21awk11:47:292
267049943,0cyclictest16690-21awk08:12:312
267049943,0cyclictest14635-21awk10:52:302
267049940,0cyclictest0-21swapper/207:15:402
266999944,0cyclictest0-21swapper/109:31:381
266999943,0cyclictest6009-21awk07:42:331
266999943,0cyclictest3346-21snmpd11:37:291
266999943,0cyclictest3346-21snmpd11:13:431
266999943,0cyclictest3346-21snmpd10:57:281
266999943,0cyclictest20602-21awk07:07:361
266999943,0cyclictest14720-21awk10:17:381
266999943,0cyclictest14023-21lldpd11:43:541
266999943,0cyclictest14023-21lldpd11:32:531
266999943,0cyclictest14023-21lldpd10:48:531
266999943,0cyclictest14023-21lldpd10:44:531
266999943,0cyclictest14023-21lldpd09:58:521
266999943,0cyclictest14023-21lldpd06:47:191
266999943,0cyclictest14023-21lldpd06:27:191
266889944,0cyclictest0-21swapper/011:00:130
266889944,0cyclictest0-21swapper/010:12:360
266889944,0cyclictest0-21swapper/010:12:360
266889943,0cyclictest9867-21awk06:37:340
266889943,0cyclictest5649-21awk09:37:420
266889943,0cyclictest3070-21irqbalance10:02:180
266889943,0cyclictest3070-21irqbalance09:58:280
266889943,0cyclictest3070-21irqbalance09:57:080
266889943,0cyclictest3070-21irqbalance09:27:380
266889943,0cyclictest3070-21irqbalance09:03:080
266889943,0cyclictest3070-21irqbalance08:55:380
266889943,0cyclictest3070-21irqbalance07:07:180
266889943,0cyclictest3070-21irqbalance07:02:290
266889943,0cyclictest3070-21irqbalance06:48:090
266889943,0cyclictest3070-21irqbalance06:42:490
266889943,0cyclictest23795-21awk08:17:340
266889943,0cyclictest19234-21awk11:12:300
266889943,0cyclictest15094-21awk08:32:410
266889943,0cyclictest14023-21lldpd07:15:490
266889940,0cyclictest0-21swapper/007:58:550
259940,0migration/28919-21taskset09:57:182
259940,0migration/24155-21runrttasks10:47:122
259940,0migration/230837-21kthreadcore06:27:322
259940,0migration/223403-21munin-node07:32:302
259940,0migration/215971-21kthreadcore11:27:292
259940,0migration/213632-21kthreadcore07:22:372
259940,0migration/212545-21cpuspeed_turbos07:02:272
249940,0watchdog/283150irq/26-0000:00:06:37:122
249940,0watchdog/223723-21fschecks_count06:47:282
249940,0watchdog/217524-21irqstats06:42:322
249940,0watchdog/210662-21kthreadcore08:47:312
249940,0watchdog/20-21swapper/208:22:362
249940,0watchdog/20-21swapper/208:02:322
20518243,0sleep220650-21awk07:07:362
179940,0migration/19706-21kthreadcore09:37:451
179940,0migration/16666-21kthreadcore08:02:321
179940,0migration/14419-21taskset09:02:181
179940,0migration/132397-21kthreadcore07:12:341
179940,0migration/130072-21taskset09:48:461
179940,0migration/127454-21munin-node10:07:291
179940,0migration/125934-21kthreadcore07:32:341
179940,0migration/125775-21sh10:04:301
179940,0migration/12125-21kthreadcore06:52:381
179940,0migration/121005-21kthreadcore06:42:331
179940,0migration/119984-21kthreadcore08:12:341
179940,0migration/118563-21kthreadcore11:27:301
179940,0migration/11561-21kthreadcore08:42:331
179940,0migration/11474-21aten2_r1power_e07:17:261
179940,0migration/113669-21kthreadcore07:22:371
179940,0migration/112561-21munin-node07:47:281
169940,0watchdog/18185-21ssh09:56:151
169940,0watchdog/15301-21idleruntime10:12:311
169940,0watchdog/15301-21idleruntime10:12:311
169940,0watchdog/130251-21sh10:24:151
169940,0watchdog/129759-21ssh08:37:471
169940,0watchdog/121621-21latency07:27:361
169940,0watchdog/115656-21ssh11:07:431
169940,0watchdog/113420-21ssh10:52:191
169940,0watchdog/1121152sleep108:29:161
169940,0watchdog/10-21swapper/108:47:271
169940,0watchdog/10-21swapper/108:22:321
169940,0watchdog/10-21swapper/106:37:121
169940,0watchdog/10-21swapper/106:32:121
149940,0migration/05374-21df_inode07:42:300
149940,0migration/04957-21kthreadcore11:02:290
149940,0migration/03388-21kthreadcore08:02:310
149940,0migration/029084-21kthreadcore09:32:320
149940,0migration/015556-21kthreadcore10:52:330
28005030,0irq/28-eth0-rx-0-21swapper/308:12:333
266999933,0cyclictest14023-21lldpd07:02:191
266999932,0cyclictest28520-21awk09:32:301
266999932,0cyclictest14023-21lldpd09:17:201
266999932,0cyclictest14023-21lldpd07:57:491
266999930,0cyclictest9271-21tune2fs08:07:281
266889933,0cyclictest20393-21awk07:52:340
266889932,0cyclictest5350-21awk11:22:340
266889932,0cyclictest3346-21snmpd07:35:430
266889932,0cyclictest3346-21snmpd06:29:040
266889932,0cyclictest31013-21awk08:22:340
266889932,0cyclictest16627-21awk08:12:310
266889932,0cyclictest15438-21sensors08:47:360
266889932,0cyclictest14023-21lldpd07:17:190
266889930,0cyclictest1865-21cat08:42:330
266889930,0cyclictest11042-21cut08:07:330
169930,0watchdog/14545-21diskmemload11:53:391
169930,0watchdog/14545-21diskmemload10:37:111
169930,0watchdog/10-21swapper/108:57:311
169930,0watchdog/10-21swapper/108:52:311
149930,0migration/015966-21sh11:08:210
266889920,0cyclictest0-21swapper/007:37:120
266889920,0cyclictest0-21swapper/007:22:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional