You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-05-17 - 22:54

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack1slot2.osadl.org (updated Tue May 17, 2022 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
597829089,0sleep00-21swapper/006:56:260
592827675,0sleep20-21swapper/206:55:522
586724838,0sleep30-21swapper/306:55:133
6225994646,0cyclictest25620-21md1_resync09:11:471
6224994444,0cyclictest25622-21md0_resync09:11:480
584423725,0sleep10-21swapper/106:54:551
6224991313,0cyclictest3346-21snmpd11:06:050
8315070,0irq/26-0000:00:25531-21users11:43:463
8315070,0irq/26-0000:00:0-21swapper/311:08:053
27170,0ktimersoftd/20-21swapper/209:15:592
19170,0ktimersoftd/10-21swapper/109:15:591
8315060,0irq/26-0000:00:5406-21iostat_ios08:03:353
8315060,0irq/26-0000:00:5239-21sed08:48:173
8315060,0irq/26-0000:00:4856-21perl10:23:313
8315060,0irq/26-0000:00:4068-21sh11:14:043
8315060,0irq/26-0000:00:3601-21yum07:58:453
8315060,0irq/26-0000:00:30285-21hwlatdetect09:43:323
8315060,0irq/26-0000:00:2576-21sh10:20:373
8315060,0irq/26-0000:00:23707-21rm10:49:483
8315060,0irq/26-0000:00:22282-21diskstats10:33:303
8315060,0irq/26-0000:00:1627-21gltestperf11:33:303
8315060,0irq/26-0000:00:14704-21munin-node07:03:333
8315060,0irq/26-0000:00:14649-21awk09:13:393
8315060,0irq/26-0000:00:12701-21grep09:48:533
8315060,0irq/26-0000:00:11959-21grep09:33:313
8315060,0irq/26-0000:00:11862-21df08:08:323
8315060,0irq/26-0000:00:10750-21iostat_ios10:08:353
62269966,0cyclictest0-21swapper/210:16:032
62269960,0cyclictest0-21swapper/208:37:312
62259966,0cyclictest2431-21nvme07:38:351
62259960,0cyclictest21283-21ssh08:59:321
62259960,0cyclictest0-21swapper/109:50:181
62249965,0cyclictest21612-21ssh11:23:440
62249960,0cyclictest0-21swapper/012:18:360
62249960,0cyclictest0-21swapper/007:33:050
35160,0ktimersoftd/30-21swapper/308:36:063
339960,0migration/37928-21sh09:12:483
339960,0migration/3563-21sh09:23:493
339960,0migration/326492-21sh12:21:243
339960,0migration/320322-21sh11:08:223
339960,0migration/316860-21sh11:39:253
339960,0migration/31487-21sh12:23:403
339960,0migration/314502-21sh11:22:303
329960,0watchdog/323933-21aten2_r1power_v09:58:293
28015060,0irq/29-eth0-tx-6442-21munin-node06:58:320
149960,0migration/027466-21kthreadcore12:03:370
149960,0migration/02181-21sh07:13:430
149960,0migration/019691-21hddtemp_smartct11:43:310
8315050,0irq/26-0000:00:5324-21latency_hist08:48:173
8315050,0irq/26-0000:00:29177-21sed07:13:333
8315050,0irq/26-0000:00:26174-21grep10:53:363
8315050,0irq/26-0000:00:2466-21ssh11:49:273
8315050,0irq/26-0000:00:24172-21swap07:28:443
8315050,0irq/26-0000:00:23083-21ssh12:03:253
8315050,0irq/26-0000:00:21947-21hwlatdetect07:08:353
8315050,0irq/26-0000:00:20033-21irqstats10:13:333
8315050,0irq/26-0000:00:17635-21ssh12:17:133
8315050,0irq/26-0000:00:16830-21kthreadcore07:48:363
8315050,0irq/26-0000:00:16594-21kthreadcore09:53:363
8315050,0irq/26-0000:00:14962-21munin-node11:58:333
8315050,0irq/26-0000:00:0-21swapper/310:03:353
8315050,0irq/26-0000:00:0-21swapper/308:18:373
8315050,0irq/26-0000:00:0-21swapper/307:23:373
62269955,0cyclictest27862-1kworker/2:1H11:31:532
62269955,0cyclictest0-21swapper/211:13:462
62269955,0cyclictest0-21swapper/209:53:402
62269950,0cyclictest0-21swapper/212:00:262
62269950,0cyclictest0-21swapper/210:00:462
62259955,0cyclictest0-21swapper/110:27:121
62259955,0cyclictest0-21swapper/110:10:291
62259955,0cyclictest0-21swapper/109:58:211
62259950,0cyclictest0-21swapper/110:41:421
62259950,0cyclictest0-21swapper/110:33:111
62259950,0cyclictest0-21swapper/109:39:441
62249955,0cyclictest2431-21nvme11:10:100
62249955,0cyclictest0-21swapper/010:55:310
62249955,0cyclictest0-21swapper/010:24:280
62249955,0cyclictest0-21swapper/010:21:560
62249955,0cyclictest0-21swapper/010:03:390
62249955,0cyclictest0-21swapper/009:42:310
62249950,0cyclictest0-21swapper/011:48:580
62249950,0cyclictest0-21swapper/011:00:340
62249950,0cyclictest0-21swapper/010:01:080
62249950,0cyclictest0-21swapper/008:38:590
4150,0ktimersoftd/00-21swapper/011:37:460
339950,0migration/3929-21gltestperf07:38:323
339950,0migration/39152-21fschecks_time08:28:323
339950,0migration/36116-21irqrtprio11:53:413
339950,0migration/34693-21kthreadcore10:58:463
339950,0migration/332351-21aten2_r1power_e12:08:273
339950,0migration/329944-21taskset09:04:193
339950,0migration/327531-21kthreadcore08:38:353
339950,0migration/327491-21runrttasks09:39:223
339950,0migration/320587-21kthreadcore08:13:353
339950,0migration/31648-21kthreadcore10:38:373
339950,0migration/311587-21kthreadcore06:58:373
329950,0watchdog/322385-21diskstats07:53:303
329950,0watchdog/30-21swapper/310:43:453
329950,0watchdog/30-21swapper/310:28:333
28015050,0irq/29-eth0-tx-4258-21munin-node08:03:340
28015050,0irq/29-eth0-tx-30352-21munin-node08:43:360
28015050,0irq/29-eth0-tx-2774-21munin-node09:28:392
28015050,0irq/29-eth0-tx-27707-21munin-node12:23:402
28015050,0irq/29-eth0-tx-27337-21ssh09:38:582
28015050,0irq/29-eth0-tx-14488-21ssh08:58:370
28015050,0irq/29-eth0-tx-13007-21munin-node10:28:352
28015050,0irq/29-eth0-tx-11100-21cut10:08:362
28015050,0irq/29-eth0-tx-0-21swapper/211:50:172
28015050,0irq/29-eth0-tx-0-21swapper/211:43:362
28015050,0irq/29-eth0-tx-0-21swapper/211:40:282
28015050,0irq/29-eth0-tx-0-21swapper/210:23:242
28015050,0irq/29-eth0-tx-0-21swapper/210:22:162
28015050,0irq/29-eth0-tx-0-21swapper/210:04:182
28015050,0irq/29-eth0-tx-0-21swapper/008:50:430
28015050,0irq/29-eth0-tx-0-21swapper/008:11:470
28005050,0irq/28-eth0-rx-3165-21diskstats09:28:303
28005050,0irq/28-eth0-rx-21312-21munin-node11:23:383
28005050,0irq/28-eth0-rx-0-21swapper/311:32:423
28005050,0irq/28-eth0-rx-0-21swapper/309:18:403
28005050,0irq/28-eth0-rx-0-21swapper/308:58:373
27150,0ktimersoftd/20-21swapper/208:38:282
259950,0migration/29361-21sh10:59:352
259950,0migration/231652-21kthreadcore09:23:382
259950,0migration/229161-21missed_timers12:03:392
259950,0migration/226043-21gltestperf07:33:332
259950,0migration/223643-21fschecks_time09:03:322
259950,0migration/215068-21fschecks_time08:58:392
259950,0migration/211849-21timerwakeupswit08:48:432
259950,0migration/211385-21kthreadcore10:43:462
259950,0migration/210228-21kthreadcore09:48:402
249950,0watchdog/220012-21kthreadcore11:23:372
249950,0watchdog/20-21swapper/207:18:342
179950,0migration/17079-21switchtime10:03:451
179950,0migration/130373-21kthreadcore07:13:371
179950,0migration/129145-21missed_timers12:03:391
179950,0migration/119903-21irqrtprio10:13:331
179950,0migration/115478-21kthreadcore07:03:351
179950,0migration/113893-21kthreadcore08:28:371
179950,0migration/111372-21kthreadcore11:18:361
169950,0watchdog/129699-21ssh10:36:171
169950,0watchdog/12253-21chrt09:27:501
149950,0migration/09419-21idleruntime08:28:330
149950,0migration/09412-21kthreadcore10:43:450
149950,0migration/06027-21missed_timers12:08:360
149950,0migration/03661-21kthreadcore08:23:350
149950,0migration/032474-21kthreadcore07:58:360
149950,0migration/032020-21kthreadcore11:13:440
149950,0migration/031519-21kthreadcore09:43:350
149950,0migration/031194-21sh11:29:140
149950,0migration/029733-3199-raid-check10:36:230
149950,0migration/028262-21kthreadcore08:18:350
149950,0migration/027538-21latency_hist12:23:160
149950,0migration/026075-21gltestperf07:33:330
149950,0migration/023167-21timerwakeupswit10:48:550
149950,0migration/022191-21kthreadcore08:13:350
149950,0migration/022098-21kthreadcore10:13:340
149950,0migration/021664-21latency_hist10:33:170
149950,0migration/019499-21sh09:36:310
149950,0migration/015164-21kthreadcore07:23:370
149950,0migration/013724-21kthreadcore11:38:370
149950,0migration/011996-21kthreadcore12:14:010
149950,0migration/011892-21kthreadcore11:18:360
149950,0migration/010771-21kthreadcore09:13:370
8315040,0irq/26-0000:00:0-21swapper/307:43:343
8315040,0irq/26-0000:00:0-21swapper/307:43:343
8315040,0irq/26-0000:00:0-21swapper/307:33:313
62269943,0cyclictest3346-21snmpd10:48:272
62269943,0cyclictest3346-21snmpd08:25:262
62269943,0cyclictest31907-21awk10:38:352
62269943,0cyclictest3070-21irqbalance11:18:232
62269943,0cyclictest3070-21irqbalance11:03:542
62269943,0cyclictest3070-21irqbalance10:33:442
62269943,0cyclictest3070-21irqbalance08:18:442
62269943,0cyclictest23030-21awk07:53:322
62269943,0cyclictest18858-21awk07:28:342
62269943,0cyclictest12665-21awk08:08:352
62269943,0cyclictest10277-21kworker/2:007:50:232
62269943,0cyclictest10277-21kworker/2:007:46:312
62269943,0cyclictest10277-21kworker/2:007:46:312
62259944,0cyclictest0-21swapper/111:29:001
62259944,0cyclictest0-21swapper/109:34:131
62259944,0cyclictest0-21swapper/107:30:371
62259943,0cyclictest3346-21snmpd12:17:271
62259943,0cyclictest3346-21snmpd11:38:271
62259943,0cyclictest3346-21snmpd11:33:271
62259943,0cyclictest3346-21snmpd10:18:261
62259943,0cyclictest3346-21snmpd08:54:111
62259943,0cyclictest3070-21irqbalance10:43:341
62259943,0cyclictest23069-21awk07:53:331
62259943,0cyclictest17663-21awk10:48:431
62259943,0cyclictest16324-21awk11:23:351
62259943,0cyclictest14023-21lldpd09:03:491
62259940,0cyclictest0-21swapper/108:49:361
62249944,0cyclictest0-21swapper/009:28:290
62249944,0cyclictest0-21swapper/007:57:120
62249944,0cyclictest0-21swapper/007:10:140
62249943,0cyclictest31524-21sadc09:08:170
62249943,0cyclictest3070-21irqbalance11:58:330
62249943,0cyclictest3070-21irqbalance11:56:430
62249943,0cyclictest27266-21awk09:23:340
62249943,0cyclictest18566-21awk09:18:430
62249943,0cyclictest14836-21awk07:03:340
329940,0watchdog/30-21swapper/307:18:343
28015040,0irq/29-eth0-tx-9191-21ssh12:13:212
28015040,0irq/29-eth0-tx-15826-21cut08:33:290
28015040,0irq/29-eth0-tx-13972-21diskmemload12:08:192
28015040,0irq/29-eth0-tx-0-21swapper/212:18:282
28015040,0irq/29-eth0-tx-0-21swapper/211:54:512
28015040,0irq/29-eth0-tx-0-21swapper/211:33:252
28015040,0irq/29-eth0-tx-0-21swapper/211:08:442
28015040,0irq/29-eth0-tx-0-21swapper/210:53:352
28015040,0irq/29-eth0-tx-0-21swapper/209:43:212
28015040,0irq/29-eth0-tx-0-21swapper/209:33:262
28015040,0irq/29-eth0-tx-0-21swapper/209:18:452
28015040,0irq/29-eth0-tx-0-21swapper/209:10:392
28015040,0irq/29-eth0-tx-0-21swapper/007:48:400
28015040,0irq/29-eth0-tx-0-21swapper/007:43:400
28015040,0irq/29-eth0-tx-0-21swapper/007:43:400
28005040,0irq/28-eth0-rx-0-21swapper/308:23:363
259940,0migration/232442-21runrttasks07:38:262
259940,0migration/223629-21kthreadcore08:13:362
259940,0migration/215302-21taskset08:31:032
249940,0watchdog/27567-21if_eth1.306:58:342
249940,0watchdog/20-21swapper/207:13:382
179940,0migration/1771-21gltestperf12:08:301
179940,0migration/13494-21irqcore10:58:451
179940,0migration/131512-21kthreadcore08:18:361
179940,0migration/131200-21kthreadcore10:53:421
179940,0migration/130674-21irqrtprio07:58:351
179940,0migration/129836-21fschecks_count09:43:301
179940,0migration/127852-21taskset07:11:061
179940,0migration/126329-21kthreadcore08:38:351
179940,0migration/124852-21missed_timers11:43:381
179940,0migration/119927-21taskset11:07:591
179940,0migration/119816-21kthreadcore09:53:371
179940,0migration/119758-21kthreadcore11:58:361
179940,0migration/118425-21runrttasks12:18:281
179940,0migration/116497-21kthreadcore07:48:351
179940,0migration/115620-21kthreadcore08:08:381
179940,0migration/110574-21aten2_r1power_c07:23:301
169940,0watchdog/15295-21if_eth008:03:341
169940,0watchdog/14185-21hwlatdetect07:18:341
169940,0watchdog/131403-21if_eth108:43:341
169940,0watchdog/129143-21sed12:23:371
169940,0watchdog/127291-21ssh11:09:011
169940,0watchdog/12514-21cut11:13:451
169940,0watchdog/1188722sleep108:13:221
169940,0watchdog/117385-21ssh09:18:301
169940,0watchdog/112025-21ssh11:55:051
169940,0watchdog/11147-21kthreadcore11:48:371
169940,0watchdog/110021-21ssh09:29:491
169940,0watchdog/10-21swapper/106:58:341
149940,0migration/05293-21kthreadcore07:38:360
62269933,0cyclictest10277-21kworker/2:008:53:232
62269933,0cyclictest10277-21kworker/2:008:43:272
62269933,0cyclictest10277-21kworker/2:008:03:192
62269933,0cyclictest10277-21kworker/2:007:58:212
62269933,0cyclictest10277-21kworker/2:007:08:292
62269933,0cyclictest10277-21kworker/2:007:03:332
62259932,0cyclictest7445-21sensors08:23:401
62259932,0cyclictest7404-21date07:43:191
62259932,0cyclictest7404-21date07:43:191
62259932,0cyclictest26154-21awk07:33:341
62259932,0cyclictest22030-21sensors08:33:401
62249933,0cyclictest31882-21awk10:38:350
62249932,0cyclictest7050-21awk09:48:370
62249932,0cyclictest3346-21snmpd10:09:110
62249932,0cyclictest3346-21snmpd08:55:430
62249932,0cyclictest3346-21snmpd07:21:000
62249932,0cyclictest3070-21irqbalance09:53:340
329930,0watchdog/30-21swapper/308:53:183
259930,0migration/214974-21kthreadcore07:23:372
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional