You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-02-04 - 23:14

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Sat Feb 04, 2023 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2486726564,0sleep00-21swapper/006:22:370
2507424635,0sleep30-21swapper/306:25:043
2483723827,0sleep10-21swapper/106:22:181
2513223026,0sleep20-21swapper/206:25:432
25246991111,0cyclictest0-21swapper/207:45:312
25246991010,0cyclictest0-21swapper/206:56:212
25246991010,0cyclictest0-21swapper/206:43:332
252469999,0cyclictest0-21swapper/206:35:332
252449999,0cyclictest0-21swapper/006:56:220
252449999,0cyclictest0-21swapper/006:55:320
252449990,0cyclictest0-21swapper/006:33:320
252449990,0cyclictest0-21swapper/006:27:320
252449988,0cyclictest0-21swapper/006:41:220
252459970,0cyclictest0-21swapper/110:48:441
8315060,0irq/26-0000:00:28092-21date09:51:142
339960,0migration/35266-21sh11:22:323
339960,0migration/32854-21sh09:52:183
339960,0migration/32531-21sh10:28:303
339960,0migration/32531-21sh10:28:303
339960,0migration/321535-21sh10:40:023
339960,0migration/319008-21unixbench-2d08:31:413
339960,0migration/318082-21sh09:44:283
28005060,0irq/28-eth0-rx-29440-21perl08:21:273
28005060,0irq/28-eth0-rx-24627-21unixbench_singl07:06:413
28005060,0irq/28-eth0-rx-22640-21cut10:56:323
28005060,0irq/28-eth0-rx-14968-21yum10:51:433
252469966,0cyclictest26439-21awk06:26:302
252469966,0cyclictest0-21swapper/209:24:372
252449966,0cyclictest0-21swapper/007:45:320
149960,0migration/018092-21latency_hist10:21:140
8315050,0irq/26-0000:00:25191-21crond11:36:132
4150,0ktimersoftd/00-21swapper/009:56:520
4150,0ktimersoftd/00-21swapper/008:48:460
339950,0migration/39838-21smart_sda07:41:493
339950,0migration/36295-21kthreadcore11:06:313
339950,0migration/34121-21kthreadcore11:41:323
339950,0migration/332727-21munin-node06:31:283
339950,0migration/332308-21sh09:34:453
339950,0migration/329584-21kthreadcore07:11:343
339950,0migration/325753-21unixbench-2d09:46:393
339950,0migration/322688-21kthreadcore11:51:333
339950,0migration/322298-21ntp_offset08:51:483
339950,0migration/322013-21df_abs07:31:333
339950,0migration/317762-21switchtime11:46:393
339950,0migration/316128-21idleruntime09:26:293
339950,0migration/312274-21kthreadcore06:36:423
339950,0migration/310874-21kthreadcore08:46:333
339950,0migration/310635-21kthreadcore08:06:333
339950,0migration/310611-21taskset06:59:353
329950,0watchdog/36265-21pmu-power10:11:373
329950,0watchdog/34132-21ssh10:31:213
329950,0watchdog/33831-21cut09:01:413
329950,0watchdog/328322-21uptime06:46:423
329950,0watchdog/327666-21uname08:36:383
329950,0watchdog/322434-21munin-node10:41:333
329950,0watchdog/318904-21gltestperf07:51:303
329950,0watchdog/317778-21sh07:51:143
329950,0watchdog/30-21swapper/309:11:333
28015050,0irq/29-eth0-tx-8139-21cut10:31:332
28015050,0irq/29-eth0-tx-3540-21ssh08:26:242
28015050,0irq/29-eth0-tx-24596-21ssh10:57:492
28015050,0irq/29-eth0-tx-21842-21munin-node06:46:382
28015050,0irq/29-eth0-tx-13209-21ssh09:59:492
28015050,0irq/29-eth0-tx-11831-21gltestperf09:06:282
28015050,0irq/29-eth0-tx-11046-21sh09:41:282
28015050,0irq/29-eth0-tx-0-21swapper/211:32:042
28015050,0irq/29-eth0-tx-0-21swapper/211:01:222
28015050,0irq/29-eth0-tx-0-21swapper/208:46:222
28005050,0irq/28-eth0-rx-8403-21ssh09:38:103
28005050,0irq/28-eth0-rx-26897-21kthreadcore08:56:393
28005050,0irq/28-eth0-rx-24949-21kthreadcore11:16:333
28005050,0irq/28-eth0-rx-22340-21mdstat11:31:353
28005050,0irq/28-eth0-rx-21150-40yum-updatesd-he08:15:003
28005050,0irq/28-eth0-rx-19779-21kthreadcore10:01:333
28005050,0irq/28-eth0-rx-17907-21sadc07:06:143
28005050,0irq/28-eth0-rx-12631-21ssh09:58:333
28005050,0irq/28-eth0-rx-12130-21ssh09:06:313
28005050,0irq/28-eth0-rx-0-21swapper/311:36:313
28005050,0irq/28-eth0-rx-0-21swapper/311:26:253
28005050,0irq/28-eth0-rx-0-21swapper/311:11:353
28005050,0irq/28-eth0-rx-0-21swapper/311:02:313
28005050,0irq/28-eth0-rx-0-21swapper/310:21:373
28005050,0irq/28-eth0-rx-0-21swapper/310:16:323
28005050,0irq/28-eth0-rx-0-21swapper/310:10:543
28005050,0irq/28-eth0-rx-0-21swapper/309:26:093
28005050,0irq/28-eth0-rx-0-21swapper/309:17:063
28005050,0irq/28-eth0-rx-0-21swapper/308:45:403
28005050,0irq/28-eth0-rx-0-21swapper/308:16:283
252469955,0cyclictest0-21swapper/210:06:432
252469955,0cyclictest0-21swapper/208:53:152
252469954,0cyclictest15865-21kworker/2:109:30:572
252469950,0cyclictest0-21swapper/209:11:232
252459955,0cyclictest3254-21diskmemload10:51:431
252459950,0cyclictest0-21swapper/111:27:401
252459950,0cyclictest0-21swapper/108:29:121
252449950,0cyclictest0-21swapper/010:57:060
252449950,0cyclictest0-21swapper/010:54:060
252449950,0cyclictest0-21swapper/010:02:520
259950,0migration/232205-21kthreadcore11:21:372
259950,0migration/223907-21kthreadcore09:46:322
259950,0migration/221489-21idleruntime11:51:292
259950,0migration/221279-21kthreadcore07:51:342
259950,0migration/217186-21sh10:19:322
259950,0migration/215226-21sh10:52:172
259950,0migration/214348-21if_eth010:36:292
249950,0watchdog/26263-21grep10:11:372
249950,0watchdog/22921-21cut11:41:292
249950,0watchdog/213392-21proc_pri07:21:382
249950,0watchdog/211776-21if_err_eth007:46:302
249950,0watchdog/20-21swapper/210:01:332
169950,0watchdog/118899-21gltestperf07:51:301
149950,0migration/08191-21latency11:41:340
149950,0migration/07679-21fschecks_count08:06:280
149950,0migration/07097-21kthreadcore10:31:320
149950,0migration/06616-21kthreadcore11:06:310
149950,0migration/0654-21kthreadcore08:41:550
149950,0migration/06340-21kthreadcore09:36:340
149950,0migration/04409-21kthreadcore09:01:420
149950,0migration/03565-21kthreadcore11:21:390
149950,0migration/032523-21kthreadcore07:36:340
149950,0migration/030648-21timerandwakeup09:31:390
149950,0migration/028368-21kthreadcore11:36:320
149950,0migration/028193-21kthreadcore07:56:330
149950,0migration/027672-21kthreadcore10:06:360
149950,0migration/026923-21kthreadcore08:36:340
149950,0migration/024983-21kthreadcore11:51:330
149950,0migration/022800-21irqcore08:16:300
149950,0migration/022541-21kthreadcore09:11:330
149950,0migration/021662-21kthreadcore10:21:340
149950,0migration/021034-21kthreadcore07:06:330
149950,0migration/02001-21runrttasks11:04:210
149950,0migration/019383-21sh09:10:230
149950,0migration/017790-21kthreadcore08:51:450
149950,0migration/015915-21kthreadcore08:31:330
149950,0migration/014892-21kthreadcore11:46:320
149950,0migration/013414-21idleruntime11:11:360
149950,0migration/012095-21kthreadcore11:26:320
8315040,0irq/26-0000:00:0-21swapper/208:11:302
8315040,0irq/26-0000:00:0-21swapper/208:01:142
339940,0migration/328702-21latency_hist07:36:143
339940,0migration/32523-21kthreadcore08:01:323
329940,0watchdog/3970-21munin-node07:16:303
329940,0watchdog/39610-21kthreadcore07:21:343
329940,0watchdog/30-21swapper/306:51:303
329940,0watchdog/30-21swapper/306:26:303
28015040,0irq/29-eth0-tx-3705-21ssh10:30:562
28015040,0irq/29-eth0-tx-3705-21ssh10:30:562
28015040,0irq/29-eth0-tx-30862-21munin-node08:41:432
28015040,0irq/29-eth0-tx-21416-21ssh11:16:222
28015040,0irq/29-eth0-tx-14253-21kthreadcore11:11:392
28015040,0irq/29-eth0-tx-0-21swapper/210:46:162
28015040,0irq/29-eth0-tx-0-21swapper/209:16:262
28015040,0irq/29-eth0-tx-0-21swapper/208:57:492
28015040,0irq/29-eth0-tx-0-21swapper/208:31:282
28005040,0irq/28-eth0-rx-0-21swapper/310:46:253
28005040,0irq/28-eth0-rx-0-21swapper/308:29:403
252469944,0cyclictest0-21swapper/209:31:152
252469944,0cyclictest0-21swapper/206:38:372
252469943,0cyclictest4380-21awk11:06:302
252469943,0cyclictest3346-21snmpd08:16:432
252469943,0cyclictest3346-21snmpd08:03:132
252469943,0cyclictest3070-21irqbalance08:22:002
252469943,0cyclictest29918-21awk07:36:322
252469943,0cyclictest2922-21awk09:01:402
252469943,0cyclictest26428-21awk07:11:312
252459944,0cyclictest0-21swapper/111:32:241
252459944,0cyclictest0-21swapper/109:59:511
252459944,0cyclictest0-21swapper/108:01:371
252459943,0cyclictest7050-21awk09:21:301
252459943,0cyclictest3346-21snmpd11:06:281
252459943,0cyclictest3346-21snmpd08:52:581
252459943,0cyclictest3346-21snmpd08:11:431
252459943,0cyclictest3346-21snmpd07:38:431
252459943,0cyclictest3278-21awk11:41:311
252459943,0cyclictest3070-21irqbalance08:42:001
252459943,0cyclictest30056-21awk06:51:321
252459943,0cyclictest26378-21awk07:11:311
252459943,0cyclictest26217-21awk08:56:371
252459943,0cyclictest22965-21awk06:46:311
252459943,0cyclictest21572-21awk11:51:311
252459943,0cyclictest15290-21awk10:01:311
252459943,0cyclictest14023-21lldpd11:48:491
252459943,0cyclictest14023-21lldpd11:24:191
252459943,0cyclictest14023-21lldpd11:11:481
252459943,0cyclictest14023-21lldpd11:01:181
252459943,0cyclictest14023-21lldpd11:00:181
252459943,0cyclictest14023-21lldpd10:41:181
252459943,0cyclictest14023-21lldpd09:51:411
252459943,0cyclictest14023-21lldpd09:43:411
252459943,0cyclictest14023-21lldpd07:02:101
252459943,0cyclictest1047-21awk07:16:301
252449944,0cyclictest0-21swapper/010:44:290
252449944,0cyclictest0-21swapper/009:26:360
252449943,0cyclictest8633-21awk06:36:400
252449943,0cyclictest8320-21awk07:21:310
252449943,0cyclictest3346-21snmpd09:42:430
252449943,0cyclictest3070-21irqbalance10:50:090
252449943,0cyclictest3070-21irqbalance10:28:300
252449943,0cyclictest3070-21irqbalance10:28:300
252449943,0cyclictest3070-21irqbalance09:55:200
252449943,0cyclictest3070-21irqbalance08:15:000
252449943,0cyclictest3070-21irqbalance08:01:500
252449943,0cyclictest3070-21irqbalance07:01:410
252449943,0cyclictest30290-21awk09:16:290
252449943,0cyclictest29751-21awk08:21:290
252449943,0cyclictest26326-21awk08:56:380
252449943,0cyclictest17493-21awk11:31:320
259940,0migration/28104-21hddtemp_smartct08:06:292
259940,0migration/27354-21aten2_r1power_e11:26:252
259940,0migration/232509-21aten2_r1power_e07:16:272
259940,0migration/222512-21aten2_r1power_p10:41:312
259940,0migration/222171-21fschecks_count07:31:342
259940,0migration/219728-21irqrtprio10:21:332
249940,0watchdog/24494-21kthreadcore09:36:332
249940,0watchdog/224428-21tr08:36:342
249940,0watchdog/217098-21kthreadcore11:46:332
249940,0watchdog/20-21swapper/207:26:142
249940,0watchdog/20-21swapper/206:51:302
169940,0watchdog/19609-21kthreadcore07:21:341
169940,0watchdog/16264-21pmu-power10:11:371
169940,0watchdog/15242-21kthreadcore09:36:331
169940,0watchdog/126406-21idleruntime06:26:301
169940,0watchdog/124122-21grep09:11:331
169940,0watchdog/114473-21latency_hist07:26:141
169940,0watchdog/112783-21kthreadcore06:36:421
169940,0watchdog/10-21swapper/111:16:331
169940,0watchdog/10-21swapper/110:26:331
169940,0watchdog/10-21swapper/110:26:331
169940,0watchdog/10-21swapper/109:01:411
169940,0watchdog/10-21swapper/108:31:341
169940,0watchdog/10-21swapper/107:31:341
169940,0watchdog/10-21swapper/106:56:341
149940,0migration/08116-21ksmtuned10:14:350
149940,0migration/022802-21irqcore11:16:310
149940,0migration/022204-21df_abs06:46:280
149940,0migration/015943-21irqrtprio07:26:320
8315030,0irq/26-0000:00:0-21swapper/207:06:192
329930,0watchdog/30-21swapper/307:26:143
329930,0watchdog/30-21swapper/306:41:383
252469933,0cyclictest15865-21kworker/2:107:01:292
252459933,0cyclictest14023-21lldpd06:43:091
252459933,0cyclictest0-21swapper/109:16:201
252459933,0cyclictest0-21swapper/107:41:351
252459932,0cyclictest8293-21awk08:06:311
252459932,0cyclictest30514-21sensors09:31:361
252459932,0cyclictest29234-21cut07:56:341
252459932,0cyclictest28125-21grep10:06:361
252459932,0cyclictest22448-21awk08:36:311
252459932,0cyclictest19297-21awk10:21:291
252459932,0cyclictest16153-21awk09:26:291
252459932,0cyclictest14023-21lldpd10:36:161
252459932,0cyclictest14023-21lldpd08:17:401
252459932,0cyclictest11790-21awk07:46:301
252459932,0cyclictest10098-21tail10:16:281
252459930,0cyclictest19496-21df09:46:261
252459930,0cyclictest18852-21perl07:06:291
252459930,0cyclictest0-21swapper/106:31:291
252449932,0cyclictest7029-21awk09:21:300
252449932,0cyclictest3070-21irqbalance07:46:300
252449932,0cyclictest3070-21irqbalance07:31:400
252449932,0cyclictest3070-21irqbalance07:16:410
252449932,0cyclictest3070-21irqbalance07:11:510
252449932,0cyclictest19498-21irqstats07:51:330
252449930,0cyclictest4211-21perl08:26:280
252449930,0cyclictest26077-21ssh09:47:060
169930,0watchdog/10-21swapper/111:36:131
169930,0watchdog/10-21swapper/110:31:211
169930,0watchdog/10-21swapper/109:06:331
169930,0watchdog/10-21swapper/108:46:341
169930,0watchdog/10-21swapper/108:21:261
149930,0migration/021749-21sh10:40:300
28005020,0irq/28-eth0-rx-0-21swapper/307:56:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional