You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-07-29 - 17:47

Intel(R) Xeon(R) CPU E3-1270 V2 @ 3.50GHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Thu Jul 29, 2021 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291450540,0irq/31-eth10-21swapper/107:19:331
1101623929,0sleep30-21swapper/307:22:433
1072923626,0sleep20-21swapper/207:19:262
1072723026,0sleep00-21swapper/007:19:240
111169980,0cyclictest5640-21grep10:28:263
111169974,0cyclictest0-21swapper/309:54:113
111169970,0cyclictest0-21swapper/312:43:593
111119976,0cyclictest0-21swapper/010:38:010
35160,0ktimersoftd/30-21swapper/311:37:373
149960,0migration/030693-21sh10:55:360
149960,0migration/01467-21proc_pri10:58:290
149960,0migration/01109-21sh12:06:250
111169966,0cyclictest0-21swapper/311:32:533
111169960,0cyclictest0-21swapper/312:33:153
111139960,0cyclictest0-21swapper/212:13:592
111119960,0cyclictest0-21swapper/010:14:500
329950,0watchdog/38629-21ntpq12:13:233
329950,0watchdog/327545-21gltestperf08:58:153
28425050,0irq/33-eth0-tx-30750-21vmstat11:28:262
28425050,0irq/33-eth0-tx-26408-21munin-node07:53:200
28425050,0irq/33-eth0-tx-1840-21tune2fs11:33:242
28425050,0irq/33-eth0-tx-17728-21munin-run10:43:012
28425050,0irq/33-eth0-tx-17575-21perl10:08:192
28425050,0irq/33-eth0-tx-14442-21munin-node11:13:172
28425050,0irq/33-eth0-tx-13839-21munin-node07:28:240
28425050,0irq/33-eth0-tx-11067-21port_sshpriv11:08:192
28425050,0irq/33-eth0-tx-0-21swapper/212:03:102
28425050,0irq/33-eth0-tx-0-21swapper/211:55:452
28425050,0irq/33-eth0-tx-0-21swapper/211:23:562
28425050,0irq/33-eth0-tx-0-21swapper/210:28:172
28425050,0irq/33-eth0-tx-0-21swapper/210:16:322
28425050,0irq/33-eth0-tx-0-21swapper/209:39:542
28425050,0irq/33-eth0-tx-0-21swapper/209:31:182
28425050,0irq/33-eth0-tx-0-21swapper/009:23:090
28425050,0irq/33-eth0-tx-0-21swapper/008:59:510
28425050,0irq/33-eth0-tx-0-21swapper/008:54:500
28425050,0irq/33-eth0-tx-0-21swapper/008:47:430
28425050,0irq/33-eth0-tx-0-21swapper/008:28:140
28425050,0irq/33-eth0-tx-0-21swapper/007:50:550
28425050,0irq/33-eth0-tx-0-21swapper/007:41:330
27150,0ktimersoftd/20-21swapper/211:59:102
259950,0migration/231793-21sh09:48:322
259950,0migration/229067-21sh10:19:362
259950,0migration/227696-21needreboot12:33:222
259950,0migration/22721-21fschecks_count12:08:162
259950,0migration/216543-21latency_hist12:23:022
249950,0watchdog/2109742sleep210:34:232
249950,0watchdog/20-21swapper/212:43:592
169950,0watchdog/16339-21pmu-power11:03:311
169950,0watchdog/10-21swapper/109:03:111
169950,0watchdog/10-21swapper/107:48:231
149950,0migration/06655-21sh12:12:350
149950,0migration/06628-21latency_hist08:18:010
149950,0migration/04989-21sh11:36:400
149950,0migration/03610-21sh10:28:050
149950,0migration/031908-21cpuspeed_turbos09:08:250
149950,0migration/02876-21sh12:42:230
149950,0migration/027960-21rt-features12:33:250
149950,0migration/026677-21sshd11:24:430
149950,0migration/026404-21latency_hist10:18:010
149950,0migration/023674-21sh09:40:100
149950,0migration/022641-21munin-node12:28:190
149950,0migration/01898-21latency_hist09:13:010
149950,0migration/013426-21munin-node10:03:270
149950,0migration/012853-21sh11:43:330
149950,0migration/010238-21sh11:42:350
111169955,0cyclictest0-21swapper/312:31:033
111169955,0cyclictest0-21swapper/312:20:013
111169955,0cyclictest0-21swapper/311:46:543
111169955,0cyclictest0-21swapper/311:39:013
111169955,0cyclictest0-21swapper/311:06:043
111169955,0cyclictest0-21swapper/310:18:283
111169955,0cyclictest0-21swapper/310:15:003
111169955,0cyclictest0-21swapper/309:41:063
111169955,0cyclictest0-21swapper/308:18:203
111169955,0cyclictest0-21swapper/307:41:453
111169950,0cyclictest0-21swapper/311:25:183
111139955,0cyclictest0-21swapper/212:50:052
111139955,0cyclictest0-21swapper/212:40:282
111139950,0cyclictest0-21swapper/207:39:402
111129955,0cyclictest6876-21diskmemload11:48:201
111129950,0cyclictest0-21swapper/112:04:541
111129950,0cyclictest0-21swapper/110:48:161
111119955,0cyclictest0-21swapper/010:39:550
111119955,0cyclictest0-21swapper/009:33:130
111119954,0cyclictest5395-21snmpd12:22:420
111119950,0cyclictest0-21swapper/012:15:180
111119950,0cyclictest0-21swapper/011:09:520
8305040,0irq/26-0000:00:1694-21latency_hist09:13:012
35140,0ktimersoftd/30-21swapper/309:10:023
329940,0watchdog/329334-21cpu09:03:113
329940,0watchdog/3284150irq/32-eth0-rx-11:59:513
329940,0watchdog/327572-21ssh10:53:153
329940,0watchdog/314400-21ssh10:03:553
329940,0watchdog/30-21swapper/312:03:033
329940,0watchdog/30-21swapper/310:58:193
329940,0watchdog/30-21swapper/310:38:273
329940,0watchdog/30-21swapper/310:34:233
329940,0watchdog/30-21swapper/309:58:193
329940,0watchdog/30-21swapper/307:48:233
28425040,0irq/33-eth0-tx-0-21swapper/212:28:092
28425040,0irq/33-eth0-tx-0-21swapper/212:23:072
28425040,0irq/33-eth0-tx-0-21swapper/211:48:192
28425040,0irq/33-eth0-tx-0-21swapper/210:53:252
28425040,0irq/33-eth0-tx-0-21swapper/210:43:102
28425040,0irq/33-eth0-tx-0-21swapper/210:24:382
28425040,0irq/33-eth0-tx-0-21swapper/210:03:182
28425040,0irq/33-eth0-tx-0-21swapper/209:58:192
28425040,0irq/33-eth0-tx-0-21swapper/209:55:062
28425040,0irq/33-eth0-tx-0-21swapper/209:43:142
28425040,0irq/33-eth0-tx-0-21swapper/209:33:132
28425040,0irq/33-eth0-tx-0-21swapper/008:38:280
28425040,0irq/33-eth0-tx-0-21swapper/008:08:250
28425040,0irq/33-eth0-tx-0-21swapper/007:23:170
28415040,0irq/32-eth0-rx-0-21swapper/312:49:393
28415040,0irq/32-eth0-rx-0-21swapper/311:51:233
28415040,0irq/32-eth0-rx-0-21swapper/311:16:343
28415040,0irq/32-eth0-rx-0-21swapper/310:44:073
28415040,0irq/32-eth0-rx-0-21swapper/310:24:063
28415040,0irq/32-eth0-rx-0-21swapper/309:51:343
28415040,0irq/32-eth0-rx-0-21swapper/309:44:443
259940,0migration/27033-21latency_hist09:23:012
259940,0migration/26748-21runrttasks09:21:402
259940,0migration/223843-21runrttasks08:50:342
259940,0migration/220488-21missed_timers08:43:212
259940,0migration/216716-21latency_hist08:38:012
249940,0watchdog/25960-21ssh11:38:072
249940,0watchdog/212374-21irqstats07:23:232
249940,0watchdog/20-21swapper/211:03:312
249940,0watchdog/20-21swapper/210:58:072
249940,0watchdog/20-21swapper/208:28:152
249940,0watchdog/20-21swapper/207:43:272
179940,0migration/11750-21sh10:25:141
179940,0migration/113339-21taskset07:25:021
179940,0migration/110999-21sh12:49:311
169940,0watchdog/15891-21ssh09:55:511
169940,0watchdog/128838-21ssh10:19:111
169940,0watchdog/122973-21sh09:38:471
169940,0watchdog/117016-21df_abs12:23:151
169940,0watchdog/11609-21ssh09:51:591
169940,0watchdog/112933-21perl10:03:191
169940,0watchdog/112029-21fschecks_count08:28:151
169940,0watchdog/10-21swapper/111:19:071
169940,0watchdog/10-21swapper/110:58:191
169940,0watchdog/10-21swapper/110:45:111
169940,0watchdog/10-21swapper/109:29:391
169940,0watchdog/10-21swapper/108:58:151
149940,0migration/05637-21iostat09:18:240
149940,0migration/05503-21timerwakeupswit12:43:320
149940,0migration/021721-21diskmemload10:46:400
149940,0migration/013447-21unixbench_multi09:28:250
111169944,0cyclictest0-21swapper/311:09:423
111169944,0cyclictest0-21swapper/310:48:153
111169944,0cyclictest0-21swapper/307:44:453
111169943,0cyclictest5620-21awk09:18:243
111169943,0cyclictest5395-21snmpd12:40:423
111169943,0cyclictest5395-21snmpd12:26:423
111169943,0cyclictest5395-21snmpd11:56:573
111169943,0cyclictest5395-21snmpd09:28:103
111169943,0cyclictest5395-21snmpd09:23:253
111169943,0cyclictest5395-21snmpd09:16:553
111169943,0cyclictest5395-21snmpd08:35:093
111169943,0cyclictest29802-21awk07:58:193
111169943,0cyclictest27280-21awk07:53:233
111169943,0cyclictest25249-21awk08:53:223
111169943,0cyclictest12683-21awk08:28:173
111139943,0cyclictest5395-21snmpd11:43:122
111139943,0cyclictest5395-21snmpd11:18:112
111139943,0cyclictest5395-21snmpd10:48:262
111139943,0cyclictest5395-21snmpd09:04:512
111139943,0cyclictest5395-21snmpd08:13:242
111139943,0cyclictest403-21awk09:08:352
111139943,0cyclictest3111-21irqbalance07:34:002
111139943,0cyclictest3111-21irqbalance07:29:002
111139943,0cyclictest26163-21sadc07:53:012
111139943,0cyclictest2467-21awk08:08:232
111139943,0cyclictest10114-21awk08:23:222
111139940,0cyclictest0-21swapper/208:57:142
111139940,0cyclictest0-21swapper/208:37:122
111129944,0cyclictest0-21swapper/110:38:491
111129944,0cyclictest0-21swapper/110:36:551
111129944,0cyclictest0-21swapper/109:26:171
111129943,0cyclictest7580-21awk08:18:181
111129943,0cyclictest5395-21snmpd12:43:421
111129943,0cyclictest5395-21snmpd12:38:541
111129943,0cyclictest5395-21snmpd12:33:271
111129943,0cyclictest5395-21snmpd11:35:421
111129943,0cyclictest5395-21snmpd10:01:401
111129943,0cyclictest5395-21snmpd09:43:231
111129943,0cyclictest5395-21snmpd09:20:101
111129943,0cyclictest5395-21snmpd09:13:551
111129943,0cyclictest5395-21snmpd08:54:091
111129943,0cyclictest5395-21snmpd08:41:091
111129943,0cyclictest5395-21snmpd08:24:091
111129943,0cyclictest5395-21snmpd08:13:591
111129943,0cyclictest5395-21snmpd08:10:441
111129943,0cyclictest5395-21snmpd08:00:541
111129943,0cyclictest5395-21snmpd07:54:161
111129943,0cyclictest3392-21awk12:08:191
111129943,0cyclictest32325-21awk08:03:161
111129943,0cyclictest22813-21awk10:13:211
111129943,0cyclictest15266-21awk08:33:171
111129943,0cyclictest11770-21awk11:43:151
111129943,0cyclictest10548-21awk11:08:151
111129942,0cyclictest5395-21snmpd07:41:351
111129940,0cyclictest26257-21sh11:58:201
111129940,0cyclictest0-21swapper/111:54:541
111129940,0cyclictest0-21swapper/111:29:441
111119944,0cyclictest0-21swapper/011:55:450
111119944,0cyclictest0-21swapper/009:45:030
111119943,0cyclictest9433-21awk12:48:200
111119943,0cyclictest5395-21snmpd11:49:420
111119943,0cyclictest5395-21snmpd11:04:560
111119943,0cyclictest5395-21snmpd10:25:410
111119943,0cyclictest5395-21snmpd10:00:550
111119943,0cyclictest5395-21snmpd09:05:240
111119943,0cyclictest5395-21snmpd08:34:090
111119943,0cyclictest5395-21snmpd08:23:240
111119943,0cyclictest5395-21snmpd08:01:240
111119943,0cyclictest5395-21snmpd07:43:240
111119943,0cyclictest5395-21snmpd07:33:390
111119943,0cyclictest5182-21awk08:13:200
111119943,0cyclictest29725-21awk11:28:160
111119943,0cyclictest22781-21awk08:48:200
111119943,0cyclictest20286-21awk11:18:200
111119943,0cyclictest17964-21awk10:08:210
111119943,0cyclictest15348-21awk11:13:190
111119940,0cyclictest31422-21munin-node08:03:240
329930,0watchdog/30-21swapper/310:08:193
329930,0watchdog/30-21swapper/308:23:273
329930,0watchdog/30-21swapper/308:13:153
329930,0watchdog/30-21swapper/307:28:033
28415030,0irq/32-eth0-rx-0-21swapper/312:08:053
28415030,0irq/32-eth0-rx-0-21swapper/311:18:103
28415030,0irq/32-eth0-rx-0-21swapper/309:33:043
249930,0watchdog/20-21swapper/208:58:152
169930,0watchdog/10-21swapper/112:18:311
169930,0watchdog/10-21swapper/111:13:031
169930,0watchdog/10-21swapper/110:53:151
169930,0watchdog/10-21swapper/110:28:271
111169932,0cyclictest31994-21perl08:03:143
111169932,0cyclictest22693-21awk08:48:193
111169932,0cyclictest17587-21strings07:33:213
111169932,0cyclictest16756-21seq08:38:093
111169930,0cyclictest19848-21perl08:43:173
111169930,0cyclictest0-21swapper/308:08:283
111139933,0cyclictest0-21swapper/208:01:062
111139932,0cyclictest8218-21sensors08:18:232
111139932,0cyclictest32412-21awk08:03:172
111139932,0cyclictest27326-21awk07:53:242
111129932,0cyclictest8284-21awk12:13:201
111129932,0cyclictest7221-21mdstat11:38:191
111129932,0cyclictest24916-21awk11:23:171
111129932,0cyclictest22663-21awk08:48:191
111129932,0cyclictest22346-21awk07:43:211
111129932,0cyclictest21536-21ssh12:28:051
111129932,0cyclictest20039-21head08:43:171
111129932,0cyclictest17746-21grep10:08:191
111129932,0cyclictest17296-21awk07:33:191
111129932,0cyclictest14808-21awk07:28:201
111129930,0cyclictest16886-21perl09:33:141
111129930,0cyclictest0-21swapper/109:08:291
111119933,0cyclictest32665-21kworker/0:212:23:470
111119933,0cyclictest32665-21kworker/0:209:53:070
111119933,0cyclictest32665-21kworker/0:209:48:110
111119932,0cyclictest5395-21snmpd11:58:120
111119932,0cyclictest23711-21awk10:48:180
111169920,0cyclictest11051-21latency_hist07:23:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional