You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-08-04 - 21:25

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.16.15-rt7 (Profile)

Latency plot of system in rack #b, slot #1
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rackbslot1.osadl.org (updated Wed Aug 04, 2021 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
208112610,0sleep10-21swapper/109:35:091
3812580,0sleep10-21swapper/108:05:221
2966825442,7sleep00-21swapper/007:09:520
2931825342,7sleep10-21swapper/107:06:321
2968424937,7sleep30-21swapper/307:09:583
2925324635,7sleep20-21swapper/207:05:562
86922190,1sleep20-21swapper/211:55:262
29816991714,2cyclictest6584-21snmpd08:19:573
29816991614,1cyclictest6584-21snmpd08:08:213
29803991513,1cyclictest328-21kworker/0:207:15:060
29816991413,0cyclictest6584-21snmpd08:29:223
29816991412,1cyclictest6584-21snmpd10:15:283
29813991412,1cyclictest6584-21snmpd12:15:412
29813991312,0cyclictest6584-21snmpd08:11:462
29813991312,0cyclictest6584-21snmpd07:29:382
29813991311,1cyclictest6584-21snmpd09:42:332
2980399130,12cyclictest4201-31munin-html09:05:300
29816991211,1cyclictest6584-21snmpd09:21:413
29816991210,1cyclictest32515-21munin-update10:50:003
2981699120,1cyclictest20833-31munin-html12:15:353
2981699120,11cyclictest8259-21/usr/share/muni09:15:143
2981699120,11cyclictest0-21swapper/310:30:383
29813991212,0cyclictest6584-21snmpd11:49:122
29813991211,0cyclictest6584-21snmpd09:01:172
29813991210,1cyclictest0-21swapper/209:05:002
2981399120,11cyclictest26317-21ntpq08:50:092
2981399120,11cyclictest23205-31munin-html11:25:292
2981399120,11cyclictest0-21swapper/208:20:272
29807991211,0cyclictest6584-21snmpd10:49:231
29803991210,1cyclictest26173-21kworker/0:110:25:350
29803991210,1cyclictest25015-21kworker/0:008:50:250
2980399120,11cyclictest32222-31munin-html11:40:300
2980399120,11cyclictest20622-31munin-graph11:20:390
2981699119,1cyclictest0-21swapper/308:35:413
2981699110,1cyclictest0-21swapper/308:30:493
2981699110,10cyclictest0-21swapper/312:14:463
2981699110,10cyclictest0-21swapper/311:24:433
2981699110,10cyclictest0-21swapper/309:47:023
2981699110,10cyclictest0-21swapper/309:47:013
2981699110,10cyclictest0-21swapper/308:55:123
2981399119,1cyclictest0-21swapper/211:15:382
2981399119,1cyclictest0-21swapper/211:09:202
2981399119,1cyclictest0-21swapper/210:15:542
2981399119,1cyclictest0-21swapper/207:34:462
29813991111,0cyclictest6584-21snmpd11:44:372
2981399110,1cyclictest0-21swapper/212:30:142
2981399110,10cyclictest0-21swapper/210:06:212
2981399110,10cyclictest0-21swapper/208:16:052
2981399110,10cyclictest0-21swapper/207:43:472
2980799119,1cyclictest6584-21snmpd12:00:081
2980799119,1cyclictest6584-21snmpd08:55:441
2980799119,1cyclictest0-21swapper/110:10:571
2980799119,1cyclictest0-21swapper/109:30:251
2980799119,1cyclictest0-21swapper/108:00:291
2980799110,10cyclictest0-21swapper/111:10:131
2980399119,1cyclictest15330-21kworker/0:012:15:080
29803991110,0cyclictest26399-21kworker/0:010:57:140
29803991110,0cyclictest26173-21kworker/0:110:15:310
29803991110,0cyclictest11098-21kworker/0:108:45:240
2980399110,1cyclictest0-21swapper/011:15:290
2981699109,0cyclictest0-21swapper/309:25:243
2981699108,1cyclictest0-21swapper/310:06:243
2981699100,1cyclictest0-21swapper/309:13:183
2981399109,0cyclictest0-21swapper/207:45:062
2981399109,0cyclictest0-21swapper/207:45:052
2981399108,1cyclictest0-21swapper/212:39:452
2981399108,1cyclictest0-21swapper/211:21:562
2981399108,1cyclictest0-21swapper/209:45:132
2981399108,1cyclictest0-21swapper/209:45:122
2981399108,1cyclictest0-21swapper/207:15:512
2981399108,1cyclictest0-21swapper/207:12:482
2981399101,1cyclictest91rcu_preempt08:00:002
2981399100,9cyclictest0-21swapper/210:34:172
2981399100,1cyclictest0-21swapper/209:58:472
2981399100,1cyclictest0-21swapper/209:21:102
2981399100,1cyclictest0-21swapper/207:36:032
2980799100,9cyclictest0-21swapper/111:32:491
2980799100,9cyclictest0-21swapper/108:45:231
2980799100,9cyclictest0-21swapper/107:43:511
2980399109,0cyclictest26173-21kworker/0:110:20:240
2980399109,0cyclictest24745-21kworker/0:308:10:130
2980399109,0cyclictest14223-21kworker/0:109:35:200
29803991010,0cyclictest6584-21snmpd09:52:160
2980399100,9cyclictest0-21swapper/012:34:550
2980399100,1cyclictest0-21swapper/011:00:470
298169998,0cyclictest9885-21kworker/3:108:23:543
298169998,0cyclictest6319-21kworker/3:111:59:223
298169997,1cyclictest6584-21snmpd08:46:533
298169997,1cyclictest6584-21snmpd08:44:313
298169997,1cyclictest0-21swapper/312:03:483
298169997,1cyclictest0-21swapper/311:01:483
298169997,1cyclictest0-21swapper/309:58:003
298169997,1cyclictest0-21swapper/309:07:283
298169995,3cyclictest6172-21kworker/3:209:40:423
298169995,3cyclictest32265-21kworker/3:007:12:013
298169995,3cyclictest2342-21kworker/3:010:10:103
298169994,4cyclictest3547-21kworker/3:011:10:313
298169990,8cyclictest0-21swapper/311:28:163
298169990,1cyclictest0-21swapper/309:31:423
298139998,0cyclictest6584-21snmpd12:22:142
298139997,1cyclictest0-21swapper/212:27:002
298139997,1cyclictest0-21swapper/211:37:162
298139997,1cyclictest0-21swapper/209:39:152
298139997,1cyclictest0-21swapper/208:57:452
298139997,1cyclictest0-21swapper/207:54:362
298139990,8cyclictest0-21swapper/207:58:592
298139990,0cyclictest0-21swapper/208:09:482
298079998,0cyclictest0-21swapper/108:30:031
298079997,1cyclictest0-21swapper/111:40:431
298039998,0cyclictest24745-21kworker/0:307:59:590
298039997,1cyclictest6584-21snmpd11:46:430
298039997,1cyclictest6584-21snmpd08:26:450
298039997,1cyclictest30082-21kworker/0:312:00:460
298039997,1cyclictest20177-21kworker/0:210:30:380
298039997,1cyclictest0-21swapper/008:56:380
298039990,1cyclictest101rcu_sched10:10:000
298169987,0cyclictest6319-21kworker/3:111:52:203
298169987,0cyclictest6172-21kworker/3:209:50:583
298169987,0cyclictest6172-21kworker/3:209:38:023
298169987,0cyclictest3547-21kworker/3:011:16:563
298169987,0cyclictest3547-21kworker/3:010:57:563
298169987,0cyclictest32265-21kworker/3:007:21:443
298169987,0cyclictest28480-21kworker/3:212:35:443
298169987,0cyclictest19737-21kworker/3:110:52:083
298169987,0cyclictest19737-21kworker/3:110:39:443
298169986,1cyclictest0-21swapper/309:02:293
298169986,1cyclictest0-21swapper/307:25:363
298169985,2cyclictest9885-21kworker/3:108:50:203
298169985,2cyclictest3547-21kworker/3:011:05:483
298169985,2cyclictest32265-21kworker/3:007:36:443
298169985,2cyclictest32265-21kworker/3:007:30:033
298169985,2cyclictest32265-21kworker/3:007:15:233
298169985,2cyclictest28480-21kworker/3:212:30:403
298169985,2cyclictest27784-21kworker/3:308:13:523
298169985,2cyclictest27784-21kworker/3:308:00:563
298169985,2cyclictest27096-21kworker/3:311:40:183
298169985,2cyclictest27096-21kworker/3:311:39:133
298169985,2cyclictest24729-21kworker/3:112:25:233
298169985,2cyclictest23803-21kworker/3:111:30:283
298169985,2cyclictest19737-21kworker/3:110:28:083
298169985,2cyclictest13432-21kworker/3:012:06:283
298169985,2cyclictest12318-21kworker/3:107:45:403
298169985,2cyclictest12318-21kworker/3:107:45:403
298169985,2cyclictest11563-21kworker/3:310:20:333
298169984,3cyclictest21488-21kworker/3:207:50:273
298139987,0cyclictest6584-21snmpd10:04:192
298139987,0cyclictest0-21swapper/210:50:082
298139986,1cyclictest33-21ksoftirqd/212:00:002
298139986,1cyclictest33-21ksoftirqd/211:50:012
298139986,1cyclictest33-21ksoftirqd/210:40:012
298139986,1cyclictest33-21ksoftirqd/208:35:012
298139986,1cyclictest0-21swapper/210:37:222
298139986,1cyclictest0-21swapper/210:26:142
298139986,1cyclictest0-21swapper/210:11:422
298139986,1cyclictest0-21swapper/209:31:482
298139985,2cyclictest33-21ksoftirqd/210:25:002
298079988,0cyclictest6584-21snmpd12:11:171
298079988,0cyclictest6584-21snmpd09:18:321
298079987,1cyclictest6584-21snmpd09:07:111
298079987,0cyclictest6584-21snmpd12:35:481
298079986,1cyclictest0-21swapper/110:01:451
298039986,1cyclictest8798-21kworker/0:109:15:290
298039986,1cyclictest6312-21kworker/0:007:25:350
298039986,1cyclictest6312-21kworker/0:007:21:200
298039986,1cyclictest4940-21kworker/0:009:20:440
298039986,1cyclictest26173-21kworker/0:109:55:320
298039986,1cyclictest26173-21kworker/0:109:45:320
298039986,1cyclictest26173-21kworker/0:109:45:320
298039986,1cyclictest24745-21kworker/0:308:05:020
298039986,1cyclictest24306-21kworker/0:207:52:520
298039986,1cyclictest22436-21kworker/0:009:40:270
298039986,1cyclictest21061-21kworker/0:007:47:480
298039986,1cyclictest21061-21kworker/0:007:47:470
298039986,1cyclictest11803-21kworker/0:207:30:280
298039986,1cyclictest11098-21kworker/0:108:40:020
30687270,5sleep029769-21/usr/share/muni10:45:130
298169976,0cyclictest13432-21kworker/3:012:20:083
298169975,2cyclictest12318-21kworker/3:107:40:553
298169974,2cyclictest27096-21kworker/3:311:45:163
298169974,2cyclictest24550-21kworker/3:107:57:003
298169974,2cyclictest2342-21kworker/3:010:01:063
298169974,2cyclictest19737-21kworker/3:110:40:423
298139977,0cyclictest6584-21snmpd08:49:202
298139975,1cyclictest33-21ksoftirqd/212:15:002
298139975,1cyclictest33-21ksoftirqd/210:55:002
298139975,1cyclictest33-21ksoftirqd/210:50:002
298139975,1cyclictest33-21ksoftirqd/209:20:002
298139975,1cyclictest0-21swapper/211:03:032
298139975,1cyclictest0-21swapper/209:26:392
298139974,2cyclictest26212-21ping11:30:302
298079976,0cyclictest6584-21snmpd11:15:061
298079976,0cyclictest0-21swapper/110:55:121
298079975,1cyclictest6584-21snmpd09:43:261
298079975,1cyclictest6584-21snmpd09:26:121
298079975,1cyclictest6584-21snmpd07:48:121
298079975,1cyclictest6584-21snmpd07:48:111
298079975,1cyclictest0-21swapper/112:05:451
298079975,1cyclictest0-21swapper/111:24:051
298079975,1cyclictest0-21swapper/110:35:391
298079975,1cyclictest0-21swapper/110:22:541
298079975,1cyclictest0-21swapper/109:57:331
298079975,1cyclictest0-21swapper/107:55:391
298079974,2cyclictest4049-21openvpn12:18:021
298079974,2cyclictest32053-21ntpdc09:00:071
298079970,6cyclictest0-21swapper/108:26:481
298079970,6cyclictest0-21swapper/107:54:221
298079970,6cyclictest0-21swapper/107:21:251
298039977,0cyclictest26173-21kworker/0:110:10:100
298039977,0cyclictest23828-21kworker/0:011:30:030
298039976,1cyclictest9209-21kworker/0:111:58:510
298039976,1cyclictest31131-21kworker/0:109:03:150
298039976,1cyclictest29679-21kworker/0:007:10:270
298039976,1cyclictest26173-21kworker/0:110:00:410
298039976,1cyclictest1870-21kworker/0:209:11:360
298039976,1cyclictest15348-21kworker/0:107:41:460
298039976,1cyclictest15330-21kworker/0:012:28:480
298039976,1cyclictest15330-21kworker/0:012:12:180
298039976,1cyclictest14663-21kworker/0:111:13:400
298039976,1cyclictest14223-21kworker/0:109:25:070
298039976,1cyclictest11803-21kworker/0:207:35:130
298039976,1cyclictest11098-21kworker/0:108:34:100
298039976,0cyclictest4132-21kworker/0:008:15:200
298039976,0cyclictest23300-21kworker/0:110:35:060
298039975,1cyclictest6153-21kworker/0:211:05:290
298039975,1cyclictest26399-21kworker/0:010:40:170
298039975,1cyclictest14663-21kworker/0:111:25:140
298039975,1cyclictest14223-21kworker/0:109:30:520
298039975,1cyclictest12241-21kworker/0:212:05:360
298039970,6cyclictest0-21swapper/011:35:200
7554260,1sleep10-21swapper/109:10:591
298139966,0cyclictest14542-21awk12:05:242
298139965,1cyclictest33-21ksoftirqd/208:35:012
298139964,1cyclictest5610-21awk07:20:182
298139964,1cyclictest33-21ksoftirqd/209:50:002
298139964,1cyclictest33-21ksoftirqd/208:45:002
298139964,1cyclictest33-21ksoftirqd/208:30:002
298139963,2cyclictest0-21swapper/211:10:312
298139961,1cyclictest91rcu_preempt09:10:332
298079964,1cyclictest9555-31munin-html08:20:281
298079964,1cyclictest28995-21grep12:30:131
298079964,1cyclictest0-21swapper/112:25:171
298079964,1cyclictest0-21swapper/111:28:271
298079964,1cyclictest0-21swapper/110:44:121
298079964,1cyclictest0-21swapper/109:52:131
298079964,1cyclictest0-21swapper/109:20:251
298079964,1cyclictest0-21swapper/108:15:591
298079964,1cyclictest0-21swapper/107:30:181
298079963,2cyclictest6584-21snmpd12:22:441
298079963,2cyclictest6584-21snmpd11:50:311
298079963,2cyclictest6584-21snmpd11:35:311
298079963,2cyclictest6584-21snmpd10:53:081
298079963,2cyclictest6584-21snmpd09:46:521
298079963,2cyclictest6584-21snmpd09:46:521
298079963,2cyclictest6584-21snmpd08:41:291
298079963,2cyclictest6584-21snmpd08:38:071
298079963,2cyclictest6584-21snmpd07:39:171
298079963,2cyclictest3663-21smart_sda08:10:281
298079962,3cyclictest666-21dump-pmu-power11:49:551
298079960,1cyclictest0-21swapper/108:51:491
298079960,1cyclictest0-21swapper/107:26:441
298039965,1cyclictest7456-21kworker/0:308:20:180
298039965,1cyclictest30082-21kworker/0:311:50:040
298039965,0cyclictest15330-21kworker/0:012:35:300
298039963,2cyclictest348-21ping10:50:020
298039962,1cyclictest91rcu_preempt12:20:000
298039962,1cyclictest91rcu_preempt07:55:080
298039962,1cyclictest0-21swapper/008:35:110
19944260,0sleep119685-31munin-graph10:25:441
298079954,0cyclictest0-21swapper/111:05:031
298079953,1cyclictest6580-21grep10:05:091
298079953,1cyclictest6357-21/usr/share/muni11:00:141
298079953,1cyclictest30288-21cat07:10:051
298079953,1cyclictest21493-21cat10:30:091
298079953,1cyclictest1394-21cat07:15:051
298079953,1cyclictest12126-21cat10:15:041
298079952,2cyclictest7166-21needreboot11:55:051
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional