You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-04-21 - 00:46

Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #a, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rackaslot0.osadl.org (updated Tue Apr 20, 2021 12:44:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
73262660,1sleep21714-21runrttasks10:37:042
90712590,1sleep00-21swapper/010:29:050
142082560,1sleep21714-21runrttasks11:32:012
8402550,0sleep00-21swapper/010:23:480
68902550,0sleep20-21swapper/209:16:292
82082530,0sleep30-21swapper/312:00:183
63852520,1sleep26378-21sshd10:53:432
3155325121,26sleep10-21swapper/107:05:471
183132510,0sleep30-21swapper/309:18:103
3119425027,19sleep20-21swapper/207:05:242
236512500,0sleep2291ktimersoftd/209:40:242
55872490,1sleep08-21rcu_preempt11:55:450
4762490,0sleep224529-21diskmemload10:44:372
3154724924,9sleep00-21swapper/007:05:420
203272490,0sleep120323-21sshd11:07:551
188352490,1sleep218832-21bash11:28:322
306022480,0sleep00-21swapper/011:42:230
94142470,0sleep10-21swapper/111:23:101
2954124724,19sleep30-21swapper/307:05:013
276632470,0sleep20-21swapper/210:48:032
276102470,0sleep00-21swapper/012:31:470
252702460,0sleep20-21swapper/210:26:552
98502450,0sleep30-21swapper/309:47:023
144252450,0sleep114418-21sshd11:44:341
64192440,0sleep00-21swapper/010:03:330
309072440,1sleep01714-21runrttasks09:49:510
32140992119,1cyclictest0-21swapper/309:25:483
32137992110,10cyclictest25398-21sshd10:10:130
3213799210,21cyclictest0-21swapper/009:55:100
3213799203,4cyclictest21812-21missed_timers12:10:190
3213799192,16cyclictest5964-21sshd10:07:380
32137991917,1cyclictest21762-21sshd11:20:250
32137991911,7cyclictest22206-21sshd11:33:030
3213799191,11cyclictest0-21swapper/012:16:580
19362190,0sleep00-21swapper/010:40:270
3213799189,8cyclictest0-21swapper/009:31:120
3213799182,4cyclictest29981-21sshd12:07:150
3213799182,3cyclictest0-21swapper/010:55:400
32137991810,7cyclictest0-21swapper/012:23:050
186672180,1sleep118666-21sshd10:55:141
3213799174,4cyclictest27440-21sshd09:40:540
3213799173,13cyclictest3861-21sshd11:51:260
3213799173,13cyclictest31995-21bash09:24:550
3213799173,13cyclictest24529-21diskmemload10:51:050
3213799173,13cyclictest17155-21sshd10:30:080
3213799172,12cyclictest0-21swapper/011:25:220
3213799172,11cyclictest24529-21diskmemload11:15:070
3213799172,11cyclictest24529-21diskmemload11:15:070
3213799172,10cyclictest114850irq/25-eth012:27:390
32137991712,4cyclictest5378-21sshd09:38:080
32137991711,5cyclictest23574-21sshd09:27:580
32137991711,5cyclictest0-21swapper/011:46:490
32137991711,5cyclictest0-21swapper/011:02:500
3213799171,12cyclictest0-21swapper/009:11:150
32137991710,6cyclictest0-21swapper/009:51:200
3213799170,14cyclictest0-21swapper/011:36:310
3213799170,13cyclictest0-21swapper/012:00:380
125782170,0sleep10-21swapper/110:46:001
93862160,1sleep21714-21runrttasks10:16:252
32140991614,1cyclictest7237-21sshd10:24:393
3213899165,11cyclictest0-21swapper/110:19:171
3213799164,5cyclictest28176-21sshd10:18:570
3213799164,11cyclictest14437-21sshd10:46:150
3213799163,4cyclictest30830-21sshd11:05:120
32137991612,3cyclictest9915-21sshd12:37:500
32137991611,4cyclictest27931-21sshd11:13:030
32137991611,4cyclictest27931-21sshd11:13:030
309472160,0sleep20-21swapper/212:24:002
244732160,0sleep20-21swapper/210:22:382
203702160,1sleep220354-21sshd12:35:062
32140991513,1cyclictest32588-21bash11:26:033
32140991510,2cyclictest9883-21sshd10:33:193
3213999150,14cyclictest0-21swapper/209:39:482
32137991512,2cyclictest29910-21sshd10:35:460
3213799150,11cyclictest0-21swapper/009:15:090
30012150,0sleep3381ktimersoftd/309:37:483
149262150,1sleep124529-21diskmemload10:08:511
3214099149,4cyclictest0-21swapper/309:41:543
3214099149,2cyclictest0-21swapper/309:54:423
3214099148,5cyclictest0-21swapper/310:25:213
32140991410,1cyclictest114850irq/25-eth011:52:143
3214099140,13cyclictest6041-21sshd10:03:313
3214099140,13cyclictest0-21swapper/312:34:303
3214099140,13cyclictest0-21swapper/311:49:143
3213999149,4cyclictest0-21swapper/212:15:092
3213999147,4cyclictest0-21swapper/211:54:552
3213999140,13cyclictest0-21swapper/212:28:582
3213999140,13cyclictest0-21swapper/210:57:352
3213999140,13cyclictest0-21swapper/210:00:382
3213899149,2cyclictest23421-21sshd10:35:051
3213899149,2cyclictest0-21swapper/112:20:491
3213899144,1cyclictest7356-21sshd09:46:421
3213899141,2cyclictest0-21swapper/111:58:231
3213899140,1cyclictest0-21swapper/109:56:411
3213899140,13cyclictest0-21swapper/112:07:451
3213899140,13cyclictest0-21swapper/111:54:001
3213899140,13cyclictest0-21swapper/109:53:521
306902140,0sleep10-21swapper/111:34:121
138752140,0sleep10-21swapper/110:42:041
102252140,0sleep30-21swapper/312:17:083
98172130,0sleep10-21swapper/110:20:411
3214099139,1cyclictest0-21swapper/311:56:183
3214099138,4cyclictest15548-21bash10:42:183
3214099138,4cyclictest11667-21cpu10:50:133
3214099138,4cyclictest0-21swapper/311:20:123
3214099138,2cyclictest25698-21sshd10:05:573
3214099138,1cyclictest26678-21uname10:35:223
3214099131,1cyclictest114850irq/25-eth011:09:033
3214099131,11cyclictest0-21swapper/311:38:343
3214099131,11cyclictest0-21swapper/311:30:073
3214099130,12cyclictest28687-21sshd10:48:113
3214099130,12cyclictest23371-21bash11:00:063
3214099130,12cyclictest20870-21sshd12:10:153
3214099130,12cyclictest15146-21sshd12:05:213
3214099130,12cyclictest14678-21sshd12:21:503
3214099130,12cyclictest12850-21sshd11:15:153
3214099130,12cyclictest12850-21sshd11:15:153
3214099130,11cyclictest0-21swapper/312:37:373
3214099130,10cyclictest9107-21sshd09:55:163
3213999137,5cyclictest0-21swapper/211:45:202
3213999131,11cyclictest0-21swapper/211:40:212
3213999131,11cyclictest0-21swapper/211:36:142
3213999130,4cyclictest0-21swapper/212:06:062
3213999130,12cyclictest24564-21bash09:49:002
3213999130,12cyclictest22804-21bash11:12:232
3213999130,12cyclictest22804-21bash11:12:232
3213999130,12cyclictest22129-21id10:34:562
3213999130,12cyclictest18413-21sshd11:57:292
3213999130,12cyclictest0-21swapper/212:13:412
3213999130,12cyclictest0-21swapper/212:00:262
3213999130,12cyclictest0-21swapper/211:16:282
3213999130,12cyclictest0-21swapper/211:16:282
3213999130,12cyclictest0-21swapper/211:06:152
3213999130,12cyclictest0-21swapper/211:02:572
3213999130,12cyclictest0-21swapper/210:05:182
3213899139,3cyclictest0-21swapper/112:14:541
3213899139,1cyclictest0-21swapper/110:01:121
3213899139,1cyclictest0-21swapper/109:15:531
3213899138,2cyclictest25927-21sshd11:25:151
3213899137,5cyclictest0-21swapper/110:11:321
3213899131,9cyclictest114850irq/25-eth009:41:131
3213899131,1cyclictest26982-21sshd11:04:421
3213899131,1cyclictest0-21swapper/109:36:321
3213899131,11cyclictest0-21swapper/111:11:081
3213899131,11cyclictest0-21swapper/111:11:081
3213899130,4cyclictest21200-21sshd11:37:021
3213899130,2cyclictest6823-21sshd12:25:051
3213899130,12cyclictest15103-21sshd09:22:221
3213899130,12cyclictest0-21swapper/112:35:401
3213899130,12cyclictest0-21swapper/112:32:371
3213899130,12cyclictest0-21swapper/111:46:511
3213899130,12cyclictest0-21swapper/109:11:001
3213899130,10cyclictest31229-21sshd09:33:081
3213799132,10cyclictest0-21swapper/008:15:130
153792130,0sleep20-21swapper/210:13:052
3214099129,2cyclictest0-21swapper/310:12:163
3214099129,2cyclictest0-21swapper/309:30:073
3214099128,3cyclictest0-21swapper/312:25:573
3214099128,2cyclictest21519-21sshd10:55:313
3214099121,10cyclictest0-21swapper/311:11:183
3214099121,10cyclictest0-21swapper/311:11:183
3214099121,10cyclictest0-21swapper/310:15:173
3214099120,1cyclictest0-21swapper/309:10:133
3214099120,12cyclictest0-21swapper/311:40:093
3214099120,11cyclictest0-21swapper/309:21:203
3213999129,2cyclictest0-21swapper/211:20:332
3213999129,2cyclictest0-21swapper/209:25:152
3213999129,2cyclictest0-21swapper/209:11:132
3213999129,2cyclictest0-21swapper/207:30:132
3213999121,10cyclictest0-21swapper/212:30:212
3213999120,3cyclictest0-21swapper/209:21:082
3213999120,11cyclictest4598-21sshd09:50:262
3213999120,11cyclictest0-21swapper/209:31:172
3213899129,2cyclictest0-21swapper/109:25:121
3213899129,2cyclictest0-21swapper/108:15:141
3213899128,3cyclictest0-21swapper/111:16:481
3213899128,3cyclictest0-21swapper/111:16:481
3213899121,9cyclictest114850irq/25-eth012:15:171
3213899121,2cyclictest0-21swapper/112:03:041
32138991211,1cyclictest9455-21kworker/1:210:25:381
3213899121,10cyclictest0-21swapper/110:50:221
32138991210,1cyclictest23660-21sshd10:30:491
3213799129,2cyclictest0-21swapper/008:10:150
3213799121,7cyclictest0-21swapper/009:00:110
3213799120,11cyclictest0-21swapper/008:25:160
3214099110,10cyclictest0-21swapper/308:50:173
3214099110,10cyclictest0-21swapper/308:45:153
3214099110,10cyclictest0-21swapper/308:17:223
3213999111,1cyclictest0-21swapper/207:55:142
32139991110,1cyclictest0-21swapper/207:42:122
3213999110,11cyclictest0-21swapper/208:35:242
3213999110,10cyclictest2225-21diskstats07:15:012
3213999110,10cyclictest0-21swapper/209:55:022
3213999110,10cyclictest0-21swapper/209:00:142
3213899119,1cyclictest0-21swapper/109:00:131
3213899118,2cyclictest0-21swapper/108:50:151
3213899118,2cyclictest0-21swapper/107:30:121
3213899110,11cyclictest0-21swapper/108:30:331
3213899110,11cyclictest0-21swapper/107:39:191
3213899110,11cyclictest0-21swapper/107:19:481
3213799119,1cyclictest0-21swapper/007:35:200
3213799113,7cyclictest2804-21munin-node07:15:180
3213799113,4cyclictest0-21swapper/008:35:150
32137991110,1cyclictest0-21swapper/007:23:410
32137991110,1cyclictest0-21swapper/007:11:290
3213799110,2cyclictest23084-21tune2fs09:05:210
3213799110,1cyclictest0-21swapper/008:09:060
3213799110,10cyclictest0-21swapper/008:40:130
3213799110,10cyclictest0-21swapper/007:40:140
3214099109,1cyclictest0-21swapper/308:33:183
3214099109,1cyclictest0-21swapper/307:12:193
3214099108,1cyclictest0-21swapper/307:40:153
3214099107,2cyclictest0-21swapper/307:30:123
3214099100,9cyclictest23041-21fschecks_time09:05:223
3214099100,9cyclictest0-21swapper/307:55:143
3214099100,1cyclictest0-21swapper/308:23:313
3214099100,1cyclictest0-21swapper/308:05:223
3214099100,10cyclictest0-21swapper/308:01:063
3213999108,1cyclictest0-21swapper/208:45:152
3213999108,1cyclictest0-21swapper/208:00:152
3213999100,1cyclictest0-21swapper/208:15:132
3213999100,1cyclictest0-21swapper/207:45:562
3213999100,10cyclictest0-21swapper/208:44:002
3213999100,10cyclictest0-21swapper/208:14:012
3213999100,10cyclictest0-21swapper/207:54:382
3213899109,1cyclictest0-21swapper/108:08:231
3213899109,1cyclictest0-21swapper/108:01:461
3213899109,1cyclictest0-21swapper/107:52:551
3213899109,1cyclictest0-21swapper/107:46:111
3213899108,1cyclictest8277-21perl07:25:151
3213899107,2cyclictest0-21swapper/109:05:271
3213899100,9cyclictest11333-21perl08:40:141
3213899100,9cyclictest0-21swapper/108:25:141
3213899100,9cyclictest0-21swapper/107:20:131
3213899100,1cyclictest0-21swapper/108:46:071
3213899100,1cyclictest0-21swapper/108:20:161
3213899100,10cyclictest0-21swapper/107:44:261
3213899100,10cyclictest0-21swapper/107:12:501
3213799109,1cyclictest0-21swapper/008:55:130
3213799104,3cyclictest0-21swapper/008:50:160
3213799100,9cyclictest0-21swapper/008:45:110
3213799100,1cyclictest6285-21diskstats08:30:130
3213799100,1cyclictest0-21swapper/008:22:140
3213799100,10cyclictest0-21swapper/007:49:420
321409998,1cyclictest0-21swapper/307:24:103
321409997,1cyclictest4243-21perl08:25:143
321409990,9cyclictest0-21swapper/308:35:233
321409990,9cyclictest0-21swapper/307:15:013
321409990,1cyclictest0-21swapper/307:25:153
321399998,1cyclictest0-21swapper/208:58:042
321399997,1cyclictest0-21swapper/208:25:142
321399990,9cyclictest0-21swapper/208:50:152
321399990,9cyclictest0-21swapper/208:21:482
321379998,1cyclictest0-21swapper/008:01:400
321379998,1cyclictest0-21swapper/007:56:400
321379992,6cyclictest0-21swapper/007:25:110
321379990,9cyclictest0-21swapper/007:50:140
321409987,1cyclictest0-21swapper/308:57:073
321409987,1cyclictest0-21swapper/308:10:593
321399988,0cyclictest5949-21conntrack07:20:132
321399988,0cyclictest0-21swapper/209:05:272
321399987,1cyclictest0-21swapper/208:30:452
321399981,6cyclictest0-21swapper/208:05:192
321399980,5cyclictest0-21swapper/207:36:192
321389987,1cyclictest0-21swapper/107:56:461
321389981,6cyclictest0-21swapper/108:35:191
321379982,5cyclictest0-21swapper/007:30:100
321409977,0cyclictest0-21swapper/309:00:133
321409975,1cyclictest19331-21apache_processe07:50:113
321409975,1cyclictest17314-21cat07:45:133
321409975,1cyclictest13147-21if_eth107:35:233
321409975,1cyclictest11490-21kernelversion08:40:183
321399970,6cyclictest0-21swapper/207:10:192
321389970,5cyclictest0-21swapper/108:55:121
321399960,5cyclictest0-21swapper/207:25:112
321389961,4cyclictest0-21swapper/108:10:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional