You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-16 - 06:53

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #a, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackaslot0.osadl.org (updated Tue Aug 16, 2022 00:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2554421020,5sleep00-21swapper/000:22:170
59672900,1sleep32783399cyclictest23:09:433
250502860,0sleep339-21ksoftirqd/323:33:053
92542850,0sleep30-21swapper/323:03:503
32792850,0sleep13259-21sshd22:23:451
109022850,0sleep00-21swapper/023:37:440
167032590,6sleep230-21ksoftirqd/200:27:492
88042570,1sleep00-21swapper/021:27:270
72652570,2sleep37264-21sshd00:30:063
40022560,2sleep228-21rcuc/200:33:022
128182560,1sleep30-21swapper/300:01:543
2727125530,21sleep30-21swapper/319:06:213
207152550,0sleep20-21swapper/223:07:572
260602540,2sleep12783199cyclictest23:33:121
2542925428,22sleep10-21swapper/119:05:031
128632540,0sleep10-21swapper/121:18:361
107302540,0sleep00-21swapper/022:08:380
293132520,1sleep329286-21sshd00:22:413
217892520,1sleep221782-21sshd00:18:482
111972520,0sleep011198-21sshd23:10:110
77142510,0sleep00-21swapper/000:04:400
48332500,1sleep20-21swapper/223:21:552
290732500,1sleep329063-21sshd21:54:423
209032500,0sleep30-21swapper/321:38:043
168012500,0sleep20-21swapper/222:49:102
2738524923,22sleep00-21swapper/019:07:420
229282490,0sleep20-21swapper/221:54:062
48442480,1sleep30-21swapper/323:43:083
2751724825,19sleep20-21swapper/219:09:202
140522480,1sleep01535-21nfsd22:12:070
11512480,0sleep10-21swapper/123:54:431
6672470,0sleep228-21rcuc/200:13:312
274312470,0sleep30-21swapper/321:23:033
274312470,0sleep30-21swapper/321:23:033
181842470,1sleep10-21swapper/123:56:111
314182450,1sleep041ktimersoftd/000:13:210
157572450,0sleep10-21swapper/123:29:191
117582450,1sleep00-21swapper/023:16:210
84372440,2sleep12783199cyclictest23:09:571
83172440,1sleep220533-21diskmemload21:55:432
77702420,1sleep37769-21sshd21:33:393
294902410,0sleep00-21swapper/022:16:440
165662380,0sleep20-21swapper/200:24:412
64692360,0sleep06467-21bash22:57:170
47582220,1sleep20-21swapper/223:15:382
2783399222,1cyclictest835-21sshd23:27:543
27830992220,1cyclictest25227-21sshd23:30:120
2783199210,20cyclictest11351-21bash23:43:441
27830992119,1cyclictest26011-21sshd22:25:470
2783199209,3cyclictest0-21swapper/122:01:181
2783199206,2cyclictest21-21ksoftirqd/100:32:381
208362200,2sleep041ktimersoftd/023:56:260
2783099199,6cyclictest0-21swapper/022:39:030
27832991814,1cyclictest118850irq/25-eth000:08:522
83132170,0sleep10-21swapper/122:39:111
2783199175,1cyclictest21-21ksoftirqd/122:13:571
2783199173,10cyclictest21-21ksoftirqd/123:48:041
27831991711,1cyclictest21-21ksoftirqd/122:34:411
2783099179,1cyclictest0-21swapper/022:41:230
2783099178,8cyclictest8146-21sshd00:36:340
239122170,0sleep10-21swapper/100:38:091
218652170,1sleep021816-21sshd21:54:000
97412160,0sleep10-21swapper/123:19:221
85582160,1sleep08562-21sshd22:33:100
299832160,1sleep337-21rcuc/322:44:193
29322160,1sleep20-21swapper/222:14:122
2783299165,1cyclictest30-21ksoftirqd/221:47:402
2783299163,9cyclictest30-21ksoftirqd/222:38:322
27831991614,1cyclictest21-21ksoftirqd/123:22:241
27831991613,1cyclictest21-21ksoftirqd/123:02:221
2783099164,9cyclictest0-21swapper/022:46:590
2783099160,11cyclictest0-21swapper/023:05:520
194932160,1sleep11776-21runrttasks21:22:171
194932160,1sleep11776-21runrttasks21:22:171
155082160,1sleep20-21swapper/223:32:132
152512160,1sleep315239-21df_inode00:05:233
123842160,2sleep119-21rcuc/123:13:211
122172160,0sleep10-21swapper/121:40:231
31652150,2sleep119-21rcuc/100:10:291
309202150,1sleep230906-21sshd23:51:232
2783299159,3cyclictest30-21ksoftirqd/222:57:372
2783299159,3cyclictest0-21swapper/222:17:392
27832991513,1cyclictest30-21ksoftirqd/222:42:222
27832991513,1cyclictest30-21ksoftirqd/221:42:162
27832991513,1cyclictest30-21ksoftirqd/200:37:522
27832991512,2cyclictest30-21ksoftirqd/222:08:002
2783199157,3cyclictest19-21rcuc/123:00:011
2783199155,1cyclictest21-21ksoftirqd/122:17:061
27831991513,1cyclictest21-21ksoftirqd/121:11:191
2783199150,3cyclictest17589-21sshd00:27:531
2783099158,4cyclictest0-21swapper/022:50:180
2783099154,10cyclictest0-21swapper/022:22:270
2783099152,1cyclictest30385-21sshd23:48:260
27830991512,2cyclictest3835-21sshd22:01:380
2783099151,1cyclictest794-21sshd21:58:140
2783099151,1cyclictest0-21swapper/023:44:430
27830991510,1cyclictest0-21swapper/000:33:090
2783099150,3cyclictest0-21swapper/021:34:130
2783099150,2cyclictest0-21swapper/021:10:130
2783099150,12cyclictest0-21swapper/023:52:590
239332150,0sleep10-21swapper/122:10:001
109562150,1sleep10-21swapper/121:30:461
2783399149,4cyclictest0-21swapper/322:01:383
2783399149,2cyclictest0-21swapper/322:27:023
2783399148,1cyclictest0-21swapper/321:41:553
2783399141,12cyclictest27947-21sshd21:10:353
2783399141,12cyclictest0-21swapper/322:45:453
2783399140,5cyclictest0-21swapper/323:52:023
2783399140,13cyclictest25425-21sshd22:13:163
2783399140,13cyclictest23152-21sshd23:35:503
2783399140,13cyclictest15118-21sshd22:36:453
2783299148,3cyclictest24923-21sshd23:56:482
2783299144,9cyclictest30-21ksoftirqd/222:25:102
2783299143,10cyclictest30-21ksoftirqd/222:03:122
27832991412,1cyclictest30-21ksoftirqd/223:45:432
27832991412,1cyclictest30-21ksoftirqd/223:10:582
27832991412,1cyclictest30-21ksoftirqd/223:01:562
27832991412,1cyclictest30-21ksoftirqd/221:28:282
27832991412,1cyclictest30-21ksoftirqd/221:22:152
27832991412,1cyclictest30-21ksoftirqd/221:22:152
2783299141,1cyclictest28-21rcuc/221:16:072
2783299140,13cyclictest0-21swapper/221:11:122
2783299140,13cyclictest0-21swapper/221:05:142
2783199149,4cyclictest0-21swapper/122:45:581
2783199149,1cyclictest13218-21kworker/1:100:16:061
2783199148,1cyclictest0-21swapper/100:22:451
2783199144,1cyclictest21-21ksoftirqd/100:08:561
27831991413,1cyclictest21-21ksoftirqd/123:38:341
27831991413,1cyclictest21-21ksoftirqd/122:41:221
27831991413,1cyclictest21-21ksoftirqd/121:38:001
27831991413,1cyclictest21-21ksoftirqd/121:26:341
27831991410,1cyclictest21-21ksoftirqd/122:50:291
2783199140,5cyclictest0-21swapper/122:25:401
2783099148,3cyclictest0-21swapper/000:25:310
2783099147,3cyclictest0-21swapper/021:20:580
2783099147,3cyclictest0-21swapper/021:20:580
2783099141,12cyclictest0-21swapper/021:36:300
2783099140,10cyclictest0-21swapper/021:41:160
56962130,1sleep35690-21bash23:21:593
30682130,0sleep119-21rcuc/119:25:011
2783399139,2cyclictest0-21swapper/323:45:443
2783399138,2cyclictest23232-21sshd22:22:343
2783399133,1cyclictest13819-21id21:46:433
2783399131,1cyclictest0-21swapper/322:05:043
27833991311,1cyclictest980-21sshd23:12:153
27833991311,1cyclictest10126-21sshd00:36:463
2783399131,11cyclictest0-21swapper/322:59:403
2783399131,11cyclictest0-21swapper/321:18:013
2783399131,11cyclictest0-21swapper/300:00:013
2783399130,2cyclictest0-21swapper/323:18:163
2783399130,12cyclictest2110-21cp22:32:333
2783399130,12cyclictest19359-21sshd21:25:223
2783399130,12cyclictest0-21swapper/321:59:083
2783399130,10cyclictest0-21swapper/322:50:193
2783299139,2cyclictest30-21ksoftirqd/200:01:532
27832991311,1cyclictest30-21ksoftirqd/222:30:262
27832991311,1cyclictest30-21ksoftirqd/222:20:242
27832991311,1cyclictest30-21ksoftirqd/221:38:562
2783299130,9cyclictest0-21swapper/223:37:592
2783299130,1cyclictest0-21swapper/222:51:382
2783299130,12cyclictest17230-21sshd21:31:242
2783299130,11cyclictest0-21swapper/223:41:472
2783299130,10cyclictest0-21swapper/223:26:202
27831991312,1cyclictest21-21ksoftirqd/121:55:071
27831991312,1cyclictest21-21ksoftirqd/121:45:061
2783199130,4cyclictest21307-21sshd21:50:381
2783199130,12cyclictest0-21swapper/100:01:081
27830991311,2cyclictest0-21swapper/021:16:170
2783099131,11cyclictest0-21swapper/019:15:000
2783099130,3cyclictest0-21swapper/023:28:380
2783099130,1cyclictest4020-21sshd00:07:250
2783099130,12cyclictest18712-21bash23:01:330
2783099130,12cyclictest0-21swapper/021:45:500
2783099130,12cyclictest0-21swapper/000:15:230
2783099130,11cyclictest0-21swapper/023:20:370
24302130,0sleep30-21swapper/322:17:153
2783399129,2cyclictest0-21swapper/319:25:163
2783399129,2cyclictest0-21swapper/300:13:353
2783399120,1cyclictest0-21swapper/300:15:413
2783399120,11cyclictest11653-21sshd00:27:183
2783299120,11cyclictest0-21swapper/220:52:482
2783299120,11cyclictest0-21swapper/219:21:442
2783199129,2cyclictest0-21swapper/120:45:181
2783399119,1cyclictest0-21swapper/321:10:013
2783399110,6cyclictest0-21swapper/320:10:183
27832991110,1cyclictest0-21swapper/220:58:162
27832991110,1cyclictest0-21swapper/220:41:032
27832991110,1cyclictest0-21swapper/220:11:132
27832991110,1cyclictest0-21swapper/219:46:082
2783299110,1cyclictest0-21swapper/219:54:502
2783299110,1cyclictest0-21swapper/219:42:112
2783299110,11cyclictest0-21swapper/220:20:312
2783299110,11cyclictest0-21swapper/220:19:282
2783299110,11cyclictest0-21swapper/219:58:542
2783299110,11cyclictest0-21swapper/219:12:332
2783299110,10cyclictest0-21swapper/219:15:022
2783199110,10cyclictest0-21swapper/120:25:161
2783199110,10cyclictest0-21swapper/119:30:171
27830991110,1cyclictest0-21swapper/020:53:550
27830991110,1cyclictest0-21swapper/019:36:040
2783099110,11cyclictest0-21swapper/020:34:520
2783099110,11cyclictest0-21swapper/020:08:440
2783099110,11cyclictest0-21swapper/019:54:540
2783099110,11cyclictest0-21swapper/019:27:000
2783099110,10cyclictest0-21swapper/019:20:160
2783399109,1cyclictest0-21swapper/320:06:133
2783399108,1cyclictest0-21swapper/320:30:193
2783399100,9cyclictest30277-21grep19:15:013
2783399100,9cyclictest0-21swapper/321:00:173
2783299109,1cyclictest0-21swapper/221:02:202
2783299109,1cyclictest0-21swapper/220:46:522
2783299100,9cyclictest0-21swapper/220:25:152
2783199100,9cyclictest4027-21fw_conntrack19:25:161
2783199100,9cyclictest19126-21perl21:05:161
2783199100,1cyclictest30339-21open_files19:15:011
2783199100,10cyclictest0-21swapper/119:45:161
2783099109,1cyclictest0-21swapper/020:40:120
2783099109,1cyclictest0-21swapper/020:14:250
2783099105,4cyclictest0-21swapper/020:55:160
2783099100,9cyclictest0-21swapper/021:00:170
2783099100,6cyclictest0-21swapper/019:40:150
2783099100,1cyclictest30160-21fschecks_time20:20:160
2783099100,10cyclictest0-21swapper/021:05:110
2783099100,10cyclictest0-21swapper/020:29:450
278339998,1cyclictest0-21swapper/320:35:373
278339998,1cyclictest0-21swapper/320:01:413
278339998,1cyclictest0-21swapper/319:50:253
278339998,1cyclictest0-21swapper/319:41:353
278339998,1cyclictest0-21swapper/319:32:123
278339990,1cyclictest0-21swapper/319:56:143
278329998,1cyclictest0-21swapper/220:09:492
278329998,1cyclictest0-21swapper/219:36:222
278329998,1cyclictest0-21swapper/219:26:092
278329990,9cyclictest0-21swapper/219:30:162
278329990,2cyclictest0-21swapper/220:30:192
278319990,9cyclictest0-21swapper/119:50:171
278319990,1cyclictest0-21swapper/120:00:181
278309999,0cyclictest0-21swapper/020:19:530
278309991,5cyclictest0-21swapper/019:55:150
278309990,8cyclictest0-21swapper/019:45:330
278309990,7cyclictest0-21swapper/019:30:260
278309990,7cyclictest0-21swapper/019:11:460
278309990,6cyclictest0-21swapper/020:35:190
278339987,1cyclictest0-21swapper/320:48:243
278339986,1cyclictest30134-21fschecks_time20:20:163
278339986,1cyclictest1276-21diskstats19:20:163
278339981,6cyclictest0-21swapper/320:25:173
278339981,6cyclictest0-21swapper/319:35:243
278339981,6cyclictest0-21swapper/319:10:163
278329987,1cyclictest0-21swapper/220:36:182
278319988,0cyclictest0-21swapper/119:55:171
278319987,1cyclictest0-21swapper/120:40:201
278319980,7cyclictest0-21swapper/120:10:231
278319980,1cyclictest0-21swapper/119:13:581
278309980,6cyclictest0-21swapper/020:00:120
278339977,0cyclictest0-21swapper/320:15:163
278339975,1cyclictest7024-21fschecks_count20:40:203
278339972,3cyclictest0-21swapper/320:55:193
278339971,5cyclictest0-21swapper/320:50:153
278329970,2cyclictest0-21swapper/220:00:232
278319974,3cyclictest21-21ksoftirqd/121:00:221
278319973,4cyclictest21-21ksoftirqd/119:40:001
278319970,7cyclictest0-21swapper/120:20:101
278319970,6cyclictest0-21swapper/120:50:151
278319970,6cyclictest0-21swapper/120:15:221
278309970,4cyclictest0-21swapper/020:45:160
278339964,1cyclictest12747-21chrt19:45:053
278319964,1cyclictest21-21ksoftirqd/120:35:161
278319964,1cyclictest10390-21apache_accesses19:40:151
278319962,2cyclictest0-21swapper/120:05:221
278319961,4cyclictest0-21swapper/120:30:181
278319960,5cyclictest0-21swapper/120:55:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional