You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2022-01-17 - 02:44

Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #4, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack4slot0.osadl.org (updated Mon Jan 17, 2022 00:44:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
395721080,0sleep33912-21sshd00:37:283
290662910,3sleep1628999cyclictest23:00:191
279072830,1sleep30-21swapper/322:36:483
279072830,1sleep30-21swapper/322:36:483
251822810,0sleep10-21swapper/122:13:131
312102750,0sleep10-21swapper/100:13:421
28442740,0sleep10-21swapper/123:34:141
251332740,0sleep20-21swapper/221:33:142
249672740,0sleep10-21swapper/123:46:241
222942730,2sleep0628499cyclictest22:56:130
173102730,0sleep00-21swapper/000:25:350
137112710,1sleep213699-21sshd22:32:022
66162700,0sleep00-21swapper/021:24:470
183672700,1sleep318360-21sshd00:02:253
17112700,0sleep00-21swapper/021:34:080
98622690,1sleep29859-21sshd21:55:032
53772690,0sleep10-21swapper/121:57:531
280672690,1sleep328051-21sshd22:13:313
274222680,0sleep00-21swapper/020:00:150
249872680,1sleep224983-21lspci23:33:102
138932680,0sleep30-21swapper/322:32:033
309992670,0sleep00-21swapper/023:53:410
279602660,0sleep00-21swapper/022:26:470
169362650,0sleep20-21swapper/222:49:022
114182590,2sleep3629499cyclictest21:48:293
576525727,9sleep00-21swapper/019:07:040
259642570,0sleep20-21swapper/223:20:042
324602560,2sleep1628999cyclictest21:27:241
575024429,9sleep20-21swapper/219:06:532
563324328,9sleep10-21swapper/119:05:451
564724227,9sleep30-21swapper/319:05:553
160052400,1sleep016004-21bash23:15:410
177202350,0sleep1231rcuc/100:09:011
68892280,0sleep06887-21sshd21:54:450
38262280,1sleep23814-21sshd22:44:212
236872280,0sleep20-21swapper/200:06:172
121732270,0sleep312170-21grepconf.sh23:25:183
214052260,0sleep021404-21bash23:32:480
163062260,1sleep216304-21sshd21:42:192
69492240,0sleep10-21swapper/100:31:091
64522240,0sleep10-21swapper/122:31:161
273202240,1sleep327317-21lspci22:00:133
124002240,1sleep312384-21sshd21:25:213
629499233,1cyclictest0-21swapper/323:24:573
320572230,1sleep332047-21lspci23:37:123
629499222,4cyclictest4312-21bash00:07:383
629499222,2cyclictest0-21swapper/320:36:153
629499220,4cyclictest0-21swapper/319:30:163
628999222,4cyclictest0-21swapper/100:18:041
628999222,2cyclictest8446-21lspci23:38:071
628999222,2cyclictest27092-21lspci21:13:411
628999222,2cyclictest16222-21sshd00:35:241
628999222,2cyclictest0-21swapper/120:50:341
628999220,4cyclictest0-21swapper/121:37:411
628999220,4cyclictest0-21swapper/121:30:511
629499211,2cyclictest32263-21lspci22:07:193
629499211,2cyclictest0-21swapper/320:03:033
629499211,1cyclictest9015-21lspci23:51:253
629499210,3cyclictest0-21swapper/322:24:143
629499210,3cyclictest0-21swapper/320:25:403
629499210,3cyclictest0-21swapper/319:59:243
629499210,2cyclictest1154-21lspci21:24:143
628999212,1cyclictest0-21swapper/120:49:161
628999210,4cyclictest0-21swapper/123:50:461
628999210,3cyclictest0-21swapper/123:08:081
628999210,2cyclictest0-21swapper/119:29:321
629499202,2cyclictest102550irq/24-eno123:15:153
629499200,3cyclictest0-21swapper/321:18:233
629499200,2cyclictest102550irq/24-eno121:40:573
629499200,2cyclictest0-21swapper/321:37:533
629499200,2cyclictest0-21swapper/319:52:083
629499200,2cyclictest0-21swapper/319:18:233
629499200,2cyclictest0-21swapper/300:20:023
628999200,3cyclictest0-21swapper/122:35:541
628999200,3cyclictest0-21swapper/122:35:541
628999200,2cyclictest0-21swapper/123:20:011
628999200,2cyclictest0-21swapper/122:48:581
628999200,2cyclictest0-21swapper/122:08:451
628999200,2cyclictest0-21swapper/121:22:231
628999200,2cyclictest0-21swapper/100:20:591
628999200,1cyclictest15107-21id22:25:271
628499200,4cyclictest0-21swapper/020:32:130
298042200,0sleep30-21swapper/323:33:393
102742200,0sleep20-21swapper/221:11:582
629499192,2cyclictest0-21swapper/300:12:403
6294991917,1cyclictest3661-21lspci21:51:063
629499190,4cyclictest0-21swapper/321:04:023
629499190,1cyclictest0-21swapper/323:59:013
629499190,1cyclictest0-21swapper/323:40:043
629499190,1cyclictest0-21swapper/322:43:203
629499190,1cyclictest0-21swapper/321:30:193
629499190,1cyclictest0-21swapper/320:43:043
629499190,1cyclictest0-21swapper/320:14:503
629499190,1cyclictest0-21swapper/319:43:203
629499190,1cyclictest0-21swapper/319:21:413
629499190,1cyclictest0-21swapper/319:13:553
629499190,1cyclictest0-21swapper/300:26:023
629499190,0cyclictest0-21swapper/322:50:423
628999190,4cyclictest0-21swapper/123:43:151
628999190,1cyclictest16434-21lspci00:28:491
628999190,1cyclictest0-21swapper/123:27:421
628999190,1cyclictest0-21swapper/122:15:091
628999190,1cyclictest0-21swapper/121:04:361
628999190,1cyclictest0-21swapper/120:39:431
628999190,1cyclictest0-21swapper/120:32:571
628999190,1cyclictest0-21swapper/120:21:091
628999190,1cyclictest0-21swapper/120:14:331
628999190,1cyclictest0-21swapper/120:09:531
628999190,1cyclictest0-21swapper/120:02:121
628999190,1cyclictest0-21swapper/119:52:241
628999190,1cyclictest0-21swapper/119:39:051
628999190,1cyclictest0-21swapper/119:33:171
628999190,1cyclictest0-21swapper/119:21:251
628999190,1cyclictest0-21swapper/100:01:271
628499190,4cyclictest0-21swapper/022:01:350
628499190,4cyclictest0-21swapper/019:42:520
628499190,1cyclictest0-21swapper/019:21:080
230592190,0sleep20-21swapper/222:03:032
629499181,1cyclictest10740-21lspci21:55:113
629499180,3cyclictest0-21swapper/320:20:063
629499180,2cyclictest3818-21grepconf.sh23:47:333
629499180,1cyclictest4689-21sshd22:27:443
629499180,1cyclictest11698-21lspci22:18:273
628999181,1cyclictest0-21swapper/122:50:251
628999180,3cyclictest0-21swapper/122:41:551
628999180,3cyclictest0-21swapper/119:19:441
628999180,2cyclictest8403-21lspci21:41:291
628999180,2cyclictest26439-21bash22:23:181
628999180,2cyclictest21029-21bash21:16:221
628999180,2cyclictest0-21swapper/120:58:591
628999180,1cyclictest0-21swapper/121:54:051
628999180,1cyclictest0-21swapper/119:59:001
628499180,3cyclictest0-21swapper/020:22:000
628499180,0cyclictest0-21swapper/022:31:560
628499180,0cyclictest0-21swapper/000:39:280
326772180,0sleep032678-21bash22:47:190
629499170,2cyclictest0-21swapper/323:02:073
629499170,2cyclictest0-21swapper/322:48:263
629499170,2cyclictest0-21swapper/321:12:043
629499170,2cyclictest0-21swapper/320:33:353
629499170,2cyclictest0-21swapper/320:18:313
628999170,2cyclictest6897-21munin-run20:30:001
628499170,1cyclictest0-21swapper/023:25:150
628499170,1cyclictest0-21swapper/000:13:160
230342170,0sleep00-21swapper/023:06:230
629499162,2cyclictest0-21swapper/320:47:573
6294991614,1cyclictest0-21swapper/322:57:363
629499160,1cyclictest0-21swapper/319:48:423
629499160,1cyclictest0-21swapper/319:35:053
629499160,1cyclictest0-21swapper/300:15:443
6293991613,2cyclictest28938-21sshd00:30:112
628999160,4cyclictest0-21swapper/120:40:141
628999160,1cyclictest0-21swapper/123:55:161
628999160,1cyclictest0-21swapper/123:15:091
628999160,1cyclictest0-21swapper/122:56:071
628999160,1cyclictest0-21swapper/120:19:131
628999160,1cyclictest0-21swapper/119:48:021
628499165,1cyclictest16244-21sshd00:21:570
628499165,10cyclictest10463-21sshd21:58:250
628499160,3cyclictest0-21swapper/020:14:150
628499160,3cyclictest0-21swapper/019:29:230
628499160,1cyclictest0-21swapper/019:19:190
628499160,1cyclictest0-21swapper/000:15:430
628499160,15cyclictest18647-21sshd21:42:340
628499160,0cyclictest0-21swapper/021:15:040
629499150,1cyclictest0-21swapper/323:11:233
629499150,1cyclictest0-21swapper/321:05:213
629499150,1cyclictest0-21swapper/300:31:523
629499150,10cyclictest18431-21sshd23:05:553
629399154,2cyclictest17633-21sshd23:39:042
629399153,10cyclictest20932-21sshd22:56:042
6293991513,1cyclictest2042-21bash22:14:112
6293991513,1cyclictest16929-21grepconf.sh23:48:542
629399150,3cyclictest31677-21sshd00:37:002
629399150,13cyclictest8969-21sshd23:01:322
628499155,5cyclictest593-21systemd-journal20:50:000
6284991513,1cyclictest25839-21nfsd23:35:210
628499150,6cyclictest23920-21kworker/0:122:17:270
628499150,1cyclictest0-21swapper/022:50:240
628499150,0cyclictest0-21swapper/020:51:430
628499150,0cyclictest0-21swapper/019:12:020
6293991412,1cyclictest6061-21bash21:24:432
6293991412,1cyclictest321ktimersoftd/219:15:172
6293991412,1cyclictest25776-21id23:53:092
629399141,1cyclictest30074-21sshd00:00:192
6293991411,2cyclictest28317-21in:imjournal22:38:112
6293991411,2cyclictest28317-21in:imjournal22:38:112
629399140,5cyclictest25143-21kworker/2:100:22:102
629399140,13cyclictest11947-21sshd00:25:062
628999141,2cyclictest6880-21lspci22:04:441
628999140,1cyclictest0-21swapper/121:48:181
628499141,4cyclictest0-21swapper/023:45:540
6284991411,2cyclictest21221-21sshd23:12:530
628499140,1cyclictest0-21swapper/019:37:030
628499140,1cyclictest0-21swapper/019:32:410
629399138,4cyclictest7544-21sshd22:18:012
629399138,4cyclictest32380-21kworker/2:223:58:022
629399133,1cyclictest17751-21bash22:29:062
629399132,9cyclictest321ktimersoftd/221:50:282
6293991311,1cyclictest12671-21bash21:18:502
6293991311,1cyclictest0-21swapper/223:41:192
629399131,10cyclictest1278-21runrttasks22:07:202
629399130,3cyclictest4292-21sshd22:51:022
629399130,2cyclictest18240-21sshd23:19:172
629399130,12cyclictest10108-21kworker/2:221:38:422
629399130,11cyclictest15507-21sshd23:05:352
628999139,1cyclictest0-21swapper/123:14:181
628499138,4cyclictest0-21swapper/022:09:190
628499133,1cyclictest9-21ksoftirqd/023:41:030
6284991311,1cyclictest21038-21sshd21:39:270
6284991311,1cyclictest16739-21bash23:22:260
6284991311,1cyclictest16244-21sshd00:08:340
628499130,1cyclictest12319-21sshd22:35:140
628499130,1cyclictest12319-21sshd22:35:140
628499130,1cyclictest0-21swapper/000:30:200
629499120,1cyclictest0-21swapper/320:05:173
629499120,10cyclictest15584-21df20:50:193
629399122,1cyclictest722-40kipmi021:25:192
629399120,2cyclictest26776-21sshd23:26:452
629399120,2cyclictest25835-21nfsd23:10:032
629399120,10cyclictest7125-21/usr/sbin/munin20:30:252
629399120,10cyclictest27366-21sshd22:20:082
629399120,10cyclictest23302-21cstates19:50:142
629399120,10cyclictest21413-21df19:45:112
629399120,10cyclictest20274-21proc_pri21:00:242
629399120,10cyclictest12058-21sshd00:15:032
629399120,10cyclictest1053-21sshd00:10:352
629399120,10cyclictest10433-21grep21:45:082
628999120,1cyclictest0-21swapper/121:05:201
628999120,1cyclictest0-21swapper/119:41:211
628999120,1cyclictest0-21swapper/119:13:171
628499128,3cyclictest27802-21sshd00:00:010
628499122,1cyclictest680-21bash22:20:380
6284991210,1cyclictest27443-21/usr/sbin/munin22:10:160
6284991210,1cyclictest20392-21id22:42:420
628499120,11cyclictest25840-21nfsd21:45:290
628499120,11cyclictest13737-21sshd23:55:170
628499120,0cyclictest1815-21grep23:00:460
628499120,0cyclictest0-21swapper/021:27:300
628499120,0cyclictest0-21swapper/019:50:480
282332120,0sleep00-21swapper/021:13:480
629499110,2cyclictest13095-21grep19:25:123
629499110,10cyclictest17709-21ipmitool20:55:153
629399118,2cyclictest9933-21ntp_states20:35:232
629399118,2cyclictest6521-21cat19:10:122
629399110,9cyclictest3836-21ntp_kernel_pll_20:20:192
629399110,9cyclictest15535-21cstates20:50:192
629399110,9cyclictest0-21swapper/220:30:002
629399110,9cyclictest0-21swapper/220:05:212
629399110,1cyclictest31364-21cpu20:10:132
629399110,10cyclictest27772-21memory20:00:202
629399110,10cyclictest13272-21smartctl19:25:122
629399110,10cyclictest12362-21timerwakeupswit20:40:222
629399110,10cyclictest0-21swapper/220:55:182
628499119,1cyclictest0-21swapper/020:25:140
628499110,10cyclictest25391-21cat19:55:120
628499110,10cyclictest19688-21diskstats21:00:170
628499110,0cyclictest0-21swapper/019:49:510
629399108,1cyclictest1338-21hddtemp_smartct20:15:152
629399108,1cyclictest0-21swapper/221:05:202
629399108,1cyclictest0-21swapper/219:20:132
629399100,9cyclictest0-21swapper/219:55:122
629399100,9cyclictest0-21swapper/219:30:162
629399100,1cyclictest0-21swapper/220:45:122
629399100,1cyclictest0-21swapper/219:40:132
629399100,1cyclictest0-21swapper/219:35:132
628499108,1cyclictest91002chrt20:35:100
628499108,1cyclictest11627-21interrupts20:40:150
628499108,1cyclictest0-21swapper/020:15:130
628499100,9cyclictest0-21swapper/021:05:200
628499100,9cyclictest0-21swapper/020:55:120
628499100,1cyclictest29435-21ls20:05:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional