You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-05-12 - 14:16

Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #4, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack4slot0.osadl.org (updated Wed May 12, 2021 12:44:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1558621250,1sleep0101ktimersoftd/011:44:190
2449521150,0sleep00-21swapper/012:13:210
1119521050,0sleep111197-21grep11:51:511
144102820,0sleep00-21swapper/007:05:190
47452710,1sleep341-21ksoftirqd/310:00:523
8042700,1sleep20-21swapper/211:22:312
70942690,1sleep10-21swapper/110:21:231
259852690,1sleep00-21swapper/010:26:540
241942660,0sleep30-21swapper/311:20:073
67732610,1sleep233-21ksoftirqd/212:25:582
67732610,1sleep233-21ksoftirqd/212:25:582
1571325926,9sleep20-21swapper/207:09:472
1547725726,9sleep30-21swapper/307:07:433
139562510,0sleep00-21swapper/012:36:500
1395124624,15sleep10-21swapper/107:05:081
284522270,0sleep20-21swapper/209:18:362
171882270,0sleep0101ktimersoftd/009:25:120
273312260,0sleep00-21swapper/009:58:100
309072250,0sleep10-21swapper/109:29:081
171572250,2sleep317150-21lspci12:20:143
90882230,0sleep30-21swapper/312:35:243
178502230,0sleep00-21swapper/011:53:460
288532210,0sleep20-21swapper/210:54:262
250522210,14sleep3101050irq/24-eno111:37:583
94482200,0sleep20-21swapper/211:51:202
1585299198,1cyclictest0-21swapper/212:15:572
1585299198,1cyclictest0-21swapper/212:15:562
1585299190,18cyclictest4964-21sshd11:59:072
1584399180,16cyclictest29923-21sshd11:30:260
54712170,0sleep20-21swapper/211:41:222
1585399170,1cyclictest0-21swapper/307:50:163
15852991714,1cyclictest591-21systemd-journal12:23:402
15852991714,1cyclictest26327-21sshd11:38:202
1585299170,12cyclictest21177-21sshd12:38:572
1584999171,13cyclictest0-21swapper/111:03:281
13632170,3sleep11584999cyclictest10:19:581
15853991614,1cyclictest12737-21bash12:27:453
15853991614,1cyclictest12737-21bash12:27:443
15853991613,2cyclictest24875-21sshd09:36:593
15853991610,5cyclictest3166-21sshd09:10:273
1585299168,2cyclictest18525-21grep12:11:362
1585299168,2cyclictest1846-21sshd10:20:042
1585299168,1cyclictest23392-21grep10:06:252
1585299162,1cyclictest8279-21cp11:33:262
15852991611,4cyclictest5116-21bash09:21:092
15852991610,5cyclictest1442-21snmpd12:33:372
1585299160,9cyclictest0-21swapper/210:42:342
1585299160,1cyclictest4677-21bash10:56:282
1585299160,1cyclictest19353-21/usr/sbin/munin10:25:132
1585299160,1cyclictest15188-21lspci12:01:442
1585299160,1cyclictest0-21swapper/210:46:542
1585299160,1cyclictest0-21swapper/209:28:392
1585299160,15cyclictest31561-21sshd12:06:122
1585299160,15cyclictest11463-21lspci10:03:032
1585299160,14cyclictest21828-21timerwakeupswit11:10:202
1585299160,11cyclictest0-21swapper/209:36:042
1585299160,10cyclictest6277-21sensors10:30:192
1585299160,10cyclictest31447-21in:imjournal11:46:482
1585299160,10cyclictest2964-21sshd09:40:162
1585299160,10cyclictest19946-21bash09:55:322
15849991613,2cyclictest12795-21sshd12:01:021
1584999160,15cyclictest5264-21sshd09:40:351
288082150,0sleep10-21swapper/110:27:431
1585399153,2cyclictest0-21swapper/311:49:053
15853991512,2cyclictest30062-21bash10:45:383
1585399150,2cyclictest24669-21sshd09:17:153
1585299158,1cyclictest5524-21cp10:10:422
1585299158,1cyclictest0-21swapper/210:16:342
1585299158,1cyclictest0-21swapper/209:30:222
1585299158,1cyclictest0-21swapper/208:04:592
1585299157,2cyclictest102550irq/26-enp2s0-r07:10:132
1585299155,2cyclictest4240-21idleruntime-cro08:00:002
1585299150,9cyclictest25612-21bash10:35:322
1585299150,9cyclictest0-21swapper/211:06:012
1585299150,1cyclictest3149-21grep09:10:262
1585299150,1cyclictest12362-21sshd11:25:302
1585299150,1cyclictest12164-21sshd11:16:382
1585299150,1cyclictest0-21swapper/209:46:082
1585299150,1cyclictest0-21swapper/208:10:002
1585299150,1cyclictest0-21swapper/207:25:202
1585299150,14cyclictest6805-21sshd09:51:122
1585299150,10cyclictest23101-21sshd11:01:382
1584999158,6cyclictest4768-21sshd10:56:291
1584999158,6cyclictest28461-21sshd12:31:551
1584999157,7cyclictest24116-21sshd12:39:461
1584999154,10cyclictest26639-21nfsd09:55:471
15849991513,1cyclictest12696-21sshd10:40:541
1584399154,2cyclictest9916-21sed10:40:170
15843991511,2cyclictest5573-21sshd10:56:450
15843991511,2cyclictest2761-21hwlatdetect07:55:120
1585399144,1cyclictest31447-21in:imjournal11:34:163
1585399142,2cyclictest1443-21snmpd10:34:053
15853991411,2cyclictest41-21ksoftirqd/309:55:003
15853991411,2cyclictest1572-21bash12:15:433
15853991411,2cyclictest1572-21bash12:15:423
1585399140,2cyclictest591-21systemd-journal10:50:113
1585399140,1cyclictest0-21swapper/307:15:233
1585399140,13cyclictest0-21swapper/312:02:143
1585399140,13cyclictest0-21swapper/311:50:213
1585399140,12cyclictest22478-21sshd10:25:513
1585399140,10cyclictest4247-21sshd11:41:013
1585299147,1cyclictest0-21swapper/207:20:152
1585299147,1cyclictest0-21swapper/207:20:142
1585299140,9cyclictest26639-21nfsd08:20:452
1585299140,9cyclictest0-21swapper/209:05:312
1585299140,9cyclictest0-21swapper/209:00:342
1585299140,9cyclictest0-21swapper/207:15:002
1585299140,1cyclictest0-21swapper/207:35:302
1585299140,13cyclictest0-21swapper/208:55:372
1584999143,10cyclictest28634-21sshd10:08:111
15849991412,1cyclictest10072-21bash10:12:211
15849991411,2cyclictest23723-21munin-run11:20:011
1584999140,13cyclictest0-21swapper/111:35:141
1584999140,12cyclictest12755-21sshd09:53:281
1584399144,9cyclictest5862-21sshd10:10:510
1584399143,10cyclictest14315-21sshd09:24:160
1584399142,2cyclictest15559-21bash12:28:350
1584399142,2cyclictest15559-21bash12:28:350
15843991410,2cyclictest12349-21sshd11:25:290
1584399140,1cyclictest0-21swapper/012:00:120
1584399140,1cyclictest0-21swapper/007:20:310
1584399140,1cyclictest0-21swapper/007:20:300
1585399139,3cyclictest9160-21sshd09:41:583
1585399139,3cyclictest7758-21sshd10:11:313
1585399138,4cyclictest1377-21lldpd09:59:393
1585399138,3cyclictest17062-21cpuspeed_turbos11:00:113
1585399133,1cyclictest31836-21nscd10:42:173
1585399133,1cyclictest23207-21bash10:16:213
1585399132,1cyclictest21788-21kworker/3:212:05:243
1585399132,10cyclictest19216-21sshd12:11:483
15853991311,1cyclictest6941-21needreboot11:15:153
15853991311,1cyclictest0-21swapper/311:10:053
15853991311,1cyclictest0-21swapper/310:05:263
15853991310,2cyclictest9551-21kworker/3:111:26:273
1585399130,12cyclictest0-21swapper/310:22:083
1585399130,11cyclictest101050irq/24-eno109:25:253
1585399130,10cyclictest23634-21chrt08:45:193
1585399130,10cyclictest23634-21chrt08:45:193
1585299130,1cyclictest0-21swapper/208:40:212
1585299130,1cyclictest0-21swapper/208:35:442
1584999139,2cyclictest2265-21lspci09:20:181
1584999133,1cyclictest27454-21lspci11:56:121
1584999133,1cyclictest0-21swapper/112:09:381
1584999132,10cyclictest3492-21bash12:25:141
1584999132,10cyclictest3492-21bash12:25:131
15849991311,1cyclictest25500-21rm09:37:091
15849991311,1cyclictest21358-21sshd12:21:091
15849991311,1cyclictest0-21swapper/112:15:221
15849991311,1cyclictest0-21swapper/112:15:211
1584999130,2cyclictest3343-21sshd11:49:531
1584999130,12cyclictest1268-21runrttasks11:22:421
1584999130,11cyclictest0-21swapper/111:25:261
1584399132,10cyclictest6446-21bash11:15:120
15843991311,1cyclictest30472-21munin-plugin-st07:45:000
15843991311,1cyclictest25795-21sshd11:55:420
15843991311,1cyclictest0-21swapper/011:08:460
15843991311,1cyclictest0-21swapper/008:30:220
15843991310,2cyclictest26251-21tty09:47:100
1584399130,2cyclictest0-21swapper/009:33:250
1584399130,1cyclictest0-21swapper/008:40:160
1584399130,12cyclictest0-21swapper/007:50:180
1584399130,11cyclictest0-21swapper/011:10:130
1584399130,10cyclictest3694-21bash10:47:240
143602130,1sleep1231rcuc/110:50:241
1585399128,1cyclictest0-21swapper/312:33:193
1585399121,1cyclictest0-21swapper/307:10:183
15853991210,1cyclictest28718-21latency_hist09:00:003
15853991210,1cyclictest0-21swapper/309:21:493
1585399120,9cyclictest0-21swapper/308:05:233
1585399120,2cyclictest8216-21lspci10:57:333
1585399120,2cyclictest591-21systemd-journal11:58:153
1585399120,1cyclictest0-21swapper/311:05:523
1585399120,11cyclictest8287-21lspci09:31:513
1585399120,11cyclictest31470-21memory09:05:163
1585399120,10cyclictest3716-21users07:55:223
1585399120,10cyclictest29699-21ntp_states09:00:243
1585399120,10cyclictest23896-21/usr/sbin/munin10:35:163
1585399120,10cyclictest21012-21diskstats08:40:133
1585399120,10cyclictest19700-21/usr/sbin/munin09:45:143
1585399120,10cyclictest0-21swapper/307:35:183
1585299121,10cyclictest24880-21cpuspeed_turbos08:50:122
1585299120,9cyclictest0-21swapper/207:40:372
1585299120,11cyclictest0-21swapper/207:30:132
1585299120,10cyclictest833-21latency07:50:172
1584999129,2cyclictest28762-21grep10:36:261
1584999129,2cyclictest23550-21ntp_states08:45:191
1584999129,2cyclictest23550-21ntp_states08:45:181
1584999129,2cyclictest0-21swapper/109:18:201
1584999122,1cyclictest2082-21sshd09:30:121
1584999122,1cyclictest0-21swapper/111:11:071
1584999121,10cyclictest1851-21consoletype11:31:331
1584999120,2cyclictest4917-21wc10:30:121
1584999120,2cyclictest19007-21hddtemp_smartct08:35:171
1584999120,2cyclictest14225-21bash09:14:211
1584999120,1cyclictest26639-21nfsd10:00:281
1584999120,11cyclictest873-21sshd11:40:151
1584999120,11cyclictest22864-21sshd12:12:531
1584999120,11cyclictest0-21swapper/107:45:151
1584999120,10cyclictest20004-21gawk09:45:181
1584999120,10cyclictest0-21swapper/108:55:111
1584399129,2cyclictest7790-21sshd12:08:490
1584399129,2cyclictest17952-21grep11:35:530
1584399129,2cyclictest16404-21if_eno107:10:140
1584399129,1cyclictest0-21swapper/009:15:230
1584399122,1cyclictest718-40kipmi010:05:150
1584399122,1cyclictest2572-21bash09:50:020
15843991210,1cyclictest72322chrt10:39:410
15843991210,1cyclictest5273-21tty10:30:130
15843991210,1cyclictest26639-21nfsd10:50:500
15843991210,1cyclictest0-21swapper/010:02:220
1584399120,2cyclictest3572-21sshd12:34:110
1584399120,2cyclictest31447-21in:imjournal10:22:260
1584399120,2cyclictest3139-21sshd12:16:110
1584399120,2cyclictest3139-21sshd12:16:110
1584399120,2cyclictest29158-21bash11:47:570
1584399120,2cyclictest29011-21cpuspeed_turbos09:00:130
1584399120,2cyclictest2817-21sshd09:40:140
1584399120,2cyclictest24466-21cpu11:20:120
1584399120,2cyclictest19947-21/usr/sbin/munin09:35:220
1584399120,2cyclictest19160-21sshd11:00:290
1584399120,2cyclictest18318-21grep07:15:010
1584399120,10cyclictest6982-21cut08:05:180
1584399120,10cyclictest591-21systemd-journal10:18:100
1584399120,10cyclictest27472-21grep08:55:160
1584399120,10cyclictest18963-21sshd12:20:270
1584399120,10cyclictest14943-21df08:25:120
1584399120,10cyclictest13663-21timerandwakeup08:20:270
1584399120,10cyclictest1149-21sshd09:10:110
1584399120,10cyclictest0-21swapper/007:30:120
1585399119,1cyclictest0-21swapper/308:10:183
1585399110,2cyclictest22763-21expr07:25:123
1585399110,2cyclictest0-21swapper/308:30:123
1585399110,1cyclictest24782-21cpuspeed_turbos07:30:103
1585399110,1cyclictest0-21swapper/308:25:133
1585399110,1cyclictest0-21swapper/308:20:203
1585399110,10cyclictest5753-21users08:00:213
1585399110,10cyclictest31023-21hddtemp_smartct07:45:123
1585399110,10cyclictest25062-21df08:50:123
1585399110,10cyclictest11698-21expr08:15:213
1585299119,1cyclictest0-21swapper/208:25:132
1585299119,1cyclictest0-21swapper/208:10:122
1585299110,2cyclictest0-21swapper/208:17:082
1585299110,10cyclictest23023-21expr08:45:132
1585299110,10cyclictest23023-21expr08:45:122
1584999119,1cyclictest9692-21timerwakeupswit08:10:201
1584999119,1cyclictest0-21swapper/111:05:091
1584999119,1cyclictest0-21swapper/108:40:161
1584999119,1cyclictest0-21swapper/107:40:111
1584999118,2cyclictest27809-21sed07:35:211
1584999118,2cyclictest13776-21sed08:20:271
1584999111,1cyclictest0-21swapper/109:00:251
1584999110,9cyclictest0-21swapper/109:05:121
1584999110,2cyclictest22538-21/usr/sbin/munin07:25:111
1584999110,2cyclictest1497-21sed07:50:221
1584999110,1cyclictest6800-21df08:05:121
1584999110,1cyclictest2740-21hddtemp_smartct07:55:121
1584999110,1cyclictest18376-21smart_sda07:15:011
1584999110,1cyclictest0-21swapper/108:50:181
1584999110,1cyclictest0-21swapper/108:30:141
1584999110,1cyclictest0-21swapper/108:00:111
1584999110,1cyclictest0-21swapper/107:20:271
1584999110,1cyclictest0-21swapper/107:20:261
1584999110,10cyclictest26638-21nfsd10:45:021
1584999110,10cyclictest25165-21sed07:30:151
1584999110,10cyclictest16848-21ntp_states07:10:191
1584999110,10cyclictest15467-21ntp_states08:25:181
1584999110,10cyclictest11866-21users08:15:221
1584399119,1cyclictest0-21swapper/008:00:120
1584399118,2cyclictest31228-21forks09:05:120
1584399110,9cyclictest0-21swapper/007:45:200
1584399110,2cyclictest8816-21cstates08:10:110
1584399110,1cyclictest26699-21cpuspeed_turbos07:35:100
1584399110,1cyclictest0-21swapper/008:35:250
1584399110,10cyclictest22942-21expr08:45:130
1584399110,10cyclictest22942-21expr08:45:130
1584399110,10cyclictest0-21swapper/007:25:200
1585399108,1cyclictest0-21swapper/308:35:163
1585399108,1cyclictest0-21swapper/307:40:033
1585399108,1cyclictest0-21swapper/307:20:223
1585399108,1cyclictest0-21swapper/307:20:223
1585299108,1cyclictest0-21swapper/208:30:122
1585299108,1cyclictest0-21swapper/207:45:202
1584399108,1cyclictest0-21swapper/008:50:110
1584399100,9cyclictest0-21swapper/008:15:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional