You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-08-05 - 06:38

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #1, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack1slot0.osadl.org (updated Thu Aug 05, 2021 00:43:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
326332830,1sleep2820-21dbus-daemon22:19:222
193902630,0sleep20-21swapper/223:09:312
296222610,0sleep20-21swapper/223:38:232
26592610,0sleep10-21swapper/121:27:491
2892026036,7sleep30-21swapper/319:07:523
23772600,1sleep3401ktimersoftd/322:23:013
23772600,1sleep3401ktimersoftd/322:23:003
110492600,0sleep20-21swapper/200:03:562
296472590,1sleep1241ktimersoftd/123:55:251
2902825734,7sleep10-21swapper/119:09:071
135372570,0sleep20-21swapper/223:53:462
265042560,1sleep026503-21grep00:38:310
246312560,0sleep20-21swapper/223:58:272
141382560,1sleep314139-21grep23:26:153
215252550,0sleep021521-21sshd22:38:520
132942550,0sleep10-21swapper/122:17:171
249152540,0sleep00-21swapper/022:28:510
2883825241,7sleep20-21swapper/219:06:542
30992510,1sleep23092-21bash00:10:002
157842510,1sleep2321ktimersoftd/222:48:322
14362510,0sleep30-21swapper/322:40:113
2877724837,7sleep00-21swapper/019:06:080
267622460,0sleep20-21swapper/223:44:572
196932450,2sleep22922199cyclictest21:40:122
98912250,0sleep00-21swapper/023:05:030
139662240,0sleep00-21swapper/022:55:140
312812190,0sleep20-21swapper/223:03:532
192552190,0sleep20-21swapper/221:57:202
23162180,0sleep02313-21lspci00:13:260
211652170,1sleep3401ktimersoftd/300:34:133
100082160,0sleep00-21swapper/021:21:120
81142150,0sleep20-21swapper/219:35:172
292802150,0sleep20-21swapper/200:27:432
261632150,0sleep10-21swapper/123:03:211
67662140,0sleep20-21swapper/221:24:382
2922199140,9cyclictest10863-21sshd21:32:182
2922199140,1cyclictest1-21systemd22:24:002
2922199140,1cyclictest1-21systemd22:23:592
2922199140,10cyclictest32247-21sshd23:21:132
251042140,0sleep30-21swapper/300:15:583
2922199130,9cyclictest16855-21sshd23:47:122
2922199130,1cyclictest24876-21sshd22:11:402
82262120,0sleep30-21swapper/319:35:193
29228991210,1cyclictest18309-21sshd00:11:373
2922899120,1cyclictest4160-21sshd23:07:503
2922199125,4cyclictest0-21swapper/222:01:142
2922199123,4cyclictest10792-21lspci22:27:192
2922199120,2cyclictest820-21dbus-daemon21:49:502
2922199120,2cyclictest12597-21rm22:51:322
2922199120,1cyclictest9085-21cp22:30:292
2922199120,1cyclictest4447-21bash21:52:172
2922199120,1cyclictest30068-21chrt20:31:002
2922199120,1cyclictest1-21systemd00:16:082
2922199120,1cyclictest10695-21sshd22:06:432
2921599129,2cyclictest27423-21sshd21:23:151
29215991211,0cyclictest6827-21lspci00:25:101
29215991210,1cyclictest8553-21grepconf.sh22:30:251
2921599120,11cyclictest14976-21sshd00:18:371
2921599120,10cyclictest27699-21kworker/1:022:54:421
2921599120,0cyclictest0-21swapper/121:10:401
2920899120,1cyclictest9995-21sshd21:59:500
233662120,0sleep10-21swapper/121:54:231
2922899119,1cyclictest0-21swapper/323:36:533
2922899110,10cyclictest9578-21bash21:24:593
2922899110,10cyclictest0-21swapper/300:35:403
2922899110,10cyclictest0-21swapper/300:07:063
2922199110,2cyclictest11961-21tty21:25:142
2922199110,1cyclictest929-21gdbus23:16:512
2922199110,1cyclictest801-21systemd-logind21:12:192
2922199110,1cyclictest3897-21cp21:35:042
2922199110,1cyclictest32105-21bash00:35:242
2922199110,1cyclictest2725-21bash23:11:082
2922199110,1cyclictest23688-21consoletype22:35:342
2922199110,1cyclictest21564-21gdbus21:15:362
2922199110,1cyclictest20750-21tty23:30:282
2922199110,1cyclictest18352-21sshd00:30:112
2922199110,1cyclictest1-21systemd22:56:522
2922199110,1cyclictest0-21swapper/223:27:352
2922199110,1cyclictest0-21swapper/200:10:562
2922199110,10cyclictest20876-21sshd00:22:582
2922199110,10cyclictest20517-21sshd22:42:072
2921599119,1cyclictest18299-21bash23:33:441
2921599119,1cyclictest10115-21cpuspeed21:05:111
2921599119,1cyclictest0-21swapper/121:36:011
2921599119,1cyclictest0-21swapper/121:15:151
2921599111,3cyclictest131rcu_sched22:01:191
29215991110,0cyclictest0-21swapper/121:57:191
29215991110,0cyclictest0-21swapper/100:03:351
2921599110,10cyclictest0-21swapper/123:53:351
2921599110,10cyclictest0-21swapper/123:38:141
2921599110,10cyclictest0-21swapper/121:44:251
2921599110,10cyclictest0-21swapper/100:38:281
2921599110,10cyclictest0-21swapper/100:21:591
2920899119,1cyclictest0-21swapper/000:29:310
2920899110,10cyclictest23332-21sshd00:01:440
2920899110,10cyclictest0-21swapper/023:44:380
2920899110,10cyclictest0-21swapper/019:55:210
2922899109,0cyclictest5828-21lspci21:52:253
2922899109,0cyclictest0-21swapper/323:34:393
2922899109,0cyclictest0-21swapper/322:55:153
2922899109,0cyclictest0-21swapper/322:26:213
2922899109,0cyclictest0-21swapper/322:18:163
2922899109,0cyclictest0-21swapper/319:50:223
2922899100,9cyclictest0-21swapper/321:25:193
2922899100,9cyclictest0-21swapper/300:28:343
2922899100,0cyclictest0-21swapper/320:34:573
2922199100,9cyclictest0-21swapper/219:56:162
2922199100,1cyclictest18316-21users20:00:352
2922199100,1cyclictest13702-21sed19:50:172
2922199100,0cyclictest0-21swapper/220:55:032
2921599109,0cyclictest0-21swapper/119:15:521
2921599100,9cyclictest0-21swapper/123:48:451
2921599100,9cyclictest0-21swapper/122:43:011
2921599100,1cyclictest9136-21bash21:32:071
2921599100,1cyclictest20993-21kworker/1:223:15:011
2921599100,0cyclictest16721-21sshd22:55:271
2921599100,0cyclictest0-21swapper/123:08:021
2921599100,0cyclictest0-21swapper/122:37:421
2920899109,0cyclictest30862-21sshd23:24:370
2920899109,0cyclictest0-21swapper/023:55:270
2920899109,0cyclictest0-21swapper/023:11:220
2920899109,0cyclictest0-21swapper/023:02:100
2920899109,0cyclictest0-21swapper/022:06:560
2920899109,0cyclictest0-21swapper/021:34:590
2920899109,0cyclictest0-21swapper/020:00:220
2920899108,1cyclictest27521-21lspci21:54:500
292289998,0cyclictest0-21swapper/323:58:123
292289998,0cyclictest0-21swapper/323:10:243
292289998,0cyclictest0-21swapper/323:00:043
292289998,0cyclictest0-21swapper/322:07:473
292219998,0cyclictest0-21swapper/219:30:162
292219993,5cyclictest1398-21snmpd20:13:572
292219991,1cyclictest5883-21systemd-cgroups20:54:552
292219990,1cyclictest31106-21latency_hist19:15:002
292219990,1cyclictest1398-21snmpd20:21:032
292159998,0cyclictest31726-21sshd22:22:411
292159998,0cyclictest31726-21sshd22:22:401
292159998,0cyclictest0-21swapper/123:12:061
292159998,0cyclictest0-21swapper/119:33:381
292159997,1cyclictest28518-21bash23:41:381
292159997,1cyclictest25-21ksoftirqd/122:11:371
292159995,1cyclictest25-21ksoftirqd/122:26:571
292159990,8cyclictest15002-21kworker/1:300:13:121
292159990,1cyclictest131rcu_sched21:46:311
292089998,0cyclictest0-21swapper/023:15:230
292089998,0cyclictest0-21swapper/022:21:580
292089998,0cyclictest0-21swapper/022:21:570
292089997,1cyclictest820-21dbus-daemon22:30:500
292089997,1cyclictest3326-21lspci23:45:460
292089993,3cyclictest1397-21snmpd19:20:000
22072290,0sleep30-21swapper/300:01:363
16870290,2sleep32922899cyclictest22:14:193
292289987,0cyclictest0-21swapper/320:39:563
292289986,1cyclictest7593-21lspci21:45:453
292289986,1cyclictest15234-21lspci23:40:183
292289980,7cyclictest31268-21sshd21:41:253
292289980,7cyclictest29203-21sshd23:48:333
292289980,7cyclictest16865-21sshd00:22:313
292289980,7cyclictest16596-21sshd21:57:033
292219987,0cyclictest0-21swapper/220:40:202
292219983,4cyclictest1398-21snmpd21:09:522
292219983,4cyclictest1398-21snmpd20:48:092
292219983,4cyclictest1398-21snmpd20:06:262
292219983,4cyclictest1398-21snmpd19:17:402
292219983,4cyclictest1397-21snmpd20:15:402
292219983,4cyclictest1397-21snmpd19:40:322
292219983,4cyclictest1397-21snmpd19:24:542
292219982,5cyclictest319952sleep220:35:362
292219982,5cyclictest1398-21snmpd21:00:062
292219982,1cyclictest34162sleep219:25:132
292219982,1cyclictest11442-21expr19:45:152
292159987,0cyclictest0-21swapper/122:07:591
292159987,0cyclictest0-21swapper/120:08:011
292159986,1cyclictest820-21dbus-daemon23:21:571
292159980,7cyclictest4158-21kworker/1:200:30:271
292159980,7cyclictest28108-21sshd00:05:391
292089987,0cyclictest0-21swapper/023:29:550
292089987,0cyclictest0-21swapper/022:12:550
292089987,0cyclictest0-21swapper/019:48:530
292089986,1cyclictest9937-21lspci00:32:520
292089986,1cyclictest25628-21rm00:23:330
292089980,7cyclictest7075-21sshd21:13:490
292089980,7cyclictest3823-21sshd21:41:570
292089980,7cyclictest28697-21sshd22:53:190
292089980,7cyclictest20042-21sshd22:42:050
292289976,0cyclictest0-21swapper/321:30:093
292289975,1cyclictest3770-21lspci21:35:033
292289970,6cyclictest8370-21sshd22:37:263
292289970,6cyclictest3201-21sshd22:50:313
292289970,6cyclictest29064-21sshd22:46:283
292289970,6cyclictest28468-21sshd21:12:333
292289970,6cyclictest26485-21sshd23:17:123
292289970,6cyclictest26335-21sshd20:22:523
292289970,6cyclictest25439-21sshd22:32:163
292289970,6cyclictest20658-21sshd23:20:043
292289970,6cyclictest14842-21sshd23:50:213
292289970,6cyclictest13328-21sshd22:00:113
292289970,6cyclictest1147-21sshd21:16:413
292219970,2cyclictest26842-21cat20:25:122
292159970,6cyclictest2473-21sshd23:25:001
292159970,6cyclictest12935-21kworker/1:422:45:061
292159970,6cyclictest0-21swapper/120:33:121
292159970,6cyclictest0-21swapper/119:46:331
292159970,6cyclictest0-21swapper/119:26:121
292089976,1cyclictest12499-21lspci21:25:160
292089976,0cyclictest0-21swapper/022:15:000
292089976,0cyclictest0-21swapper/020:24:050
292089976,0cyclictest0-21swapper/020:09:330
292089975,1cyclictest3769-21lspci21:35:020
292089975,1cyclictest24896-21lspci22:45:590
292089970,6cyclictest7163-21sshd21:17:200
292089970,6cyclictest7034-21sshd21:45:400
292089970,6cyclictest4458-21kworker/0:123:35:220
292089970,6cyclictest21991-21sshd22:01:010
292089970,6cyclictest21369-21sshd00:05:020
292089970,6cyclictest20512-21sshd23:30:260
292089970,6cyclictest16478-21sshd00:15:060
292089970,6cyclictest15012-21sshd23:50:210
292289963,2cyclictest678-21sshd19:15:263
292289963,2cyclictest1398-21snmpd20:42:013
292289963,2cyclictest1398-21snmpd20:15:473
292289963,2cyclictest1398-21snmpd20:01:023
292289963,2cyclictest1398-21snmpd19:33:343
292289963,2cyclictest1398-21snmpd19:27:473
292289963,2cyclictest1398-21snmpd19:10:243
292289963,2cyclictest1397-21snmpd20:51:443
292159965,0cyclictest0-21swapper/119:53:001
292159964,1cyclictest29399-21sshd19:10:031
292159964,1cyclictest1219-21runrttasks20:38:271
292159963,2cyclictest1398-21snmpd21:02:061
292159963,2cyclictest1398-21snmpd19:36:251
292159961,1cyclictest131rcu_sched20:19:541
292159960,5cyclictest3662-21sshd20:46:521
292159960,0cyclictest0-21swapper/120:22:331
292089963,2cyclictest1398-21snmpd19:32:370
292089963,2cyclictest1397-21snmpd21:02:480
292089963,2cyclictest1397-21snmpd20:58:190
292089963,2cyclictest1397-21snmpd19:50:470
292089962,3cyclictest833-21polkitd20:45:000
292289953,1cyclictest27106-21hddtemp_smartct20:25:143
292289953,1cyclictest19588-21missed_timers20:05:143
292289953,1cyclictest1398-21snmpd21:00:183
292289953,1cyclictest1398-21snmpd20:55:063
292289953,1cyclictest1398-21snmpd19:55:443
292289953,1cyclictest1398-21snmpd19:21:303
292289953,1cyclictest1397-21snmpd21:05:043
292289953,1cyclictest1397-21snmpd20:45:383
292289953,1cyclictest1397-21snmpd20:10:253
292289953,1cyclictest1397-21snmpd19:40:073
292289953,1cyclictest11396-21cstates19:45:163
292159953,1cyclictest6557-21cat20:55:161
292159953,1cyclictest4617-21cat20:50:171
292159953,1cyclictest20993-21cpuspeed20:10:111
292159953,1cyclictest1398-21snmpd20:41:121
292159953,1cyclictest1398-21snmpd19:40:161
292159953,1cyclictest1397-21snmpd20:25:201
292159953,1cyclictest1397-21snmpd19:55:011
292159953,1cyclictest1397-21snmpd19:20:361
292159950,1cyclictest17044-21/usr/sbin/munin20:00:261
292089953,1cyclictest7726-21cat19:35:140
292089953,1cyclictest29890-21timerwakeupswit20:30:320
292089953,1cyclictest1398-21snmpd21:05:270
292089953,1cyclictest1398-21snmpd20:52:030
292089953,1cyclictest1398-21snmpd20:35:130
292089953,1cyclictest1398-21snmpd20:25:050
292089953,1cyclictest1398-21snmpd19:10:540
292089953,1cyclictest1397-21snmpd20:16:140
292089953,1cyclictest1397-21snmpd20:11:490
292089952,2cyclictest3738-21cat19:25:140
292089952,2cyclictest2617-21unixbench_multi19:20:210
292089952,2cyclictest2397-21expr20:45:150
292089952,2cyclictest10131-21wc19:40:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional