You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2022-01-17 - 01:59

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #1, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack1slot0.osadl.org (updated Mon Jan 17, 2022 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2592521040,2sleep01518099cyclictest22:58:510
268732870,0sleep0111rcuc/023:07:250
81422620,0sleep10-21swapper/123:00:081
1467526251,7sleep20-21swapper/219:05:272
1467526251,7sleep20-21swapper/219:05:262
286292610,0sleep30-21swapper/321:19:223
257752610,0sleep30-21swapper/321:30:453
115812610,0sleep00-21swapper/021:32:210
321882600,0sleep30-21swapper/300:13:143
31282600,1sleep00-21swapper/000:13:310
280202590,1sleep1241ktimersoftd/121:59:231
63262580,0sleep30-21swapper/321:48:553
284252580,0sleep20-21swapper/221:19:212
225362580,0sleep10-21swapper/122:21:341
235672570,1sleep123561-21sshd23:38:211
1499125734,7sleep10-21swapper/119:08:551
1499125734,7sleep10-21swapper/119:08:541
136632570,0sleep30-21swapper/300:37:103
136632570,0sleep30-21swapper/300:37:103
133942570,1sleep213388-21bash22:52:002
127242570,0sleep20-21swapper/222:09:262
324312560,0sleep21219-21runrttasks00:35:562
324312560,0sleep21219-21runrttasks00:35:552
318612560,0sleep00-21swapper/023:30:320
318612560,0sleep00-21swapper/023:30:310
282432560,0sleep20-21swapper/200:27:032
274852560,0sleep20-21swapper/200:21:182
253092560,0sleep30-21swapper/322:38:513
205522560,0sleep10-21swapper/121:12:441
181552560,0sleep20-21swapper/221:49:592
169982560,0sleep00-21swapper/021:49:530
137532560,0sleep00-21swapper/023:43:100
92002550,0sleep10-21swapper/121:37:481
71682550,0sleep00-21swapper/023:00:020
295592550,0sleep10-21swapper/100:21:291
290732550,0sleep00-21swapper/000:07:140
26062550,0sleep30-21swapper/300:24:523
256182550,1sleep125-21ksoftirqd/119:35:131
1476225533,7sleep00-21swapper/019:06:120
1476225533,7sleep00-21swapper/019:06:120
112712540,0sleep20-21swapper/222:29:082
238722510,0sleep0101ktimersoftd/022:27:210
174872500,1sleep217463-21sshd23:23:412
1503125038,7sleep30-21swapper/319:09:213
1503125038,7sleep30-21swapper/319:09:203
306952490,0sleep30-21swapper/323:39:013
72612480,0sleep20-21swapper/223:31:142
72612480,0sleep20-21swapper/223:31:142
280362440,0sleep30-21swapper/300:04:213
152092440,0sleep00-21swapper/022:46:290
278942370,0sleep00-21swapper/022:41:530
108332240,0sleep310827-21lspci22:03:333
61442200,0sleep3391rcuc/322:14:303
314882170,1sleep331490-21bash00:15:573
218232170,0sleep20-21swapper/222:24:242
122222170,0sleep112168-21sshd22:34:531
122072170,0sleep10-21swapper/121:29:381
109422170,1sleep010943-21bash00:00:010
179022160,0sleep3391rcuc/321:35:403
123972160,0sleep00-21swapper/023:48:430
100392160,0sleep10-21swapper/123:05:531
48702150,0sleep00-21swapper/022:11:320
38992150,0sleep00-21swapper/000:39:110
38992150,0sleep00-21swapper/000:39:110
254832150,0sleep30-21swapper/321:59:093
1519399143,2cyclictest33-21ksoftirqd/223:18:512
1518099146,2cyclictest9-21ksoftirqd/023:15:400
1518099145,2cyclictest9-21ksoftirqd/022:08:310
1518099142,3cyclictest9-21ksoftirqd/023:11:310
300792130,1sleep01-21systemd00:18:440
15193991311,1cyclictest833-21polkitd21:31:192
1518099135,1cyclictest9-21ksoftirqd/022:24:270
1518099134,3cyclictest9-21ksoftirqd/000:30:450
1518099133,2cyclictest9-21ksoftirqd/021:13:140
1518099133,2cyclictest9-21ksoftirqd/000:25:030
1518099132,2cyclictest9-21ksoftirqd/000:02:400
15180991313,0cyclictest17352-21kworker/0:019:15:130
225132120,0sleep00-21swapper/021:41:460
221752120,0sleep10-21swapper/121:50:171
15199991210,1cyclictest12765-21lspci21:11:563
1519999120,1cyclictest5527-21lspci21:26:083
1519999120,1cyclictest10845-21sshd21:52:073
1519999120,11cyclictest820-21dbus-daemon22:20:033
1519399129,2cyclictest19159-21grep21:50:022
15185991210,1cyclictest9326-21bash21:17:361
15185991210,1cyclictest1171-21lspci21:48:281
1518599120,1cyclictest17672-21sshd21:41:201
1518599120,11cyclictest801-21systemd-logind23:22:551
1518599120,11cyclictest11799-21bash21:20:401
1518599120,11cyclictest0-21swapper/122:37:221
1518099126,3cyclictest9-21ksoftirqd/021:18:530
1518099124,3cyclictest9-21ksoftirqd/023:21:340
1518099123,3cyclictest9-21ksoftirqd/022:36:240
1518099123,2cyclictest9-21ksoftirqd/021:55:390
1518099122,2cyclictest9-21ksoftirqd/021:51:130
1518099121,1cyclictest9-21ksoftirqd/021:25:150
15180991211,0cyclictest17352-21kworker/0:019:25:280
1518099120,11cyclictest2389-21sshd22:34:000
188322110,1sleep3401ktimersoftd/322:41:043
15199991110,0cyclictest0-21swapper/322:53:303
1519999110,10cyclictest0-21swapper/323:17:513
1519399119,1cyclictest820-21dbus-daemon21:40:592
1519399119,1cyclictest3523-21sshd23:11:062
1519399111,1cyclictest33-21ksoftirqd/223:55:362
15193991110,0cyclictest3687-21lspci00:30:322
15193991110,0cyclictest33-21ksoftirqd/222:20:012
15193991110,0cyclictest0-21swapper/222:48:482
1519399110,8cyclictest4262-21kworker/2:322:01:162
1519399110,10cyclictest3991-21sshd22:59:472
1519399110,10cyclictest0-21swapper/221:56:212
1518599119,1cyclictest24628-21lspci22:41:351
1518599119,1cyclictest11477-21nfsd420:20:171
1518599110,1cyclictest32569-21sshd22:48:011
1518599110,10cyclictest12855-21consoletype23:43:061
1518099115,2cyclictest9-21ksoftirqd/023:27:200
1518099115,2cyclictest9-21ksoftirqd/022:50:140
1518099115,1cyclictest9-21ksoftirqd/022:15:200
1518099113,2cyclictest9-21ksoftirqd/021:37:060
1518099113,1cyclictest9-21ksoftirqd/021:21:110
1518099111,4cyclictest1-21systemd23:38:250
15180991111,0cyclictest658-21kworker/0:020:12:060
15180991111,0cyclictest27616-21kworker/0:221:07:530
15180991110,0cyclictest658-21kworker/0:020:01:510
1518099110,1cyclictest9-21ksoftirqd/000:20:130
5102100,0sleep10-21swapper/123:50:281
1519999109,0cyclictest0-21swapper/323:32:153
1519999109,0cyclictest0-21swapper/323:32:153
1519999109,0cyclictest0-21swapper/321:41:283
1519999109,0cyclictest0-21swapper/300:09:223
1519999100,9cyclictest0-21swapper/320:31:333
1519999100,0cyclictest0-21swapper/322:16:123
1519399108,1cyclictest820-21dbus-daemon21:26:472
1519399108,1cyclictest28122-21bash22:10:422
1519399105,3cyclictest33-21ksoftirqd/223:47:012
1519399105,2cyclictest33-21ksoftirqd/223:27:302
1519399101,1cyclictest131rcu_sched21:39:532
1519399100,7cyclictest698-21kworker/2:200:10:442
1519399100,1cyclictest131rcu_sched22:35:132
1519399100,1cyclictest131rcu_sched00:02:112
1519399100,0cyclictest131rcu_sched23:44:062
1519399100,0cyclictest0-21swapper/222:43:292
1519399100,0cyclictest0-21swapper/200:17:562
1518599109,0cyclictest0-21swapper/123:32:071
1518599109,0cyclictest0-21swapper/123:32:061
1518599109,0cyclictest0-21swapper/122:09:561
1518599109,0cyclictest0-21swapper/100:03:071
1518599100,9cyclictest0-21swapper/122:56:451
1518599100,0cyclictest0-21swapper/100:09:151
1518099109,0cyclictest658-21kworker/0:019:56:590
1518099105,1cyclictest9-21ksoftirqd/022:01:100
15180991010,0cyclictest27616-21kworker/0:221:04:360
1518099100,2cyclictest9-21ksoftirqd/023:54:310
151999998,0cyclictest0-21swapper/323:43:133
151999998,0cyclictest0-21swapper/322:45:533
151999998,0cyclictest0-21swapper/320:01:023
151999990,0cyclictest0-21swapper/321:20:493
151939998,0cyclictest3691-21lspci23:36:352
151939998,0cyclictest0-21swapper/221:21:282
151939998,0cyclictest0-21swapper/220:35:172
151939992,1cyclictest33-21ksoftirqd/223:00:202
151939991,1cyclictest33-21ksoftirqd/223:52:212
151939990,3cyclictest22894-21kworker/2:322:31:322
151939990,2cyclictest131rcu_sched00:07:442
151859998,0cyclictest0-21swapper/123:17:451
151859998,0cyclictest0-21swapper/100:10:421
151809999,0cyclictest16304-21kworker/0:020:36:210
151809999,0cyclictest10515-21kworker/0:120:34:480
151809994,1cyclictest9-21ksoftirqd/020:10:000
151999987,0cyclictest0-21swapper/323:12:423
151999987,0cyclictest0-21swapper/319:55:343
151999987,0cyclictest0-21swapper/319:51:113
151999987,0cyclictest0-21swapper/319:46:113
151999986,1cyclictest18709-21lspci22:58:113
151999980,7cyclictest762-21sshd22:30:593
151999980,7cyclictest7198-21sshd23:45:233
151999980,7cyclictest32290-21sshd23:50:243
151999980,7cyclictest31241-21sshd00:30:123
151999980,7cyclictest24309-21sshd23:58:193
151999980,7cyclictest20209-21sshd00:29:143
151999980,7cyclictest16097-21sshd23:09:213
151999980,7cyclictest13055-21sshd23:26:053
151939986,1cyclictest33-21ksoftirqd/221:10:242
151939980,7cyclictest4588-21sshd23:05:232
151859980,7cyclictest729-21sshd22:11:101
151859980,7cyclictest5714-21sshd23:25:251
151859980,7cyclictest4294-21sshd23:45:101
151859980,7cyclictest3811-21sshd23:11:031
151859980,7cyclictest2675-21sshd22:51:021
151859980,7cyclictest1662-21sshd22:16:541
151809988,0cyclictest29233-21kworker/0:219:46:360
151809988,0cyclictest25309-21kworker/0:119:42:470
151809988,0cyclictest16304-21kworker/0:020:47:050
151999976,0cyclictest0-21swapper/320:58:213
151999970,6cyclictest30424-21sshd22:25:103
151999970,6cyclictest25100-21sshd19:32:333
151999970,6cyclictest19280-21sshd23:01:023
151999970,6cyclictest13395-21sshd23:20:253
151999970,6cyclictest11954-21sshd22:06:273
151999970,0cyclictest0-21swapper/320:53:313
151939970,1cyclictest131rcu_sched19:25:012
151859975,1cyclictest29418-21lspci00:15:471
151859972,1cyclictest121rcu_preempt00:25:581
151859970,6cyclictest8524-21sshd00:30:571
151859970,6cyclictest4391-21sshd22:00:091
151859970,6cyclictest2939-21sshd22:25:291
151859970,6cyclictest24185-21sshd00:35:141
151859970,6cyclictest24185-21sshd00:35:141
151859970,6cyclictest19014-21sshd23:55:021
151859970,6cyclictest16885-21sshd21:30:021
151809977,0cyclictest658-21kworker/0:020:17:170
151809977,0cyclictest25309-21kworker/0:119:37:260
151809977,0cyclictest16304-21kworker/0:020:57:190
151809977,0cyclictest16304-21kworker/0:020:52:280
151809977,0cyclictest16304-21kworker/0:020:41:440
151809977,0cyclictest10515-21kworker/0:120:26:170
31580260,4sleep131701-21df19:50:141
151999965,0cyclictest0-21swapper/320:22:413
151999965,0cyclictest0-21swapper/319:39:153
151999965,0cyclictest0-21swapper/319:13:263
151999963,2cyclictest1398-21snmpd21:06:403
151999963,2cyclictest1398-21snmpd21:00:293
151999963,2cyclictest1397-21snmpd20:15:493
151939964,1cyclictest24908-21sshd19:30:322
151939964,1cyclictest13955-21sshd20:25:332
151939963,2cyclictest1398-21snmpd19:25:022
151939963,2cyclictest1398-21snmpd19:10:242
151939963,2cyclictest1397-21snmpd20:32:252
151939963,2cyclictest1397-21snmpd19:47:542
151939961,1cyclictest131rcu_sched20:00:002
151939960,5cyclictest21918-21sshd20:47:312
151939960,5cyclictest0-21swapper/220:56:162
151939960,5cyclictest0-21swapper/220:17:322
151939960,3cyclictest19491-21munin-run19:20:002
151859964,1cyclictest7206-21sed20:10:151
151859964,1cyclictest26703-21cat21:00:191
151859963,2cyclictest1398-21snmpd19:41:451
151809966,0cyclictest17352-21kworker/0:019:30:490
151809965,0cyclictest9-21ksoftirqd/019:15:000
151809963,2cyclictest1398-21snmpd20:22:110
151809960,5cyclictest29233-21kworker/0:219:50:320
151999953,1cyclictest7281-21cat20:10:163
151999953,1cyclictest5698-21ntp_offset20:05:213
151999953,1cyclictest1398-21snmpd20:40:073
151999953,1cyclictest1398-21snmpd19:40:573
151999953,1cyclictest1398-21snmpd19:20:243
151999953,1cyclictest1397-21snmpd20:45:553
151999953,1cyclictest1397-21snmpd20:35:143
151999953,1cyclictest13065-21cat20:25:163
151999952,2cyclictest22859-21tail19:25:193
151999952,2cyclictest1398-21snmpd19:17:023
151939953,1cyclictest28664-21cat21:05:192
151939953,1cyclictest27898-21cat19:40:142
151939953,1cyclictest1398-21snmpd20:50:052
151939953,1cyclictest1398-21snmpd20:21:042
151939953,1cyclictest1398-21snmpd20:10:062
151939953,1cyclictest1397-21snmpd21:01:092
151939953,1cyclictest1397-21snmpd20:07:342
151939953,1cyclictest1397-21snmpd19:50:262
151939953,1cyclictest1397-21snmpd19:35:352
151939952,2cyclictest19472-21switchtime20:40:202
151939952,0cyclictest3056-21grep20:00:152
151859953,1cyclictest9216-21cat20:15:151
151859953,1cyclictest3347-21cat20:00:181
151859953,1cyclictest24515-21tr20:55:121
151859953,1cyclictest16666-21cpuspeed_turbos20:35:131
151859953,1cyclictest15800-21cat19:10:141
151859953,1cyclictest1398-21snmpd20:51:051
151859953,1cyclictest1398-21snmpd19:45:101
151859953,1cyclictest1398-21snmpd19:25:141
151859953,1cyclictest1398-21snmpd19:15:111
151859953,1cyclictest1397-21snmpd20:31:251
151859953,1cyclictest1397-21snmpd20:05:061
151859953,1cyclictest13090-21cat20:25:151
151859952,2cyclictest29158-21gunzip21:05:231
151859952,2cyclictest1398-21snmpd19:20:501
151859952,2cyclictest1003-21grep19:55:111
151859951,1cyclictest913-21gdbus20:40:011
151859950,4cyclictest23629-21/usr/sbin/munin19:30:171
151859950,0cyclictest0-21swapper/120:45:291
151809955,0cyclictest17352-21kworker/0:019:21:490
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional