You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2022-01-26 - 11:47

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #1, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rack1slot0.osadl.org (updated Wed Jan 26, 2022 00:43:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2610221060,0sleep026101-21grep23:30:460
312232880,0sleep20-21swapper/221:53:582
137662730,2sleep22504599cyclictest22:29:392
82702650,1sleep132710-21kworker/1:322:03:221
122252650,0sleep012223-21sshd21:26:110
2475726250,7sleep20-21swapper/219:08:212
224632610,0sleep20-21swapper/222:04:392
157112600,1sleep315712-21bash21:29:383
271352590,0sleep10-21swapper/122:48:011
266012590,1sleep3391rcuc/322:42:143
203912590,0sleep30-21swapper/321:23:583
116272590,0sleep10-21swapper/100:21:071
201722580,0sleep10-21swapper/121:50:081
296532570,0sleep00-21swapper/000:05:320
2463925734,7sleep00-21swapper/019:07:030
2458325734,7sleep30-21swapper/319:06:223
230332570,0sleep123037-21bash21:44:391
221882570,0sleep00-21swapper/023:04:460
170432570,0sleep20-21swapper/221:29:462
117932570,0sleep20-21swapper/200:06:512
313642560,0sleep10-21swapper/123:22:421
313642560,0sleep10-21swapper/123:22:421
290902560,0sleep20-21swapper/223:11:012
266732550,0sleep20-21swapper/221:36:192
266732550,0sleep20-21swapper/221:36:192
209072550,0sleep20-21swapper/221:24:012
192592550,0sleep30-21swapper/322:35:493
140632550,0sleep10-21swapper/122:52:341
252782540,0sleep00-21swapper/021:15:220
2402725139,7sleep10-21swapper/119:05:281
310962500,0sleep30-21swapper/300:22:533
63212450,1sleep1241ktimersoftd/123:37:401
154282430,0sleep00-21swapper/023:50:010
148552250,2sleep32505399cyclictest00:04:193
135242250,0sleep20-21swapper/223:01:042
161272190,13sleep395750irq/24-eno1-rx-23:15:353
47192170,0sleep00-21swapper/022:37:260
37682170,0sleep00-21swapper/023:20:170
37682170,0sleep00-21swapper/023:20:170
299802170,0sleep20-21swapper/222:13:532
187942170,1sleep318790-21lspci00:16:013
128912170,0sleep00-21swapper/021:46:340
93872160,0sleep3820-21dbus-daemon23:40:473
278292160,0sleep30-21swapper/300:25:263
255572160,0sleep2311rcuc/221:59:092
238432160,0sleep00-21swapper/023:27:460
226732160,2sleep322667-21sshd21:18:083
213182160,0sleep00-21swapper/022:53:130
44852150,0sleep10-21swapper/100:11:541
310132150,0sleep1231rcuc/122:05:221
2503299144,2cyclictest9-21ksoftirqd/021:59:590
25032991414,0cyclictest4298-21kworker/0:022:02:450
2503299140,12cyclictest801-21systemd-logind23:42:270
127842140,0sleep30-21swapper/322:32:243
25032991312,0cyclictest16117-21kworker/0:023:15:410
25032991311,1cyclictest5970-21kworker/0:222:49:220
2505399120,11cyclictest20505-21lspci23:47:343
2505399120,11cyclictest1-21systemd21:46:383
2505399120,11cyclictest0-21swapper/300:13:533
25045991210,1cyclictest13459-21consoletype00:04:112
2503899120,1cyclictest0-21swapper/122:41:441
2503899120,11cyclictest23623-21grep21:27:181
25032991212,0cyclictest5530-21kworker/0:000:12:340
2503299120,11cyclictest23717-21lspci23:36:190
180882120,0sleep20-21swapper/222:10:002
25053991110,0cyclictest0-21swapper/323:07:213
2505399110,10cyclictest22935-21sshd23:36:153
2505399110,10cyclictest0-21swapper/323:20:263
2505399110,10cyclictest0-21swapper/323:20:263
2505399110,10cyclictest0-21swapper/300:30:213
2504599119,1cyclictest27597-21lspci22:16:302
2504599119,1cyclictest13367-21sshd22:52:292
25045991110,0cyclictest0-21swapper/223:54:402
2504599110,10cyclictest0-21swapper/223:41:352
2504599110,10cyclictest0-21swapper/222:37:582
2504599110,10cyclictest0-21swapper/222:22:182
2503899119,1cyclictest28253-21lspci21:33:361
25038991110,0cyclictest17369-21sshd23:32:551
25038991110,0cyclictest0-21swapper/123:40:311
2503899110,1cyclictest5316-21sshd00:03:261
2503299116,1cyclictest9-21ksoftirqd/022:25:200
25032991110,0cyclictest30524-21lspci22:05:200
25032991110,0cyclictest28450-21kworker/0:200:35:260
25032991110,0cyclictest22246-21lspci22:10:190
2503299110,10cyclictest0-21swapper/021:52:490
289942100,0sleep0111rcuc/000:25:310
2505399109,0cyclictest0-21swapper/323:14:373
2505399109,0cyclictest0-21swapper/322:56:033
2505399109,0cyclictest0-21swapper/322:19:153
2505399109,0cyclictest0-21swapper/319:35:383
2505399108,1cyclictest41-21ksoftirqd/300:35:193
2505399100,9cyclictest0-21swapper/322:06:383
2505399100,5cyclictest131rcu_sched21:52:163
2504599109,0cyclictest0-21swapper/223:20:352
2504599109,0cyclictest0-21swapper/223:20:352
2504599109,0cyclictest0-21swapper/223:16:142
2504599109,0cyclictest0-21swapper/222:48:472
2504599109,0cyclictest0-21swapper/221:03:502
2504599109,0cyclictest0-21swapper/220:37:512
2504599108,1cyclictest801-21systemd-logind00:34:132
2504599108,1cyclictest21342-21lspci23:07:312
2504599100,9cyclictest0-21swapper/219:15:292
2504599100,0cyclictest0-21swapper/223:34:002
2504599100,0cyclictest0-21swapper/221:47:052
2503899109,0cyclictest0-21swapper/122:22:221
2503899109,0cyclictest0-21swapper/122:16:331
2503899109,0cyclictest0-21swapper/121:21:011
2503899100,0cyclictest19687-21sshd21:11:391
2503899100,0cyclictest0-21swapper/100:18:471
2503299109,0cyclictest3417-21kworker/0:221:42:070
2503299109,0cyclictest0-21swapper/022:15:160
2503299108,1cyclictest7397-21kworker/0:121:21:190
2503299108,1cyclictest1828-21kworker/0:322:31:520
2503299108,1cyclictest17180-21kworker/0:221:33:160
25032991010,0cyclictest490-21kworker/0:423:58:320
2503299100,9cyclictest32700-21kworker/0:200:16:590
2503299100,9cyclictest30311-21kworker/0:300:21:140
250539998,0cyclictest32448-21sensors19:25:263
250539998,0cyclictest0-21swapper/322:27:483
250539997,1cyclictest41-21ksoftirqd/323:00:163
250539997,1cyclictest29110-21unixbench_singl00:05:293
250459998,0cyclictest0-21swapper/223:58:422
250459998,0cyclictest0-21swapper/220:44:572
250459998,0cyclictest0-21swapper/200:19:312
250459997,1cyclictest0-21swapper/223:35:272
250459990,0cyclictest13247-21sshd22:43:552
250389998,1cyclictest0-21swapper/120:01:301
250389998,0cyclictest0-21swapper/123:52:271
250389998,0cyclictest0-21swapper/122:29:281
250389998,0cyclictest0-21swapper/119:25:431
250389998,0cyclictest0-21swapper/100:37:301
250389997,1cyclictest1468-21lspci22:34:181
250389997,1cyclictest10516-21lspci23:46:391
250389997,1cyclictest10450-21sshd23:15:101
250389997,1cyclictest0-21swapper/123:06:271
250329999,0cyclictest4298-21kworker/0:021:38:190
250329999,0cyclictest4298-21kworker/0:021:38:180
250329999,0cyclictest24894-21kworker/0:019:40:260
250329998,0cyclictest5970-21kworker/0:222:40:420
250329998,0cyclictest24894-21kworker/0:019:30:220
250329998,0cyclictest0-21swapper/022:58:130
250329998,0cyclictest0-21swapper/022:21:230
250329994,2cyclictest9-21ksoftirqd/000:02:260
250329991,1cyclictest131rcu_sched23:47:420
250329990,8cyclictest13795-21kworker/0:223:11:100
250329990,1cyclictest22258-21nscd21:14:110
250329990,1cyclictest131rcu_sched23:08:290
250539987,0cyclictest0-21swapper/322:49:463
250539986,1cyclictest41-21ksoftirqd/323:51:273
250539980,7cyclictest9158-21sshd23:55:133
250539980,7cyclictest3807-21sshd22:00:083
250539980,7cyclictest31388-21sshd22:14:003
250539980,7cyclictest30886-21kworker/3:321:30:383
250539980,7cyclictest28838-21sshd23:31:023
250539980,7cyclictest27702-21sshd22:50:523
250539980,7cyclictest23931-21sshd21:12:133
250539980,7cyclictest14061-21sshd21:35:153
250539980,7cyclictest14061-21sshd21:35:153
250539980,7cyclictest13921-21sshd22:23:563
250539980,7cyclictest12344-21kworker/3:021:55:173
250459987,0cyclictest0-21swapper/219:45:222
250459987,0cyclictest0-21swapper/200:36:482
250459980,7cyclictest28285-21sshd21:18:392
250459980,7cyclictest2607-21sshd00:11:432
250459980,7cyclictest24456-21sshd22:30:312
250459980,7cyclictest0-21swapper/220:21:312
250389987,0cyclictest0-21swapper/123:10:481
250389987,0cyclictest0-21swapper/122:13:191
250389987,0cyclictest0-21swapper/121:36:111
250389987,0cyclictest0-21swapper/121:36:101
250389987,0cyclictest0-21swapper/100:07:281
250389986,1cyclictest0-21swapper/121:45:421
250389980,7cyclictest3557-21sshd00:26:091
250389980,7cyclictest22664-21sshd23:56:211
250389980,7cyclictest20329-21sshd00:30:281
250329981,3cyclictest820-21dbus-daemon20:20:010
250329980,1cyclictest131rcu_sched00:30:360
250539975,1cyclictest41-21ksoftirqd/321:40:483
250539970,6cyclictest25211-21sshd23:25:043
250539970,6cyclictest16105-21sshd20:06:503
250459976,0cyclictest0-21swapper/220:28:382
250459976,0cyclictest0-21swapper/219:32:182
250459976,0cyclictest0-21swapper/219:20:502
250459976,0cyclictest0-21swapper/219:10:302
250459975,1cyclictest5967-21lspci21:31:362
250459975,1cyclictest1-21systemd21:10:462
250459970,6cyclictest9870-21sshd00:26:432
250459970,6cyclictest8351-21sshd21:43:202
250459970,6cyclictest5457-21sshd23:49:082
250459970,6cyclictest31618-21sshd23:25:322
250459970,6cyclictest15362-21sshd22:55:282
250459970,6cyclictest11424-21sshd00:21:062
250389976,0cyclictest0-21swapper/121:15:021
250389976,0cyclictest0-21swapper/120:56:181
250389976,0cyclictest0-21swapper/120:47:541
250389975,1cyclictest29779-21sensors23:25:231
250389975,1cyclictest1398-21snmpd20:20:041
250389970,6cyclictest7645-21sshd21:10:011
250389970,6cyclictest17344-21sshd22:55:401
250389970,6cyclictest16645-21sshd22:35:351
250389970,6cyclictest12636-21sshd21:55:111
250389970,6cyclictest12213-21sshd23:00:571
250539963,2cyclictest1398-21snmpd20:41:293
250539963,2cyclictest1398-21snmpd19:13:413
250539963,2cyclictest1397-21snmpd21:04:073
250539963,2cyclictest1397-21snmpd20:10:553
250539960,5cyclictest0-21swapper/319:30:313
250459965,0cyclictest0-21swapper/220:15:532
250459965,0cyclictest0-21swapper/220:03:462
250459963,2cyclictest1398-21snmpd21:08:552
250459963,2cyclictest1397-21snmpd19:57:352
250459963,2cyclictest1397-21snmpd19:54:462
250389965,0cyclictest0-21swapper/120:44:021
250389965,0cyclictest0-21swapper/119:36:521
250389965,0cyclictest0-21swapper/119:21:131
250389963,2cyclictest1397-21snmpd20:25:391
250329966,0cyclictest24894-21kworker/0:019:35:150
250329964,1cyclictest9-21ksoftirqd/020:55:010
250329963,2cyclictest1398-21snmpd20:28:240
250329963,2cyclictest1398-21snmpd20:20:430
250329963,2cyclictest1398-21snmpd19:45:540
250329963,2cyclictest1397-21snmpd20:05:290
250329960,1cyclictest131rcu_sched20:45:000
250539953,1cyclictest30951-21grep20:45:233
250539953,1cyclictest29963-21cat19:20:203
250539953,1cyclictest28067-21sadc20:40:013
250539953,1cyclictest24859-21cat20:30:203
250539953,1cyclictest22878-21cat20:25:213
250539953,1cyclictest1398-21snmpd21:06:043
250539953,1cyclictest1398-21snmpd20:51:113
250539953,1cyclictest1398-21snmpd20:21:513
250539953,1cyclictest1397-21snmpd20:55:073
250539953,1cyclictest1397-21snmpd20:15:293
250539953,1cyclictest1397-21snmpd20:00:073
250539953,1cyclictest1397-21snmpd19:50:113
250539953,1cyclictest1397-21snmpd19:40:163
250539953,1cyclictest11169-21idleruntime19:55:213
250539952,2cyclictest7950-21cat19:45:253
250539952,2cyclictest27663-21cat19:15:163
250459954,0cyclictest0-21swapper/220:50:362
250459953,1cyclictest2216-21cat20:55:192
250459953,1cyclictest1398-21snmpd20:45:132
250459953,1cyclictest1398-21snmpd19:40:332
250459953,1cyclictest1397-21snmpd20:10:252
250459953,1cyclictest1397-21snmpd20:05:412
250459953,1cyclictest1219-21runrttasks20:30:102
250459952,2cyclictest32672-21grep19:25:282
250459952,2cyclictest1397-21snmpd19:35:212
250389953,1cyclictest696-21processes20:50:251
250389953,1cyclictest4725-21switchtime21:00:261
250389953,1cyclictest25514-21entropy19:10:171
250389953,1cyclictest1398-21snmpd20:35:511
250389953,1cyclictest1398-21snmpd20:16:131
250389953,1cyclictest1398-21snmpd19:30:521
250389953,1cyclictest1397-21snmpd19:45:011
250389953,1cyclictest1397-21snmpd19:40:481
250389953,1cyclictest1397-21snmpd19:16:091
250389953,1cyclictest1219-21runrttasks20:05:501
250389952,2cyclictest9476-21needreboot19:50:211
250389952,2cyclictest25697-21who20:30:281
250389952,2cyclictest1398-21snmpd20:10:331
250389950,1cyclictest10817-21/usr/sbin/munin19:55:281
250329953,1cyclictest9-21ksoftirqd/021:00:000
250329953,1cyclictest9200-21hddtemp_smartct19:50:190
250329953,1cyclictest3486-21kworker/0:221:05:230
250329953,1cyclictest29941-21cat19:20:190
250329953,1cyclictest28013-21cat19:15:210
250329953,1cyclictest26802-21cat20:35:210
250329953,1cyclictest25624-21cat19:10:180
250329953,1cyclictest25025-21missed_timers20:30:220
250329953,1cyclictest16992-21cut20:10:180
250329953,1cyclictest1398-21snmpd20:00:320
250329953,1cyclictest1397-21snmpd20:56:130
250329953,1cyclictest1397-21snmpd20:41:390
250329953,1cyclictest1397-21snmpd19:25:100
250329953,1cyclictest11239-21cat19:55:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional