You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-03 - 07:48

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #4, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rack4slot0.osadl.org (updated Sun Jul 03, 2022 00:44:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1379121160,0sleep10-21swapper/122:29:131
276202930,2sleep20-21swapper/219:05:172
285322880,0sleep00-21swapper/023:03:110
4622800,1sleep0111rcuc/022:42:030
205102790,0sleep20-21swapper/222:01:092
16992790,0sleep00-21swapper/022:06:150
83032770,1sleep20-21swapper/223:33:232
114862770,1sleep01278-21runrttasks22:03:470
52422750,0sleep2311rcuc/200:34:102
45762730,0sleep00-21swapper/022:24:350
45762730,0sleep00-21swapper/022:24:340
138142730,0sleep00-21swapper/021:20:510
45932720,1sleep125838-21nfsd23:50:581
258112720,0sleep00-21swapper/021:50:570
112242710,0sleep00-21swapper/000:26:510
98512700,0sleep09848-21lspci23:58:500
280572700,1sleep128056-21sshd21:36:501
241272700,0sleep10-21swapper/121:22:001
58622690,2sleep32892999cyclictest23:11:283
219452670,1sleep121941-21lspci23:06:001
142782660,2sleep32892999cyclictest22:25:393
120732640,0sleep10-21swapper/121:31:261
212862630,1sleep1231rcuc/122:37:101
2851425726,10sleep00-21swapper/019:07:090
221492570,0sleep30-21swapper/322:37:173
2859825625,9sleep30-21swapper/319:08:043
191012550,0sleep019083-21sshd21:17:490
238562470,2sleep12892099cyclictest21:25:331
2846824423,13sleep10-21swapper/119:06:421
206532440,1sleep220652-21sshd21:21:362
4160-21360,1/usr/sbin/munin311rcuc/223:40:122
28925993013,3cyclictest25840-21nfsd22:14:392
2892599267,1cyclictest593-21systemd-journal22:38:532
2892599262,4cyclictest31563-21bash23:25:142
34422250,1sleep33430-21lspci22:53:123
2892099243,3cyclictest16914-21sshd00:11:541
55322230,0sleep00-21swapper/021:45:120
2892999233,2cyclictest0-21swapper/320:52:563
2892999233,2cyclictest0-21swapper/320:09:073
2892999233,1cyclictest0-21swapper/321:39:213
28929992319,2cyclictest102550irq/24-eno100:18:023
149942230,0sleep20-21swapper/223:01:412
107112230,1sleep310707-21bash00:30:473
2892999223,1cyclictest0-21swapper/321:48:353
2892999222,2cyclictest0-21swapper/323:51:393
2892999222,2cyclictest0-21swapper/321:27:223
2892999222,2cyclictest0-21swapper/320:48:223
2892999222,1cyclictest0-21swapper/321:43:203
2892599223,1cyclictest0-21swapper/222:52:462
2892599222,2cyclictest0-21swapper/220:33:542
2892599222,2cyclictest0-21swapper/219:56:252
2892599222,2cyclictest0-21swapper/219:56:252
2892599220,2cyclictest0-21swapper/200:20:142
2892599220,1cyclictest0-21swapper/220:01:422
2892599220,1cyclictest0-21swapper/220:01:412
2892099221,3cyclictest28772-21lspci23:32:031
2892099220,4cyclictest0-21swapper/120:55:151
2891999221,3cyclictest0-21swapper/020:10:430
2891999220,4cyclictest0-21swapper/022:52:040
2891999220,4cyclictest0-21swapper/020:43:030
89432210,1sleep18945-21bash23:29:531
2892999212,1cyclictest0-21swapper/320:23:473
2892999212,1cyclictest0-21swapper/319:34:443
2892999210,3cyclictest0-21swapper/320:41:373
2892999210,3cyclictest0-21swapper/320:31:083
2892999210,3cyclictest0-21swapper/320:28:193
2892599216,2cyclictest25840-21nfsd21:30:342
2892599213,2cyclictest0-21swapper/222:20:332
2892599213,2cyclictest0-21swapper/222:20:332
2892599211,2cyclictest2821-21sshd00:29:522
2892599211,2cyclictest0-21swapper/223:07:382
2892599211,2cyclictest0-21swapper/221:05:222
2892599211,2cyclictest0-21swapper/220:14:302
2892599211,2cyclictest0-21swapper/219:51:242
2892599210,3cyclictest0-21swapper/219:16:042
2892599210,3cyclictest0-21swapper/219:10:072
2892599210,0cyclictest0-21swapper/221:04:352
2892099213,3cyclictest1929-21lspci00:02:041
2892099210,3cyclictest0-21swapper/122:47:581
2892099210,2cyclictest13539-21sshd22:14:481
2891999211,2cyclictest0-21swapper/019:29:230
2891999210,3cyclictest0-21swapper/023:49:310
2891999210,3cyclictest0-21swapper/023:35:000
231462210,1sleep1241ktimersoftd/121:47:051
102712210,0sleep00-21swapper/022:18:020
2892999203,2cyclictest0-21swapper/323:00:143
2892999201,1cyclictest0-21swapper/321:58:213
2892999200,2cyclictest2670-21sshd22:56:423
2892999200,2cyclictest15122-21sshd00:11:433
2892999200,2cyclictest0-21swapper/321:03:543
2892999200,2cyclictest0-21swapper/319:37:493
2892599209,2cyclictest26892-21sshd23:39:062
2892599209,2cyclictest26892-21sshd23:39:062
2892599203,2cyclictest0-21swapper/223:51:252
2892599203,2cyclictest0-21swapper/221:50:422
2892599202,2cyclictest0-21swapper/223:46:122
2892599202,0cyclictest0-21swapper/200:16:462
2892599200,2cyclictest0-21swapper/223:58:142
2892599200,2cyclictest0-21swapper/220:45:542
2892099201,1cyclictest0-21swapper/119:24:471
2892099200,2cyclictest25840-21nfsd00:06:521
2892099200,2cyclictest0-21swapper/122:24:241
2892099200,2cyclictest0-21swapper/122:24:231
2892099200,2cyclictest0-21swapper/119:15:301
2892099200,1cyclictest0-21swapper/100:15:161
2891999208,1cyclictest9-21ksoftirqd/021:40:000
28919992017,1cyclictest101ktimersoftd/022:25:450
2891999201,1cyclictest29005-21lspci00:01:250
2891999201,1cyclictest1278-21runrttasks20:23:440
2891999200,2cyclictest19043-21sshd00:35:460
2891999200,2cyclictest0-21swapper/023:51:300
2891999200,2cyclictest0-21swapper/023:11:510
2891999200,2cyclictest0-21swapper/022:58:230
2891999200,2cyclictest0-21swapper/020:28:520
2891999200,2cyclictest0-21swapper/019:17:040
161872200,0sleep10-21swapper/100:39:271
43742190,14sleep3102550irq/24-eno123:58:133
2892999193,1cyclictest0-21swapper/322:40:423
2892999192,2cyclictest0-21swapper/319:17:563
2892999192,2cyclictest0-21swapper/300:06:453
2892999191,2cyclictest0-21swapper/319:29:303
2892999190,4cyclictest0-21swapper/319:51:383
2892999190,4cyclictest0-21swapper/319:23:233
2892999190,1cyclictest0-21swapper/323:22:113
2892999190,1cyclictest0-21swapper/323:18:233
2892999190,1cyclictest0-21swapper/323:07:493
2892999190,1cyclictest0-21swapper/322:48:313
2892999190,1cyclictest0-21swapper/322:09:513
2892999190,1cyclictest0-21swapper/321:15:143
2892999190,1cyclictest0-21swapper/321:14:503
2892999190,1cyclictest0-21swapper/321:05:143
2892999190,1cyclictest0-21swapper/320:36:473
2892999190,1cyclictest0-21swapper/320:15:043
2892999190,1cyclictest0-21swapper/319:47:493
2892999190,1cyclictest0-21swapper/300:36:203
2892599192,2cyclictest0-21swapper/219:34:342
2892599190,4cyclictest24807-21sshd22:44:472
2892599190,2cyclictest0-21swapper/219:38:432
2892599190,1cyclictest0-21swapper/223:19:242
2892599190,1cyclictest0-21swapper/222:46:582
2892599190,1cyclictest0-21swapper/221:56:592
2892599190,1cyclictest0-21swapper/220:36:462
2892599190,1cyclictest0-21swapper/220:23:052
2892599190,1cyclictest0-21swapper/220:16:042
2892599190,1cyclictest0-21swapper/219:24:212
2892099193,3cyclictest0-21swapper/122:42:411
2892099191,2cyclictest0-21swapper/123:23:421
2892099190,3cyclictest0-21swapper/123:00:151
2892099190,1cyclictest31916-21sshd21:40:531
2892099190,1cyclictest26935-21bash00:28:521
2892099190,1cyclictest13873-21sshd21:56:481
2892099190,1cyclictest0-21swapper/123:49:401
2892099190,1cyclictest0-21swapper/123:14:091
2892099190,1cyclictest0-21swapper/122:58:161
2892099190,1cyclictest0-21swapper/122:50:161
2892099190,1cyclictest0-21swapper/122:34:271
2892099190,1cyclictest0-21swapper/122:34:271
2892099190,1cyclictest0-21swapper/122:03:331
2892099190,1cyclictest0-21swapper/121:19:261
2892099190,1cyclictest0-21swapper/121:09:301
2892099190,1cyclictest0-21swapper/121:01:111
2892099190,1cyclictest0-21swapper/120:36:401
2892099190,1cyclictest0-21swapper/120:14:501
2892099190,1cyclictest0-21swapper/119:33:341
2892099190,1cyclictest0-21swapper/119:14:491
2892099190,1cyclictest0-21swapper/100:32:251
2891999190,3cyclictest9-21ksoftirqd/019:21:140
2891999190,1cyclictest0-21swapper/023:09:270
2891999190,1cyclictest0-21swapper/022:47:480
2891999190,1cyclictest0-21swapper/022:36:250
2891999190,1cyclictest0-21swapper/021:41:000
2891999190,1cyclictest0-21swapper/021:02:300
2891999190,1cyclictest0-21swapper/020:46:390
2891999190,1cyclictest0-21swapper/020:38:100
2891999190,1cyclictest0-21swapper/020:32:040
2891999190,1cyclictest0-21swapper/020:15:080
2891999190,1cyclictest0-21swapper/019:56:000
2891999190,1cyclictest0-21swapper/019:55:590
2891999190,1cyclictest0-21swapper/019:46:110
2891999190,1cyclictest0-21swapper/019:13:570
2892999182,2cyclictest0-21swapper/321:50:033
2892999181,2cyclictest14040-21lspci22:11:133
2892999181,2cyclictest0-21swapper/319:14:383
2892999180,2cyclictest26406-21lspci23:42:363
2892999180,1cyclictest0-21swapper/320:04:433
2892999180,1cyclictest0-21swapper/320:04:423
2892999180,1cyclictest0-21swapper/300:26:283
2892999180,1cyclictest0-21swapper/300:00:303
2892599181,2cyclictest28154-21ntp_states23:10:192
2892599181,2cyclictest0-21swapper/221:40:252
2892599180,2cyclictest15537-21sshd21:35:252
2892599180,2cyclictest0-21swapper/221:10:152
2892599180,1cyclictest0-21swapper/222:16:122
2892599180,1cyclictest0-21swapper/221:16:442
2892599180,1cyclictest0-21swapper/200:09:242
2892599180,0cyclictest0-21swapper/222:30:152
2892599180,0cyclictest0-21swapper/222:30:152
2892599180,0cyclictest0-21swapper/220:58:392
2892599180,0cyclictest0-21swapper/220:08:292
2892599180,0cyclictest0-21swapper/219:48:542
2892599180,0cyclictest0-21swapper/219:27:122
2892099180,3cyclictest0-21swapper/120:52:441
2892099180,3cyclictest0-21swapper/119:44:171
2892099180,2cyclictest593-21systemd-journal21:51:521
2892099180,2cyclictest0-21swapper/120:49:121
2892099180,1cyclictest0-21swapper/122:17:041
2892099180,1cyclictest0-21swapper/120:43:291
2892099180,1cyclictest0-21swapper/120:19:011
2892099180,1cyclictest0-21swapper/120:05:041
2892099180,0cyclictest0-21swapper/120:00:001
2892099180,0cyclictest0-21swapper/120:00:001
2892099180,0cyclictest0-21swapper/119:53:031
2892099180,0cyclictest0-21swapper/119:47:531
2891999186,2cyclictest9-21ksoftirqd/021:25:190
2891999181,4cyclictest121rcu_preempt23:29:520
2891999181,3cyclictest9-21ksoftirqd/022:15:000
2891999181,1cyclictest0-21swapper/021:05:540
2891999180,2cyclictest24963-21lspci00:32:340
2891999180,1cyclictest0-21swapper/020:52:410
2891999180,0cyclictest0-21swapper/019:50:150
188302180,0sleep30-21swapper/322:00:583
2892999170,0cyclictest0-21swapper/319:57:183
2892999170,0cyclictest0-21swapper/319:57:183
2892599170,2cyclictest0-21swapper/221:45:272
2892599170,0cyclictest0-21swapper/220:50:472
2892099173,1cyclictest0-21swapper/119:36:301
2892099170,1cyclictest0-21swapper/100:22:281
2891999176,5cyclictest867-21systemd-logind20:09:590
2891999176,3cyclictest9-21ksoftirqd/022:32:140
2891999176,3cyclictest9-21ksoftirqd/022:32:130
28919991713,3cyclictest1624-21sshd23:39:540
28919991713,3cyclictest1624-21sshd23:39:540
2891999171,1cyclictest15436-21idleruntime-cro23:45:000
2891999170,2cyclictest0-21swapper/023:20:560
2892999162,2cyclictest0-21swapper/322:33:463
2892999162,2cyclictest0-21swapper/322:33:463
28929991613,2cyclictest9686-21sshd21:24:003
2892999160,1cyclictest0-21swapper/323:30:353
2892999160,1cyclictest0-21swapper/322:20:523
2892999160,1cyclictest0-21swapper/322:20:523
2892999160,1cyclictest0-21swapper/321:31:253
2892999160,1cyclictest0-21swapper/300:20:473
2892599162,2cyclictest26782-21sshd22:09:022
2892599162,2cyclictest0-21swapper/219:40:052
2892599162,2cyclictest0-21swapper/200:35:152
2892599160,3cyclictest0-21swapper/222:25:122
2892599160,1cyclictest16836-21sshd21:28:252
2892599160,1cyclictest0-21swapper/223:20:292
2892599160,1cyclictest0-21swapper/222:58:282
2892599160,1cyclictest0-21swapper/200:03:152
2892099160,1cyclictest0-21swapper/123:17:061
2892099160,1cyclictest0-21swapper/120:33:491
2892099160,1cyclictest0-21swapper/120:25:141
2892099160,15cyclictest17364-21sshd21:10:241
2891999166,1cyclictest9-21ksoftirqd/021:31:430
2891999165,3cyclictest9-21ksoftirqd/021:57:540
2891999165,3cyclictest9-21ksoftirqd/000:22:440
2891999160,2cyclictest9-21ksoftirqd/021:12:300
2891999160,1cyclictest0-21swapper/000:18:100
2891999160,1cyclictest0-21swapper/000:05:250
229162160,0sleep10-21swapper/122:08:371
2892999151,2cyclictest0-21swapper/320:55:063
2892999150,3cyclictest0-21swapper/323:28:573
2892999150,0cyclictest0-21swapper/322:15:363
2892599150,0cyclictest0-21swapper/220:42:532
2892599150,0cyclictest0-21swapper/220:29:252
2892099150,1cyclictest23866-21/usr/sbin/munin23:35:181
2892099150,1cyclictest23866-21/usr/sbin/munin23:35:171
2892099150,0cyclictest0-21swapper/119:28:041
2891999151,2cyclictest3625-21lspci00:10:180
2891999151,11cyclictest121rcu_preempt23:18:190
2891999150,3cyclictest0-21swapper/020:02:580
2891999150,3cyclictest0-21swapper/020:02:580
2891999150,3cyclictest0-21swapper/019:40:460
2891999150,1cyclictest0-21swapper/019:36:120
2891999150,1cyclictest0-21swapper/019:34:110
2892999142,2cyclictest18592-21sshd23:48:543
2892999142,1cyclictest0-21swapper/319:40:273
2892999140,0cyclictest0-21swapper/323:36:243
2892999140,0cyclictest0-21swapper/323:36:243
28920991412,1cyclictest14166-21bash23:55:391
28920991411,2cyclictest15191-21in:imjournal23:42:421
2891999140,0cyclictest0-21swapper/020:56:090
2892599131,3cyclictest2348-21lspci00:14:052
2892099128,2cyclictest17604-21/usr/sbin/munin20:00:121
2892099128,2cyclictest17604-21/usr/sbin/munin20:00:111
2892099120,10cyclictest0-21swapper/120:20:181
2892999110,0cyclictest0-21swapper/320:12:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional