You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-17 - 05:57

Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #4, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack4slot0.osadl.org (updated Sun Oct 17, 2021 00:44:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1147021170,0sleep011469-21bash22:58:420
240842770,2sleep2723799cyclictest21:52:562
6962750,1sleep2321ktimersoftd/222:21:092
44832730,0sleep10-21swapper/122:48:131
219272700,0sleep20-21swapper/222:35:282
176042670,0sleep10-21swapper/122:43:051
200832660,0sleep00-21swapper/022:52:170
124962610,0sleep10-21swapper/122:59:001
683225625,10sleep30-21swapper/319:07:193
683625425,9sleep20-21swapper/219:07:222
708224429,9sleep10-21swapper/119:09:581
532824225,10sleep00-21swapper/019:05:030
291082420,1sleep229110-21bash21:11:052
48292270,0sleep00-21swapper/000:31:420
724099252,4cyclictest0-21swapper/319:33:133
724099240,1cyclictest0-21swapper/322:36:393
723799240,1cyclictest0-21swapper/200:13:102
723799233,2cyclictest0-21swapper/219:42:142
723799232,2cyclictest0-21swapper/223:13:402
723199233,2cyclictest0-21swapper/121:53:241
724099223,1cyclictest0-21swapper/323:31:583
724099223,1cyclictest0-21swapper/323:31:573
724099220,4cyclictest0-21swapper/323:10:243
724099220,4cyclictest0-21swapper/322:52:203
724099220,4cyclictest0-21swapper/319:37:083
724099220,4cyclictest0-21swapper/300:15:173
723799223,1cyclictest0-21swapper/223:59:092
723199222,2cyclictest0-21swapper/120:54:061
723199222,2cyclictest0-21swapper/120:03:541
318692220,0sleep00-21swapper/000:21:410
143932220,0sleep20-21swapper/222:42:092
724099216,0cyclictest0-21swapper/320:40:373
724099211,2cyclictest0-21swapper/322:12:283
724099210,3cyclictest0-21swapper/323:55:183
724099210,3cyclictest0-21swapper/323:22:343
724099210,3cyclictest0-21swapper/320:55:513
724099210,3cyclictest0-21swapper/320:34:003
724099210,2cyclictest0-21swapper/321:52:273
723799213,2cyclictest0-21swapper/221:18:552
723799211,2cyclictest0-21swapper/223:19:512
723799210,3cyclictest0-21swapper/221:55:432
723799210,3cyclictest0-21swapper/221:00:472
723199210,5cyclictest0-21swapper/122:35:211
723199210,4cyclictest0-21swapper/119:35:521
723199210,3cyclictest0-21swapper/122:24:361
723199210,3cyclictest0-21swapper/120:22:311
723199210,1cyclictest0-21swapper/123:25:171
302742210,0sleep20-21swapper/222:03:142
287312210,0sleep10-21swapper/100:03:421
226662210,16sleep3102550irq/24-eno123:18:563
724099202,2cyclictest0-21swapper/323:07:443
724099200,3cyclictest0-21swapper/323:25:243
724099200,3cyclictest0-21swapper/320:14:213
724099200,2cyclictest0-21swapper/321:36:463
724099200,2cyclictest0-21swapper/319:45:203
723799203,2cyclictest0-21swapper/222:06:032
723799200,2cyclictest0-21swapper/223:48:202
723799200,2cyclictest0-21swapper/222:56:292
723799200,2cyclictest0-21swapper/222:51:542
723799200,2cyclictest0-21swapper/219:25:212
723799200,2cyclictest0-21swapper/219:25:202
723199201,1cyclictest0-21swapper/120:15:151
723199200,2cyclictest0-21swapper/122:30:301
723199200,2cyclictest0-21swapper/120:30:071
723199200,2cyclictest0-21swapper/119:57:121
223772200,0sleep00-21swapper/019:45:180
724099190,3cyclictest0-21swapper/319:16:313
724099190,1cyclictest0-21swapper/323:51:023
724099190,1cyclictest0-21swapper/323:46:583
724099190,1cyclictest0-21swapper/323:40:393
724099190,1cyclictest0-21swapper/323:01:023
724099190,1cyclictest0-21swapper/322:20:343
724099190,1cyclictest0-21swapper/322:17:403
724099190,1cyclictest0-21swapper/321:59:183
724099190,1cyclictest0-21swapper/321:32:293
724099190,1cyclictest0-21swapper/321:16:133
724099190,1cyclictest0-21swapper/321:06:463
724099190,1cyclictest0-21swapper/321:03:053
724099190,1cyclictest0-21swapper/320:52:003
724099190,1cyclictest0-21swapper/320:47:193
724099190,1cyclictest0-21swapper/320:47:193
724099190,1cyclictest0-21swapper/320:29:443
724099190,1cyclictest0-21swapper/320:22:353
724099190,1cyclictest0-21swapper/320:01:503
724099190,1cyclictest0-21swapper/319:56:213
724099190,1cyclictest0-21swapper/319:51:243
724099190,1cyclictest0-21swapper/319:44:533
724099190,1cyclictest0-21swapper/319:29:593
724099190,1cyclictest0-21swapper/319:29:593
724099190,1cyclictest0-21swapper/319:20:433
724099190,1cyclictest0-21swapper/300:40:003
724099190,1cyclictest0-21swapper/300:34:233
724099190,1cyclictest0-21swapper/300:22:033
724099190,1cyclictest0-21swapper/300:14:263
724099190,1cyclictest0-21swapper/300:01:513
723799193,2cyclictest0-21swapper/220:05:092
723799192,2cyclictest0-21swapper/221:46:052
723799190,4cyclictest2266-21nfsd00:00:322
723799190,4cyclictest10198-21sshd22:49:492
723799190,3cyclictest0-21swapper/223:40:182
723799190,3cyclictest0-21swapper/222:30:272
723799190,1cyclictest0-21swapper/221:43:292
723799190,1cyclictest0-21swapper/221:20:252
723799190,1cyclictest0-21swapper/221:05:472
723799190,1cyclictest0-21swapper/220:45:192
723799190,1cyclictest0-21swapper/220:45:192
723799190,1cyclictest0-21swapper/220:17:122
723799190,1cyclictest0-21swapper/220:12:052
723799190,1cyclictest0-21swapper/219:54:212
723799190,1cyclictest0-21swapper/200:29:252
723799190,1cyclictest0-21swapper/200:18:542
723199190,3cyclictest0-21swapper/122:51:401
723199190,2cyclictest12837-21sshd23:16:081
723199190,1cyclictest0-21swapper/123:57:301
723199190,1cyclictest0-21swapper/123:46:251
723199190,1cyclictest0-21swapper/123:37:211
723199190,1cyclictest0-21swapper/123:23:111
723199190,1cyclictest0-21swapper/122:27:571
723199190,1cyclictest0-21swapper/122:16:301
723199190,1cyclictest0-21swapper/122:06:121
723199190,1cyclictest0-21swapper/122:02:421
723199190,1cyclictest0-21swapper/121:40:061
723199190,1cyclictest0-21swapper/121:37:291
723199190,1cyclictest0-21swapper/121:30:431
723199190,1cyclictest0-21swapper/121:10:281
723199190,1cyclictest0-21swapper/121:03:121
723199190,1cyclictest0-21swapper/120:11:151
723199190,1cyclictest0-21swapper/119:44:201
723199190,1cyclictest0-21swapper/119:25:011
723199190,1cyclictest0-21swapper/119:25:011
723199190,1cyclictest0-21swapper/119:18:161
723199190,1cyclictest0-21swapper/100:36:161
723199190,1cyclictest0-21swapper/100:23:531
723199190,1cyclictest0-21swapper/100:06:131
214332190,0sleep00-21swapper/022:26:440
724099182,1cyclictest0-21swapper/321:10:303
7240991817,0cyclictest102550irq/24-eno122:41:053
724099181,2cyclictest0-21swapper/322:03:593
724099180,3cyclictest0-21swapper/320:15:513
724099180,2cyclictest0-21swapper/300:29:463
724099180,1cyclictest0-21swapper/319:13:583
724099180,0cyclictest0-21swapper/322:26:153
724099180,0cyclictest0-21swapper/322:09:573
724099180,0cyclictest0-21swapper/321:44:073
724099180,0cyclictest0-21swapper/321:26:083
724099180,0cyclictest0-21swapper/321:21:043
7237991816,1cyclictest9021-21lspci23:32:262
7237991816,1cyclictest9021-21lspci23:32:262
723799180,3cyclictest15601-21sshd22:16:312
723799180,3cyclictest0-21swapper/222:25:582
723799180,3cyclictest0-21swapper/221:35:192
723799180,3cyclictest0-21swapper/220:36:482
723799180,2cyclictest2488-21bash00:22:322
723799180,2cyclictest0-21swapper/223:03:112
723799180,1cyclictest0-21swapper/223:08:352
723799180,1cyclictest0-21swapper/220:56:572
723799180,0cyclictest0-21swapper/223:21:452
723799180,0cyclictest0-21swapper/220:43:282
723799180,0cyclictest0-21swapper/219:22:552
723199180,3cyclictest7746-21sshd00:32:331
723199180,2cyclictest30833-21sshd22:12:001
723199180,2cyclictest0-21swapper/100:25:251
723199180,1cyclictest2637-21sshd23:05:011
723199180,1cyclictest0-21swapper/121:47:201
723199180,1cyclictest0-21swapper/121:28:471
723199180,1cyclictest0-21swapper/120:45:521
723199180,1cyclictest0-21swapper/120:45:521
723199180,1cyclictest0-21swapper/120:41:221
7230991816,1cyclictest9-21ksoftirqd/022:43:200
724099170,2cyclictest0-21swapper/322:57:193
724099170,0cyclictest0-21swapper/320:08:143
723799170,2cyclictest0-21swapper/222:10:072
723799170,2cyclictest0-21swapper/200:33:162
723199170,3cyclictest0-21swapper/123:31:211
723199170,3cyclictest0-21swapper/123:31:201
723199170,2cyclictest0-21swapper/120:28:481
259392170,0sleep00-21swapper/023:28:240
7240991614,1cyclictest9216-21kworker/3:000:05:423
724099160,1cyclictest0-21swapper/322:31:373
724099160,1cyclictest0-21swapper/321:49:443
723799160,5cyclictest2663-21lspci00:05:242
723799160,4cyclictest0-21swapper/220:27:092
723799160,4cyclictest0-21swapper/220:02:542
723799160,3cyclictest27964-21idleruntime-cro20:00:002
723799160,1cyclictest0-21swapper/221:33:582
723799160,1cyclictest0-21swapper/221:27:402
723799160,1cyclictest0-21swapper/219:37:142
723799160,0cyclictest0-21swapper/220:20:202
723199161,1cyclictest0-21swapper/120:36:351
723199160,1cyclictest0-21swapper/123:50:331
723199160,1cyclictest0-21swapper/123:10:331
723199160,1cyclictest0-21swapper/121:58:421
723199160,1cyclictest0-21swapper/121:21:361
723199160,1cyclictest0-21swapper/121:17:501
723199160,1cyclictest0-21swapper/120:55:121
723199160,1cyclictest0-21swapper/120:09:231
723199160,1cyclictest0-21swapper/119:51:141
723199160,1cyclictest0-21swapper/100:16:261
723199160,1cyclictest0-21swapper/100:13:431
723099164,10cyclictest13886-21unixbench_singl00:25:260
724099150,1cyclictest28147-21grepconf.sh22:45:453
724099150,0cyclictest0-21swapper/323:37:093
723799150,3cyclictest0-21swapper/220:31:342
723799150,2cyclictest0-21swapper/219:18:252
723799150,0cyclictest0-21swapper/223:26:142
723799150,0cyclictest0-21swapper/220:51:312
723799150,0cyclictest0-21swapper/219:34:152
723799150,0cyclictest0-21swapper/219:13:342
723799150,0cyclictest0-21swapper/200:36:322
723199150,3cyclictest0-21swapper/121:05:191
723199150,1cyclictest0-21swapper/123:01:351
723199150,1cyclictest0-21swapper/119:22:071
723099147,1cyclictest121rcu_preempt23:33:030
723099147,1cyclictest121rcu_preempt23:33:030
723099143,10cyclictest19244-21bash23:00:350
7230991414,0cyclictest0-21swapper/021:05:180
7230991413,0cyclictest722-40kipmi022:45:250
7230991412,1cyclictest29908-21sshd22:37:440
723099140,1cyclictest121rcu_preempt23:35:330
723099140,13cyclictest3901-21kworker/0:221:43:040
146132140,0sleep00-21swapper/022:33:410
724099130,3cyclictest0-21swapper/320:37:523
723099139,2cyclictest1510-21snmpd21:51:080
723099138,4cyclictest3901-21kworker/0:221:27:480
723099133,1cyclictest31317-21sshd23:46:590
7230991311,1cyclictest5588-21sshd22:05:190
7230991311,1cyclictest29375-21lspci22:11:340
7230991311,1cyclictest28847-21lspci22:20:170
7230991311,1cyclictest26842-21sshd23:54:360
7230991311,1cyclictest24530-21kworker/0:100:12:270
7230991311,1cyclictest23230-21missed_timers23:10:270
7230991311,1cyclictest2151-21bash23:21:590
7230991311,1cyclictest14294-21bash21:33:050
723799120,1cyclictest0-21swapper/223:36:212
723799120,10cyclictest23001-21proc_pri19:45:262
723799120,0cyclictest0-21swapper/223:50:032
723199120,1cyclictest0-21swapper/119:31:561
723199120,1cyclictest0-21swapper/119:12:461
723199120,10cyclictest7435-21sshd23:40:331
723099129,2cyclictest3507-21sshd21:55:560
723099129,2cyclictest30411-21sshd21:45:490
723099128,2cyclictest27957-21sadc20:00:000
723099121,1cyclictest0-21swapper/000:35:160
7230991211,0cyclictest722-40kipmi019:25:180
7230991211,0cyclictest722-40kipmi019:25:180
7230991210,1cyclictest13977-21lspci23:42:260
723099120,3cyclictest0-21swapper/021:11:220
723099120,2cyclictest8653-21grep00:15:240
723099120,2cyclictest4573-21sshd23:05:310
723099120,2cyclictest2777-21bash23:56:390
723099120,2cyclictest22353-21sshd21:35:140
723099120,2cyclictest19013-21sshd23:17:540
723099120,2cyclictest18658-21rm21:16:570
723099120,2cyclictest1856-21bash21:21:030
723099120,11cyclictest0-21swapper/022:00:180
723099120,10cyclictest1576-21lspci00:05:120
723099120,10cyclictest1273-21sshd00:00:240
723099120,10cyclictest12174-21cat19:20:180
723099120,10cyclictest0-21swapper/021:05:010
723199110,10cyclictest0-21swapper/119:45:161
723099119,1cyclictest0-21swapper/020:55:210
723099118,2cyclictest11278-21date20:35:260
723099110,2cyclictest9856-21munin-run20:35:000
723099110,2cyclictest9393-21grep22:15:030
723099110,2cyclictest20348-21expr19:40:150
723099110,2cyclictest16209-21/usr/sbin/munin20:50:220
723099110,2cyclictest0-21swapper/020:24:290
723099110,2cyclictest0-21swapper/019:35:110
723099110,1cyclictest593-21systemd-journal20:19:590
723099110,1cyclictest2266-21nfsd20:45:080
723099110,1cyclictest2266-21nfsd20:45:070
723099110,1cyclictest10494-21memory19:15:190
723099110,10cyclictest8503-21sed19:10:220
723099110,10cyclictest367-21sed20:10:220
723099110,10cyclictest31379-21timerwakeupswit20:05:300
723099110,10cyclictest28998-21ntpq20:00:210
723099110,10cyclictest0-21swapper/020:26:530
723099110,10cyclictest0-21swapper/019:30:170
723099108,1cyclictest11861-21kworker/0:320:40:080
723099100,1cyclictest0-21swapper/019:50:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional