You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-09-29 - 04:30

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.16.15-rt7 (Profile)

Latency plot of system in rack #b, slot #1
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackbslot1.osadl.org (updated Wed Sep 29, 2021 00:44:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
399726238,7sleep00-21swapper/019:08:000
376925934,8sleep10-21swapper/119:05:561
405225139,7sleep30-21swapper/319:08:273
383824836,7sleep20-21swapper/219:06:252
127712190,0sleep00-21swapper/020:15:150
209382170,1sleep120869-21munin-update21:25:001
4345991614,1cyclictest6584-21snmpd22:56:523
4325991614,1cyclictest6584-21snmpd00:12:370
4345991311,1cyclictest6584-21snmpd19:32:053
4345991311,1cyclictest6584-21snmpd00:36:073
4338991311,1cyclictest6584-21snmpd23:20:022
4338991311,1cyclictest22159-31munin-html19:35:322
4332991311,1cyclictest6584-21snmpd22:53:041
4325991312,0cyclictest6584-21snmpd22:52:220
4325991311,1cyclictest6584-21snmpd19:24:560
4345991210,1cyclictest8638-21diskstats22:50:183
4345991210,1cyclictest0-21swapper/322:15:073
4345991210,1cyclictest0-21swapper/319:25:353
434599120,11cyclictest0-21swapper/323:15:393
433899120,0cyclictest23996-21latency_hist21:30:002
433299121,4cyclictest24-21ksoftirqd/121:10:001
4332991210,1cyclictest418-31munin-html22:35:321
433299120,11cyclictest0-21swapper/123:15:161
4325991211,0cyclictest6584-21snmpd20:54:040
4325991210,1cyclictest6584-21snmpd21:48:420
4325991210,1cyclictest0-21swapper/021:05:180
432599120,1cyclictest6825-31munin-html23:40:280
432599120,1cyclictest23997-21latency_hist21:30:000
432599120,11cyclictest9841-31munin-html19:15:310
434599119,1cyclictest0-21swapper/321:41:033
434599110,10cyclictest0-21swapper/323:55:033
434599110,10cyclictest0-21swapper/321:34:453
434599110,10cyclictest0-21swapper/321:15:513
434599110,10cyclictest0-21swapper/300:12:303
433899119,1cyclictest6584-21snmpd20:03:512
433899119,1cyclictest0-21swapper/220:45:262
433899110,10cyclictest0-21swapper/222:30:432
433899110,10cyclictest0-21swapper/200:00:092
433299119,1cyclictest0-21swapper/123:00:351
433299119,1cyclictest0-21swapper/120:40:351
433299119,1cyclictest0-21swapper/100:10:481
433299110,10cyclictest0-21swapper/123:23:491
433299110,10cyclictest0-21swapper/121:12:041
433299110,10cyclictest0-21swapper/120:27:491
433299110,10cyclictest0-21swapper/100:35:201
432599119,1cyclictest6584-21snmpd22:31:130
432599119,1cyclictest0-21swapper/021:51:000
432599119,1cyclictest0-21swapper/019:59:550
432599113,4cyclictest26279-21perf20:40:000
432599110,1cyclictest13808-21/usr/share/muni23:55:090
432599110,0cyclictest17938-31munin-graph22:10:420
434599108,1cyclictest0-21swapper/323:01:463
434599108,1cyclictest0-21swapper/300:00:523
434599100,9cyclictest0-21swapper/322:22:423
434599100,1cyclictest0-21swapper/300:21:123
433899108,1cyclictest33-21ksoftirqd/222:30:012
433899108,1cyclictest0-21swapper/223:15:362
433899108,1cyclictest0-21swapper/220:25:152
433899100,9cyclictest0-21swapper/221:57:072
433299108,1cyclictest0-21swapper/123:56:171
433299108,1cyclictest0-21swapper/123:45:271
433299108,1cyclictest0-21swapper/120:32:401
433299108,1cyclictest0-21swapper/119:28:041
433299100,9cyclictest0-21swapper/123:07:301
433299100,9cyclictest0-21swapper/122:43:561
433299100,9cyclictest0-21swapper/122:08:051
433299100,1cyclictest0-21swapper/119:50:111
432599109,0cyclictest0-21swapper/022:15:040
432599108,1cyclictest2097-21kworker/0:000:30:430
432599100,1cyclictest0-21swapper/020:41:540
5362291,1sleep24253-21/usr/share/muni22:45:152
43459998,0cyclictest0-21swapper/319:21:043
43459997,1cyclictest0-21swapper/323:40:023
43459997,1cyclictest0-21swapper/320:37:163
43459997,1cyclictest0-21swapper/320:32:543
43459997,1cyclictest0-21swapper/320:15:483
43459990,1cyclictest0-21swapper/323:07:553
43459990,1cyclictest0-21swapper/319:58:123
43389998,1cyclictest6584-21snmpd21:06:502
43389997,1cyclictest0-21swapper/223:41:452
43389990,8cyclictest0-21swapper/223:53:032
43329998,1cyclictest6584-21snmpd20:45:431
43329997,1cyclictest0-21swapper/123:53:211
43329997,1cyclictest0-21swapper/123:37:321
43329997,1cyclictest0-21swapper/123:34:381
43329997,1cyclictest0-21swapper/122:27:081
43329997,1cyclictest0-21swapper/122:10:501
43329997,1cyclictest0-21swapper/121:31:161
43329997,1cyclictest0-21swapper/120:21:021
43329997,1cyclictest0-21swapper/120:11:211
43329991,1cyclictest101rcu_sched21:00:001
43329990,1cyclictest0-21swapper/122:02:361
43329990,1cyclictest0-21swapper/100:27:251
43259997,1cyclictest6584-21snmpd22:03:240
43259997,1cyclictest6584-21snmpd21:20:320
43259997,1cyclictest22241-21kworker/0:220:00:270
43259997,1cyclictest20953-21kworker/0:222:20:320
43259990,8cyclictest0-21swapper/021:36:280
7194280,0sleep20-21swapper/221:00:142
43459987,0cyclictest0-21swapper/323:50:033
43459987,0cyclictest0-21swapper/323:45:033
43459986,1cyclictest0-21swapper/322:47:533
43459986,1cyclictest0-21swapper/322:30:173
43459986,1cyclictest0-21swapper/322:11:333
43459986,1cyclictest0-21swapper/322:02:313
43459986,1cyclictest0-21swapper/321:47:403
43459986,1cyclictest0-21swapper/320:42:263
43459986,1cyclictest0-21swapper/320:21:163
43459986,1cyclictest0-21swapper/300:16:123
43459980,7cyclictest0-21swapper/321:11:103
43459980,7cyclictest0-21swapper/320:52:273
43459980,1cyclictest0-21swapper/323:27:073
43459980,1cyclictest0-21swapper/322:08:493
43389986,1cyclictest6584-21snmpd21:45:052
43389986,1cyclictest0-21swapper/200:36:012
43329987,0cyclictest231ktimersoftd/120:54:091
43329986,1cyclictest28845-21/usr/share/muni00:20:011
43329986,1cyclictest24-21ksoftirqd/121:35:001
43329986,1cyclictest24-21ksoftirqd/120:05:011
43329986,1cyclictest24-21ksoftirqd/119:15:011
43329986,1cyclictest0-21swapper/121:57:061
43329986,1cyclictest0-21swapper/121:41:071
43329986,1cyclictest0-21swapper/119:57:091
43329986,1cyclictest0-21swapper/119:41:101
43329980,7cyclictest0-21swapper/121:16:451
43329980,7cyclictest0-21swapper/119:34:521
43329980,7cyclictest0-21swapper/100:19:311
43329980,1cyclictest101rcu_sched00:05:001
43329980,1cyclictest0-21swapper/121:47:451
43329980,1cyclictest0-21swapper/120:18:551
43329980,1cyclictest0-21swapper/119:47:561
43259987,0cyclictest19040-31munin-graph00:00:590
43259986,1cyclictest7-21ksoftirqd/019:50:000
43259984,3cyclictest801-21gdbus23:17:000
43259980,8cyclictest0-21swapper/022:06:100
20368280,6sleep220361-31munin-graph21:20:412
43459975,1cyclictest6584-21snmpd20:29:583
43459975,1cyclictest0-21swapper/323:35:023
43459975,1cyclictest0-21swapper/321:24:053
43459975,1cyclictest0-21swapper/320:01:063
43459975,1cyclictest0-21swapper/319:11:423
43459975,1cyclictest0-21swapper/300:09:353
43459974,2cyclictest6584-21snmpd21:50:363
43459970,6cyclictest27075-21sshd23:20:243
43459970,6cyclictest0-21swapper/320:13:423
43459970,1cyclictest0-21swapper/323:32:083
43389976,1cyclictest6584-21snmpd23:37:572
43389975,1cyclictest22612-31munin-graph20:30:342
43389975,1cyclictest0-21swapper/220:09:352
43389975,1cyclictest0-21swapper/219:59:062
43389975,1cyclictest0-21swapper/219:40:022
43389975,1cyclictest0-21swapper/219:13:242
43389975,1cyclictest0-21swapper/200:13:532
43389974,2cyclictest24590-21sort21:30:072
43389970,6cyclictest0-21swapper/223:14:472
43389970,6cyclictest0-21swapper/223:09:272
43389970,6cyclictest0-21swapper/223:04:082
43389970,6cyclictest0-21swapper/222:58:482
43389970,6cyclictest0-21swapper/221:54:232
43389970,6cyclictest0-21swapper/221:44:122
43389970,6cyclictest0-21swapper/221:39:322
43389970,6cyclictest0-21swapper/219:19:132
43389970,1cyclictest0-21swapper/221:18:322
43389970,1cyclictest0-21swapper/219:49:342
43389970,1cyclictest0-21swapper/200:23:532
43329975,1cyclictest6584-21snmpd00:05:131
43329975,1cyclictest24-21ksoftirqd/123:10:011
43329975,1cyclictest24-21ksoftirqd/119:40:011
43329975,1cyclictest24-21ksoftirqd/119:10:011
43329974,2cyclictest0-21swapper/122:45:311
43329970,4cyclictest748-21polkitd22:33:001
43259977,0cyclictest10690-21kworker/0:219:25:480
43259975,1cyclictest7-21ksoftirqd/023:50:000
43259975,1cyclictest7-21ksoftirqd/023:25:010
43259975,1cyclictest6584-21snmpd22:28:010
43259975,1cyclictest6584-21snmpd19:53:230
43259975,1cyclictest0-21swapper/019:36:200
43259975,1cyclictest0-21swapper/019:31:210
43259975,1cyclictest0-21swapper/019:11:280
43259974,2cyclictest801-21gdbus21:05:000
43259970,7cyclictest0-21swapper/020:32:410
43259970,6cyclictest0-21swapper/021:19:510
43259970,6cyclictest0-21swapper/020:21:530
43259970,1cyclictest0-21swapper/019:41:210
11045270,1sleep111126-21hddtemp_smartct22:55:111
43459966,0cyclictest6584-21snmpd20:49:193
43459964,1cyclictest32549-31munin-html22:35:333
43459964,1cyclictest26681-31munin-html22:25:313
43459964,1cyclictest1349-21grep00:25:303
43459964,1cyclictest0-21swapper/321:38:183
43459964,1cyclictest0-21swapper/319:52:133
43459964,1cyclictest0-21swapper/319:45:263
43459964,1cyclictest0-21swapper/300:32:023
43459963,2cyclictest6584-21snmpd22:43:053
43459963,2cyclictest6584-21snmpd21:57:183
43459963,2cyclictest6584-21snmpd21:02:433
43459963,2cyclictest6584-21snmpd20:57:343
43389965,0cyclictest9108-31munin-html22:50:352
43389965,0cyclictest0-21swapper/220:20:342
43389964,1cyclictest801-21gdbus22:17:002
43389964,1cyclictest6584-21snmpd23:31:092
43389964,1cyclictest33-21ksoftirqd/222:01:012
43389964,1cyclictest0-21swapper/222:21:402
43389964,1cyclictest0-21swapper/220:37:512
43389963,2cyclictest87450irq/34-eno2-rx-22:10:322
43389963,2cyclictest6584-21snmpd22:05:002
43389963,2cyclictest6584-21snmpd21:12:082
43389963,2cyclictest6584-21snmpd20:59:072
43389963,2cyclictest6584-21snmpd20:50:312
43389963,2cyclictest6584-21snmpd19:51:442
43389963,2cyclictest6584-21snmpd19:32:442
43389963,2cyclictest6584-21snmpd19:28:052
43389963,2cyclictest6584-21snmpd00:08:572
43389963,2cyclictest0-21swapper/219:22:342
43389962,3cyclictest801-21gdbus23:29:552
43389961,1cyclictest91rcu_preempt00:25:302
43389961,1cyclictest101rcu_sched00:00:002
43389960,5cyclictest4010-21perf22:45:002
43389960,1cyclictest13548-31munin-html20:15:322
43389960,1cyclictest0-21swapper/200:31:002
43329965,1cyclictest6584-21snmpd22:17:331
43329965,0cyclictest0-21swapper/100:30:201
43329964,1cyclictest24-21ksoftirqd/123:25:001
43329964,1cyclictest24-21ksoftirqd/122:25:001
43329964,1cyclictest0-21swapper/123:40:271
43329963,2cyclictest6584-21snmpd21:27:291
43329963,2cyclictest6584-21snmpd20:39:221
43329960,1cyclictest91rcu_preempt20:55:321
43259965,0cyclictest6584-21snmpd21:14:170
43259964,1cyclictest6584-21snmpd20:56:340
43259964,1cyclictest1237-21runrttasks23:10:350
43259964,1cyclictest0-21swapper/020:26:340
43259964,1cyclictest0-21swapper/000:29:240
43259964,1cyclictest0-21swapper/000:16:000
43259964,1cyclictest0-21swapper/000:08:060
43259963,2cyclictest8183-21awk21:55:220
43259963,2cyclictest6584-21snmpd23:52:480
43259963,2cyclictest6584-21snmpd23:30:170
43259963,2cyclictest6584-21snmpd23:09:350
43259963,2cyclictest6584-21snmpd23:04:320
43259963,2cyclictest6584-21snmpd20:45:190
43259963,2cyclictest15523-21kworker/0:023:20:320
43259963,2cyclictest10063-21awk20:10:180
43259962,3cyclictest6257-31munin-html22:45:330
43259961,4cyclictest30937-21cat21:40:110
43259961,1cyclictest7-21ksoftirqd/022:35:000
43259961,1cyclictest24098-21mailstats21:29:590
43259960,5cyclictest1842-21/usr/share/muni23:35:000
43259960,1cyclictest1-21systemd22:45:000
10905260,1sleep20-21swapper/220:11:252
43459953,1cyclictest87150irq/30-eno1-rx-19:35:323
43459953,1cyclictest785-21dbus-daemon23:10:003
43459953,1cyclictest23026-21proc_pri21:25:313
43459953,1cyclictest2151-21kworker/3:120:05:133
43459952,2cyclictest26758-21openvpn19:15:003
43459952,2cyclictest23453-21ping19:40:073
43459950,4cyclictest9138-21/usr/share/muni21:05:013
43389953,1cyclictest7832-21/usr/sbin/munin23:45:062
43389953,1cyclictest0-21swapper/222:35:282
43389952,2cyclictest26486-21/usr/share/muni20:40:092
43389952,2cyclictest0-21swapper/200:15:052
43329952,2cyclictest801-21gdbus19:20:011
43329952,2cyclictest2499-21/usr/share/muni20:00:121
43329950,1cyclictest91rcu_preempt21:50:011
43259953,1cyclictest29558-21cat00:20:080
43259952,2cyclictest26758-21openvpn22:55:070
43259952,2cyclictest0-21swapper/020:07:290
43259952,2cyclictest0-21swapper/000:35:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional