You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-17 - 05:38

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.16.15-rt7 (Profile)

Latency plot of system in rack #b, slot #1
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rackbslot1.osadl.org (updated Sun Oct 17, 2021 00:44:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
138932630,0sleep30-21swapper/321:18:203
26662610,0sleep20-21swapper/221:00:052
3234326036,7sleep30-21swapper/319:05:543
280572580,0sleep30-21swapper/321:40:193
3265425633,7sleep20-21swapper/219:08:412
3256925038,7sleep00-21swapper/019:07:510
3237724836,7sleep10-21swapper/119:06:041
468991912,6cyclictest6584-21snmpd23:27:333
460991513,1cyclictest6584-21snmpd22:58:532
447991514,1cyclictest6584-21snmpd21:42:250
460991414,0cyclictest6584-21snmpd21:54:212
447991414,0cyclictest6584-21snmpd23:57:520
468991311,1cyclictest13765-31munin-html22:10:293
46099130,12cyclictest11442-31munin-html23:00:322
447991311,1cyclictest6584-21snmpd20:23:220
46899125,6cyclictest6584-21snmpd22:16:373
46899120,7cyclictest1-21systemd20:35:003
46899120,1cyclictest14858-31munin-html00:00:273
46899120,1cyclictest0-21swapper/320:15:033
46899120,11cyclictest30070-31munin-html19:55:323
46899120,11cyclictest0-21swapper/321:20:173
46899120,0cyclictest0-21swapper/300:35:043
460991210,1cyclictest9148-21awk21:10:102
46099120,11cyclictest0-21swapper/219:35:382
46899119,1cyclictest892-31munin-graph00:30:493
46899119,1cyclictest21952-31munin-graph20:35:433
46899119,1cyclictest0-21swapper/323:45:173
46899119,1cyclictest0-21swapper/320:10:223
46899115,5cyclictest6584-21snmpd22:49:203
46899115,5cyclictest6584-21snmpd21:06:113
46899115,5cyclictest6584-21snmpd19:32:553
46899114,6cyclictest6584-21snmpd20:03:303
46899114,6cyclictest6584-21snmpd19:48:513
46899114,6cyclictest6584-21snmpd00:26:263
46899114,6cyclictest6584-21snmpd00:16:473
46899113,7cyclictest6584-21snmpd23:51:043
46899113,7cyclictest6584-21snmpd23:36:573
46899113,7cyclictest6584-21snmpd21:36:373
46899113,7cyclictest6584-21snmpd20:20:503
46899113,7cyclictest6584-21snmpd19:11:353
46899110,8cyclictest4487-21expr21:55:183
46899110,8cyclictest17083-21cstates19:35:123
46899110,7cyclictest21084-21grep22:25:063
46899110,7cyclictest17025-21cat00:05:123
46899110,6cyclictest14445-21cut23:05:343
46099119,1cyclictest0-21swapper/223:17:582
46099110,1cyclictest0-21swapper/221:40:162
46099110,10cyclictest0-21swapper/222:21:472
46099110,10cyclictest0-21swapper/221:07:302
46099110,10cyclictest0-21swapper/220:55:442
46099110,10cyclictest0-21swapper/220:41:012
45499119,1cyclictest9184-31munin-graph19:20:341
45499119,1cyclictest6584-21snmpd20:02:301
45499119,1cyclictest0-21swapper/119:30:441
45499110,1cyclictest0-21swapper/123:20:391
45499110,10cyclictest0-21swapper/120:10:371
45499110,10cyclictest0-21swapper/119:15:431
44799119,1cyclictest7741-31munin-html22:00:320
44799119,1cyclictest6584-21snmpd22:31:350
44799119,1cyclictest30296-21smart_sda22:40:050
44799119,1cyclictest0-21swapper/020:59:010
44799119,1cyclictest0-21swapper/019:40:520
44799119,1cyclictest0-21swapper/019:37:000
44799119,1cyclictest0-21swapper/000:00:400
44799113,4cyclictest16507-21dig20:30:010
44799110,10cyclictest0-21swapper/023:00:080
44799110,10cyclictest0-21swapper/021:20:200
44799110,10cyclictest0-21swapper/019:45:140
46899108,1cyclictest0-21swapper/320:54:563
46899103,6cyclictest7973-21awk19:20:103
46899103,6cyclictest748-21polkitd23:58:013
46899103,6cyclictest7390-21awk22:00:213
46899103,6cyclictest6584-21snmpd23:43:113
46899103,6cyclictest6584-21snmpd23:10:543
46899103,6cyclictest6584-21snmpd23:03:533
46899103,6cyclictest6584-21snmpd22:55:503
46899103,6cyclictest6584-21snmpd22:54:513
46899103,6cyclictest6584-21snmpd22:37:273
46899103,6cyclictest6584-21snmpd22:20:493
46899103,6cyclictest6584-21snmpd21:47:313
46899103,6cyclictest6584-21snmpd21:26:243
46899103,6cyclictest6584-21snmpd21:10:393
46899103,6cyclictest6584-21snmpd20:57:353
46899103,6cyclictest6584-21snmpd20:42:343
46899103,6cyclictest6584-21snmpd19:41:093
46899103,6cyclictest6584-21snmpd19:29:123
46899103,6cyclictest3303-21cat20:05:183
46899103,6cyclictest20025-21awk23:15:183
46899100,9cyclictest4704-21ping21:01:223
46899100,9cyclictest3874-21/usr/share/muni19:15:013
46899100,9cyclictest25052-21/usr/share/muni00:20:013
46899100,8cyclictest6990-21munin-node19:50:003
46899100,6cyclictest24821-21cat22:30:123
46899100,6cyclictest22255-21if_enp0s26u1u423:20:093
46899100,6cyclictest0-21swapper/321:30:103
46899100,1cyclictest0-21swapper/320:27:293
46099109,1cyclictest6584-21snmpd23:46:252
46099108,1cyclictest0-21swapper/220:13:542
46099108,1cyclictest0-21swapper/200:37:242
46099100,9cyclictest0-21swapper/222:37:072
45499108,1cyclictest6584-21snmpd22:51:261
45499108,1cyclictest0-21swapper/122:09:561
45499108,1cyclictest0-21swapper/121:52:011
45499108,1cyclictest0-21swapper/119:55:461
45499108,1cyclictest0-21swapper/119:25:541
45499108,1cyclictest0-21swapper/119:10:521
45499100,9cyclictest0-21swapper/123:15:591
45499100,9cyclictest0-21swapper/100:39:461
44799109,0cyclictest0-21swapper/020:10:060
44799108,1cyclictest0-21swapper/021:07:160
44799108,1cyclictest0-21swapper/020:39:480
44799108,1cyclictest0-21swapper/020:01:230
44799107,2cyclictest0-21swapper/023:16:250
44799100,9cyclictest0-21swapper/023:30:080
44799100,9cyclictest0-21swapper/023:21:060
44799100,9cyclictest0-21swapper/022:15:520
44799100,9cyclictest0-21swapper/020:06:130
44799100,1cyclictest0-21swapper/022:07:490
4689992,6cyclictest6584-21snmpd22:41:413
4689992,6cyclictest27453-21/usr/share/muni23:30:013
4689992,6cyclictest10292-21expr22:05:153
4689991,7cyclictest32266-21/usr/share/muni21:50:003
4689990,8cyclictest20883-21ntp_states00:10:313
4689990,5cyclictest0-21swapper/320:45:113
4609997,1cyclictest0-21swapper/223:50:352
4609997,1cyclictest0-21swapper/223:35:332
4609997,1cyclictest0-21swapper/222:25:392
4609997,1cyclictest0-21swapper/221:23:202
4609997,1cyclictest0-21swapper/220:51:212
4609990,8cyclictest0-21swapper/220:07:082
4549998,0cyclictest0-21swapper/123:00:481
4549997,1cyclictest0-21swapper/120:47:161
4549990,1cyclictest0-21swapper/100:06:211
4479998,1cyclictest6584-21snmpd23:38:340
4479997,1cyclictest25341-21kworker/0:220:45:300
4479997,1cyclictest24780-21kworker/0:100:25:280
4479997,1cyclictest0-21swapper/023:53:050
4479997,1cyclictest0-21swapper/023:43:320
4479997,1cyclictest0-21swapper/023:14:300
4479997,1cyclictest0-21swapper/023:07:520
4609986,1cyclictest6584-21snmpd21:36:242
4609986,1cyclictest0-21swapper/223:27:202
4609986,1cyclictest0-21swapper/223:24:352
4609986,1cyclictest0-21swapper/222:42:262
4609986,1cyclictest0-21swapper/221:17:212
4609980,7cyclictest0-21swapper/220:36:402
4549986,1cyclictest16584-21smartctl19:35:061
4549986,1cyclictest0-21swapper/123:55:021
4549986,1cyclictest0-21swapper/121:18:551
4549986,1cyclictest0-21swapper/121:12:561
4549986,1cyclictest0-21swapper/120:50:491
4549986,1cyclictest0-21swapper/120:32:051
4549986,1cyclictest0-21swapper/119:40:541
4549980,1cyclictest0-21swapper/120:56:001
4479986,1cyclictest23122-31munin-html22:25:330
4479986,1cyclictest0-21swapper/022:23:380
4479986,1cyclictest0-21swapper/021:51:490
4479986,1cyclictest0-21swapper/019:53:580
4479980,7cyclictest0-21swapper/022:52:220
8162270,1sleep18207-21host22:01:001
4609976,1cyclictest6584-21snmpd22:52:332
4609976,0cyclictest0-21swapper/223:30:052
4609975,1cyclictest6584-21snmpd21:28:212
4609975,1cyclictest656-21/usr/share/muni19:10:012
4609975,1cyclictest0-21swapper/222:46:392
4609975,1cyclictest0-21swapper/222:15:572
4609975,1cyclictest0-21swapper/222:07:532
4609975,1cyclictest0-21swapper/219:53:332
4609975,1cyclictest0-21swapper/219:31:452
4549976,0cyclictest0-21swapper/122:25:261
4549975,1cyclictest24-21ksoftirqd/100:14:591
4549975,1cyclictest0-21swapper/123:26:091
4549975,1cyclictest0-21swapper/123:11:181
4549975,1cyclictest0-21swapper/122:33:301
4549975,1cyclictest0-21swapper/120:44:321
4549975,1cyclictest0-21swapper/100:34:171
4549970,6cyclictest0-21swapper/122:56:551
4549970,6cyclictest0-21swapper/120:36:271
4549970,3cyclictest32666-31munin-html23:35:311
4549970,3cyclictest15270-21dig23:10:001
4549970,1cyclictest0-21swapper/121:39:351
4479976,1cyclictest6584-21snmpd22:57:270
4479976,1cyclictest6584-21snmpd20:40:590
4479976,0cyclictest0-21swapper/022:35:530
4479975,1cyclictest7-21ksoftirqd/019:20:000
4479975,1cyclictest15790-21isc-worker000121:56:530
4479975,1cyclictest0-21swapper/021:14:210
4479975,1cyclictest0-21swapper/020:53:040
4479975,1cyclictest0-21swapper/019:33:070
4479972,1cyclictest0-21swapper/021:35:130
4479970,2cyclictest0-21swapper/000:34:550
4609965,1cyclictest6584-21snmpd20:45:112
4609965,1cyclictest6584-21snmpd19:45:272
4609965,1cyclictest6584-21snmpd19:24:082
4609964,1cyclictest9843-31munin-html20:15:312
4609964,1cyclictest87450irq/34-eno2-rx-20:25:322
4609964,1cyclictest6588-21cat22:00:102
4609964,1cyclictest17301-21tr00:05:212
4609964,1cyclictest0-21swapper/222:10:582
4609963,2cyclictest6584-21snmpd22:31:072
4609963,2cyclictest6584-21snmpd21:58:372
4609963,2cyclictest6584-21snmpd21:31:392
4609963,2cyclictest6584-21snmpd19:41:402
4609963,2cyclictest6584-21snmpd00:13:202
4609963,2cyclictest0-21swapper/223:05:332
4549965,1cyclictest6584-21snmpd21:45:491
4549965,1cyclictest6584-21snmpd20:08:361
4549964,1cyclictest4520-21/usr/sbin/munin21:00:301
4549964,1cyclictest24-21ksoftirqd/121:25:011
4549964,1cyclictest0-21swapper/122:43:021
4549963,2cyclictest6584-21snmpd22:22:051
4549963,2cyclictest6584-21snmpd22:10:111
4549963,2cyclictest6584-21snmpd21:29:091
4549963,2cyclictest6584-21snmpd21:07:381
4549963,2cyclictest6584-21snmpd20:17:141
4549963,2cyclictest6584-21snmpd19:46:541
4549963,2cyclictest23043-21crond21:35:001
4549961,1cyclictest24003-21kworker/1:323:50:311
4549961,1cyclictest24003-21kworker/1:323:30:321
4549960,1cyclictest30070-31munin-html00:25:301
4549960,1cyclictest27084-31munin-html19:50:311
4549960,1cyclictest25554-21kworker/1:221:40:321
4549960,1cyclictest24061-31munin-html00:15:301
4549960,1cyclictest2314-31munin-html22:45:321
4479965,1cyclictest6584-21snmpd00:07:420
4479965,0cyclictest6584-21snmpd21:45:540
4479964,1cyclictest7-21ksoftirqd/021:25:300
4479964,1cyclictest22273-31munin-html21:30:290
4479964,1cyclictest0-21swapper/021:04:300
4479964,1cyclictest0-21swapper/020:31:050
4479963,2cyclictest785-21dbus-daemon19:12:010
4479963,2cyclictest6584-21snmpd23:48:340
4479963,2cyclictest6584-21snmpd22:48:540
4479963,2cyclictest6584-21snmpd22:11:520
4479963,2cyclictest6584-21snmpd20:17:590
4479963,2cyclictest6584-21snmpd00:35:560
4479963,2cyclictest6584-21snmpd00:12:410
14053260,1sleep00-21swapper/021:19:590
4609953,1cyclictest6584-21snmpd00:26:142
4609953,1cyclictest3844-21if_err_eno219:15:012
4609953,1cyclictest29810-21ntpq21:45:082
4609953,1cyclictest29208-21/usr/sbin/munin19:55:282
4609953,1cyclictest2170-21/usr/sbin/munin23:40:312
4609953,1cyclictest17433-21cat20:30:102
4609953,1cyclictest16099-21httpd20:20:202
4609953,1cyclictest10323-21cat23:55:062
4609953,1cyclictest0-21swapper/223:10:052
4609953,1cyclictest0-21swapper/200:14:592
4609952,2cyclictest745-21systemd-logind00:31:002
4609952,2cyclictest31092-21/usr/sbin/munin20:00:112
4609952,2cyclictest12755-21/usr/share/muni00:00:042
4609952,2cyclictest11982-21ping19:25:212
4609950,2cyclictest25580-21ntp_states00:20:072
4549955,0cyclictest6584-21snmpd21:55:111
4549953,1cyclictest1566-21in:imjournal00:20:061
4549953,1cyclictest14487-21cat00:00:181
4549953,1cyclictest10956-21needreboot20:20:051
4549952,1cyclictest91rcu_preempt23:45:031
4549950,1cyclictest27359-21if_eno222:35:071
4549950,1cyclictest16931-31munin-html22:15:281
4549950,1cyclictest14117-21latency20:25:061
4549950,1cyclictest1039-21/usr/share/muni23:40:011
4479953,1cyclictest7-21ksoftirqd/000:20:000
4479953,1cyclictest6584-21snmpd19:25:020
4479953,1cyclictest6584-21snmpd00:16:210
4479953,1cyclictest24630-21vmstat23:25:040
4479952,2cyclictest28099-21/usr/share/muni19:55:090
4479952,1cyclictest0-21swapper/019:20:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional