You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-05-12 - 13:59

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.16.15-rt7 (Profile)

Latency plot of system in rack #b, slot #1
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackbslot1.osadl.org (updated Wed May 12, 2021 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
525525935,8sleep30-21swapper/319:07:303
518725633,7sleep10-21swapper/119:06:571
549525038,7sleep00-21swapper/019:09:440
511624736,7sleep20-21swapper/219:06:092
61662220,0sleep20-21swapper/223:40:132
115242200,0sleep00-21swapper/023:50:090
78222170,1sleep3401rcuc/321:00:073
5657991513,1cyclictest22027-21snmpd19:49:583
5639991412,1cyclictest22027-21snmpd19:52:580
5657991311,1cyclictest22027-21snmpd23:18:083
5657991311,1cyclictest22027-21snmpd21:09:343
5651991311,1cyclictest22027-21snmpd19:50:222
565199130,12cyclictest8558-21memory23:45:092
565199130,12cyclictest4120-31munin-html00:30:302
5657991211,1cyclictest5308-21kworker/3:219:15:293
5651991210,1cyclictest22027-21snmpd22:13:332
5651991210,1cyclictest22027-21snmpd00:36:152
5651991210,1cyclictest0-21swapper/223:00:142
5639991212,0cyclictest22027-21snmpd22:25:540
5639991211,0cyclictest22027-21snmpd19:11:270
563999120,1cyclictest6540-31munin-graph22:45:360
563999120,1cyclictest0-21swapper/022:00:330
565799119,1cyclictest12506-31munin-graph22:55:353
565799119,1cyclictest0-21swapper/323:37:153
565799119,1cyclictest0-21swapper/323:05:473
565799119,1cyclictest0-21swapper/322:40:163
565799119,1cyclictest0-21swapper/320:10:353
5657991111,0cyclictest15048-21kworker/3:119:25:333
5657991110,1cyclictest22027-21snmpd23:21:363
5657991110,0cyclictest22027-21snmpd22:20:063
565799110,1cyclictest0-21swapper/322:27:013
565799110,10cyclictest0-21swapper/322:45:453
565799110,10cyclictest0-21swapper/300:32:383
565199119,1cyclictest22027-21snmpd20:45:512
565199119,1cyclictest0-21swapper/221:50:102
565199119,1cyclictest0-21swapper/220:22:012
565199119,1cyclictest0-21swapper/219:55:132
5651991111,0cyclictest22027-21snmpd22:09:032
5651991110,0cyclictest22027-21snmpd23:29:452
565199110,1cyclictest0-21swapper/221:35:102
565199110,10cyclictest0-21swapper/223:08:002
564399119,1cyclictest22027-21snmpd19:19:031
564399119,1cyclictest0-21swapper/123:49:471
564399119,1cyclictest0-21swapper/123:27:591
564399119,1cyclictest0-21swapper/122:05:091
564399119,1cyclictest0-21swapper/121:18:301
564399119,1cyclictest0-21swapper/121:06:131
564399110,1cyclictest0-21swapper/123:15:241
564399110,10cyclictest0-21swapper/121:39:001
563999119,1cyclictest22027-21snmpd22:40:460
563999119,1cyclictest22027-21snmpd19:25:360
563999119,1cyclictest0-21swapper/020:59:210
563999119,1cyclictest0-21swapper/020:59:210
563999110,10cyclictest0-21swapper/000:15:420
565799109,1cyclictest24473-21kworker/3:020:50:203
565799108,1cyclictest1630-21kworker/3:223:40:413
565799108,1cyclictest0-21swapper/323:32:053
565799108,1cyclictest0-21swapper/320:21:243
565799100,9cyclictest0-21swapper/321:24:033
565199109,1cyclictest22027-21snmpd20:51:452
565199109,0cyclictest22027-21snmpd22:40:492
565199108,1cyclictest0-21swapper/222:30:222
565199100,1cyclictest0-21swapper/221:56:572
565199100,0cyclictest0-21swapper/222:35:032
564399109,0cyclictest0-21swapper/123:20:041
564399109,0cyclictest0-21swapper/122:25:311
564399108,1cyclictest0-21swapper/122:39:521
564399108,1cyclictest0-21swapper/121:31:341
564399108,1cyclictest0-21swapper/121:22:431
564399108,1cyclictest0-21swapper/120:37:291
564399108,1cyclictest0-21swapper/119:30:581
564399108,1cyclictest0-21swapper/100:28:121
564399100,9cyclictest0-21swapper/120:54:461
564399100,9cyclictest0-21swapper/120:19:431
564399100,9cyclictest0-21swapper/119:28:031
564399100,9cyclictest0-21swapper/100:36:261
564399100,1cyclictest0-21swapper/119:23:031
563999109,0cyclictest0-21swapper/020:15:360
563999108,1cyclictest6967-21sed23:40:310
563999108,1cyclictest22027-21snmpd21:16:370
56579998,1cyclictest0-21swapper/323:27:593
56579997,1cyclictest5308-21kworker/3:219:20:313
56579997,1cyclictest31167-21kworker/3:000:25:463
56579997,1cyclictest0-21swapper/322:52:023
56579997,1cyclictest0-21swapper/322:30:533
56579997,1cyclictest0-21swapper/322:05:033
56579997,1cyclictest0-21swapper/320:44:003
56579997,1cyclictest0-21swapper/320:30:173
56579990,8cyclictest0-21swapper/321:58:363
56579990,8cyclictest0-21swapper/320:26:533
56579990,1cyclictest0-21swapper/321:16:473
56579990,1cyclictest0-21swapper/319:14:533
56519997,1cyclictest0-21swapper/223:50:182
56519997,1cyclictest0-21swapper/221:40:392
56519997,1cyclictest0-21swapper/219:35:212
56519997,1cyclictest0-21swapper/200:24:232
56519990,1cyclictest0-21swapper/220:11:122
56439998,1cyclictest22027-21snmpd19:39:161
56439997,1cyclictest0-21swapper/122:53:371
56439997,1cyclictest0-21swapper/121:29:381
56439997,1cyclictest0-21swapper/121:14:181
56439997,1cyclictest0-21swapper/120:31:311
56439997,1cyclictest0-21swapper/100:32:451
56439990,8cyclictest0-21swapper/123:09:351
56439990,8cyclictest0-21swapper/100:16:251
56399998,0cyclictest22027-21snmpd21:49:240
56399997,1cyclictest0-21swapper/021:00:290
56579988,0cyclictest635-40kipmi020:02:323
56579987,1cyclictest22027-21snmpd21:11:383
56579986,1cyclictest0-21swapper/322:19:173
56579986,1cyclictest0-21swapper/321:42:463
56579986,1cyclictest0-21swapper/300:21:103
56579980,7cyclictest0-21swapper/320:56:263
56579980,7cyclictest0-21swapper/320:56:263
56579980,1cyclictest0-21swapper/321:47:563
56519988,0cyclictest22027-21snmpd22:46:012
56519987,1cyclictest22027-21snmpd00:07:552
56519986,1cyclictest0-21swapper/221:13:312
56519986,1cyclictest0-21swapper/220:07:292
56519986,1cyclictest0-21swapper/219:22:362
56519986,1cyclictest0-21swapper/219:13:042
56519980,7cyclictest0-21swapper/223:59:112
56519980,7cyclictest0-21swapper/223:18:292
56519980,7cyclictest0-21swapper/222:55:532
56519980,7cyclictest0-21swapper/221:32:242
56519980,7cyclictest0-21swapper/219:27:072
56519980,7cyclictest0-21swapper/219:17:342
56519980,7cyclictest0-21swapper/200:01:362
56519980,1cyclictest0-21swapper/221:22:432
56519980,1cyclictest0-21swapper/221:02:232
56439987,0cyclictest0-21swapper/123:40:161
56439986,1cyclictest24-21ksoftirqd/122:40:001
56439986,1cyclictest17226-21awk00:00:061
56439986,1cyclictest0-21swapper/123:35:541
56439986,1cyclictest0-21swapper/120:56:421
56439986,1cyclictest0-21swapper/120:56:421
56439986,1cyclictest0-21swapper/120:45:051
56439986,1cyclictest0-21swapper/120:42:001
56439986,1cyclictest0-21swapper/119:13:031
56439985,2cyclictest13250-21/usr/share/muni23:00:011
56399986,1cyclictest7-21ksoftirqd/019:45:000
56399986,1cyclictest0-21swapper/021:55:130
56399986,1cyclictest0-21swapper/020:25:170
56399986,1cyclictest0-21swapper/020:10:360
56399983,4cyclictest15541-21dig22:10:000
56399980,7cyclictest0-21swapper/022:10:540
56399980,4cyclictest6344-21crond21:10:010
56399980,1cyclictest0-21swapper/000:08:460
56579975,1cyclictest42-21ksoftirqd/323:55:003
56579975,1cyclictest42-21ksoftirqd/300:10:003
56579975,1cyclictest42-21ksoftirqd/300:01:003
56579975,1cyclictest22027-21snmpd20:08:253
56579975,1cyclictest16546-21awk00:00:003
56579975,1cyclictest15048-21kworker/3:119:30:353
56579975,1cyclictest0-21swapper/322:12:193
56579975,1cyclictest0-21swapper/322:00:513
56579975,1cyclictest0-21swapper/300:12:373
56579970,6cyclictest0-21swapper/321:36:293
56579970,1cyclictest0-21swapper/323:48:243
56519975,1cyclictest22027-21snmpd20:37:572
56519975,1cyclictest0-21swapper/223:38:212
56519975,1cyclictest0-21swapper/223:33:122
56519975,1cyclictest0-21swapper/223:11:522
56519974,2cyclictest785-21dbus-daemon22:55:012
56519970,6cyclictest0-21swapper/220:32:112
56519970,1cyclictest0-21swapper/200:15:482
56439975,1cyclictest0-21swapper/123:31:331
56439975,1cyclictest0-21swapper/122:23:341
56439975,1cyclictest0-21swapper/122:16:371
56439975,1cyclictest0-21swapper/122:13:331
56439975,1cyclictest0-21swapper/121:55:181
56439975,1cyclictest0-21swapper/121:41:451
56439975,1cyclictest0-21swapper/120:28:071
56439975,1cyclictest0-21swapper/120:11:001
56399976,1cyclictest22027-21snmpd00:28:280
56399976,0cyclictest22027-21snmpd20:05:520
56399975,1cyclictest29716-31munin-html22:30:280
56399975,1cyclictest0-21swapper/023:17:250
56399975,1cyclictest0-21swapper/022:53:300
56399975,1cyclictest0-21swapper/021:54:050
56399975,1cyclictest0-21swapper/020:54:410
56399975,1cyclictest0-21swapper/019:22:390
56399975,1cyclictest0-21swapper/000:10:220
56399970,6cyclictest0-21swapper/023:34:310
56399970,6cyclictest0-21swapper/023:21:460
56399970,6cyclictest0-21swapper/023:06:060
56399970,6cyclictest0-21swapper/021:41:100
56399970,1cyclictest2035-21/usr/share/muni00:30:010
56399970,1cyclictest0-21swapper/022:39:190
56579965,1cyclictest29921-21kworker/3:319:55:113
56579965,1cyclictest22027-21snmpd23:11:023
56579965,1cyclictest21016-21kworker/3:219:42:533
56579965,0cyclictest22027-21snmpd22:38:193
56579965,0cyclictest22027-21snmpd00:37:423
56579964,1cyclictest87450irq/34-eno2-rx-21:30:313
56579964,1cyclictest87450irq/34-eno2-rx-19:35:013
56579964,1cyclictest24473-21kworker/3:020:45:073
56579964,1cyclictest0-21swapper/320:37:333
56579964,1cyclictest0-21swapper/320:19:183
56579963,2cyclictest748-21polkitd21:28:003
56519965,1cyclictest22027-21snmpd22:27:372
56519965,1cyclictest22027-21snmpd22:19:452
56519965,1cyclictest22027-21snmpd20:03:582
56519964,1cyclictest748-21polkitd19:50:002
56519964,1cyclictest22027-21snmpd22:24:362
56519964,1cyclictest0-21swapper/221:06:342
56519964,1cyclictest0-21swapper/220:27:302
56519964,1cyclictest0-21swapper/200:25:102
56519963,2cyclictest785-21dbus-daemon21:30:002
56519963,2cyclictest25711-21awk19:40:162
56439966,0cyclictest22027-21snmpd20:20:531
56439964,1cyclictest28041-21isc-worker000322:47:371
56439964,1cyclictest24-21ksoftirqd/120:00:001
56439964,1cyclictest24-21ksoftirqd/119:45:001
56439964,1cyclictest24-21ksoftirqd/100:25:001
56439964,1cyclictest22027-21snmpd00:07:501
56439964,1cyclictest15785-31munin-html23:55:341
56439964,1cyclictest0-21swapper/121:45:191
56439963,2cyclictest31726-21awk19:50:151
56439963,2cyclictest22870-21ntpd23:50:211
56439961,2cyclictest5998-31munin-graph21:50:341
56439960,5cyclictest3482-21kworker/1:220:00:321
56399965,1cyclictest22027-21snmpd19:48:250
56399965,1cyclictest22027-21snmpd19:31:480
56399965,1cyclictest22027-21snmpd00:35:320
56399965,0cyclictest22027-21snmpd21:29:250
56399964,1cyclictest26091-21munin-limits21:30:280
56399964,1cyclictest14909-21/usr/sbin/munin23:55:330
56399964,1cyclictest0-21swapper/023:48:530
56399964,1cyclictest0-21swapper/023:27:450
56399964,1cyclictest0-21swapper/021:21:380
56399964,1cyclictest0-21swapper/020:45:190
56399964,1cyclictest0-21swapper/019:56:520
56399964,1cyclictest0-21swapper/000:21:490
56399963,2cyclictest801-21gdbus00:00:000
56399963,2cyclictest27167-21/usr/share/muni21:35:160
56399963,2cyclictest23058-21ping19:35:210
56399963,2cyclictest22027-21snmpd23:36:570
56399963,2cyclictest22027-21snmpd23:14:200
56399963,2cyclictest22027-21snmpd22:59:490
56399963,2cyclictest22027-21snmpd22:21:390
56399963,2cyclictest22027-21snmpd21:10:280
56399963,2cyclictest19193-21awk22:15:060
56399960,5cyclictest27468-21munin-update20:40:000
56399960,1cyclictest0-21swapper/019:18:070
3136260,1sleep20-21swapper/221:46:492
27744260,1sleep20-21swapper/223:22:482
26991260,1sleep00-21swapper/020:35:380
25373260,1sleep20-21swapper/200:13:312
15259260,1sleep20-21swapper/220:17:342
56579953,1cyclictest30192-21/usr/share/muni19:50:023
56579953,1cyclictest25875-21gzip00:15:033
56579953,1cyclictest22027-21snmpd21:50:123
56579953,1cyclictest13962-21cat23:00:093
56519953,1cyclictest785-21dbus-daemon20:40:002
56519953,1cyclictest4606-21cat20:55:062
56519953,1cyclictest4606-21cat20:55:062
56519953,1cyclictest19768-21cat19:30:182
56519953,1cyclictest16319-21/usr/sbin/munin21:15:102
56519953,1cyclictest0-21swapper/222:00:012
56439953,1cyclictest8357-21cat20:05:171
56439953,1cyclictest7299-21/usr/share/muni21:00:131
56439953,1cyclictest23296-21cat00:10:061
56439953,1cyclictest22027-21snmpd23:10:231
56439953,1cyclictest10459-21ntpq22:00:101
56439952,2cyclictest22870-21ntpd22:30:031
56439950,2cyclictest27204-21/usr/sbin/munin19:45:011
56439950,2cyclictest10266-21/usr/share/muni22:55:001
56399953,1cyclictest4491-21cat20:00:090
56399953,1cyclictest16495-21cat20:20:070
56399952,2cyclictest22244-21isc-worker000020:30:060
56399952,2cyclictest13253-21/usr/sbin/munin23:00:060
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional