You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-22 - 03:07

Intel(R) Core(TM) i5-8365UE CPU @ 1.60GHz, Linux 5.10.41-rt42 (Profile)

Latency plot of system in rack #a, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8.osadl.org (updated Fri Oct 22, 2021 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
134692302273,19sleep20-21swapper/219:08:302
134352281219,21sleep30-21swapper/319:08:073
132302266213,18sleep10-21swapper/119:05:391
131952262238,15sleep00-21swapper/019:05:260
175312700,0sleep00-21swapper/020:25:230
88522610,0sleep10-21swapper/122:51:481
258452610,0sleep30-21swapper/323:41:473
263702600,0sleep20-21swapper/222:02:522
250582570,0sleep10-21swapper/122:42:001
31902560,0sleep00-21swapper/023:08:170
58462550,0sleep30-21swapper/323:50:043
102792550,0sleep00-21swapper/021:33:360
167832540,0sleep10-21swapper/119:15:171
316542170,0sleep10-21swapper/123:45:301
239472150,0sleep30-21swapper/323:20:253
88302130,0sleep00-21swapper/022:51:470
10772130,0sleep20-21swapper/222:47:162
1369199109,1cyclictest22-21ksoftirqd/123:20:001
137009997,2cyclictest21924-21kworker/3:200:29:433
136879991,4cyclictest283-21systemd-journal19:40:000
136969980,5cyclictest20893-21basename22:20:012
136919987,1cyclictest22-21ksoftirqd/100:00:011
136919983,3cyclictest283-21systemd-journal21:45:341
136879985,2cyclictest5216-21sh23:49:450
136879983,3cyclictest283-21systemd-journal22:35:530
11445280,0sleep20-21swapper/200:33:552
6893270,0sleep20-21swapper/222:10:252
27515270,0sleep00-21swapper/023:23:140
137009973,3cyclictest665-21snmpd22:54:263
137009973,3cyclictest665-21snmpd19:44:503
137009973,3cyclictest5612-21df22:30:173
137009973,3cyclictest283-21systemd-journal22:11:373
137009973,3cyclictest283-21systemd-journal21:10:013
137009973,1cyclictest0-21swapper/323:15:323
137009971,5cyclictest13759-21tr21:35:223
137009971,2cyclictest22856-21irqstats23:00:163
137009970,1cyclictest0-21swapper/323:29:063
136969973,3cyclictest23617-21cat20:40:172
136969973,1cyclictest0-21swapper/223:20:092
136969972,3cyclictest25458-21munin-run19:35:002
136969970,4cyclictest28427-21basename21:45:002
136969970,2cyclictest20074-21sh22:39:272
136919974,2cyclictest30824-21ntpq00:25:231
136919973,3cyclictest6060-21cat00:30:181
136919973,3cyclictest31641-21sensors_temp00:05:251
136919973,3cyclictest30370-21ntp_states22:45:201
136919973,3cyclictest283-21systemd-journal21:08:351
136919973,3cyclictest283-21systemd-journal20:35:001
136919973,3cyclictest283-21systemd-journal19:56:271
136919973,1cyclictest0-21swapper/123:31:121
136919972,3cyclictest283-21systemd-journal21:50:531
136919970,7cyclictest0-21swapper/122:29:191
136879976,1cyclictest1488-21grep21:05:000
136879975,1cyclictest0-21swapper/022:05:420
136879973,3cyclictest665-21snmpd20:02:110
136879972,3cyclictest283-21systemd-journal20:06:260
136879972,3cyclictest283-21systemd-journal19:26:190
136879971,3cyclictest31642-21run-parts21:00:000
137009966,0cyclictest0-21swapper/323:34:233
137009966,0cyclictest0-21swapper/322:55:113
137009963,2cyclictest5838-21ssh23:10:123
137009963,2cyclictest51850irq/136-enp4s0-21:10:073
137009963,2cyclictest516-21rs:main3
137009963,2cyclictest508-21dbus-daemon23:58:123
137009963,2cyclictest508-21dbus-daemon21:25:313
137009963,2cyclictest32312-21diskstats21:00:173
137009963,2cyclictest283-21systemd-journal20:42:303
137009963,2cyclictest17404-21sh21:17:493
137009963,2cyclictest16146-21ssh00:36:143
137009963,2cyclictest1446-21ssh23:06:543
137009963,2cyclictest10127-21ssh21:33:283
137009963,1cyclictest0-21swapper/322:15:013
137009962,3cyclictest665-21snmpd23:37:543
137009962,3cyclictest665-21snmpd22:40:133
137009962,3cyclictest665-21snmpd22:36:223
137009962,3cyclictest665-21snmpd20:31:123
137009962,3cyclictest665-21snmpd19:57:473
137009962,3cyclictest665-21snmpd19:49:533
137009962,3cyclictest665-21snmpd19:36:053
137009962,3cyclictest665-21snmpd19:25:393
137009962,3cyclictest665-21snmpd19:20:053
137009962,3cyclictest665-21snmpd00:23:563
137009962,3cyclictest30058-21cat21:45:163
137009962,3cyclictest283-21systemd-journal22:47:053
137009962,3cyclictest283-21systemd-journal22:20:393
137009962,3cyclictest21053-21expr20:35:273
137009962,3cyclictest20863-21latency_hist22:00:013
137009962,3cyclictest10183-21ntpq20:10:173
137009962,1cyclictest0-21swapper/321:21:103
137009961,4cyclictest6688-21sh00:10:183
137009961,4cyclictest30604-21cut22:25:203
137009961,4cyclictest0-21swapper/323:45:183
137009960,6cyclictest0-21swapper/322:05:533
137009960,6cyclictest0-21swapper/321:51:333
137009960,6cyclictest0-21swapper/300:18:323
137009960,5cyclictest5249-21munin-run00:10:003
137009960,5cyclictest20596-21packagekitd21:40:013
137009960,3cyclictest27279-21ssh00:03:013
137009960,3cyclictest23713-21ssh22:00:473
136969966,0cyclictest0-21swapper/223:28:292
136969966,0cyclictest0-21swapper/223:17:542
136969966,0cyclictest0-21swapper/219:15:032
136969964,2cyclictest514-21in:imuxsock20:15:012
136969963,2cyclictest9301-21sshd22:52:112
136969963,2cyclictest665-21snmpd21:27:162
136969963,2cyclictest20208-21taskset20:32:362
136969963,2cyclictest20161-21sendmail_mailst19:20:242
136969963,2cyclictest17876-21ssh00:37:412
136969963,1cyclictest0-21swapper/200:13:002
136969963,1cyclictest0-21swapper/200:00:142
136969962,3cyclictest665-21snmpd21:57:252
136969962,3cyclictest665-21snmpd21:16:312
136969962,3cyclictest665-21snmpd21:01:352
136969962,3cyclictest665-21snmpd20:58:322
136969962,3cyclictest665-21snmpd20:46:062
136969962,3cyclictest665-21snmpd19:56:532
136969962,3cyclictest665-21snmpd19:52:262
136969962,3cyclictest665-21snmpd19:27:172
136969962,3cyclictest5319-21cut21:10:202
136969962,3cyclictest28942-21ntpq19:40:212
136969962,3cyclictest283-21systemd-journal22:26:482
136969962,3cyclictest283-21systemd-journal20:28:352
136969962,3cyclictest1-21systemd20:38:452
136969962,2cyclictest22036-21grep21:20:272
136969962,1cyclictest27706-21grep19:40:002
136969962,1cyclictest0-21swapper/223:46:222
136969962,1cyclictest0-21swapper/221:47:212
136969961,4cyclictest665-21snmpd23:00:202
136969961,3cyclictest283-21systemd-journal00:27:152
136969961,2cyclictest18477-21ssh00:17:482
136969960,6cyclictest0-21swapper/222:41:052
136969960,6cyclictest0-21swapper/222:33:522
136969960,5cyclictest6077-21/usr/sbin/munin23:30:312
136919966,0cyclictest0-21swapper/122:56:591
136919966,0cyclictest0-21swapper/100:15:381
136919963,2cyclictest6869-21grep23:50:161
136919963,2cyclictest665-21snmpd22:05:391
136919963,2cyclictest514-21in:imuxsock21:13:511
136919963,2cyclictest32586-21cat21:00:191
136919963,2cyclictest31302-21chrt20:56:111
136919963,2cyclictest28580-21ssh22:04:381
136919963,2cyclictest26312-21sh00:22:431
136919963,2cyclictest23108-21sh21:20:531
136919963,2cyclictest18575-21taskset21:38:481
136919963,1cyclictest0-21swapper/122:31:391
136919962,3cyclictest7661-21sshd00:10:351
136919962,3cyclictest665-21snmpd23:26:531
136919962,3cyclictest665-21snmpd23:12:441
136919962,3cyclictest665-21snmpd23:08:421
136919962,3cyclictest665-21snmpd23:02:081
136919962,3cyclictest665-21snmpd22:18:151
136919962,3cyclictest665-21snmpd22:10:301
136919962,3cyclictest665-21snmpd20:51:191
136919962,3cyclictest665-21snmpd20:25:511
136919962,3cyclictest665-21snmpd20:17:391
136919962,3cyclictest665-21snmpd20:08:271
136919962,3cyclictest665-21snmpd19:54:171
136919962,3cyclictest665-21snmpd19:49:561
136919962,3cyclictest665-21snmpd19:26:321
136919962,3cyclictest25859-21cat20:45:191
136919962,3cyclictest23237-21ntpq00:00:171
136919962,3cyclictest14584-21cat19:10:181
136919962,3cyclictest14182-21ntpq22:35:171
136919962,2cyclictest665-21snmpd22:20:331
136919962,1cyclictest0-21swapper/121:44:101
136919962,1cyclictest0-21swapper/121:34:191
136919962,1cyclictest0-21swapper/121:25:191
136919962,1cyclictest0-21swapper/121:16:351
136919962,1cyclictest0-21swapper/100:36:001
136919961,3cyclictest283-21systemd-journal23:35:361
136919960,6cyclictest0-21swapper/123:20:041
136919960,6cyclictest0-21swapper/120:35:311
136919960,5cyclictest508-21dbus-daemon20:00:211
136919960,4cyclictest283-21systemd-journal19:21:471
136919960,2cyclictest27951-21/usr/sbin/munin19:40:191
136919960,0cyclictest0-21swapper/121:55:531
136879963,2cyclictest665-21snmpd22:30:590
136879963,2cyclictest665-21snmpd21:36:290
136879963,2cyclictest665-21snmpd20:21:380
136879963,2cyclictest665-21snmpd00:35:050
136879963,2cyclictest665-21snmpd00:22:310
136879963,2cyclictest5097-21tr21:10:180
136879963,2cyclictest508-21dbus-daemon19:16:360
136879963,2cyclictest449-21sh22:26:440
136879963,2cyclictest32640-21ssh00:26:260
136879963,2cyclictest32143-21ssh22:46:140
136879963,2cyclictest283-21systemd-journal20:19:090
136879963,2cyclictest24578-21chrt23:00:560
136879963,2cyclictest17351-21sshd22:56:530
136879963,2cyclictest16310-21sh23:55:410
136879963,1cyclictest0-21swapper/021:56:070
136879962,3cyclictest9438-21rm23:51:380
136879962,3cyclictest683-21lldpd23:42:490
136879962,3cyclictest665-21snmpd21:40:150
136879962,3cyclictest665-21snmpd21:25:530
136879962,3cyclictest665-21snmpd21:16:040
136879962,3cyclictest665-21snmpd21:05:160
136879962,3cyclictest665-21snmpd20:34:510
136879962,3cyclictest665-21snmpd19:45:480
136879962,3cyclictest665-21snmpd19:12:070
136879962,3cyclictest665-21snmpd00:30:430
136879962,3cyclictest32141-11brltty22:23:100
136879962,3cyclictest32141-11brltty00:11:580
136879962,3cyclictest30557-21expr00:05:190
136879962,3cyclictest29736-21grep21:45:140
136879962,3cyclictest283-21systemd-journal20:35:370
136879962,3cyclictest15532-21tr23:15:290
136879962,3cyclictest1-21systemd19:22:260
136879962,2cyclictest283-21systemd-journal22:03:350
136879962,1cyclictest0-21swapper/023:36:490
136879962,1cyclictest0-21swapper/022:40:230
136879962,1cyclictest0-21swapper/021:20:240
136879961,3cyclictest283-21systemd-journal23:12:410
136879961,3cyclictest283-21systemd-journal22:17:030
136879961,3cyclictest283-21systemd-journal20:52:550
136879961,3cyclictest283-21systemd-journal00:16:190
136879960,6cyclictest0-21swapper/023:34:150
136879960,6cyclictest0-21swapper/021:51:160
136879960,6cyclictest0-21swapper/000:00:410
136879960,4cyclictest283-21systemd-journal23:26:480
137009953,1cyclictest69662sleep320:05:043
137009953,1cyclictest29636-21/usr/sbin/munin20:55:163
137009953,1cyclictest25661-21fschecks_time20:45:173
137009952,2cyclictest4715-21packagekitd20:00:013
137009952,2cyclictest325-21cat19:50:143
137009952,2cyclictest23804-21expr19:30:193
137009952,2cyclictest16739-21expr20:25:183
137009952,2cyclictest14944-21head19:10:203
137009952,2cyclictest13987-21cat20:20:153
137009951,3cyclictest665-21snmpd20:15:123
137009951,3cyclictest5537-21cat00:30:163
137009951,3cyclictest27812-21df_inode20:50:163
136969952,2cyclictest8331-21cat20:05:292
136969952,2cyclictest665-21snmpd23:10:172
136969952,2cyclictest5904-21rm23:50:062
136969952,2cyclictest5175-21fschecks_count20:00:152
136969952,2cyclictest30038-21sh00:05:072
136969952,2cyclictest29683-21cron23:05:002
136969952,2cyclictest283-21systemd-journal21:05:372
136969952,2cyclictest27384-21chrt20:50:072
136969952,2cyclictest22013-21ssh23:40:002
136969952,2cyclictest21626-21sh00:20:082
136969952,2cyclictest14687-21ntpq20:20:222
136969952,2cyclictest14585-21cat19:10:172
136969952,2cyclictest14039-21sh23:55:082
136969952,2cyclictest13487-21sh22:55:082
136969952,2cyclictest12460-21ssh21:35:092
136969952,2cyclictest12101-21cat20:15:192
136969952,1cyclictest0-21swapper/222:05:012
136969952,1cyclictest0-21swapper/221:30:142
136969951,3cyclictest559-21gdbus19:50:002
136969951,3cyclictest4916-21ssh21:50:082
136969951,3cyclictest16334-21ssh23:35:442
136969951,3cyclictest0-21swapper/222:20:312
136919952,2cyclictest665-21snmpd20:10:321
136919952,2cyclictest665-21snmpd19:30:171
136919952,2cyclictest26481-21latency19:35:171
136919952,2cyclictest23305-21expr20:40:151
136919952,2cyclictest22201-21rm23:40:101
136919952,2cyclictest14466-21cat20:20:201
136879952,2cyclictest9907-21cat20:10:150
136879952,2cyclictest665-21snmpd20:40:040
136879952,2cyclictest665-21snmpd19:30:080
136879952,2cyclictest28192-21expr19:40:160
136879951,3cyclictest5583-21expr22:10:150
136879951,3cyclictest305-21cat19:50:150
136879951,3cyclictest2928-21cat19:55:170
136879951,3cyclictest26853-21sshd20:46:310
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional