You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-22 - 05:01

x86 Intel Xeon E3-1220 @3100 MHz, Linux 4.19.1-rt3 (Profile)

Latency plot of system in rack #2, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack2slot0.osadl.org (updated Wed Mar 22, 2023 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
318992950,0sleep331898-21sshd23:43:343
39902910,0sleep10-21swapper/122:39:241
238482530,1sleep123847-21awk21:55:191
141492520,0sleep30-21swapper/322:25:293
119432520,0sleep30-21swapper/323:46:313
144522500,0sleep20-21swapper/200:03:322
247312490,0sleep00-21swapper/000:22:120
253392420,0sleep20-21swapper/220:40:202
129672420,2sleep01926899cyclictest20:10:190
1889024130,6sleep20-21swapper/219:06:402
117622410,1sleep1241ktimersoftd/100:27:021
1910123210,7sleep30-21swapper/319:09:203
1839523217,9sleep00-21swapper/019:05:210
189562256,7sleep10-21swapper/119:07:311
121752190,0sleep30-21swapper/322:16:483
220462170,1sleep10-21swapper/119:15:181
304102160,0sleep1968-21sshd21:32:411
1927599140,12cyclictest573-17auditd23:03:373
19268991414,0cyclictest518-40kipmi023:36:120
134902140,0sleep113493-21bash21:52:471
19275991311,1cyclictest0-21swapper/323:17:363
1927099130,12cyclictest0-21swapper/223:39:492
19268991312,0cyclictest1707-21lspci00:08:270
19268991311,1cyclictest3191-21lspci00:25:050
19268991310,2cyclictest32611-21sshd22:05:320
244312120,2sleep01926899cyclictest21:15:060
1927599120,1cyclictest8393-21sshd21:35:173
1926999125,6cyclictest598-21avahi-daemon23:46:271
1926999120,11cyclictest0-21swapper/100:38:101
1926999120,10cyclictest26914-21bash23:01:181
156282120,0sleep00-21swapper/021:20:420
1927599110,10cyclictest19885-21grepconf.sh23:24:173
1927599110,10cyclictest0-21swapper/323:53:473
1927599110,10cyclictest0-21swapper/322:47:563
1927599110,10cyclictest0-21swapper/322:00:573
1927599110,10cyclictest0-21swapper/320:15:173
1927099119,1cyclictest16373-21lspci23:23:192
19270991110,0cyclictest0-21swapper/222:29:492
19270991110,0cyclictest0-21swapper/221:33:592
1927099110,10cyclictest0-21swapper/223:18:192
1927099110,10cyclictest0-21swapper/200:28:392
1927099110,0cyclictest0-21swapper/223:07:492
1927099110,0cyclictest0-21swapper/223:07:482
1927099110,0cyclictest0-21swapper/221:24:282
1926999119,1cyclictest0-21swapper/123:20:191
1926999119,1cyclictest0-21swapper/123:18:391
1926999118,2cyclictest0-21swapper/122:26:391
1926999115,5cyclictest598-21avahi-daemon22:48:341
1926999115,5cyclictest598-21avahi-daemon22:05:301
1926999115,5cyclictest598-21avahi-daemon21:24:211
1926999114,1cyclictest614-21dbus-daemon22:14:121
1926999114,1cyclictest598-21avahi-daemon22:52:071
1926999110,10cyclictest0-21swapper/123:41:391
1926999110,10cyclictest0-21swapper/100:09:001
1926999110,0cyclictest0-21swapper/122:32:291
19268991110,0cyclictest19424-21lspci23:40:210
1926899110,1cyclictest23744-21lspci22:11:320
1926899110,10cyclictest0-21swapper/023:16:130
1926899110,10cyclictest0-21swapper/022:44:530
1926899110,10cyclictest0-21swapper/022:35:220
1926899110,10cyclictest0-21swapper/021:38:110
1927599109,0cyclictest20836-21sshd21:54:473
1927599109,0cyclictest0-21swapper/323:27:373
1927599109,0cyclictest0-21swapper/321:25:373
1927599109,0cyclictest0-21swapper/321:17:473
1927599109,0cyclictest0-21swapper/319:20:263
1927599109,0cyclictest0-21swapper/300:19:473
1927599100,9cyclictest0-21swapper/321:09:373
1927599100,9cyclictest0-21swapper/320:40:573
1927599100,9cyclictest0-21swapper/319:29:573
1927599100,0cyclictest0-21swapper/323:58:373
1927599100,0cyclictest0-21swapper/323:10:373
1927599100,0cyclictest0-21swapper/321:41:573
1927599100,0cyclictest0-21swapper/320:00:183
1927099109,0cyclictest0-21swapper/221:38:292
1927099109,0cyclictest0-21swapper/200:08:582
1927099100,0cyclictest0-21swapper/223:52:492
1927099100,0cyclictest0-21swapper/223:14:492
1927099100,0cyclictest0-21swapper/223:04:182
1927099100,0cyclictest0-21swapper/222:48:382
1927099100,0cyclictest0-21swapper/221:47:082
1927099100,0cyclictest0-21swapper/200:36:392
1926999109,0cyclictest0-21swapper/100:15:301
1926999105,4cyclictest598-21avahi-daemon00:20:041
1926999105,1cyclictest598-21avahi-daemon23:53:271
1926999105,1cyclictest598-21avahi-daemon21:37:001
1926999104,1cyclictest633-21polkitd23:05:081
1926999104,1cyclictest633-21polkitd23:05:081
1926999104,1cyclictest633-21polkitd22:21:161
1926999104,1cyclictest614-21dbus-daemon23:35:541
1926999104,1cyclictest614-21dbus-daemon23:13:491
1926999104,1cyclictest614-21dbus-daemon21:15:561
1926999104,1cyclictest614-21dbus-daemon00:12:061
1926999104,1cyclictest614-21dbus-daemon00:03:471
1926999104,1cyclictest598-21avahi-daemon23:56:311
1926999104,1cyclictest598-21avahi-daemon23:28:101
1926999104,1cyclictest598-21avahi-daemon21:28:251
1926999104,1cyclictest598-21avahi-daemon21:10:101
1926999104,1cyclictest598-21avahi-daemon21:10:091
1926999104,1cyclictest26027-21sshd23:33:581
1926999103,1cyclictest643-21gdbus22:02:411
1926999102,2cyclictest614-21dbus-daemon00:30:211
1926999100,0cyclictest0-21swapper/121:48:081
1926899109,0cyclictest32679-21lspci23:10:550
1926899109,0cyclictest18050-21lspci22:34:500
1926899109,0cyclictest0-21swapper/023:00:320
1926899109,0cyclictest0-21swapper/021:12:220
1926899109,0cyclictest0-21swapper/021:12:210
1926899100,0cyclictest0-21swapper/022:27:320
192759998,0cyclictest0-21swapper/322:51:273
192759998,0cyclictest0-21swapper/322:32:573
192759998,0cyclictest0-21swapper/320:13:463
192759998,0cyclictest0-21swapper/319:15:373
192759998,0cyclictest0-21swapper/319:10:263
192759998,0cyclictest0-21swapper/300:34:273
192759998,0cyclictest0-21swapper/300:11:373
192709998,0cyclictest0-21swapper/222:37:382
192709998,0cyclictest0-21swapper/221:10:292
192709998,0cyclictest0-21swapper/221:10:292
192709998,0cyclictest0-21swapper/200:17:092
192709997,1cyclictest23730-21/usr/sbin/munin00:30:182
192709990,0cyclictest0-21swapper/223:57:392
192709990,0cyclictest0-21swapper/222:05:492
192699993,1cyclictest970-21snmpd21:05:581
192699993,1cyclictest643-21gdbus22:15:021
192699993,1cyclictest643-21gdbus21:40:071
192699993,1cyclictest598-21avahi-daemon22:55:081
192699993,1cyclictest598-21avahi-daemon22:42:031
192689998,0cyclictest0-21swapper/023:49:120
192689998,0cyclictest0-21swapper/022:03:310
192689998,0cyclictest0-21swapper/021:55:220
192689998,0cyclictest0-21swapper/021:30:020
192689998,0cyclictest0-21swapper/000:00:330
192689990,5cyclictest614-21dbus-daemon22:25:000
192689990,0cyclictest0-21swapper/021:49:020
192759987,0cyclictest0-21swapper/319:50:473
192759987,0cyclictest0-21swapper/319:39:463
192759987,0cyclictest0-21swapper/300:05:063
192759986,1cyclictest598-21avahi-daemon23:36:083
192759986,1cyclictest598-21avahi-daemon21:21:453
192759986,1cyclictest14878-21lspci22:58:273
192759985,2cyclictest598-21avahi-daemon23:30:393
192759985,2cyclictest598-21avahi-daemon23:06:093
192759985,2cyclictest598-21avahi-daemon23:06:083
192759985,2cyclictest598-21avahi-daemon22:38:353
192759985,2cyclictest598-21avahi-daemon22:13:323
192759985,2cyclictest598-21avahi-daemon21:49:293
192759985,2cyclictest598-21avahi-daemon00:27:543
192759985,2cyclictest598-21avahi-daemon00:23:173
192759985,2cyclictest598-21avahi-daemon00:03:163
192709987,0cyclictest0-21swapper/222:55:392
192709985,2cyclictest598-21avahi-daemon23:25:082
192709985,2cyclictest598-21avahi-daemon22:23:272
192709985,2cyclictest598-21avahi-daemon22:10:092
192709985,2cyclictest598-21avahi-daemon22:04:262
192709985,2cyclictest598-21avahi-daemon21:57:572
192709985,2cyclictest598-21avahi-daemon21:16:572
192709985,2cyclictest598-21avahi-daemon00:23:132
192699983,1cyclictest990-21grep21:00:181
192699983,1cyclictest970-21snmpd20:51:571
192699983,1cyclictest970-21snmpd20:49:551
192699983,1cyclictest970-21snmpd20:43:341
192699983,1cyclictest970-21snmpd20:22:181
192699983,1cyclictest970-21snmpd19:27:071
192699983,1cyclictest29943-21dump-pmu-power19:35:001
192689987,0cyclictest0-21swapper/023:50:530
192689987,0cyclictest0-21swapper/000:11:530
192689986,1cyclictest9-21ksoftirqd/020:19:590
192689985,2cyclictest598-21avahi-daemon23:59:430
192689985,2cyclictest598-21avahi-daemon23:27:300
192689985,2cyclictest598-21avahi-daemon23:20:300
192689985,2cyclictest598-21avahi-daemon22:58:510
192689985,2cyclictest598-21avahi-daemon22:46:430
192689985,2cyclictest598-21avahi-daemon21:44:250
192689985,2cyclictest598-21avahi-daemon00:39:080
192689985,2cyclictest598-21avahi-daemon00:30:040
192689985,2cyclictest598-21avahi-daemon00:18:350
192689984,3cyclictest1004-21kworker/0:023:31:080
192689982,1cyclictest12136-21sshd21:52:250
192759976,0cyclictest0-21swapper/322:21:063
192759976,0cyclictest0-21swapper/319:33:373
192759975,1cyclictest598-21avahi-daemon22:41:183
192759975,1cyclictest598-21avahi-daemon22:06:003
192759975,1cyclictest598-21avahi-daemon21:55:303
192759975,1cyclictest598-21avahi-daemon00:36:053
192759974,2cyclictest598-21avahi-daemon21:32:513
192709976,0cyclictest0-21swapper/222:31:492
192709975,1cyclictest598-21avahi-daemon23:40:512
192709975,1cyclictest598-21avahi-daemon22:51:422
192709975,1cyclictest598-21avahi-daemon22:16:512
192709975,1cyclictest598-21avahi-daemon21:42:032
192709975,1cyclictest598-21avahi-daemon21:00:002
192709974,2cyclictest598-21avahi-daemon23:46:092
192709974,2cyclictest598-21avahi-daemon21:51:082
192709974,2cyclictest598-21avahi-daemon21:26:202
192709974,2cyclictest598-21avahi-daemon00:11:032
192709973,3cyclictest598-21avahi-daemon23:31:062
192709973,1cyclictest0-21swapper/222:40:122
192689975,1cyclictest598-21avahi-daemon22:15:190
192689975,1cyclictest598-21avahi-daemon21:25:050
192689974,2cyclictest614-21dbus-daemon19:20:010
192689974,2cyclictest598-21avahi-daemon22:50:030
192689970,1cyclictest121rcu_preempt23:09:270
192689970,1cyclictest121rcu_preempt23:09:270
192759965,0cyclictest0-21swapper/320:25:573
192759965,0cyclictest0-21swapper/319:55:373
192759964,1cyclictest633-21polkitd21:10:093
192759964,1cyclictest633-21polkitd21:10:083
192759963,2cyclictest970-21snmpd20:49:123
192759960,5cyclictest0-21swapper/320:51:573
192709965,0cyclictest0-21swapper/220:05:492
192709965,0cyclictest0-21swapper/219:40:592
192709963,2cyclictest970-21snmpd20:15:302
192709960,5cyclictest0-21swapper/220:01:082
192699963,2cyclictest970-21snmpd19:54:251
192699963,2cyclictest970-21snmpd19:47:441
192689966,0cyclictest22029-21kworker/0:020:35:010
192689965,0cyclictest9-21ksoftirqd/020:10:000
192689965,0cyclictest19940-21cat19:10:150
192689963,2cyclictest970-21snmpd21:02:150
192689963,2cyclictest970-21snmpd20:56:340
192689963,2cyclictest970-21snmpd20:29:200
192689962,3cyclictest573-13auditd20:40:010
192759954,0cyclictest0-21swapper/320:32:473
192759953,1cyclictest970-21snmpd21:00:273
192759953,1cyclictest970-21snmpd20:58:073
192759953,1cyclictest970-21snmpd19:45:253
192759953,1cyclictest970-21snmpd19:41:353
192759953,1cyclictest633-21polkitd20:05:013
192759953,1cyclictest16306-21tr20:20:163
192709953,1cyclictest970-21snmpd20:52:072
192709953,1cyclictest970-21snmpd20:45:542
192709953,1cyclictest970-21snmpd20:35:542
192709953,1cyclictest970-21snmpd19:51:052
192709953,1cyclictest970-21snmpd19:45:582
192709953,1cyclictest970-21snmpd19:12:122
192709953,1cyclictest6429-21cat19:55:152
192709953,1cyclictest30777-21cut20:55:152
192709953,1cyclictest28466-21gltestperf19:30:142
192709953,1cyclictest25865-21crond19:29:552
192709953,1cyclictest24571-21cat19:20:202
192709953,1cyclictest20734-21cat20:30:162
192709953,1cyclictest12909-21cut20:10:172
192709952,2cyclictest970-21snmpd21:06:562
192709952,2cyclictest22719-21cat19:15:232
192709952,1cyclictest0-21swapper/219:39:442
192709950,1cyclictest16170-21/usr/sbin/munin20:20:212
192699953,1cyclictest970-21snmpd20:35:181
192699953,1cyclictest970-21snmpd20:30:541
192699953,1cyclictest970-21snmpd20:29:171
192699953,1cyclictest970-21snmpd20:11:091
192699953,1cyclictest970-21snmpd19:57:441
192699953,1cyclictest970-21snmpd19:35:341
192699953,1cyclictest970-21snmpd19:20:151
192699953,1cyclictest14619-21cat20:15:151
192699953,1cyclictest0-21swapper/120:55:451
192699952,2cyclictest970-21snmpd20:05:091
192699952,2cyclictest970-21snmpd19:42:021
192699952,2cyclictest8897-21open_files20:00:171
192689953,1cyclictest970-21snmpd20:03:140
192689953,1cyclictest970-21snmpd19:34:380
192689953,1cyclictest633-21polkitd19:39:590
192689953,1cyclictest4421-21sendmail19:21:470
192689953,1cyclictest26801-21cat20:45:160
192689953,1cyclictest2358-21cat19:45:200
192689953,1cyclictest16663-21cat20:20:200
192689953,1cyclictest1071-21lldpd19:36:030
192689952,2cyclictest970-21snmpd19:25:310
192689952,2cyclictest28720-21tail20:50:190
192689952,2cyclictest24397-21tr20:40:130
192689951,3cyclictest2490-21tr21:05:190
192689950,0cyclictest0-21swapper/019:56:020
192759942,1cyclictest22768-21cat20:35:143
192709942,1cyclictest18497-21gltestperf20:25:192
192699942,1cyclictest20557-21cat19:10:201
192689942,1cyclictest3938-21/usr/sbin/munin19:50:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional