You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-28 - 07:13

x86 Intel Xeon E3-1220 @3100 MHz, Linux 4.19.1-rt3 (Profile)

Latency plot of system in rack #2, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rack2slot0.osadl.org (updated Tue Mar 28, 2023 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
137392590,0sleep00-21swapper/021:54:410
257452530,0sleep00-21swapper/023:35:110
281882510,0sleep00-21swapper/023:51:570
275002500,0sleep00-21swapper/019:45:000
42762490,0sleep10-21swapper/123:05:151
316542490,0sleep00-21swapper/000:09:180
301412490,0sleep00-21swapper/023:20:010
291262490,0sleep00-21swapper/023:11:240
203902490,0sleep10-21swapper/123:25:231
114822470,0sleep20-21swapper/221:21:242
268242450,2sleep21270599cyclictest00:00:002
15272450,0sleep2311rcuc/221:27:042
1235724231,7sleep20-21swapper/219:07:072
1254923514,7sleep30-21swapper/319:09:283
1236723110,7sleep00-21swapper/019:07:140
1231423025,3sleep10-21swapper/119:06:351
275032210,0sleep2968-21sshd23:10:582
200672170,0sleep00-21swapper/022:04:200
241852160,0sleep30-21swapper/300:15:223
12705991515,0cyclictest518-40kipmi020:20:062
320312140,0sleep30-21swapper/321:59:143
273502140,0sleep30-21swapper/300:32:293
1269499130,11cyclictest8440-21sshd00:03:230
12705991210,1cyclictest9859-21sshd00:28:082
1270599120,11cyclictest0-21swapper/200:19:082
1270199125,6cyclictest598-21avahi-daemon23:54:071
1270199125,6cyclictest598-21avahi-daemon23:40:441
1270199125,6cyclictest598-21avahi-daemon23:35:351
1270199125,6cyclictest598-21avahi-daemon22:31:491
1270199125,6cyclictest598-21avahi-daemon22:16:461
1270199125,6cyclictest598-21avahi-daemon21:16:211
1270199125,6cyclictest598-21avahi-daemon00:35:381
1270199125,6cyclictest598-21avahi-daemon00:18:431
1270199125,6cyclictest598-21avahi-daemon00:01:101
1270199124,7cyclictest598-21avahi-daemon23:57:031
12701991210,1cyclictest576-17sedispatch00:14:091
1270199120,1cyclictest2470-21sshd22:24:171
12694991210,1cyclictest0-21swapper/022:51:130
163702110,0sleep116373-21gltestperf21:55:161
12711991110,0cyclictest0-21swapper/321:11:513
1271199110,10cyclictest0-21swapper/322:03:213
1271199110,0cyclictest0-21swapper/323:18:113
1270599114,4cyclictest633-21polkitd22:26:482
1270599114,3cyclictest33-21ksoftirqd/222:00:002
1270599110,10cyclictest17022-21lspci22:11:282
1270599110,10cyclictest0-21swapper/223:20:282
1270599110,10cyclictest0-21swapper/223:17:172
1270599110,10cyclictest0-21swapper/200:20:382
1270199119,1cyclictest0-21swapper/121:36:481
1270199118,2cyclictest0-21swapper/121:10:381
1270199115,5cyclictest598-21avahi-daemon21:30:551
1270199115,5cyclictest598-21avahi-daemon21:26:161
1270199114,6cyclictest633-21polkitd23:21:031
1270199114,6cyclictest633-21polkitd22:55:251
1270199114,6cyclictest633-21polkitd22:41:101
1270199114,6cyclictest633-21polkitd22:39:241
1270199114,6cyclictest614-21dbus-daemon22:45:221
1270199114,6cyclictest614-21dbus-daemon22:29:511
1270199114,6cyclictest598-21avahi-daemon22:52:461
1270199113,2cyclictest616-21gdbus21:40:041
1270199113,1cyclictest643-21gdbus22:07:291
12701991110,0cyclictest0-21swapper/121:20:191
12694991110,0cyclictest23848-21lspci22:37:390
12694991110,0cyclictest0-21swapper/022:46:030
1269499110,10cyclictest0-21swapper/021:22:030
1271199109,0cyclictest0-21swapper/323:08:123
1271199109,0cyclictest0-21swapper/322:44:123
1271199109,0cyclictest0-21swapper/322:08:523
1271199109,0cyclictest0-21swapper/321:17:423
1271199108,1cyclictest71250irq/24-eno1-rx-00:00:423
1271199100,0cyclictest0-21swapper/323:47:023
1271199100,0cyclictest0-21swapper/323:42:113
1271199100,0cyclictest0-21swapper/322:17:523
1271199100,0cyclictest0-21swapper/321:54:023
1270599109,0cyclictest0-21swapper/223:31:382
1270599109,0cyclictest0-21swapper/221:49:182
1270599109,0cyclictest0-21swapper/221:33:282
1270599109,0cyclictest0-21swapper/221:18:382
1270599109,0cyclictest0-21swapper/219:23:472
1270599109,0cyclictest0-21swapper/200:38:382
1270599109,0cyclictest0-21swapper/200:31:082
1270599109,0cyclictest0-21swapper/200:01:182
1270599103,4cyclictest633-21polkitd23:06:162
1270599103,3cyclictest616-21gdbus22:35:222
1270599103,1cyclictest33-21ksoftirqd/222:06:512
1270599102,3cyclictest33-21ksoftirqd/221:40:042
1270599102,1cyclictest33-21ksoftirqd/222:40:362
1270599101,1cyclictest19576-21munin-run23:50:002
1270599100,9cyclictest0-21swapper/223:02:082
1270599100,0cyclictest0-21swapper/223:44:272
1270599100,0cyclictest0-21swapper/223:39:382
1270599100,0cyclictest0-21swapper/222:55:182
1270599100,0cyclictest0-21swapper/221:51:072
1270199104,5cyclictest633-21polkitd21:50:231
1270199104,5cyclictest614-21dbus-daemon23:00:261
1270199103,6cyclictest598-21avahi-daemon00:20:191
1270199103,1cyclictest9156-21sshd00:27:571
1270199103,1cyclictest633-21polkitd23:45:211
1270199103,1cyclictest633-21polkitd23:15:141
1270199103,1cyclictest614-21dbus-daemon00:07:011
1270199103,1cyclictest598-21avahi-daemon23:30:121
1270199103,1cyclictest25614-21sadc21:00:011
1270199102,2cyclictest614-21dbus-daemon23:11:551
1270199102,2cyclictest614-21dbus-daemon22:12:151
1270199102,2cyclictest614-21dbus-daemon22:02:441
1270199102,2cyclictest614-21dbus-daemon00:31:041
1269499109,0cyclictest0-21swapper/021:34:530
1269499107,2cyclictest27572-21kworker/0:222:44:490
1269499100,0cyclictest0-21swapper/022:10:530
1269499100,0cyclictest0-21swapper/000:15:130
30350290,0sleep20-21swapper/221:10:062
127119998,0cyclictest0-21swapper/321:45:013
127119997,1cyclictest1720-21lspci22:32:023
127119997,1cyclictest16111-21bash22:35:313
127119997,1cyclictest0-21swapper/321:36:113
127119996,2cyclictest598-21avahi-daemon22:28:253
127059993,3cyclictest643-21gdbus00:10:362
127059993,2cyclictest633-21polkitd21:37:362
127059992,3cyclictest633-21polkitd22:31:102
127059992,3cyclictest614-21dbus-daemon23:50:352
127059992,1cyclictest33-21ksoftirqd/222:51:052
127059991,3cyclictest33-21ksoftirqd/222:03:282
127059991,1cyclictest33-21ksoftirqd/222:17:432
127059991,1cyclictest33-21ksoftirqd/200:08:292
127019993,1cyclictest970-21snmpd20:49:251
127019993,1cyclictest643-21gdbus21:10:011
127019993,1cyclictest619-21systemd-logind21:45:551
127019993,1cyclictest619-21systemd-logind19:29:591
127019993,1cyclictest18693-21timerandwakeup19:20:281
126949998,0cyclictest0-21swapper/023:42:030
126949998,0cyclictest0-21swapper/023:30:520
126949998,0cyclictest0-21swapper/022:19:520
126949998,0cyclictest0-21swapper/021:16:220
126949998,0cyclictest0-21swapper/020:25:120
126949997,1cyclictest23136-21bash21:48:530
126949997,1cyclictest1613-21lspci22:56:230
126949990,0cyclictest0-21swapper/020:49:430
13557280,0sleep00-21swapper/023:56:200
127119987,0cyclictest0-21swapper/322:10:413
127119987,0cyclictest0-21swapper/321:41:323
127119987,0cyclictest0-21swapper/320:03:123
127119985,2cyclictest598-21avahi-daemon23:26:033
127119985,2cyclictest598-21avahi-daemon22:56:343
127119985,2cyclictest598-21avahi-daemon22:54:163
127119985,2cyclictest598-21avahi-daemon21:30:423
127119985,2cyclictest598-21avahi-daemon00:39:093
127119984,3cyclictest1-21systemd23:31:453
127059987,0cyclictest0-21swapper/220:29:572
127059986,1cyclictest33-21ksoftirqd/219:59:592
127059982,3cyclictest33-21ksoftirqd/223:26:232
127059981,3cyclictest33-21ksoftirqd/222:45:012
127059980,1cyclictest33-21ksoftirqd/222:20:132
127019983,1cyclictest970-21snmpd20:37:111
127019983,1cyclictest970-21snmpd19:59:411
127019983,1cyclictest970-21snmpd19:40:221
127019983,1cyclictest970-21snmpd19:35:301
127019983,1cyclictest970-21snmpd19:10:141
127019982,1cyclictest970-21snmpd20:04:311
127019982,1cyclictest619-21systemd-logind20:10:011
127019982,1cyclictest5678-21cstates20:10:131
127019982,1cyclictest28166-21cat19:45:151
127019982,1cyclictest26258-21sed21:00:131
127019982,1cyclictest18496-21awk20:40:331
127019982,1cyclictest15375-21munin-node19:15:021
127019981,2cyclictest970-21snmpd19:32:461
127019980,3cyclictest22097-21grep20:50:141
127019980,2cyclictest9578-21sshd20:20:121
127019980,2cyclictest8463-21/usr/sbin/munin20:15:201
126949987,0cyclictest0-21swapper/023:27:430
126949987,0cyclictest0-21swapper/023:04:530
126949987,0cyclictest0-21swapper/021:40:120
126949987,0cyclictest0-21swapper/019:25:220
126949986,2cyclictest25904-21kworker/0:323:07:230
126949986,1cyclictest598-21avahi-daemon21:13:570
126949986,1cyclictest598-21avahi-daemon00:32:380
126949985,2cyclictest598-21avahi-daemon23:23:350
126949985,2cyclictest598-21avahi-daemon22:05:100
126949985,2cyclictest598-21avahi-daemon21:37:490
127119976,0cyclictest0-21swapper/321:04:113
127119976,0cyclictest0-21swapper/319:17:123
127119975,1cyclictest598-21avahi-daemon23:01:333
127119975,1cyclictest598-21avahi-daemon21:28:573
127119975,1cyclictest598-21avahi-daemon21:20:383
127119975,1cyclictest598-21avahi-daemon00:28:073
127119975,1cyclictest598-21avahi-daemon00:21:493
127119975,1cyclictest598-21avahi-daemon00:13:023
127119974,2cyclictest598-21avahi-daemon23:35:283
127119974,2cyclictest598-21avahi-daemon23:11:173
127119974,2cyclictest598-21avahi-daemon22:20:013
127119974,2cyclictest598-21avahi-daemon00:09:143
127119974,2cyclictest25943-21sshd22:46:133
127119973,3cyclictest598-21avahi-daemon23:20:433
127059976,0cyclictest0-21swapper/220:07:282
127059975,1cyclictest598-21avahi-daemon21:10:012
127059971,1cyclictest33-21ksoftirqd/219:34:542
127019972,1cyclictest970-21snmpd20:29:231
127019971,1cyclictest30402-21cat19:50:201
127019971,1cyclictest15317-21chrt20:33:301
126949976,0cyclictest0-21swapper/022:25:130
126949975,1cyclictest9-21ksoftirqd/019:54:590
126949975,1cyclictest598-21avahi-daemon23:45:150
126949975,1cyclictest598-21avahi-daemon21:25:050
126949975,1cyclictest598-21avahi-daemon00:29:160
126949975,1cyclictest598-21avahi-daemon00:21:390
126949975,1cyclictest598-21avahi-daemon00:12:520
126949974,2cyclictest598-21avahi-daemon21:55:130
126949974,2cyclictest598-21avahi-daemon00:35:390
127119965,0cyclictest0-21swapper/320:55:023
127119965,0cyclictest0-21swapper/320:46:113
127119965,0cyclictest0-21swapper/319:58:413
127119964,1cyclictest614-21dbus-daemon23:50:033
127119964,1cyclictest598-21avahi-daemon19:10:003
127119963,2cyclictest633-21polkitd23:55:023
127119960,5cyclictest0-21swapper/321:09:523
127119960,0cyclictest0-21swapper/319:53:023
127059965,0cyclictest311rcuc/219:29:592
127059965,0cyclictest0-21swapper/221:01:172
127059963,2cyclictest970-21snmpd20:44:512
127059963,2cyclictest970-21snmpd20:35:562
127059963,2cyclictest970-21snmpd19:15:392
127059962,1cyclictest121rcu_preempt20:55:012
127059961,1cyclictest27983-21gltestperf19:45:142
127059961,1cyclictest121rcu_preempt21:00:012
127059960,5cyclictest0-21swapper/220:33:082
127059960,5cyclictest0-21swapper/220:18:382
127059960,0cyclictest0-21swapper/220:12:172
126949964,1cyclictest614-21dbus-daemon22:20:180
126949964,1cyclictest598-21avahi-daemon19:20:250
126949962,2cyclictest25318-21sshd22:30:010
126949960,5cyclictest0-21swapper/020:51:120
126949960,5cyclictest0-21swapper/020:17:120
126949960,1cyclictest11382-21munin-run20:25:010
127119953,1cyclictest970-21snmpd20:50:343
127119953,1cyclictest970-21snmpd20:30:033
127119953,1cyclictest970-21snmpd20:26:003
127119953,1cyclictest970-21snmpd20:20:143
127119953,1cyclictest970-21snmpd20:17:233
127119953,1cyclictest970-21snmpd19:46:453
127119953,1cyclictest970-21snmpd19:41:463
127119953,1cyclictest970-21snmpd19:35:003
127119953,1cyclictest970-21snmpd19:26:043
127119953,1cyclictest598-21avahi-daemon20:09:273
127119953,1cyclictest22066-21cat19:30:153
127119953,1cyclictest18177-21grep19:20:233
127119953,1cyclictest16141-21cat20:35:163
127119952,2cyclictest5787-21taskset20:10:143
127059953,1cyclictest970-21snmpd19:36:212
127059952,2cyclictest970-21snmpd19:54:202
127059952,2cyclictest1078-21ntpd20:45:222
127059950,3cyclictest0-21swapper/220:00:282
127059950,1cyclictest23356-21kworker/2:019:40:012
127059950,1cyclictest131rcu_sched19:10:002
126949953,1cyclictest970-21snmpd21:05:010
126949953,1cyclictest970-21snmpd20:57:160
126949953,1cyclictest970-21snmpd20:40:590
126949953,1cyclictest970-21snmpd20:33:210
126949953,1cyclictest970-21snmpd19:55:050
126949953,1cyclictest970-21snmpd19:46:210
126949953,1cyclictest970-21snmpd19:33:550
126949953,1cyclictest970-21snmpd19:10:410
126949953,1cyclictest598-21avahi-daemon19:37:570
126949953,1cyclictest3962-21cat20:05:150
126949953,1cyclictest26331-21mii-tool21:00:140
126949953,1cyclictest1928-21cat20:00:220
126949953,1cyclictest1071-21lldpd19:16:120
126949952,2cyclictest614-21dbus-daemon20:10:000
127119942,1cyclictest17811-21ls20:40:243
126949942,1cyclictest16084-21hwlatdetect20:35:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional