You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-16 - 06:59

x86 Intel Xeon E3-1220 @3100 MHz, Linux 4.19.1-rt3 (Profile)

Latency plot of system in rack #2, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack2slot0.osadl.org (updated Tue Aug 16, 2022 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
38632560,2sleep12710299cyclictest23:09:161
33592550,0sleep10-21swapper/122:36:351
228192550,0sleep30-21swapper/323:46:023
243342540,0sleep10-21swapper/121:12:571
130302510,0sleep00-21swapper/000:31:540
236252500,0sleep10-21swapper/121:29:031
319372480,0sleep20-21swapper/221:30:532
55372470,0sleep20-21swapper/223:17:372
257912470,0sleep20-21swapper/221:53:522
189332440,2sleep32711499cyclictest22:24:323
110552400,0sleep20-21swapper/222:14:292
161022390,0sleep00-21swapper/021:10:440
2694823423,6sleep20-21swapper/219:09:232
2682623221,7sleep10-21swapper/119:07:491
2683323110,7sleep30-21swapper/319:07:563
2678822513,7sleep00-21swapper/019:07:200
35292160,0sleep00-21swapper/000:29:440
48232140,0sleep30-21swapper/323:33:413
27102991412,1cyclictest633-21polkitd21:43:071
27114991311,1cyclictest573-17auditd22:28:273
156742130,0sleep00-21swapper/021:18:570
310632120,0sleep30-21swapper/323:15:473
27114991210,1cyclictest614-21dbus-daemon21:50:183
2710799120,1cyclictest22738-21sshd21:20:262
111592120,0sleep20-21swapper/200:31:242
2711499110,1cyclictest0-21swapper/322:38:273
2711499110,10cyclictest21462-21needreboot21:20:173
2711499110,10cyclictest0-21swapper/323:21:283
2711499110,10cyclictest0-21swapper/323:10:073
2711499110,10cyclictest0-21swapper/300:06:073
2710799119,1cyclictest26583-21bash21:37:462
2710799113,1cyclictest633-21polkitd23:35:312
27107991110,0cyclictest6631-21lspci23:34:112
27107991110,0cyclictest0-21swapper/219:40:162
27107991110,0cyclictest0-21swapper/200:19:472
2710799110,10cyclictest0-21swapper/223:59:262
2710799110,10cyclictest0-21swapper/222:54:162
2710799110,10cyclictest0-21swapper/222:25:062
27102991110,0cyclictest0-21swapper/122:45:101
2710299110,1cyclictest0-21swapper/122:41:591
2710299110,10cyclictest0-21swapper/123:28:591
2710299110,10cyclictest0-21swapper/122:16:491
2710299110,10cyclictest0-21swapper/100:36:391
2710299110,10cyclictest0-21swapper/100:32:301
2710299110,10cyclictest0-21swapper/100:02:291
2709799110,10cyclictest0-21swapper/022:23:280
2709799110,10cyclictest0-21swapper/021:45:270
2709799110,10cyclictest0-21swapper/021:25:070
2709799110,10cyclictest0-21swapper/000:24:290
2709799110,0cyclictest0-21swapper/023:50:490
2709799110,0cyclictest0-21swapper/023:50:480
2711499109,0cyclictest0-21swapper/323:28:373
2711499109,0cyclictest0-21swapper/322:41:383
2711499109,0cyclictest0-21swapper/321:44:583
2711499109,0cyclictest0-21swapper/321:35:473
2711499109,0cyclictest0-21swapper/321:11:573
2711499109,0cyclictest0-21swapper/300:04:483
2711499108,1cyclictest5338-21lspci22:12:573
2711499100,0cyclictest0-21swapper/322:00:473
2711499100,0cyclictest0-21swapper/300:30:373
2711499100,0cyclictest0-21swapper/300:25:083
2710799109,0cyclictest0-21swapper/222:19:572
2710799109,0cyclictest0-21swapper/222:09:372
2710799109,0cyclictest0-21swapper/221:57:262
2710799109,0cyclictest0-21swapper/221:48:362
2710799109,0cyclictest0-21swapper/221:14:162
2710799109,0cyclictest0-21swapper/220:45:162
2710799102,3cyclictest33-21ksoftirqd/221:40:372
2710799102,3cyclictest131rcu_sched23:10:142
2710799100,0cyclictest0-21swapper/223:22:572
2710799100,0cyclictest0-21swapper/223:00:362
2710299109,0cyclictest0-21swapper/123:10:301
2710299109,0cyclictest0-21swapper/121:24:491
2710299109,0cyclictest0-21swapper/121:16:401
2710299109,0cyclictest0-21swapper/100:14:391
2710299100,9cyclictest0-21swapper/122:32:201
2710299100,9cyclictest0-21swapper/122:22:001
2710299100,0cyclictest0-21swapper/122:11:391
2710299100,0cyclictest0-21swapper/122:02:591
2709799109,0cyclictest0-21swapper/022:41:380
2709799109,0cyclictest0-21swapper/021:54:170
2709799109,0cyclictest0-21swapper/000:12:180
2709799108,1cyclictest12964-21sshd23:11:190
2709799101,2cyclictest643-21gdbus22:32:440
2709799101,1cyclictest619-21systemd-logind23:30:000
2709799100,9cyclictest0-21swapper/023:21:090
2709799100,0cyclictest0-21swapper/023:36:490
2709799100,0cyclictest0-21swapper/021:34:280
271149998,0cyclictest0-21swapper/323:37:173
271149998,0cyclictest0-21swapper/323:06:583
271149998,0cyclictest0-21swapper/322:33:383
271079998,0cyclictest4344-21strings19:30:172
271079998,0cyclictest0-21swapper/223:40:172
271079998,0cyclictest0-21swapper/222:30:172
271079998,0cyclictest0-21swapper/220:10:162
271079998,0cyclictest0-21swapper/219:55:162
271079998,0cyclictest0-21swapper/200:39:062
271079998,0cyclictest0-21swapper/200:10:162
271079994,3cyclictest33-21ksoftirqd/221:15:222
271079993,2cyclictest33-21ksoftirqd/222:45:572
271079993,1cyclictest33-21ksoftirqd/219:10:002
271079992,1cyclictest33-21ksoftirqd/222:42:142
271079991,3cyclictest633-21polkitd22:39:432
271079991,2cyclictest131rcu_sched22:59:552
271079991,1cyclictest121rcu_preempt00:00:112
271079990,0cyclictest0-21swapper/223:52:092
271079990,0cyclictest0-21swapper/223:52:082
271029998,0cyclictest0-21swapper/123:39:101
271029998,0cyclictest0-21swapper/123:18:591
271029998,0cyclictest0-21swapper/122:51:291
271029998,0cyclictest0-21swapper/100:21:291
271029997,1cyclictest614-21dbus-daemon23:51:541
271029997,1cyclictest614-21dbus-daemon23:51:531
270979998,0cyclictest0-21swapper/022:04:370
270979997,1cyclictest20550-21systemd-cgroups22:16:370
270979996,0cyclictest9-21ksoftirqd/020:00:000
270979993,3cyclictest633-21polkitd23:43:130
270979993,2cyclictest633-21polkitd22:45:170
270979993,1cyclictest9-21ksoftirqd/021:20:550
270979992,2cyclictest616-21gdbus22:09:080
270979992,1cyclictest614-21dbus-daemon21:57:040
270979991,2cyclictest9-21ksoftirqd/023:31:080
270979991,1cyclictest9-21ksoftirqd/023:02:060
270979991,1cyclictest9-21ksoftirqd/022:24:590
270979991,1cyclictest131rcu_sched21:35:460
270979991,1cyclictest131rcu_sched00:18:040
271149987,0cyclictest0-21swapper/319:36:273
271149987,0cyclictest0-21swapper/300:38:583
271149987,0cyclictest0-21swapper/300:16:583
271149986,1cyclictest26052-21bash22:18:083
271149985,2cyclictest598-21avahi-daemon23:42:493
271149985,2cyclictest598-21avahi-daemon23:02:283
271149985,2cyclictest598-21avahi-daemon22:57:483
271149985,2cyclictest598-21avahi-daemon22:05:523
271149985,2cyclictest598-21avahi-daemon21:28:103
271149985,2cyclictest598-21avahi-daemon00:11:113
271079987,0cyclictest0-21swapper/220:59:562
271079987,0cyclictest0-21swapper/200:25:172
271079983,2cyclictest662-21sshd23:08:252
271079982,3cyclictest33-21ksoftirqd/222:00:002
271079981,3cyclictest33-21ksoftirqd/220:45:012
271079981,1cyclictest131rcu_sched21:25:242
271079981,0cyclictest121rcu_preempt23:45:232
271079980,2cyclictest131rcu_sched00:22:162
271079980,1cyclictest33-21ksoftirqd/220:30:002
271079980,1cyclictest33-21ksoftirqd/200:06:462
271079980,1cyclictest121rcu_preempt23:27:342
271029987,0cyclictest0-21swapper/122:06:301
271029987,0cyclictest0-21swapper/121:32:301
271029987,0cyclictest0-21swapper/100:27:001
271029985,2cyclictest598-21avahi-daemon23:03:351
271029985,2cyclictest598-21avahi-daemon21:50:211
271029985,2cyclictest598-21avahi-daemon00:17:311
270979987,0cyclictest0-21swapper/022:55:390
270979986,1cyclictest9-21ksoftirqd/020:45:010
270979986,1cyclictest9-21ksoftirqd/019:50:010
270979982,1cyclictest9-21ksoftirqd/023:55:280
270979982,1cyclictest9-21ksoftirqd/022:51:180
270979982,1cyclictest9-21ksoftirqd/000:38:090
270979982,1cyclictest643-21gdbus22:35:000
270979982,1cyclictest4660-21kworker/0:122:12:480
270979981,1cyclictest9-21ksoftirqd/023:09:180
270979981,1cyclictest131rcu_sched23:45:450
270979981,1cyclictest131rcu_sched00:06:570
270979981,1cyclictest131rcu_sched00:00:100
270979980,2cyclictest9-21ksoftirqd/023:15:120
270979980,1cyclictest131rcu_sched21:40:450
271149975,1cyclictest614-21dbus-daemon00:20:483
271149975,1cyclictest598-21avahi-daemon23:50:483
271149975,1cyclictest598-21avahi-daemon23:50:473
271149975,1cyclictest598-21avahi-daemon22:51:523
271149975,1cyclictest598-21avahi-daemon21:45:123
271149975,1cyclictest598-21avahi-daemon21:15:413
271149975,1cyclictest32683-21systemd-cgroups21:31:043
271149974,2cyclictest614-21dbus-daemon21:05:013
271149974,2cyclictest598-21avahi-daemon23:56:533
271149974,2cyclictest598-21avahi-daemon22:45:023
271149974,2cyclictest598-21avahi-daemon21:55:433
271079976,0cyclictest0-21swapper/220:50:172
271079976,0cyclictest0-21swapper/219:18:162
271079970,1cyclictest131rcu_sched22:20:202
271029976,0cyclictest0-21swapper/123:44:591
271029976,0cyclictest0-21swapper/122:25:291
271029975,1cyclictest633-21polkitd23:45:221
271029975,1cyclictest598-21avahi-daemon21:36:001
271029974,2cyclictest643-21gdbus00:05:431
271029974,2cyclictest598-21avahi-daemon23:24:231
271029974,2cyclictest598-21avahi-daemon22:55:191
271029974,2cyclictest598-21avahi-daemon21:55:341
271029974,2cyclictest598-21avahi-daemon21:46:331
271029974,2cyclictest1-21systemd00:00:001
270979975,1cyclictest9-21ksoftirqd/020:29:590
270979975,1cyclictest9-21ksoftirqd/019:15:010
270979970,1cyclictest131rcu_sched19:25:000
270979970,0cyclictest121rcu_preempt19:35:000
271149965,0cyclictest0-21swapper/319:55:173
271149964,1cyclictest643-21gdbus21:10:003
271149964,1cyclictest598-21avahi-daemon21:00:003
271149964,1cyclictest21740-21idleruntime-cro20:15:003
271149963,2cyclictest970-21snmpd20:21:353
271149960,5cyclictest0-21swapper/319:33:383
271079964,1cyclictest970-21snmpd20:24:392
271079963,2cyclictest970-21snmpd20:38:422
271079961,3cyclictest678-21gdbus19:49:552
271029964,1cyclictest970-21snmpd20:15:481
271029964,1cyclictest633-21polkitd19:15:001
271029964,1cyclictest598-21avahi-daemon23:31:231
271029964,1cyclictest32268-21grep19:20:151
271029963,2cyclictest970-21snmpd21:06:261
271029963,2cyclictest970-21snmpd20:59:311
270979965,0cyclictest9-21ksoftirqd/020:10:010
270979964,1cyclictest9-21ksoftirqd/021:10:010
270979961,2cyclictest121rcu_preempt19:15:170
270979960,1cyclictest9-21ksoftirqd/020:39:540
270979960,1cyclictest121rcu_preempt20:20:000
7869250,0sleep0111rcuc/019:40:140
271149953,1cyclictest970-21snmpd20:45:363
271149953,1cyclictest970-21snmpd20:42:303
271149953,1cyclictest970-21snmpd20:00:333
271149953,1cyclictest970-21snmpd19:50:413
271149953,1cyclictest970-21snmpd19:50:413
271149953,1cyclictest970-21snmpd19:48:313
271149953,1cyclictest618-21lsmd20:35:163
271149953,1cyclictest32375-21cat19:20:153
271149953,1cyclictest27675-21gltestperf19:10:183
271149953,1cyclictest22444-21mii-tool20:15:153
271149953,1cyclictest18392-21cat20:05:183
271149952,2cyclictest970-21snmpd19:18:363
271149952,2cyclictest3213-21chrt19:27:523
271149952,2cyclictest28172-21cpuspeed_turbos20:30:153
271149950,4cyclictest0-21swapper/320:50:383
271079953,1cyclictest970-21snmpd21:00:042
271079953,1cyclictest970-21snmpd20:17:122
271079953,1cyclictest970-21snmpd20:03:182
271079953,1cyclictest970-21snmpd19:38:402
271079953,1cyclictest970-21snmpd19:27:042
271079953,1cyclictest33-21ksoftirqd/219:21:272
271079953,1cyclictest28377-21fw_packets20:30:162
271079952,2cyclictest970-21snmpd20:05:082
271079951,0cyclictest12505-21nfsd419:50:182
271079951,0cyclictest12505-21nfsd419:50:172
271079950,1cyclictest10579-21latency21:05:182
271029953,1cyclictest970-21snmpd20:33:241
271029953,1cyclictest970-21snmpd20:26:551
271029953,1cyclictest970-21snmpd20:10:231
271029953,1cyclictest970-21snmpd19:55:081
271029953,1cyclictest970-21snmpd19:50:121
271029953,1cyclictest970-21snmpd19:50:111
271029953,1cyclictest966-21runrttasks20:09:201
271029953,1cyclictest9129-21sshd19:40:261
271029953,1cyclictest8440-21cat21:00:151
271029953,1cyclictest6171-21cat19:35:151
271029953,1cyclictest4189-21grep20:50:121
271029953,1cyclictest32653-21cat20:40:201
271029953,1cyclictest2946-21switchtime20:45:211
271029953,1cyclictest1002-21rpcbind20:36:001
271029952,2cyclictest970-21snmpd20:20:261
271029952,2cyclictest970-21snmpd19:45:351
271029950,0cyclictest0-21swapper/119:30:091
271029950,0cyclictest0-21swapper/119:27:191
270979954,0cyclictest7457-21kworker/0:120:55:110
270979954,0cyclictest7457-21kworker/0:120:50:020
270979953,1cyclictest970-21snmpd21:00:220
270979953,1cyclictest970-21snmpd20:02:440
270979953,1cyclictest6169-21cat19:35:150
270979953,1cyclictest598-21avahi-daemon19:50:380
270979953,1cyclictest598-21avahi-daemon19:50:380
270979953,1cyclictest4421-21sendmail20:14:280
270979953,1cyclictest31921-21/usr/sbin/munin19:20:150
270979953,1cyclictest28403-21gltestperf20:30:170
270979952,2cyclictest970-21snmpd20:46:240
270979951,1cyclictest121rcu_preempt20:20:190
271149942,1cyclictest970-21snmpd20:25:133
271149940,3cyclictest7717-21/usr/sbin/munin19:40:153
271029942,1cyclictest29371-21sensors19:15:001
271029942,1cyclictest0-21swapper/120:00:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional