You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-02-08 - 05:01

x86 Intel Xeon E3-1220 @3100 MHz, Linux 4.19.1-rt3 (Profile)

Latency plot of system in rack #2, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack2slot0.osadl.org (updated Wed Feb 08, 2023 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
144352720,1sleep2321ktimersoftd/200:35:252
305872550,0sleep30-21swapper/300:39:503
300502530,0sleep10-21swapper/122:53:161
263522530,1sleep3391rcuc/322:19:413
203602530,0sleep00-21swapper/020:20:010
47902500,0sleep00-21swapper/021:17:030
226912500,0sleep10-21swapper/123:07:451
148392500,0sleep20-21swapper/221:35:472
22292490,0sleep30-21swapper/321:16:213
140532480,0sleep20-21swapper/200:02:542
245612470,0sleep00-21swapper/000:38:110
194642410,0sleep10-21swapper/122:01:251
171922370,1sleep32360899cyclictest22:41:393
2330723423,6sleep10-21swapper/119:07:391
2327223114,7sleep00-21swapper/019:07:140
2327923018,7sleep20-21swapper/219:07:202
2348222816,7sleep30-21swapper/319:09:533
287912210,0sleep00-21swapper/023:42:010
178652180,2sleep22360799cyclictest21:12:112
286252170,0sleep30-21swapper/321:31:073
163852160,0sleep10-21swapper/123:14:301
69212150,0sleep10-21swapper/121:50:161
23606991514,0cyclictest33150irq/27-ahci[00000:31:581
23606991514,0cyclictest29226-21lspci21:39:411
23602991515,0cyclictest518-40kipmi023:54:020
122512150,0sleep10-21swapper/122:48:461
23602991413,0cyclictest5355-21lspci21:41:350
81182130,1sleep1643-21gdbus21:34:211
187392130,0sleep018741-21unixbench_singl21:20:260
285182120,0sleep20-21swapper/222:03:552
23608991210,1cyclictest23807-21bash21:38:143
2360799120,1cyclictest23925-21latency_hist00:30:012
2360799120,1cyclictest1-21systemd21:50:002
2360799120,10cyclictest18205-21sshd23:15:002
23606991211,0cyclictest33150irq/27-ahci[00022:34:211
2360699120,11cyclictest24327-21sshd00:21:421
276842110,0sleep30-21swapper/323:33:423
2360899119,1cyclictest0-21swapper/323:51:343
23608991110,0cyclictest0-21swapper/323:44:233
23608991110,0cyclictest0-21swapper/322:00:043
23608991110,0cyclictest0-21swapper/300:22:533
2360899110,10cyclictest0-21swapper/322:53:233
2360899110,10cyclictest0-21swapper/322:26:533
2360899110,10cyclictest0-21swapper/319:50:333
23607991110,0cyclictest0-21swapper/222:53:412
23607991110,0cyclictest0-21swapper/221:42:502
2360799110,1cyclictest7810-21sshd23:53:112
2360799110,10cyclictest0-21swapper/222:11:402
23606991110,0cyclictest0-21swapper/123:24:021
2360699110,10cyclictest14810-21sshd22:16:321
2360299119,1cyclictest29777-21sshd23:58:470
23602991110,0cyclictest0-21swapper/000:04:260
2360299110,1cyclictest21527-21sshd22:10:160
2360299110,1cyclictest0-21swapper/000:25:360
2360299110,10cyclictest0-21swapper/021:34:360
2360299110,0cyclictest0-21swapper/022:02:160
2360299110,0cyclictest0-21swapper/000:12:560
290372100,1sleep233-21ksoftirqd/222:28:262
2360899109,0cyclictest0-21swapper/323:35:333
2360899109,0cyclictest0-21swapper/322:48:433
2360899109,0cyclictest0-21swapper/300:07:133
2360899100,0cyclictest0-21swapper/323:57:133
2360899100,0cyclictest0-21swapper/323:22:133
2360899100,0cyclictest0-21swapper/323:07:133
2360899100,0cyclictest0-21swapper/300:27:043
2360799109,0cyclictest0-21swapper/223:32:502
2360799109,0cyclictest0-21swapper/222:36:212
2360799109,0cyclictest0-21swapper/222:08:302
2360799109,0cyclictest0-21swapper/200:21:402
2360799109,0cyclictest0-21swapper/200:18:512
2360799108,1cyclictest10809-21sshd23:21:002
2360799100,0cyclictest0-21swapper/223:07:302
2360799100,0cyclictest0-21swapper/222:19:302
2360799100,0cyclictest0-21swapper/221:24:102
2360699109,0cyclictest33150irq/27-ahci[00000:03:341
2360699109,0cyclictest0-21swapper/123:25:321
2360699109,0cyclictest0-21swapper/123:03:021
2360699109,0cyclictest0-21swapper/119:30:021
2360699109,0cyclictest0-21swapper/100:13:121
2360699100,9cyclictest0-21swapper/123:51:531
2360699100,9cyclictest0-21swapper/122:21:121
2360699100,0cyclictest0-21swapper/123:35:521
2360699100,0cyclictest0-21swapper/100:16:031
2360299109,0cyclictest0-21swapper/022:49:060
2360299109,0cyclictest0-21swapper/021:55:360
2360299109,0cyclictest0-21swapper/021:05:260
2360299109,0cyclictest0-21swapper/020:59:560
2360299100,0cyclictest0-21swapper/023:22:260
2360299100,0cyclictest0-21swapper/023:01:260
2360299100,0cyclictest0-21swapper/022:07:060
236089998,0cyclictest0-21swapper/322:39:143
236089998,0cyclictest0-21swapper/321:53:133
236089998,0cyclictest0-21swapper/321:49:433
236089998,0cyclictest0-21swapper/321:29:433
236089998,0cyclictest0-21swapper/321:13:243
236089998,0cyclictest0-21swapper/300:34:033
236089998,0cyclictest0-21swapper/300:17:143
236079998,0cyclictest0-21swapper/223:43:112
236079998,0cyclictest0-21swapper/223:19:312
236079998,0cyclictest0-21swapper/222:49:002
236079998,0cyclictest0-21swapper/221:30:002
236079998,0cyclictest0-21swapper/220:26:312
236079998,0cyclictest0-21swapper/219:39:302
236069998,0cyclictest20453-21lspci23:40:061
236069998,0cyclictest0-21swapper/123:49:021
236069998,0cyclictest0-21swapper/120:30:021
236069998,0cyclictest0-21swapper/119:35:321
236069998,0cyclictest0-21swapper/100:39:521
236069992,3cyclictest633-21polkitd00:06:091
236029998,0cyclictest0-21swapper/023:25:260
236029998,0cyclictest0-21swapper/021:28:570
236029998,0cyclictest0-21swapper/019:59:560
236029996,2cyclictest614-21dbus-daemon23:17:170
236029995,1cyclictest9-21ksoftirqd/000:20:000
236029994,1cyclictest9-21ksoftirqd/022:32:390
236029993,2cyclictest633-21polkitd23:12:000
236089987,0cyclictest0-21swapper/323:25:143
236089987,0cyclictest0-21swapper/322:58:033
236089987,0cyclictest0-21swapper/321:58:233
236089987,0cyclictest0-21swapper/320:30:333
236089987,0cyclictest0-21swapper/300:04:243
236089985,2cyclictest598-21avahi-daemon23:14:303
236089985,2cyclictest598-21avahi-daemon22:12:523
236079987,0cyclictest0-21swapper/222:58:202
236079987,0cyclictest0-21swapper/221:57:002
236079987,0cyclictest0-21swapper/200:32:502
236079985,2cyclictest598-21avahi-daemon23:45:362
236079985,2cyclictest598-21avahi-daemon23:39:482
236079985,2cyclictest598-21avahi-daemon23:27:312
236079985,2cyclictest598-21avahi-daemon22:31:062
236079985,2cyclictest598-21avahi-daemon21:53:062
236079985,2cyclictest598-21avahi-daemon21:18:542
236079985,2cyclictest598-21avahi-daemon00:14:522
236069987,0cyclictest0-21swapper/122:28:531
236069987,0cyclictest0-21swapper/121:48:321
236069987,0cyclictest0-21swapper/119:56:421
236069985,2cyclictest614-21dbus-daemon21:19:011
236069985,2cyclictest598-21avahi-daemon22:06:311
236069985,2cyclictest598-21avahi-daemon21:43:211
236029987,0cyclictest0-21swapper/022:16:370
236029987,0cyclictest0-21swapper/021:45:060
236029987,0cyclictest0-21swapper/021:37:260
236029986,1cyclictest0-21swapper/023:35:460
236029983,1cyclictest9-21ksoftirqd/023:30:110
236029983,1cyclictest9-21ksoftirqd/022:51:260
236029983,1cyclictest5686-21sshd00:17:020
236029982,2cyclictest614-21dbus-daemon22:35:580
236029981,2cyclictest131rcu_sched23:47:550
236029981,1cyclictest9-21ksoftirqd/022:58:470
236029981,1cyclictest9-21ksoftirqd/022:20:290
236029981,1cyclictest9-21ksoftirqd/000:34:290
236029981,1cyclictest619-21systemd-logind00:08:040
236029981,1cyclictest576-17sedispatch21:54:070
236029980,2cyclictest17741-21sshd21:12:100
236029980,1cyclictest614-21dbus-daemon23:05:290
236089976,0cyclictest0-21swapper/323:48:433
236089976,0cyclictest0-21swapper/320:52:333
236089975,1cyclictest598-21avahi-daemon23:15:173
236089975,1cyclictest598-21avahi-daemon22:31:133
236089975,1cyclictest598-21avahi-daemon22:20:063
236089975,1cyclictest598-21avahi-daemon22:06:223
236089975,1cyclictest598-21avahi-daemon19:10:013
236089974,2cyclictest598-21avahi-daemon23:01:383
236089974,2cyclictest598-21avahi-daemon21:42:363
236089973,1cyclictest598-21avahi-daemon21:20:063
236079975,1cyclictest598-21avahi-daemon23:01:422
236079975,1cyclictest25402-21lspci00:05:342
236079974,2cyclictest598-21avahi-daemon22:44:232
236079974,2cyclictest598-21avahi-daemon21:31:192
236079973,3cyclictest614-21dbus-daemon22:20:152
236079973,3cyclictest598-21avahi-daemon23:58:312
236069977,0cyclictest518-40kipmi019:40:411
236069976,0cyclictest0-21swapper/121:55:321
236069976,0cyclictest0-21swapper/121:10:521
236069975,1cyclictest598-21avahi-daemon23:56:311
236069975,1cyclictest598-21avahi-daemon23:15:171
236069975,1cyclictest598-21avahi-daemon22:59:451
236069975,1cyclictest598-21avahi-daemon22:40:391
236069975,1cyclictest598-21avahi-daemon21:25:171
236069975,1cyclictest598-21avahi-daemon21:21:101
236069975,1cyclictest598-21avahi-daemon00:26:101
236069973,1cyclictest25-21ksoftirqd/119:50:001
236069971,1cyclictest25-21ksoftirqd/123:30:031
236069970,6cyclictest25098-21kworker/1:122:13:311
236069970,1cyclictest27290-21systemd-cgroups22:35:581
236029975,1cyclictest9-21ksoftirqd/020:50:000
236029971,2cyclictest9-21ksoftirqd/022:40:430
236029970,1cyclictest614-21dbus-daemon22:25:380
12054270,0sleep30-21swapper/300:10:223
236089965,0cyclictest0-21swapper/319:36:443
236089963,2cyclictest970-21snmpd20:24:413
236079965,0cyclictest0-21swapper/221:00:412
236079965,0cyclictest0-21swapper/219:49:402
236079963,2cyclictest970-21snmpd19:25:032
236079960,5cyclictest0-21swapper/220:18:002
236079960,5cyclictest0-21swapper/219:23:002
236069965,0cyclictest0-21swapper/120:10:221
236069964,1cyclictest970-21snmpd20:09:461
236069960,5cyclictest0-21swapper/119:28:131
236069960,5cyclictest0-21swapper/119:14:421
236029965,0cyclictest0-21swapper/020:25:060
236029965,0cyclictest0-21swapper/020:06:460
236029965,0cyclictest0-21swapper/019:30:260
236029964,1cyclictest4421-21sendmail19:50:200
236029963,2cyclictest970-21snmpd20:12:070
236029961,1cyclictest30762-21gltestperf19:25:230
32488250,0sleep30-21swapper/319:30:193
236089953,1cyclictest970-21snmpd21:07:003
236089953,1cyclictest970-21snmpd20:55:183
236089953,1cyclictest970-21snmpd20:46:443
236089953,1cyclictest970-21snmpd20:27:053
236089953,1cyclictest970-21snmpd20:10:313
236089953,1cyclictest970-21snmpd20:06:433
236089953,1cyclictest970-21snmpd20:01:073
236089953,1cyclictest970-21snmpd19:42:273
236089953,1cyclictest970-21snmpd19:25:123
236089953,1cyclictest970-21snmpd19:23:573
236089953,1cyclictest6618-21hddtemp_smartct19:45:183
236089953,1cyclictest5206-21awk21:00:213
236089953,1cyclictest27350-21processes19:15:263
236089953,1cyclictest18933-21mii-tool20:15:173
236089952,2cyclictest26671-21tr20:35:203
236089950,2cyclictest28913-21df_inode20:40:143
236079953,1cyclictest970-21snmpd20:55:092
236079953,1cyclictest970-21snmpd20:51:032
236079953,1cyclictest970-21snmpd20:46:142
236079953,1cyclictest970-21snmpd20:32:532
236079953,1cyclictest970-21snmpd20:07:132
236079953,1cyclictest970-21snmpd19:51:222
236079953,1cyclictest970-21snmpd19:32:302
236079953,1cyclictest970-21snmpd19:15:232
236079953,1cyclictest633-21polkitd21:10:002
236079953,1cyclictest633-21polkitd19:55:002
236079953,1cyclictest4702-21cat19:40:222
236079953,1cyclictest29099-21cat20:40:162
236079953,1cyclictest26999-21hddtemp_smartct20:35:222
236079953,1cyclictest24656-21ntp_states19:10:182
236079953,1cyclictest16883-21cat20:10:192
236079953,1cyclictest12838-21cat20:00:202
236079952,2cyclictest970-21snmpd20:20:112
236069953,1cyclictest970-21snmpd20:57:331
236069953,1cyclictest970-21snmpd20:42:501
236069953,1cyclictest970-21snmpd20:36:561
236069953,1cyclictest970-21snmpd20:22:201
236069953,1cyclictest970-21snmpd20:16:311
236069953,1cyclictest970-21snmpd19:15:051
236069953,1cyclictest8766-21cat19:50:181
236069953,1cyclictest633-21polkitd20:25:011
236069953,1cyclictest31170-21cat20:45:171
236069952,2cyclictest29246-21ntp_kernel_err19:20:201
236069950,4cyclictest5548-21switchtime21:00:261
236029954,0cyclictest0-21swapper/019:10:060
236029953,1cyclictest970-21snmpd20:53:040
236029953,1cyclictest970-21snmpd20:40:560
236029953,1cyclictest970-21snmpd20:36:260
236029953,1cyclictest970-21snmpd20:34:050
236029953,1cyclictest970-21snmpd20:22:080
236029953,1cyclictest970-21snmpd19:46:030
236029953,1cyclictest970-21snmpd19:23:450
236029953,1cyclictest9-21ksoftirqd/019:40:000
236029952,2cyclictest970-21snmpd21:00:030
236029952,2cyclictest970-21snmpd20:01:190
236029952,2cyclictest970-21snmpd19:16:120
236029951,1cyclictest5053-21tr19:40:240
10941250,0sleep30-21swapper/319:55:253
236069942,1cyclictest970-21snmpd21:05:151
236069942,1cyclictest0-21swapper/120:50:131
236069942,1cyclictest0-21swapper/120:00:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional