You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-06-25 - 03:11

x86 Intel Xeon E3-1220 @3100 MHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #1, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rack1slot0.osadl.org (updated Sat Jun 25, 2022 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11762870,0sleep20-21swapper/222:15:302
291372850,0sleep00-21swapper/023:13:170
291372850,0sleep00-21swapper/023:13:160
265482640,0sleep10-21swapper/121:57:181
88122610,0sleep10-21swapper/122:49:011
3155926150,7sleep20-21swapper/219:09:382
171092610,0sleep00-21swapper/023:55:270
97482600,1sleep39745-21lspci22:16:113
319382600,0sleep031935-21lspci22:02:450
85062590,1sleep38495-21bash00:05:193
131632590,0sleep00-21swapper/023:17:050
283052580,1sleep128303-21grepconf.sh21:44:511
251322580,2sleep23170299cyclictest21:44:362
249712580,1sleep324967-21bash23:30:343
208382580,0sleep00-21swapper/023:40:210
201282580,1sleep120125-21lspci00:03:381
146272580,0sleep30-21swapper/321:18:253
133052580,1sleep213300-21lspci22:56:512
117012580,0sleep0111rcuc/023:34:400
67692570,0sleep0101ktimersoftd/023:26:380
3152425734,7sleep10-21swapper/119:09:131
238012570,0sleep30-21swapper/322:37:343
61862560,1sleep16181-21lspci22:13:231
191562560,1sleep019154-21kworker/u8:222:24:320
106122550,0sleep00-21swapper/000:16:150
83912540,2sleep13168599cyclictest23:04:051
66122540,1sleep31-21systemd23:39:203
3118825432,7sleep00-21swapper/019:05:310
30012540,0sleep30-21swapper/322:25:443
18772540,0sleep10-21swapper/123:33:531
106752540,0sleep00-21swapper/021:48:320
3142425240,7sleep30-21swapper/319:08:023
302692510,0sleep20-21swapper/222:38:042
131772500,0sleep00-21swapper/021:18:180
144092420,1sleep0101ktimersoftd/000:35:330
37852280,0sleep03788-21bash00:10:170
63592220,0sleep30-21swapper/323:24:093
237362210,1sleep323738-21bash23:45:403
320652200,0sleep10-21swapper/122:43:161
193352200,0sleep30-21swapper/300:33:213
193352200,0sleep30-21swapper/300:33:213
308082190,0sleep20-21swapper/223:05:452
286612190,1sleep028662-21lspci00:04:210
209862190,0sleep30-21swapper/322:50:003
202192190,0sleep10-21swapper/100:27:581
161052190,1sleep116108-21bash23:42:351
107552190,0sleep00-21swapper/022:36:300
92112180,0sleep00-21swapper/022:33:550
50872180,0sleep20-21swapper/222:05:402
87322170,1sleep08726-21sshd00:05:210
50702170,0sleep20-21swapper/223:49:242
273912170,0sleep30-21swapper/322:10:013
188882170,1sleep018889-21bash22:27:010
3170299165,2cyclictest33-21ksoftirqd/223:41:152
31702991614,1cyclictest33-21ksoftirqd/221:30:372
218292160,0sleep2311rcuc/222:47:302
178732160,0sleep317868-21sshd21:36:243
165532160,0sleep2311rcuc/223:55:252
3170299155,3cyclictest33-21ksoftirqd/222:52:022
3170299155,1cyclictest33-21ksoftirqd/223:19:322
174642150,0sleep3401ktimersoftd/300:11:293
3170299134,1cyclictest33-21ksoftirqd/221:52:102
3170299134,1cyclictest33-21ksoftirqd/200:27:032
3170299133,3cyclictest33-21ksoftirqd/221:29:432
3170299133,2cyclictest33-21ksoftirqd/223:35:512
3170299132,4cyclictest33-21ksoftirqd/222:03:212
3170299131,4cyclictest33-21ksoftirqd/200:12:552
308612130,1sleep1241ktimersoftd/120:29:571
182772130,2sleep03167699cyclictest22:42:120
13992130,1sleep3391rcuc/322:58:293
3171099129,2cyclictest8937-21id22:31:173
3171099120,1cyclictest6475-21lspci21:48:103
3171099120,1cyclictest16549-21lspci22:09:103
3170299124,1cyclictest33-21ksoftirqd/221:20:522
3170299123,2cyclictest33-21ksoftirqd/222:41:362
3170299123,2cyclictest33-21ksoftirqd/221:55:052
3170299122,2cyclictest33-21ksoftirqd/221:37:412
3170299122,2cyclictest33-21ksoftirqd/221:11:262
3170299121,3cyclictest33-21ksoftirqd/223:25:452
3170299121,2cyclictest33-21ksoftirqd/200:31:262
3170299121,2cyclictest33-21ksoftirqd/200:31:252
31702991210,1cyclictest12308-21lspci22:26:302
3170299120,1cyclictest820-21dbus-daemon00:21:082
3170299120,1cyclictest5205-21bash23:11:232
3170299120,1cyclictest5205-21bash23:11:222
3170299120,1cyclictest25213-21bash23:30:352
31685991210,1cyclictest7804-21sshd22:51:251
31685991210,1cyclictest28116-21consoletype21:29:421
31685991210,1cyclictest11709-21sshd22:08:481
3168599120,1cyclictest32423-21sshd23:59:191
248042120,0sleep024806-21bash21:52:070
3171099119,1cyclictest6883-21sshd22:53:533
31710991110,0cyclictest0-21swapper/323:27:193
31710991110,0cyclictest0-21swapper/322:01:343
3170299119,1cyclictest33-21ksoftirqd/200:06:022
3170299113,1cyclictest33-21ksoftirqd/200:35:082
3170299111,3cyclictest33-21ksoftirqd/222:31:132
3170299111,2cyclictest33-21ksoftirqd/223:01:372
3170299111,2cyclictest33-21ksoftirqd/200:15:032
3170299111,1cyclictest131rcu_sched21:17:542
3170299110,3cyclictest1-21systemd22:21:582
3170299110,1cyclictest131rcu_sched21:47:582
3170299110,10cyclictest0-21swapper/222:12:452
3168599119,1cyclictest28489-21lspci23:18:181
3168599110,10cyclictest0-21swapper/123:11:551
3168599110,10cyclictest0-21swapper/123:11:541
3168599110,10cyclictest0-21swapper/122:29:461
3168599110,10cyclictest0-21swapper/100:15:521
3167699119,1cyclictest25324-21lspci23:02:510
31676991110,0cyclictest0-21swapper/000:32:170
31676991110,0cyclictest0-21swapper/000:32:160
3167699110,1cyclictest0-21swapper/022:15:300
3167699110,10cyclictest0-21swapper/023:45:550
322692100,0sleep0111rcuc/000:29:010
3171099109,0cyclictest0-21swapper/323:11:493
3171099109,0cyclictest0-21swapper/323:11:483
3171099109,0cyclictest0-21swapper/319:40:183
3171099109,0cyclictest0-21swapper/300:20:073
3171099108,1cyclictest1503-21lspci21:50:163
3171099100,0cyclictest0-21swapper/323:41:123
3171099100,0cyclictest0-21swapper/322:44:023
3170299109,0cyclictest0-21swapper/220:16:592
3170299102,2cyclictest33-21ksoftirqd/223:52:402
3170299101,1cyclictest33-21ksoftirqd/223:20:562
3170299100,9cyclictest0-21swapper/220:46:232
3170299100,1cyclictest131rcu_sched00:00:032
3168599109,0cyclictest0-21swapper/121:30:111
3168599109,0cyclictest0-21swapper/121:15:301
3168599109,0cyclictest0-21swapper/100:37:191
3168599108,1cyclictest22627-21sshd23:27:531
3168599100,0cyclictest0-21swapper/122:34:371
3168599100,0cyclictest0-21swapper/122:19:161
3167699109,0cyclictest0-21swapper/023:24:380
3167699109,0cyclictest0-21swapper/021:59:230
3167699100,9cyclictest28439-21kworker/0:222:58:090
3167699100,1cyclictest131rcu_sched00:21:360
3167699100,0cyclictest0-21swapper/021:10:460
317109998,0cyclictest18858-21lspci23:07:233
317109998,0cyclictest0-21swapper/321:55:263
317109998,0cyclictest0-21swapper/321:42:033
317109990,8cyclictest2732-21sshd23:16:163
317109990,8cyclictest2328-21sshd21:27:373
317109990,0cyclictest0-21swapper/320:50:133
317029990,0cyclictest0-21swapper/220:20:132
316859998,0cyclictest0-21swapper/123:50:111
316859998,0cyclictest0-21swapper/123:05:171
316859998,0cyclictest0-21swapper/122:04:441
316769997,1cyclictest27516-21lspci23:35:500
316769993,1cyclictest9-21ksoftirqd/021:32:520
316769991,1cyclictest131rcu_sched21:28:000
316769990,8cyclictest0-21swapper/019:21:470
316769990,7cyclictest29460-21kworker/0:323:52:560
316769990,1cyclictest21017-21lspci22:11:590
317109987,0cyclictest0-21swapper/321:07:003
317109987,0cyclictest0-21swapper/321:07:003
317109987,0cyclictest0-21swapper/320:20:593
317109987,0cyclictest0-21swapper/319:59:313
317109986,1cyclictest8921-21lspci23:54:533
317109986,1cyclictest7918-21lspci00:18:473
317109986,1cyclictest24648-21lspci23:00:153
317109986,1cyclictest20112-21lspci23:58:203
317109980,7cyclictest20981-21sshd22:22:063
317109980,7cyclictest0-21swapper/319:28:503
317029987,0cyclictest0-21swapper/219:21:562
317029981,1cyclictest33-21ksoftirqd/220:55:002
317029980,7cyclictest0-21swapper/220:09:342
316859987,1cyclictest17003-21lspci21:46:271
316859987,0cyclictest0-21swapper/123:21:261
316859987,0cyclictest0-21swapper/122:56:341
316859986,1cyclictest8068-21lspci22:23:391
316859986,1cyclictest7445-21lspci00:21:241
316859986,1cyclictest27513-21lspci23:35:501
316859986,1cyclictest19197-21lspci21:10:271
316859986,1cyclictest18828-21lspci21:51:381
316859986,1cyclictest12679-21lspci00:13:471
316859980,7cyclictest22965-21sshd22:37:291
316859980,7cyclictest17114-21sshd23:45:121
316859980,7cyclictest14161-21sshd21:36:061
316859980,7cyclictest11160-21sshd00:08:161
316769987,0cyclictest0-21swapper/019:47:270
316769986,1cyclictest9-21ksoftirqd/021:37:000
316769986,1cyclictest2446-21lspci22:45:580
316769984,1cyclictest9-21ksoftirqd/021:43:150
316769980,7cyclictest2822-21kworker/0:021:22:570
316769980,7cyclictest20250-21kworker/0:022:50:190
316769980,1cyclictest131rcu_sched23:05:160
317109976,1cyclictest4268-21lspci21:12:293
317109975,1cyclictest7035-21lspci00:35:023
317109975,1cyclictest3179-21lspci21:20:073
317109973,1cyclictest1-21systemd00:25:303
317109970,6cyclictest15557-21sshd00:00:303
317109970,6cyclictest1465-21sshd21:30:103
317109970,6cyclictest0-21swapper/320:03:523
317109970,0cyclictest0-21swapper/320:56:013
317029976,0cyclictest0-21swapper/219:46:292
317029973,1cyclictest33-21ksoftirqd/219:20:012
317029970,6cyclictest24713-21sshd20:11:272
317029970,6cyclictest0-21swapper/221:05:352
317029970,6cyclictest0-21swapper/221:05:352
316859976,0cyclictest0-21swapper/120:55:091
316859976,0cyclictest0-21swapper/119:25:221
316859972,2cyclictest32329-21polkitd00:32:291
316859972,2cyclictest32329-21polkitd00:32:281
316859970,6cyclictest1984-21sshd21:22:321
316859970,6cyclictest16953-21sshd19:51:551
316859970,6cyclictest0-21swapper/120:42:531
316859970,6cyclictest0-21swapper/119:40:531
316859970,0cyclictest0-21swapper/119:59:171
316769970,6cyclictest28761-21sshd22:05:040
8912260,0sleep20-21swapper/220:55:142
317109965,0cyclictest0-21swapper/321:00:043
317109965,0cyclictest0-21swapper/319:37:333
317109964,1cyclictest2980-21hddtemp_smartct20:40:143
317109963,2cyclictest1398-21snmpd19:45:123
317109963,2cyclictest1397-21snmpd20:36:163
317109963,2cyclictest1397-21snmpd20:25:253
317109963,2cyclictest1397-21snmpd19:19:263
317029965,0cyclictest0-21swapper/220:31:412
317029964,1cyclictest16741-21tail19:50:192
317029964,1cyclictest1397-21snmpd20:29:312
317029963,2cyclictest1398-21snmpd19:31:212
317029960,2cyclictest13381-21turbostat19:45:002
316859965,0cyclictest0-21swapper/120:37:331
316859965,0cyclictest0-21swapper/119:11:011
316859964,1cyclictest21750-21grep20:05:141
316859963,2cyclictest1397-21snmpd20:47:411
316859963,2cyclictest1397-21snmpd20:33:191
316859960,5cyclictest28510-21sshd20:20:271
316859960,0cyclictest0-21swapper/120:15:261
316769964,1cyclictest11896-21sshd21:02:100
316769963,2cyclictest1397-21snmpd21:05:560
316769963,2cyclictest1397-21snmpd21:05:560
316769963,2cyclictest1397-21snmpd20:34:040
316769963,2cyclictest1397-21snmpd20:23:250
316769960,5cyclictest0-21swapper/020:42:210
316769960,5cyclictest0-21swapper/019:37:360
316769960,1cyclictest6317-21crond20:50:000
317109953,1cyclictest8898-21sort19:30:203
317109953,1cyclictest4991-21cat20:45:133
317109953,1cyclictest32102-21cstates19:10:113
317109953,1cyclictest31791-21cut20:30:163
317109953,1cyclictest25698-21cat20:15:123
317109953,1cyclictest1398-21snmpd20:10:373
317109952,2cyclictest5124-21sed19:20:193
317109952,2cyclictest21651-21cat20:05:123
317109952,2cyclictest15936-21cat19:50:123
317029953,1cyclictest17491-21/usr/sbin/munin19:55:182
317029953,1cyclictest1398-21snmpd20:40:412
317029953,1cyclictest1398-21snmpd19:25:142
317029953,1cyclictest1397-21snmpd20:37:132
317029953,1cyclictest10153-21cat19:35:132
317029952,2cyclictest19619-21cut20:00:122
317029952,2cyclictest1398-21snmpd19:10:052
317029951,0cyclictest33-21ksoftirqd/221:00:002
316859953,1cyclictest4321-21irqstats19:20:131
316859953,1cyclictest2367-21cat19:15:141
316859953,1cyclictest1398-21snmpd21:00:171
316859953,1cyclictest1398-21snmpd20:00:271
316859953,1cyclictest1398-21snmpd19:36:241
316859953,1cyclictest1398-21snmpd19:30:031
316859953,1cyclictest1397-21snmpd21:05:301
316859953,1cyclictest1397-21snmpd21:05:291
316859953,1cyclictest1397-21snmpd20:51:291
316859953,1cyclictest1397-21snmpd20:10:591
316859953,1cyclictest1397-21snmpd19:45:241
316769953,1cyclictest6680-21cpuspeed_turbos20:50:120
316769953,1cyclictest29171-21/usr/sbin/munin20:25:100
316769953,1cyclictest23424-21cpuspeed20:10:110
316769953,1cyclictest22254-21awk20:05:180
316769953,1cyclictest1967-21head20:35:210
316769953,1cyclictest15631-21cpuspeed_turbos19:50:100
316769953,1cyclictest1398-21snmpd20:55:300
316769953,1cyclictest1398-21snmpd19:55:180
316769953,1cyclictest1398-21snmpd19:15:500
316769953,1cyclictest1397-21snmpd20:15:070
316769953,1cyclictest1397-21snmpd20:00:480
316769953,1cyclictest1397-21snmpd19:33:120
316769953,1cyclictest11963-21df19:40:110
316769952,2cyclictest32270-21basename19:10:130
316769952,2cyclictest1397-21snmpd19:25:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional