You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-04 - 02:27

x86 Intel Xeon i3-1578L v5 @2000 MHz, Linux 4.19.246-rt110 (Profile)

Latency plot of system in rack #5, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3.osadl.org (updated Sun Dec 04, 2022 00:50:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26722994114,26cyclictest28226-21ssh21:30:266
26687993837,1cyclictest0-21swapper/121:30:231
26681993810,0cyclictest0-21swapper/021:00:150
2669799370,0cyclictest0-21swapper/220:30:182
26715993434,0cyclictest0-21swapper/520:25:215
26687993333,0cyclictest0-21swapper/119:45:131
26687993333,0cyclictest0-21swapper/119:45:121
26710993130,0cyclictest0-21swapper/419:50:124
26697993117,6cyclictest121rcu_preempt21:31:372
2670399301,11cyclictest0-21swapper/300:10:183
26687993010,5cyclictest0-21swapper/119:15:221
26710992929,0cyclictest0-21swapper/420:15:174
26681992928,1cyclictest0-21swapper/019:50:120
26728992828,0cyclictest0-21swapper/720:29:407
2669799289,15cyclictest33-21ksoftirqd/219:40:232
2669799287,5cyclictest33-21ksoftirqd/220:35:212
26697992810,13cyclictest33-21ksoftirqd/220:29:592
26681992827,1cyclictest0-21swapper/020:15:170
26715992710,17cyclictest0-21swapper/523:04:495
26687992727,0cyclictest0-21swapper/100:11:261
26728992626,0cyclictest0-21swapper/719:40:187
26697992610,4cyclictest0-21swapper/220:40:122
26697992610,4cyclictest0-21swapper/220:40:112
26722992524,1cyclictest0-21swapper/622:54:296
26722992523,2cyclictest0-21swapper/622:15:266
2669799259,15cyclictest33-21ksoftirqd/219:35:222
26728992416,5cyclictest721ktimersoftd/720:50:207
26722992424,0cyclictest0-21swapper/620:20:146
26715992424,0cyclictest0-21swapper/523:15:235
26715992424,0cyclictest0-21swapper/521:00:145
26697992412,11cyclictest0-21swapper/219:10:222
26697992411,12cyclictest0-21swapper/220:55:162
26722992313,8cyclictest729-21cron19:15:016
26710992322,0cyclictest0-21swapper/420:30:174
26697992323,0cyclictest0-21swapper/219:45:122
26697992323,0cyclictest0-21swapper/219:45:122
26697992321,1cyclictest32378-21ls19:15:212
26697992317,5cyclictest21357-21perf19:59:592
26687992310,4cyclictest0-21swapper/119:55:221
26681992314,3cyclictest101ktimersoftd/021:35:010
2672299220,17cyclictest0-21swapper/619:55:216
2669799227,8cyclictest0-21swapper/223:15:272
26697992221,1cyclictest321ktimersoftd/220:46:402
26697992217,4cyclictest321ktimersoftd/219:20:112
26687992221,1cyclictest1739-21sendmail-mta23:09:431
26687992221,0cyclictest0-21swapper/119:10:221
26681992220,1cyclictest0-21swapper/020:30:180
26722992110,6cyclictest0-21swapper/620:30:236
26715992110,1cyclictest0-21swapper/519:45:225
26715992110,1cyclictest0-21swapper/519:45:225
26710992121,0cyclictest0-21swapper/420:50:184
26710992110,3cyclictest0-21swapper/419:45:214
26710992110,3cyclictest0-21swapper/419:45:214
26703992121,0cyclictest0-21swapper/319:15:113
26703992110,3cyclictest0-21swapper/319:20:113
26687992110,5cyclictest0-21swapper/119:40:121
2668199210,9cyclictest10737-21cron19:40:000
26728992015,5cyclictest729-21cron23:40:017
26728992010,4cyclictest0-21swapper/720:40:117
26728992010,4cyclictest0-21swapper/720:40:117
26722992018,2cyclictest0-21swapper/623:37:496
26722992013,4cyclictest641ktimersoftd/619:35:226
2672299200,6cyclictest0-21swapper/619:15:116
26715992010,8cyclictest0-21swapper/522:55:155
26715992010,8cyclictest0-21swapper/521:34:485
26715992010,4cyclictest0-21swapper/520:15:235
26710992010,5cyclictest49-21ksoftirqd/419:20:124
2669799206,11cyclictest0-21swapper/220:15:232
26687992010,3cyclictest0-21swapper/120:15:221
26681992019,1cyclictest0-21swapper/020:50:180
26681992010,4cyclictest0-21swapper/019:15:000
26728991910,4cyclictest0-21swapper/720:35:137
26728991910,4cyclictest0-21swapper/719:15:117
26728991910,3cyclictest0-21swapper/719:10:217
26728991910,0cyclictest0-21swapper/722:30:227
26715991919,0cyclictest76350irq/125-eth022:20:245
26715991916,3cyclictest0-21swapper/521:18:515
26715991910,4cyclictest0-21swapper/521:35:005
26687991910,3cyclictest0-21swapper/120:20:241
26687991910,3cyclictest0-21swapper/119:35:191
26681991910,2cyclictest0-21swapper/019:40:200
26722991810,3cyclictest0-21swapper/623:15:236
26715991810,3cyclictest0-21swapper/520:50:225
26710991817,0cyclictest16199-1kworker/u17:000:20:294
26710991810,6cyclictest0-21swapper/421:32:374
26703991810,3cyclictest0-21swapper/319:45:193
26703991810,3cyclictest0-21swapper/319:45:193
2669799184,13cyclictest0-21swapper/220:50:212
26687991818,0cyclictest0-21swapper/122:55:161
26687991812,2cyclictest15421-21uptime20:45:231
26687991810,2cyclictest0-21swapper/120:30:221
26687991810,1cyclictest0-21swapper/120:35:201
26687991810,0cyclictest0-21swapper/123:03:251
26681991810,4cyclictest0-21swapper/019:55:220
26681991810,3cyclictest0-21swapper/022:55:070
26728991710,5cyclictest0-21swapper/721:00:157
26728991710,4cyclictest0-21swapper/700:25:147
2672299175,7cyclictest0-21swapper/619:45:186
2672299175,7cyclictest0-21swapper/619:45:186
26722991715,0cyclictest6823-21nscd21:11:156
26722991712,4cyclictest703-21nscd23:41:206
26722991710,5cyclictest0-21swapper/620:25:236
26722991710,3cyclictest0-21swapper/620:45:216
26722991710,3cyclictest0-21swapper/619:40:226
26715991710,4cyclictest0-21swapper/519:40:235
26710991710,4cyclictest0-21swapper/419:15:174
26710991710,2cyclictest0-21swapper/423:15:264
26710991710,2cyclictest0-21swapper/422:55:164
26710991710,2cyclictest0-21swapper/420:20:224
26710991710,0cyclictest0-21swapper/420:35:094
26710991710,0cyclictest0-21swapper/400:10:394
26703991715,2cyclictest703-21nscd23:15:343
26703991710,2cyclictest0-21swapper/320:55:213
26703991710,2cyclictest0-21swapper/320:15:213
2669799176,10cyclictest33-21ksoftirqd/200:05:162
26697991716,1cyclictest24940-21ssh23:51:592
26697991715,2cyclictest321ktimersoftd/222:55:032
26697991712,5cyclictest0-21swapper/220:20:152
26687991717,0cyclictest0-21swapper/120:25:151
26687991710,3cyclictest0-21swapper/123:15:251
2668799170,16cyclictest0-21swapper/100:22:141
26681991710,4cyclictest0-21swapper/020:35:160
26681991710,3cyclictest0-21swapper/020:00:120
26681991710,1cyclictest0-21swapper/019:10:210
26728991610,6cyclictest0-21swapper/720:15:197
26728991610,4cyclictest0-21swapper/723:15:327
26728991610,4cyclictest0-21swapper/721:31:147
26728991610,3cyclictest0-21swapper/720:30:237
26728991610,3cyclictest0-21swapper/719:45:227
26728991610,3cyclictest0-21swapper/719:45:227
26728991610,0cyclictest0-21swapper/723:43:007
26722991610,3cyclictest0-21swapper/622:59:516
26722991610,3cyclictest0-21swapper/621:53:466
26722991610,2cyclictest0-21swapper/620:35:146
26722991610,0cyclictest0-21swapper/619:20:116
26715991615,1cyclictest76350irq/125-eth023:07:445
26715991610,4cyclictest0-21swapper/519:10:225
26715991610,3cyclictest0-21swapper/520:55:185
26715991610,3cyclictest0-21swapper/520:20:155
26715991610,2cyclictest0-21swapper/520:35:115
26715991610,0cyclictest0-21swapper/519:39:545
26710991610,3cyclictest0-21swapper/423:45:124
26710991610,3cyclictest0-21swapper/420:47:394
26710991610,2cyclictest0-21swapper/422:33:114
26710991610,2cyclictest0-21swapper/421:35:014
26710991610,0cyclictest0-21swapper/423:10:514
26703991610,4cyclictest0-21swapper/322:05:133
26703991610,4cyclictest0-21swapper/320:35:163
26703991610,0cyclictest0-21swapper/321:42:113
2669799169,1cyclictest121rcu_preempt00:33:152
26697991615,1cyclictest321ktimersoftd/222:47:142
26697991615,1cyclictest27635-21kworker/2:222:35:232
26687991614,1cyclictest703-21nscd22:27:371
26687991610,6cyclictest0-21swapper/123:39:211
26687991610,6cyclictest0-21swapper/123:33:331
26687991610,4cyclictest0-21swapper/120:50:201
26687991610,3cyclictest0-21swapper/122:40:231
26687991610,0cyclictest0-21swapper/123:49:511
26687991610,0cyclictest0-21swapper/123:29:581
26687991610,0cyclictest0-21swapper/121:51:261
26681991612,0cyclictest25144-21kworker/u16:223:22:290
26681991610,6cyclictest0-21swapper/020:20:200
26681991610,4cyclictest0-21swapper/022:21:520
26681991610,3cyclictest0-21swapper/019:20:120
26681991610,0cyclictest0-21swapper/023:15:260
26681991610,0cyclictest0-21swapper/021:52:260
26728991510,5cyclictest0-21swapper/700:34:297
26728991510,5cyclictest0-21swapper/700:21:267
26728991510,5cyclictest0-21swapper/700:14:537
26728991510,3cyclictest0-21swapper/719:50:117
26728991510,3cyclictest0-21swapper/719:20:127
26728991510,2cyclictest0-21swapper/720:45:187
26728991510,2cyclictest0-21swapper/700:02:407
26722991510,5cyclictest0-21swapper/623:20:346
26722991510,5cyclictest0-21swapper/600:28:096
26722991510,3cyclictest0-21swapper/620:50:196
26722991510,2cyclictest0-21swapper/623:05:166
26722991510,2cyclictest0-21swapper/621:05:216
26722991510,2cyclictest0-21swapper/620:55:186
26722991510,0cyclictest0-21swapper/623:59:336
26722991510,0cyclictest0-21swapper/623:49:376
26722991510,0cyclictest0-21swapper/621:42:156
26722991510,0cyclictest0-21swapper/600:22:216
26722991510,0cyclictest0-21swapper/600:06:326
2671599152,5cyclictest0-21swapper/520:45:225
26715991513,1cyclictest16422-21apt-get19:50:125
26715991511,0cyclictest21174-21ls19:55:225
26715991510,5cyclictest0-21swapper/522:36:085
26715991510,5cyclictest0-21swapper/500:15:325
26715991510,3cyclictest0-21swapper/519:15:015
26715991510,2cyclictest0-21swapper/520:30:175
26715991510,0cyclictest0-21swapper/523:57:075
26715991510,0cyclictest0-21swapper/521:56:105
26710991510,5cyclictest0-21swapper/422:47:214
26710991510,5cyclictest0-21swapper/422:15:284
26710991510,3cyclictest0-21swapper/419:55:214
26710991510,3cyclictest0-21swapper/400:25:214
26710991510,2cyclictest0-21swapper/419:40:184
26710991510,2cyclictest0-21swapper/419:35:204
26710991510,0cyclictest0-21swapper/420:25:254
26710991510,0cyclictest0-21swapper/400:05:174
26703991513,1cyclictest26281-21diskmemload22:12:143
26703991510,2cyclictest0-21swapper/321:32:583
26703991510,2cyclictest0-21swapper/320:30:233
26703991510,2cyclictest0-21swapper/319:55:193
26697991514,1cyclictest6880-21ssh23:00:102
26697991514,1cyclictest321ktimersoftd/223:55:462
26697991514,1cyclictest321ktimersoftd/223:45:222
26697991514,1cyclictest321ktimersoftd/223:38:122
26697991514,1cyclictest321ktimersoftd/223:27:172
26697991514,1cyclictest321ktimersoftd/223:09:502
26697991514,1cyclictest321ktimersoftd/222:07:262
26697991514,1cyclictest321ktimersoftd/200:25:052
26697991513,1cyclictest321ktimersoftd/200:00:182
26697991512,2cyclictest1606-21ssh21:35:002
26697991511,1cyclictest5984-21kworker/2:100:39:552
26697991510,0cyclictest0-21swapper/200:15:152
26687991515,0cyclictest0-21swapper/121:05:001
26687991510,5cyclictest0-21swapper/122:37:591
26687991510,4cyclictest0-21swapper/123:44:541
26687991510,3cyclictest0-21swapper/120:55:201
26687991510,0cyclictest0-21swapper/123:50:031
26687991510,0cyclictest0-21swapper/120:00:111
26687991510,0cyclictest0-21swapper/100:15:221
26681991514,1cyclictest996-21snmpd23:42:200
26681991511,4cyclictest803-21gmain23:47:440
26681991510,4cyclictest0-21swapper/000:31:310
26681991510,3cyclictest0-21swapper/022:12:560
26681991510,2cyclictest0-21swapper/023:55:220
26681991510,1cyclictest0-21swapper/022:05:290
26681991510,0cyclictest0-21swapper/023:39:560
26681991510,0cyclictest0-21swapper/022:36:330
26681991510,0cyclictest0-21swapper/021:55:570
26681991510,0cyclictest0-21swapper/021:39:310
2668199150,0cyclictest0-21swapper/023:29:000
26728991414,0cyclictest0-21swapper/722:36:387
26728991414,0cyclictest0-21swapper/722:02:357
26728991410,4cyclictest0-21swapper/723:50:157
26728991410,4cyclictest0-21swapper/722:52:267
26728991410,4cyclictest0-21swapper/722:22:267
26728991410,4cyclictest0-21swapper/721:55:177
26728991410,4cyclictest0-21swapper/720:10:217
26728991410,4cyclictest0-21swapper/719:35:197
26728991410,4cyclictest0-21swapper/700:19:037
26728991410,3cyclictest0-21swapper/722:55:147
26728991410,2cyclictest0-21swapper/723:20:187
26728991410,1cyclictest0-21swapper/719:55:227
26728991410,0cyclictest0-21swapper/723:55:307
26728991410,0cyclictest0-21swapper/723:30:207
26728991410,0cyclictest0-21swapper/723:11:047
26728991410,0cyclictest0-21swapper/723:06:447
26728991410,0cyclictest0-21swapper/723:02:537
26728991410,0cyclictest0-21swapper/721:54:017
26728991410,0cyclictest0-21swapper/721:40:567
26728991410,0cyclictest0-21swapper/721:35:037
26728991410,0cyclictest0-21swapper/721:26:587
26728991410,0cyclictest0-21swapper/721:10:017
26728991410,0cyclictest0-21swapper/720:55:207
26728991410,0cyclictest0-21swapper/700:35:447
26728991410,0cyclictest0-21swapper/700:08:287
26722991414,0cyclictest0-21swapper/622:26:126
26722991414,0cyclictest0-21swapper/621:57:116
26722991410,4cyclictest0-21swapper/620:16:336
26722991410,3cyclictest0-21swapper/621:35:006
26722991410,0cyclictest0-21swapper/623:34:056
26722991410,0cyclictest0-21swapper/623:27:226
26722991410,0cyclictest0-21swapper/623:11:236
26722991410,0cyclictest0-21swapper/622:49:276
26722991410,0cyclictest0-21swapper/622:36:326
26722991410,0cyclictest0-21swapper/622:10:206
26722991410,0cyclictest0-21swapper/621:27:566
26722991410,0cyclictest0-21swapper/619:29:066
26722991410,0cyclictest0-21swapper/600:35:166
26722991410,0cyclictest0-21swapper/600:31:526
26722991410,0cyclictest0-21swapper/600:15:196
26722991410,0cyclictest0-21swapper/600:10:226
26722991410,0cyclictest0-21swapper/600:01:106
26715991414,0cyclictest0-21swapper/519:30:115
26715991413,1cyclictest703-21nscd23:45:155
26715991411,1cyclictest76350irq/125-eth000:01:545
26715991410,4cyclictest0-21swapper/522:45:265
26715991410,4cyclictest0-21swapper/522:32:505
26715991410,4cyclictest0-21swapper/522:03:205
26715991410,4cyclictest0-21swapper/521:44:035
26715991410,4cyclictest0-21swapper/500:10:265
26715991410,3cyclictest0-21swapper/519:25:205
26715991410,0cyclictest0-21swapper/523:40:085
26715991410,0cyclictest0-21swapper/522:40:225
26715991410,0cyclictest0-21swapper/522:25:255
26715991410,0cyclictest0-21swapper/522:18:525
26715991410,0cyclictest0-21swapper/521:51:575
26715991410,0cyclictest0-21swapper/520:10:215
26715991410,0cyclictest0-21swapper/500:35:165
26710991414,0cyclictest17550irq/127-ahci[0019:31:364
26710991414,0cyclictest0-21swapper/423:52:584
26710991413,1cyclictest1006-1kworker/4:0H23:37:544
26710991410,4cyclictest0-21swapper/423:55:214
26710991410,4cyclictest0-21swapper/423:05:254
26710991410,4cyclictest0-21swapper/422:52:214
26710991410,4cyclictest0-21swapper/422:26:274
26710991410,4cyclictest0-21swapper/422:08:004
26710991410,4cyclictest0-21swapper/422:02:344
26710991410,1cyclictest121rcu_preempt22:21:494
26710991410,1cyclictest0-21swapper/421:45:114
26710991410,1cyclictest0-21swapper/420:55:214
26710991410,1cyclictest0-21swapper/400:30:204
26710991410,0cyclictest0-21swapper/423:30:204
26710991410,0cyclictest0-21swapper/423:01:494
26710991410,0cyclictest0-21swapper/422:41:114
26710991410,0cyclictest0-21swapper/422:35:464
26710991410,0cyclictest0-21swapper/422:11:054
26710991410,0cyclictest0-21swapper/421:21:534
26710991410,0cyclictest0-21swapper/421:17:264
26710991410,0cyclictest0-21swapper/400:18:544
2671099141,2cyclictest0-21swapper/420:40:124
2671099141,2cyclictest0-21swapper/420:40:114
26703991414,0cyclictest0-21swapper/321:45:193
26703991413,1cyclictest28846-21ssh21:51:103
26703991413,0cyclictest16199-1kworker/u17:000:20:413
26703991410,4cyclictest0-21swapper/323:27:503
26703991410,4cyclictest0-21swapper/322:20:243
26703991410,3cyclictest0-21swapper/321:00:013
26703991410,3cyclictest0-21swapper/320:50:183
26703991410,3cyclictest0-21swapper/319:40:193
26703991410,2cyclictest0-21swapper/320:45:213
26703991410,2cyclictest0-21swapper/319:10:223
26703991410,1cyclictest996-21snmpd22:46:183
26703991410,0cyclictest0-21swapper/323:52:403
26703991410,0cyclictest0-21swapper/323:42:013
26703991410,0cyclictest0-21swapper/323:35:253
26703991410,0cyclictest0-21swapper/321:35:553
26703991410,0cyclictest0-21swapper/321:24:033
2669799149,3cyclictest33-21ksoftirqd/222:15:272
2669799148,4cyclictest121rcu_preempt22:14:562
26697991414,0cyclictest321ktimersoftd/200:13:462
26697991413,1cyclictest6990-21ssh00:21:402
26697991413,1cyclictest4111-21ps21:15:202
26697991413,1cyclictest321ktimersoftd/223:43:492
26697991413,1cyclictest321ktimersoftd/223:20:222
26697991413,1cyclictest321ktimersoftd/222:52:452
26697991413,1cyclictest321ktimersoftd/221:55:412
26697991413,1cyclictest321ktimersoftd/221:00:212
26697991410,0cyclictest0-21swapper/223:10:202
26697991410,0cyclictest0-21swapper/222:21:482
26697991410,0cyclictest0-21swapper/221:40:112
26687991414,0cyclictest0-21swapper/100:31:421
26687991410,4cyclictest803-21gmain21:55:041
26687991410,4cyclictest0-21swapper/122:45:531
26687991410,4cyclictest0-21swapper/121:21:461
26687991410,4cyclictest0-21swapper/100:39:191
26687991410,3cyclictest0-21swapper/121:35:011
26687991410,2cyclictest0-21swapper/123:11:531
26687991410,0cyclictest0-21swapper/123:55:221
26687991410,0cyclictest0-21swapper/123:20:181
26687991410,0cyclictest0-21swapper/122:05:151
26687991410,0cyclictest0-21swapper/121:45:231
26687991410,0cyclictest0-21swapper/121:40:591
26687991410,0cyclictest0-21swapper/121:25:541
26687991410,0cyclictest0-21swapper/121:15:211
26687991410,0cyclictest0-21swapper/121:11:031
26687991410,0cyclictest0-21swapper/100:00:221
26681991413,1cyclictest4300-21ssh22:16:510
26681991413,1cyclictest26281-21diskmemload22:03:360
26681991413,1cyclictest19536-21ssh22:47:040
26681991412,1cyclictest24665-21cstates20:05:150
26681991410,4cyclictest0-21swapper/022:30:120
26681991410,4cyclictest0-21swapper/000:39:500
26681991410,2cyclictest0-21swapper/019:45:190
26681991410,2cyclictest0-21swapper/019:45:190
26681991410,1cyclictest9-21ksoftirqd/020:25:230
26681991410,0cyclictest0-21swapper/021:28:410
26681991410,0cyclictest0-21swapper/021:05:120
26681991410,0cyclictest0-21swapper/020:48:390
26681991410,0cyclictest0-21swapper/020:40:170
26681991410,0cyclictest0-21swapper/020:40:160
26728991313,0cyclictest0-21swapper/723:46:547
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional