You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-09 - 00:20

x86 Intel Xeon i3-1578L v5 @2000 MHz, Linux 4.19.246-rt110 (Profile)

Latency plot of system in rack #5, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3.osadl.org (updated Mon Aug 08, 2022 12:49:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
581699420,0cyclictest0-21swapper/609:25:266
5811994140,0cyclictest0-21swapper/108:15:121
5815994039,0cyclictest0-21swapper/508:30:135
5810994040,0cyclictest0-21swapper/011:40:260
5817993838,0cyclictest0-21swapper/707:10:017
581599370,37cyclictest0-21swapper/508:35:225
581699360,35cyclictest22416-21apt-get08:40:156
581299360,0cyclictest0-21swapper/207:55:132
5810993620,9cyclictest121rcu_preempt07:10:180
5811993533,1cyclictest30736-21apt-get07:55:131
5813993434,0cyclictest0-21swapper/307:50:223
5817993333,0cyclictest0-21swapper/709:10:227
5813993333,0cyclictest0-21swapper/307:15:173
5810993231,0cyclictest0-21swapper/007:35:150
5810993225,6cyclictest101ktimersoftd/008:05:290
5817993131,0cyclictest0-21swapper/707:20:157
5810993127,3cyclictest0-21swapper/008:20:110
5810993121,6cyclictest101ktimersoftd/008:30:190
5816993028,1cyclictest0-21swapper/607:39:296
5810993025,4cyclictest0-21swapper/007:55:130
581099296,22cyclictest0-21swapper/008:35:280
581099296,22cyclictest0-21swapper/008:00:260
5811992810,0cyclictest0-21swapper/107:27:561
581099286,21cyclictest0-21swapper/007:45:130
5810992828,0cyclictest0-21swapper/008:55:020
5810992815,12cyclictest111rcuc/008:40:210
5810992815,12cyclictest0-21swapper/009:10:000
5812992710,0cyclictest0-21swapper/207:35:432
581099278,15cyclictest0-21swapper/008:10:240
5810992719,7cyclictest8332-21ssh09:50:320
5810992718,8cyclictest0-21swapper/008:15:200
5810992710,0cyclictest0-21swapper/010:15:530
5817992626,0cyclictest0-21swapper/708:15:127
5816992610,16cyclictest0-21swapper/608:00:166
5815992626,0cyclictest0-21swapper/507:10:135
5810992619,2cyclictest101ktimersoftd/007:25:280
5816992510,6cyclictest0-21swapper/607:40:346
581099257,0cyclictest0-21swapper/007:20:300
5810992511,7cyclictest0-21swapper/007:40:360
5816992423,1cyclictest0-21swapper/608:55:136
581499241,9cyclictest121rcu_preempt08:35:284
5812992424,0cyclictest0-21swapper/208:45:202
5810992416,5cyclictest9-21ksoftirqd/009:14:370
5815992310,3cyclictest0-21swapper/508:10:255
5814992323,0cyclictest0-21swapper/411:40:264
5813992315,8cyclictest703-21nscd11:41:563
5810992322,1cyclictest101ktimersoftd/012:20:200
5817992222,0cyclictest0-21swapper/712:05:307
5817992210,4cyclictest0-21swapper/708:40:247
5816992221,1cyclictest0-21swapper/609:07:556
5815992222,0cyclictest76350irq/125-eth007:32:575
5813992222,0cyclictest0-21swapper/308:05:193
5813992212,5cyclictest0-21swapper/307:30:393
5812992222,0cyclictest0-21swapper/208:35:162
5812992210,5cyclictest0-21swapper/208:05:242
5811992220,1cyclictest1739-21sendmail-mta11:17:431
5810992222,0cyclictest101ktimersoftd/010:55:250
5810992221,1cyclictest0-21swapper/010:51:040
5810992220,1cyclictest101ktimersoftd/007:50:010
5810992217,1cyclictest101ktimersoftd/008:50:200
5810992215,6cyclictest9108-21cpu07:15:010
5816992121,0cyclictest0-21swapper/611:25:006
5816992110,4cyclictest0-21swapper/608:35:266
5816992110,3cyclictest0-21swapper/611:25:146
5815992110,6cyclictest0-21swapper/509:10:485
5814992118,2cyclictest481ktimersoftd/407:40:364
5813992110,3cyclictest0-21swapper/308:55:013
5811992110,4cyclictest0-21swapper/108:50:231
581099218,12cyclictest0-21swapper/008:25:230
5810992114,7cyclictest15330-1kworker/0:0H11:07:080
5817992010,4cyclictest0-21swapper/709:50:317
5816992010,5cyclictest0-21swapper/609:53:126
5815992010,2cyclictest0-21swapper/508:05:195
5814992010,2cyclictest0-21swapper/408:40:254
5813992011,8cyclictest744-21in:imuxsock09:45:433
5813992010,5cyclictest0-21swapper/308:45:213
5813992010,2cyclictest0-21swapper/307:45:143
5811992020,0cyclictest0-21swapper/108:45:201
5811992010,4cyclictest0-21swapper/109:50:321
5811992010,3cyclictest0-21swapper/108:10:261
5811992010,10cyclictest0-21swapper/108:03:301
581099205,11cyclictest0-21swapper/008:45:250
5810992015,4cyclictest0-21swapper/007:30:160
5810992014,5cyclictest0-21swapper/009:25:260
581099201,16cyclictest27892-21sed11:25:010
5817991919,0cyclictest0-21swapper/708:55:017
5817991917,1cyclictest0-21swapper/708:00:007
5817991910,5cyclictest0-21swapper/708:10:247
5817991910,3cyclictest0-21swapper/708:00:207
5816991913,3cyclictest8620-21timerandwakeup09:10:276
5816991910,8cyclictest0-21swapper/608:45:216
5816991910,3cyclictest0-21swapper/607:10:236
5816991910,2cyclictest0-21swapper/607:30:166
5815991919,0cyclictest0-21swapper/507:45:135
5815991910,3cyclictest0-21swapper/509:51:395
5815991910,3cyclictest0-21swapper/508:40:255
5815991910,3cyclictest0-21swapper/507:25:305
5814991910,6cyclictest0-21swapper/407:10:224
5814991910,5cyclictest0-21swapper/409:11:424
5813991915,4cyclictest707-21nscd10:40:583
5813991915,4cyclictest707-21nscd10:40:573
5813991910,6cyclictest0-21swapper/309:10:273
5812991910,5cyclictest0-21swapper/207:20:292
5812991910,3cyclictest0-21swapper/209:10:282
5812991910,3cyclictest0-21swapper/208:30:142
5812991910,1cyclictest0-21swapper/211:42:172
5811991910,4cyclictest0-21swapper/111:42:021
5811991910,3cyclictest0-21swapper/107:10:221
581099198,11cyclictest0-21swapper/011:25:030
5810991919,0cyclictest9-21ksoftirqd/009:55:140
5817991818,0cyclictest0-21swapper/709:15:167
5817991810,5cyclictest0-21swapper/707:40:267
5817991810,2cyclictest0-21swapper/708:05:187
5817991810,0cyclictest0-21swapper/709:30:087
581799180,5cyclictest0-21swapper/708:30:137
5816991810,4cyclictest0-21swapper/611:44:146
5815991818,0cyclictest0-21swapper/508:00:165
5814991818,0cyclictest32637-21ls07:55:224
5814991816,1cyclictest11821-21apt-get08:20:114
5814991813,5cyclictest0-21swapper/407:15:444
5814991810,2cyclictest0-21swapper/408:50:234
5813991816,2cyclictest703-21nscd09:53:123
5813991810,3cyclictest0-21swapper/311:25:003
5813991810,3cyclictest0-21swapper/307:20:293
5813991810,1cyclictest0-21swapper/307:55:133
5813991810,0cyclictest0-21swapper/308:15:203
5812991810,0cyclictest0-21swapper/208:15:212
5811991810,7cyclictest0-21swapper/107:15:431
5811991810,3cyclictest0-21swapper/108:05:251
5811991810,2cyclictest0-21swapper/111:25:131
5811991810,2cyclictest0-21swapper/108:35:171
5810991816,1cyclictest101ktimersoftd/012:10:280
5817991717,0cyclictest0-21swapper/708:45:157
5817991710,7cyclictest703-21nscd11:40:247
5817991710,5cyclictest0-21swapper/709:05:237
5817991710,2cyclictest0-21swapper/708:50:227
5817991710,0cyclictest0-21swapper/707:50:287
5816991710,2cyclictest0-21swapper/607:20:316
5816991710,2cyclictest0-21swapper/607:15:426
5816991710,1cyclictest0-21swapper/608:25:226
5816991710,0cyclictest0-21swapper/612:15:196
5816991710,0cyclictest0-21swapper/608:15:216
5816991710,0cyclictest0-21swapper/608:10:246
5815991717,0cyclictest76350irq/125-eth010:40:055
5815991717,0cyclictest76350irq/125-eth010:40:055
5815991710,3cyclictest0-21swapper/508:50:245
5815991710,2cyclictest0-21swapper/511:44:475
5815991710,2cyclictest0-21swapper/507:35:195
5814991710,7cyclictest0-21swapper/409:51:144
5813991710,4cyclictest0-21swapper/308:00:183
5813991710,3cyclictest0-21swapper/308:30:243
5813991710,3cyclictest0-21swapper/307:40:133
5812991713,1cyclictest27663-21latency_hist11:25:002
5812991712,3cyclictest9073-21latency_hist08:15:002
5812991710,4cyclictest0-21swapper/207:30:372
5812991710,2cyclictest0-21swapper/207:40:342
5812991710,0cyclictest0-21swapper/209:54:592
581199177,8cyclictest0-21swapper/109:11:481
5811991716,1cyclictest0-21swapper/110:33:231
5811991710,7cyclictest0-21swapper/112:25:221
5811991710,7cyclictest0-21swapper/108:55:001
5811991710,4cyclictest0-21swapper/109:25:251
5811991710,3cyclictest0-21swapper/108:25:221
5811991710,2cyclictest0-21swapper/108:20:111
5811991710,2cyclictest0-21swapper/107:45:131
5810991716,1cyclictest101ktimersoftd/011:53:490
5810991716,1cyclictest101ktimersoftd/011:36:400
5810991716,1cyclictest101ktimersoftd/011:18:090
5810991716,1cyclictest101ktimersoftd/010:35:450
5810991716,1cyclictest101ktimersoftd/010:32:050
5810991715,1cyclictest726-21avahi-daemon11:45:500
5810991713,1cyclictest101ktimersoftd/010:07:450
5810991710,3cyclictest0-21swapper/009:15:160
5817991616,0cyclictest0-21swapper/711:18:007
5817991616,0cyclictest0-21swapper/707:30:167
5817991610,4cyclictest0-21swapper/709:55:097
5817991610,3cyclictest0-21swapper/712:36:017
5817991610,3cyclictest0-21swapper/708:25:217
5817991610,3cyclictest0-21swapper/707:35:437
5817991610,3cyclictest0-21swapper/707:25:307
5816991610,2cyclictest0-21swapper/608:20:116
5816991610,1cyclictest0-21swapper/607:25:316
5816991610,0cyclictest0-21swapper/609:35:126
5816991610,0cyclictest0-21swapper/608:50:196
5815991615,1cyclictest16435-21ssh10:56:395
5815991610,5cyclictest0-21swapper/508:15:215
5815991610,0cyclictest0-21swapper/507:15:295
5814991613,2cyclictest0-21swapper/407:20:304
5814991610,2cyclictest0-21swapper/408:15:004
5813991613,2cyclictest803-21gmain11:49:203
5813991610,4cyclictest0-21swapper/310:57:473
5813991610,4cyclictest0-21swapper/308:40:243
5813991610,4cyclictest0-21swapper/308:35:263
5813991610,2cyclictest0-21swapper/309:25:233
5813991610,2cyclictest0-21swapper/307:35:223
5813991610,0cyclictest0-21swapper/309:55:283
5813991610,0cyclictest0-21swapper/308:10:253
5813991610,0cyclictest0-21swapper/307:10:133
5812991615,1cyclictest17281-21ssh11:17:462
5812991610,3cyclictest0-21swapper/208:40:232
5812991610,3cyclictest0-21swapper/207:15:012
5812991610,2cyclictest0-21swapper/209:05:232
5812991610,1cyclictest0-21swapper/208:00:232
5811991610,6cyclictest0-21swapper/111:37:021
5811991610,6cyclictest0-21swapper/107:30:531
5811991610,4cyclictest0-21swapper/109:05:271
5811991610,3cyclictest0-21swapper/108:30:181
5811991610,2cyclictest0-21swapper/107:35:191
5811991610,0cyclictest0-21swapper/112:06:511
5811991610,0cyclictest0-21swapper/111:07:261
5811991610,0cyclictest0-21swapper/107:54:261
581099169,3cyclictest121rcu_preempt09:30:270
5810991615,1cyclictest101ktimersoftd/012:40:000
5810991615,1cyclictest101ktimersoftd/011:30:130
5810991615,1cyclictest101ktimersoftd/011:03:330
5810991615,1cyclictest101ktimersoftd/010:46:050
5810991615,1cyclictest101ktimersoftd/010:24:170
5810991615,1cyclictest101ktimersoftd/009:48:410
5810991615,1cyclictest101ktimersoftd/009:39:050
5810991613,1cyclictest101ktimersoftd/010:40:180
5810991613,1cyclictest101ktimersoftd/010:40:170
5810991610,1cyclictest0-21swapper/012:15:170
5810991610,0cyclictest0-21swapper/010:27:120
5817991514,1cyclictest11795-21nscd10:33:067
5817991514,0cyclictest25296-1kworker/u17:110:10:107
5817991510,5cyclictest0-21swapper/712:33:357
5817991510,5cyclictest0-21swapper/710:06:477
5817991510,5cyclictest0-21swapper/709:45:297
5817991510,4cyclictest0-21swapper/712:10:377
5817991510,2cyclictest0-21swapper/707:45:147
5817991510,2cyclictest0-21swapper/707:15:017
5817991510,0cyclictest0-21swapper/711:28:597
5816991513,1cyclictest0-21swapper/607:45:136
5816991510,5cyclictest0-21swapper/611:08:256
5816991510,4cyclictest0-21swapper/609:15:156
5816991510,2cyclictest0-21swapper/609:55:156
5816991510,2cyclictest0-21swapper/608:05:186
5816991510,0cyclictest99-21kswapd012:05:246
5816991510,0cyclictest0-21swapper/610:55:516
5816991510,0cyclictest0-21swapper/610:51:266
5816991510,0cyclictest0-21swapper/610:00:186
5816991510,0cyclictest0-21swapper/609:32:206
5816991510,0cyclictest0-21swapper/607:55:146
5815991515,0cyclictest76350irq/125-eth007:40:015
5815991514,1cyclictest5298-21ssh12:31:315
5815991510,5cyclictest0-21swapper/511:30:425
5815991510,3cyclictest0-21swapper/509:05:275
5815991510,0cyclictest0-21swapper/511:16:445
5815991510,0cyclictest0-21swapper/508:55:015
5814991514,1cyclictest29312-21ls11:45:214
5814991514,1cyclictest1002-21sshd10:50:034
5814991513,1cyclictest18159-21ssh11:18:354
5814991512,0cyclictest31479-21ls10:05:294
5814991510,5cyclictest0-21swapper/408:05:264
5814991510,4cyclictest11976-21kworker/4:011:00:204
5814991510,3cyclictest0-21swapper/409:05:244
5814991510,2cyclictest0-21swapper/408:30:244
5814991510,2cyclictest0-21swapper/408:15:134
5814991510,0cyclictest0-21swapper/411:24:114
5813991512,3cyclictest25197-21nscd11:25:133
5813991512,3cyclictest0-21swapper/308:20:213
5813991510,5cyclictest0-21swapper/311:55:273
5813991510,4cyclictest0-21swapper/308:25:193
5813991510,0cyclictest0-21swapper/312:07:203
5813991510,0cyclictest0-21swapper/311:15:253
5813991510,0cyclictest0-21swapper/309:15:053
5813991510,0cyclictest0-21swapper/309:05:263
5812991515,0cyclictest0-21swapper/207:25:232
5812991510,5cyclictest0-21swapper/210:22:592
5812991510,5cyclictest0-21swapper/210:18:312
5812991510,3cyclictest0-21swapper/209:25:222
5812991510,2cyclictest0-21swapper/207:10:222
5811991513,0cyclictest15567-21ls09:35:221
5811991510,5cyclictest0-21swapper/111:22:431
5811991510,5cyclictest0-21swapper/110:05:331
5811991510,3cyclictest0-21swapper/107:40:251
5811991510,2cyclictest0-21swapper/108:45:001
5811991510,0cyclictest0-21swapper/112:21:351
5811991510,0cyclictest0-21swapper/111:45:161
5811991510,0cyclictest0-21swapper/110:39:301
5811991510,0cyclictest0-21swapper/110:11:171
5811991510,0cyclictest0-21swapper/110:04:031
5811991510,0cyclictest0-21swapper/109:17:011
5810991515,0cyclictest101ktimersoftd/009:00:170
5810991514,1cyclictest101ktimersoftd/012:30:330
5810991514,1cyclictest101ktimersoftd/012:25:120
5810991514,1cyclictest101ktimersoftd/011:13:210
5810991514,1cyclictest101ktimersoftd/009:40:250
5810991514,1cyclictest101ktimersoftd/009:20:170
5810991511,1cyclictest131rcu_sched10:10:210
5810991511,1cyclictest121rcu_preempt12:05:190
5810991510,3cyclictest9-21ksoftirqd/011:55:180
5810991510,1cyclictest131rcu_sched10:00:060
5810991510,0cyclictest0-21swapper/012:01:010
5817991414,0cyclictest0-21swapper/712:20:017
5817991414,0cyclictest0-21swapper/711:21:537
5817991411,3cyclictest5801-21cyclictest11:56:167
5817991410,4cyclictest0-21swapper/711:05:077
5817991410,4cyclictest0-21swapper/710:36:167
5817991410,3cyclictest0-21swapper/712:16:307
5817991410,1cyclictest0-21swapper/712:25:367
5817991410,0cyclictest0-21swapper/712:00:247
5817991410,0cyclictest0-21swapper/711:45:037
5817991410,0cyclictest0-21swapper/711:32:417
5817991410,0cyclictest0-21swapper/709:35:287
5817991410,0cyclictest0-21swapper/709:25:267
5817991410,0cyclictest0-21swapper/709:21:137
5817991410,0cyclictest0-21swapper/708:35:227
5817991410,0cyclictest0-21swapper/708:20:127
5816991410,4cyclictest0-21swapper/610:35:416
5816991410,4cyclictest0-21swapper/610:25:176
5816991410,4cyclictest0-21swapper/609:45:106
5816991410,4cyclictest0-21swapper/608:30:136
5816991410,3cyclictest0-21swapper/611:54:096
5816991410,0cyclictest0-21swapper/612:32:166
5816991410,0cyclictest0-21swapper/612:25:126
5816991410,0cyclictest0-21swapper/612:13:196
5816991410,0cyclictest0-21swapper/612:01:396
5816991410,0cyclictest0-21swapper/611:56:136
5816991410,0cyclictest0-21swapper/611:49:166
5816991410,0cyclictest0-21swapper/611:37:076
5816991410,0cyclictest0-21swapper/611:00:006
5816991410,0cyclictest0-21swapper/610:45:266
5816991410,0cyclictest0-21swapper/610:17:016
5816991410,0cyclictest0-21swapper/610:14:106
5816991410,0cyclictest0-21swapper/610:05:196
5816991410,0cyclictest0-21swapper/609:20:526
5816991410,0cyclictest0-21swapper/607:50:256
5815991414,0cyclictest76350irq/125-eth008:29:075
5815991413,1cyclictest0-21swapper/511:56:105
5815991410,4cyclictest0-21swapper/512:12:185
5815991410,3cyclictest0-21swapper/511:25:025
5815991410,0cyclictest0-21swapper/510:24:115
5814991414,0cyclictest0-21swapper/408:25:184
5814991413,1cyclictest26831-21ssh12:05:034
5814991413,1cyclictest19283-21ssh10:18:344
5814991412,1cyclictest27203-21ssh09:42:454
5814991410,4cyclictest0-21swapper/410:44:214
5814991410,4cyclictest0-21swapper/410:44:204
5814991410,4cyclictest0-21swapper/410:25:234
5814991410,3cyclictest0-21swapper/407:45:134
5814991410,2cyclictest0-21swapper/411:50:254
5814991410,2cyclictest0-21swapper/407:35:004
5814991410,1cyclictest17550irq/127-ahci[0011:57:424
5814991410,0cyclictest0-21swapper/409:45:284
5814991410,0cyclictest0-21swapper/409:32:264
5814991410,0cyclictest0-21swapper/408:55:124
5814991410,0cyclictest0-21swapper/408:00:174
581499140,11cyclictest49-21ksoftirqd/407:30:164
581399148,5cyclictest0-21swapper/312:35:253
5813991410,4cyclictest0-21swapper/312:17:543
5813991410,4cyclictest0-21swapper/311:00:163
5813991410,4cyclictest0-21swapper/309:40:533
5813991410,3cyclictest0-21swapper/307:25:253
5813991410,1cyclictest19693-21cstates12:00:183
5813991410,0cyclictest0-21swapper/312:32:233
5813991410,0cyclictest0-21swapper/312:30:003
5813991410,0cyclictest0-21swapper/312:11:033
5813991410,0cyclictest0-21swapper/311:51:213
5813991410,0cyclictest0-21swapper/311:35:333
5813991410,0cyclictest0-21swapper/311:06:203
5813991410,0cyclictest0-21swapper/310:45:153
5813991410,0cyclictest0-21swapper/310:22:023
5813991410,0cyclictest0-21swapper/309:30:373
5813991410,0cyclictest0-21swapper/308:50:153
5812991414,0cyclictest0-21swapper/208:20:182
5812991410,4cyclictest0-21swapper/209:15:022
5812991410,2cyclictest0-21swapper/208:50:232
5812991410,1cyclictest32100-21/usr/sbin/munin08:55:182
5812991410,0cyclictest0-21swapper/212:30:152
5812991410,0cyclictest0-21swapper/212:05:272
5812991410,0cyclictest0-21swapper/211:46:032
5812991410,0cyclictest0-21swapper/210:35:112
5812991410,0cyclictest0-21swapper/207:50:252
5811991410,4cyclictest0-21swapper/112:37:531
5811991410,4cyclictest0-21swapper/109:00:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional