You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-31 - 04:50

x86 Intel Xeon i3-1578L v5 @2000 MHz, Linux 4.19.246-rt110 (Profile)

Latency plot of system in rack #5, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3.osadl.org (updated Fri Mar 31, 2023 00:49:02)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
14369994645,1cyclictest0-21swapper/320:35:183
1437099450,0cyclictest0-21swapper/419:25:254
14368994342,0cyclictest0-21swapper/219:30:222
14366994343,0cyclictest0-21swapper/021:55:310
14366994310,31cyclictest121rcu_preempt19:10:220
14368994139,1cyclictest15850-21needreboot19:10:182
14368993937,1cyclictest0-21swapper/220:30:302
14373993837,0cyclictest0-21swapper/720:35:187
14370993837,1cyclictest0-21swapper/419:35:014
14370993837,0cyclictest0-21swapper/420:00:214
14367993837,0cyclictest0-21swapper/119:15:301
14373993735,1cyclictest19357-21cut19:15:267
1437399370,37cyclictest0-21swapper/719:10:237
14368993534,1cyclictest0-21swapper/219:25:262
14366993417,16cyclictest0-21swapper/019:25:290
14366993414,18cyclictest121rcu_preempt22:31:540
14366993410,23cyclictest0-21swapper/023:18:370
14373993310,0cyclictest0-21swapper/700:30:227
14372993232,0cyclictest0-21swapper/622:35:136
1437299320,1cyclictest0-21swapper/620:05:026
14371993232,0cyclictest20957-21strings23:35:185
14368993232,0cyclictest0-21swapper/223:35:222
14366993230,1cyclictest0-21swapper/022:35:220
14366993218,14cyclictest0-21swapper/020:30:280
14366993210,19cyclictest0-21swapper/022:51:250
14372993028,1cyclictest31576-21apt-get19:40:126
14367993030,0cyclictest0-21swapper/119:25:291
14367993028,1cyclictest26503-21turbostat23:20:001
14371992929,0cyclictest0-21swapper/500:15:245
14366992910,14cyclictest0-21swapper/019:20:320
14370992827,1cyclictest996-21snmpd20:05:224
14373992625,0cyclictest0-21swapper/723:35:227
14368992626,0cyclictest0-21swapper/222:30:322
14368992624,1cyclictest1004-21lldpd20:00:212
1436899260,26cyclictest0-21swapper/223:17:462
14366992625,1cyclictest0-21swapper/000:15:270
14371992524,1cyclictest0-21swapper/520:00:125
14369992525,0cyclictest0-21swapper/319:30:213
14366992511,13cyclictest121rcu_preempt21:42:390
14366992510,13cyclictest121rcu_preempt19:30:170
14366992510,12cyclictest121rcu_preempt23:35:190
14370992424,0cyclictest0-21swapper/422:15:214
14369992424,0cyclictest0-21swapper/319:55:223
14368992424,0cyclictest0-21swapper/219:35:022
14367992322,1cyclictest0-21swapper/119:30:171
14366992323,0cyclictest0-21swapper/000:30:270
14366992310,11cyclictest121rcu_preempt23:43:340
14371992221,1cyclictest1004-21lldpd20:45:145
14369992222,0cyclictest76350irq/125-eth021:41:243
14369992222,0cyclictest0-21swapper/320:40:233
14369992221,1cyclictest76350irq/125-eth021:56:013
14366992210,10cyclictest121rcu_preempt23:52:140
14366992210,10cyclictest121rcu_preempt23:08:220
14366992210,10cyclictest121rcu_preempt22:45:300
14366992210,10cyclictest121rcu_preempt21:17:590
14366992210,10cyclictest121rcu_preempt21:13:300
14366992210,10cyclictest121rcu_preempt20:11:190
14366992210,10cyclictest121rcu_preempt00:11:170
14366992210,10cyclictest0-21swapper/020:00:120
14372992120,1cyclictest0-21swapper/622:15:186
14372992119,2cyclictest703-21nscd21:13:386
14371992115,1cyclictest996-21snmpd19:35:005
14369992121,0cyclictest0-21swapper/319:10:183
14367992111,9cyclictest17550irq/127-ahci[0021:55:421
1436799210,1cyclictest0-21swapper/122:34:521
1436699212,18cyclictest721-21systemd-logind19:15:010
1436699212,15cyclictest12450-21cat20:05:220
14366992111,8cyclictest121rcu_preempt22:00:230
14366992110,9cyclictest121rcu_preempt23:57:350
14366992110,9cyclictest121rcu_preempt22:14:190
14366992110,9cyclictest121rcu_preempt21:05:550
14366992110,9cyclictest121rcu_preempt20:38:070
14366992110,9cyclictest121rcu_preempt20:29:310
14366992110,9cyclictest121rcu_preempt19:44:030
14366992110,9cyclictest0-21swapper/020:00:010
14366992110,8cyclictest121rcu_preempt20:48:320
14366992110,5cyclictest121rcu_preempt23:30:140
14373992010,7cyclictest0-21swapper/721:44:017
14370992020,0cyclictest0-21swapper/423:19:174
14370992010,6cyclictest0-21swapper/422:31:134
14369992019,1cyclictest0-21swapper/320:30:303
14369992010,5cyclictest0-21swapper/322:33:243
14367992019,1cyclictest1680-21ntpd21:41:051
14367992019,1cyclictest0-21swapper/120:05:131
14366992012,6cyclictest996-21snmpd22:43:140
14366992012,6cyclictest1004-21lldpd21:47:050
14366992011,6cyclictest121rcu_preempt22:22:030
14366992011,6cyclictest121rcu_preempt22:07:590
14366992011,6cyclictest121rcu_preempt20:51:420
14366992011,6cyclictest121rcu_preempt00:01:160
14366992010,8cyclictest121rcu_preempt00:07:380
14366992010,8cyclictest121rcu_preempt00:07:380
14366992010,8cyclictest0-21swapper/022:26:460
14366992010,8cyclictest0-21swapper/019:37:270
14366992010,7cyclictest121rcu_preempt22:15:540
14373991910,4cyclictest0-21swapper/719:25:297
14373991910,0cyclictest0-21swapper/720:41:157
1437399190,19cyclictest0-21swapper/720:02:237
14372991918,1cyclictest0-21swapper/619:35:366
14372991917,1cyclictest28807-21date20:40:006
14371991913,3cyclictest561ktimersoftd/520:10:015
14369991919,0cyclictest76350irq/125-eth019:39:573
14369991917,1cyclictest76350irq/125-eth022:52:203
14368991910,3cyclictest0-21swapper/222:51:382
14367991910,2cyclictest0-21swapper/122:52:011
14366991911,6cyclictest121rcu_preempt23:25:590
14366991911,6cyclictest121rcu_preempt23:23:320
14366991911,6cyclictest121rcu_preempt23:10:130
14366991911,6cyclictest121rcu_preempt23:00:180
14366991911,6cyclictest121rcu_preempt21:35:040
14366991911,6cyclictest121rcu_preempt21:30:320
14366991911,6cyclictest121rcu_preempt21:00:270
14366991911,6cyclictest121rcu_preempt20:55:120
14366991911,6cyclictest121rcu_preempt20:15:220
14366991911,6cyclictest121rcu_preempt00:35:300
14366991911,6cyclictest121rcu_preempt00:20:240
14366991911,5cyclictest121rcu_preempt21:50:090
14366991910,8cyclictest0-21swapper/020:40:230
14366991910,7cyclictest996-21snmpd21:25:430
14366991910,7cyclictest32465-21ssh21:23:000
14366991910,7cyclictest20611-21perf20:25:000
14366991910,7cyclictest121rcu_preempt23:45:000
14366991910,7cyclictest1004-21lldpd19:50:020
14366991910,7cyclictest1004-21lldpd19:45:070
14366991910,6cyclictest121rcu_preempt22:55:000
14366991910,6cyclictest0-21swapper/000:25:140
14373991816,1cyclictest0-21swapper/719:30:227
14372991815,2cyclictest21590-21kworker/u16:219:15:326
14372991810,0cyclictest0-21swapper/621:58:416
14370991810,2cyclictest0-21swapper/422:51:494
14368991817,1cyclictest996-21snmpd00:16:512
14367991810,3cyclictest0-21swapper/120:30:251
14367991810,3cyclictest0-21swapper/119:10:231
14373991715,1cyclictest18506-21cat22:55:147
14373991710,7cyclictest0-21swapper/722:34:007
14373991710,4cyclictest0-21swapper/723:17:577
14373991710,2cyclictest0-21swapper/719:55:207
14372991710,3cyclictest0-21swapper/622:31:146
14372991710,2cyclictest0-21swapper/622:52:036
14371991716,1cyclictest996-21snmpd21:43:275
14371991713,4cyclictest27844-1kworker/u17:021:21:425
14371991710,3cyclictest0-21swapper/519:55:215
14371991710,2cyclictest0-21swapper/522:54:335
14371991710,2cyclictest0-21swapper/519:10:185
14370991717,0cyclictest0-21swapper/419:20:294
14370991716,1cyclictest1004-21lldpd20:35:224
14370991710,4cyclictest0-21swapper/421:45:134
14369991717,0cyclictest76350irq/125-eth019:16:033
14369991716,1cyclictest725-21ptp4l22:17:123
14369991716,1cyclictest0-21swapper/319:25:253
14369991710,5cyclictest0-21swapper/300:30:243
14369991710,0cyclictest0-21swapper/323:10:163
14368991717,0cyclictest0-21swapper/220:05:012
14368991710,4cyclictest0-21swapper/221:56:212
14368991710,4cyclictest0-21swapper/200:30:252
14368991710,3cyclictest0-21swapper/221:44:592
14368991710,3cyclictest0-21swapper/219:55:182
14367991717,0cyclictest0-21swapper/123:07:291
14367991710,4cyclictest0-21swapper/122:55:011
14367991710,3cyclictest0-21swapper/120:35:231
14367991710,3cyclictest0-21swapper/119:55:181
14373991615,1cyclictest25332-21ssh00:36:427
14373991610,6cyclictest32283-21ssh23:03:307
14373991610,3cyclictest0-21swapper/722:51:047
14372991616,0cyclictest0-21swapper/619:10:226
14372991614,0cyclictest18882-1kworker/u17:223:19:586
14372991610,5cyclictest0-21swapper/600:20:226
14372991610,4cyclictest0-21swapper/621:41:556
14372991610,3cyclictest0-21swapper/600:29:346
14372991610,2cyclictest0-21swapper/623:10:226
14371991610,3cyclictest0-21swapper/523:17:195
14370991613,2cyclictest12273-21diskmemload21:40:294
14370991610,3cyclictest0-21swapper/421:56:524
14369991616,0cyclictest76350irq/125-eth023:15:513
14369991614,1cyclictest76350irq/125-eth020:00:213
14369991610,5cyclictest0-21swapper/300:07:463
14369991610,5cyclictest0-21swapper/300:07:463
14369991610,3cyclictest0-21swapper/300:15:243
14368991615,1cyclictest996-21snmpd21:50:342
14368991615,1cyclictest321ktimersoftd/220:40:222
14368991610,6cyclictest0-21swapper/222:20:202
14368991610,3cyclictest0-21swapper/223:10:212
14367991610,6cyclictest0-21swapper/120:20:211
14367991610,2cyclictest0-21swapper/100:30:271
14367991610,1cyclictest0-21swapper/122:00:231
14367991610,0cyclictest0-21swapper/121:47:471
14373991513,0cyclictest12273-21diskmemload23:24:277
14373991510,4cyclictest0-21swapper/721:56:497
14373991510,3cyclictest0-21swapper/721:45:037
14372991514,1cyclictest641ktimersoftd/622:00:266
14372991510,4cyclictest0-21swapper/620:40:216
14372991510,2cyclictest0-21swapper/619:30:186
14372991510,0cyclictest0-21swapper/623:21:266
14372991510,0cyclictest0-21swapper/621:30:276
14372991510,0cyclictest0-21swapper/620:00:186
14371991515,0cyclictest22766-21ls19:20:335
14371991513,0cyclictest996-21snmpd19:15:315
14371991513,0cyclictest11983-21ls21:05:235
14371991510,5cyclictest0-21swapper/522:09:395
14371991510,4cyclictest0-21swapper/521:57:335
14371991510,3cyclictest0-21swapper/500:30:245
14371991510,2cyclictest0-21swapper/522:30:385
14371991510,0cyclictest0-21swapper/523:20:165
14371991510,0cyclictest0-21swapper/522:23:315
14371991510,0cyclictest0-21swapper/522:01:415
14371991510,0cyclictest0-21swapper/500:03:245
14370991510,3cyclictest0-21swapper/422:55:134
14370991510,0cyclictest0-21swapper/423:12:224
14370991510,0cyclictest0-21swapper/419:15:024
14370991510,0cyclictest0-21swapper/400:29:154
14369991515,0cyclictest0-21swapper/323:35:223
14369991513,1cyclictest76350irq/125-eth022:35:113
14369991511,1cyclictest76350irq/125-eth000:24:483
14369991510,5cyclictest0-21swapper/323:00:313
14369991510,0cyclictest0-21swapper/323:46:313
14369991510,0cyclictest0-21swapper/322:41:063
14368991515,0cyclictest0-21swapper/221:19:022
14368991514,1cyclictest803-21gmain22:43:052
14368991512,1cyclictest12710-21kworker/u16:121:20:242
14368991510,5cyclictest0-21swapper/221:39:272
14368991510,5cyclictest0-21swapper/200:20:252
14368991510,4cyclictest0-21swapper/222:00:262
14368991510,3cyclictest0-21swapper/223:25:192
14368991510,3cyclictest0-21swapper/222:55:112
14368991510,3cyclictest0-21swapper/222:35:202
14368991510,3cyclictest0-21swapper/221:45:092
14368991510,3cyclictest0-21swapper/220:35:222
14368991510,3cyclictest0-21swapper/200:03:332
14368991510,2cyclictest0-21swapper/222:15:202
14368991510,2cyclictest0-21swapper/219:15:282
14367991513,1cyclictest1680-21ntpd21:07:491
14367991510,5cyclictest0-21swapper/123:47:031
14367991510,2cyclictest0-21swapper/120:10:131
14367991510,2cyclictest0-21swapper/119:35:011
14367991510,0cyclictest0-21swapper/123:58:161
14367991510,0cyclictest0-21swapper/123:39:121
14367991510,0cyclictest0-21swapper/122:47:581
14367991510,0cyclictest0-21swapper/121:53:511
14367991510,0cyclictest0-21swapper/100:25:161
14367991510,0cyclictest0-21swapper/100:06:091
14367991510,0cyclictest0-21swapper/100:06:091
14373991413,1cyclictest9814-21ssh00:27:417
14373991413,1cyclictest23011-21ssh21:16:557
14373991413,1cyclictest1826-21ssh00:03:277
14373991410,4cyclictest0-21swapper/723:50:197
14373991410,4cyclictest0-21swapper/723:27:047
14373991410,4cyclictest0-21swapper/722:15:457
14373991410,4cyclictest0-21swapper/721:52:287
14373991410,4cyclictest0-21swapper/719:20:347
14373991410,3cyclictest0-21swapper/722:35:207
14373991410,2cyclictest0-21swapper/700:15:267
14373991410,0cyclictest996-21snmpd00:22:087
14373991410,0cyclictest0-21swapper/723:49:257
14373991410,0cyclictest0-21swapper/723:43:457
14373991410,0cyclictest0-21swapper/722:14:567
14373991410,0cyclictest0-21swapper/722:06:197
14373991410,0cyclictest0-21swapper/722:01:037
14373991410,0cyclictest0-21swapper/720:34:097
14373991410,0cyclictest0-21swapper/700:07:407
14373991410,0cyclictest0-21swapper/700:07:407
14372991410,4cyclictest0-21swapper/623:54:476
14372991410,4cyclictest0-21swapper/622:25:436
14372991410,4cyclictest0-21swapper/621:45:016
14372991410,4cyclictest0-21swapper/621:21:336
14372991410,4cyclictest0-21swapper/600:30:026
14372991410,1cyclictest12273-21diskmemload22:13:216
14372991410,0cyclictest996-21snmpd23:07:176
14372991410,0cyclictest0-21swapper/623:38:206
14372991410,0cyclictest0-21swapper/623:30:216
14372991410,0cyclictest0-21swapper/623:25:456
14372991410,0cyclictest0-21swapper/622:55:556
14372991410,0cyclictest0-21swapper/622:40:236
14372991410,0cyclictest0-21swapper/621:51:086
14372991410,0cyclictest0-21swapper/621:25:076
14372991410,0cyclictest0-21swapper/621:19:356
14372991410,0cyclictest0-21swapper/621:00:276
14372991410,0cyclictest0-21swapper/620:30:266
14372991410,0cyclictest0-21swapper/619:58:136
14372991410,0cyclictest0-21swapper/619:20:346
14372991410,0cyclictest0-21swapper/600:35:126
14372991410,0cyclictest0-21swapper/600:15:396
14372991410,0cyclictest0-21swapper/600:12:076
14371991412,1cyclictest996-21snmpd21:54:365
14371991411,1cyclictest725-21ptp4l23:49:265
14371991410,4cyclictest0-21swapper/523:00:245
14371991410,4cyclictest0-21swapper/522:44:425
14371991410,3cyclictest0-21swapper/522:55:055
14371991410,1cyclictest996-21snmpd23:53:165
14371991410,1cyclictest0-21swapper/521:45:125
14371991410,0cyclictest0-21swapper/500:36:065
14371991410,0cyclictest0-21swapper/500:09:165
14371991410,0cyclictest0-21swapper/500:09:165
14370991414,0cyclictest996-21snmpd19:12:504
14370991412,0cyclictest23202-21kworker/u16:323:57:484
14370991410,4cyclictest0-21swapper/423:35:244
14370991410,4cyclictest0-21swapper/421:32:524
14370991410,4cyclictest0-21swapper/420:43:164
14370991410,4cyclictest0-21swapper/400:36:264
14370991410,0cyclictest0-21swapper/422:26:504
14370991410,0cyclictest0-21swapper/400:30:404
14369991414,0cyclictest0-21swapper/321:24:593
14369991411,1cyclictest703-21nscd21:39:453
14369991410,4cyclictest0-21swapper/323:28:143
14369991410,4cyclictest0-21swapper/322:55:063
14369991410,4cyclictest0-21swapper/321:31:213
14369991410,4cyclictest0-21swapper/321:18:233
14369991410,4cyclictest0-21swapper/320:50:173
14369991410,4cyclictest0-21swapper/319:40:123
14369991410,2cyclictest0-21swapper/320:05:183
14369991410,1cyclictest0-21swapper/300:28:573
14369991410,0cyclictest0-21swapper/323:42:493
14369991410,0cyclictest0-21swapper/322:47:023
14369991410,0cyclictest0-21swapper/321:45:123
14368991414,0cyclictest22992-21ls00:35:282
14368991414,0cyclictest0-21swapper/221:11:252
14368991410,4cyclictest0-21swapper/223:45:192
14368991410,4cyclictest0-21swapper/223:44:592
14368991410,4cyclictest0-21swapper/222:27:362
14368991410,4cyclictest0-21swapper/221:30:002
14368991410,4cyclictest0-21swapper/220:55:212
14368991410,2cyclictest0-21swapper/200:05:232
14368991410,2cyclictest0-21swapper/200:05:232
14368991410,1cyclictest0-21swapper/219:20:312
14367991410,4cyclictest0-21swapper/122:20:201
14367991410,4cyclictest0-21swapper/122:15:161
14367991410,4cyclictest0-21swapper/122:08:451
14367991410,4cyclictest0-21swapper/121:11:431
14367991410,4cyclictest0-21swapper/120:55:191
14367991410,4cyclictest0-21swapper/100:12:561
14367991410,2cyclictest0-21swapper/120:00:221
14367991410,2cyclictest0-21swapper/119:20:311
14367991410,1cyclictest0-21swapper/123:50:281
14367991410,0cyclictest0-21swapper/123:42:061
14367991410,0cyclictest0-21swapper/123:34:261
14367991410,0cyclictest0-21swapper/123:20:481
14367991410,0cyclictest0-21swapper/123:10:231
14367991410,0cyclictest0-21swapper/122:28:181
14367991410,0cyclictest0-21swapper/121:21:211
14367991410,0cyclictest0-21swapper/121:19:501
14367991410,0cyclictest0-21swapper/100:35:231
14367991410,0cyclictest0-21swapper/100:17:061
14373991311,1cyclictest996-21snmpd23:57:137
14373991311,1cyclictest996-21snmpd22:40:157
14373991311,1cyclictest996-21snmpd22:20:247
14373991311,1cyclictest996-21snmpd21:31:287
14373991311,1cyclictest996-21snmpd21:25:227
14373991311,1cyclictest996-21snmpd20:53:267
14373991311,1cyclictest996-21snmpd20:45:087
14373991310,3cyclictest0-21swapper/723:32:327
14373991310,3cyclictest0-21swapper/719:35:187
14373991310,3cyclictest0-21swapper/700:11:047
14373991310,0cyclictest1004-21lldpd23:10:187
14373991310,0cyclictest0-21swapper/723:07:087
14373991310,0cyclictest0-21swapper/722:27:077
14373991310,0cyclictest0-21swapper/721:35:277
14373991310,0cyclictest0-21swapper/721:11:397
14373991310,0cyclictest0-21swapper/720:20:167
14373991310,0cyclictest0-21swapper/720:05:187
14372991313,0cyclictest31531-21ls22:20:266
14372991312,1cyclictest703-21nscd21:35:036
14372991312,1cyclictest24773-21kworker/u16:000:00:586
14372991312,1cyclictest16789-21if_enp3s020:15:166
14372991312,1cyclictest12710-21kworker/u16:122:06:126
14372991311,1cyclictest996-21snmpd20:50:186
14372991310,3cyclictest0-21swapper/623:47:056
14372991310,3cyclictest0-21swapper/623:42:536
14372991310,3cyclictest0-21swapper/623:00:316
14372991310,3cyclictest0-21swapper/619:25:246
14372991310,3cyclictest0-21swapper/600:05:186
14372991310,3cyclictest0-21swapper/600:05:186
14372991310,2cyclictest0-21swapper/623:59:426
14372991310,0cyclictest0-21swapper/621:05:196
14372991310,0cyclictest0-21swapper/620:20:146
14371991312,1cyclictest996-21snmpd21:14:075
14371991312,1cyclictest996-21snmpd00:10:505
14371991312,1cyclictest30567-21ps19:35:215
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional