You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-20 - 19:38

x86 Intel Atom E3845 @1910 MHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackdslot3.osadl.org (updated Mon Mar 20, 2023 12:46:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2459621620,9sleep2489399cyclictest09:13:192
8255911600,5ptp4l401ktimersoftd/307:07:303
2452621600,7sleep0489199cyclictest09:12:370
8255911150,2ptp4l9837-21ls12:30:333
223029774,10sleep00-21swapper/007:05:010
825591910,2ptp4l20341-21sh09:33:443
825591910,2ptp4l20341-21sh09:33:443
251312870,8sleep2489399cyclictest09:34:582
251312870,8sleep2489399cyclictest09:34:572
4891998612,21cyclictest13654-21spawn09:26:280
4891998612,21cyclictest13654-21spawn09:26:270
325082770,6sleep0489199cyclictest09:32:200
325082770,6sleep0489199cyclictest09:32:200
160182730,5sleep316009-21sh09:07:303
489299701,7cyclictest25-21ksoftirqd/109:26:461
489299701,7cyclictest25-21ksoftirqd/109:26:461
44402700,4sleep14449-21sh09:33:161
44402700,4sleep14449-21sh09:33:151
4892996932,10cyclictest25-21ksoftirqd/109:01:571
489199691,25cyclictest4089-21tee09:01:290
4892996811,10cyclictest26372-21sort09:05:351
825591670,20ptp4l32221-21spawn09:25:573
825591670,20ptp4l32221-21spawn09:25:563
245952670,8sleep1489299cyclictest09:13:181
191532670,5sleep341-21ksoftirqd/308:45:453
489399663,6cyclictest321ktimersoftd/209:05:552
489199660,24cyclictest21443-21taskset11:47:450
825591650,18ptp4l401ktimersoftd/307:15:283
489399650,9cyclictest321ktimersoftd/210:20:002
489299646,8cyclictest25-21ksoftirqd/111:10:001
246072640,3sleep30-21swapper/309:13:283
489299636,8cyclictest25591-21tune2fs09:15:221
489299635,10cyclictest25-21ksoftirqd/110:40:191
489299634,7cyclictest25-21ksoftirqd/111:05:001
489199630,23cyclictest11592-21seq11:27:310
4893996228,7cyclictest33-21ksoftirqd/209:20:292
489299626,9cyclictest25-21ksoftirqd/111:40:011
489299626,9cyclictest25-21ksoftirqd/109:35:341
489299625,9cyclictest25-21ksoftirqd/110:50:221
489299623,8cyclictest25-21ksoftirqd/111:40:041
489299621,8cyclictest25-21ksoftirqd/109:20:041
11012620,4sleep31102-21sh09:01:223
4893996113,6cyclictest33-21ksoftirqd/209:03:322
489299614,10cyclictest25-21ksoftirqd/108:59:051
4892996127,11cyclictest25-21ksoftirqd/110:00:191
489299611,8cyclictest25-21ksoftirqd/110:40:001
489299610,9cyclictest25-21ksoftirqd/108:45:011
489399605,9cyclictest33-21ksoftirqd/209:26:002
489399605,9cyclictest33-21ksoftirqd/209:25:592
489399604,8cyclictest321ktimersoftd/209:35:502
489399603,8cyclictest321ktimersoftd/208:45:262
489399602,8cyclictest321ktimersoftd/210:40:002
489299604,9cyclictest25-21ksoftirqd/110:25:221
489299603,7cyclictest25-21ksoftirqd/111:10:201
489299603,10cyclictest25-21ksoftirqd/111:15:171
4892996027,10cyclictest25-21ksoftirqd/111:45:221
4892996027,10cyclictest25-21ksoftirqd/111:20:271
489199600,23cyclictest29263-21runrttasks09:47:510
489199600,22cyclictest14168-21runrttasks11:31:430
489399596,6cyclictest32344-21latency_hist11:05:002
489399596,5cyclictest16973-21apt-config11:40:002
489399595,8cyclictest33-21ksoftirqd/209:00:012
489399593,6cyclictest33-21ksoftirqd/212:10:362
4893995923,6cyclictest33-21ksoftirqd/209:45:172
489299595,7cyclictest18412-21memory08:45:271
489299595,11cyclictest25-21ksoftirqd/110:05:241
489299594,10cyclictest25-21ksoftirqd/109:45:241
489299591,7cyclictest25-21ksoftirqd/110:15:181
489399585,4cyclictest9874-21apt-get08:30:002
489299585,9cyclictest25-21ksoftirqd/112:30:001
489299583,9cyclictest25-21ksoftirqd/111:29:341
489299580,8cyclictest25-21ksoftirqd/110:45:241
489299580,8cyclictest25-21ksoftirqd/110:45:241
489399576,7cyclictest2770-21grep12:15:322
489399576,7cyclictest2770-21grep12:15:322
489399575,7cyclictest33-21ksoftirqd/210:25:002
4893995726,8cyclictest33-21ksoftirqd/208:30:022
489299574,9cyclictest25-21ksoftirqd/110:35:001
489299572,7cyclictest25-21ksoftirqd/108:53:021
489299570,7cyclictest25-21ksoftirqd/111:35:001
489299570,14cyclictest25-21ksoftirqd/112:05:341
489299570,10cyclictest25-21ksoftirqd/109:55:281
489399566,8cyclictest33-21ksoftirqd/210:30:312
489399566,7cyclictest33-21ksoftirqd/211:45:262
489399565,8cyclictest33-21ksoftirqd/209:55:192
489399565,5cyclictest1144-21missed_timers11:05:262
4893995621,6cyclictest33-21ksoftirqd/212:00:232
489299568,4cyclictest25-21ksoftirqd/112:15:141
489299568,4cyclictest25-21ksoftirqd/112:15:131
489299564,10cyclictest25-21ksoftirqd/110:20:171
489299562,8cyclictest25-21ksoftirqd/112:20:311
489299562,7cyclictest25-21ksoftirqd/112:05:001
489299562,7cyclictest25-21ksoftirqd/109:50:301
489299561,8cyclictest25-21ksoftirqd/112:10:201
489299561,8cyclictest25-21ksoftirqd/107:25:301
489299561,7cyclictest10631-21whetstone-doubl08:25:311
489299560,7cyclictest25-21ksoftirqd/111:55:221
489299560,7cyclictest25-21ksoftirqd/110:10:171
489299560,7cyclictest11585-21sshd12:35:301
825591550,1ptp4l401ktimersoftd/308:59:063
489399557,8cyclictest33-21ksoftirqd/208:10:202
489399557,7cyclictest33-21ksoftirqd/212:20:232
489399556,8cyclictest33-21ksoftirqd/210:50:222
489399556,7cyclictest33-21ksoftirqd/211:00:012
489399556,7cyclictest33-21ksoftirqd/210:25:162
489399556,6cyclictest33-21ksoftirqd/210:40:012
489399556,5cyclictest24306-21strings10:45:322
489399556,5cyclictest24306-21strings10:45:312
489399555,9cyclictest33-21ksoftirqd/211:40:152
489399555,10cyclictest33-21ksoftirqd/210:00:262
489399554,7cyclictest33-21ksoftirqd/211:50:212
489399554,7cyclictest33-21ksoftirqd/210:14:592
489399553,8cyclictest33-21ksoftirqd/211:30:012
489399551,8cyclictest321ktimersoftd/208:05:132
489299554,10cyclictest25-21ksoftirqd/111:50:181
489299553,10cyclictest25-21ksoftirqd/110:55:171
489299551,7cyclictest25-21ksoftirqd/112:30:321
489299550,9cyclictest25-21ksoftirqd/109:45:001
489199551,24cyclictest6866-21grep09:35:240
489199551,24cyclictest18982-21seq11:40:560
489199550,25cyclictest23838-21taskset11:53:430
489399547,7cyclictest33-21ksoftirqd/207:55:272
489399545,8cyclictest33-21ksoftirqd/209:40:252
489399544,8cyclictest33-21ksoftirqd/207:30:172
489399543,8cyclictest33-21ksoftirqd/208:00:232
4893995421,6cyclictest33-21ksoftirqd/212:35:232
4893995416,3cyclictest33-21ksoftirqd/211:20:232
489399540,9cyclictest321ktimersoftd/210:05:172
489399540,10cyclictest321ktimersoftd/208:35:232
489299544,6cyclictest10631-21whetstone-doubl07:20:321
489299543,6cyclictest10631-21whetstone-doubl07:30:291
489299540,7cyclictest25-21ksoftirqd/107:50:131
4894995342,6cyclictest41-21ksoftirqd/311:20:133
489399537,8cyclictest33-21ksoftirqd/208:50:132
489399537,6cyclictest33-21ksoftirqd/212:25:222
489399536,8cyclictest33-21ksoftirqd/209:50:282
489399535,8cyclictest33-21ksoftirqd/208:20:162
489399534,8cyclictest33-21ksoftirqd/209:20:002
489399534,7cyclictest33-21ksoftirqd/212:30:302
489399534,7cyclictest33-21ksoftirqd/211:10:242
489399533,8cyclictest33-21ksoftirqd/211:30:242
489399532,7cyclictest33-21ksoftirqd/208:15:272
489399532,7cyclictest33-21ksoftirqd/208:15:262
489299536,4cyclictest25-21ksoftirqd/107:10:281
489299532,6cyclictest25-21ksoftirqd/108:05:301
489299532,6cyclictest25-21ksoftirqd/107:55:261
4892995312,6cyclictest25-21ksoftirqd/108:24:551
489199536,26cyclictest2029-21taskset11:07:520
489199531,24cyclictest31603-21seq09:53:050
489399527,8cyclictest33-21ksoftirqd/207:15:242
489399526,4cyclictest33-21ksoftirqd/207:10:142
489399524,8cyclictest33-21ksoftirqd/208:44:412
489399524,8cyclictest33-21ksoftirqd/207:50:142
489399524,8cyclictest33-21ksoftirqd/207:45:302
489399524,7cyclictest33-21ksoftirqd/207:25:002
489199520,24cyclictest25932-21seq11:55:450
4894995145,4cyclictest41-21ksoftirqd/312:05:003
4894995140,3cyclictest41-21ksoftirqd/309:35:283
489399516,7cyclictest33-21ksoftirqd/211:15:152
489399514,7cyclictest33-21ksoftirqd/207:40:012
489399513,8cyclictest33-21ksoftirqd/211:55:172
489199510,26cyclictest5573-21seq12:24:560
489199510,26cyclictest27113-21taskset09:44:570
489199510,23cyclictest6609-21seq11:16:150
489399503,8cyclictest33-21ksoftirqd/207:40:212
489399503,7cyclictest33-21ksoftirqd/212:05:212
489199501,25cyclictest29673-21seq10:56:390
489199500,26cyclictest9259-21runrttasks11:22:210
489199500,24cyclictest30776-21seq12:08:050
489199500,23cyclictest610-21seq12:12:150
489399493,7cyclictest33-21ksoftirqd/207:25:122
4891994932,13cyclictest9-21ksoftirqd/012:28:490
489199491,25cyclictest10433-21seq12:34:550
489199490,26cyclictest20179-21seq10:37:010
489199490,24cyclictest25633-21/usr/sbin/munin09:15:230
4894994843,3cyclictest41-21ksoftirqd/310:19:593
4894994843,3cyclictest41-21ksoftirqd/308:55:003
4892994834,4cyclictest25-21ksoftirqd/107:40:111
489199480,23cyclictest20928-21cat10:40:130
489299472,4cyclictest25-21ksoftirqd/108:30:131
489199471,22cyclictest15908-21latency_hist10:30:000
489199470,25cyclictest12528-21runrttasks12:36:590
489199470,23cyclictest1551-21diskstats12:15:190
489199470,23cyclictest1551-21diskstats12:15:180
489199461,22cyclictest3817-21taskset10:01:420
4894994539,4cyclictest41-21ksoftirqd/308:30:003
4894994539,3cyclictest41-21ksoftirqd/312:10:003
4892994531,4cyclictest25-21ksoftirqd/108:01:381
489199451,22cyclictest0-21swapper/008:53:470
489199450,24cyclictest17146-21pwd08:44:530
440024514,18sleep20-21swapper/207:07:292
825591440,1ptp4l401ktimersoftd/307:20:313
489199440,20cyclictest10631-21whetstone-doubl08:24:550
4894994338,3cyclictest41-21ksoftirqd/308:15:013
4894994337,4cyclictest41-21ksoftirqd/311:05:243
4894994337,3cyclictest41-21ksoftirqd/309:20:233
4894994336,4cyclictest41-21ksoftirqd/309:15:513
4894994336,4cyclictest41-21ksoftirqd/307:45:003
4892994338,3cyclictest25-21ksoftirqd/107:45:001
489299430,2cyclictest0-21swapper/107:16:361
489199433,4cyclictest182122execl08:45:390
489199430,23cyclictest1100-21runrttasks12:00:430
434424313,19sleep10-21swapper/107:06:431
825591420,1ptp4l401ktimersoftd/310:32:553
489499424,4cyclictest0-21swapper/312:35:373
4894994236,4cyclictest41-21ksoftirqd/310:40:003
4894994235,4cyclictest41-21ksoftirqd/310:25:153
4894994235,4cyclictest41-21ksoftirqd/308:00:003
4892994237,3cyclictest25-21ksoftirqd/108:15:151
4892994237,3cyclictest25-21ksoftirqd/108:15:141
4892994235,4cyclictest25-21ksoftirqd/107:35:121
489199421,24cyclictest17923-21seq10:32:510
4894994135,4cyclictest41-21ksoftirqd/308:05:283
4894994135,3cyclictest41-21ksoftirqd/308:25:003
4894994135,3cyclictest41-21ksoftirqd/307:25:213
4894994134,4cyclictest41-21ksoftirqd/311:45:283
4894994134,4cyclictest41-21ksoftirqd/310:05:243
4894994134,4cyclictest41-21ksoftirqd/307:30:213
489199411,23cyclictest31975-21seq11:01:100
489199411,23cyclictest2485-21seq11:10:080
489199411,21cyclictest0-21swapper/011:37:190
4894994035,3cyclictest41-21ksoftirqd/308:35:253
4894994035,3cyclictest41-21ksoftirqd/307:54:593
4894994034,3cyclictest41-21ksoftirqd/310:24:593
4894994028,10cyclictest41-21ksoftirqd/308:15:143
4894994028,10cyclictest41-21ksoftirqd/308:15:143
4892994035,3cyclictest25-21ksoftirqd/108:10:291
4891994017,20cyclictest23436-21tee09:05:280
489199401,23cyclictest22427-21runrttasks08:55:100
489199401,20cyclictest0-21swapper/010:19:500
489199400,24cyclictest6327-21seq10:09:020
4894993932,4cyclictest41-21ksoftirqd/308:43:563
4894993932,4cyclictest41-21ksoftirqd/308:00:253
4892993934,3cyclictest25-21ksoftirqd/108:35:221
489199390,26cyclictest10631-21whetstone-doubl07:52:170
489199390,24cyclictest25199-21taskset10:48:540
489199390,24cyclictest25199-21taskset10:48:530
825591380,1ptp4l401ktimersoftd/310:00:183
4894993833,3cyclictest41-21ksoftirqd/311:40:003
4894993833,3cyclictest41-21ksoftirqd/310:00:013
4894993833,3cyclictest41-21ksoftirqd/309:50:413
4894993832,4cyclictest41-21ksoftirqd/310:40:153
4894993831,4cyclictest41-21ksoftirqd/312:20:003
4894993831,4cyclictest41-21ksoftirqd/312:19:593
4894993831,4cyclictest41-21ksoftirqd/307:45:213
4891993831,4cyclictest9-21ksoftirqd/010:11:400
489199381,18cyclictest10631-21whetstone-doubl07:30:310
489199380,20cyclictest1100-21runrttasks10:52:100
4894993732,3cyclictest41-21ksoftirqd/307:35:243
4894993731,4cyclictest41-21ksoftirqd/312:00:003
4894993731,4cyclictest41-21ksoftirqd/308:30:213
4894993731,3cyclictest41-21ksoftirqd/311:11:253
4894993730,4cyclictest41-21ksoftirqd/311:51:233
4894993730,4cyclictest41-21ksoftirqd/311:28:423
4894993730,4cyclictest41-21ksoftirqd/311:01:503
4894993730,4cyclictest41-21ksoftirqd/310:53:473
489199371,17cyclictest0-21swapper/009:20:310
825591360,1ptp4l401ktimersoftd/311:30:193
4894993630,3cyclictest41-21ksoftirqd/307:10:213
4894993629,5cyclictest41-21ksoftirqd/310:14:403
4891993620,13cyclictest9-21ksoftirqd/009:57:190
4894993429,3cyclictest41-21ksoftirqd/311:40:193
4894993429,3cyclictest41-21ksoftirqd/310:45:273
4894993429,3cyclictest41-21ksoftirqd/310:45:263
4894993428,4cyclictest41-21ksoftirqd/309:49:353
4894993428,3cyclictest41-21ksoftirqd/309:40:043
4894993427,4cyclictest41-21ksoftirqd/311:15:013
825591330,1ptp4l401ktimersoftd/312:10:473
4894993327,4cyclictest41-21ksoftirqd/312:25:393
489199334,23cyclictest11396-21apt-get10:20:140
4894993227,3cyclictest41-21ksoftirqd/310:55:203
4894993226,4cyclictest41-21ksoftirqd/312:20:133
489199321,19cyclictest10631-21whetstone-doubl07:16:360
4891993010,17cyclictest10631-21whetstone-doubl07:38:540
489199281,23cyclictest10631-21whetstone-doubl08:40:000
489199271,23cyclictest11088-21cut07:20:280
489199271,22cyclictest10631-21whetstone-doubl08:05:140
489199271,22cyclictest10631-21whetstone-doubl07:25:210
489199270,24cyclictest9440-21grep08:25:310
489199270,24cyclictest1468-21sed08:10:210
489199261,21cyclictest10631-21whetstone-doubl07:50:000
489199261,21cyclictest10631-21whetstone-doubl07:40:110
489199261,21cyclictest10631-21whetstone-doubl07:10:170
489199260,23cyclictest10631-21whetstone-doubl08:30:210
489199260,23cyclictest10631-21whetstone-doubl08:00:370
489199260,22cyclictest10631-21whetstone-doubl07:55:010
489199250,22cyclictest10631-21whetstone-doubl08:15:200
489199250,22cyclictest10631-21whetstone-doubl08:15:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional