You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-24 - 04:27

x86 Intel Atom E3845 @1910 MHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rackdslot2.osadl.org (updated Fri Mar 24, 2023 00:46:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
554981500,54rtkit-daemon0-21swapper/219:05:322
1339911200,4ptp4l401ktimersoftd/319:08:333
3283211482,21sleep10-21swapper/119:08:351
3082210887,10sleep00-21swapper/019:06:030
119102700,2sleep20-21swapper/219:25:262
368099621,27cyclictest24592-21ls23:20:212
367899614,3cyclictest121rcu_preempt20:30:130
33922613,44sleep20-21swapper/219:10:012
155682580,2sleep20-21swapper/200:10:272
133991580,1ptp4l391rcuc/319:20:553
3678995548,4cyclictest9-21ksoftirqd/000:10:050
368099541,24cyclictest19055-21apt-get23:10:192
367899546,7cyclictest9-21ksoftirqd/020:10:010
3678995444,5cyclictest9-21ksoftirqd/022:55:180
368099530,28cyclictest30746-21awk21:15:272
367899538,8cyclictest9-21ksoftirqd/000:20:200
3678995344,5cyclictest9-21ksoftirqd/020:10:120
3678995243,4cyclictest9-21ksoftirqd/022:20:140
367899516,7cyclictest9-21ksoftirqd/000:00:190
367899516,5cyclictest9-21ksoftirqd/019:20:300
223402510,4sleep322345-21fschecks_count00:25:223
3680995010,21cyclictest17810-21cat19:40:132
3679994942,4cyclictest25-21ksoftirqd/121:45:011
3679994942,4cyclictest25-21ksoftirqd/121:45:001
367899494,10cyclictest9-21ksoftirqd/023:00:340
3678994931,7cyclictest9-21ksoftirqd/019:30:300
368099480,2cyclictest0-21swapper/223:00:362
367899485,8cyclictest9-21ksoftirqd/021:45:340
367899483,9cyclictest9-21ksoftirqd/020:35:250
368099471,27cyclictest2488-21perl21:25:202
368099470,2cyclictest0-21swapper/221:00:282
368099470,27cyclictest5569-21systemctl21:30:282
368099470,27cyclictest14740-21ls21:50:302
368099470,26cyclictest29120-21cat22:20:312
367899472,8cyclictest9-21ksoftirqd/022:40:360
368099460,23cyclictest0-21swapper/222:25:022
3679994631,4cyclictest25-21ksoftirqd/100:10:001
3679994627,7cyclictest25-21ksoftirqd/122:35:281
367899463,5cyclictest9-21ksoftirqd/021:50:380
367899462,10cyclictest9-21ksoftirqd/021:55:130
367899457,9cyclictest9-21ksoftirqd/021:30:160
367899453,8cyclictest121rcu_preempt23:40:270
3678994532,4cyclictest9-21ksoftirqd/020:20:000
3678994527,7cyclictest9-21ksoftirqd/000:05:190
367899450,3cyclictest121rcu_preempt22:20:010
133991450,1ptp4l401ktimersoftd/321:55:163
133991450,1ptp4l401ktimersoftd/321:16:413
133991450,1ptp4l401ktimersoftd/319:41:183
133991450,1ptp4l401ktimersoftd/300:34:533
3679994434,6cyclictest25-21ksoftirqd/121:35:121
367899447,8cyclictest9-21ksoftirqd/022:10:220
367899446,9cyclictest121rcu_preempt20:40:210
367899441,5cyclictest9-21ksoftirqd/021:40:310
367899441,5cyclictest9-21ksoftirqd/021:40:300
367899441,5cyclictest9-21ksoftirqd/019:10:170
3678994410,5cyclictest9-21ksoftirqd/022:05:010
3680994328,12cyclictest33-21ksoftirqd/219:33:532
3679994338,3cyclictest25-21ksoftirqd/123:49:591
3679994337,4cyclictest25-21ksoftirqd/100:30:371
3679994336,4cyclictest25-21ksoftirqd/120:05:001
367899436,4cyclictest9-21ksoftirqd/022:50:200
367899436,4cyclictest121rcu_preempt19:15:160
3678994337,3cyclictest9-21ksoftirqd/021:20:140
3678994335,5cyclictest9-21ksoftirqd/023:55:250
367899432,11cyclictest9-21ksoftirqd/019:40:010
367899431,9cyclictest9-21ksoftirqd/021:15:160
367899431,8cyclictest9-21ksoftirqd/020:50:380
133991430,1ptp4l401ktimersoftd/320:10:013
3679994236,4cyclictest25-21ksoftirqd/120:25:211
3679994236,4cyclictest25-21ksoftirqd/120:25:201
3679994236,4cyclictest25-21ksoftirqd/119:20:261
3679994235,4cyclictest25-21ksoftirqd/123:20:001
3679994235,4cyclictest25-21ksoftirqd/120:30:311
367999421,5cyclictest0-21swapper/100:25:161
367899428,5cyclictest9-21ksoftirqd/023:25:010
367899424,4cyclictest9-21ksoftirqd/019:40:190
3678994235,4cyclictest9-21ksoftirqd/000:35:220
3678994234,4cyclictest9-21ksoftirqd/022:30:340
3678994227,6cyclictest9-21ksoftirqd/023:30:260
367899421,5cyclictest9-21ksoftirqd/021:10:230
133991420,1ptp4l401ktimersoftd/300:35:213
368199411,37cyclictest30325-21cron23:35:003
368099415,18cyclictest55498rtkit-daemon20:30:302
3680994125,13cyclictest33-21ksoftirqd/220:02:472
3679994135,3cyclictest25-21ksoftirqd/100:05:001
3679994134,4cyclictest25-21ksoftirqd/120:40:151
367899417,4cyclictest9-21ksoftirqd/020:00:160
3678994135,3cyclictest9-21ksoftirqd/021:00:280
3678994134,4cyclictest9-21ksoftirqd/023:10:230
367899412,5cyclictest9-21ksoftirqd/023:25:250
367899412,5cyclictest9-21ksoftirqd/021:25:180
3678994119,8cyclictest9-21ksoftirqd/023:15:350
367899410,3cyclictest121rcu_preempt22:35:270
164532410,9sleep1367999cyclictest00:10:381
368099401,23cyclictest0-21swapper/220:50:372
368099401,23cyclictest0-21swapper/219:20:212
368099401,22cyclictest0-21swapper/221:10:302
3679994034,4cyclictest25-21ksoftirqd/123:05:001
3679994034,4cyclictest25-21ksoftirqd/122:15:331
3679994034,4cyclictest25-21ksoftirqd/121:20:311
3679994033,4cyclictest25-21ksoftirqd/122:45:321
3679994033,4cyclictest25-21ksoftirqd/120:10:001
3679994033,4cyclictest25-21ksoftirqd/120:00:001
3679994033,4cyclictest25-21ksoftirqd/119:30:001
3679994028,4cyclictest25-21ksoftirqd/119:35:311
367899406,9cyclictest9-21ksoftirqd/020:45:250
3678994035,3cyclictest9-21ksoftirqd/021:40:000
3678994035,3cyclictest9-21ksoftirqd/019:55:350
3678994034,4cyclictest9-21ksoftirqd/022:05:290
3678994034,3cyclictest9-21ksoftirqd/023:10:010
3678994033,5cyclictest9-21ksoftirqd/020:20:220
367899402,4cyclictest121rcu_preempt20:55:370
367899401,5cyclictest9-21ksoftirqd/022:45:160
367899400,9cyclictest121rcu_preempt19:45:180
368099392,30cyclictest4895-21apt-get22:40:202
368099390,24cyclictest55498rtkit-daemon22:00:132
3679993933,4cyclictest25-21ksoftirqd/123:40:331
3679993933,4cyclictest25-21ksoftirqd/119:50:201
3679993933,4cyclictest25-21ksoftirqd/100:00:001
3679993933,3cyclictest25-21ksoftirqd/122:00:011
3679993932,4cyclictest25-21ksoftirqd/123:30:191
3679993932,4cyclictest25-21ksoftirqd/100:15:351
3679993932,4cyclictest25-21ksoftirqd/100:15:351
3678993934,3cyclictest9-21ksoftirqd/000:20:010
3678993934,3cyclictest9-21ksoftirqd/000:20:010
3678993933,3cyclictest9-21ksoftirqd/022:25:010
3678993932,4cyclictest9-21ksoftirqd/023:55:000
3678993932,4cyclictest9-21ksoftirqd/000:25:190
367899390,5cyclictest9-21ksoftirqd/023:45:220
133991390,1ptp4l401ktimersoftd/320:15:193
368199381,34cyclictest4725-21grep19:10:243
368099383,27cyclictest7609-21apt-get23:55:162
368099383,26cyclictest21365-21apt-get23:15:182
368099382,32cyclictest1396-21grep23:40:272
368099380,23cyclictest0-21swapper/219:50:192
367999384,8cyclictest121rcu_preempt21:15:161
367999383,4cyclictest121rcu_preempt21:30:211
3679993833,3cyclictest25-21ksoftirqd/121:00:001
3679993832,4cyclictest25-21ksoftirqd/122:25:001
3679993832,4cyclictest25-21ksoftirqd/122:10:251
3679993832,4cyclictest25-21ksoftirqd/122:00:221
3679993832,4cyclictest25-21ksoftirqd/121:15:001
3679993832,4cyclictest25-21ksoftirqd/120:50:321
3679993832,4cyclictest25-21ksoftirqd/119:20:011
3679993832,4cyclictest25-21ksoftirqd/100:35:221
3679993832,3cyclictest25-21ksoftirqd/100:25:011
367999381,5cyclictest0-21swapper/119:35:001
367899385,9cyclictest121rcu_preempt23:35:290
3678993831,4cyclictest9-21ksoftirqd/019:50:230
3678993831,4cyclictest9-21ksoftirqd/019:25:120
368199373,28cyclictest0-21swapper/322:40:203
368199373,26cyclictest18476-21apt-get22:00:133
3681993730,4cyclictest41-21ksoftirqd/300:17:013
3681993730,4cyclictest41-21ksoftirqd/300:17:003
368199372,27cyclictest9939-21apt-get00:00:153
368199370,4cyclictest4138-21awk22:35:383
368099373,28cyclictest25411-21apt-get22:15:142
3679993732,3cyclictest25-21ksoftirqd/121:50:001
3679993732,3cyclictest25-21ksoftirqd/121:05:011
3679993732,3cyclictest25-21ksoftirqd/120:45:311
3679993731,4cyclictest25-21ksoftirqd/123:35:341
3679993731,4cyclictest25-21ksoftirqd/122:45:001
3679993731,4cyclictest25-21ksoftirqd/122:30:001
3679993731,3cyclictest25-21ksoftirqd/122:35:001
3679993731,3cyclictest25-21ksoftirqd/120:20:251
3679993730,4cyclictest25-21ksoftirqd/120:35:191
3678993732,3cyclictest9-21ksoftirqd/021:05:300
3678993731,3cyclictest9-21ksoftirqd/020:25:200
3678993731,3cyclictest9-21ksoftirqd/020:25:200
368199363,28cyclictest26009-21apt-get23:25:183
368199363,28cyclictest0-21swapper/323:15:193
368199363,28cyclictest0-21swapper/322:25:213
368199362,27cyclictest6523-21apt-get20:25:163
368199362,27cyclictest6523-21apt-get20:25:163
368199362,26cyclictest9563-21apt-get22:50:153
368199362,26cyclictest13834-21apt-get21:50:193
3681993614,18cyclictest11115-21bc20:35:163
368199360,4cyclictest24319-21sed22:10:303
368199360,2cyclictest0-21swapper/320:45:133
368099369,23cyclictest13880-21latency_hist23:00:002
368099363,25cyclictest16746-21apt-get23:05:182
368099362,26cyclictest2555-21apt-get22:35:182
3679993631,3cyclictest25-21ksoftirqd/123:30:011
3679993631,3cyclictest25-21ksoftirqd/121:30:001
3679993631,3cyclictest25-21ksoftirqd/120:15:161
3679993631,3cyclictest25-21ksoftirqd/119:10:121
3679993630,4cyclictest25-21ksoftirqd/123:05:011
3679993629,4cyclictest25-21ksoftirqd/122:10:001
3679993620,4cyclictest25-21ksoftirqd/121:50:341
3678993630,4cyclictest9-21ksoftirqd/000:30:240
368199353,28cyclictest0-21swapper/320:20:173
368199353,27cyclictest0-21swapper/323:50:213
368199353,27cyclictest0-21swapper/323:05:193
368199353,27cyclictest0-21swapper/322:15:153
368199353,26cyclictest27380-21apt-get21:10:213
368199353,26cyclictest0-21swapper/320:05:123
368199353,25cyclictest12250-21apt-get00:05:143
3681993529,4cyclictest41-21ksoftirqd/300:11:273
3681993527,4cyclictest22789-21cron22:10:013
368199352,28cyclictest0-21swapper/323:55:173
368199350,4cyclictest13028-21/usr/sbin/munin19:30:303
368099353,27cyclictest29324-21apt-get20:05:122
3680993528,4cyclictest33-21ksoftirqd/221:24:252
3680993528,4cyclictest33-21ksoftirqd/220:24:192
368099352,28cyclictest20408-21apt-get20:55:142
368099352,27cyclictest0-21swapper/200:30:152
368099352,27cyclictest0-21swapper/200:05:142
368099352,25cyclictest5252-21apt-get23:50:212
368099352,25cyclictest28325-21apt-get23:30:172
3680993514,18cyclictest7751-21phc2sys-jitter21:35:262
368099351,31cyclictest1804-21grep22:30:332
368099351,29cyclictest0-21swapper/200:00:152
3679993529,3cyclictest25-21ksoftirqd/121:05:311
3679993529,3cyclictest25-21ksoftirqd/120:10:301
3679993529,3cyclictest25-21ksoftirqd/119:45:011
31122350,6sleep2368099cyclictest20:15:282
3681993428,3cyclictest41-21ksoftirqd/320:53:253
3681993427,4cyclictest41-21ksoftirqd/321:47:153
3681993427,4cyclictest41-21ksoftirqd/319:27:323
3681993413,18cyclictest26527-21wc21:05:313
368199340,4cyclictest13417-21acpi20:40:143
368199340,3cyclictest263972chrt19:56:253
368199340,2cyclictest0-21swapper/319:50:143
3680993428,3cyclictest33-21ksoftirqd/221:06:292
3680993426,5cyclictest33-21ksoftirqd/200:24:112
368099342,26cyclictest9207-21apt-get21:40:152
368099342,26cyclictest9207-21apt-get21:40:152
3680993415,16cyclictest30562-21/usr/sbin/munin23:35:232
368099340,5cyclictest4665-21taskset23:46:422
368099340,5cyclictest1232-21taskset20:12:352
368099340,2cyclictest0-21swapper/221:55:182
3679993429,3cyclictest25-21ksoftirqd/123:10:361
3679993429,3cyclictest25-21ksoftirqd/122:59:591
3679993428,4cyclictest25-21ksoftirqd/119:45:331
3679993428,3cyclictest25-21ksoftirqd/122:55:001
3679993420,4cyclictest25-21ksoftirqd/123:20:181
368199335,25cyclictest1623-21cron21:24:593
368199333,23cyclictest8829-21apt-get20:30:133
3681993326,4cyclictest41-21ksoftirqd/323:11:223
3681993326,4cyclictest41-21ksoftirqd/323:04:193
368199332,27cyclictest0-21swapper/321:40:163
368199332,27cyclictest0-21swapper/321:40:153
368199332,26cyclictest0-21swapper/320:55:133
3681993313,17cyclictest31834-21grep23:35:293
3681993313,17cyclictest28845-21cut22:20:273
368199331,28cyclictest6532-21cat21:35:013
368199330,4cyclictest970-21fschecks_count23:40:223
368199330,2cyclictest0-21swapper/322:30:183
368099333,25cyclictest15784-21apt-get20:45:142
3680993327,3cyclictest33-21ksoftirqd/219:45:552
368099332,26cyclictest0-21swapper/223:25:182
368099331,27cyclictest0-21swapper/222:50:152
367999331,3cyclictest0-21swapper/123:50:431
368199322,23cyclictest15410-21apt-get19:35:153
368199321,27cyclictest6568-21tune2fs19:15:223
368199321,27cyclictest22200-21wc19:50:003
368199321,27cyclictest12047-21tr22:55:203
368099327,21cyclictest15473-21cron20:45:012
3680993225,4cyclictest33-21ksoftirqd/220:35:062
368099322,25cyclictest6523-21apt-get20:25:152
368099322,25cyclictest6523-21apt-get20:25:152
368099322,24cyclictest0-21swapper/219:35:152
368099321,27cyclictest24979-21grep19:55:222
368099321,27cyclictest13543-21ls21:50:012
3680993211,18cyclictest24529-21timerandwakeup22:10:332
3680993210,19cyclictest7816-21seq19:15:562
368099320,28cyclictest21471-21cut22:05:262
133991320,1ptp4l391rcuc/321:38:063
368199315,22cyclictest3333-21fschecks_count23:45:223
368199315,22cyclictest19679-21cpuspeed00:20:163
3681993113,15cyclictest23685-21missed_timers21:00:243
368199311,26cyclictest2468-21cut21:25:203
368199310,3cyclictest26936-21/usr/sbin/munin20:00:183
368199310,26cyclictest9252-21cron22:50:003
368099317,20cyclictest27868-21cut00:35:582
368099311,24cyclictest0-21swapper/200:25:172
368199301,26cyclictest23986-21df23:20:153
368099301,25cyclictest18089-21irqrtprio00:15:252
368099301,25cyclictest18089-21irqrtprio00:15:252
368099300,25cyclictest9253-21cron22:50:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional