You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-13 - 01:58

x86 Intel Atom E3845 @1910 MHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackdslot3.osadl.org (updated Fri Aug 12, 2022 12:46:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2681424030,4sleep30-21swapper/307:05:143
2680421490,7sleep02904699cyclictest11:40:160
285532135113,9sleep20-21swapper/207:07:242
576981240,42rtkit-daemon11993-21apache207:07:091
284472836,19sleep00-21swapper/007:06:000
29048997813,14cyclictest7673-21sed11:00:002
29048997512,15cyclictest25385-21latency_hist09:20:002
29048997511,15cyclictest20822-21bc09:10:122
29048997412,13cyclictest29977-21cut07:10:222
2904799736,3cyclictest121rcu_preempt08:15:121
2904999725,8cyclictest41-21ksoftirqd/307:20:053
2904899729,16cyclictest787-21cut09:35:182
2904899729,16cyclictest787-21cut09:35:182
2904899719,14cyclictest6741-21latency_hist08:40:012
29048997113,13cyclictest27179-21iwlist11:40:202
29049997028,10cyclictest41-21ksoftirqd/309:15:003
2904899709,13cyclictest5801-21cut12:05:002
29048997013,14cyclictest10410-21grep08:45:252
29048997011,14cyclictest2746-21chrt10:47:472
29048997011,13cyclictest3741-21needreboot09:40:262
29048997011,13cyclictest24869-21ls08:10:012
29048997010,14cyclictest17318-21grep09:00:262
2904899698,15cyclictest3417-21grep10:50:152
29048996911,13cyclictest31196-21cut11:50:012
2904899689,13cyclictest13052-21tune2fs11:10:222
29048996812,13cyclictest25867-21sed11:35:332
29048996812,12cyclictest28497-21seq10:35:042
29048996811,13cyclictest30680-21fschecks_time09:30:262
29048996810,13cyclictest4107-21cpuspeed_turbos07:25:142
29048996711,14cyclictest23900-21chrt11:34:432
29048996711,14cyclictest22278-21latency10:20:262
291602660,7sleep32904999cyclictest10:35:203
2904899669,14cyclictest11379-21ps11:05:272
2904899669,14cyclictest11379-21ps11:05:272
2904899669,11cyclictest0-21swapper/207:30:312
2904899668,13cyclictest21225-21grep11:25:312
81712650,4sleep12904799cyclictest08:40:281
2904999657,9cyclictest41-21ksoftirqd/311:20:003
2904999656,10cyclictest41-21ksoftirqd/308:00:263
2904999655,8cyclictest41-21ksoftirqd/309:25:303
2904999655,6cyclictest41-21ksoftirqd/308:59:593
2904899659,15cyclictest13963-21cat07:45:182
2904899659,14cyclictest705-21wc10:45:012
2904899659,14cyclictest18259-21sed10:10:302
2904899658,15cyclictest2523-21cut07:20:272
29048996514,14cyclictest8879-21/usr/sbin/munin09:50:382
29048996512,14cyclictest31528-21chrt08:23:152
29048996510,12cyclictest11088-21sed09:55:312
2904999647,9cyclictest41-21ksoftirqd/310:10:263
2904899648,12cyclictest19044-21mailstats12:30:312
2904999638,9cyclictest41-21ksoftirqd/309:05:103
2904999637,10cyclictest41-21ksoftirqd/310:05:003
2904999636,9cyclictest41-21ksoftirqd/307:59:593
2904999635,8cyclictest41-21ksoftirqd/310:55:253
2904899639,14cyclictest6261-21cpuspeed_turbos12:05:202
2904899638,18cyclictest15139-21mailstats08:55:262
2904899638,13cyclictest15675-21cat12:25:182
2904899638,12cyclictest0-21swapper/212:10:312
29048996310,13cyclictest15115-21cut11:15:152
2904999627,9cyclictest41-21ksoftirqd/311:50:183
2904999627,9cyclictest41-21ksoftirqd/310:30:203
2904999627,9cyclictest41-21ksoftirqd/308:55:013
2904999627,10cyclictest41-21ksoftirqd/312:20:183
2904999626,10cyclictest41-21ksoftirqd/307:35:263
2904999625,8cyclictest41-21ksoftirqd/309:20:173
2904999624,9cyclictest41-21ksoftirqd/307:10:273
2904999624,8cyclictest41-21ksoftirqd/311:55:253
2904999624,7cyclictest41-21ksoftirqd/311:40:013
2904899629,12cyclictest392-21timedrift11:50:292
2904899629,12cyclictest202532chrt09:06:222
2904899628,13cyclictest27831-21basename09:25:142
2904899628,12cyclictest0-21swapper/208:25:242
29048996212,20cyclictest21916-21chrt12:37:372
2904999617,9cyclictest41-21ksoftirqd/308:45:283
2904999617,13cyclictest41-21ksoftirqd/309:40:193
2904999616,9cyclictest41-21ksoftirqd/312:30:323
2904999616,10cyclictest41-21ksoftirqd/310:40:183
2904999615,8cyclictest41-21ksoftirqd/308:40:223
2904999615,8cyclictest41-21ksoftirqd/307:30:173
2904999614,8cyclictest41-21ksoftirqd/310:05:193
2904999614,8cyclictest41-21ksoftirqd/309:30:223
2904999613,9cyclictest41-21ksoftirqd/308:10:133
2904999613,9cyclictest41-21ksoftirqd/308:10:133
2904999613,7cyclictest41-21ksoftirqd/310:15:153
2904899619,13cyclictest17822-21cat07:55:002
29048996112,13cyclictest9723-21chrt11:02:322
29048996112,13cyclictest18136-21latency11:20:242
29048996110,14cyclictest10648-21/usr/sbin/munin12:15:152
29048996110,11cyclictest0-21swapper/207:40:152
2904999606,9cyclictest41-21ksoftirqd/309:35:233
2904999606,9cyclictest41-21ksoftirqd/309:35:223
2904999606,9cyclictest41-21ksoftirqd/307:15:213
2904999606,10cyclictest41-21ksoftirqd/310:50:283
2904999604,9cyclictest41-21ksoftirqd/308:20:193
2904999604,8cyclictest41-21ksoftirqd/308:35:183
2904999604,7cyclictest41-21ksoftirqd/311:45:243
2904999602,7cyclictest41-21ksoftirqd/310:29:593
2904899609,13cyclictest5683-21if_enp4s009:45:212
2904899608,13cyclictest462-21timerandwakeup07:15:292
2904999597,9cyclictest41-21ksoftirqd/312:35:133
2904999597,9cyclictest41-21ksoftirqd/307:50:133
2904999597,10cyclictest41-21ksoftirqd/312:20:003
2904999596,11cyclictest41-21ksoftirqd/308:30:273
2904999595,9cyclictest41-21ksoftirqd/311:10:313
2904999594,8cyclictest41-21ksoftirqd/307:40:263
2904999593,7cyclictest41-21ksoftirqd/308:25:143
29048995912,14cyclictest19285-21sendmail_mailtr07:55:272
2904699591,3cyclictest0-21swapper/010:15:130
2904999586,9cyclictest41-21ksoftirqd/307:25:143
2904999585,7cyclictest41-21ksoftirqd/311:00:213
2904999584,7cyclictest41-21ksoftirqd/311:05:203
2904999584,7cyclictest41-21ksoftirqd/311:05:193
2904999582,8cyclictest41-21ksoftirqd/310:45:203
2904899588,13cyclictest3429-21latency_hist12:00:002
2904899588,12cyclictest16319-21chrt10:08:292
2904899587,14cyclictest12648-21smartctl10:00:232
2904899587,12cyclictest0-21swapper/208:31:072
29048995810,12cyclictest27049-21cat10:30:272
2904799589,4cyclictest25-21ksoftirqd/110:30:291
226492580,2sleep10-21swapper/108:05:121
2904999576,9cyclictest41-21ksoftirqd/311:25:243
2904999575,9cyclictest41-21ksoftirqd/311:20:203
2904999575,7cyclictest41-21ksoftirqd/308:05:183
2904999573,8cyclictest41-21ksoftirqd/309:50:323
2904999572,8cyclictest41-21ksoftirqd/311:30:363
2904899578,11cyclictest0-21swapper/212:24:342
2904899577,13cyclictest25674-21sshd10:29:082
2904899577,10cyclictest0-21swapper/208:50:532
249652570,2sleep10-21swapper/110:25:301
2904999566,9cyclictest41-21ksoftirqd/312:12:113
2904999565,9cyclictest41-21ksoftirqd/310:20:283
2904999564,9cyclictest41-21ksoftirqd/311:45:013
2904999564,7cyclictest41-21ksoftirqd/309:55:173
2904999563,8cyclictest41-21ksoftirqd/312:00:333
2904999563,6cyclictest41-21ksoftirqd/307:45:173
2904999562,8cyclictest41-21ksoftirqd/309:15:323
2904899569,14cyclictest0-21swapper/208:12:132
2904899569,14cyclictest0-21swapper/208:12:122
2904899568,15cyclictest21254-21latency_hist10:20:012
2904899568,14cyclictest7170-21df_abs08:40:162
2904899568,14cyclictest21196-21grep08:00:222
29047995645,6cyclictest25-21ksoftirqd/109:45:141
2904699561,3cyclictest0-21swapper/009:00:130
2904999553,7cyclictest41-21ksoftirqd/312:05:253
2904899559,14cyclictest27700-21sed09:25:002
29048995511,18cyclictest27406-21ls08:15:142
2904999543,8cyclictest41-21ksoftirqd/309:45:193
2904999543,7cyclictest41-21ksoftirqd/309:00:193
2904899539,14cyclictest9317-21sh07:35:192
2904999525,10cyclictest41-21ksoftirqd/308:15:293
29047995245,4cyclictest25-21ksoftirqd/110:00:001
29047995239,11cyclictest25-21ksoftirqd/108:05:001
2904999513,7cyclictest41-21ksoftirqd/312:25:253
315782500,2sleep10-21swapper/107:15:141
2904799508,8cyclictest25-21ksoftirqd/112:25:251
29047994942,4cyclictest25-21ksoftirqd/112:20:001
29047994942,4cyclictest25-21ksoftirqd/110:40:011
29047994942,4cyclictest25-21ksoftirqd/110:15:011
29046994942,4cyclictest9-21ksoftirqd/012:29:590
29047994841,4cyclictest25-21ksoftirqd/107:50:201
29046994841,4cyclictest9-21ksoftirqd/010:45:000
29046994841,4cyclictest9-21ksoftirqd/007:39:590
29047994741,3cyclictest25-21ksoftirqd/109:30:001
29046994742,3cyclictest9-21ksoftirqd/012:25:000
29046994740,4cyclictest9-21ksoftirqd/009:09:590
29046994728,4cyclictest9-21ksoftirqd/011:00:000
2904799469,10cyclictest25-21ksoftirqd/111:45:281
29047994641,3cyclictest25-21ksoftirqd/112:00:001
29047994640,4cyclictest25-21ksoftirqd/108:50:141
29047994640,3cyclictest25-21ksoftirqd/111:10:001
29047994640,3cyclictest25-21ksoftirqd/111:10:001
29047994640,3cyclictest25-21ksoftirqd/109:20:001
29047994639,4cyclictest25-21ksoftirqd/111:50:041
29047994639,4cyclictest25-21ksoftirqd/111:20:001
2904699464,7cyclictest121rcu_preempt10:25:000
29046994641,3cyclictest9-21ksoftirqd/008:59:590
29046994641,3cyclictest9-21ksoftirqd/008:09:590
29046994639,4cyclictest9-21ksoftirqd/010:29:590
29047994539,3cyclictest25-21ksoftirqd/111:15:011
2904799453,8cyclictest25-21ksoftirqd/110:20:001
29047994538,4cyclictest25-21ksoftirqd/110:05:181
29047994538,3cyclictest25-21ksoftirqd/112:05:211
2904699454,5cyclictest121rcu_preempt10:45:270
29046994538,4cyclictest9-21ksoftirqd/008:30:000
2904799448,5cyclictest25-21ksoftirqd/111:00:381
29047994438,4cyclictest25-21ksoftirqd/107:25:211
29047994438,3cyclictest25-21ksoftirqd/110:55:001
29047994437,4cyclictest25-21ksoftirqd/109:40:141
29047994437,4cyclictest25-21ksoftirqd/109:40:011
29047994437,4cyclictest25-21ksoftirqd/109:40:001
29047994437,4cyclictest25-21ksoftirqd/109:15:001
29047994437,4cyclictest25-21ksoftirqd/108:30:121
29047994437,4cyclictest25-21ksoftirqd/107:20:201
29047994436,5cyclictest25-21ksoftirqd/112:00:331
29047994436,5cyclictest25-21ksoftirqd/109:05:171
29047994436,5cyclictest25-21ksoftirqd/108:00:001
29046994438,4cyclictest9-21ksoftirqd/009:45:210
2904699443,4cyclictest121rcu_preempt08:30:300
2904799436,5cyclictest25-21ksoftirqd/112:10:291
29047994337,4cyclictest25-21ksoftirqd/110:20:251
29047994337,4cyclictest25-21ksoftirqd/109:30:391
29047994333,7cyclictest25-21ksoftirqd/109:50:311
29046994337,3cyclictest9-21ksoftirqd/012:40:000
29046994337,3cyclictest9-21ksoftirqd/010:00:270
29046994337,3cyclictest9-21ksoftirqd/009:25:240
29046994336,4cyclictest9-21ksoftirqd/011:55:000
29046994336,4cyclictest9-21ksoftirqd/008:40:000
2904699431,4cyclictest121rcu_preempt10:35:010
29047994237,3cyclictest25-21ksoftirqd/111:40:011
29047994236,4cyclictest25-21ksoftirqd/110:40:211
29047994236,3cyclictest25-21ksoftirqd/111:40:151
29047994236,3cyclictest25-21ksoftirqd/111:25:221
29047994235,4cyclictest25-21ksoftirqd/109:20:221
29047994235,4cyclictest25-21ksoftirqd/108:40:001
29047994235,4cyclictest25-21ksoftirqd/107:45:171
29047994235,4cyclictest25-21ksoftirqd/107:35:121
29047994228,10cyclictest25-21ksoftirqd/112:35:001
29046994236,4cyclictest9-21ksoftirqd/011:50:000
29046994236,4cyclictest9-21ksoftirqd/008:00:210
29046994236,3cyclictest9-21ksoftirqd/010:40:000
29046994236,3cyclictest9-21ksoftirqd/008:25:000
29046994235,4cyclictest9-21ksoftirqd/012:20:000
29046994231,4cyclictest9-21ksoftirqd/007:25:140
2904699422,4cyclictest121rcu_preempt07:10:180
29046994223,4cyclictest9-21ksoftirqd/007:45:270
29047994136,3cyclictest25-21ksoftirqd/108:30:001
29047994135,4cyclictest25-21ksoftirqd/112:35:151
29047994135,4cyclictest25-21ksoftirqd/108:50:001
29047994135,4cyclictest25-21ksoftirqd/107:44:591
29047994135,3cyclictest25-21ksoftirqd/109:00:201
29047994134,4cyclictest25-21ksoftirqd/110:05:001
29047994134,4cyclictest25-21ksoftirqd/108:25:011
29046994135,4cyclictest9-21ksoftirqd/008:50:170
29046994135,3cyclictest9-21ksoftirqd/012:00:190
29046994134,4cyclictest9-21ksoftirqd/011:59:590
2904699411,3cyclictest0-21swapper/011:28:220
2904799408,10cyclictest25-21ksoftirqd/111:20:161
2904799406,4cyclictest25-21ksoftirqd/108:10:261
2904799406,4cyclictest25-21ksoftirqd/108:10:251
2904799405,5cyclictest25-21ksoftirqd/110:45:211
29047994035,3cyclictest25-21ksoftirqd/111:35:011
29047994035,3cyclictest25-21ksoftirqd/108:59:591
29047994023,5cyclictest25-21ksoftirqd/107:15:001
29046994035,3cyclictest9-21ksoftirqd/007:20:010
29046994034,4cyclictest9-21ksoftirqd/007:30:230
29046994034,3cyclictest9-21ksoftirqd/011:20:000
29046994034,3cyclictest9-21ksoftirqd/010:00:000
2904799395,4cyclictest25-21ksoftirqd/107:30:271
29047993932,5cyclictest25-21ksoftirqd/112:22:321
29047993931,5cyclictest25-21ksoftirqd/110:55:371
29046993934,3cyclictest9-21ksoftirqd/009:40:160
29046993934,3cyclictest9-21ksoftirqd/009:20:000
29046993933,4cyclictest9-21ksoftirqd/007:45:010
29046993933,4cyclictest9-21ksoftirqd/007:20:170
29046993920,5cyclictest9-21ksoftirqd/009:20:380
2904699391,4cyclictest121rcu_preempt12:05:360
2904699391,3cyclictest0-21swapper/012:14:100
2904699391,3cyclictest0-21swapper/009:14:070
2904699383,3cyclictest121rcu_preempt10:05:150
29046993832,3cyclictest9-21ksoftirqd/011:20:320
29046993831,4cyclictest9-21ksoftirqd/009:35:000
29046993819,5cyclictest9-21ksoftirqd/010:10:290
2904699381,3cyclictest0-21swapper/008:47:170
29046993731,3cyclictest9-21ksoftirqd/011:33:390
29046993731,3cyclictest9-21ksoftirqd/008:13:520
29046993731,3cyclictest9-21ksoftirqd/008:13:510
29046993730,4cyclictest9-21ksoftirqd/011:08:160
29046993730,4cyclictest9-21ksoftirqd/011:08:160
29046993730,4cyclictest9-21ksoftirqd/007:50:120
29046993719,4cyclictest9-21ksoftirqd/010:50:310
29046993719,4cyclictest9-21ksoftirqd/008:40:250
29046993631,3cyclictest9-21ksoftirqd/012:30:250
29046993631,3cyclictest9-21ksoftirqd/011:10:190
29046993630,4cyclictest9-21ksoftirqd/008:15:280
29046993630,3cyclictest9-21ksoftirqd/009:35:220
29046993630,3cyclictest9-21ksoftirqd/009:35:210
29046993620,4cyclictest9-21ksoftirqd/011:35:170
29046993619,4cyclictest9-21ksoftirqd/011:00:260
29046993619,4cyclictest9-21ksoftirqd/007:55:140
29046993529,3cyclictest9-21ksoftirqd/009:51:510
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional