You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-05-24 - 11:22

Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7 (updated Fri May 20, 2022 00:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
564721339,20sleep30-21swapper/319:03:153
2859621130,5chrt28597-21ssh22:23:113
6991210617,60sleep07014-21apt-get19:08:040
7248991022,71cyclictest12740-21apt-get21:58:143
7227999672,17cyclictest25505-21munin-plugin-st20:03:040
7227999416,56cyclictest7161-21uniq21:48:240
7248998867,15cyclictest20026-21ssh23:53:323
7248998649,9cyclictest141rcu_preempt20:28:063
724899864,49cyclictest20714-21apt-get21:18:153
7227998647,23cyclictest392-21runrttasks22:09:460
7227998647,23cyclictest392-21runrttasks22:09:460
7240998512,65cyclictest20714-21apt-get21:18:222
7227998527,23cyclictest13695-21/usr/sbin/munin21:58:470
7240998441,21cyclictest33-21ksoftirqd/223:29:052
7240998441,21cyclictest33-21ksoftirqd/223:29:052
7227998432,18cyclictest24336-21ssh21:23:280
7227998426,30cyclictest12493-21ssh00:35:080
7227998425,22cyclictest19125-21ssh23:53:050
7227998423,34cyclictest0-21swapper/019:13:090
7227998318,41cyclictest6697-21ssh22:38:550
7227998242,23cyclictest14130-21diskmemload00:03:300
7227998242,23cyclictest14130-21diskmemload00:03:300
7227998239,27cyclictest0-21swapper/021:09:500
7227998239,27cyclictest0-21swapper/021:09:500
7227998227,21cyclictest0-21swapper/022:58:170
7227998227,21cyclictest0-21swapper/022:58:170
7240998148,13cyclictest33-21ksoftirqd/219:58:332
7227998140,25cyclictest0-21swapper/000:16:320
7227998124,15cyclictest0-21swapper/022:28:150
7227998115,41cyclictest0-21swapper/022:43:060
7248998030,26cyclictest141rcu_preempt21:17:363
724099809,46cyclictest9440-21diskstats19:13:152
7227998031,24cyclictest0-21swapper/022:56:220
7227998029,18cyclictest12880-21cstates21:03:200
7227998024,21cyclictest759-21grep20:23:370
7227998022,31cyclictest32147-21ssh21:36:210
706628014,35sleep20-21swapper/219:08:062
7248997953,17cyclictest6565-21sh21:48:033
7248997934,14cyclictest141rcu_preempt23:38:273
7240997951,13cyclictest33-21ksoftirqd/221:32:412
7232997922,28cyclictest14130-21diskmemload23:43:111
7227997932,28cyclictest14130-21diskmemload23:21:080
7227997926,32cyclictest0-21swapper/023:25:270
7227997923,23cyclictest29872-21grep20:13:360
7227997921,19cyclictest6574-21munin-run21:48:040
7227997921,11cyclictest2110-21apt-get00:18:150
7227997918,38cyclictest11776-21runrttasks21:55:310
7227997918,38cyclictest11776-21runrttasks21:55:310
7240997842,11cyclictest33-21ksoftirqd/221:44:072
7232997856,13cyclictest25172-21munin-plugin-st00:03:031
7227997838,23cyclictest6604-21taskset00:24:540
7227997837,24cyclictest14130-21diskmemload21:22:190
7227997827,31cyclictest14130-21diskmemload22:13:190
7227997825,44cyclictest16247-21sort19:33:240
7227997821,36cyclictest27992-21kworker/u8:221:16:130
7227997821,32cyclictest9649-21ssh00:30:150
7240997711,24cyclictest0-21swapper/221:58:142
7232997725,34cyclictest11382-21ssh22:48:041
7227997726,28cyclictest2030-21rm21:39:150
7227997722,32cyclictest4470-21ssh22:36:170
7240997647,16cyclictest0-21swapper/220:33:362
7227997652,17cyclictest0-21swapper/022:23:060
7227997624,30cyclictest12523-21ssh23:42:060
7248997552,14cyclictest973-21fschecks_count21:38:313
7240997548,7cyclictest33-21ksoftirqd/200:08:182
7240997547,13cyclictest33-21ksoftirqd/221:12:422
7240997547,13cyclictest33-21ksoftirqd/221:12:422
7227997531,17cyclictest0-21swapper/022:24:070
7227997529,16cyclictest0-21swapper/022:06:310
7227997526,30cyclictest14130-21diskmemload23:33:170
7227997522,21cyclictest24227-21/usr/sbin/munin23:08:220
7227997521,20cyclictest27188-21ls23:13:180
7227997425,20cyclictest14556-21chrt19:28:210
7227997425,20cyclictest14556-21chrt19:28:210
7227997424,25cyclictest14130-21diskmemload23:57:510
7227997424,18cyclictest29548-21sed00:08:350
7227997423,22cyclictest27996-21sh21:28:400
7227997423,19cyclictest28681-21kworker/0:020:28:340
7227997419,21cyclictest22066-21taskset23:03:400
704027413,34sleep1365-21dbus-daemon19:08:051
7240997336,13cyclictest33-21ksoftirqd/221:17:042
7227997327,38cyclictest7668-21systemd-run20:48:050
7227997324,30cyclictest6235-21runrttasks23:31:500
7227997324,30cyclictest6235-21runrttasks23:31:500
7227997324,18cyclictest7708-21grep19:08:490
7227997323,19cyclictest13241-21sh23:43:190
7248997232,12cyclictest141rcu_preempt22:44:193
7248997219,26cyclictest141rcu_preempt19:33:233
7240997251,14cyclictest6028-21idleruntime-cro20:43:032
7240997243,7cyclictest33-21ksoftirqd/220:08:232
7240997243,7cyclictest33-21ksoftirqd/220:08:232
7227997225,18cyclictest11851-21sh22:48:210
7240997146,16cyclictest17163-21threads23:48:362
7240997138,11cyclictest33-21ksoftirqd/222:31:582
7232997115,22cyclictest141rcu_preempt21:33:141
7227997122,27cyclictest14130-21diskmemload23:59:230
7248997042,13cyclictest141rcu_preempt22:37:253
7248997039,10cyclictest141rcu_preempt19:48:373
7248997039,10cyclictest141rcu_preempt19:48:373
7248997028,23cyclictest611-21in:imklog00:13:353
7248997014,20cyclictest41-21ksoftirqd/319:13:333
7240997052,11cyclictest359-21latency_hist21:38:032
7232997055,7cyclictest0-21swapper/119:58:031
7227997017,20cyclictest0-21swapper/020:08:240
7227997017,20cyclictest0-21swapper/020:08:240
7248996916,14cyclictest0-21swapper/323:28:333
7248996916,14cyclictest0-21swapper/323:28:333
7240996944,14cyclictest0-21swapper/223:58:062
7240996939,13cyclictest33-21ksoftirqd/200:23:062
7240996935,13cyclictest33-21ksoftirqd/221:51:502
7240996935,11cyclictest33-21ksoftirqd/219:08:442
7232996943,9cyclictest0-21swapper/119:08:451
7227996923,17cyclictest9813-21idleruntime20:53:250
7227996920,18cyclictest18352-21wc19:38:340
7227996920,18cyclictest18352-21wc19:38:340
7227996915,23cyclictest22355-21latency_hist19:53:060
7227996915,23cyclictest22355-21latency_hist19:53:060
7232996847,12cyclictest27992-21kworker/u8:221:28:341
7232996817,28cyclictest20975-21diskmemload22:10:011
7232996817,28cyclictest20975-21diskmemload22:10:011
7227996820,16cyclictest392-21runrttasks19:24:160
7227996819,31cyclictest392-21runrttasks19:46:200
7248996740,8cyclictest141rcu_preempt23:15:503
7248996726,24cyclictest2035-21kworker/u8:123:28:063
7248996715,32cyclictest25557-21apt-get00:03:153
7248996715,32cyclictest25557-21apt-get00:03:153
7240996727,13cyclictest33-21ksoftirqd/222:28:052
7240996719,14cyclictest33-21ksoftirqd/222:07:232
7232996743,15cyclictest10905-21sendmail19:18:031
722799677,35cyclictest0-21swapper/019:54:250
7227996721,18cyclictest11609-21latency19:18:290
7227996721,18cyclictest11609-21latency19:18:290
7227996719,17cyclictest30648-21taskset20:18:180
7227996719,17cyclictest30648-21taskset20:18:180
7248996647,12cyclictest7293-21sh21:48:273
7248996645,8cyclictest141rcu_preempt22:51:093
7248996641,9cyclictest141rcu_preempt20:23:073
7248996641,9cyclictest141rcu_preempt20:23:073
7248996613,18cyclictest41-21ksoftirqd/322:18:373
7240996644,16cyclictest23723-21idleruntime-cro23:08:032
724099664,33cyclictest0-21swapper/222:43:162
7240996643,16cyclictest29894-21idleruntime-cro23:18:032
7240996628,15cyclictest0-21swapper/220:58:342
7227996620,17cyclictest4369-21cron20:38:030
7227996618,20cyclictest5970-21taskset20:42:340
7227996616,19cyclictest0-21swapper/020:58:140
7248996540,9cyclictest141rcu_preempt20:33:363
7248996537,8cyclictest141rcu_preempt23:03:133
7248996537,8cyclictest141rcu_preempt20:43:383
7248996537,8cyclictest141rcu_preempt20:03:263
7248996529,19cyclictest0-21swapper/321:08:363
7248996529,19cyclictest0-21swapper/321:08:363
7240996541,8cyclictest33-21ksoftirqd/200:33:062
7240996540,16cyclictest276-21systemd-journal22:08:202
7240996540,16cyclictest276-21systemd-journal22:08:202
7240996531,10cyclictest33-21ksoftirqd/219:53:302
7240996522,16cyclictest33-21ksoftirqd/221:43:062
723299659,17cyclictest0-21swapper/122:48:331
7232996550,9cyclictest1897-21ntp_kernel_err20:28:291
7232996548,10cyclictest16510-21fschecks_count22:03:231
7232996546,12cyclictest4186-21rm23:28:221
7232996546,12cyclictest4186-21rm23:28:221
7232996510,16cyclictest0-21swapper/122:23:351
7227996519,18cyclictest26837-21taskset20:04:480
7248996435,7cyclictest141rcu_preempt00:28:353
7248996424,10cyclictest0-21swapper/319:08:493
7248996417,18cyclictest41-21ksoftirqd/323:18:343
7248996416,19cyclictest41-21ksoftirqd/322:38:373
7248996414,10cyclictest0-21swapper/321:55:383
7248996414,10cyclictest0-21swapper/321:55:383
7240996412,31cyclictest0-21swapper/223:33:362
7232996446,12cyclictest9014-21unixbench_multi00:28:371
7248996333,9cyclictest141rcu_preempt19:33:063
7248996333,9cyclictest141rcu_preempt19:33:063
7248996313,29cyclictest41-21ksoftirqd/300:33:243
7240996339,8cyclictest33-21ksoftirqd/223:18:232
7240996330,10cyclictest33-21ksoftirqd/220:53:352
7240996321,20cyclictest611-21in:imklog22:48:272
7232996346,10cyclictest22528-21cpuspeed22:13:191
7227996318,17cyclictest9080-21taskset20:51:030
7248996234,14cyclictest0-21swapper/320:58:233
7248996232,14cyclictest141rcu_preempt23:59:243
7248996232,13cyclictest141rcu_preempt23:43:203
7248996230,8cyclictest141rcu_preempt22:59:283
7248996230,8cyclictest141rcu_preempt22:59:283
7248996227,9cyclictest141rcu_preempt22:53:333
7248996226,7cyclictest141rcu_preempt21:23:193
7248996221,25cyclictest141rcu_preempt23:50:443
7248996220,20cyclictest41-21ksoftirqd/322:08:373
7248996220,20cyclictest41-21ksoftirqd/322:08:373
7248996219,11cyclictest0-21swapper/300:25:463
724099623,37cyclictest25229-21munin-run22:18:032
7240996223,12cyclictest0-21swapper/223:58:192
7240996222,26cyclictest33-21ksoftirqd/222:38:062
7240996220,11cyclictest141rcu_preempt00:13:372
7232996246,10cyclictest610-21in:imuxsock22:03:031
7232996216,15cyclictest0-21swapper/120:33:351
7248996111,25cyclictest41-21ksoftirqd/322:13:343
7248996111,22cyclictest41-21ksoftirqd/322:32:373
7240996140,7cyclictest33-21ksoftirqd/219:38:342
7240996140,7cyclictest33-21ksoftirqd/219:38:342
7240996138,8cyclictest33-21ksoftirqd/200:34:272
7240996128,11cyclictest33-21ksoftirqd/200:26:102
7240996120,9cyclictest0-21swapper/222:57:042
7232996114,16cyclictest0-21swapper/120:18:361
7232996114,16cyclictest0-21swapper/120:18:361
724899607,29cyclictest29601-21kworker/3:121:36:273
7248996039,8cyclictest141rcu_preempt23:13:063
7248996026,9cyclictest141rcu_preempt19:58:233
7248996023,20cyclictest0-21swapper/320:48:303
7248996016,18cyclictest41-21ksoftirqd/323:38:023
7248996015,17cyclictest41-21ksoftirqd/320:38:333
7248996014,13cyclictest0-21swapper/300:18:353
7248996011,22cyclictest41-21ksoftirqd/320:23:293
7240996046,9cyclictest3640-21idleruntime-cro23:28:032
7240996039,16cyclictest13650-21cut23:43:282
7240996036,14cyclictest0-21swapper/222:58:292
7240996036,14cyclictest0-21swapper/222:58:292
7240996030,16cyclictest180-21pr/ttyS020:48:362
7248995930,9cyclictest141rcu_preempt19:23:203
7248995924,8cyclictest141rcu_preempt22:08:053
7248995916,19cyclictest27839-21ssh21:28:383
7248995914,22cyclictest41-21ksoftirqd/321:03:373
724099595,30cyclictest0-21swapper/222:22:522
7240995943,8cyclictest0-21swapper/223:08:182
7240995940,12cyclictest32017-21systemd-run20:23:052
7240995940,12cyclictest32017-21systemd-run20:23:052
7240995935,16cyclictest0-21swapper/220:13:242
7240995931,10cyclictest141rcu_preempt19:28:282
7240995931,10cyclictest141rcu_preempt19:28:282
7232995934,16cyclictest6567-21idleruntime-cro21:48:041
7232995917,16cyclictest392-21runrttasks23:02:431
7232995917,16cyclictest392-21runrttasks23:02:431
7248995815,27cyclictest41-21ksoftirqd/300:08:343
7248995812,9cyclictest141rcu_preempt19:18:373
7248995812,9cyclictest141rcu_preempt19:18:373
7240995838,7cyclictest33-21ksoftirqd/221:55:392
7240995838,7cyclictest33-21ksoftirqd/221:55:392
7240995819,31cyclictest0-21swapper/220:43:122
7232995840,11cyclictest0-21swapper/123:18:031
7232995837,10cyclictest0-21swapper/123:43:061
7232995817,8cyclictest3668-21systemd-run23:28:041
7232995812,18cyclictest14130-21diskmemload00:15:491
7232995810,18cyclictest946-21chrt20:25:381
7248995738,12cyclictest611-21in:imklog20:11:343
7248995738,12cyclictest611-21in:imklog20:11:343
7248995720,8cyclictest141rcu_preempt20:13:193
7248995714,15cyclictest41-21ksoftirqd/319:48:043
724099579,22cyclictest141rcu_preempt19:43:372
7240995738,12cyclictest12841-21cpuspeed21:03:202
7240995733,12cyclictest0-21swapper/219:48:242
7240995733,12cyclictest0-21swapper/219:48:242
7240995730,15cyclictest0-21swapper/219:33:232
723299578,18cyclictest1073-21taskset22:30:291
7232995739,12cyclictest1-21systemd20:58:231
7232995734,15cyclictest21940-21unixbench_multi23:03:361
7232995724,22cyclictest2362-21apt-get22:33:131
7232995717,16cyclictest0-21swapper/123:38:031
7248995630,6cyclictest0-21swapper/319:42:473
7248995630,6cyclictest0-21swapper/319:42:473
7248995625,14cyclictest141rcu_preempt19:53:343
7248995612,24cyclictest41-21ksoftirqd/320:58:063
7240995642,7cyclictest0-21swapper/200:03:432
7240995642,7cyclictest0-21swapper/200:03:432
7240995624,7cyclictest0-21swapper/220:28:342
7240995623,10cyclictest611-21in:imklog22:38:362
7232995625,20cyclictest16307-21apt-get23:48:161
724099558,18cyclictest21489-21kworker/2:221:24:212
7240995537,12cyclictest276-21systemd-journal23:38:212
7240995524,8cyclictest33-21ksoftirqd/219:18:252
7240995524,8cyclictest33-21ksoftirqd/219:18:252
723299557,18cyclictest14130-21diskmemload21:21:451
723299555,17cyclictest29122-21cstates20:13:201
723299554,20cyclictest10960-21apt-get00:33:181
7232995539,9cyclictest20319-21idleruntime-cro21:18:031
7232995535,14cyclictest8377-21/usr/sbin/munin20:48:291
7232995529,18cyclictest1664-21sed21:38:441
7240995425,13cyclictest611-21in:imklog20:03:262
723299549,15cyclictest15610-21taskset21:08:481
723299549,15cyclictest15610-21taskset21:08:481
723299547,18cyclictest14130-21diskmemload00:26:581
7232995438,9cyclictest0-21swapper/122:53:231
7232995438,9cyclictest0-21swapper/119:43:041
7232995438,9cyclictest0-21swapper/119:43:041
7232995431,16cyclictest7111-21fschecks_count21:48:241
7232995430,17cyclictest6333-21cat22:38:361
7240995333,10cyclictest0-21swapper/219:28:032
723299539,18cyclictest14130-21diskmemload21:55:481
723299539,18cyclictest14130-21diskmemload21:55:481
7232995334,9cyclictest0-21swapper/100:18:241
7232995330,12cyclictest0-21swapper/122:23:061
7232995330,12cyclictest0-21swapper/119:33:071
7232995330,12cyclictest0-21swapper/119:33:071
7240995230,12cyclictest0-21swapper/220:23:372
723299529,16cyclictest14130-21diskmemload21:23:491
723299525,16cyclictest10194-21sensors_temp20:53:341
7232995237,7cyclictest0-21swapper/123:53:361
7232995235,9cyclictest17368-21modprobe19:38:031
7232995231,13cyclictest0-21swapper/123:08:231
7232995230,10cyclictest0-21swapper/119:43:201
7232995229,12cyclictest0-21swapper/120:43:251
7232995228,11cyclictest0-21swapper/120:08:251
7232995228,11cyclictest0-21swapper/120:08:251
7232995227,15cyclictest27094-21cron20:08:031
723299515,12cyclictest25-21ksoftirqd/100:03:071
723299515,12cyclictest25-21ksoftirqd/100:03:071
7232995128,12cyclictest0-21swapper/119:53:071
723299508,20cyclictest29606-21/usr/sbin/munin00:08:361
7232995030,10cyclictest0-21swapper/119:18:371
7232995030,10cyclictest0-21swapper/119:18:371
7232995028,11cyclictest0-21swapper/119:48:351
7232994928,12cyclictest0-21swapper/121:08:071
7232994927,11cyclictest4677-21apt-get20:38:141
7232994927,11cyclictest0-21swapper/123:18:271
7232994821,19cyclictest25708-21latency_hist20:03:071
7232994629,9cyclictest12800-21apt-get19:23:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional