You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-16 - 05:32

ARM Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7 (updated Tue Aug 16, 2022 00:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
278592915,13sleep00-21swapper/018:58:400
28352998729,10cyclictest0-21swapper/221:17:132
28347998746,14cyclictest141rcu_preempt22:24:451
274642866,54sleep20-21swapper/218:57:522
2817728519,38sleep10-21swapper/119:02:111
28352998426,10cyclictest0-21swapper/221:37:442
28340998432,20cyclictest141rcu_preempt22:33:220
278202845,13sleep30-21swapper/318:58:103
28352998333,43cyclictest0-21swapper/219:52:102
28352998332,44cyclictest29694-21ssh22:42:102
28352998324,21cyclictest19200-21unixbench_multi23:17:442
28352998318,14cyclictest540-21cut21:52:372
28357998258,15cyclictest4485-21unixbench_multi22:52:453
28352998231,44cyclictest0-21swapper/200:12:462
28352998231,44cyclictest0-21swapper/200:12:462
28352998229,44cyclictest0-21swapper/223:57:122
2835299813,19cyclictest22676-21grep20:22:342
28352998127,10cyclictest0-21swapper/221:17:102
28352998124,9cyclictest0-21swapper/200:07:452
28352998118,12cyclictest0-21swapper/219:27:322
2835299803,23cyclictest6772-21grep22:02:482
28352998030,43cyclictest0-21swapper/219:37:282
28352998018,11cyclictest0-21swapper/222:47:442
28352998018,11cyclictest0-21swapper/222:47:442
2835299793,23cyclictest22627-21grep00:17:472
2835299793,22cyclictest3742-21fschecks_count22:52:322
28352997918,12cyclictest0-21swapper/222:32:122
28352997911,16cyclictest0-21swapper/220:22:132
28347997943,15cyclictest141rcu_preempt23:53:331
2835299786,21cyclictest6472-21modprobe23:52:102
282592780,6sleep028252-21ssh00:27:450
2835299774,23cyclictest25478-21modprobe20:32:092
2835299773,23cyclictest4119-21grep21:02:482
2835299773,23cyclictest4119-21grep21:02:482
2835299773,20cyclictest31544-21sed20:47:452
2835299773,20cyclictest31544-21sed20:47:452
2835299772,24cyclictest29370-21timerwakeupswit19:02:482
2835299772,24cyclictest29370-21timerwakeupswit19:02:482
28352997717,13cyclictest0-21swapper/223:26:202
28352997713,19cyclictest3184-21grep19:22:312
28352997713,19cyclictest3184-21grep19:22:312
2835299764,21cyclictest32055-21latency_hist19:12:122
2835299763,20cyclictest19125-21latency_hist20:12:132
28352997618,12cyclictest0-21swapper/223:47:122
28352997616,12cyclictest10140-21sendmail_mailqu23:02:502
2835299753,23cyclictest392-21runrttasks22:00:092
2835299753,22cyclictest598-21grep19:12:442
2835299753,22cyclictest598-21grep19:12:442
28352997526,9cyclictest0-21swapper/222:59:172
2835299752,22cyclictest12966-21unixbench_multi22:12:442
28352997514,13cyclictest19001-21unixbench_singl21:27:512
28347997552,16cyclictest0-21swapper/123:17:131
2835799743,45cyclictest410-21snmpd23:52:103
2835299745,21cyclictest32496-21fschecks_time20:52:322
2835299743,22cyclictest16536-21ps20:02:442
28340997436,16cyclictest141rcu_preempt00:08:200
28340997435,9cyclictest141rcu_preempt21:57:410
28340997434,10cyclictest141rcu_preempt21:17:140
28340997425,28cyclictest0-21swapper/022:38:150
28357997330,25cyclictest0-21swapper/321:37:213
28357997322,19cyclictest9347-21fschecks_time21:12:333
2835299733,22cyclictest15574-21ntp_kernel_pll_21:22:422
2835299733,22cyclictest15574-21ntp_kernel_pll_21:22:422
2835299733,20cyclictest2232-21grep20:57:442
2835299732,22cyclictest20874-21/usr/sbin/munin21:32:392
28347997355,10cyclictest11620-21cron22:12:091
28347997334,13cyclictest141rcu_preempt21:22:321
28347997334,13cyclictest141rcu_preempt21:22:321
28340997318,12cyclictest141rcu_preempt22:29:260
28357997251,13cyclictest20681-21cron22:27:093
2835299725,9cyclictest0-21swapper/219:20:552
28352997244,19cyclictest392-21runrttasks23:37:342
28352997244,19cyclictest392-21runrttasks23:37:342
28352997221,9cyclictest0-21swapper/221:47:102
28347997237,13cyclictest141rcu_preempt23:22:401
28347997235,8cyclictest141rcu_preempt20:22:431
28340997254,12cyclictest3155-21runrttasks22:52:090
28340997254,12cyclictest3155-21runrttasks22:52:090
28357997127,18cyclictest0-21swapper/321:04:273
28357997127,18cyclictest0-21swapper/321:04:273
28352997126,12cyclictest0-21swapper/222:32:242
28352997125,11cyclictest0-21swapper/223:58:112
28352997125,11cyclictest0-21swapper/223:58:112
28347997139,15cyclictest141rcu_preempt21:34:421
28357997047,13cyclictest0-21swapper/321:47:553
28357997021,19cyclictest31495-21taskset22:43:553
28352997051,11cyclictest10871-21runrttasks19:47:122
28352997012,11cyclictest0-21swapper/200:23:212
28347997040,7cyclictest141rcu_preempt21:19:331
28347997019,10cyclictest0-21swapper/119:27:451
28347997015,29cyclictest25-21ksoftirqd/121:41:101
28347997011,14cyclictest25-21ksoftirqd/121:46:481
2834099706,41cyclictest141rcu_preempt21:52:390
28340997048,16cyclictest21321-21latency_hist00:17:110
28340997048,16cyclictest21321-21latency_hist00:17:110
28340997047,14cyclictest11909-21timerwakeupswit19:47:440
2834099704,11cyclictest12-21ksoftirqd/019:02:470
2834099704,11cyclictest12-21ksoftirqd/019:02:470
28340997035,29cyclictest10462-21unixbench_multi23:02:530
28357996954,9cyclictest25596-21latency_hist20:32:123
28357996940,20cyclictest17799-21latency_hist21:27:133
28357996940,20cyclictest17799-21latency_hist21:27:133
2835299693,23cyclictest29761-21/usr/sbin/munin21:47:512
2835299693,22cyclictest392-21runrttasks20:00:542
28340996941,8cyclictest141rcu_preempt00:22:390
28357996821,18cyclictest28491-21ssh21:44:393
28357996819,22cyclictest844-21chrt20:53:103
28352996832,8cyclictest141rcu_preempt23:27:452
28352996832,8cyclictest141rcu_preempt23:27:452
28352996810,16cyclictest27396-21/usr/sbin/munin00:27:392
28347996837,15cyclictest141rcu_preempt00:18:501
28347996833,13cyclictest141rcu_preempt20:27:481
28347996830,10cyclictest141rcu_preempt20:07:131
28347996810,14cyclictest241ktimers/100:22:261
28340996841,12cyclictest141rcu_preempt22:27:120
2835799678,39cyclictest0-21swapper/323:57:323
2835799678,39cyclictest0-21swapper/323:57:323
28357996748,12cyclictest22192-21unixbench_multi23:22:493
28357996740,18cyclictest16137-21/usr/sbin/munin20:02:353
28357996740,15cyclictest0-21swapper/320:07:313
28357996723,15cyclictest0-21swapper/323:08:353
28357996718,20cyclictest5598-21taskset19:27:523
2835299672,20cyclictest30419-21latency_hist20:47:112
28347996745,12cyclictest0-21swapper/122:42:091
28347996737,7cyclictest141rcu_preempt20:52:271
28340996740,12cyclictest141rcu_preempt19:17:320
28340996718,44cyclictest0-21swapper/023:57:200
28340996718,44cyclictest0-21swapper/023:57:200
28357996630,26cyclictest30560-21apt-get19:07:243
28357996617,19cyclictest20107-21seq21:31:013
28347996619,14cyclictest25-21ksoftirqd/123:37:091
28340996646,8cyclictest141rcu_preempt20:07:120
28340996637,13cyclictest141rcu_preempt23:55:120
2834099663,43cyclictest141rcu_preempt19:32:450
2834099663,43cyclictest141rcu_preempt19:32:450
28340996631,10cyclictest141rcu_preempt20:27:090
28340996630,13cyclictest141rcu_preempt21:37:340
28340996629,12cyclictest141rcu_preempt23:27:310
28340996629,12cyclictest141rcu_preempt23:27:310
28340996623,24cyclictest141rcu_preempt23:10:000
28340996610,13cyclictest141rcu_preempt23:17:420
28357996545,14cyclictest0-21swapper/320:42:103
28357996545,14cyclictest0-21swapper/320:42:103
28357996537,17cyclictest276-21systemd-journal19:12:363
28357996537,17cyclictest276-21systemd-journal19:12:363
28357996517,27cyclictest141rcu_preempt20:17:323
28357996517,18cyclictest13127-21timerandwakeup21:18:083
28352996542,12cyclictest0-21swapper/220:32:492
28347996543,13cyclictest12280-21ssh22:12:331
28347996541,9cyclictest141rcu_preempt20:33:141
28347996515,12cyclictest0-21swapper/100:12:431
28347996515,12cyclictest0-21swapper/100:12:431
28340996534,8cyclictest141rcu_preempt20:12:350
28340996520,25cyclictest141rcu_preempt00:17:290
28357996438,13cyclictest11090-21apt-get19:47:243
2835299643,19cyclictest12611-21fschecks_count23:07:382
28347996446,12cyclictest31514-21unixbench_multi20:47:451
28347996446,12cyclictest31514-21unixbench_multi20:47:451
28347996431,8cyclictest141rcu_preempt19:57:381
28347996423,22cyclictest25-21ksoftirqd/122:17:401
28347996419,23cyclictest141rcu_preempt20:19:151
28347996414,18cyclictest25-21ksoftirqd/119:47:111
28340996438,7cyclictest141rcu_preempt21:07:340
28340996434,9cyclictest141rcu_preempt20:42:420
28340996432,8cyclictest141rcu_preempt23:27:110
28340996431,14cyclictest141rcu_preempt21:16:340
28340996420,11cyclictest12-21ksoftirqd/023:32:280
28340996419,7cyclictest0-21swapper/000:03:130
28340996417,21cyclictest141rcu_preempt23:12:440
28340996413,31cyclictest12-21ksoftirqd/023:47:120
28357996341,16cyclictest0-21swapper/320:22:263
28357996341,15cyclictest0-21swapper/322:37:463
28357996325,10cyclictest19969-21ssh00:13:283
28357996325,10cyclictest19969-21ssh00:13:283
28357996317,18cyclictest2803-21diskmemload23:42:033
28357996317,18cyclictest2803-21diskmemload23:42:033
28357996316,20cyclictest10537-21diskmemload22:09:043
28357996314,22cyclictest141rcu_preempt22:27:453
28352996335,9cyclictest0-21swapper/220:37:332
28352996335,9cyclictest0-21swapper/220:37:332
28352996323,18cyclictest0-21swapper/222:46:582
28347996343,8cyclictest141rcu_preempt22:46:001
2834799634,23cyclictest25-21ksoftirqd/100:02:091
2834799634,23cyclictest25-21ksoftirqd/100:02:091
2834799633,31cyclictest25-21ksoftirqd/121:57:371
28347996332,9cyclictest141rcu_preempt19:22:541
28347996332,9cyclictest141rcu_preempt19:22:541
28347996331,12cyclictest141rcu_preempt23:06:181
28347996323,21cyclictest30500-21fschecks_time23:37:331
28347996323,21cyclictest30500-21fschecks_time23:37:331
28340996332,9cyclictest141rcu_preempt22:42:280
28340996321,24cyclictest141rcu_preempt23:48:120
28340996314,23cyclictest141rcu_preempt21:02:480
28340996314,23cyclictest141rcu_preempt21:02:480
28357996244,12cyclictest27163-21unin-run20:37:113
28357996239,12cyclictest0-21swapper/319:17:323
28357996234,15cyclictest0-21swapper/300:02:383
28357996227,25cyclictest0-21swapper/322:32:343
28357996220,18cyclictest0-21swapper/323:52:273
28347996242,14cyclictest11748-21unin-run21:17:101
28347996241,8cyclictest141rcu_preempt00:28:211
28347996241,13cyclictest26688-21sh22:37:091
28347996211,29cyclictest25-21ksoftirqd/100:04:531
28340996240,8cyclictest141rcu_preempt19:52:330
28340996237,7cyclictest141rcu_preempt20:47:340
28340996237,7cyclictest141rcu_preempt20:47:340
28340996229,9cyclictest141rcu_preempt19:42:280
28340996229,11cyclictest141rcu_preempt21:47:540
28340996222,11cyclictest141rcu_preempt19:27:420
28340996214,11cyclictest141rcu_preempt19:07:190
28357996116,17cyclictest410-21snmpd21:52:453
28357996116,14cyclictest0-21swapper/323:01:383
28352996134,12cyclictest276-21systemd-journal21:07:352
2834799616,25cyclictest25-21ksoftirqd/120:07:351
28347996138,14cyclictest15036-21kworker/u8:422:28:251
28347996118,13cyclictest141rcu_preempt21:30:431
28340996139,7cyclictest141rcu_preempt21:42:400
28340996126,13cyclictest141rcu_preempt20:17:320
2835799608,22cyclictest141rcu_preempt23:32:383
28357996044,8cyclictest0-21swapper/322:47:453
28357996044,8cyclictest0-21swapper/322:47:453
28357996039,11cyclictest27418-21apt-get00:27:283
28357996014,18cyclictest27112-21munin-run00:27:113
28357996011,21cyclictest12113-21df22:12:283
28347996036,13cyclictest0-21swapper/121:52:131
28347996032,8cyclictest141rcu_preempt19:02:391
28347996032,8cyclictest141rcu_preempt19:02:391
28347996029,12cyclictest141rcu_preempt19:07:431
28340996034,12cyclictest141rcu_preempt20:37:120
28340996034,10cyclictest141rcu_preempt21:27:320
28340996033,13cyclictest141rcu_preempt22:52:330
28340996028,12cyclictest141rcu_preempt22:57:470
28340996025,16cyclictest141rcu_preempt20:37:330
28340996025,16cyclictest141rcu_preempt20:37:330
28340996020,20cyclictest141rcu_preempt21:33:270
28340996014,11cyclictest141rcu_preempt20:27:370
28357995935,13cyclictest0-21swapper/323:27:353
28357995935,13cyclictest0-21swapper/323:27:353
28357995921,28cyclictest12027-21diskmemload23:07:093
28357995913,20cyclictest10531-21runrttasks19:43:433
28352995941,6cyclictest141rcu_preempt23:33:172
2834799597,28cyclictest25-21ksoftirqd/120:12:221
28347995932,8cyclictest141rcu_preempt23:47:411
28347995932,7cyclictest141rcu_preempt20:57:351
28347995931,8cyclictest141rcu_preempt00:07:231
28347995920,24cyclictest141rcu_preempt21:07:401
28347995917,21cyclictest25-21ksoftirqd/122:57:111
28347995916,19cyclictest141rcu_preempt22:58:471
28347995913,15cyclictest25-21ksoftirqd/122:06:181
2834099598,23cyclictest12-21ksoftirqd/022:02:440
28340995935,9cyclictest141rcu_preempt19:57:440
28340995934,9cyclictest141rcu_preempt19:12:270
28340995934,9cyclictest141rcu_preempt19:12:270
28340995924,18cyclictest141rcu_preempt22:22:100
28340995911,23cyclictest141rcu_preempt22:08:590
323102580,6sleep00-21swapper/023:40:380
323102580,6sleep00-21swapper/023:40:380
28357995840,9cyclictest2803-21diskmemload21:57:463
28357995836,14cyclictest0-21swapper/300:22:113
28357995825,24cyclictest4496-21grep19:27:133
28357995825,24cyclictest4496-21grep19:27:133
28357995820,14cyclictest141rcu_preempt22:21:293
28352995829,9cyclictest0-21swapper/200:03:502
2834799587,27cyclictest0-21swapper/120:41:471
2834799587,27cyclictest0-21swapper/120:41:471
28347995827,11cyclictest141rcu_preempt23:42:301
2834799582,21cyclictest12492-21munin-run19:52:121
28357995741,8cyclictest0-21swapper/319:02:353
28357995741,8cyclictest0-21swapper/319:02:353
28357995732,13cyclictest0-21swapper/320:57:463
28357995728,18cyclictest0-21swapper/319:42:093
28357995722,14cyclictest0-21swapper/320:47:253
28357995722,14cyclictest0-21swapper/320:47:253
28352995738,12cyclictest276-21systemd-journal23:12:262
2834799579,21cyclictest25-21ksoftirqd/121:57:101
28347995717,16cyclictest25-21ksoftirqd/119:37:131
28347995717,16cyclictest25-21ksoftirqd/119:37:131
28347995716,21cyclictest25-21ksoftirqd/123:28:071
28347995716,21cyclictest25-21ksoftirqd/123:28:071
28347995716,18cyclictest25-21ksoftirqd/119:37:321
28340995738,9cyclictest0-21swapper/019:42:090
28340995734,12cyclictest141rcu_preempt20:57:320
28340995722,21cyclictest141rcu_preempt22:13:090
28340995713,22cyclictest0-21swapper/021:27:130
28357995639,11cyclictest6335-21fschecks_count21:07:343
28357995638,11cyclictest18371-21systemd00:12:123
28357995635,10cyclictest0-21swapper/323:17:113
28347995628,8cyclictest141rcu_preempt23:08:261
28347995618,8cyclictest141rcu_preempt20:42:231
28347995612,8cyclictest241ktimers/122:47:461
28347995612,8cyclictest241ktimers/122:47:461
28347995611,21cyclictest25-21ksoftirqd/121:07:101
28347995611,21cyclictest25-21ksoftirqd/121:07:101
28340995631,7cyclictest141rcu_preempt21:25:360
2835799558,16cyclictest0-21swapper/319:58:543
28357995530,17cyclictest6497-21fschecks_time19:32:343
28357995530,17cyclictest6497-21fschecks_time19:32:343
28357995527,17cyclictest0-21swapper/320:42:223
28357995522,25cyclictest14082-21idleruntime-cro19:57:103
28357995521,24cyclictest3612-21date23:47:123
28357995518,29cyclictest19487-21cut20:12:323
28352995530,7cyclictest141rcu_preempt19:32:342
28352995530,7cyclictest141rcu_preempt19:32:342
28347995523,12cyclictest141rcu_preempt19:19:151
28347995514,19cyclictest25-21ksoftirqd/119:12:341
28347995514,19cyclictest25-21ksoftirqd/119:12:341
28347995513,20cyclictest25-21ksoftirqd/123:17:501
28340995530,16cyclictest0-21swapper/020:12:100
28340995514,19cyclictest12-21ksoftirqd/019:22:410
28340995514,19cyclictest12-21ksoftirqd/019:22:410
28352995435,12cyclictest10823-21in:imklog22:20:342
28352995431,7cyclictest141rcu_preempt22:22:202
28352995429,12cyclictest0-21swapper/222:12:122
28352995427,8cyclictest33-21ksoftirqd/219:52:442
28357995327,18cyclictest21201-21grep21:32:393
28357995326,19cyclictest7120-21tail22:02:523
28357995325,19cyclictest18590-21/usr/sbin/munin23:17:333
28347995317,14cyclictest25-21ksoftirqd/119:52:361
28352995233,11cyclictest0-21swapper/220:12:412
28340995233,12cyclictest1180-21awk20:57:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional