You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-02-04 - 23:59

ARM Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7 (updated Sat Feb 04, 2023 12:44:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
311059910338,49cyclictest0-21swapper/011:51:500
31105999946,27cyclictest14850-21sh09:21:240
31105999944,28cyclictest15435-21ssh09:22:000
31105999944,28cyclictest15435-21ssh09:22:000
31105999741,28cyclictest23076-21ssh10:27:230
31105999740,30cyclictest24609-21ssh11:25:100
31105999639,19cyclictest13554-21ls11:06:390
31105999636,35cyclictest5035-21awk09:01:560
31105999447,15cyclictest9804-21awk07:36:580
31105999446,17cyclictest12134-21unixbench_singl11:57:080
31105999441,31cyclictest7077-21taskset09:07:020
31105999441,31cyclictest7077-21taskset09:07:020
31105999244,16cyclictest1-21systemd10:06:220
31105999239,21cyclictest27086-21grep08:31:440
31105999235,17cyclictest27067-21rm10:35:060
31105999234,33cyclictest0-21swapper/010:46:260
31105999136,32cyclictest1737-21grep08:51:550
31105999136,32cyclictest1737-21grep08:51:550
31105999066,9cyclictest10665-21apt-get07:41:350
31105999066,9cyclictest10665-21apt-get07:41:350
31105999043,17cyclictest10861-21/usr/sbin/munin11:01:480
31105999039,17cyclictest17878-21wc08:01:570
31118998969,13cyclictest32667-21grep07:11:222
31105998958,20cyclictest0-21swapper/007:06:260
31105998944,35cyclictest4165-21/usr/sbin/munin07:21:560
31105998943,14cyclictest2644-21/usr/sbin/munin07:16:430
31105998940,20cyclictest5269-21grep11:46:440
31105998940,17cyclictest5638-21diskmemload12:21:370
31105998933,19cyclictest0-21swapper/012:13:360
31105998837,19cyclictest27663-21if_err_eth009:41:510
31105998837,19cyclictest27663-21if_err_eth009:41:510
31105998837,18cyclictest6518-21cat07:26:580
31105998834,29cyclictest5638-21diskmemload12:07:430
31105998833,17cyclictest32150-21sh10:42:300
31105998831,30cyclictest17938-21ssh10:19:460
31105998767,11cyclictest3876-21apt-get09:56:340
31105998767,11cyclictest3876-21apt-get09:56:340
31105998741,15cyclictest12852-21wc07:46:540
31105998739,17cyclictest24388-21ssh09:36:420
31105998734,20cyclictest27290-21rm11:29:200
31105998647,31cyclictest19563-21ssh11:16:410
31105998647,31cyclictest19563-21ssh11:16:410
31105998641,15cyclictest2098-21ssh09:52:260
31105998640,16cyclictest415-21snmpd10:52:510
31105998639,16cyclictest0-21swapper/009:48:410
31105998637,19cyclictest10062-21sh11:01:120
31118998557,18cyclictest23441-21head11:22:012
31105998541,35cyclictest19308-21cat11:16:260
31105998539,14cyclictest31629-21systemd-run12:31:240
31105998538,16cyclictest20271-21rm09:29:510
31105998538,16cyclictest20271-21rm09:29:510
31105998536,14cyclictest0-21swapper/007:06:420
31105998534,29cyclictest13192-21cat10:11:450
31105998534,14cyclictest0-21swapper/012:16:440
253792850,17sleep3401ktimers/308:26:403
31105998447,28cyclictest30988-21cron11:36:220
31105998441,35cyclictest7607-21cut07:31:450
31105998439,11cyclictest18850-21ssh10:21:340
31105998436,17cyclictest15397-21ssh12:02:540
31105998346,29cyclictest32707-21unixbench_singl11:37:390
31105998336,16cyclictest18780-21taskset08:06:360
31105998335,19cyclictest30986-21expr08:41:560
31105998334,18cyclictest16945-21latency_hist08:01:240
31105998245,29cyclictest9158-21df_inode09:11:420
31105998239,34cyclictest23646-21/usr/sbin/munin08:21:590
31105998235,19cyclictest25359-21cstates08:26:390
31118998147,12cyclictest52-21kcompactd007:56:572
31118998056,17cyclictest3914-21idleruntime-cro07:21:222
31112998055,15cyclictest3549-21munin-plugin-st09:56:221
31105998056,16cyclictest15431-21grep07:56:250
31105998056,16cyclictest15431-21grep07:56:250
31105998038,36cyclictest28513-21open_files10:36:540
31105998033,16cyclictest22347-21cat08:16:460
31105997934,18cyclictest10186-21fschecks_time10:06:450
31123997861,10cyclictest31583-21munin-plugin-st08:46:233
31118997735,23cyclictest141rcu_preempt09:22:032
31118997735,23cyclictest141rcu_preempt09:22:032
3111299772,50cyclictest1479-21/usr/sbin/munin10:46:491
31105997749,22cyclictest21932-21latency_hist08:16:240
31105997749,18cyclictest21623-21/usr/sbin/munin09:31:590
31105997728,20cyclictest2011-21sh07:13:230
31105997636,33cyclictest429-21ssh12:31:580
31123997536,28cyclictest0-21swapper/311:36:223
31105997532,14cyclictest0-21swapper/008:56:490
31105997531,16cyclictest0-21swapper/008:36:590
31123997456,10cyclictest19887-21unixbench_singl10:21:583
3112399742,59cyclictest21824-21ssh10:26:333
31118997446,8cyclictest141rcu_preempt12:16:342
31105997456,13cyclictest8150irq/69-2188000.11:41:370
305082749,38sleep230598-21unixbench_singl07:02:102
31118997357,10cyclictest19873-21unixbench_multi10:21:582
31118997350,11cyclictest0-21swapper/208:11:222
3112399723,61cyclictest0-21swapper/312:06:373
31118997224,28cyclictest0-21swapper/211:11:022
3112399715,54cyclictest22976-21apt-get12:16:343
31123997133,29cyclictest0-21swapper/311:51:243
31105997145,16cyclictest32425-21/usr/sbin/munin08:46:550
3112399702,57cyclictest10456-21apt-get11:01:343
2984427011,29sleep30-21swapper/307:01:503
31123996912,52cyclictest21596-21ssh09:31:593
31118996940,19cyclictest28818-21latency11:31:492
31118996916,26cyclictest0-21swapper/210:11:552
3112399685,54cyclictest1966-21apt-get11:41:373
3112399683,55cyclictest8752-21modprobe07:36:233
3112399682,57cyclictest30875-21rm10:41:353
31123996811,48cyclictest0-21swapper/308:31:583
31118996846,11cyclictest0-21swapper/212:06:442
31118996833,12cyclictest141rcu_preempt12:33:422
31118996826,12cyclictest180-21pr/ttyS007:51:342
31118996826,12cyclictest180-21pr/ttyS007:51:342
3112399675,52cyclictest7508-21ssh10:56:383
31123996745,12cyclictest0-21swapper/308:01:223
3112399673,57cyclictest16909-21unixbench_singl10:16:593
3112399673,56cyclictest2629-21grep10:47:093
3112399673,55cyclictest13996-21apt-get12:01:373
3112399673,54cyclictest27306-21apt-get09:41:403
3112399673,54cyclictest27306-21apt-get09:41:403
3112399673,54cyclictest25232-21grep10:31:563
3112399672,56cyclictest13136-21timerwakeupswit09:17:103
31118996740,18cyclictest10831-21timerwakeupswit10:06:582
31118996714,28cyclictest16045-21df_inode10:16:432
3112399663,55cyclictest1811-21unixbench_multi07:11:583
3112399663,54cyclictest4615-21mailstats09:56:553
3112399663,54cyclictest4615-21mailstats09:56:553
3112399663,54cyclictest25351-21unixbench_singl09:37:023
3112399663,53cyclictest22373-21/usr/sbin/munin11:22:013
31123996626,30cyclictest0-21swapper/310:06:253
3112399662,55cyclictest18590-21latency_hist08:06:253
3112399662,55cyclictest1237-21fschecks_time08:51:433
3112399662,55cyclictest1237-21fschecks_time08:51:433
3112399662,54cyclictest2116-21cron12:36:223
31123996622,36cyclictest0-21swapper/309:16:233
31118996644,12cyclictest22764-21runrttasks12:16:232
31112996646,11cyclictest13128-21idleruntime-cro11:06:221
3112399654,54cyclictest0-21swapper/309:26:363
3112399654,54cyclictest0-21swapper/309:26:363
3112399653,55cyclictest32734-21grep11:37:393
3112399653,53cyclictest19242-21ssh11:16:253
3112399653,53cyclictest10106-21sort10:06:443
3112399653,41cyclictest4643-21kernelversion09:01:463
3112399652,55cyclictest31580-21modprobe12:31:223
3112399652,54cyclictest8847-21cat09:11:253
3112399652,54cyclictest8847-21cat09:11:253
3112399652,54cyclictest10839-21sort07:41:433
3112399652,54cyclictest10839-21sort07:41:433
3112399652,53cyclictest16328-21wc11:11:273
31123996525,33cyclictest0-21swapper/307:06:463
3112399652,52cyclictest31845-21apt-get08:46:343
31123996516,40cyclictest20164-21sh11:16:563
31123996516,40cyclictest20164-21sh11:16:563
31118996543,14cyclictest180-21pr/ttyS008:31:572
31118996533,10cyclictest0-21swapper/211:19:512
31118996533,10cyclictest0-21swapper/211:19:512
31118996532,11cyclictest141rcu_preempt10:40:192
31118996528,13cyclictest552-21in:imklog07:16:262
3111299653,47cyclictest31426-21apt-get11:36:451
31112996515,24cyclictest141rcu_preempt08:46:251
3112399645,48cyclictest20305-21latency_hist08:11:253
3112399643,54cyclictest0-21swapper/310:11:353
3112399643,53cyclictest0-21swapper/309:21:453
3112399643,53cyclictest0-21swapper/309:21:453
3112399643,52cyclictest6717-21unixbench_multi07:26:593
3112399643,52cyclictest3508-21sed07:17:023
3112399643,52cyclictest24281-21find08:22:013
3112399643,51cyclictest10819-21latency_hist11:56:253
31123996426,29cyclictest0-21swapper/310:51:593
3112399642,54cyclictest14794-21grep07:51:573
3112399642,54cyclictest14794-21grep07:51:573
3112399642,54cyclictest12054-21grep11:57:073
3112399642,50cyclictest397-21runrttasks08:57:113
3112399642,10cyclictest28585-21apt-get08:36:343
31123996420,34cyclictest0-21swapper/309:51:273
3111899648,36cyclictest20832-21apt-get09:31:332
31118996442,14cyclictest5638-21diskmemload09:21:252
31118996442,12cyclictest0-21swapper/212:02:032
31118996436,10cyclictest180-21pr/ttyS011:41:262
31118996419,10cyclictest0-21swapper/211:37:252
31118996413,25cyclictest0-21swapper/208:51:332
31118996413,25cyclictest0-21swapper/208:51:332
31112996443,14cyclictest10585-21munin-plugin-st11:56:221
302172647,35sleep130262-21processes07:02:041
3112399633,52cyclictest26666-21find12:21:563
3112399633,52cyclictest22930-21tail08:16:593
3112399633,51cyclictest30602-21tune2fs09:46:453
3112399633,51cyclictest20592-21fschecks_count08:11:453
3112399633,50cyclictest12263-21/usr/sbin/munin07:46:343
3112399632,51cyclictest25908-21ssh11:26:503
31118996333,9cyclictest141rcu_preempt11:26:442
3111299633,22cyclictest3607-21sed08:56:571
3112399624,50cyclictest20367-21kernelversion12:11:493
3112399622,53cyclictest4698-21awk07:21:573
3111899629,34cyclictest25151-21ssh10:31:532
31118996229,10cyclictest141rcu_preempt10:29:592
31118996215,29cyclictest0-21swapper/209:51:442
31112996240,15cyclictest22639-21sendmail12:16:211
31112996239,12cyclictest0-21swapper/111:06:591
31112996238,14cyclictest278-21systemd-journal07:16:431
3111299623,26cyclictest5638-21diskmemload11:46:041
3112399614,47cyclictest28079-21/usr/sbin/munin10:36:443
3112399613,49cyclictest9544-21ntpq07:36:513
3111899616,30cyclictest0-21swapper/208:21:412
3111899614,31cyclictest0-21swapper/208:41:582
31118996140,11cyclictest0-21swapper/210:06:222
31118996139,13cyclictest26997-21kworker/u8:311:12:062
31118996137,15cyclictest0-21swapper/210:51:592
31118996137,13cyclictest0-21swapper/209:11:222
31118996137,13cyclictest0-21swapper/209:11:222
31118996135,10cyclictest141rcu_preempt11:51:542
31118996124,18cyclictest0-21swapper/212:26:232
31118996115,26cyclictest0-21swapper/211:56:462
31112996144,12cyclictest4675-21ntp_kernel_pll_07:21:561
31112996136,16cyclictest26094-21runrttasks08:26:571
31118996042,9cyclictest0-21swapper/209:06:232
31118996036,8cyclictest141rcu_preempt09:40:432
31118996012,24cyclictest0-21swapper/211:47:142
31112996038,12cyclictest0-21swapper/107:56:581
31112996034,16cyclictest0-21swapper/109:22:161
31112996034,16cyclictest0-21swapper/109:22:161
31118995936,12cyclictest0-21swapper/207:31:242
31118995935,17cyclictest31095-21cyclictest09:56:562
31118995935,17cyclictest31095-21cyclictest09:56:562
31118995933,14cyclictest0-21swapper/209:42:042
31118995933,14cyclictest0-21swapper/209:42:042
3111899593,20cyclictest141rcu_preempt11:06:242
31118995931,18cyclictest9235-21cut09:11:442
31112995941,11cyclictest22092-21kworker/u8:111:22:011
31112995940,12cyclictest552-21in:imklog10:19:511
31112995933,14cyclictest0-21swapper/107:46:581
3111299593,21cyclictest9639-21ps09:11:551
3111899586,13cyclictest0-21swapper/210:56:382
31118995842,11cyclictest1-21systemd09:31:232
31118995842,11cyclictest1-21systemd09:31:232
31118995832,16cyclictest9328-21fschecks_time07:36:462
31118995822,13cyclictest141rcu_preempt10:46:562
31118995814,19cyclictest141rcu_preempt09:01:252
31118995810,27cyclictest0-21swapper/209:46:312
31112995842,10cyclictest31948-21unixbench_singl10:41:591
31112995840,10cyclictest552-21in:imklog10:25:381
31112995839,13cyclictest7954-21unixbench_singl10:01:581
3111299582,35cyclictest9943-21grep07:36:591
31112995738,13cyclictest26228-21fschecks_time12:21:461
31112995738,12cyclictest0-21swapper/110:56:251
31112995735,12cyclictest0-21swapper/110:32:101
31112995730,17cyclictest15738-21ssh10:16:251
3111899563,27cyclictest141rcu_preempt07:31:452
31118995625,9cyclictest0-21swapper/210:41:292
31118995622,14cyclictest141rcu_preempt12:31:242
31118995612,11cyclictest141rcu_preempt07:26:262
3111299564,22cyclictest32660-21idleruntime-cro07:11:221
31112995637,13cyclictest8821-21latency_hist09:11:251
31112995637,13cyclictest8821-21latency_hist09:11:251
31112995633,14cyclictest4678-21latency09:01:461
3111299562,16cyclictest278-21systemd-journal08:02:001
31112995611,21cyclictest141rcu_preempt07:11:451
3111899554,22cyclictest0-21swapper/208:36:362
31118995533,11cyclictest0-21swapper/208:46:502
31112995536,12cyclictest0-21swapper/107:46:221
31112995536,12cyclictest0-21swapper/107:46:221
31112995535,14cyclictest29426-21unixbench_singl11:32:001
31112995535,13cyclictest30824-21sh09:46:511
31112995534,12cyclictest0-21swapper/111:11:391
3111299553,17cyclictest10127-21fschecks_time10:06:451
3111299552,30cyclictest18996-21fschecks_time08:06:441
3111299552,17cyclictest12432-21fschecks_count09:16:571
31118995433,10cyclictest31095-21cyclictest07:41:392
31118995433,10cyclictest31095-21cyclictest07:41:392
31118995420,16cyclictest33-21ksoftirqd/208:16:452
31112995410,27cyclictest141rcu_preempt09:27:001
31112995410,27cyclictest141rcu_preempt09:27:001
3111899536,8cyclictest141rcu_preempt08:11:452
31118995312,13cyclictest141rcu_preempt08:26:442
3111299536,24cyclictest141rcu_preempt11:46:571
3111299534,9cyclictest141rcu_preempt12:01:031
31112995336,10cyclictest31704-21sh12:31:251
31112995334,10cyclictest0-21swapper/110:31:221
31112995330,14cyclictest180-21pr/ttyS007:56:251
31112995330,14cyclictest180-21pr/ttyS007:56:251
3111299532,21cyclictest14406-21/usr/sbin/munin12:01:511
3111299532,15cyclictest1-21systemd12:11:221
3111899524,21cyclictest141rcu_preempt08:01:462
31112995234,11cyclictest24049-21unixbench_singl12:16:581
3111299523,20cyclictest24134-21latency_hist09:36:291
31112995231,13cyclictest0-21swapper/108:16:221
31112995229,12cyclictest0-21swapper/108:51:441
31112995229,12cyclictest0-21swapper/108:51:441
31112995228,13cyclictest0-21swapper/109:46:241
31112995228,13cyclictest0-21swapper/109:46:241
3111299522,24cyclictest141rcu_preempt11:19:421
3111299522,24cyclictest141rcu_preempt11:19:421
3111299522,21cyclictest32622-21ssh12:31:551
31112995211,12cyclictest141rcu_preempt11:31:251
3111299513,22cyclictest0-21swapper/107:31:411
31118995026,12cyclictest0-21swapper/207:46:442
31112995034,8cyclictest0-21swapper/110:39:031
31112995023,15cyclictest0-21swapper/108:22:041
3111299502,18cyclictest9956-21sh11:00:541
3111299502,16cyclictest27843-21grep08:31:581
31112995020,7cyclictest141rcu_preempt09:56:441
31112995020,7cyclictest141rcu_preempt09:56:441
3111299493,23cyclictest0-21swapper/108:37:071
31112994928,11cyclictest0-21swapper/108:51:241
3111299487,8cyclictest6330-21ntp_states07:26:541
31112994827,11cyclictest0-21swapper/109:36:471
31112994826,12cyclictest0-21swapper/108:16:451
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional