You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-05-27 - 18:20

Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rackbslot7 (updated Fri May 27, 2022 12:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25660213614,92sleep10-21swapper/107:03:361
265769911569,16cyclictest29390-21/usr/sbin/munin09:52:430
265769911046,32cyclictest21862-21/usr/sbin/munin10:33:050
265769910648,22cyclictest6357-21cat11:52:350
265769910648,22cyclictest6357-21cat11:52:350
265769910643,15cyclictest13901-21/usr/sbin/munin09:28:110
265769910643,15cyclictest13901-21/usr/sbin/munin09:28:110
26411210613,66sleep026423-21modprobe07:07:340
265769910544,35cyclictest28756-21sed07:12:450
265769910356,18cyclictest10354-21runrttasks09:22:120
265769910145,21cyclictest10307-21/usr/sbin/munin11:57:590
265769910143,23cyclictest14210-21/usr/sbin/munin11:12:560
265769910137,19cyclictest13775-21rm11:12:160
265769910039,25cyclictest19460-21ssh09:36:450
38632990,15sleep33862-21proc_pri10:03:033
38632990,15sleep33862-21proc_pri10:03:033
26576999944,23cyclictest28506-21grep12:27:550
26576999941,27cyclictest32070-21ssh10:48:060
26576999941,27cyclictest32070-21ssh10:48:060
26576999860,19cyclictest12853-21apt-get12:02:460
26576999848,30cyclictest29374-21ssh10:43:370
26576999842,33cyclictest17296-21ssh11:17:390
26576999751,20cyclictest29430-21ssh11:37:330
26576999750,27cyclictest392-21runrttasks07:38:450
26576999741,18cyclictest1865-21grep07:28:050
26576999741,18cyclictest1865-21grep07:28:050
26576999740,22cyclictest20699-21/usr/sbin/munin11:22:540
26576999734,45cyclictest0-21swapper/011:01:520
26576999649,18cyclictest10092-21ssh11:05:410
26576999649,18cyclictest10092-21ssh11:05:410
26576999647,18cyclictest25848-21xargs10:38:060
26576999643,21cyclictest11056-21sort09:22:560
26576999546,22cyclictest11940-21grep07:58:120
26576999544,19cyclictest16334-21/usr/sbin/munin12:07:560
26576999541,20cyclictest19956-21/usr/sbin/munin12:13:070
26576999528,49cyclictest16625-21switchtime10:23:260
2531729522,35sleep20-21swapper/207:03:252
26597999450,23cyclictest141rcu_preempt12:18:033
26576999442,31cyclictest7717-21sh10:09:000
26576999442,22cyclictest30224-21cat11:37:590
26576999438,17cyclictest309-21rpcbind12:18:050
26576999432,22cyclictest1858-21cat09:07:570
26576999426,48cyclictest0-21swapper/008:27:570
26576999348,16cyclictest20160-21grep08:23:070
26576999339,18cyclictest20175-21grep09:37:490
26576999339,17cyclictest26512-21/usr/sbin/munin09:47:520
26576999337,16cyclictest14404-21rm10:20:150
26576999243,19cyclictest339-21/usr/sbin/munin07:23:060
26576999242,30cyclictest3645-21rm10:55:020
26576999242,20cyclictest17451-21latency_hist08:17:360
26576999242,20cyclictest17451-21latency_hist08:17:360
26576999242,18cyclictest3417-21/usr/sbin/munin10:02:540
26576999242,18cyclictest3417-21/usr/sbin/munin10:02:540
26576999241,17cyclictest6764-21/usr/sbin/munin11:52:530
26576999232,19cyclictest21550-21modprobe10:32:320
26576999142,20cyclictest32641-21ssh09:57:550
26576999142,20cyclictest32641-21ssh09:57:550
26576999142,20cyclictest22484-21/usr/sbin/munin08:32:440
26576999141,21cyclictest3521-21/usr/sbin/munin07:33:050
26576999140,21cyclictest30905-21fschecks_time08:57:570
26576999139,31cyclictest25647-21sh11:30:530
26576999139,31cyclictest25647-21sh11:30:530
26576999041,19cyclictest7836-21/usr/sbin/munin07:47:510
26576999041,18cyclictest25248-21sh12:22:450
26576999040,20cyclictest27283-21runrttasks07:08:010
26576999039,17cyclictest0-21swapper/008:47:520
26576999037,29cyclictest32235-21/usr/sbin/munin09:02:540
26576998946,13cyclictest30926-21/usr/sbin/munin07:18:000
26576998940,19cyclictest3039-21ssh11:47:190
26576998940,19cyclictest3039-21ssh11:47:190
26576998938,21cyclictest13603-21/usr/sbin/munin08:03:240
26576998937,20cyclictest15652-21cron08:12:340
26576998933,28cyclictest24935-21sort08:38:060
26576998932,17cyclictest0-21swapper/012:35:100
26576998932,15cyclictest0-21swapper/009:47:310
26576998932,15cyclictest0-21swapper/009:47:310
26576998839,21cyclictest10987-21rm10:14:280
26576998838,30cyclictest6280-21ssh09:14:390
26576998837,17cyclictest0-21swapper/008:18:140
26576998837,16cyclictest9341-21/usr/sbin/munin07:52:560
26576998837,16cyclictest9341-21/usr/sbin/munin07:52:560
26576998831,18cyclictest7515-21systemd-run07:47:350
26582998744,37cyclictest4233-21sendmail07:37:331
26576998734,22cyclictest30179-21taskset08:55:440
2659799862,62cyclictest15656-21kworker/u8:008:17:503
26597998622,24cyclictest0-21swapper/311:49:573
26597998622,24cyclictest0-21swapper/311:49:573
2658299863,20cyclictest27377-21apt-get08:47:461
26576998637,18cyclictest26190-21latency08:42:590
26582998522,53cyclictest0-21swapper/111:02:331
2658299843,20cyclictest23198-21switchtime08:33:051
2658299842,14cyclictest3687-21ssh11:47:541
2658299842,14cyclictest3687-21ssh11:47:541
26597998361,15cyclictest27118-21munin-run08:47:333
26597998331,25cyclictest141rcu_preempt12:24:263
2658299833,18cyclictest26721-21ssh11:32:501
26597998244,9cyclictest141rcu_preempt11:03:453
26597998244,9cyclictest141rcu_preempt11:03:453
26590998234,12cyclictest141rcu_preempt11:23:492
26597998121,32cyclictest611-21in:imklog11:26:173
26597998042,13cyclictest141rcu_preempt09:34:353
26590997939,10cyclictest141rcu_preempt07:54:252
26590997939,10cyclictest141rcu_preempt07:54:252
26590997936,23cyclictest141rcu_preempt09:16:222
2658299792,13cyclictest14235-21ssh11:12:461
2659099783,45cyclictest17806-21df08:17:522
2658299783,18cyclictest1190-21ssh09:07:331
26597997747,21cyclictest23049-21latency_hist09:42:363
26597997728,27cyclictest141rcu_preempt08:57:523
2658299777,10cyclictest0-21swapper/110:33:231
78552760,5sleep30-21swapper/310:09:223
26597997630,25cyclictest141rcu_preempt11:45:153
26597997630,25cyclictest141rcu_preempt11:45:153
26590997654,14cyclictest2558-21kworker/u8:208:02:362
26590997633,13cyclictest141rcu_preempt08:03:272
2658299764,15cyclictest16846-21sed10:23:271
2658299763,16cyclictest545-21head07:23:071
2658299763,16cyclictest12171-21apt10:17:331
2658299763,15cyclictest392-21runrttasks07:07:481
2658299763,14cyclictest27406-21chrt12:25:581
2658299762,15cyclictest10732-21idleruntime-cro07:57:341
2658299762,15cyclictest10732-21idleruntime-cro07:57:341
26597997535,8cyclictest141rcu_preempt11:33:413
26597997528,27cyclictest141rcu_preempt10:15:493
26590997532,19cyclictest611-21in:imklog07:34:002
2658299754,14cyclictest14048-21sh08:07:331
26582997539,24cyclictest28583-21cstates12:27:561
2658299753,63cyclictest21854-21grep08:28:081
2658299753,11cyclictest0-21swapper/107:48:001
26582997528,37cyclictest0-21swapper/109:02:331
2658299752,16cyclictest10990-21/usr/sbin/munin07:58:101
2658299752,15cyclictest32179-21cron09:57:331
2658299752,15cyclictest23505-21/usr/sbin/munin11:27:541
2658299752,15cyclictest23505-21/usr/sbin/munin11:27:541
2659799746,54cyclictest0-21swapper/312:27:493
26597997440,13cyclictest141rcu_preempt07:07:543
26597997429,23cyclictest117-21pr/ttymxc012:14:533
26597997417,19cyclictest41-21ksoftirqd/310:48:073
26597997417,19cyclictest41-21ksoftirqd/310:48:073
26590997438,15cyclictest141rcu_preempt10:18:122
26590997437,18cyclictest141rcu_preempt09:10:592
26590997432,24cyclictest141rcu_preempt11:36:412
26590997428,9cyclictest141rcu_preempt07:08:032
26590997426,24cyclictest0-21swapper/209:28:002
26590997426,24cyclictest0-21swapper/209:28:002
2658299744,60cyclictest29731-21head07:13:081
2658299744,60cyclictest10803-21sh11:07:331
2658299744,60cyclictest10803-21sh11:07:331
2658299743,9cyclictest21831-21awk12:17:331
2658299743,63cyclictest16940-21unixbench_multi12:08:061
2658299743,18cyclictest18086-21unixbench_singl09:33:111
2658299743,17cyclictest1608-21rm10:52:351
2658299743,17cyclictest1608-21rm10:52:351
2658299743,16cyclictest28375-21cut10:42:551
2658299743,14cyclictest7292-21/usr/sbin/munin11:53:091
2658299743,13cyclictest26721-21grep08:43:091
2658299743,13cyclictest19492-21cut08:22:551
2658299742,13cyclictest14955-21mailstats08:08:041
2658299742,11cyclictest22469-21ssh12:17:541
26597997342,18cyclictest141rcu_preempt10:54:413
26597997331,18cyclictest611-21in:imklog12:10:433
26597997329,24cyclictest141rcu_preempt11:09:023
26597997329,19cyclictest141rcu_preempt08:36:323
26590997319,27cyclictest0-21swapper/210:32:412
2658299736,60cyclictest0-21swapper/111:07:571
2658299734,14cyclictest5458-21xargs09:13:071
2658299734,13cyclictest11911-21unixbench_singl09:23:101
2658299733,62cyclictest13145-21tune2fs12:02:541
2658299733,59cyclictest24249-21grep09:43:081
2658299733,59cyclictest24249-21grep09:43:081
2658299733,17cyclictest17302-21runrttasks08:17:331
2658299733,17cyclictest17302-21runrttasks08:17:331
2658299733,16cyclictest10801-21unixbench_singl11:58:071
2658299733,15cyclictest29914-21sed08:53:071
2658299733,11cyclictest0-21swapper/112:32:511
2658299733,10cyclictest19212-21kernelversion10:27:571
26582997327,34cyclictest0-21swapper/109:57:591
26582997327,34cyclictest0-21swapper/109:57:591
2658299732,12cyclictest7735-21ssh09:17:451
26597997237,16cyclictest22455-21ssh10:33:003
26597997235,21cyclictest141rcu_preempt10:38:073
26597997231,19cyclictest141rcu_preempt09:57:593
26597997231,19cyclictest141rcu_preempt09:57:593
2659099728,38cyclictest15843-21ssh10:23:082
26590997255,11cyclictest28587-21fschecks_time07:12:432
26590997239,9cyclictest141rcu_preempt09:03:072
26590997237,17cyclictest141rcu_preempt09:58:352
26590997237,17cyclictest141rcu_preempt09:58:352
26590997237,10cyclictest141rcu_preempt08:32:502
26590997228,23cyclictest611-21in:imklog11:55:522
26590997223,11cyclictest141rcu_preempt07:19:292
2658299724,60cyclictest24313-21grep08:37:541
2658299724,59cyclictest2107-21unixbench_multi07:28:081
2658299724,59cyclictest2107-21unixbench_multi07:28:081
2658299724,55cyclictest3220-21ssh10:02:461
2658299724,55cyclictest3220-21ssh10:02:461
2658299723,58cyclictest17582-21grep11:17:541
2658299723,19cyclictest27779-21apt10:42:331
2658299723,16cyclictest2118-21cut10:52:541
2658299723,16cyclictest19020-21modprobe08:22:331
2658299723,12cyclictest6574-21tune2fs10:07:531
2658299723,12cyclictest27312-21ssh09:48:081
26597997128,15cyclictest141rcu_preempt08:07:563
26597997127,21cyclictest0-21swapper/309:50:363
26597997125,22cyclictest611-21in:imklog07:17:353
26590997137,8cyclictest141rcu_preempt10:29:212
26590997133,18cyclictest141rcu_preempt11:47:592
26590997133,18cyclictest141rcu_preempt11:47:592
26590997131,9cyclictest141rcu_preempt11:31:502
26590997131,9cyclictest141rcu_preempt11:31:502
26590997130,13cyclictest141rcu_preempt11:41:092
26590997124,12cyclictest0-21swapper/211:15:212
2658299714,59cyclictest3138-21latency_hist11:47:351
2658299714,59cyclictest3138-21latency_hist11:47:351
2658299714,18cyclictest32512-21idleruntime-cro11:42:331
2658299713,16cyclictest20409-21/usr/sbin/munin11:22:581
2658299713,15cyclictest13901-21/usr/sbin/munin09:28:071
2658299713,15cyclictest13901-21/usr/sbin/munin09:28:071
2658299713,13cyclictest15292-21apt-get10:22:341
2658299713,12cyclictest1556-21grep09:07:501
26597997044,8cyclictest141rcu_preempt09:24:353
26597997035,12cyclictest141rcu_preempt10:43:413
26597997033,18cyclictest117-21pr/ttymxc007:27:363
26597997029,19cyclictest141rcu_preempt11:41:093
26597997026,30cyclictest141rcu_preempt09:15:213
26590997040,8cyclictest141rcu_preempt10:40:232
26590997039,11cyclictest141rcu_preempt08:14:332
26590997039,11cyclictest141rcu_preempt08:14:332
26590997020,19cyclictest141rcu_preempt09:36:042
2658299707,7cyclictest0-21swapper/107:37:501
2658299703,58cyclictest6356-21grep07:42:551
2658299703,58cyclictest20582-21ssh09:37:581
26597996954,8cyclictest0-21swapper/311:52:533
26597996947,14cyclictest24186-21proc_pri11:28:043
26597996947,14cyclictest24186-21proc_pri11:28:043
26597996944,9cyclictest141rcu_preempt12:01:433
26597996933,15cyclictest180-21pr/ttyS009:53:153
26597996929,21cyclictest141rcu_preempt11:17:553
26597996927,27cyclictest141rcu_preempt09:03:013
26597996923,22cyclictest0-21swapper/307:27:593
26597996923,22cyclictest0-21swapper/307:27:593
26597996913,23cyclictest41-21ksoftirqd/310:28:053
26590996939,16cyclictest141rcu_preempt10:10:472
26590996935,8cyclictest141rcu_preempt12:13:032
26590996931,21cyclictest52-21kcompactd012:06:032
26590996927,13cyclictest141rcu_preempt10:17:272
26590996924,18cyclictest141rcu_preempt10:04:172
26590996924,18cyclictest141rcu_preempt10:04:172
26590996916,30cyclictest0-21swapper/209:20:262
26597996839,7cyclictest141rcu_preempt12:06:583
26597996831,22cyclictest141rcu_preempt09:29:523
26597996831,22cyclictest141rcu_preempt09:29:523
26597996810,34cyclictest15669-21ssh10:22:453
26590996829,9cyclictest141rcu_preempt07:32:022
26590996829,9cyclictest141rcu_preempt07:32:022
26590996828,9cyclictest141rcu_preempt10:57:382
26590996826,23cyclictest0-21swapper/207:42:332
26597996734,16cyclictest141rcu_preempt08:37:563
26597996734,14cyclictest141rcu_preempt12:33:343
26597996729,19cyclictest611-21in:imklog09:20:583
26597996726,9cyclictest141rcu_preempt08:27:143
26597996725,11cyclictest141rcu_preempt11:15:113
26597996716,23cyclictest41-21ksoftirqd/309:42:553
26597996716,23cyclictest41-21ksoftirqd/309:42:553
26590996734,18cyclictest141rcu_preempt10:55:252
26590996732,23cyclictest141rcu_preempt08:28:022
26590996721,18cyclictest141rcu_preempt09:23:222
26590996715,25cyclictest0-21swapper/211:09:552
26590996712,29cyclictest17340-21apt-get11:17:482
2658299672,18cyclictest31161-21grep07:18:061
26597996622,18cyclictest0-21swapper/307:47:583
26590996636,16cyclictest141rcu_preempt09:45:432
26590996636,16cyclictest141rcu_preempt09:45:432
26590996635,8cyclictest141rcu_preempt07:27:022
26590996634,8cyclictest141rcu_preempt11:06:392
26590996634,8cyclictest141rcu_preempt11:06:392
26590996632,7cyclictest141rcu_preempt12:20:412
26590996621,7cyclictest0-21swapper/208:42:572
26590996621,10cyclictest0-21swapper/212:07:492
26590996620,8cyclictest141rcu_preempt11:47:242
26590996620,8cyclictest141rcu_preempt11:47:242
26597996540,12cyclictest141rcu_preempt08:02:473
26597996539,7cyclictest141rcu_preempt11:00:493
26590996539,14cyclictest141rcu_preempt12:02:282
26590996534,8cyclictest141rcu_preempt09:52:572
26590996530,21cyclictest180-21pr/ttyS009:51:442
26590996529,8cyclictest141rcu_preempt12:27:402
26590996529,8cyclictest141rcu_preempt12:24:192
26590996526,13cyclictest141rcu_preempt10:48:512
26590996526,13cyclictest141rcu_preempt10:48:512
26597996438,17cyclictest11144-21runrttasks07:57:553
26597996436,11cyclictest141rcu_preempt07:47:363
26597996429,8cyclictest141rcu_preempt07:55:033
26597996429,8cyclictest141rcu_preempt07:55:033
26597996428,21cyclictest141rcu_preempt07:37:593
26597996424,13cyclictest141rcu_preempt09:07:513
26590996436,7cyclictest141rcu_preempt09:37:502
26597996334,11cyclictest141rcu_preempt07:32:453
26597996318,25cyclictest0-21swapper/307:17:443
26590996326,11cyclictest141rcu_preempt10:45:242
250802639,30sleep324735-21apt-get07:02:453
26597996229,9cyclictest141rcu_preempt10:17:563
26597996226,18cyclictest141rcu_preempt08:13:043
26597996226,18cyclictest141rcu_preempt08:13:043
26597996213,29cyclictest41-21ksoftirqd/308:52:343
26590996232,8cyclictest141rcu_preempt07:49:472
26590996228,17cyclictest141rcu_preempt07:43:032
26590996225,19cyclictest611-21in:imklog08:22:522
26590996223,10cyclictest141rcu_preempt08:57:362
26590996128,8cyclictest141rcu_preempt08:07:552
26590996120,9cyclictest141rcu_preempt12:32:532
26590996111,21cyclictest33-21ksoftirqd/208:37:502
26597996038,8cyclictest141rcu_preempt08:32:283
26590996032,7cyclictest141rcu_preempt08:57:532
26590996025,9cyclictest141rcu_preempt08:47:542
26597995715,9cyclictest401ktimers/308:57:363
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional