You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-05-28 - 22:35

ARM Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7 (updated Sun May 28, 2023 12:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
287252986,15sleep10-21swapper/107:03:131
29175999475,11cyclictest0-21swapper/312:32:193
29175999475,11cyclictest0-21swapper/312:32:193
29166999229,10cyclictest0-21swapper/208:12:312
29175999133,33cyclictest3793-21taskset09:06:243
2836929015,34sleep00-21swapper/007:02:350
29166998929,11cyclictest19594-21ssh11:22:102
29166998818,16cyclictest0-21swapper/210:07:312
2830728833,31sleep30-21swapper/307:02:343
29166998627,12cyclictest0-21swapper/209:47:102
29166998627,11cyclictest0-21swapper/211:56:522
29153998355,19cyclictest2230-21munin-plugin-st09:01:491
29166998130,16cyclictest0-21swapper/209:12:162
29166998056,16cyclictest29497-21grep11:37:322
29145998032,25cyclictest6658-21ssh10:05:500
29145998025,19cyclictest19050-21cpuspeed09:32:110
29166997957,15cyclictest916-21sendmail-mta07:57:162
29145997926,15cyclictest16491-21cstates10:22:100
29145997926,15cyclictest16491-21cstates10:22:100
29145997915,30cyclictest11733-21ssh12:02:280
29166997857,14cyclictest5188-21grep09:07:292
2916699783,15cyclictest22268-21/usr/sbin/munin10:32:262
29153997856,14cyclictest16820-21fschecks_count11:17:141
29145997852,16cyclictest22785-21ssh12:22:160
29145997852,16cyclictest22785-21ssh12:22:160
29145997849,21cyclictest18495-21apt-get08:12:090
29175997725,18cyclictest14109-21vmstat09:22:343
29166997712,44cyclictest1767-21find10:52:082
29145997728,20cyclictest5018-21fschecks_count10:57:200
29145997725,18cyclictest0-21swapper/012:27:220
29175997625,22cyclictest10217-21taskset12:01:253
29166997656,13cyclictest13276-21idleruntime-cro11:11:492
29145997631,15cyclictest7693-21ntp_states09:12:260
29145997626,20cyclictest12145-21rm09:20:270
29145997622,20cyclictest22449-21taskset11:26:580
29145997622,20cyclictest22449-21taskset11:26:580
29145997529,18cyclictest18283-21rm09:30:560
29145997524,20cyclictest31634-21grep08:52:100
29145997523,18cyclictest1066-21modprobe09:56:490
29145997519,18cyclictest13742-21ssh11:12:120
29145997431,17cyclictest22178-21idleruntime-cro11:26:490
29145997423,20cyclictest8578-21date07:41:510
29145997419,23cyclictest14055-21/usr/sbin/munin10:17:310
29153997311,11cyclictest3856-21diskmemload09:52:151
29145997327,18cyclictest12721-21rm10:16:110
29145997325,19cyclictest24633-21rm10:36:100
29145997325,14cyclictest32082-21cat10:47:300
29145997324,27cyclictest23206-21runrttasks09:37:510
29145997324,27cyclictest23206-21runrttasks09:37:510
29175997247,17cyclictest31888-21timerandwakeup09:52:423
29175997225,20cyclictest4148-21sh11:50:553
2915399723,41cyclictest1966-21df_inode10:52:171
29145997224,38cyclictest10430-21chrt12:01:510
29145997223,17cyclictest5453-21latency_hist07:31:520
287462725,45sleep20-21swapper/207:03:302
149522720,5sleep10-21swapper/110:18:461
29175997120,17cyclictest26070-21ssh11:32:263
29175997119,19cyclictest337-21dbus-daemon12:06:503
29166997123,41cyclictest13541-21cstates10:17:102
29166997111,42cyclictest30893-21/usr/sbin/munin09:52:042
29145997129,11cyclictest3272-21ssh09:59:360
29145997125,17cyclictest19614-21ssh12:17:100
29145997122,18cyclictest0-21swapper/011:20:480
29145997120,18cyclictest2976-21sed07:22:270
29145997120,18cyclictest2976-21sed07:22:270
29175997053,11cyclictest16878-21timerandwakeup09:27:293
29153997051,13cyclictest23070-21unixbench_singl09:37:341
29153997051,13cyclictest23070-21unixbench_singl09:37:341
29145997044,18cyclictest0-21swapper/011:45:500
29145997044,18cyclictest0-21swapper/011:45:500
29145997033,29cyclictest13623-21/usr/sbin/munin07:57:220
29145997027,13cyclictest28657-21cstates12:32:180
29145997027,13cyclictest28657-21cstates12:32:180
29175996953,10cyclictest5232-21unixbench_singl09:07:303
29175996921,20cyclictest30179-21sh09:50:583
2916699695,29cyclictest0-21swapper/212:22:072
2916699695,29cyclictest0-21swapper/212:22:072
29166996945,17cyclictest27556-21idleruntime-cro09:46:492
29166996943,16cyclictest15724-21latency_hist09:26:512
29145996922,19cyclictest20513-21ssh10:27:450
29175996822,19cyclictest3856-21diskmemload12:13:073
29175996817,15cyclictest0-21swapper/310:32:063
29166996819,44cyclictest0-21swapper/212:32:012
29166996819,44cyclictest0-21swapper/212:32:012
29153996851,12cyclictest23574-21unixbench_singl12:22:301
29153996851,12cyclictest23574-21unixbench_singl12:22:301
29153996844,13cyclictest28245-21latency_hist11:36:521
29145996847,14cyclictest8298-21sh10:07:310
29175996742,18cyclictest4132-21latency_hist10:01:523
29175996722,16cyclictest3856-21diskmemload11:38:043
29175996719,17cyclictest3856-21diskmemload10:02:133
29175996717,19cyclictest10745-21mailstats09:17:313
29166996743,16cyclictest31679-21systemd-journal12:29:162
29166996741,16cyclictest0-21swapper/207:41:492
29166996734,14cyclictest122-21pr/ttymxc011:12:272
29145996745,17cyclictest21946-21fschecks_count08:22:150
29145996730,29cyclictest6061-21runrttasks11:53:010
29175996619,19cyclictest21308-21taskset08:19:463
29175996619,18cyclictest9058-21sh11:03:143
29175996617,17cyclictest20195-21ssh11:22:273
29175996613,16cyclictest7022-21latency_hist10:06:523
29166996648,10cyclictest0-21swapper/211:42:292
29166996648,10cyclictest0-21swapper/211:42:292
29153996624,10cyclictest141rcu_preempt12:30:521
29145996625,32cyclictest0-21swapper/011:02:310
29145996624,15cyclictest29498-21sh09:48:550
29145996623,35cyclictest829-21tune2fs07:17:160
29145996619,19cyclictest30483-21rm10:45:540
29145996619,14cyclictest0-21swapper/012:12:310
29166996547,11cyclictest32200-21runrttasks10:47:312
29153996545,11cyclictest29716-21latency_hist08:46:521
29153996540,17cyclictest32751-21idleruntime-cro07:16:501
29145996538,20cyclictest25261-21apt-get10:37:080
29145996538,20cyclictest24962-21runrttasks09:42:080
29145996524,34cyclictest3859-21modprobe09:06:490
29145996520,17cyclictest2919-21users10:52:430
29145996520,10cyclictest25454-21apt-get11:32:090
29145996519,20cyclictest6539-21sh09:11:190
29175996417,19cyclictest15254-21sh12:08:573
29175996417,19cyclictest15254-21sh12:08:573
29175996416,18cyclictest493-21apt-get08:56:553
2916699647,43cyclictest400-21runrttasks08:57:102
29166996446,12cyclictest122-21pr/ttymxc008:02:322
29166996445,11cyclictest362-21in:imklog11:06:522
29166996445,10cyclictest0-21swapper/209:57:332
29153996444,14cyclictest4481-21munin-plugin-st11:51:491
29153996443,15cyclictest26009-21head09:42:291
29153996443,13cyclictest0-21swapper/112:16:511
2915399642,34cyclictest25-21ksoftirqd/110:23:271
2915399642,34cyclictest25-21ksoftirqd/110:23:271
29145996446,12cyclictest17232-21kernelversion08:07:190
29145996424,16cyclictest16463-21latency_hist12:11:520
29145996424,16cyclictest16463-21latency_hist12:11:520
29145996416,20cyclictest29214-21switchtime11:37:290
29175996338,18cyclictest9418-21tr07:42:293
29175996338,18cyclictest9418-21tr07:42:293
29175996334,17cyclictest7284-21cut09:12:163
29175996330,23cyclictest10845-21ssh11:07:143
29175996323,15cyclictest11631-21chrt10:13:053
29166996335,13cyclictest31679-21systemd-journal12:17:012
29153996347,10cyclictest20426-21unixbench_singl10:27:381
29153996344,13cyclictest11702-21ssh11:07:321
29145996337,18cyclictest11416-21/usr/sbin/munin11:07:290
29145996330,24cyclictest24343-21unixbench_singl08:27:310
29145996325,14cyclictest30695-21modprobe07:11:500
29175996238,15cyclictest3856-21diskmemload11:01:523
29175996219,16cyclictest26144-21diskmemload09:42:483
29166996239,14cyclictest0-21swapper/208:26:492
29166996238,16cyclictest362-21in:imklog10:02:282
29166996237,16cyclictest0-21swapper/207:07:242
29153996240,16cyclictest19892-21cut12:17:181
29153996223,17cyclictest141rcu_preempt12:32:471
29153996223,17cyclictest141rcu_preempt12:32:471
29145996235,19cyclictest13412-21latency_hist07:56:520
29145996234,20cyclictest16430-21chrt08:04:470
29145996216,22cyclictest0-21swapper/011:46:530
29145996215,16cyclictest0-21swapper/008:17:050
29175996137,15cyclictest31371-21diskstats10:47:153
29175996136,18cyclictest10384-21apt-get07:47:053
29175996134,19cyclictest24993-21apt-get08:32:163
29175996134,19cyclictest14041-21/usr/sbin/munin10:17:303
29175996133,19cyclictest24329-21unixbench_singl08:27:313
29175996133,19cyclictest20350-21ssh09:33:193
29175996118,18cyclictest23822-21ssh11:28:103
29175996118,18cyclictest23822-21ssh11:28:103
29166996146,9cyclictest24717-21idleruntime-cro08:31:492
29166996142,12cyclictest6863-21sendmail07:36:502
29166996142,12cyclictest6863-21sendmail07:36:502
29166996141,13cyclictest0-21swapper/211:21:522
29166996139,15cyclictest31679-21systemd-journal07:51:522
29166996128,10cyclictest122-21pr/ttymxc011:32:322
29153996145,10cyclictest6864-21latency_hist07:36:501
29153996145,10cyclictest6864-21latency_hist07:36:501
29153996138,14cyclictest3836-21kworker/u8:309:16:491
29153996138,14cyclictest0-21swapper/110:16:531
2915399613,49cyclictest30505-21grep09:51:491
2915399613,43cyclictest3544-21apt-get07:27:001
29153996131,17cyclictest52-21kcompactd010:47:131
29145996131,19cyclictest29664-21sh08:46:500
29175996041,10cyclictest19340-21apt-get10:27:123
29175996037,15cyclictest26112-21timerandwakeup10:37:303
29175996036,18cyclictest7395-21munin-plugin-st11:56:493
29175996036,16cyclictest13621-21apt-get11:12:083
29175996035,18cyclictest28093-21/usr/sbin/munin12:31:493
29175996025,26cyclictest29881-21taskset07:07:203
29175996015,19cyclictest400-21runrttasks07:35:273
29175996015,19cyclictest400-21runrttasks07:35:273
2916699605,27cyclictest0-21swapper/209:06:522
29166996040,12cyclictest0-21swapper/208:36:502
29166996036,15cyclictest31679-21systemd-journal11:47:152
2916699602,30cyclictest22004-21apt-get09:37:082
2916699602,30cyclictest22004-21apt-get09:37:082
29166996021,19cyclictest0-21swapper/207:27:342
29153996046,8cyclictest14080-21unixbench_singl09:22:321
29153996042,12cyclictest10458-21latency_hist12:01:511
2915399603,25cyclictest3856-21diskmemload09:06:491
29153996027,13cyclictest0-21swapper/110:35:201
29153996025,15cyclictest141rcu_preempt12:02:221
29153996024,17cyclictest141rcu_preempt11:27:461
29153996024,17cyclictest141rcu_preempt11:27:461
2915399602,31cyclictest25-21ksoftirqd/107:17:211
29153996014,18cyclictest0-21swapper/110:41:511
29145996038,15cyclictest400-21runrttasks09:25:160
29175995944,9cyclictest31550-21apt-get08:52:053
29175995942,9cyclictest28238-21apt-get10:42:143
29175995936,17cyclictest13342-21apt07:56:503
29175995931,20cyclictest31366-21apt-get11:42:053
29175995931,20cyclictest31366-21apt-get11:42:053
29166995938,15cyclictest362-21in:imklog10:37:302
29166995937,12cyclictest0-21swapper/209:27:302
29166995934,14cyclictest0-21swapper/212:07:202
29166995934,14cyclictest0-21swapper/212:07:202
29166995923,13cyclictest141rcu_preempt08:07:152
29153995941,9cyclictest0-21swapper/111:56:491
2915399593,34cyclictest28738-21fschecks_time11:37:181
29153995927,17cyclictest141rcu_preempt10:01:411
2915399592,33cyclictest3856-21diskmemload10:42:331
29153995911,8cyclictest231rcuc/108:02:261
29145995934,15cyclictest8953-21fschecks_count07:42:180
29145995934,15cyclictest8953-21fschecks_count07:42:180
29145995929,19cyclictest31551-21/usr/sbin/munin07:12:130
29145995921,12cyclictest0-21swapper/008:57:260
29175995832,18cyclictest337-21dbus-daemon07:26:513
29175995832,18cyclictest337-21dbus-daemon07:26:513
29175995832,18cyclictest30780-21cat08:47:423
29175995831,17cyclictest13813-21/usr/sbin/munin07:57:153
29175995830,18cyclictest17327-21/usr/sbin/munin08:07:203
29175995823,26cyclictest585-21apt-get07:17:063
29175995818,13cyclictest0-21swapper/308:38:213
29175995813,16cyclictest19237-21ssh11:21:493
29175995813,15cyclictest418-21snmpd10:56:253
29166995839,13cyclictest11376-21tail10:12:342
29166995832,14cyclictest0-21swapper/209:21:522
2916699583,11cyclictest141rcu_preempt07:21:502
29166995830,8cyclictest141rcu_preempt12:13:472
29166995830,8cyclictest0-21swapper/211:01:522
29166995827,13cyclictest141rcu_preempt11:59:402
29153995843,8cyclictest0-21swapper/110:57:211
29153995838,12cyclictest0-21swapper/108:56:501
29153995826,19cyclictest52-21kcompactd007:22:191
29153995826,19cyclictest52-21kcompactd007:22:191
29153995826,15cyclictest141rcu_preempt11:24:271
29153995825,7cyclictest141rcu_preempt09:07:001
29145995839,10cyclictest24970-21/usr/sbin/munin08:32:450
29175995733,16cyclictest400-21runrttasks07:31:523
29175995733,16cyclictest31210-21tune2fs07:11:593
29175995728,20cyclictest18479-21basename08:12:043
29175995724,24cyclictest28549-21fschecks_time08:42:193
29175995714,16cyclictest18181-21ssh10:24:343
29175995714,16cyclictest18181-21ssh10:24:343
29166995736,13cyclictest13518-21kworker/u8:311:31:522
29166995736,13cyclictest13518-21kworker/u8:311:31:522
2916699573,13cyclictest0-21swapper/210:27:382
29153995728,14cyclictest141rcu_preempt09:27:531
29153995726,15cyclictest141rcu_preempt08:32:271
29153995724,6cyclictest141rcu_preempt10:03:301
29145995743,7cyclictest26704-21apt-get08:37:130
29145995735,16cyclictest6415-21grep07:32:270
29145995735,16cyclictest6415-21grep07:32:270
29145995732,17cyclictest10558-21grep07:47:130
29175995638,11cyclictest21984-21ssh09:37:083
29175995638,11cyclictest21984-21ssh09:37:083
29175995631,19cyclictest22634-21rm12:22:113
29175995631,19cyclictest22634-21rm12:22:113
29175995631,19cyclictest19729-21cstates12:17:133
29175995624,22cyclictest16620-21idleruntime-cro08:06:493
29175995617,31cyclictest21996-21fschecks_time08:22:163
29166995639,10cyclictest16038-21kworker/u8:007:16:502
2915399569,23cyclictest25-21ksoftirqd/112:09:491
2915399569,23cyclictest25-21ksoftirqd/112:09:491
2915399565,13cyclictest0-21swapper/108:01:521
29153995629,12cyclictest141rcu_preempt09:17:291
29145995632,17cyclictest30180-21fschecks_time08:47:280
29166995538,11cyclictest31679-21systemd-journal12:04:582
29166995530,13cyclictest0-21swapper/207:42:322
29166995530,13cyclictest0-21swapper/207:42:322
29166995518,13cyclictest362-21in:imklog09:35:042
29153995536,10cyclictest31658-21cut11:42:151
29153995536,10cyclictest31658-21cut11:42:151
2915399553,14cyclictest31277-21sh08:51:491
29153995529,12cyclictest141rcu_preempt08:07:211
29153995523,16cyclictest141rcu_preempt11:03:491
29153995512,21cyclictest241ktimers/109:32:521
29166995434,9cyclictest0-21swapper/208:17:292
29166995434,10cyclictest0-21swapper/208:42:182
2916699543,13cyclictest30603-21grep08:47:402
2915399544,24cyclictest25-21ksoftirqd/108:22:311
2915399544,24cyclictest25-21ksoftirqd/107:51:501
29153995435,12cyclictest0-21swapper/111:16:491
29153995431,11cyclictest0-21swapper/107:41:491
29153995428,13cyclictest141rcu_preempt08:41:511
29153995426,13cyclictest141rcu_preempt08:20:061
29175995328,17cyclictest7341-21fschecks_count07:37:203
29166995333,13cyclictest16767-21ssh10:22:182
29166995333,13cyclictest16767-21ssh10:22:182
29166995319,11cyclictest141rcu_preempt08:52:162
29153995322,15cyclictest141rcu_preempt10:10:421
29153995310,11cyclictest25-21ksoftirqd/107:42:301
29153995310,11cyclictest25-21ksoftirqd/107:42:301
29153995228,12cyclictest141rcu_preempt08:27:161
29153995225,14cyclictest141rcu_preempt07:07:181
29153995221,16cyclictest141rcu_preempt07:52:101
29166995124,16cyclictest0-21swapper/208:37:392
29166995121,12cyclictest141rcu_preempt10:42:332
29153995114,16cyclictest25-21ksoftirqd/108:12:251
29166994929,12cyclictest31679-21systemd-journal07:52:142
29166994629,8cyclictest0-21swapper/207:22:162
29166994629,8cyclictest0-21swapper/207:22:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional