You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-01 - 19:11

ARM Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7 (updated Fri Jul 01, 2022 12:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1137321100,4chrt11375-21ssh10:00:530
32628210919,44sleep00-21swapper/007:00:310
311210617,56sleep2325-21grep07:00:372
16639910056,34cyclictest7452-21munin-run12:35:122
1645999984,9cyclictest32141-21rsync11:28:190
1671999641,9cyclictest0-21swapper/309:05:353
1671999641,9cyclictest0-21swapper/309:05:353
1671999424,15cyclictest0-21swapper/309:50:473
1671999141,10cyclictest27158-21latency08:20:373
1671999141,10cyclictest27158-21latency08:20:373
1671999029,8cyclictest0-21swapper/308:00:133
1671999029,8cyclictest0-21swapper/308:00:133
1663999047,35cyclictest31341-21unixbench_multi11:25:532
166399892,44cyclictest10049-21apt-get11:45:372
1651998954,18cyclictest0-21swapper/111:28:371
164599883,51cyclictest24866-21/usr/sbin/munin11:15:480
164599883,51cyclictest24866-21/usr/sbin/munin11:15:480
1645998746,24cyclictest9667-21apt-get11:45:230
12722875,14sleep10-21swapper/107:02:191
167199852,50cyclictest32394-21ssh11:28:363
167199843,15cyclictest31321-21latency_hist10:35:153
166399842,13cyclictest27521-21grep11:20:342
1671998328,10cyclictest0-21swapper/308:35:383
167199831,9cyclictest141rcu_preempt10:40:473
1663998141,31cyclictest0-21swapper/210:35:132
166399813,21cyclictest18554-21/usr/sbin/munin07:55:362
166399813,21cyclictest18554-21/usr/sbin/munin07:55:362
1663998120,51cyclictest0-21swapper/210:05:242
1645998113,50cyclictest18798-21apt-get12:00:270
1645998113,47cyclictest27589-21sh09:35:240
1645998062,11cyclictest5577-21apt-key08:55:150
166399792,13cyclictest16514-21wc11:55:382
35492780,8sleep1241ktimers/107:10:211
166399783,18cyclictest31351-21wc09:40:442
1663997831,14cyclictest0-21swapper/210:05:152
167199772,64cyclictest25178-21sshd08:15:463
166399766,10cyclictest0-21swapper/209:31:292
166399763,19cyclictest4080-21cut09:50:162
166399763,12cyclictest15564-21fschecks_time07:45:342
1663997613,45cyclictest141rcu_preempt11:00:552
164599769,19cyclictest0-21swapper/012:20:330
166399759,59cyclictest15561-21sh09:15:362
166399752,19cyclictest8237-21sed09:55:472
166399752,19cyclictest8237-21sed09:55:472
1645997537,23cyclictest392-21runrttasks08:46:240
167199743,49cyclictest13687-21/usr/sbin/munin07:40:323
167199743,49cyclictest13687-21/usr/sbin/munin07:40:323
1671997422,29cyclictest41-21ksoftirqd/309:10:453
1671997422,29cyclictest41-21ksoftirqd/309:10:453
164599743,49cyclictest15995-21apt-get10:10:250
1645997419,16cyclictest0-21swapper/009:10:340
1645997419,16cyclictest0-21swapper/009:10:340
166399739,15cyclictest0-21swapper/209:10:472
166399739,15cyclictest0-21swapper/209:10:472
166399736,19cyclictest23976-21ssh11:15:162
166399734,18cyclictest19418-21ssh12:01:032
166399733,19cyclictest23635-21ssh10:21:082
166399733,19cyclictest23635-21ssh10:21:082
166399733,19cyclictest13216-21fschecks_count11:50:342
166399733,13cyclictest5464-21cat07:15:162
166399733,12cyclictest0-21swapper/208:45:522
166399732,13cyclictest12981-21grep07:35:472
166399725,14cyclictest31351-21cat08:35:132
166399725,14cyclictest31351-21cat08:35:132
166399724,17cyclictest5783-21fschecks_time07:15:342
166399724,17cyclictest5005-21unixbench_multi08:50:452
166399723,20cyclictest9686-21tail07:25:512
166399723,17cyclictest2877-21processes10:40:442
166399723,15cyclictest28658-21tail09:36:062
166399723,13cyclictest26155-21grep12:10:462
166399722,20cyclictest3581-21idleruntime-cro11:35:132
166399722,19cyclictest24927-21latency_hist12:10:152
166399722,18cyclictest17581-21fschecks_count11:05:342
166399722,15cyclictest22714-21unixbench_multi08:05:472
166399722,15cyclictest22714-21unixbench_multi08:05:472
166399722,14cyclictest27646-21tail08:20:462
166399722,14cyclictest27646-21tail08:20:462
1663997218,13cyclictest4356-21cut11:35:382
1663997210,15cyclictest18989-21proc_pri09:20:442
164599726,33cyclictest16196-21ssh11:55:300
1645997217,23cyclictest23895-21sed11:15:150
167199713,47cyclictest23972-21sshd11:15:163
167199712,40cyclictest5831-21apt-get08:55:253
166399713,61cyclictest0-21swapper/210:40:152
166399713,19cyclictest26146-21grep08:16:392
166399713,16cyclictest17128-21cut07:50:332
166399713,16cyclictest11909-21latency_hist09:10:152
166399713,16cyclictest11909-21latency_hist09:10:152
166399713,14cyclictest2316-21latency_hist08:45:152
166399713,13cyclictest8109-21unixbench_singl07:20:462
166399712,18cyclictest7701-21grep09:00:352
166399712,17cyclictest31847-21sendmail_mailqu12:20:422
166399712,17cyclictest24786-21latency_hist08:15:132
166399712,16cyclictest11941-21/usr/sbin/munin10:55:412
166399712,13cyclictest2011-21grep07:05:352
164599717,16cyclictest0-21swapper/012:05:220
1645997147,15cyclictest8799-21diskmemload10:45:440
1645997141,20cyclictest0-21swapper/011:45:160
1645997141,20cyclictest0-21swapper/011:45:160
1645997125,15cyclictest28542-21apt-get10:30:230
1645997117,20cyclictest6301-21latency08:56:200
1645997114,29cyclictest19216-21timerandwakeup09:20:470
1671997045,16cyclictest8974-21latency_hist09:05:163
1671997037,11cyclictest141rcu_preempt10:16:413
166399709,10cyclictest0-21swapper/212:15:252
166399704,14cyclictest6275-21grep10:45:472
166399703,20cyclictest6819-21unixbench_singl08:56:302
166399703,19cyclictest392-21runrttasks12:30:152
166399703,17cyclictest11329-21grep07:30:462
166399703,17cyclictest11329-21grep07:30:462
166399703,16cyclictest9662-21cron11:45:132
166399703,16cyclictest9662-21cron11:45:132
166399703,16cyclictest28208-21cron10:30:122
166399703,15cyclictest22142-21apt-get10:20:152
1663997028,35cyclictest0-21swapper/208:25:392
1663997028,34cyclictest0-21swapper/207:40:342
1663997028,34cyclictest0-21swapper/207:40:342
166399702,18cyclictest7022-21latency_hist09:55:152
1645997010,18cyclictest0-21swapper/009:25:350
1645997010,18cyclictest0-21swapper/009:25:350
1671996934,13cyclictest141rcu_preempt09:25:293
1671996934,13cyclictest141rcu_preempt09:25:293
166399693,61cyclictest0-21swapper/211:16:082
166399693,61cyclictest0-21swapper/211:16:082
166399693,15cyclictest31870-21tune2fs08:35:352
166399693,13cyclictest21096-21grep08:00:472
166399693,13cyclictest21096-21grep08:00:472
166399692,14cyclictest17664-21ssh10:11:232
1651996952,11cyclictest28224-21apt10:30:121
1645996949,12cyclictest131ktimers/010:05:340
1645996945,14cyclictest14140-21ssh11:00:150
1645996921,26cyclictest7940-21timerandwakeup07:20:450
1645996917,23cyclictest67-21kswapd008:15:500
3122226810,10sleep131310-21sh12:20:281
1671996840,20cyclictest52-21kcompactd007:55:163
1671996834,17cyclictest141rcu_preempt08:35:153
1671996834,17cyclictest141rcu_preempt08:35:153
166399683,16cyclictest24171-21rm09:30:152
166399683,16cyclictest24171-21rm09:30:152
1645996842,14cyclictest8150irq/69-2188000.10:15:240
1645996829,12cyclictest0-21swapper/011:05:130
1645996816,21cyclictest0-21swapper/008:40:150
1645996811,21cyclictest28764-21chrt12:15:410
1671996744,16cyclictest8495-21idleruntime-cro07:25:133
1671996742,15cyclictest0-21swapper/307:15:463
166399673,13cyclictest8170-21taskset10:50:162
1651996752,7cyclictest0-21swapper/112:10:131
1645996743,13cyclictest1108-21apt-get09:45:270
1645996738,12cyclictest9450irq/72-mmc211:30:270
1645996732,26cyclictest16934-21apt-get07:50:250
1645996718,16cyclictest2343-21/usr/sbin/munin12:25:410
1645996711,33cyclictest24476-21sh10:23:160
1645996711,33cyclictest24476-21sh10:23:160
1671996651,8cyclictest0-21swapper/308:30:133
1671996649,10cyclictest3188-21basename07:10:133
1671996643,17cyclictest11258-21grep10:00:473
1671996617,24cyclictest1093-21idleruntime08:40:343
1645996642,14cyclictest0-21swapper/011:55:150
1645996616,19cyclictest8206-21/usr/sbin/munin09:55:470
1645996616,19cyclictest8206-21/usr/sbin/munin09:55:470
1645996611,17cyclictest0-21swapper/009:05:150
92826512,21sleep30-21swapper/307:00:553
1671996546,13cyclictest12691-21munin-plugin-st11:50:133
1671996541,16cyclictest12143-21threads10:55:443
207602640,10sleep1241ktimers/111:10:341
1671996426,11cyclictest141rcu_preempt11:39:233
1645996438,16cyclictest0-21swapper/007:35:130
1645996438,16cyclictest0-21swapper/007:35:130
1671996347,10cyclictest7980-21tail11:40:453
1671996347,10cyclictest7980-21tail11:40:453
1671996345,13cyclictest24793-21sendmail08:15:133
167199632,41cyclictest16391-21apt-get10:10:253
1651996340,14cyclictest2727-21unixbench_multi12:25:451
1651996340,11cyclictest0-21swapper/107:55:151
1645996315,20cyclictest4140-21acpi09:50:240
1645996314,18cyclictest12714-21sort07:35:450
1645996313,20cyclictest6859-21cron07:20:130
105092630,9sleep0161rcuc/009:06:360
105092630,9sleep0161rcuc/009:06:360
167199625,24cyclictest16031-21apt-config10:10:143
1671996235,17cyclictest0-21swapper/310:00:133
1671996235,17cyclictest0-21swapper/310:00:133
1651996248,8cyclictest8799-21diskmemload11:15:481
1651996248,8cyclictest8799-21diskmemload11:15:481
1651996241,11cyclictest0-21swapper/110:00:341
1651996234,9cyclictest141rcu_preempt12:35:121
1651996234,13cyclictest241ktimers/110:50:271
1651996222,23cyclictest141rcu_preempt11:55:461
1651996210,33cyclictest3583-21munin-run11:35:131
164599626,20cyclictest20102-21runrttasks11:10:130
164599626,16cyclictest0-21swapper/008:00:430
164599626,16cyclictest0-21swapper/008:00:430
1645996232,10cyclictest0-21swapper/011:35:440
1645996212,17cyclictest9720-21awk07:25:510
1645996212,16cyclictest976-21sed09:45:160
1671996144,11cyclictest5457-21modprobe08:55:133
1671996142,10cyclictest14011-21unixbench_multi11:50:473
1651996141,10cyclictest0-21swapper/107:45:471
1651996140,14cyclictest611-21in:imklog11:01:181
1651996138,12cyclictest0-21swapper/107:55:421
1651996138,12cyclictest0-21swapper/107:55:421
1651996137,9cyclictest141rcu_preempt09:05:241
1651996137,9cyclictest141rcu_preempt09:05:241
1651996114,27cyclictest25-21ksoftirqd/110:34:491
1651996112,30cyclictest13974-21fschecks_time07:40:411
1651996112,30cyclictest13974-21fschecks_time07:40:411
1645996134,16cyclictest0-21swapper/009:35:120
1645996127,9cyclictest0-21swapper/011:23:150
1645996113,19cyclictest28444-21cat08:25:290
249052600,8chrt0-21swapper/109:30:441
167199608,31cyclictest3619-21modprobe11:35:133
1671996042,11cyclictest611-21in:imklog07:35:293
1671996040,12cyclictest180-21pr/ttyS010:30:153
1671996038,15cyclictest611-21in:imklog12:15:363
1645996030,21cyclictest8799-21diskmemload09:15:260
1645996011,18cyclictest19197-21ps07:55:530
1645996011,18cyclictest19197-21ps07:55:530
1645996010,13cyclictest31771-21diskstats10:35:310
165199593,35cyclictest10049-21apt-get11:45:231
1651995926,18cyclictest0-21swapper/109:45:431
1651995924,13cyclictest141rcu_preempt10:48:341
1645995935,16cyclictest25498-21/usr/sbin/munin10:25:360
1645995916,14cyclictest0-21swapper/010:54:280
1645995911,19cyclictest5078-21cut10:45:150
1671995842,10cyclictest276-21systemd-journal12:20:353
165199587,27cyclictest5464-21cron08:55:131
1651995842,9cyclictest0-21swapper/109:50:471
165199584,13cyclictest141rcu_preempt08:40:131
1651995841,10cyclictest14121-21latency_hist11:00:141
1651995837,11cyclictest0-21swapper/111:20:471
1651995815,23cyclictest25-21ksoftirqd/111:35:351
1651995814,12cyclictest141rcu_preempt11:55:141
164599585,25cyclictest0-21swapper/007:05:470
1645995837,12cyclictest0-21swapper/008:20:460
1645995837,12cyclictest0-21swapper/008:20:460
1645995811,18cyclictest30898-21sed08:30:450
1645995811,18cyclictest30898-21sed08:30:450
1671995733,14cyclictest0-21swapper/312:10:463
1651995741,8cyclictest0-21swapper/112:20:121
1651995741,8cyclictest0-21swapper/110:35:341
1651995731,17cyclictest15602-21fschecks_time09:15:371
164599574,16cyclictest24782-21apt-get08:15:170
1645995734,13cyclictest0-21swapper/008:05:480
1645995734,13cyclictest0-21swapper/008:05:480
1645995713,15cyclictest5484-21mailstats12:30:430
167199568,11cyclictest141rcu_preempt09:45:463
167199564,27cyclictest41-21ksoftirqd/312:29:303
1671995639,10cyclictest0-21swapper/307:25:513
1671995638,12cyclictest10971-21processes07:30:423
1671995638,12cyclictest10971-21processes07:30:423
1671995636,10cyclictest0-21swapper/312:30:463
1671995618,12cyclictest141rcu_preempt08:48:513
1651995640,8cyclictest0-21swapper/111:05:401
1651995639,11cyclictest18775-21latency09:20:391
1651995638,11cyclictest0-21swapper/107:40:131
164599563,21cyclictest410-21snmpd07:10:200
167199554,38cyclictest24127-21ssh11:15:193
167199554,38cyclictest24127-21ssh11:15:193
1671995539,10cyclictest276-21systemd-journal09:38:433
1671995530,14cyclictest0-21swapper/309:20:153
1671995523,12cyclictest41-21ksoftirqd/311:06:583
167199552,16cyclictest32650-21grep10:35:483
165199555,13cyclictest22185-21kworker/u8:110:45:151
1651995536,13cyclictest0-21swapper/110:05:381
1651995534,9cyclictest0-21swapper/109:00:391
1651995534,11cyclictest0-21swapper/108:40:331
1651995521,12cyclictest141rcu_preempt07:20:161
165199545,29cyclictest25-21ksoftirqd/108:05:481
165199545,29cyclictest25-21ksoftirqd/108:05:481
165199543,33cyclictest19197-21apt-get12:00:551
165199543,33cyclictest16612-21ssh10:10:591
1651995432,11cyclictest0-21swapper/109:25:341
1651995432,11cyclictest0-21swapper/109:25:341
1651995422,17cyclictest25-21ksoftirqd/111:41:311
1651995422,17cyclictest25-21ksoftirqd/111:41:311
165199542,14cyclictest22858-21ssh10:20:551
165199542,14cyclictest22858-21ssh10:20:551
1651995420,13cyclictest25-21ksoftirqd/109:40:351
1651995418,15cyclictest25-21ksoftirqd/112:10:421
1651995411,26cyclictest141rcu_preempt09:55:421
1651995411,26cyclictest141rcu_preempt09:55:421
1671995337,7cyclictest0-21swapper/311:55:473
1671995336,11cyclictest611-21in:imklog10:21:363
1671995336,11cyclictest611-21in:imklog10:21:363
1671995331,12cyclictest0-21swapper/312:05:153
1671995310,21cyclictest41-21ksoftirqd/308:05:343
1671995310,21cyclictest41-21ksoftirqd/308:05:343
1671995310,16cyclictest29965-21latency_hist11:25:123
165199536,25cyclictest0-21swapper/107:10:161
1651995338,8cyclictest0-21swapper/108:15:131
165199533,39cyclictest1952-21kworker/u8:208:45:271
1645995326,19cyclictest15521-21fschecks_count07:45:340
164599532,13cyclictest25429-21fschecks_time12:10:330
167199528,12cyclictest41-21ksoftirqd/309:20:393
1671995232,11cyclictest117-21pr/ttymxc011:01:513
1671995231,11cyclictest0-21swapper/310:50:463
1651995237,8cyclictest0-21swapper/108:35:151
1651995237,8cyclictest0-21swapper/108:35:151
165199523,23cyclictest27183-21apt-get09:35:181
1651995232,12cyclictest0-21swapper/108:20:121
164599529,16cyclictest2382-21latency_hist08:45:160
1645995219,18cyclictest0-21swapper/007:40:270
1645995219,18cyclictest0-21swapper/007:40:270
167199513,13cyclictest3450-21ls07:10:193
1671995115,18cyclictest141rcu_preempt10:45:353
165199516,15cyclictest25-21ksoftirqd/107:25:511
165199514,35cyclictest3434-21munin-node08:55:221
1651995134,10cyclictest0-21swapper/107:35:131
1651995134,10cyclictest0-21swapper/107:35:131
1671995036,9cyclictest25503-21unixbench_multi09:30:543
1671995030,9cyclictest0-21swapper/309:40:463
165199504,23cyclictest25-21ksoftirqd/109:10:291
165199504,23cyclictest25-21ksoftirqd/109:10:291
1651995035,9cyclictest276-21systemd-journal08:20:291
1651995035,9cyclictest276-21systemd-journal08:20:291
1651995018,19cyclictest141rcu_preempt08:02:421
1651995018,19cyclictest141rcu_preempt08:02:421
1651995014,9cyclictest141rcu_preempt10:20:121
167199495,12cyclictest0-21swapper/308:00:473
167199495,12cyclictest0-21swapper/308:00:473
1671994934,8cyclictest0-21swapper/312:05:333
1651994728,9cyclictest0-21swapper/108:30:131
165199463,16cyclictest7390-21fschecks_time07:20:331
1671994521,12cyclictest0-21swapper/307:45:433
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional