You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-28 - 14:00

ARM Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7 (updated Tue Mar 28, 2023 00:44:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2288029126,31sleep00-21swapper/018:58:080
24349998663,14cyclictest11877-21idleruntime-cro20:02:530
2285528611,37sleep10-21swapper/118:58:051
24363998332,43cyclictest24520-21kworker/u8:122:07:541
24363998320,13cyclictest0-21swapper/120:12:531
24363998221,12cyclictest31268-21cron23:52:531
2438199819,59cyclictest23842-21diskstats22:43:223
24373998148,27cyclictest11534-21unixbench_multi00:13:292
24363998125,9cyclictest0-21swapper/121:13:261
24363998122,12cyclictest0-21swapper/122:48:211
24363998121,8cyclictest0-21swapper/100:03:361
2434999813,43cyclictest20307-21/usr/sbin/munin20:28:140
24363998023,49cyclictest0-21swapper/119:28:361
24363998023,49cyclictest0-21swapper/119:28:361
24381997954,15cyclictest25888-21idleruntime-cro19:07:543
24363997925,7cyclictest0-21swapper/119:13:281
24363997921,10cyclictest0-21swapper/123:32:531
24363997921,10cyclictest0-21swapper/123:32:531
2438199783,66cyclictest31355-21munin-run23:52:543
24373997856,15cyclictest17406-21sendmail22:32:532
24373997856,15cyclictest17406-21sendmail22:32:532
24363997826,7cyclictest0-21swapper/123:38:191
2436399774,21cyclictest28157-21cron20:52:541
2436399774,19cyclictest32331-21grep23:53:331
2436399774,18cyclictest28706-21fschecks_time20:53:151
24363997728,41cyclictest0-21swapper/121:22:531
24363997714,10cyclictest0-21swapper/120:33:381
24349997748,18cyclictest2674-21latency19:33:230
2437399764,9cyclictest25991-21apt-get23:43:192
2437399764,9cyclictest25991-21apt-get23:43:192
2436399764,19cyclictest11053-21fschecks_count21:23:161
2436399763,20cyclictest3182-21grep19:33:331
24363997628,8cyclictest0-21swapper/121:28:301
2437399753,14cyclictest20326-21apt-get20:28:072
2436399758,47cyclictest17798-21rm22:33:051
24363997552,14cyclictest8841-21idleruntime-cro22:17:541
24363997525,10cyclictest0-21swapper/121:58:261
24363997515,10cyclictest0-21swapper/100:28:261
2437399743,62cyclictest32434-21ps21:03:272
2437399743,62cyclictest32434-21ps21:03:272
2436399744,19cyclictest12339-21fschecks_count20:03:171
24363997432,30cyclictest0-21swapper/119:38:111
2436399743,22cyclictest20547-21fschecks_count20:28:221
2436399743,21cyclictest11442-21unixbench_multi19:58:291
24363997423,7cyclictest0-21swapper/122:57:561
24363997423,7cyclictest0-21swapper/122:57:561
2436399742,21cyclictest24887-21kernelversion19:03:181
2436399742,20cyclictest16908-21latency_hist23:27:561
24363997417,12cyclictest8534-21acpi23:13:051
24381997351,15cyclictest9059-21latency_hist22:17:573
2437399733,6cyclictest0-21swapper/221:08:092
2437399733,62cyclictest24375-21latency_hist19:02:582
2436399733,19cyclictest15069-21cut22:28:151
2436399733,19cyclictest15069-21cut22:28:151
24349997336,13cyclictest141rcu_preempt00:13:300
2437399722,15cyclictest32339-21apt-get22:58:062
2436399723,21cyclictest22833-21switchtime21:43:291
24363997227,7cyclictest0-21swapper/121:48:161
24363997224,39cyclictest0-21swapper/122:43:071
24363997223,8cyclictest0-21swapper/121:09:081
24363997223,7cyclictest0-21swapper/121:42:561
24363997219,11cyclictest0-21swapper/123:08:481
24363997219,11cyclictest0-21swapper/122:12:561
24349997253,11cyclictest11074-21apt-get23:17:530
24349997238,21cyclictest141rcu_preempt20:58:290
2437399713,6cyclictest0-21swapper/222:43:142
2437399713,14cyclictest31189-21munin-run19:22:542
2437399713,10cyclictest6915-21apt-get19:47:542
2437399712,12cyclictest22872-21grep20:33:402
24373997117,47cyclictest8594-21cron19:52:542
24363997114,13cyclictest0-21swapper/100:23:181
24373997053,9cyclictest13896-21fschecks_count21:28:152
24373997035,10cyclictest141rcu_preempt22:27:542
24373997035,10cyclictest141rcu_preempt22:27:542
2437399703,14cyclictest11902-21ssh23:18:382
2437399702,58cyclictest18002-21sh22:33:192
2437399702,15cyclictest13434-21cpuspeed00:18:112
2437399702,13cyclictest1538-21unixbench_multi19:28:452
2437399702,13cyclictest1538-21unixbench_multi19:28:452
2436399704,20cyclictest28631-21ssh21:53:281
2436399703,22cyclictest16533-21/usr/sbin/munin21:33:291
24349997045,17cyclictest18889-21fschecks_time20:23:160
24349997042,18cyclictest131ktimers/022:53:120
24349997042,18cyclictest131ktimers/022:53:120
24373996954,9cyclictest5326-21kworker/u8:023:07:532
24373996954,9cyclictest5326-21kworker/u8:023:07:532
2437399694,11cyclictest19328-21apt-get21:38:092
2437399693,9cyclictest3955-21apt-get19:38:102
2437399693,9cyclictest25225-21/usr/sbin/munin19:03:272
2437399693,12cyclictest30986-21ntpq21:58:272
2437399693,12cyclictest29769-21unixbench_multi23:48:372
2437399693,11cyclictest28546-21/usr/sbin/munin19:13:242
2437399692,14cyclictest13060-21sshd21:27:032
2437399692,12cyclictest31932-21/usr/sbin/munin19:23:212
2436399696,24cyclictest0-21swapper/123:47:561
2436399696,24cyclictest0-21swapper/123:47:561
24363996950,12cyclictest913-21unixbench_multi22:58:291
24363996921,8cyclictest0-21swapper/123:03:041
24363996921,8cyclictest0-21swapper/123:03:041
24363996915,12cyclictest0-21swapper/100:11:571
24363996913,18cyclictest8959-21cpu19:53:171
24363996913,18cyclictest8959-21cpu19:53:171
24349996946,9cyclictest141rcu_preempt22:38:270
24349996945,15cyclictest4720-21idleruntime-cro21:12:530
24373996844,14cyclictest2124-21sh23:58:042
24373996844,14cyclictest2124-21sh23:58:042
2437399683,12cyclictest23837-21fschecks_time20:38:212
2437399683,12cyclictest23837-21fschecks_time20:38:212
2437399683,10cyclictest22882-21ssh23:38:092
2437399682,58cyclictest993-21fschecks_time22:03:162
2437399682,14cyclictest19987-21idleruntime-cro20:27:542
2437399682,14cyclictest19226-21latency_hist21:37:572
2437399682,14cyclictest12521-21/usr/sbin/munin20:03:202
2437399682,11cyclictest28792-21/usr/sbin/munin20:53:172
2437399682,11cyclictest28319-21/usr/sbin/munin21:53:252
2437399682,11cyclictest26547-21/usr/sbin/munin22:48:102
24363996835,8cyclictest25-21ksoftirqd/120:43:171
2436399683,20cyclictest7522-21fschecks_time19:48:281
2436399682,18cyclictest15771-21kernelversion20:13:191
24349996851,10cyclictest3755-21unixbench_multi23:03:290
24349996851,10cyclictest3755-21unixbench_multi23:03:290
24349996822,24cyclictest141rcu_preempt23:58:360
24349996822,24cyclictest141rcu_preempt23:58:360
2437399674,54cyclictest7391-21modprobe00:07:532
2437399674,54cyclictest26626-21wc20:47:552
24373996738,14cyclictest141rcu_preempt23:27:542
2437399673,56cyclictest8020-21sort00:08:212
2437399673,56cyclictest5904-21sort23:08:142
2437399673,56cyclictest31031-21grep20:58:302
2437399673,55cyclictest9868-21unixbench_singl19:53:372
2437399673,55cyclictest9868-21unixbench_singl19:53:372
2437399673,55cyclictest26230-21fschecks_count19:07:592
2437399673,12cyclictest8295-21/usr/sbin/munin21:18:172
2437399672,13cyclictest27335-21ntpq20:48:222
2437399672,13cyclictest2502-21fschecks_time19:33:202
2437399672,13cyclictest22547-21munin-plugin-st23:37:532
2437399672,13cyclictest21172-21/usr/sbin/munin22:38:212
2437399672,13cyclictest19460-21/usr/sbin/munin00:28:212
2437399672,13cyclictest11691-21latency_hist22:22:552
2437399672,12cyclictest17313-21cut23:28:162
2437399672,12cyclictest17313-21cut23:28:162
2437399672,12cyclictest16795-21sort20:17:552
2437399672,10cyclictest10717-21grep19:58:152
24363996720,15cyclictest12156-21df22:23:181
24363996720,15cyclictest12156-21df22:23:181
24363996710,14cyclictest31540-21latency_hist21:02:561
24349996737,10cyclictest141rcu_preempt22:58:110
24349996733,21cyclictest141rcu_preempt23:12:560
24349996730,14cyclictest141rcu_preempt22:22:530
24349996723,35cyclictest23084-21unixbench_singl21:43:310
24349996715,8cyclictest141rcu_preempt23:25:060
24381996647,13cyclictest23808-21fschecks_time20:38:203
24381996647,13cyclictest23808-21fschecks_time20:38:203
2437399663,9cyclictest29471-21ssh22:53:122
2437399663,9cyclictest29471-21ssh22:53:122
2437399663,54cyclictest25125-21ssh21:48:162
2437399663,13cyclictest6543-21rm22:13:142
2437399663,12cyclictest17139-21grep00:23:362
2437399662,12cyclictest4464-21sed22:08:292
2437399662,12cyclictest31651-21/usr/sbin/munin23:53:162
24349996643,15cyclictest29024-21proc_pri20:53:240
24349996637,9cyclictest141rcu_preempt22:06:170
24381996548,11cyclictest11652-21idleruntime-cro22:22:533
2437399653,13cyclictest17359-21/usr/sbin/munin20:18:172
2437399652,9cyclictest8738-21/usr/sbin/munin23:13:152
24349996546,11cyclictest17438-21http22:32:530
24349996546,11cyclictest17438-21http22:32:530
24349996539,12cyclictest141rcu_preempt19:38:380
24349996528,14cyclictest7622-21idleruntime-cro21:17:540
24349996524,8cyclictest0-21swapper/023:43:170
24349996524,8cyclictest0-21swapper/023:43:170
24349996521,21cyclictest141rcu_preempt21:23:260
24381996437,7cyclictest141rcu_preempt21:42:093
2437399644,52cyclictest22194-21ssh21:43:122
2437399643,53cyclictest5654-21/usr/sbin/munin21:13:232
2437399642,16cyclictest13754-21apt-get20:08:062
2436399643,19cyclictest397-21runrttasks19:26:251
2434999645,29cyclictest1-21systemd23:52:540
24381996345,12cyclictest552-21in:imklog23:22:543
2438199633,26cyclictest3361-21ssh23:03:233
2438199633,26cyclictest3361-21ssh23:03:233
2436399634,19cyclictest26531-21fschecks_time19:08:051
24349996344,10cyclictest0-21swapper/021:32:530
24349996337,20cyclictest31729-21cpu23:53:160
24349996333,13cyclictest141rcu_preempt22:52:560
24349996331,12cyclictest0-21swapper/021:42:560
24349996328,20cyclictest141rcu_preempt20:03:220
24349996316,28cyclictest0-21swapper/022:09:100
24381996213,24cyclictest141rcu_preempt22:23:223
24381996213,24cyclictest141rcu_preempt22:23:223
24349996238,10cyclictest141rcu_preempt19:03:150
24349996235,15cyclictest0-21swapper/019:52:530
2434999623,41cyclictest18435-21ssh22:33:350
24349996233,11cyclictest141rcu_preempt00:07:530
24349996225,27cyclictest0-21swapper/022:47:560
24381996142,12cyclictest0-21swapper/319:58:293
24381996136,15cyclictest27131-21ntp_kernel_pll_22:48:273
2438199613,50cyclictest7265-21apt-get19:48:063
24381996117,27cyclictest31427-21diskmemload23:43:533
24381996117,27cyclictest31427-21diskmemload23:43:533
24363996139,13cyclictest7408-21kworker/u8:100:22:561
24363996113,22cyclictest141rcu_preempt22:22:531
24349996142,11cyclictest0-21swapper/022:23:280
24349996142,11cyclictest0-21swapper/022:23:280
24349996136,17cyclictest7932-21fschecks_count00:08:200
24349996129,13cyclictest141rcu_preempt23:28:120
24349996129,13cyclictest141rcu_preempt23:28:120
2434999612,15cyclictest31733-21/usr/sbin/munin21:03:230
2434999612,15cyclictest31733-21/usr/sbin/munin21:03:230
24381996033,14cyclictest0-21swapper/323:53:373
24381996029,7cyclictest141rcu_preempt19:31:133
24381996029,7cyclictest141rcu_preempt19:31:133
24363996036,14cyclictest0-21swapper/120:23:281
24349996042,11cyclictest278-21systemd-journal21:37:550
24349996041,10cyclictest0-21swapper/020:08:300
24349996036,11cyclictest141rcu_preempt20:18:100
24349996035,15cyclictest0-21swapper/021:18:290
24349996032,10cyclictest141rcu_preempt20:43:190
24349996031,9cyclictest141rcu_preempt19:08:050
24349996027,19cyclictest52-21kcompactd000:23:250
24349996025,20cyclictest141rcu_preempt21:57:560
2425126011,29sleep31-21systemd19:02:563
24381995943,10cyclictest23216-21munin-plugin-st22:42:533
24381995935,15cyclictest20598-21cut20:28:293
24381995930,7cyclictest141rcu_preempt20:12:213
2434999599,26cyclictest12-21ksoftirqd/000:18:290
2434999594,32cyclictest12-21ksoftirqd/020:38:210
2434999594,32cyclictest12-21ksoftirqd/020:38:210
24349995914,10cyclictest22529-21modprobe23:37:530
24381995838,13cyclictest6027-21idleruntime-cro22:12:533
24381995830,8cyclictest141rcu_preempt23:39:563
24381995830,6cyclictest141rcu_preempt23:13:353
24381995815,18cyclictest141rcu_preempt21:03:063
24381995815,18cyclictest141rcu_preempt21:03:063
24363995826,10cyclictest25-21ksoftirqd/122:38:091
24349995832,18cyclictest161rcuc/019:28:150
24349995832,18cyclictest161rcuc/019:28:150
2434999583,16cyclictest12-21ksoftirqd/021:48:160
2438199579,25cyclictest141rcu_preempt19:47:563
24381995737,7cyclictest141rcu_preempt19:25:583
24381995736,7cyclictest141rcu_preempt19:53:543
24381995736,7cyclictest141rcu_preempt19:53:543
24381995730,7cyclictest141rcu_preempt20:05:443
24381995730,7cyclictest141rcu_preempt19:22:373
24363995738,13cyclictest552-21in:imklog21:03:221
24363995738,13cyclictest552-21in:imklog21:03:221
24363995731,16cyclictest278-21systemd-journal23:37:341
2434999575,13cyclictest141rcu_preempt21:58:290
2438199566,26cyclictest141rcu_preempt00:07:573
2438199565,25cyclictest141rcu_preempt21:13:153
24381995637,12cyclictest0-21swapper/323:28:313
24381995637,12cyclictest0-21swapper/323:28:313
24381995632,14cyclictest6698-21unixbench_singl23:08:283
24381995630,7cyclictest141rcu_preempt22:37:093
24381995629,6cyclictest141rcu_preempt20:57:293
24381995628,6cyclictest141rcu_preempt20:14:403
24381995628,18cyclictest13013-21modprobe00:17:533
24381995628,16cyclictest0-21swapper/321:57:543
24381995621,17cyclictest141rcu_preempt22:57:403
24381995621,17cyclictest141rcu_preempt22:57:403
24381995612,20cyclictest0-21swapper/320:20:033
2434999568,10cyclictest22929-21grep20:33:400
2434999567,29cyclictest12-21ksoftirqd/022:13:200
24349995626,9cyclictest141rcu_preempt20:48:280
24349995626,9cyclictest12-21ksoftirqd/023:20:150
24349995618,18cyclictest141rcu_preempt23:38:100
24349995614,23cyclictest12-21ksoftirqd/019:47:560
2438199558,22cyclictest141rcu_preempt21:37:563
24381995539,10cyclictest25634-21switchtime21:48:293
24381995530,12cyclictest0-21swapper/319:12:543
24381995529,16cyclictest19378-21latency00:28:193
24381995528,11cyclictest141rcu_preempt00:27:333
24381995525,18cyclictest0-21swapper/320:33:403
2434999553,19cyclictest397-21runrttasks19:18:200
24349995521,13cyclictest141rcu_preempt19:53:360
24349995521,13cyclictest141rcu_preempt19:53:360
24381995430,14cyclictest0-21swapper/321:02:563
24381995429,6cyclictest141rcu_preempt22:01:133
24381995420,16cyclictest141rcu_preempt22:28:463
24381995420,16cyclictest141rcu_preempt22:28:463
24381995418,16cyclictest41-21ksoftirqd/321:23:053
24363995435,10cyclictest0-21swapper/119:43:191
24349995434,10cyclictest0-21swapper/000:28:290
2438199532,15cyclictest41-21ksoftirqd/300:03:553
24381995312,12cyclictest141rcu_preempt20:23:153
24363995337,8cyclictest25-21ksoftirqd/119:18:171
24349995334,7cyclictest141rcu_preempt19:23:260
2438199528,20cyclictest41-21ksoftirqd/323:26:023
2438199523,20cyclictest32582-21fschecks_count22:58:153
24381995231,12cyclictest0-21swapper/319:37:563
24381995228,11cyclictest0-21swapper/319:38:273
24381995226,7cyclictest141rcu_preempt21:21:573
24381995226,13cyclictest141rcu_preempt21:45:003
24381995211,19cyclictest141rcu_preempt21:08:163
24349995217,11cyclictest0-21swapper/019:13:460
24381995130,14cyclictest24337-21cyclictest21:28:153
24381995130,10cyclictest0-21swapper/322:03:163
2436399516,11cyclictest141rcu_preempt20:18:211
24363995128,15cyclictest278-21systemd-journal23:18:371
24349995127,12cyclictest0-21swapper/020:13:290
2438199508,14cyclictest141rcu_preempt20:43:163
24381995034,10cyclictest552-21in:imklog19:13:013
2438199503,24cyclictest14160-21ssh00:18:303
24381995020,7cyclictest141rcu_preempt23:59:173
24381995020,7cyclictest141rcu_preempt23:59:173
2436399506,20cyclictest0-21swapper/120:38:331
2436399506,20cyclictest0-21swapper/120:38:331
24363995029,13cyclictest0-21swapper/123:58:231
24363995029,13cyclictest0-21swapper/123:58:231
2438199493,37cyclictest26982-21fschecks_count20:48:153
24381994920,6cyclictest141rcu_preempt23:36:563
24363994928,9cyclictest0-21swapper/100:13:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional