You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-06-02 - 10:47

ARM Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7 (updated Fri Jun 02, 2023 00:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7471213420,76sleep17509-21grep19:02:511
7471213420,76sleep17509-21grep19:02:511
289932930,6sleep00-21swapper/021:28:090
73672878,33sleep30-21swapper/319:02:453
73672878,33sleep30-21swapper/319:02:453
8702998615,15cyclictest2377-21grep00:27:141
8696998647,18cyclictest141rcu_preempt19:47:060
8702998345,32cyclictest0-21swapper/123:36:311
870299813,13cyclictest400-21runrttasks21:31:331
8702998130,10cyclictest0-21swapper/122:56:331
8696998044,7cyclictest141rcu_preempt19:11:580
8714997953,16cyclictest25274-21idleruntime-cro19:56:313
8702997924,12cyclictest0-21swapper/119:57:051
726927911,46sleep07235-21apt-get19:02:090
726927911,46sleep07235-21apt-get19:02:090
8702997723,10cyclictest0-21swapper/119:41:331
8702997722,10cyclictest0-21swapper/122:57:061
8702997722,10cyclictest0-21swapper/122:57:061
8702997646,19cyclictest15654-21idleruntime-cro21:06:311
870299764,23cyclictest0-21swapper/123:56:321
8702997633,19cyclictest141rcu_preempt00:22:061
8702997619,11cyclictest0-21swapper/123:47:141
8702997525,7cyclictest0-21swapper/119:11:331
8702997522,9cyclictest0-21swapper/123:20:331
79992759,30sleep20-21swapper/219:03:092
79992759,30sleep20-21swapper/219:03:092
8707997454,13cyclictest12299-21modprobe19:16:312
870299744,26cyclictest5879-21sh00:34:241
870299744,26cyclictest5879-21sh00:34:241
8702997417,11cyclictest0-21swapper/120:12:091
8702997417,11cyclictest0-21swapper/120:12:091
8707997353,13cyclictest15641-21latency_hist21:06:312
8707997347,16cyclictest1-21systemd20:56:332
8707997347,16cyclictest1-21systemd20:56:332
8702997327,9cyclictest0-21swapper/120:02:131
8707997224,34cyclictest321ktimers/223:42:022
8702997220,10cyclictest0-21swapper/121:13:221
8702997220,10cyclictest0-21swapper/121:13:221
8702997217,10cyclictest0-21swapper/119:22:061
8702997217,10cyclictest0-21swapper/119:22:061
8696997239,14cyclictest141rcu_preempt21:40:060
8707997147,13cyclictest0-21swapper/221:36:312
8707997146,7cyclictest141rcu_preempt22:24:472
8707997139,7cyclictest141rcu_preempt22:42:452
8702997124,10cyclictest0-21swapper/122:36:401
8702997124,10cyclictest0-21swapper/122:36:401
8696997149,13cyclictest15606-21kworker/u8:122:06:320
8696997144,17cyclictest21875-21sh21:16:530
8714997037,9cyclictest141rcu_preempt20:02:133
8707997050,14cyclictest31778-21kworker/u8:320:16:312
8707997050,14cyclictest31778-21kworker/u8:320:16:312
8707997049,12cyclictest16840-21sh23:56:312
8702997035,14cyclictest141rcu_preempt23:56:581
8702997035,14cyclictest141rcu_preempt23:56:581
8702997018,11cyclictest0-21swapper/100:09:001
8702997017,11cyclictest0-21swapper/120:16:571
8702997014,11cyclictest0-21swapper/120:07:121
8714996929,8cyclictest141rcu_preempt23:12:143
8702996951,12cyclictest31679-21systemd-journal23:11:331
8702996951,11cyclictest23670-21systemd-run19:51:311
8702996950,12cyclictest18240-21sendmail_mailqu23:02:131
8702996949,10cyclictest0-21swapper/123:46:331
8702996918,30cyclictest27230-21sed22:22:011
8707996840,16cyclictest0-21swapper/219:47:132
870299689,33cyclictest25-21ksoftirqd/123:28:061
8702996824,22cyclictest141rcu_preempt21:52:101
8702996818,12cyclictest0-21swapper/121:08:151
8702996818,12cyclictest0-21swapper/121:08:151
8714996711,20cyclictest141rcu_preempt21:06:523
8714996711,20cyclictest141rcu_preempt21:06:523
8707996752,9cyclictest10535-21unixbench_multi21:52:112
8707996749,11cyclictest23858-21unixbench_singl00:07:162
8707996747,13cyclictest13933-21sshd21:58:332
8707996722,17cyclictest0-21swapper/220:36:522
8702996726,12cyclictest141rcu_preempt22:47:071
8696996749,10cyclictest5563-21ssh21:44:230
8696996746,14cyclictest19666-21idleruntime-cro00:01:310
8696996746,14cyclictest19666-21idleruntime-cro00:01:310
8696996744,7cyclictest141rcu_preempt23:31:060
8707996644,12cyclictest0-21swapper/222:16:542
8707996644,11cyclictest141rcu_preempt21:09:282
8707996644,11cyclictest141rcu_preempt21:09:282
8702996650,11cyclictest0-21swapper/120:56:331
8702996650,11cyclictest0-21swapper/120:56:331
8696996649,9cyclictest1212-21ssh22:32:140
8696996637,14cyclictest141rcu_preempt23:51:020
8714996536,8cyclictest141rcu_preempt21:42:033
8707996538,17cyclictest0-21swapper/222:31:312
8707996525,21cyclictest362-21in:imklog22:54:542
870299654,29cyclictest313-21fschecks_count22:31:571
8702996511,24cyclictest25-21ksoftirqd/122:16:331
8702996511,11cyclictest185-21pr/ttyS000:06:331
8696996537,13cyclictest141rcu_preempt22:51:320
8714996449,9cyclictest20696-21cron22:11:313
8714996449,9cyclictest20696-21cron22:11:313
8714996436,21cyclictest3431-21munin-plugin-st21:41:313
8702996429,14cyclictest141rcu_preempt20:46:341
8696996438,7cyclictest141rcu_preempt00:17:240
8696996434,7cyclictest141rcu_preempt00:08:580
8714996346,11cyclictest11971-21fschecks_time22:51:583
8714996329,13cyclictest362-21in:imklog22:39:423
8714996329,13cyclictest362-21in:imklog22:39:423
8714996323,18cyclictest0-21swapper/322:28:533
8707996344,12cyclictest3449-21sh00:30:242
8707996340,12cyclictest0-21swapper/220:17:112
870799632,43cyclictest247-21jbd2/mmcblk2p2-23:11:552
870299632,37cyclictest24132-21ssh22:16:581
8696996343,10cyclictest0-21swapper/023:11:310
8696996341,13cyclictest9198-21ssh23:42:150
8696996338,8cyclictest141rcu_preempt22:45:090
8696996337,13cyclictest141rcu_preempt21:36:340
8696996332,10cyclictest141rcu_preempt20:52:020
8696996332,10cyclictest141rcu_preempt20:52:020
8696996324,12cyclictest362-21in:imklog20:22:130
8714996240,16cyclictest12043-21idleruntime-cro21:56:313
8714996239,12cyclictest0-21swapper/322:51:343
8714996232,9cyclictest141rcu_preempt00:32:033
8714996232,9cyclictest141rcu_preempt00:32:033
8714996226,10cyclictest41-21ksoftirqd/320:11:563
8714996226,10cyclictest41-21ksoftirqd/320:11:563
8714996223,8cyclictest41-21ksoftirqd/323:19:013
8707996247,10cyclictest400-21runrttasks23:57:212
8707996247,10cyclictest400-21runrttasks23:57:212
8707996245,11cyclictest24385-21idleruntime-cro21:21:312
8707996243,11cyclictest0-21swapper/223:06:302
8707996241,11cyclictest0-21swapper/222:56:592
8707996241,11cyclictest0-21swapper/222:56:592
8707996212,21cyclictest2368-21latency_hist23:31:332
8702996211,14cyclictest141rcu_preempt21:17:111
8696996248,8cyclictest19065-21ssh22:07:150
8696996248,8cyclictest19065-21ssh22:07:150
8696996244,8cyclictest0-21swapper/000:32:140
8696996244,8cyclictest0-21swapper/000:32:140
8696996243,13cyclictest3739-21unixbench_multi20:27:150
8696996227,7cyclictest0-21swapper/020:47:130
8696996225,11cyclictest12-21ksoftirqd/019:36:590
8714996145,10cyclictest20090-21idleruntime-cro23:06:303
8714996129,11cyclictest362-21in:imklog21:51:333
8714996116,20cyclictest0-21swapper/300:22:083
870299614,34cyclictest25-21ksoftirqd/123:37:211
8702996125,11cyclictest141rcu_preempt21:46:231
870299612,26cyclictest23605-21ssh23:11:591
8702996119,10cyclictest141rcu_preempt22:07:021
8702996119,10cyclictest141rcu_preempt22:07:021
8696996142,9cyclictest0-21swapper/019:31:310
8696996142,10cyclictest13950-21munin-plugin-st19:21:310
8696996141,11cyclictest0-21swapper/021:52:120
8696996130,8cyclictest0-21swapper/021:26:320
8696996130,13cyclictest31679-21systemd-journal00:26:120
8696996113,13cyclictest12-21ksoftirqd/023:01:570
871499604,9cyclictest141rcu_preempt23:07:173
8714996038,12cyclictest0-21swapper/321:31:583
8714996023,20cyclictest41-21ksoftirqd/322:04:353
8714996021,10cyclictest141rcu_preempt22:21:303
8707996035,13cyclictest141rcu_preempt19:57:252
8707996033,15cyclictest0-21swapper/200:36:332
8707996033,15cyclictest0-21swapper/200:36:332
8707996032,14cyclictest141rcu_preempt22:08:442
8707996032,14cyclictest141rcu_preempt22:08:442
870299603,34cyclictest16441-21find19:27:101
870299603,28cyclictest0-21swapper/120:40:501
8702996031,20cyclictest0-21swapper/121:37:191
8702996014,17cyclictest25-21ksoftirqd/119:46:311
8702996014,17cyclictest25-21ksoftirqd/119:46:311
8702996012,8cyclictest141rcu_preempt19:36:341
8702996012,8cyclictest141rcu_preempt19:36:341
8714995942,11cyclictest31679-21systemd-journal22:36:343
8714995940,12cyclictest12378-21latency_hist19:16:333
8714995938,12cyclictest31679-21systemd-journal23:38:113
8714995938,12cyclictest0-21swapper/323:01:333
8714995938,12cyclictest0-21swapper/323:01:333
8714995937,12cyclictest0-21swapper/320:32:013
8714995936,13cyclictest0-21swapper/319:37:003
8714995933,12cyclictest0-21swapper/300:02:043
8707995938,12cyclictest7966-21grep20:41:582
8707995938,10cyclictest16053-21fschecks_time19:27:002
8707995937,7cyclictest141rcu_preempt23:08:102
8707995937,15cyclictest19877-21timerwakeupswit21:12:132
8707995937,15cyclictest19877-21timerwakeupswit21:12:132
8707995934,13cyclictest0-21swapper/219:22:092
8707995934,13cyclictest0-21swapper/219:22:092
8707995928,6cyclictest141rcu_preempt22:16:002
8707995928,17cyclictest362-21in:imklog00:03:372
8707995926,22cyclictest0-21swapper/221:37:042
870799592,19cyclictest0-21swapper/221:22:012
8702995936,13cyclictest0-21swapper/119:17:101
8702995928,8cyclictest141rcu_preempt22:02:041
8696995942,10cyclictest31804-21systemd-run23:26:310
8696995942,10cyclictest31804-21systemd-run23:26:310
8696995941,11cyclictest32580-21ntp_offset20:17:040
8696995936,7cyclictest141rcu_preempt21:07:130
8696995936,7cyclictest141rcu_preempt21:07:130
8696995935,15cyclictest31679-21systemd-journal00:28:460
8696995912,14cyclictest12-21ksoftirqd/023:12:020
8714995841,10cyclictest8460-21idleruntime-cro22:46:303
8714995840,12cyclictest27978-21modprobe00:16:303
8714995831,7cyclictest141rcu_preempt22:12:113
8714995822,9cyclictest141rcu_preempt19:36:313
8714995822,9cyclictest141rcu_preempt19:36:313
8714995820,16cyclictest41-21ksoftirqd/321:27:063
8714995816,21cyclictest362-21in:imklog23:54:573
8707995837,13cyclictest0-21swapper/220:11:322
8707995833,12cyclictest141rcu_preempt22:50:502
870799583,19cyclictest6707-21ssh23:37:282
8707995827,11cyclictest141rcu_preempt20:27:282
870299582,28cyclictest25-21ksoftirqd/121:26:131
8702995820,18cyclictest141rcu_preempt20:47:061
8696995844,8cyclictest12671-21fschecks_time21:56:580
8696995833,13cyclictest6898-21runrttasks21:46:580
8696995813,11cyclictest12-21ksoftirqd/019:42:070
8696995813,11cyclictest12-21ksoftirqd/019:42:070
67542580,6sleep00-21swapper/020:37:210
8714995744,7cyclictest0-21swapper/300:21:313
8714995734,16cyclictest0-21swapper/321:12:003
8714995734,16cyclictest0-21swapper/321:12:003
8707995738,13cyclictest362-21in:imklog22:01:562
8707995735,8cyclictest141rcu_preempt23:19:242
8707995733,13cyclictest141rcu_preempt19:19:392
8707995728,14cyclictest0-21swapper/220:26:342
870299573,21cyclictest12668-21taskset21:56:581
870299572,32cyclictest3536-21switchtime20:27:131
8696995743,8cyclictest29297-21ssh22:26:310
8696995739,12cyclictest7990-21fschecks_count20:41:580
8696995716,12cyclictest141rcu_preempt22:27:120
8714995623,19cyclictest141rcu_preempt19:07:053
870799569,18cyclictest33-21ksoftirqd/220:01:342
8707995637,7cyclictest141rcu_preempt21:42:482
8707995636,7cyclictest141rcu_preempt21:47:232
8707995636,13cyclictest22034-21latency_hist19:46:342
8707995636,13cyclictest22034-21latency_hist19:46:342
8707995635,10cyclictest0-21swapper/219:36:342
8707995635,10cyclictest0-21swapper/219:36:342
8707995632,7cyclictest141rcu_preempt23:26:092
8707995632,7cyclictest141rcu_preempt23:26:092
8707995630,9cyclictest141rcu_preempt00:25:012
8707995627,6cyclictest141rcu_preempt00:13:192
8707995626,13cyclictest141rcu_preempt23:49:512
870299563,30cyclictest15638-21diskmemload22:27:151
8702995629,7cyclictest141rcu_preempt00:13:301
870299562,28cyclictest12914-21fschecks_time20:57:001
8702995614,25cyclictest141rcu_preempt00:21:341
8696995638,13cyclictest12-21ksoftirqd/022:16:530
8696995638,12cyclictest31775-21munin-plugin-st20:16:310
8696995638,12cyclictest31775-21munin-plugin-st20:16:310
8696995633,12cyclictest12-21ksoftirqd/022:37:270
8696995633,12cyclictest12-21ksoftirqd/022:37:270
871499555,8cyclictest141rcu_preempt19:47:123
8714995540,9cyclictest185-21pr/ttyS020:07:103
8714995539,8cyclictest0-21swapper/319:26:323
8714995539,8cyclictest0-21swapper/319:26:323
8714995538,11cyclictest362-21in:imklog23:26:513
8714995536,14cyclictest0-21swapper/323:36:313
871499553,29cyclictest6285-21cstates20:36:593
8714995532,12cyclictest0-21swapper/321:26:313
8714995521,13cyclictest41-21ksoftirqd/319:57:143
8714995514,17cyclictest141rcu_preempt19:16:523
870799556,20cyclictest10188-21ls20:47:132
8707995536,10cyclictest0-21swapper/222:31:592
8707995532,11cyclictest0-21swapper/221:26:572
870299559,23cyclictest6926-21fschecks_time21:46:591
8702995513,12cyclictest6007-21latency_hist20:36:341
8696995540,8cyclictest0-21swapper/022:56:320
8696995539,9cyclictest12360-21kworker/u8:321:01:310
8696995537,11cyclictest0-21swapper/019:36:320
8696995537,11cyclictest0-21swapper/019:36:320
8696995537,10cyclictest16895-21ssh23:00:450
8696995537,10cyclictest16895-21ssh23:00:450
8696995534,11cyclictest0-21swapper/020:36:340
8696995526,10cyclictest31679-21systemd-journal21:15:020
8696995526,10cyclictest31679-21systemd-journal21:15:020
8696995525,11cyclictest362-21in:imklog23:55:540
8696995515,15cyclictest362-21in:imklog20:07:080
8714995439,8cyclictest8685-21cyclictest20:21:553
8714995436,11cyclictest337-21dbus-daemon00:31:313
8714995432,13cyclictest17329-21latency_hist19:31:343
8714995419,18cyclictest0-21swapper/320:57:113
8714995415,21cyclictest11697-21kworker/3:221:58:083
8714995411,20cyclictest0-21swapper/323:25:373
8714995411,20cyclictest0-21swapper/323:25:373
870799549,22cyclictest33-21ksoftirqd/219:10:112
8707995431,13cyclictest0-21swapper/222:36:592
8707995431,13cyclictest0-21swapper/222:36:592
8707995431,13cyclictest0-21swapper/220:57:152
8707995428,7cyclictest141rcu_preempt23:33:492
870299549,23cyclictest141rcu_preempt19:51:571
869699547,25cyclictest141rcu_preempt19:56:590
8696995427,7cyclictest141rcu_preempt23:19:230
8696995427,7cyclictest141rcu_preempt22:15:300
8696995420,13cyclictest31679-21systemd-journal00:03:110
57432542,42sleep05750-21diskstats23:37:000
8714995339,9cyclictest8685-21cyclictest00:09:543
8714995335,12cyclictest31679-21systemd-journal23:58:333
8714995335,12cyclictest31679-21systemd-journal23:58:333
8707995325,18cyclictest5782-21modprobe20:36:312
8707995313,8cyclictest0-21swapper/219:40:132
870299536,27cyclictest141rcu_preempt21:32:121
8702995323,16cyclictest141rcu_preempt23:25:231
8702995323,16cyclictest141rcu_preempt23:25:231
8702995316,14cyclictest6199-21fschecks_time22:41:581
8696995326,21cyclictest362-21in:imklog23:31:530
8696995325,10cyclictest12-21ksoftirqd/019:52:030
869699532,19cyclictest15639-21modprobe21:06:310
8714995237,9cyclictest185-21pr/ttyS021:04:313
8714995235,10cyclictest31679-21systemd-journal23:47:053
8714995214,9cyclictest141rcu_preempt20:42:133
870299522,12cyclictest10775-21fschecks_time19:11:391
869699528,13cyclictest141rcu_preempt20:06:310
325362522,40sleep332581-21ntp_offset20:17:043
8714995138,7cyclictest0-21swapper/323:43:033
8714995135,9cyclictest31747-21kworker/u8:020:51:303
8707995121,6cyclictest141rcu_preempt00:16:452
8696995130,14cyclictest27972-21modprobe00:16:300
8714995026,15cyclictest122-21pr/ttymxc019:42:033
8714995026,15cyclictest122-21pr/ttymxc019:42:033
8702995033,11cyclictest0-21swapper/120:22:141
869699509,7cyclictest141rcu_preempt19:21:470
869699509,7cyclictest141rcu_preempt19:21:470
8714994933,10cyclictest31679-21systemd-journal20:52:183
8714994933,10cyclictest31679-21systemd-journal20:52:183
8714994927,10cyclictest0-21swapper/320:27:133
8714994926,13cyclictest0-21swapper/321:17:093
871499487,16cyclictest362-21in:imklog22:22:143
870799482,12cyclictest33-21ksoftirqd/219:52:082
8696994833,9cyclictest31679-21systemd-journal19:08:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional