You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-05-31 - 03:01

ARM Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7 (updated Wed May 31, 2023 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19704211413,66sleep319719-21systemd-run19:06:413
19704211413,66sleep319719-21systemd-run19:06:413
1892021137,77sleep10-21swapper/119:02:241
1892021137,77sleep10-21swapper/119:02:241
19791991019,53cyclictest7874-21sh21:30:020
1979199951,66cyclictest21061-21rm22:51:360
19791999313,49cyclictest26780-21diskmemload23:45:210
1979199902,58cyclictest10325-21ssh23:28:090
1979199902,58cyclictest10325-21ssh23:28:090
1979199893,59cyclictest11518-21ssh22:33:570
1979199893,59cyclictest11518-21ssh22:33:570
1979199884,77cyclictest24377-21grep23:52:300
1979199883,58cyclictest0-21swapper/022:09:440
1979199883,58cyclictest0-21swapper/022:09:440
1979199882,80cyclictest11741-21cpuspeed21:37:040
1979199882,77cyclictest8856-21ssh21:32:040
1979199877,51cyclictest141rcu_preempt23:07:170
1979199874,75cyclictest2857-21latency_hist21:21:410
1979199874,75cyclictest2857-21latency_hist21:21:410
19791998630,49cyclictest8350irq/69-2188000.21:46:580
1979199853,74cyclictest0-21swapper/019:06:440
1979199853,74cyclictest0-21swapper/019:06:440
1979199853,55cyclictest400-21runrttasks22:28:360
1979199853,54cyclictest12058-21ssh00:30:030
1979199852,43cyclictest29548-21ssh23:06:130
19791998514,46cyclictest8396-21kworker/u8:521:58:400
1979199843,73cyclictest22748-21ssh23:51:050
1979199843,73cyclictest22748-21ssh23:51:050
1979199842,52cyclictest16052-21cat22:42:100
1979499833,26cyclictest0-21swapper/223:37:042
1979199834,72cyclictest14311-21awk23:36:390
1979199834,72cyclictest14311-21awk23:36:390
1979199832,50cyclictest18901-21taskset20:41:460
19799998265,11cyclictest20292-21munin-plugin-st20:46:393
1979199826,68cyclictest30656-21/usr/sbin/munin21:12:230
1979199824,72cyclictest0-21swapper/023:41:380
1979199824,72cyclictest0-21swapper/023:16:590
1979199824,72cyclictest0-21swapper/023:16:590
1979199823,55cyclictest0-21swapper/000:01:380
1979199814,70cyclictest0-21swapper/021:42:010
1979199814,70cyclictest0-21swapper/021:42:010
1979199814,68cyclictest13789-21latency_hist20:26:420
1979199814,68cyclictest13789-21latency_hist20:26:420
1979199813,72cyclictest26812-21/usr/sbin/munin21:06:400
1979199813,71cyclictest5119-21kernelversion00:17:090
1979199813,68cyclictest3163-21cat23:16:420
1979199812,47cyclictest8163-21/usr/sbin/munin00:22:180
1979199805,67cyclictest14260-21cut20:27:080
1979199802,71cyclictest5341-21rm21:25:530
1979199802,69cyclictest4194-21ssh22:21:590
1979199802,52cyclictest25972-21ntp_states22:02:130
19792997957,14cyclictest5688-21munin-run20:01:391
1979199793,69cyclictest22498-21unixbench_singl22:52:240
1979199792,69cyclictest26917-21apt-get19:27:000
1979199792,69cyclictest0-21swapper/019:42:000
1979199792,50cyclictest1815-21cat00:11:420
19794997827,22cyclictest1212-21rm00:10:112
1979199784,66cyclictest29934-21munin-run19:36:390
1979199784,65cyclictest0-21swapper/000:12:010
1979199784,65cyclictest0-21swapper/000:12:010
1979199783,66cyclictest0-21swapper/000:02:010
1979199782,69cyclictest14323-21ssh22:38:350
1979199782,68cyclictest28199-21/usr/sbin/munin21:07:220
19791997819,53cyclictest6674-21/usr/sbin/munin23:22:150
1979299773,57cyclictest19658-21kworker/u8:319:52:001
1979299773,57cyclictest19658-21kworker/u8:319:52:001
19791997725,46cyclictest0-21swapper/019:56:400
19791997725,46cyclictest0-21swapper/019:56:400
1979199772,49cyclictest31078-21chrt22:12:050
1979199772,47cyclictest20516-21/usr/sbin/munin21:52:350
19791997722,45cyclictest0-21swapper/019:22:210
19791997721,49cyclictest1982-21ps22:17:210
139582770,4sleep10-21swapper/122:37:301
19799997627,22cyclictest14715-21/usr/sbin/munin23:36:543
1979999762,40cyclictest5977-21cat23:21:423
1979999762,40cyclictest5977-21cat23:21:423
19794997627,21cyclictest31528-21taskset19:41:322
1979199763,65cyclictest17544-21fschecks_time20:37:090
1979199763,63cyclictest24350-21apt-get22:57:010
1979199763,63cyclictest24350-21apt-get22:57:010
1979199762,50cyclictest9441-21grep20:12:090
1979199762,50cyclictest9441-21grep20:12:090
19794997527,19cyclictest21698-21taskset20:49:182
19794997527,18cyclictest26714-21chrt21:06:072
19794997526,18cyclictest7532-21taskset23:23:212
19792997534,19cyclictest141rcu_preempt22:15:431
1979199755,62cyclictest26780-21diskmemload00:32:220
1979199753,60cyclictest993-21apt-get19:46:570
1979199752,65cyclictest6408-21needreboot20:02:120
1979199752,65cyclictest6408-21needreboot20:02:120
19794997426,20cyclictest15318-21taskset20:30:462
1979199744,63cyclictest17031-21ls20:36:390
1979199744,63cyclictest17031-21ls20:36:390
1979199743,64cyclictest21852-21munin-node19:11:490
1979199743,64cyclictest21852-21munin-node19:11:490
1979199743,63cyclictest20913-21sed20:47:060
1979199742,62cyclictest22173-21apt-get20:52:010
19799997315,31cyclictest400-21runrttasks23:36:033
19799997315,31cyclictest400-21runrttasks23:36:033
1979299736,49cyclictest26780-21diskmemload23:37:021
19792997328,24cyclictest141rcu_preempt00:12:351
19792997328,24cyclictest141rcu_preempt00:12:351
19792997322,29cyclictest0-21swapper/120:15:411
19792997322,29cyclictest0-21swapper/120:15:411
19794997248,14cyclictest4321-21unixbench_singl23:17:212
19794997248,14cyclictest4321-21unixbench_singl23:17:212
19794997247,15cyclictest29539-21diskmemload21:11:392
19794997225,18cyclictest18443-21sh21:48:092
19794997224,21cyclictest29711-21chrt19:34:122
19794997223,20cyclictest15036-21runrttasks00:35:452
19792997239,12cyclictest141rcu_preempt19:23:191
19792997223,10cyclictest141rcu_preempt00:05:371
19792997217,13cyclictest0-21swapper/123:54:441
19792997216,25cyclictest0-21swapper/122:55:261
1979199724,61cyclictest0-21swapper/019:07:110
19794997140,22cyclictest25306-21apt-get19:22:022
19792997139,10cyclictest141rcu_preempt23:47:381
19792997139,10cyclictest141rcu_preempt23:47:381
19792997130,8cyclictest141rcu_preempt21:42:531
19792997130,8cyclictest141rcu_preempt21:42:531
19792997116,29cyclictest25-21ksoftirqd/122:06:211
1979199712,61cyclictest7550-21/usr/sbin/munin20:07:210
1979199712,60cyclictest11422-21ps20:17:220
19799997015,47cyclictest337-21dbus-daemon19:33:453
19794997023,19cyclictest24707-21taskset19:17:472
19794997022,20cyclictest2085-21chrt19:48:432
19792997028,10cyclictest141rcu_preempt22:28:221
1979199704,57cyclictest23669-21/usr/sbin/munin19:17:060
1979199704,57cyclictest0-21swapper/020:56:590
1979199702,61cyclictest31541-21modprobe19:41:400
1979999692,41cyclictest418-21snmpd19:18:323
19794996951,11cyclictest12569-21latency_hist00:31:382
19794996925,17cyclictest704-21sh22:15:572
19794996924,18cyclictest6989-21runrttasks22:26:482
19794996919,19cyclictest27604-21ssh22:06:082
19792996949,14cyclictest11763-21unixbench_multi20:17:281
19792996932,10cyclictest141rcu_preempt23:09:101
19792996930,9cyclictest241ktimers/121:16:391
19792996929,12cyclictest0-21swapper/120:22:031
19792996929,12cyclictest0-21swapper/120:22:031
19792996928,14cyclictest141rcu_preempt22:16:421
19792996914,23cyclictest0-21swapper/121:36:141
1979199694,56cyclictest4333-21/usr/sbin/munin19:56:560
19794996850,11cyclictest20117-21idleruntime-cro23:46:382
19794996823,16cyclictest21189-21chrt19:09:592
19794996822,19cyclictest7042-21chrt20:03:382
19794996822,19cyclictest7042-21chrt20:03:382
19794996820,19cyclictest27457-21sh23:57:552
19792996849,10cyclictest11467-21idleruntime-cro23:31:391
19792996849,10cyclictest11467-21idleruntime-cro23:31:391
19792996837,8cyclictest141rcu_preempt23:57:371
19792996836,7cyclictest141rcu_preempt20:01:511
19792996836,7cyclictest141rcu_preempt20:01:511
19792996828,20cyclictest141rcu_preempt00:08:181
19792996821,29cyclictest0-21swapper/121:05:451
19792996813,26cyclictest0-21swapper/121:10:341
19799996747,14cyclictest400-21runrttasks00:24:213
19799996742,17cyclictest25189-21sh22:01:423
19799996730,30cyclictest0-21swapper/300:01:403
19794996729,29cyclictest11592-21cat23:31:412
19794996729,29cyclictest11592-21cat23:31:412
19794996721,19cyclictest11955-21taskset20:19:022
19792996740,8cyclictest141rcu_preempt22:57:211
19792996740,8cyclictest141rcu_preempt22:57:211
19792996727,11cyclictest141rcu_preempt21:31:051
19792996716,21cyclictest0-21swapper/100:21:201
19799996649,11cyclictest18637-21unixbench_singl23:42:203
19799996646,13cyclictest32365-21http21:16:403
1979999663,52cyclictest13529-21/usr/sbin/munin00:32:183
19799996628,31cyclictest0-21swapper/320:06:403
19799996628,31cyclictest0-21swapper/320:06:403
1979999662,57cyclictest9546-21uname20:12:123
1979999662,57cyclictest9546-21uname20:12:123
19794996650,10cyclictest4508-21runrttasks00:16:412
19794996650,10cyclictest4508-21runrttasks00:16:412
19794996639,18cyclictest14393-21apt-get21:41:582
19794996639,18cyclictest14393-21apt-get21:41:582
19794996621,18cyclictest5611-21sh22:23:232
19792996649,11cyclictest5785-21idleruntime-cro23:21:381
19792996649,11cyclictest5785-21idleruntime-cro23:21:381
19792996644,14cyclictest12614-21switchtime23:32:201
19792996644,14cyclictest12614-21switchtime23:32:201
19792996639,12cyclictest141rcu_preempt21:41:111
19792996635,8cyclictest141rcu_preempt22:51:201
19792996624,12cyclictest141rcu_preempt23:14:031
19792996623,28cyclictest0-21swapper/100:32:081
19792996622,22cyclictest0-21swapper/120:48:461
19792996611,28cyclictest25-21ksoftirqd/122:24:001
19799996546,12cyclictest22101-21sshd22:52:203
19799996543,14cyclictest25881-21kworker/u8:122:16:413
1979999653,53cyclictest20558-21/usr/sbin/munin19:07:193
19799996527,29cyclictest0-21swapper/322:22:193
19794996522,16cyclictest8028-21chrt21:30:302
19794996520,17cyclictest23065-21chrt19:12:542
19794996520,17cyclictest23065-21chrt19:12:542
19792996541,11cyclictest141rcu_preempt19:17:121
19792996534,10cyclictest141rcu_preempt23:26:411
19792996532,17cyclictest141rcu_preempt21:55:361
19792996532,12cyclictest141rcu_preempt23:03:591
19792996531,8cyclictest141rcu_preempt20:07:191
19792996529,19cyclictest141rcu_preempt23:46:321
19792996516,9cyclictest0-21swapper/121:19:361
19792996516,9cyclictest0-21swapper/121:19:361
1896926511,24sleep20-21swapper/219:02:252
1896926511,24sleep20-21swapper/219:02:252
19799996444,13cyclictest185-21pr/ttyS021:32:093
1979999642,55cyclictest0-21swapper/322:47:003
1979999642,54cyclictest25487-21cut19:22:103
1979999642,53cyclictest30173-21cstates23:07:063
19794996436,18cyclictest569-21/usr/sbin/munin21:17:132
19794996436,18cyclictest569-21/usr/sbin/munin21:17:132
19792996415,27cyclictest0-21swapper/120:55:261
19792996414,26cyclictest0-21swapper/100:26:361
19792996411,13cyclictest0-21swapper/122:36:021
19792996411,13cyclictest0-21swapper/122:36:021
1979999634,49cyclictest8406-21latency_hist21:31:423
1979999633,53cyclictest29908-21timerwakeupswit00:02:213
1979999632,53cyclictest28287-21ssh22:07:023
1979999632,53cyclictest28287-21ssh22:07:023
1979999632,53cyclictest12765-21latency_hist00:31:413
1979999632,52cyclictest22595-21/usr/sbin/munin19:12:193
1979999632,52cyclictest22595-21/usr/sbin/munin19:12:193
1979999632,51cyclictest4764-21grep00:17:013
1979999632,34cyclictest141rcu_preempt22:31:563
1979999632,34cyclictest141rcu_preempt22:31:563
19794996334,22cyclictest11885-21/usr/sbin/munin23:32:022
19794996334,22cyclictest11885-21/usr/sbin/munin23:32:022
19794996314,22cyclictest22184-21sh21:56:302
19792996332,15cyclictest141rcu_preempt21:22:151
19792996332,14cyclictest141rcu_preempt19:07:281
19792996328,8cyclictest141rcu_preempt00:27:141
19792996323,10cyclictest0-21swapper/119:32:071
19792996315,21cyclictest0-21swapper/121:59:021
19799996240,12cyclictest0-21swapper/300:11:393
1979999623,52cyclictest4233-21grep21:22:263
1979999622,53cyclictest32207-21/usr/sbin/munin19:42:133
1979999622,52cyclictest2009-21ssh00:12:013
1979999622,52cyclictest2009-21ssh00:12:013
1979999622,52cyclictest18248-21/usr/sbin/munin20:37:243
1979999621,53cyclictest417-21ssh23:12:013
19794996244,11cyclictest30200-21fschecks_time21:12:102
19794996242,10cyclictest17322-21apt-get20:37:022
19794996237,17cyclictest23982-21/usr/sbin/munin23:52:262
19794996236,17cyclictest11910-21sort21:37:092
19794996234,19cyclictest8777-21apt-get21:31:512
19794996234,18cyclictest16234-21/usr/sbin/munin20:32:182
19794996234,18cyclictest16234-21/usr/sbin/munin20:32:182
19794996233,19cyclictest18221-21modprobe22:46:392
19794996218,17cyclictest26780-21diskmemload22:07:302
19794996218,17cyclictest26780-21diskmemload22:07:302
1979299623,36cyclictest25-21ksoftirqd/119:46:031
19792996230,8cyclictest141rcu_preempt19:27:161
19792996230,12cyclictest141rcu_preempt19:37:101
19792996230,11cyclictest141rcu_preempt22:42:361
1979999618,46cyclictest27278-21expr21:07:023
19799996140,11cyclictest0-21swapper/319:51:403
1979999613,48cyclictest2858-21/usr/sbin/munin19:52:113
1979999613,48cyclictest2858-21/usr/sbin/munin19:52:113
19799996127,25cyclictest0-21swapper/322:21:403
1979999612,52cyclictest25026-21ssh22:57:203
1979999612,52cyclictest25026-21ssh22:57:203
1979999612,52cyclictest16710-21taskset21:46:323
1979999612,52cyclictest16710-21taskset21:46:323
1979999612,51cyclictest8961-21/usr/sbin/munin23:26:553
1979999612,51cyclictest8961-21/usr/sbin/munin23:26:553
1979999612,51cyclictest20431-21/usr/sbin/munin23:46:563
1979999612,51cyclictest20431-21/usr/sbin/munin23:46:563
1979999612,50cyclictest542-21/usr/sbin/munin21:17:103
1979999612,50cyclictest542-21/usr/sbin/munin21:17:103
1979999612,50cyclictest26926-21/usr/sbin/munin19:27:053
19794996141,9cyclictest31829-21apt-get19:42:002
19794996136,17cyclictest4235-21unixbench_singl21:22:262
19794996136,16cyclictest7145-21grep00:21:382
19794996134,17cyclictest27451-21/usr/sbin/munin19:27:182
19794996134,17cyclictest22795-21/usr/sbin/munin20:52:192
19794996133,18cyclictest8010-21/usr/sbin/munin00:22:132
19794996131,22cyclictest11992-21ssh22:35:202
19794996131,22cyclictest11992-21ssh22:35:202
19794996121,16cyclictest26780-21diskmemload22:00:472
19792996132,9cyclictest141rcu_preempt22:10:221
19792996132,9cyclictest141rcu_preempt22:10:221
19792996118,11cyclictest0-21swapper/120:58:131
1979999608,40cyclictest0-21swapper/322:38:193
1979999604,47cyclictest25621-21fschecks_count21:02:043
1979999603,48cyclictest11042-21fschecks_count20:17:113
1979999602,51cyclictest7598-21grep22:27:173
1979999602,51cyclictest16067-21latency20:32:113
1979999602,51cyclictest16067-21latency20:32:113
1979999602,51cyclictest15879-21runrttasks22:42:063
1979999602,51cyclictest0-21swapper/323:22:003
1979999602,50cyclictest29008-21rm23:04:353
1979999602,48cyclictest4488-21/usr/sbin/munin19:57:053
19794996038,13cyclictest1316-21apt-get22:16:582
19794996034,18cyclictest10088-21grep20:12:232
19794996034,18cyclictest10088-21grep20:12:232
19794996033,19cyclictest30264-21grep23:07:102
19794996033,19cyclictest23983-21fschecks_count20:57:042
19794996032,19cyclictest21322-21date22:51:432
19794996032,17cyclictest7710-21/usr/sbin/munin20:07:022
19794996030,23cyclictest21212-21/usr/sbin/munin23:47:182
19794996030,23cyclictest21212-21/usr/sbin/munin23:47:182
19792996033,8cyclictest141rcu_preempt21:49:281
19792996033,8cyclictest141rcu_preempt20:32:161
19792996033,8cyclictest141rcu_preempt20:32:161
19792996021,25cyclictest141rcu_preempt19:12:201
19792996021,25cyclictest141rcu_preempt19:12:201
19799995940,12cyclictest362-21in:imklog21:51:393
1979999593,48cyclictest30830-21find19:37:203
1979999593,46cyclictest12451-21apt-get20:21:593
1979999593,46cyclictest12451-21apt-get20:21:593
1979999592,49cyclictest24094-21/usr/sbin/munin20:57:063
1979999592,49cyclictest20774-21grep20:47:033
1979999592,49cyclictest19900-21cat21:52:053
1979999592,49cyclictest11747-21/usr/sbin/munin21:37:043
1979999592,48cyclictest7861-21/usr/sbin/munin20:07:073
1979999592,48cyclictest25603-21fschecks_count22:02:053
1979999592,48cyclictest23739-21/usr/sbin/munin23:52:183
19794995941,13cyclictest19785-21cyclictest19:52:252
19794995941,13cyclictest19785-21cyclictest19:52:252
19794995939,13cyclictest0-21swapper/222:52:002
19794995933,17cyclictest19104-21grep20:42:062
19794995930,22cyclictest25063-21find22:57:202
19794995930,22cyclictest25063-21find22:57:202
19794995930,21cyclictest13890-21date20:26:422
19794995930,21cyclictest13890-21date20:26:422
19794995919,16cyclictest400-21runrttasks23:04:572
1979999583,45cyclictest14759-21switchtime20:27:203
19794995833,18cyclictest13841-21grep22:37:232
19794995832,19cyclictest5255-21sed19:57:222
19794995830,20cyclictest422-21/usr/sbin/munin23:12:082
19794995827,23cyclictest29754-21/usr/sbin/munin00:02:202
19792995839,10cyclictest0-21swapper/119:47:221
19792995837,8cyclictest141rcu_preempt20:44:061
1979999572,46cyclictest22173-21apt-get20:51:573
19792995720,10cyclictest0-21swapper/120:37:231
19792995711,30cyclictest0-21swapper/120:28:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional