You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-04-01 - 15:26

ARM Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7 (updated Sat Apr 01, 2023 12:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
212859910836,38cyclictest0-21swapper/011:17:550
212859910745,30cyclictest5328-21/usr/sbin/munin12:03:040
212859910745,30cyclictest5328-21/usr/sbin/munin12:03:040
212859910541,21cyclictest9920-21latency_hist11:17:380
212859910044,30cyclictest429-21diskmemload11:01:490
21301999977,8cyclictest33-21ksoftirqd/207:02:552
21285999942,21cyclictest3827-21grep11:07:390
21285999938,18cyclictest31002-21fschecks_time09:08:020
21285999937,18cyclictest19726-21date12:27:370
21285999845,17cyclictest397-21runrttasks07:48:050
21285999845,17cyclictest397-21runrttasks07:48:050
21285999840,28cyclictest4335-21ssh09:18:480
21285999840,28cyclictest4335-21ssh09:18:480
21285999831,28cyclictest14848-21fschecks_count09:37:570
21285999745,18cyclictest1-21systemd11:52:360
21285999745,18cyclictest1-21systemd11:52:360
21285999738,18cyclictest19911-21sed08:38:090
21285999645,11cyclictest24019-21sort09:53:030
21285999542,20cyclictest9030-21cut09:28:060
21285999538,20cyclictest559-21apt-get07:37:350
21285999454,33cyclictest19296-21ssh11:33:020
21285999443,20cyclictest21618-21grep08:43:130
21285999438,26cyclictest14201-21df_inode12:18:030
1973729410,49sleep219738-21cpu06:57:552
21285999342,18cyclictest27474-21diskmemload12:07:500
21285999339,32cyclictest700-21if_eth010:08:060
21301999231,16cyclictest0-21swapper/210:02:352
21285999243,27cyclictest15937-21latency_hist08:27:380
21285999242,21cyclictest16460-21grep08:28:060
21285999240,19cyclictest21582-21ssh10:42:590
21285999232,36cyclictest0-21swapper/011:38:310
21301999129,11cyclictest0-21swapper/212:28:002
21285999141,20cyclictest9203-21ssh10:22:380
21285999140,20cyclictest18620-21fschecks_count10:38:030
21285999040,19cyclictest3988-21ssh11:07:400
21285999040,17cyclictest15806-21ssh11:27:360
21285999040,17cyclictest15806-21ssh11:27:360
21285999039,24cyclictest13695-21ssh12:17:380
21285998941,17cyclictest30087-21apt-get10:57:370
21285998941,16cyclictest24185-21dpkg08:52:370
21285998938,18cyclictest30387-21latency_hist09:07:380
21285998928,28cyclictest31626-21fschecks_count11:53:090
21301998828,12cyclictest0-21swapper/212:13:012
21285998841,20cyclictest25118-21mailstats08:53:230
21285998839,21cyclictest2932-21sed07:43:040
21285998836,19cyclictest30159-21expr07:28:030
21285998743,35cyclictest24638-21fschecks_count10:48:020
21285998734,21cyclictest2248-21/usr/sbin/munin11:58:060
21285998662,17cyclictest16142-21ssh11:27:570
21285998647,32cyclictest0-21swapper/010:13:470
21285998646,16cyclictest21205-21latency09:48:050
21285998642,36cyclictest6798-21unixbench_singl07:53:230
21285998642,36cyclictest6798-21unixbench_singl07:53:230
21285998638,17cyclictest18537-21mailstats08:33:180
21285998634,17cyclictest0-21swapper/007:13:060
21311998560,15cyclictest22700-21idleruntime-cro12:32:353
21301998522,11cyclictest0-21swapper/209:37:372
21285998541,16cyclictest11091-21sendmail08:12:350
21285998539,19cyclictest22670-21cron12:32:350
21285998537,26cyclictest0-21swapper/007:03:200
21285998537,16cyclictest32081-21ssh10:06:440
21285998529,23cyclictest0-21swapper/009:35:230
2128599852,54cyclictest24033-21kworker/u8:208:58:040
21291998438,34cyclictest0-21swapper/107:02:551
21285998461,16cyclictest397-21runrttasks10:29:090
21285998444,32cyclictest17088-21ssh10:34:590
21285998439,17cyclictest27643-21sort07:18:160
21285998439,16cyclictest0-21swapper/011:47:300
21285998434,18cyclictest18570-21ssh09:43:130
21285998434,18cyclictest18570-21ssh09:43:130
2109228413,43sleep021115-21apt-config07:02:370
21285998339,14cyclictest415-21snmpd09:58:020
21285998337,19cyclictest1348-21ssh09:13:240
21285998238,16cyclictest9517-21cstates10:23:000
21285998236,17cyclictest9453-21latency_hist08:07:380
2130199813,30cyclictest30915-21apt-get11:52:452
21291998136,34cyclictest0-21swapper/108:57:591
21285998155,20cyclictest0-21swapper/007:38:010
21285998155,20cyclictest0-21swapper/007:38:010
21285998127,17cyclictest0-21swapper/009:23:100
21301997959,13cyclictest23342-21cron09:52:352
21285997931,17cyclictest13324-21cat08:18:090
21285997931,17cyclictest13324-21cat08:18:090
21285997854,17cyclictest28968-21ps07:23:100
21285997834,16cyclictest12399-21taskset08:13:440
21285997829,18cyclictest25075-21sed07:12:380
21301997757,14cyclictest6180-21fschecks_time09:23:002
21301997748,12cyclictest33-21ksoftirqd/209:57:372
21301997646,13cyclictest33-21ksoftirqd/207:08:022
21291997634,14cyclictest141rcu_preempt11:38:121
21301997545,12cyclictest33-21ksoftirqd/210:43:022
21301997510,26cyclictest0-21swapper/208:52:532
21285997528,20cyclictest7538-21cpuspeed07:58:000
2105027510,35sleep321083-21munin-run07:02:363
21311997412,26cyclictest0-21swapper/311:20:473
21301997453,8cyclictest33-21ksoftirqd/212:27:352
21301997442,9cyclictest33-21ksoftirqd/207:22:382
21311997326,20cyclictest6129-21fschecks_time07:53:113
21311997326,20cyclictest6129-21fschecks_time07:53:113
21311997221,20cyclictest7001-21/usr/sbin/munin10:18:163
21291997254,11cyclictest3859-21munin-run07:47:351
21291997232,12cyclictest141rcu_preempt12:03:121
21291997232,12cyclictest141rcu_preempt12:03:121
21291997221,29cyclictest0-21swapper/109:52:341
21291997215,39cyclictest5849-21apt-get07:52:561
21291997215,39cyclictest5849-21apt-get07:52:561
21311997012,24cyclictest397-21runrttasks08:47:433
21301997046,17cyclictest3735-21munin-plugin-st11:07:352
2130199702,45cyclictest415-21snmpd10:58:002
21301997024,13cyclictest141rcu_preempt10:13:122
21291997038,13cyclictest141rcu_preempt12:08:201
208002705,44sleep10-21swapper/106:59:201
21311996945,13cyclictest116-21pr/ttymxc011:10:113
21301996948,15cyclictest33-21ksoftirqd/211:33:232
21291996946,17cyclictest9790-21unixbench_multi09:28:201
21291996940,11cyclictest141rcu_preempt11:02:311
2129199693,23cyclictest10130-21apt-get11:17:571
21311996839,16cyclictest24057-21kworker/u8:010:52:383
21311996820,21cyclictest369-21dbus-daemon11:02:363
21311996813,28cyclictest24972-21ntp_states08:53:183
21311996813,24cyclictest8817-21/usr/sbin/munin08:02:373
21301996841,10cyclictest33-21ksoftirqd/208:28:192
21301996837,8cyclictest33-21ksoftirqd/207:17:382
21291996839,9cyclictest141rcu_preempt07:23:141
21291996820,31cyclictest24051-21apt-get08:52:461
21311996713,25cyclictest10266-21open_files08:08:103
21311996712,30cyclictest15941-21ssh09:38:473
21311996712,20cyclictest19654-21ssh09:45:383
21311996712,20cyclictest19654-21ssh09:45:383
21301996751,10cyclictest22435-21idleruntime-cro08:47:352
21301996742,17cyclictest30088-21munin-plugin-st10:57:372
21291996745,12cyclictest0-21swapper/110:38:171
21291996736,12cyclictest141rcu_preempt10:43:061
21291996735,15cyclictest415-21snmpd12:21:121
21311996635,21cyclictest25239-21apt-get11:42:543
21311996620,19cyclictest9290-21netstat09:28:123
21311996618,18cyclictest397-21runrttasks10:54:303
21311996617,18cyclictest15730-21taskset08:25:073
21311996615,21cyclictest16133-21apt-get11:27:583
21311996614,18cyclictest27474-21diskmemload11:17:023
21311996613,34cyclictest0-21swapper/311:33:053
21311996613,22cyclictest18090-21latency_hist10:37:373
21301996642,17cyclictest0-21swapper/212:12:382
21301996638,11cyclictest33-21ksoftirqd/209:42:492
21301996638,11cyclictest33-21ksoftirqd/209:42:492
2130199663,34cyclictest30236-21fschecks_count07:28:072
21301996628,15cyclictest33-21ksoftirqd/211:38:062
21291996649,11cyclictest6752-21latency_hist11:12:351
21291996648,11cyclictest553-21rs:main1
21291996640,11cyclictest141rcu_preempt07:58:191
21291996624,11cyclictest141rcu_preempt11:03:061
21311996513,19cyclictest0-21swapper/308:43:153
21301996512,21cyclictest141rcu_preempt09:13:392
21311996439,13cyclictest11625-21apt-get09:32:543
21311996416,16cyclictest9249-21runrttasks08:05:123
21301996441,14cyclictest28665-21runrttasks09:03:152
21301996429,13cyclictest33-21ksoftirqd/207:43:132
21291996446,11cyclictest16321-21timerwakeupswit10:33:141
21291996441,12cyclictest0-21swapper/111:23:201
21291996441,12cyclictest0-21swapper/111:23:201
21291996434,17cyclictest26141-21kworker/u8:309:37:321
21291996430,12cyclictest141rcu_preempt10:07:411
21291996430,12cyclictest141rcu_preempt09:22:361
21291996430,12cyclictest141rcu_preempt09:22:361
21291996426,20cyclictest0-21swapper/109:41:041
2131199632,42cyclictest31598-21diskstats11:53:083
21311996316,19cyclictest6430-21ssh12:04:233
21311996316,19cyclictest6430-21ssh12:04:233
21311996314,20cyclictest30915-21apt-get11:52:373
21311996314,20cyclictest30915-21apt-get11:52:373
21311996314,19cyclictest27474-21diskmemload09:24:323
21311996314,19cyclictest23217-21cat11:38:193
21301996344,13cyclictest8206-21runrttasks07:58:182
21301996344,12cyclictest7425-21grep10:18:222
21301996338,17cyclictest5045-21runrttasks07:48:162
21301996338,17cyclictest5045-21runrttasks07:48:162
21301996332,13cyclictest33-21ksoftirqd/212:02:572
21301996332,13cyclictest33-21ksoftirqd/212:02:572
21291996336,15cyclictest141rcu_preempt12:17:381
21291996334,17cyclictest0-21swapper/109:12:361
21291996329,18cyclictest141rcu_preempt10:27:381
21291996323,25cyclictest0-21swapper/107:37:521
21291996323,25cyclictest0-21swapper/107:37:521
21291996319,26cyclictest25-21ksoftirqd/108:08:071
21291996316,26cyclictest25-21ksoftirqd/110:55:221
21311996233,19cyclictest0-21swapper/310:42:553
21311996231,23cyclictest10396-21grep10:23:203
21311996229,20cyclictest13141-21apt-get11:22:583
21311996229,20cyclictest13141-21apt-get11:22:583
21311996227,14cyclictest141rcu_preempt11:07:403
21311996214,19cyclictest11345-21taskset08:12:433
21311996212,20cyclictest22161-21grep07:03:253
21311996212,18cyclictest27474-21diskmemload09:13:443
21301996244,13cyclictest18789-21grep11:32:352
21301996244,11cyclictest0-21swapper/208:17:352
21301996217,18cyclictest33-21ksoftirqd/207:38:112
21301996217,18cyclictest33-21ksoftirqd/207:38:112
21291996239,12cyclictest0-21swapper/111:48:161
21291996239,12cyclictest0-21swapper/111:48:161
21291996230,18cyclictest141rcu_preempt11:43:081
21291996227,11cyclictest141rcu_preempt10:02:361
21311996113,19cyclictest25215-21sh09:54:233
2130199619,40cyclictest28113-21kworker/u8:007:53:002
2130199619,40cyclictest28113-21kworker/u8:007:53:002
2130199618,10cyclictest0-21swapper/207:33:002
21301996137,15cyclictest10475-21threads08:08:132
21301996121,10cyclictest141rcu_preempt10:37:382
21291996144,11cyclictest1578-21idleruntime-cro11:57:351
21291996131,11cyclictest141rcu_preempt07:18:151
21291996130,9cyclictest141rcu_preempt08:27:381
2131199607,18cyclictest23971-21memory07:08:073
21311996019,13cyclictest0-21swapper/309:08:493
21311996017,17cyclictest10307-21sh12:11:473
21311996016,18cyclictest27474-21diskmemload10:09:123
21311996014,16cyclictest16607-21sh12:22:293
2130199608,13cyclictest543-21kworker/2:109:12:362
21301996035,14cyclictest0-21swapper/211:47:372
21301996031,14cyclictest116-21pr/ttymxc008:58:302
2129199607,36cyclictest25457-21unixbench_singl10:48:171
21291996032,20cyclictest231rcuc/109:23:141
21291996029,13cyclictest141rcu_preempt08:37:381
21291996026,7cyclictest141rcu_preempt09:03:031
21291996024,7cyclictest141rcu_preempt10:13:051
21291996017,21cyclictest141rcu_preempt11:28:061
21311995942,10cyclictest26919-21timerandwakeup08:58:303
21311995915,16cyclictest27474-21diskmemload12:00:363
21311995912,17cyclictest25702-21ntp_states07:13:103
21311995911,20cyclictest27474-21diskmemload12:13:403
21311995911,19cyclictest3190-21sensors_temp07:43:123
21301995937,12cyclictest0-21swapper/212:18:192
21301995936,14cyclictest33-21ksoftirqd/210:08:182
21301995934,16cyclictest552-21in:imklog09:28:202
21291995923,8cyclictest141rcu_preempt08:42:371
21291995921,16cyclictest141rcu_preempt12:28:001
21291995916,13cyclictest141rcu_preempt10:08:041
2131199588,18cyclictest19965-21ntpq08:38:123
21311995833,16cyclictest18922-21taskset10:38:083
21311995832,14cyclictest0-21swapper/307:28:213
21311995826,24cyclictest14312-21modprobe08:22:363
21311995826,24cyclictest14312-21modprobe08:22:363
21311995814,18cyclictest23366-21/usr/sbin/munin09:52:363
21311995814,17cyclictest27474-21diskmemload10:00:273
21301995836,12cyclictest0-21swapper/208:33:202
21301995834,14cyclictest0-21swapper/211:13:182
21301995822,10cyclictest141rcu_preempt11:47:562
21301995822,10cyclictest141rcu_preempt11:47:562
21301995817,14cyclictest5698-21latency_hist09:22:382
21301995817,14cyclictest5698-21latency_hist09:22:382
2129199588,26cyclictest21270-21cyclictest09:44:181
2129199588,26cyclictest21270-21cyclictest09:44:181
21291995839,10cyclictest0-21swapper/108:22:361
21291995839,10cyclictest0-21swapper/108:22:361
2129199583,36cyclictest22948-21grep08:48:051
2129199583,24cyclictest25-21ksoftirqd/107:12:351
21291995814,17cyclictest141rcu_preempt11:14:361
21311995736,15cyclictest552-21in:imklog08:28:153
2130199579,9cyclictest1937-21apt-get11:57:572
21301995733,13cyclictest0-21swapper/211:08:052
21291995736,14cyclictest379-21systemd-logind08:17:361
21291995734,12cyclictest0-21swapper/111:33:121
21291995728,16cyclictest0-21swapper/107:48:161
21291995728,16cyclictest0-21swapper/107:48:161
21311995618,30cyclictest0-21swapper/312:23:073
21311995611,9cyclictest141rcu_preempt09:22:363
21311995611,9cyclictest141rcu_preempt09:22:363
21311995610,19cyclictest30287-21ssh09:07:213
21301995637,12cyclictest0-21swapper/210:07:402
21301995635,14cyclictest15774-21munin-plugin-st11:27:352
21301995635,14cyclictest15774-21munin-plugin-st11:27:352
21301995633,13cyclictest0-21swapper/210:32:362
21301995622,12cyclictest33-21ksoftirqd/210:38:102
2130199562,13cyclictest141rcu_preempt10:23:202
21291995639,10cyclictest4591-21idleruntime-cro12:02:351
21291995626,13cyclictest141rcu_preempt08:43:201
21291995625,9cyclictest141rcu_preempt07:12:591
21291995615,25cyclictest0-21swapper/107:28:131
21291995611,30cyclictest0-21swapper/109:53:151
21311995530,13cyclictest0-21swapper/308:37:363
21311995526,20cyclictest1728-21/usr/sbin/munin07:38:233
21311995526,20cyclictest1728-21/usr/sbin/munin07:38:233
21311995511,19cyclictest29790-21ssh10:02:593
21301995523,12cyclictest141rcu_preempt10:50:142
21301995511,20cyclictest141rcu_preempt07:23:122
21291995537,10cyclictest0-21swapper/112:24:271
21291995523,8cyclictest141rcu_preempt10:32:381
21291995516,17cyclictest141rcu_preempt09:14:361
2131199549,17cyclictest26939-21df_inode07:18:013
21311995436,13cyclictest552-21in:imklog07:23:023
21311995422,24cyclictest3599-21fschecks_count10:13:053
21301995437,9cyclictest33-21ksoftirqd/208:48:042
21301995437,11cyclictest15691-21unixbench_singl09:38:122
21301995430,7cyclictest33-21ksoftirqd/208:38:052
2129199546,26cyclictest0-21swapper/107:32:501
21291995424,10cyclictest141rcu_preempt08:28:061
21311995332,9cyclictest4130-21apt-get07:47:533
21311995332,9cyclictest4130-21apt-get07:47:533
21311995320,24cyclictest31868-21tune2fs07:33:003
21301995333,12cyclictest0-21swapper/208:23:182
21311995217,26cyclictest13604-21ssh10:28:433
2130199527,22cyclictest141rcu_preempt08:03:472
2129199514,20cyclictest25-21ksoftirqd/108:07:371
21301995013,18cyclictest141rcu_preempt11:18:192
21301994610,13cyclictest141rcu_preempt08:18:212
21301994610,13cyclictest141rcu_preempt08:18:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional