You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-11-28 - 09:30

ARM Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7 (updated Mon Nov 28, 2022 00:44:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2716829416,50sleep227175-21latencyplot19:07:542
2701929117,47sleep327062-21awk19:07:513
27239998939,19cyclictest32739-21chrt19:23:041
27228998529,21cyclictest6575-21basename19:42:500
27228998227,19cyclictest32741-21taskset19:23:060
27228998147,17cyclictest6409-21taskset21:20:410
27239998027,21cyclictest25953-21taskset22:20:421
27250997933,7cyclictest141rcu_preempt20:58:222
2725099793,33cyclictest33-21ksoftirqd/221:23:112
27250997932,8cyclictest141rcu_preempt19:58:222
27228997943,27cyclictest31184-21/usr/sbin/munin20:58:180
27228997929,19cyclictest8454-21taskset19:48:080
2725099787,43cyclictest5325-21cstates19:38:132
27250997811,41cyclictest5437-21ls22:58:132
27239997829,19cyclictest21046-21chrt22:05:101
27228997831,18cyclictest12541-21awk23:18:260
27228997827,19cyclictest20595-21sed20:23:260
27228997825,19cyclictest5344-21systemd-run00:37:510
27250997632,7cyclictest141rcu_preempt22:53:232
27228997627,20cyclictest8096-21seq23:05:070
27228997625,29cyclictest29090-21taskset20:50:240
27250997535,13cyclictest141rcu_preempt21:12:532
27250997515,21cyclictest11944-21cpu23:18:112
27239997535,14cyclictest0-21swapper/120:05:501
27228997526,18cyclictest396-21runrttasks22:33:160
27228997525,18cyclictest6431-21runrttasks22:59:210
27228997525,18cyclictest0-21swapper/023:33:060
27250997427,25cyclictest0-21swapper/223:48:132
27239997448,18cyclictest4957-21apt-key19:37:521
27239997448,18cyclictest4957-21apt-key19:37:521
27228997430,36cyclictest29677-21grep00:12:520
27228997430,36cyclictest29677-21grep00:12:520
27228997429,17cyclictest19937-21apt-get22:03:120
27228997427,13cyclictest0-21swapper/023:23:170
27228997425,19cyclictest27423-21seq22:23:480
27228997422,30cyclictest15910-21runrttasks20:08:310
27250997343,22cyclictest1-21systemd21:22:512
27250997338,8cyclictest141rcu_preempt00:02:502
27250997332,19cyclictest141rcu_preempt23:03:102
27228997330,35cyclictest24475-21sendmail-msp20:37:500
27228997327,18cyclictest8011-21seq21:24:360
27228997326,19cyclictest32392-21taskset22:40:140
27228997324,17cyclictest0-21swapper/019:34:300
27228997324,17cyclictest0-21swapper/019:34:300
27228997323,21cyclictest12871-21taskset21:39:380
27228997318,21cyclictest25162-21latency20:38:250
2725099726,46cyclictest3370-21kworker/u8:000:23:102
27250997246,16cyclictest18094-21kworker/u8:223:37:502
27250997232,19cyclictest141rcu_preempt20:53:302
27250997228,18cyclictest0-21swapper/221:13:022
27250997228,18cyclictest0-21swapper/221:13:022
27250997227,9cyclictest0-21swapper/200:13:102
27239997249,15cyclictest16812-21/usr/sbin/munin20:13:151
27239997240,23cyclictest24582-21munin-run23:57:501
27239997225,18cyclictest32337-21taskset22:39:301
27239997223,17cyclictest1507-21taskset22:44:371
27228997251,8cyclictest13409-21apt-get21:43:090
27228997231,14cyclictest2777-21sed19:28:290
27228997224,21cyclictest3067-21taskset21:09:350
27228997223,20cyclictest17973-21taskset21:57:320
27250997142,13cyclictest15457-21ntpq20:08:222
27239997155,10cyclictest6600-21idleruntime-cro21:22:501
27239997139,22cyclictest3938-21taskset00:33:021
27239997123,20cyclictest27990-21taskset22:27:531
27228997131,33cyclictest24542-21idleruntime-cro22:17:510
27228997131,33cyclictest24542-21idleruntime-cro22:17:510
27228997125,18cyclictest19347-21seq21:59:030
27228997125,16cyclictest22792-21taskset22:11:580
27228997124,20cyclictest4618-21taskset21:13:520
27228997124,20cyclictest4618-21taskset21:13:520
27228997124,17cyclictest1-21systemd23:47:520
27228997122,18cyclictest30877-21runrttasks20:57:410
27228997121,17cyclictest0-21swapper/020:13:140
27262997051,12cyclictest27026-21kworker/u8:220:57:513
27250997035,10cyclictest141rcu_preempt20:13:092
27250997032,14cyclictest141rcu_preempt00:37:522
27250997030,8cyclictest141rcu_preempt22:02:532
27239997045,16cyclictest13853-21/usr/sbin/munin21:43:221
27239997045,16cyclictest12140-21/usr/sbin/munin19:58:181
27228997044,17cyclictest11693-21apt-get19:58:030
27228997029,32cyclictest8530-21/usr/sbin/munin21:28:030
27228997028,35cyclictest0-21swapper/021:48:140
27228997028,35cyclictest0-21swapper/021:48:140
27228997028,34cyclictest24755-21date23:57:550
27228997026,17cyclictest0-21swapper/000:17:100
27228997025,16cyclictest0-21swapper/022:48:120
27228997024,19cyclictest19296-21taskset20:20:130
27228997020,21cyclictest18375-21/usr/sbin/munin23:38:170
27250996934,15cyclictest141rcu_preempt22:22:512
27250996924,21cyclictest141rcu_preempt21:51:272
27250996924,21cyclictest141rcu_preempt21:51:272
27250996918,26cyclictest0-21swapper/219:52:532
27239996947,14cyclictest31644-21memory20:58:171
27239996946,16cyclictest30426-21unixbench_singl19:13:271
27239996944,16cyclictest6314-21unixbench_multi22:58:431
27228996926,15cyclictest26321-21taskset20:43:060
27228996925,17cyclictest22649-21taskset20:30:470
27228996923,16cyclictest0-21swapper/023:59:360
27228996922,17cyclictest13295-21/usr/sbin/munin20:03:240
27262996823,15cyclictest0-21swapper/322:01:433
27250996840,13cyclictest0-21swapper/220:43:322
27250996830,8cyclictest141rcu_preempt22:28:242
27250996823,22cyclictest141rcu_preempt20:03:142
27250996811,26cyclictest321ktimers/222:48:252
27239996842,18cyclictest2037-21apt-get22:48:051
27239996840,19cyclictest26341-21apt-get20:43:091
27239996827,16cyclictest0-21swapper/119:08:301
27239996823,17cyclictest9828-21taskset23:10:341
27239996822,18cyclictest22786-21taskset20:32:351
27228996850,11cyclictest8186-21munin-plugin-st19:47:510
27228996836,24cyclictest2179-21cat00:27:520
27228996823,18cyclictest1597-21taskset22:45:460
27262996751,9cyclictest24578-21munin-plugin-st23:57:503
27250996746,8cyclictest141rcu_preempt20:33:132
27250996733,9cyclictest141rcu_preempt20:18:242
2725099673,29cyclictest0-21swapper/223:08:152
27250996732,7cyclictest141rcu_preempt19:28:212
27250996727,6cyclictest0-21swapper/223:38:172
27239996743,17cyclictest29811-21fschecks_time20:53:181
27239996743,16cyclictest7183-21fschecks_count23:03:151
27239996742,15cyclictest21805-21grep23:48:121
27239996741,17cyclictest11208-21tail23:13:331
27239996740,18cyclictest8446-21apt-get19:48:051
27239996737,19cyclictest12686-21/usr/sbin/munin23:18:281
27239996723,17cyclictest396-21runrttasks19:40:561
27239996723,16cyclictest396-21runrttasks21:54:501
27239996723,16cyclictest396-21runrttasks19:43:431
27228996741,18cyclictest31450-21/usr/sbin/munin00:18:250
27228996740,17cyclictest4150-21/usr/sbin/munin22:53:180
27228996723,16cyclictest0-21swapper/023:10:250
27228996723,15cyclictest0-21swapper/023:48:080
27228996721,25cyclictest0-21swapper/019:20:070
270112675,40sleep00-21swapper/019:07:440
27262996645,12cyclictest0-21swapper/323:17:523
27250996636,17cyclictest141rcu_preempt23:13:202
27250996631,12cyclictest141rcu_preempt23:43:172
27250996629,9cyclictest141rcu_preempt19:53:082
27250996629,9cyclictest141rcu_preempt19:53:082
27250996628,9cyclictest141rcu_preempt21:57:522
27250996625,9cyclictest0-21swapper/222:33:332
2725099662,25cyclictest0-21swapper/219:08:472
27239996641,16cyclictest16030-21unixbench_singl21:48:341
27239996641,16cyclictest16030-21unixbench_singl21:48:341
27239996640,20cyclictest4323-21switchtime21:13:251
27239996640,20cyclictest4323-21switchtime21:13:251
27239996638,18cyclictest23475-21/usr/sbin/munin22:13:221
27239996638,18cyclictest23475-21/usr/sbin/munin22:13:221
27239996637,21cyclictest20956-21grep23:43:261
27239996621,18cyclictest396-21runrttasks23:35:571
27228996649,8cyclictest26485-21apt-get00:03:070
27228996638,21cyclictest27492-21taskset19:08:400
2709326610,29sleep127136-21munin-run19:07:531
27262996548,10cyclictest5303-21munin-plugin-st00:37:503
27262996540,17cyclictest30287-21netstat00:13:163
27250996533,8cyclictest141rcu_preempt23:28:082
27250996528,20cyclictest0-21swapper/222:13:322
27250996528,20cyclictest0-21swapper/222:13:322
27250996523,19cyclictest0-21swapper/222:43:262
27250996520,23cyclictest141rcu_preempt19:33:202
27250996520,23cyclictest141rcu_preempt19:33:202
27239996545,12cyclictest363-21apt-get21:03:061
27239996542,15cyclictest2404-21/usr/sbin/munin19:28:211
27239996541,16cyclictest19388-21taskset20:21:231
27239996538,19cyclictest8765-21fschecks_count21:28:131
27239996537,20cyclictest21776-21fschecks_time22:08:131
27239996536,21cyclictest31370-21fschecks_time19:18:141
27239996536,21cyclictest21129-21apt20:27:501
27239996532,23cyclictest3332-21sh21:12:501
27228996536,22cyclictest26136-21idleruntime-cro22:22:500
27228996525,13cyclictest11339-21taskset21:35:580
27228996525,13cyclictest11339-21taskset21:35:580
27228996522,17cyclictest11166-21taskset19:55:030
27228996522,17cyclictest11166-21taskset19:55:030
27228996522,16cyclictest30561-21runrttasks19:14:350
27228996521,16cyclictest0-21swapper/023:16:110
2725099644,29cyclictest396-21runrttasks21:42:012
27250996433,8cyclictest141rcu_preempt21:42:532
27250996432,12cyclictest141rcu_preempt21:07:372
27250996430,13cyclictest141rcu_preempt20:48:102
27250996429,7cyclictest141rcu_preempt22:23:172
27250996428,8cyclictest0-21swapper/220:28:422
2725099642,40cyclictest396-21runrttasks22:41:242
27250996415,10cyclictest0-21swapper/223:53:242
27239996445,9cyclictest15148-21apt-get23:28:031
27239996438,19cyclictest10388-21grep21:33:211
27239996438,19cyclictest10388-21grep21:33:211
27239996438,18cyclictest3871-21fschecks_time22:53:121
27239996438,18cyclictest25424-21/usr/sbin/munin23:58:211
27239996437,19cyclictest23685-21sshd20:33:271
27239996436,20cyclictest32163-21switchtime00:18:241
27239996436,20cyclictest31174-21cron00:17:491
27239996435,19cyclictest18953-21/usr/sbin/munin23:38:241
27262996341,15cyclictest0-21swapper/322:57:503
27250996339,13cyclictest0-21swapper/200:08:352
27250996339,13cyclictest0-21swapper/200:08:352
27250996334,9cyclictest141rcu_preempt22:08:232
27250996334,8cyclictest141rcu_preempt19:44:162
27250996333,11cyclictest141rcu_preempt00:03:272
27250996331,10cyclictest141rcu_preempt20:38:162
27250996322,9cyclictest0-21swapper/221:28:062
27239996342,13cyclictest11548-21latency_hist19:57:531
27239996342,13cyclictest11548-21latency_hist19:57:531
27239996341,14cyclictest28326-21ls00:08:161
27239996341,14cyclictest28326-21ls00:08:161
27239996340,15cyclictest31154-21cut22:37:521
27239996338,16cyclictest6936-21apt-get21:23:051
27239996338,15cyclictest15363-21/usr/sbin/munin20:08:171
27239996338,14cyclictest28924-21grep20:48:291
27239996337,18cyclictest27832-21latency_hist22:27:521
27239996335,16cyclictest26656-21grep00:03:151
27239996332,21cyclictest18713-21/usr/sbin/munin21:58:171
27239996318,18cyclictest396-21runrttasks20:41:431
27228996341,14cyclictest28025-21apt-get22:28:080
27228996336,20cyclictest16115-21sed23:28:260
27228996323,14cyclictest1529-21taskset21:05:370
27250996234,7cyclictest141rcu_preempt00:18:132
27250996226,11cyclictest360-21in:imklog19:18:192
27239996241,14cyclictest0-21swapper/100:28:071
27239996234,18cyclictest13759-21/usr/sbin/munin23:23:141
27239996220,18cyclictest396-21runrttasks00:24:471
27262996138,12cyclictest0-21swapper/321:53:313
27250996127,18cyclictest0-21swapper/219:23:122
27250996127,12cyclictest141rcu_preempt22:03:282
27250996119,9cyclictest0-21swapper/219:12:582
27250996110,23cyclictest33-21ksoftirqd/223:27:262
27239996139,15cyclictest12739-21unixbench_singl21:38:261
27228996140,15cyclictest0-21swapper/000:28:070
27250996039,13cyclictest180-21pr/ttyS021:33:102
27250996039,13cyclictest180-21pr/ttyS021:33:102
27250995926,17cyclictest141rcu_preempt00:28:072
27262995837,14cyclictest0-21swapper/320:32:533
27250995830,8cyclictest141rcu_preempt20:23:162
27262995738,12cyclictest22522-21unixbench_multi23:48:263
27262995734,12cyclictest0-21swapper/300:22:533
27262995722,12cyclictest141rcu_preempt19:43:063
27262995642,7cyclictest0-21swapper/321:23:283
27262995640,8cyclictest0-21swapper/320:12:503
27262995633,12cyclictest0-21swapper/322:48:263
27262995633,12cyclictest0-21swapper/319:32:503
27262995615,14cyclictest0-21swapper/322:42:523
27262995534,14cyclictest15958-21grep21:48:343
27262995534,14cyclictest15958-21grep21:48:343
27262995531,13cyclictest0-21swapper/300:12:533
27262995531,13cyclictest0-21swapper/300:12:533
2726299552,22cyclictest17772-21chrt23:33:513
27262995438,10cyclictest396-21runrttasks20:23:093
27262995435,13cyclictest274-21systemd-journal23:12:533
27262995434,13cyclictest25153-21kernelversion22:18:143
27262995433,10cyclictest0-21swapper/319:09:083
27262995433,10cyclictest0-21swapper/300:02:493
27262995430,15cyclictest0-21swapper/319:17:513
2726299542,21cyclictest29214-21chrt22:31:073
2726299541,21cyclictest8724-21fschecks_count21:28:123
27262995338,8cyclictest0-21swapper/321:47:503
27262995334,12cyclictest0-21swapper/323:28:263
27262995333,13cyclictest11491-21kthreadd21:37:503
27262995333,13cyclictest11491-21kthreadd21:37:503
27262995330,12cyclictest0-21swapper/322:43:263
27262995329,12cyclictest0-21swapper/321:38:143
27262995230,12cyclictest0-21swapper/322:07:533
27262995129,11cyclictest0-21swapper/323:02:513
27262995129,11cyclictest0-21swapper/320:58:123
27262995128,12cyclictest0-21swapper/320:22:543
27262995125,19cyclictest1726-21sh21:07:513
2726299512,21cyclictest14779-21runrttasks23:26:453
27262995036,7cyclictest0-21swapper/323:43:253
27262995035,8cyclictest0-21swapper/319:18:283
27262995029,10cyclictest0-21swapper/319:23:113
27262995026,12cyclictest0-21swapper/300:28:133
27262994932,10cyclictest24436-21sh20:37:503
27262994928,11cyclictest0-21swapper/320:52:533
27262994927,11cyclictest0-21swapper/323:18:173
27262994926,10cyclictest0-21swapper/320:13:143
2726299492,17cyclictest22601-21chrt22:09:293
2726299491,19cyclictest27448-21runrttasks22:24:093
27262994834,7cyclictest0-21swapper/321:08:073
27262994833,9cyclictest14688-21systemd20:07:523
2726299483,19cyclictest7874-21grep23:03:293
27262994830,8cyclictest0-21swapper/321:18:143
27262994828,10cyclictest0-21swapper/320:43:193
2726299481,18cyclictest27810-21taskset00:06:203
2726299478,31cyclictest0-21swapper/321:17:533
2726299478,31cyclictest0-21swapper/321:17:533
27262994733,7cyclictest0-21swapper/322:37:503
27262994727,9cyclictest0-21swapper/320:38:203
27262994724,12cyclictest0-21swapper/319:48:053
27262994724,11cyclictest0-21swapper/322:13:203
27262994724,11cyclictest0-21swapper/322:13:203
27262994722,15cyclictest0-21swapper/319:33:313
27262994722,15cyclictest0-21swapper/319:33:313
2726299472,17cyclictest396-21runrttasks19:39:393
2726299464,32cyclictest396-21runrttasks00:27:503
27262994632,7cyclictest0-21swapper/319:57:503
27262994632,7cyclictest0-21swapper/319:57:503
27262994626,10cyclictest0-21swapper/323:38:263
27262994216,12cyclictest0-21swapper/319:58:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional