You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-05 - 01:32

ARM Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7 (updated Sun Dec 04, 2022 12:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27288216530,22sleep30-21swapper/307:03:063
2817899987,22cyclictest28041-21latency_hist07:07:280
2817899973,16cyclictest3015-21taskset07:27:500
267042957,17sleep10-21swapper/107:02:391
2817899933,15cyclictest17858-21fschecks_time08:12:470
2817899929,12cyclictest0-21swapper/007:47:270
2817899922,18cyclictest19468-21chrt11:37:270
2817899914,19cyclictest396-21runrttasks09:05:110
2817899913,14cyclictest14290-21/usr/sbin/munin09:43:000
2817899903,19cyclictest2550-21cut07:27:240
2817899903,11cyclictest30171-21kernelversion07:12:340
2817899894,18cyclictest26682-21grep08:38:000
2817899893,24cyclictest6116-21apt-key10:57:250
2817899893,18cyclictest11337-21grep09:32:470
2817899893,17cyclictest13320-21grep09:37:580
2817899883,16cyclictest396-21runrttasks10:59:030
2817899883,14cyclictest9721-21fschecks_time07:48:080
28178998810,71cyclictest0-21swapper/008:47:260
2817899874,19cyclictest23173-21fschecks_count11:47:590
2817899874,19cyclictest23173-21fschecks_count11:47:590
28178998713,10cyclictest0-21swapper/009:12:450
2817899864,17cyclictest30278-21apt08:52:240
2817899864,17cyclictest30278-21apt08:52:240
2817899863,21cyclictest14431-21munin-run11:22:240
2817899863,21cyclictest14431-21munin-run11:22:240
2817899863,20cyclictest9307-21latency_hist09:27:270
2817899863,18cyclictest396-21runrttasks10:02:370
2817899863,16cyclictest16788-21timerwakeupswit09:48:030
2817899862,21cyclictest2889-21grep12:23:130
2817899854,13cyclictest31896-21apt08:57:240
2817899853,21cyclictest29704-21grep07:12:250
2817899853,21cyclictest16920-21grep08:08:130
2817899853,18cyclictest1149-21apt-get10:42:330
2817899853,18cyclictest11250-21latency_hist11:12:260
2817899853,17cyclictest29137-21cron12:07:240
2817899853,15cyclictest26744-21grep10:18:040
28178998513,16cyclictest274-21systemd-journal08:37:240
28178998513,16cyclictest274-21systemd-journal08:37:240
2677228512,35sleep20-21swapper/207:02:452
2817899846,13cyclictest0-21swapper/009:02:240
2817899846,13cyclictest0-21swapper/009:02:240
2817899844,17cyclictest30356-21grep12:07:580
2817899843,20cyclictest7041-21grep07:38:120
2817899843,16cyclictest31190-21wc10:32:550
2817899843,15cyclictest25882-21modprobe11:57:230
2817899843,13cyclictest3790-21mailstats10:48:020
2817899843,13cyclictest3790-21mailstats10:48:020
28185998332,14cyclictest141rcu_preempt12:17:471
2817899835,18cyclictest396-21runrttasks09:57:270
2817899835,18cyclictest396-21runrttasks09:57:270
2817899834,17cyclictest4635-21tune2fs07:32:450
2817899833,21cyclictest3713-21cut12:27:510
2817899833,21cyclictest3713-21cut12:27:510
2817899833,20cyclictest15674-21apt-key08:07:250
2817899833,19cyclictest27670-21fschecks_time10:22:500
2817899833,19cyclictest26284-21cstates11:57:420
2817899833,19cyclictest10791-21cron09:32:240
2817899832,19cyclictest32392-21sendmail12:17:240
2817899832,19cyclictest32392-21sendmail12:17:240
2817899824,20cyclictest11348-21/usr/sbin/munin11:12:510
2817899824,15cyclictest725-21unixbench_multi10:38:000
2817899824,15cyclictest725-21unixbench_multi10:38:000
2817899823,19cyclictest25023-21timerwakeupswit10:13:330
2817899823,18cyclictest21229-21/usr/sbin/munin11:42:590
2817899823,18cyclictest21229-21/usr/sbin/munin11:42:590
2817899823,16cyclictest4439-21grep09:12:260
2817899822,18cyclictest1-21systemd11:07:250
28178998153,18cyclictest6485-21cron12:37:230
2817899813,21cyclictest19201-21/usr/sbin/munin09:57:580
2817899813,16cyclictest20600-21idleruntime-cro08:22:240
2817899812,24cyclictest0-21swapper/007:52:540
2817899812,18cyclictest478-21grep12:17:470
2817899803,20cyclictest541-21grep07:18:010
2817899803,19cyclictest29240-21sort10:27:540
2817899803,19cyclictest29240-21sort10:27:540
2817899803,19cyclictest20568-21unixbench_multi11:38:260
2817899803,17cyclictest22692-21sort08:27:480
2817899803,16cyclictest14674-21/usr/sbin/munin11:22:570
2817899803,16cyclictest12864-21sort07:57:470
2817899802,21cyclictest16724-21cat11:27:530
2817899802,20cyclictest7205-21unixbench_singl09:18:030
28194997943,12cyclictest141rcu_preempt07:49:212
28185997942,19cyclictest141rcu_preempt07:32:411
2817899794,19cyclictest21031-21fschecks_count08:22:460
2817899793,11cyclictest23181-21cat10:08:130
28203997658,11cyclictest32458-21fschecks_count10:37:463
28203997658,11cyclictest32458-21fschecks_count10:37:463
28203997552,14cyclictest12412-21idleruntime-cro07:57:243
28203997527,20cyclictest0-21swapper/311:33:583
28203997423,22cyclictest22648-21fschecks_count10:07:563
28203997423,21cyclictest25367-21taskset08:37:013
28203997423,21cyclictest25367-21taskset08:37:013
28203997420,22cyclictest16991-21/usr/sbin/munin11:28:003
28185997422,12cyclictest0-21swapper/110:30:381
28185997422,12cyclictest0-21swapper/110:30:381
28185997421,30cyclictest0-21swapper/109:26:361
28185997351,14cyclictest0-21swapper/112:37:231
28185997327,20cyclictest0-21swapper/109:32:471
28203997251,13cyclictest1092-21idleruntime-cro09:02:243
28203997251,13cyclictest1092-21idleruntime-cro09:02:243
28185997228,25cyclictest141rcu_preempt11:04:291
28203997125,28cyclictest0-21swapper/308:50:363
28203997125,28cyclictest0-21swapper/308:50:363
28203997121,19cyclictest25372-21chrt10:16:143
28185997140,13cyclictest141rcu_preempt08:22:281
28185997138,12cyclictest141rcu_preempt07:22:451
28185997126,7cyclictest0-21swapper/108:27:591
2820399706,25cyclictest396-21runrttasks12:04:503
28203997045,13cyclictest0-21swapper/307:42:593
28185997039,10cyclictest141rcu_preempt10:23:021
28185997034,9cyclictest141rcu_preempt11:25:021
28203996926,20cyclictest30694-21/usr/sbin/munin08:52:423
28203996916,23cyclictest23965-21latency_hist08:32:273
28203996913,18cyclictest396-21runrttasks11:24:243
28203996912,19cyclictest396-21runrttasks10:07:133
28185996945,17cyclictest18966-21sendmail08:17:241
28185996945,14cyclictest25903-21idleruntime-cro11:57:231
28185996934,9cyclictest0-21swapper/111:00:341
28185996934,11cyclictest141rcu_preempt09:08:501
28185996928,14cyclictest141rcu_preempt09:53:041
28185996928,14cyclictest141rcu_preempt09:53:041
28203996828,21cyclictest14449-21df08:02:433
28203996822,16cyclictest0-21swapper/309:42:123
28194996848,11cyclictest28747-21kworker/u8:110:27:242
28194996833,8cyclictest141rcu_preempt07:25:222
28194996830,14cyclictest141rcu_preempt11:22:552
28194996818,24cyclictest33-21ksoftirqd/207:12:362
2818599688,22cyclictest141rcu_preempt11:57:511
28185996837,10cyclictest141rcu_preempt08:52:461
28185996828,19cyclictest141rcu_preempt07:48:071
2820399675,35cyclictest0-21swapper/309:12:263
28203996713,20cyclictest4921-21/usr/sbin/munin07:32:513
2819499674,34cyclictest33-21ksoftirqd/212:28:112
2819499674,34cyclictest33-21ksoftirqd/212:28:112
28194996734,9cyclictest141rcu_preempt11:07:262
28194996734,7cyclictest141rcu_preempt09:28:492
28194996730,20cyclictest141rcu_preempt08:37:582
28185996733,9cyclictest141rcu_preempt09:48:031
28185996724,22cyclictest141rcu_preempt07:42:361
28185996719,24cyclictest141rcu_preempt10:13:181
28185996717,16cyclictest0-21swapper/112:22:431
28185996715,10cyclictest25-21ksoftirqd/107:38:021
28203996647,11cyclictest28675-21munin-run08:47:243
28203996628,9cyclictest1580-21munin-run12:22:243
28203996622,24cyclictest396-21runrttasks09:53:233
28203996622,24cyclictest396-21runrttasks09:53:233
28203996615,24cyclictest2740-21chrt10:46:173
28194996625,11cyclictest141rcu_preempt09:16:442
28194996614,24cyclictest321ktimers/212:37:242
28194996613,44cyclictest321ktimers/207:37:462
28194996613,20cyclictest33-21ksoftirqd/209:58:062
28185996639,8cyclictest141rcu_preempt09:17:251
28185996635,10cyclictest141rcu_preempt10:44:341
28185996633,12cyclictest141rcu_preempt10:53:041
28185996630,10cyclictest141rcu_preempt10:03:001
28185996623,23cyclictest141rcu_preempt09:02:441
28185996615,11cyclictest0-21swapper/110:49:411
28185996615,11cyclictest0-21swapper/110:49:411
28185996614,19cyclictest25-21ksoftirqd/111:18:011
28185996614,19cyclictest25-21ksoftirqd/111:18:011
28203996546,11cyclictest1-21systemd07:12:283
28203996519,18cyclictest10785-21latency_hist07:52:273
28203996518,18cyclictest21924-21seq08:24:043
28203996511,21cyclictest113422fw_packets09:32:483
28194996542,12cyclictest0-21swapper/211:42:232
28194996530,16cyclictest0-21swapper/212:09:162
28194996512,8cyclictest33-21ksoftirqd/209:18:012
28185996544,12cyclictest32404-21idleruntime-cro12:17:241
28185996544,12cyclictest32404-21idleruntime-cro12:17:241
28185996534,12cyclictest141rcu_preempt09:21:231
28185996529,16cyclictest141rcu_preempt08:42:361
2818599652,50cyclictest19657-21runrttasks11:37:451
28185996523,17cyclictest141rcu_preempt09:00:121
28185996523,17cyclictest141rcu_preempt09:00:121
28185996514,9cyclictest0-21swapper/112:02:491
2820399647,28cyclictest396-21runrttasks09:15:283
28203996416,20cyclictest15927-21seq09:47:423
28203996414,23cyclictest28633-21chrt10:26:163
28194996447,8cyclictest311rcuc/207:52:422
28194996430,10cyclictest141rcu_preempt08:37:092
28194996430,10cyclictest141rcu_preempt08:37:092
28194996427,10cyclictest141rcu_preempt11:00:212
28194996423,19cyclictest141rcu_preempt07:45:312
28194996423,11cyclictest0-21swapper/208:18:012
28194996421,21cyclictest141rcu_preempt08:57:422
28194996421,21cyclictest141rcu_preempt08:57:422
28194996413,25cyclictest321ktimers/212:02:242
28185996449,9cyclictest12413-21sh07:57:241
28185996433,9cyclictest141rcu_preempt08:22:341
28185996415,23cyclictest25-21ksoftirqd/107:57:531
28203996345,11cyclictest30347-21cron10:32:243
28203996345,11cyclictest30347-21cron10:32:243
28203996321,17cyclictest8880-21taskset09:23:403
28203996319,16cyclictest20689-21taskset11:38:473
28203996313,20cyclictest7571-21latency_hist09:22:263
28194996335,9cyclictest141rcu_preempt10:42:412
28194996328,8cyclictest141rcu_preempt12:27:262
28194996325,18cyclictest141rcu_preempt10:07:282
28194996324,9cyclictest141rcu_preempt08:03:002
28194996316,13cyclictest141rcu_preempt12:12:452
28194996316,13cyclictest141rcu_preempt12:12:452
28194996313,21cyclictest33-21ksoftirqd/211:32:512
28185996342,12cyclictest4160-21munin-plugin-st07:32:241
28185996335,12cyclictest141rcu_preempt09:32:261
28185996332,9cyclictest141rcu_preempt10:17:411
28185996332,9cyclictest141rcu_preempt09:42:491
28185996331,17cyclictest141rcu_preempt08:07:361
28185996328,9cyclictest141rcu_preempt07:17:381
28185996327,8cyclictest0-21swapper/108:32:471
28185996327,8cyclictest0-21swapper/108:32:471
28185996326,8cyclictest141rcu_preempt10:32:461
28185996323,26cyclictest141rcu_preempt11:30:191
28203996241,11cyclictest14514-21latency_hist11:22:263
28203996241,11cyclictest14514-21latency_hist11:22:263
28203996214,19cyclictest4610-21taskset12:29:113
28203996214,19cyclictest4610-21taskset12:29:113
28203996213,19cyclictest5581-21/usr/sbin/munin12:32:513
28194996230,12cyclictest141rcu_preempt11:48:002
28194996230,12cyclictest141rcu_preempt11:48:002
28185996237,7cyclictest141rcu_preempt12:07:481
28185996220,18cyclictest141rcu_preempt11:17:231
28185996219,14cyclictest141rcu_preempt11:07:561
28203996139,10cyclictest0-21swapper/311:48:133
28203996139,10cyclictest0-21swapper/311:48:133
28203996115,15cyclictest0-21swapper/309:29:373
28203996111,22cyclictest1473-21cat09:02:403
28194996135,8cyclictest141rcu_preempt08:22:512
28194996134,8cyclictest141rcu_preempt10:13:182
28194996130,7cyclictest141rcu_preempt08:12:482
28194996129,9cyclictest141rcu_preempt09:08:482
28194996122,20cyclictest141rcu_preempt07:11:182
28194996115,12cyclictest141rcu_preempt07:34:072
28185996143,12cyclictest274-21systemd-journal08:42:241
28185996135,10cyclictest141rcu_preempt10:42:241
28185996135,10cyclictest141rcu_preempt10:42:241
28185996130,11cyclictest52-21kcompactd009:37:531
28185996129,20cyclictest0-21swapper/110:08:271
28185996115,13cyclictest0-21swapper/111:34:561
28203996015,16cyclictest4355-21taskset10:51:223
28203996015,16cyclictest4355-21taskset10:51:223
28203996010,24cyclictest20284-21chrt09:59:013
28203996010,23cyclictest31614-21taskset07:15:573
28203996010,21cyclictest14310-21apt-get09:42:373
28194996025,14cyclictest141rcu_preempt11:18:022
28194996025,14cyclictest141rcu_preempt11:18:022
28194996023,21cyclictest141rcu_preempt07:27:502
28185996041,7cyclictest141rcu_preempt07:12:291
28185996034,14cyclictest141rcu_preempt07:07:521
28185996023,20cyclictest141rcu_preempt12:27:411
28185996023,20cyclictest141rcu_preempt12:27:411
28185996021,18cyclictest141rcu_preempt08:02:571
2820399598,22cyclictest24901-21irqstats11:52:563
28203995941,12cyclictest21423-21fschecks_count11:42:453
28203995941,12cyclictest21423-21fschecks_count11:42:453
28203995940,12cyclictest0-21swapper/308:08:133
28203995919,16cyclictest0-21swapper/311:12:403
28203995912,24cyclictest0-21swapper/308:18:433
28194995936,6cyclictest141rcu_preempt10:54:262
28194995935,13cyclictest0-21swapper/208:42:592
2819499593,29cyclictest0-21swapper/207:18:152
28194995927,13cyclictest141rcu_preempt10:47:292
28194995927,13cyclictest141rcu_preempt10:47:292
28194995918,21cyclictest33-21ksoftirqd/208:12:242
28194995918,17cyclictest33-21ksoftirqd/211:07:382
28194995912,23cyclictest33-21ksoftirqd/208:27:502
28194995912,22cyclictest33-21ksoftirqd/209:42:262
28185995912,28cyclictest141rcu_preempt09:57:411
28203995810,20cyclictest1185-21/usr/sbin/munin07:22:363
2819499587,25cyclictest33-21ksoftirqd/208:47:592
2819499587,25cyclictest33-21ksoftirqd/208:47:592
28194995841,9cyclictest321ktimers/208:57:242
28194995833,7cyclictest141rcu_preempt11:45:172
28194995833,7cyclictest141rcu_preempt11:45:172
2819499583,32cyclictest33-21ksoftirqd/209:02:422
28194995832,16cyclictest0-21swapper/212:22:242
28194995829,8cyclictest141rcu_preempt10:02:532
28194995819,9cyclictest141rcu_preempt10:32:272
2818599589,23cyclictest25-21ksoftirqd/111:52:261
2818599589,23cyclictest25-21ksoftirqd/111:52:261
2818599583,25cyclictest0-21swapper/111:45:311
2818599583,25cyclictest0-21swapper/111:45:311
28185995812,27cyclictest25-21ksoftirqd/108:47:361
28185995812,27cyclictest25-21ksoftirqd/108:47:361
28194995729,7cyclictest141rcu_preempt09:42:422
28194995715,9cyclictest0-21swapper/212:02:522
28194995713,12cyclictest33-21ksoftirqd/207:58:022
28194995638,8cyclictest0-21swapper/210:32:592
28194995617,18cyclictest33-21ksoftirqd/211:55:002
28194995617,17cyclictest33-21ksoftirqd/209:52:262
28194995617,17cyclictest33-21ksoftirqd/209:32:412
2820399557,19cyclictest27123-21munin-run10:22:243
28203995539,9cyclictest0-21swapper/307:32:243
28203995510,19cyclictest7555-21sshd10:59:333
2819499557,23cyclictest141rcu_preempt09:55:242
2819499557,23cyclictest141rcu_preempt09:55:242
2819499557,22cyclictest8326-21sed09:22:562
28194995528,7cyclictest141rcu_preempt10:28:362
28194995527,8cyclictest141rcu_preempt10:17:562
28194995526,11cyclictest141rcu_preempt11:13:332
28194995514,19cyclictest33-21ksoftirqd/210:38:002
28194995514,19cyclictest33-21ksoftirqd/210:38:002
2820399549,18cyclictest10829-21taskset11:09:133
2820399548,17cyclictest26529-21irqstats11:57:493
2820399546,21cyclictest18131-21ntp_states08:12:533
2820399546,19cyclictest8596-21ntp_states11:02:583
28203995423,23cyclictest32043-21apt-get07:17:383
28203995416,12cyclictest4884-21diskstats10:52:513
2820399537,18cyclictest2478-21ntp_states12:23:073
28194995314,11cyclictest33-21ksoftirqd/211:27:522
2820399526,16cyclictest396-21runrttasks07:39:163
28203995229,12cyclictest0-21swapper/310:32:593
28203995224,20cyclictest26224-21ntpq08:37:533
28203995112,11cyclictest32445-21munin-run12:17:253
28203995112,11cyclictest32445-21munin-run12:17:253
28203994923,17cyclictest396-21runrttasks12:07:353
28203994820,17cyclictest13446-21/usr/sbin/munin07:57:593
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional