You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-06-27 - 05:30

ARM Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7 (updated Mon Jun 27, 2022 00:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26590214813,107sleep026029-21/usr/sbin/munin19:01:080
26590214813,107sleep026029-21/usr/sbin/munin19:01:080
2752399993,14cyclictest14508-21latency_hist21:25:330
2752399968,9cyclictest0-21swapper/000:29:060
2752399963,10cyclictest16457-21ls22:21:050
2752399962,18cyclictest28302-21/usr/sbin/munin23:35:570
2752399952,15cyclictest10523-21tail22:11:170
2752399952,11cyclictest32052-21ssh23:41:020
2752399943,16cyclictest20425-21cron21:35:310
2752399943,16cyclictest20425-21cron21:35:310
2752399933,20cyclictest29308-21munin-node19:10:420
2752399923,10cyclictest1442-21grep21:56:110
2752399923,10cyclictest1442-21grep21:56:110
2752399922,14cyclictest11483-21/usr/sbin/munin00:00:590
2752399913,16cyclictest24459-21latency_hist22:35:340
2752399913,16cyclictest24459-21latency_hist22:35:340
2752399904,14cyclictest392-21runrttasks00:06:320
2752399904,14cyclictest32513-21apt21:55:310
2752399902,10cyclictest15304-21ntp_states21:26:000
2752399902,10cyclictest14286-21ssh23:11:090
2752399894,12cyclictest23107-21if_eth000:20:540
2752399894,12cyclictest23107-21if_eth000:20:540
27523998936,46cyclictest18228-21sendmail20:15:310
2752399884,17cyclictest6788-21idleruntime-cro19:40:310
2752399884,10cyclictest18511-21apt-get20:15:470
2752399883,8cyclictest20766-21apt-get21:35:440
2752399883,13cyclictest30553-21ssh22:45:450
2752399883,12cyclictest28876-21ntpq20:45:570
2752399883,11cyclictest25701-21/usr/sbin/munin23:30:550
27523998830,50cyclictest0-21swapper/021:50:320
2752399882,79cyclictest20535-21sort20:20:560
2752399882,15cyclictest10053-21ssh21:16:580
2752399882,15cyclictest10053-21ssh21:16:580
2752399874,11cyclictest10155-21/usr/sbin/munin23:06:020
2752399874,10cyclictest0-21swapper/019:06:040
2752399873,14cyclictest5153-21fschecks_count23:50:510
2752399873,11cyclictest5822-21ssh21:11:190
2752399872,12cyclictest392-21runrttasks22:36:430
2752399872,12cyclictest392-21runrttasks22:36:430
2752399872,10cyclictest4162-21ssh22:55:460
2752399872,10cyclictest4162-21ssh22:55:460
2752399866,72cyclictest0-21swapper/023:25:470
2752399864,13cyclictest29675-21sed00:31:050
2752399864,12cyclictest1086-21apt-get22:50:450
2752399864,12cyclictest1086-21apt-get22:50:450
2752399863,16cyclictest8058-21sort23:55:480
2752399862,13cyclictest12192-21fschecks_count19:55:510
2752399853,14cyclictest22033-21grep20:26:100
2752399852,17cyclictest18816-21fschecks_time22:25:530
2752399844,70cyclictest13827-21fschecks_count20:00:510
2752399843,22cyclictest26716-21latency_hist20:40:350
2752399843,17cyclictest17210-21grep23:16:050
2752399843,15cyclictest24797-21munin-run20:35:310
2752399843,13cyclictest31631-21/usr/sbin/munin20:56:020
2752399842,10cyclictest825-21apt-get21:00:430
27523998419,59cyclictest8150irq/69-2188000.23:20:520
27548998326,27cyclictest141rcu_preempt23:34:063
27532998352,10cyclictest141rcu_preempt22:29:131
27532998330,22cyclictest141rcu_preempt20:16:141
2752399834,71cyclictest7809-21cat19:41:030
2752399833,17cyclictest7914-21cat23:01:030
2752399833,16cyclictest4122-21proc_pri22:01:070
2752399833,16cyclictest30207-21sort20:50:520
2752399832,8cyclictest3126-21grep19:26:060
2752399832,8cyclictest2181-21diskmemload00:15:340
2752399824,69cyclictest765-21sort19:20:520
2752399824,69cyclictest765-21sort19:20:520
2752399824,20cyclictest9271-21latency_hist22:10:340
2752399824,15cyclictest10344-21/usr/sbin/munin19:51:010
2752399823,17cyclictest28126-21idleruntime-cro20:45:310
2752399823,15cyclictest28056-21cut22:41:290
2752399823,15cyclictest2225-21sort23:45:530
2752399823,15cyclictest19980-21grep00:15:510
2752399823,12cyclictest23866-21apt-get21:40:430
27523998225,47cyclictest0-21swapper/022:15:570
2752399822,17cyclictest2948-21wc21:05:550
2752399814,14cyclictest3782-21/usr/sbin/munin19:31:050
2752399813,16cyclictest32299-21grep19:16:050
2752399813,13cyclictest9062-21perl19:46:190
27548998023,37cyclictest0-21swapper/322:02:053
2752399803,15cyclictest16025-21grep20:06:030
27548997963,9cyclictest11173-21unixbench_multi23:06:053
27548997850,19cyclictest30236-21fschecks_time20:50:523
27532997857,13cyclictest26428-21latency_hist20:40:311
27532997744,8cyclictest141rcu_preempt22:30:531
27532997744,8cyclictest141rcu_preempt22:30:531
27532997726,26cyclictest141rcu_preempt22:23:561
27548997641,12cyclictest141rcu_preempt22:11:193
27548997636,7cyclictest141rcu_preempt22:06:033
27548997629,19cyclictest0-21swapper/323:21:413
27548997537,10cyclictest141rcu_preempt21:48:103
27543997554,14cyclictest31117-21sendmail19:15:312
27543997550,15cyclictest29745-21idleruntime-cro20:50:312
27543997549,17cyclictest14386-21idleruntime-cro21:25:312
260592759,31sleep20-21swapper/219:00:492
260592759,31sleep20-21swapper/219:00:492
27548997450,17cyclictest30224-21rm22:45:313
27548997425,26cyclictest141rcu_preempt21:53:263
27548997418,19cyclictest41-21ksoftirqd/323:10:533
2754899733,39cyclictest5599-21apt-get21:10:513
27548997323,24cyclictest0-21swapper/320:48:033
27532997339,8cyclictest141rcu_preempt22:07:341
27548997229,25cyclictest117-21pr/ttymxc023:38:103
27548997226,21cyclictest141rcu_preempt00:15:433
27548997218,18cyclictest0-21swapper/320:10:343
27548997217,35cyclictest41-21ksoftirqd/320:40:343
27532997238,12cyclictest141rcu_preempt00:10:551
27532997225,25cyclictest141rcu_preempt21:58:371
27532997225,25cyclictest141rcu_preempt21:58:371
27548997113,32cyclictest19409-21kworker/3:021:38:553
27543997137,16cyclictest141rcu_preempt00:13:022
27543997127,9cyclictest141rcu_preempt22:26:012
27543997125,20cyclictest611-21in:imklog21:52:422
27548997043,8cyclictest141rcu_preempt22:35:313
27548997043,8cyclictest141rcu_preempt22:35:313
27548997035,13cyclictest141rcu_preempt20:30:583
27548997034,13cyclictest141rcu_preempt00:31:053
27548997020,10cyclictest141rcu_preempt20:10:563
27548997013,29cyclictest41-21ksoftirqd/319:55:513
27548997012,18cyclictest0-21swapper/319:45:473
27543997038,13cyclictest141rcu_preempt23:27:162
27543997034,16cyclictest141rcu_preempt00:00:342
27532997038,8cyclictest141rcu_preempt21:21:051
27532997031,16cyclictest141rcu_preempt22:41:011
27548996942,13cyclictest14671-21ssh21:25:413
27548996933,13cyclictest141rcu_preempt22:56:423
27548996933,13cyclictest141rcu_preempt22:56:423
27548996922,26cyclictest141rcu_preempt22:51:003
27548996922,26cyclictest141rcu_preempt22:51:003
27548996919,22cyclictest0-21swapper/323:41:573
27548996915,21cyclictest0-21swapper/322:20:423
27532996942,7cyclictest141rcu_preempt23:01:021
27532996936,7cyclictest141rcu_preempt23:57:061
27532996929,16cyclictest141rcu_preempt21:39:271
27548996837,9cyclictest141rcu_preempt21:33:003
27548996837,9cyclictest141rcu_preempt21:33:003
27548996833,14cyclictest141rcu_preempt20:01:033
27548996830,11cyclictest141rcu_preempt00:26:333
27548996826,7cyclictest141rcu_preempt00:00:233
27548996818,20cyclictest141rcu_preempt22:27:273
27548996816,19cyclictest22918-21kworker/3:020:30:263
27548996815,20cyclictest41-21ksoftirqd/320:45:323
27548996813,33cyclictest0-21swapper/319:38:053
27548996812,17cyclictest41-21ksoftirqd/320:21:043
27543996849,12cyclictest17217-21unixbench_multi23:16:052
27543996844,16cyclictest321ktimers/221:16:082
27543996844,16cyclictest321ktimers/221:16:082
27543996836,15cyclictest141rcu_preempt20:10:322
27543996830,13cyclictest141rcu_preempt20:42:422
27532996839,7cyclictest141rcu_preempt23:40:181
27532996835,8cyclictest141rcu_preempt21:06:231
27532996827,14cyclictest141rcu_preempt22:12:411
27532996825,17cyclictest141rcu_preempt23:32:131
27532996825,12cyclictest141rcu_preempt19:39:321
27548996741,7cyclictest141rcu_preempt21:21:133
27548996739,7cyclictest141rcu_preempt21:40:433
27548996737,12cyclictest3784-21kworker/u8:023:51:403
27548996719,23cyclictest0-21swapper/323:27:383
27548996714,30cyclictest41-21ksoftirqd/319:30:313
27543996743,7cyclictest141rcu_preempt23:42:182
27543996739,7cyclictest141rcu_preempt22:36:492
27543996739,7cyclictest141rcu_preempt22:36:492
27543996719,22cyclictest611-21in:imklog22:50:192
27532996733,7cyclictest141rcu_preempt00:29:191
27532996726,16cyclictest141rcu_preempt21:41:421
27532996725,15cyclictest141rcu_preempt00:23:071
27532996725,15cyclictest141rcu_preempt00:23:071
27532996714,22cyclictest241ktimers/120:06:031
2639826716,29sleep10-21swapper/119:01:011
2639826716,29sleep10-21swapper/119:01:011
27548996638,12cyclictest141rcu_preempt21:07:063
27548996634,14cyclictest141rcu_preempt00:05:353
27548996631,16cyclictest141rcu_preempt19:40:563
27548996625,7cyclictest141rcu_preempt00:05:313
27548996616,26cyclictest41-21ksoftirqd/319:25:333
27548996616,26cyclictest41-21ksoftirqd/319:25:333
27543996634,15cyclictest141rcu_preempt22:16:372
27543996633,13cyclictest141rcu_preempt21:26:292
27543996631,16cyclictest141rcu_preempt00:30:332
27543996630,13cyclictest141rcu_preempt20:17:052
27543996626,14cyclictest141rcu_preempt19:33:572
27543996618,19cyclictest0-21swapper/222:57:132
27543996618,19cyclictest0-21swapper/222:57:132
2753299665,43cyclictest5667-21ssh21:10:521
27532996641,7cyclictest141rcu_preempt23:19:541
27532996636,7cyclictest141rcu_preempt00:31:021
27532996631,16cyclictest141rcu_preempt22:17:551
27532996626,15cyclictest141rcu_preempt23:13:031
27532996622,23cyclictest141rcu_preempt23:26:231
27548996544,8cyclictest141rcu_preempt22:35:473
27548996544,8cyclictest141rcu_preempt22:35:473
27548996538,11cyclictest141rcu_preempt22:16:403
27548996529,8cyclictest141rcu_preempt19:15:333
27548996515,24cyclictest41-21ksoftirqd/321:00:323
27548996514,26cyclictest0-21swapper/323:18:093
27548996511,32cyclictest41-21ksoftirqd/319:20:063
27548996511,22cyclictest141rcu_preempt22:48:183
27543996540,6cyclictest141rcu_preempt23:50:012
27543996533,9cyclictest321ktimers/200:20:172
27543996532,10cyclictest141rcu_preempt19:40:382
27543996531,12cyclictest141rcu_preempt20:36:352
27543996528,19cyclictest52-21kcompactd019:56:272
27543996525,18cyclictest611-21in:imklog23:01:452
27543996524,17cyclictest180-21pr/ttyS023:21:292
27532996551,7cyclictest0-21swapper/122:55:561
27532996551,7cyclictest0-21swapper/122:55:561
27532996539,8cyclictest141rcu_preempt23:51:531
27532996532,17cyclictest141rcu_preempt20:40:511
27532996526,12cyclictest141rcu_preempt19:17:131
27532996525,21cyclictest141rcu_preempt21:27:541
27532996525,12cyclictest141rcu_preempt21:18:281
27532996525,12cyclictest141rcu_preempt21:18:281
27532996521,14cyclictest141rcu_preempt00:06:031
27532996517,9cyclictest141rcu_preempt22:46:031
27548996438,8cyclictest141rcu_preempt00:21:053
27548996438,8cyclictest141rcu_preempt00:21:053
27548996435,8cyclictest141rcu_preempt23:47:043
27548996431,13cyclictest141rcu_preempt23:04:003
27548996431,11cyclictest141rcu_preempt21:00:533
27548996429,14cyclictest141rcu_preempt21:18:113
27548996429,14cyclictest141rcu_preempt21:18:113
27548996414,24cyclictest141rcu_preempt21:57:123
27548996414,24cyclictest141rcu_preempt21:57:123
27548996411,10cyclictest41-21ksoftirqd/320:15:573
27543996433,11cyclictest141rcu_preempt22:31:112
27543996433,11cyclictest141rcu_preempt22:31:112
27543996430,15cyclictest141rcu_preempt23:35:022
27543996429,15cyclictest141rcu_preempt23:36:182
27543996426,9cyclictest141rcu_preempt20:52:582
27532996448,10cyclictest10102-21munin-plugin-st19:50:311
27532996442,14cyclictest24793-21latency_hist20:35:311
27532996436,8cyclictest141rcu_preempt19:50:531
27532996434,7cyclictest141rcu_preempt21:34:281
27532996434,7cyclictest141rcu_preempt21:34:281
27532996431,16cyclictest141rcu_preempt19:40:491
27532996427,16cyclictest141rcu_preempt19:11:031
27532996426,7cyclictest141rcu_preempt22:40:331
27532996426,7cyclictest141rcu_preempt22:40:331
27543996330,7cyclictest9851-21ssh23:58:242
27543996329,8cyclictest141rcu_preempt20:29:362
27543996329,10cyclictest141rcu_preempt19:49:322
27543996328,8cyclictest141rcu_preempt19:55:222
27543996326,11cyclictest611-21in:imklog21:41:222
27543996323,11cyclictest141rcu_preempt19:21:592
27543996323,11cyclictest141rcu_preempt19:21:592
27532996334,7cyclictest141rcu_preempt23:42:381
27532996332,8cyclictest141rcu_preempt22:00:421
27532996330,17cyclictest141rcu_preempt19:20:481
27532996330,17cyclictest141rcu_preempt19:20:481
27532996329,8cyclictest141rcu_preempt20:00:571
27532996325,17cyclictest141rcu_preempt23:45:551
27532996325,17cyclictest141rcu_preempt19:09:021
265022639,22sleep30-21swapper/319:01:033
265022639,22sleep30-21swapper/319:01:033
27548996243,7cyclictest141rcu_preempt00:11:463
27548996223,8cyclictest141rcu_preempt19:51:033
27548996217,25cyclictest0-21swapper/319:10:233
27543996237,7cyclictest141rcu_preempt20:30:422
27543996234,7cyclictest141rcu_preempt21:10:512
27543996233,7cyclictest141rcu_preempt23:11:042
27543996233,14cyclictest141rcu_preempt22:14:452
27543996220,13cyclictest141rcu_preempt23:05:482
27543996218,17cyclictest0-21swapper/219:37:042
27532996228,15cyclictest141rcu_preempt22:50:421
27532996228,15cyclictest141rcu_preempt22:50:421
27532996222,8cyclictest141rcu_preempt20:53:311
27532996221,9cyclictest141rcu_preempt20:14:081
27532996212,22cyclictest0-21swapper/100:15:541
27548996128,8cyclictest141rcu_preempt19:35:343
27543996134,8cyclictest141rcu_preempt22:21:042
27543996134,8cyclictest141rcu_preempt19:15:572
27543996133,7cyclictest141rcu_preempt22:54:112
27543996133,7cyclictest141rcu_preempt22:54:112
27543996130,7cyclictest141rcu_preempt22:05:412
27543996128,9cyclictest141rcu_preempt22:41:172
27543996124,8cyclictest141rcu_preempt20:59:182
27532996136,8cyclictest141rcu_preempt19:55:511
27532996132,8cyclictest141rcu_preempt20:26:191
27532996124,12cyclictest141rcu_preempt19:34:571
27532996123,8cyclictest141rcu_preempt20:59:471
27532996119,13cyclictest141rcu_preempt21:51:401
27543996033,8cyclictest141rcu_preempt21:58:172
27543996033,8cyclictest141rcu_preempt21:58:172
27543996031,8cyclictest141rcu_preempt21:48:002
27543996022,10cyclictest141rcu_preempt21:31:062
27543996022,10cyclictest141rcu_preempt21:31:062
27543996018,22cyclictest611-21in:imklog00:34:342
27543996014,24cyclictest611-21in:imklog21:06:342
27532996027,15cyclictest141rcu_preempt00:02:061
27532996026,16cyclictest141rcu_preempt23:20:571
27532996012,21cyclictest25-21ksoftirqd/121:47:541
27543995933,7cyclictest141rcu_preempt20:13:122
27543995930,8cyclictest141rcu_preempt23:54:292
27543995924,21cyclictest52-21kcompactd019:30:332
27543995924,16cyclictest141rcu_preempt00:06:082
27532995926,8cyclictest141rcu_preempt20:46:051
27543995839,7cyclictest141rcu_preempt22:03:492
27543995831,7cyclictest141rcu_preempt21:36:012
27543995827,8cyclictest141rcu_preempt00:22:042
27543995827,8cyclictest141rcu_preempt00:22:042
2753299583,22cyclictest21470-21apt-get20:25:341
27532995812,9cyclictest25-21ksoftirqd/123:05:561
27543995733,7cyclictest141rcu_preempt20:21:022
27543995629,8cyclictest141rcu_preempt20:01:002
27532995528,7cyclictest141rcu_preempt21:01:041
27532995526,8cyclictest141rcu_preempt19:25:491
27543995415,18cyclictest52-21kcompactd019:07:582
27543995330,7cyclictest141rcu_preempt21:03:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional