You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-09-30 - 19:12
/usr/bin/Xorg /usr/bin/Xorg

ARM Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7 (updated Fri Sep 30, 2022 12:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1076211617,58sleep11164-21grep07:03:151
1076211617,58sleep11164-21grep07:03:151
48829826,33sleep00-21swapper/007:02:570
48829826,33sleep00-21swapper/007:02:570
1760999039,43cyclictest2181-21munin-plugin-st08:47:282
1760999039,43cyclictest2181-21munin-plugin-st08:47:282
1764998965,14cyclictest24892-21sendmail08:17:273
76128710,36sleep30-21swapper/307:03:093
76128710,36sleep30-21swapper/307:03:093
1752998767,12cyclictest5357-21munin-plugin-st07:17:281
1744998734,36cyclictest0-21swapper/010:12:240
176099853,54cyclictest1780-21apt-get12:27:382
176099844,20cyclictest397-21runrttasks10:47:302
1760998434,42cyclictest0-21swapper/210:23:092
1760998434,42cyclictest0-21swapper/210:23:092
176099822,70cyclictest17696-21apt-get11:07:382
1744998260,14cyclictest6855-21unixbench_singl10:48:040
1744998148,23cyclictest334-21dbus-daemon09:22:290
1764998058,14cyclictest7974-21kworker/u8:012:37:273
176099802,20cyclictest5420-21tail09:53:062
176099802,20cyclictest5420-21tail09:53:062
175299803,17cyclictest25070-21cat08:17:301
174499808,25cyclictest0-21swapper/011:07:460
1744998055,16cyclictest6990-21idleruntime-cro07:22:270
1744998011,48cyclictest16227-21ssh11:57:380
176099793,18cyclictest16341-21unixbench_multi07:48:022
176099793,18cyclictest16341-21unixbench_multi07:48:022
1760997921,51cyclictest7054-21sh09:02:272
175299793,16cyclictest8535-21unixbench_multi09:58:061
1744997918,38cyclictest397-21runrttasks12:25:520
176099784,19cyclictest5891-21sh12:32:572
176099783,21cyclictest27146-21uname08:22:522
176099783,19cyclictest16379-21modprobe10:12:272
176099783,14cyclictest15517-21ssh11:02:542
1744997819,22cyclictest23747-21cut08:12:490
1764997749,18cyclictest334-21dbus-daemon08:32:593
175299774,18cyclictest13086-21grep07:38:071
175299773,14cyclictest2412-21awk10:42:271
1752997719,52cyclictest21357-21fschecks_count11:12:481
1744997744,25cyclictest0-21swapper/012:12:540
176099767,16cyclictest0-21swapper/207:12:502
176099764,18cyclictest29345-21unixbench_singl08:28:022
176099764,18cyclictest27919-21sort09:37:512
176099763,19cyclictest15920-21mailstats09:18:002
176099763,18cyclictest2168-21fschecks_count07:08:182
176099763,17cyclictest13089-21sed07:38:072
1760997630,39cyclictest0-21swapper/211:57:292
176099762,17cyclictest23897-21munin-plugin-st11:17:272
1760997619,50cyclictest397-21runrttasks09:27:442
175299763,17cyclictest12577-21ps10:58:001
175299763,17cyclictest12577-21ps10:58:001
175299763,14cyclictest5419-21grep09:53:061
175299763,14cyclictest5419-21grep09:53:061
1752997631,37cyclictest0-21swapper/109:47:521
174499766,53cyclictest0-21swapper/011:43:000
1744997621,25cyclictest23542-21chrt12:08:250
1744997620,24cyclictest0-21swapper/011:23:030
1744997617,24cyclictest28181-21latency_hist08:27:300
1764997558,10cyclictest19195-21unixbench_singl11:08:123
176099753,19cyclictest17311-21ntp_offset10:12:562
176099753,17cyclictest22149-21ssh12:07:292
176099753,14cyclictest8675-21diskmemload10:58:002
176099753,14cyclictest8675-21diskmemload10:58:002
1760997519,9cyclictest0-21swapper/210:02:452
175299758,11cyclictest11004-21kernelversion10:02:531
175299753,21cyclictest31929-21latency_hist10:37:301
175299753,19cyclictest728-21latency_hist08:42:301
175299753,14cyclictest27041-21sed10:28:031
175299752,18cyclictest16559-21latency_hist10:12:301
175299752,16cyclictest764-21cat11:32:301
175299752,16cyclictest764-21cat11:32:301
1744997522,24cyclictest7597-21grep09:02:490
1744997521,32cyclictest397-21runrttasks09:53:400
1744997521,32cyclictest397-21runrttasks09:53:400
1744997521,22cyclictest19498-21grep07:58:010
1744997521,22cyclictest19498-21grep07:58:010
1744997520,28cyclictest31982-21ssh09:43:380
1764997434,12cyclictest0-21swapper/312:28:013
176099743,19cyclictest10015-21tail09:08:042
176099743,18cyclictest6988-21munin-plugin-st07:22:272
176099743,18cyclictest4937-21tail08:53:032
176099743,18cyclictest19846-21ssh10:17:452
176099743,18cyclictest19846-21ssh10:17:452
176099743,16cyclictest22980-21ssh12:08:022
176099743,16cyclictest12366-21grep09:12:502
1760997430,36cyclictest0-21swapper/208:12:502
176099742,21cyclictest1263-21taskset11:32:492
176099742,19cyclictest29492-21grep12:18:022
176099742,18cyclictest32602-21sed08:38:042
175299743,19cyclictest31776-21grep12:22:501
175299743,19cyclictest10782-21fschecks_count07:32:571
175299743,19cyclictest10782-21fschecks_count07:32:571
175299743,17cyclictest15447-21fschecks_count09:17:501
175299743,15cyclictest4093-21sort11:37:451
1752997421,10cyclictest16605-21fschecks_time11:57:591
1752997415,9cyclictest0-21swapper/112:03:011
1744997441,23cyclictest20804-21unixbench_multi10:18:040
1744997441,23cyclictest20804-21unixbench_multi10:18:040
1744997413,43cyclictest397-21runrttasks12:30:490
16692749,42sleep21634-21systemd-run07:07:312
16692749,42sleep21634-21systemd-run07:07:312
176099734,15cyclictest31777-21fschecks_count12:22:502
176099733,15cyclictest8663-21modprobe09:07:282
176099733,15cyclictest10767-21fschecks_count07:32:572
176099733,15cyclictest10767-21fschecks_count07:32:572
176099732,17cyclictest19277-21tail09:23:032
1760997316,11cyclictest17651-21sendmail_mailqu07:52:582
175299734,47cyclictest2175-21apt-get12:27:431
175299734,20cyclictest5889-21grep12:32:571
175299732,18cyclictest8680-21latency_hist07:27:301
175299732,18cyclictest8680-21latency_hist07:27:301
175299732,18cyclictest23569-21/usr/sbin/munin10:23:021
175299732,18cyclictest23569-21/usr/sbin/munin10:23:021
175299732,15cyclictest9678-21proc_pri09:08:001
175299732,15cyclictest13102-21grep09:13:031
175299732,14cyclictest21257-21unixbench_singl08:03:011
1752997316,11cyclictest0-21swapper/107:48:041
1752997316,11cyclictest0-21swapper/107:48:041
1752997313,11cyclictest25216-21latency_hist12:12:301
1744997323,17cyclictest23376-21cat10:22:590
1744997323,17cyclictest23376-21cat10:22:590
1744997322,21cyclictest418-21sshd11:21:230
1744997322,21cyclictest418-21sshd11:21:230
1744997322,13cyclictest19476-21sendmail10:17:270
1744997321,21cyclictest10885-21fschecks_time10:02:500
176099724,21cyclictest25643-21fschecks_time12:12:502
176099724,20cyclictest17296-21unixbench_multi11:58:112
176099724,15cyclictest12829-21latency_hist11:52:292
176099724,14cyclictest9123-21fschecks_time07:27:512
176099723,21cyclictest26949-21latency_hist11:22:272
176099723,21cyclictest26949-21latency_hist11:22:272
176099723,17cyclictest7829-21ssh09:57:532
176099723,15cyclictest31442-21ssh09:43:002
176099722,19cyclictest4558-21awk11:37:552
175299723,17cyclictest29832-21latency_hist08:32:291
175299723,16cyclictest5510-21ssh10:47:291
175299723,16cyclictest28703-21fschecks_count12:17:481
175299723,15cyclictest24094-21idleruntime-cro09:32:281
1752997224,42cyclictest0-21swapper/108:43:051
1752997224,42cyclictest0-21swapper/108:43:051
175299722,19cyclictest31458-21grep09:43:001
175299722,18cyclictest16978-21fschecks_count10:12:501
175299722,17cyclictest11544-21cron10:57:271
175299722,16cyclictest8205-21sed09:03:011
1752997212,13cyclictest5415-21munin-run08:57:281
1744997219,30cyclictest27949-21rm10:30:250
1744997218,22cyclictest4005-1kworker/0:2H11:07:290
1744997213,18cyclictest21434-21fschecks_time11:12:480
1764997152,12cyclictest8675-21diskmemload12:17:263
176099714,18cyclictest3342-21unixbench_multi08:48:032
176099714,17cyclictest7867-21sendmail_mailqu07:23:022
176099714,17cyclictest7867-21sendmail_mailqu07:23:022
176099714,17cyclictest5764-21/usr/sbin/munin10:47:542
176099713,21cyclictest26294-21fschecks_time10:27:502
176099713,19cyclictest24944-21fschecks_time09:33:162
176099713,15cyclictest21659-21idleruntime-cro08:07:272
176099713,14cyclictest30908-21unixbench_multi08:33:012
1760997123,40cyclictest0-21swapper/210:37:502
1752997151,10cyclictest0-21swapper/111:07:271
175299714,14cyclictest0-21swapper/107:57:291
175299714,12cyclictest0-21swapper/110:52:301
175299713,20cyclictest18739-21cpuspeed07:57:431
175299713,20cyclictest18739-21cpuspeed07:57:431
175299713,19cyclictest3343-21unixbench_multi08:48:031
175299713,19cyclictest22743-21latency_hist10:22:311
175299713,19cyclictest22743-21latency_hist10:22:311
175299713,17cyclictest30945-21sed08:33:011
175299713,17cyclictest14421-21sendmail_mailqu07:43:011
175299713,15cyclictest2870-21unixbench_multi07:08:351
175299712,18cyclictest6544-21unixbench_multi07:18:061
175299712,16cyclictest25351-21sort08:17:491
1752997118,46cyclictest0-21swapper/109:22:451
1752997117,7cyclictest0-21swapper/111:52:391
1752997115,13cyclictest10271-21apt-key07:32:281
174499719,18cyclictest0-21swapper/011:47:530
1744997117,19cyclictest30804-21timerandwakeup08:33:000
176499705,27cyclictest24484-21apt-get09:33:093
1764997047,17cyclictest11547-21diskmemload10:03:023
1764997031,9cyclictest0-21swapper/311:17:293
176099704,13cyclictest14630-21timerwakeupswit07:43:042
176099703,15cyclictest8816-21chrt10:52:392
176099703,15cyclictest25349-21grep08:17:492
176099703,13cyclictest30589-21fschecks_time11:27:492
176099703,13cyclictest30589-21fschecks_time11:27:492
176099702,20cyclictest2327-21unixbench_multi09:48:012
176099702,19cyclictest30173-21latency_hist11:27:292
175299703,20cyclictest27358-21ps08:22:581
175299703,18cyclictest28278-21unixbench_multi11:23:031
175299703,18cyclictest28142-21sendmail12:17:271
175299703,18cyclictest23373-21latency_hist08:12:321
175299703,18cyclictest23373-21latency_hist08:12:321
175299703,16cyclictest358-21in:imuxsock11:22:271
175299703,16cyclictest358-21in:imuxsock11:22:271
175299703,14cyclictest25326-21wc09:33:251
1744997025,21cyclictest25644-21awk08:17:540
1744997024,14cyclictest4445-21ssh11:37:530
1744997019,21cyclictest16157-21latency_hist11:57:300
1744997017,21cyclictest11993-21latency_hist09:12:300
1744997011,18cyclictest0-21swapper/009:12:530
1764996918,17cyclictest41-21ksoftirqd/310:12:423
176099693,18cyclictest29454-21uname10:32:522
176099692,14cyclictest21955-21cpu08:07:452
176099692,14cyclictest21955-21cpu08:07:452
175299693,20cyclictest27826-21fschecks_count09:37:501
175299693,15cyclictest5931-21fschecks_time08:57:501
175299692,17cyclictest8084-21grep11:43:021
175299692,13cyclictest1564-21ssh11:32:571
1744996950,12cyclictest32339-21fschecks_count10:37:490
1744996917,23cyclictest21886-21sed09:27:540
1744996915,17cyclictest2767-21ssh10:42:450
1764996850,11cyclictest0-21swapper/310:43:033
1764996838,11cyclictest41-21ksoftirqd/307:52:513
176099684,15cyclictest7279-21fschecks_count11:42:482
176099682,14cyclictest18849-21grep07:57:482
176099682,14cyclictest18849-21grep07:57:482
175299682,15cyclictest10363-21ssh11:47:501
1752996820,7cyclictest0-21swapper/111:08:101
1744996820,19cyclictest19814-21/usr/sbin/munin12:02:540
1744996818,29cyclictest26791-21taskset09:36:170
1744996818,17cyclictest8675-21diskmemload09:37:540
1744996817,21cyclictest32158-21/usr/sbin/munin08:37:560
1744996816,34cyclictest10177-21ssh10:53:440
1744996816,18cyclictest29341-21fschecks_time10:32:500
1744996815,19cyclictest1978-21uname09:47:580
1744996814,26cyclictest8675-21diskmemload11:31:590
1744996814,26cyclictest8675-21diskmemload11:31:590
1744996814,19cyclictest30050-21rm12:19:250
1764996751,11cyclictest1735-21cyclictest08:58:023
1764996749,12cyclictest26531-21munin-plugin-st08:22:283
1764996748,13cyclictest31773-21munin-run10:37:273
1764996732,20cyclictest41-21ksoftirqd/312:22:093
1764996725,13cyclictest141rcu_preempt11:07:293
1744996719,18cyclictest8598-21latency_hist07:27:300
1744996719,18cyclictest8598-21latency_hist07:27:300
1744996718,19cyclictest12375-21sort07:37:520
1744996717,18cyclictest914-21ls08:42:470
1744996717,18cyclictest914-21ls08:42:470
1764996631,8cyclictest0-21swapper/310:39:153
1744996641,16cyclictest0-21swapper/009:57:430
1744996616,21cyclictest2612-21ssh11:34:400
1744996610,17cyclictest0-21swapper/007:37:310
1744996610,17cyclictest0-21swapper/007:37:310
1764996544,14cyclictest7164-21latency_hist09:57:273
1764996544,14cyclictest7164-21latency_hist09:57:273
1764996525,24cyclictest41-21ksoftirqd/310:51:443
1764996518,23cyclictest41-21ksoftirqd/309:52:193
1744996516,18cyclictest3808-21modprobe08:52:270
1744996514,18cyclictest0-21swapper/007:57:270
1744996513,23cyclictest13427-21ssh10:59:120
1744996513,23cyclictest13427-21ssh10:59:120
1744996512,21cyclictest0-21swapper/009:23:000
1764996437,16cyclictest0-21swapper/308:23:023
1764996420,8cyclictest41-21ksoftirqd/308:27:523
1764996415,11cyclictest0-21swapper/309:45:363
1744996447,9cyclictest28402-21apt-get08:27:420
1744996421,15cyclictest21670-21modprobe08:07:270
1744996417,20cyclictest14723-21sed07:43:050
176499633,29cyclictest22538-21apt-get10:22:273
176499633,29cyclictest22538-21apt-get10:22:273
1764996324,24cyclictest141rcu_preempt09:29:383
1764996319,26cyclictest141rcu_preempt09:20:113
1764996316,20cyclictest359-21in:imklog12:27:263
1744996317,15cyclictest0-21swapper/007:48:010
1744996317,15cyclictest0-21swapper/007:48:010
1744996311,21cyclictest1540-21apt-get07:07:400
1764996225,22cyclictest41-21ksoftirqd/309:59:593
1764996225,11cyclictest141rcu_preempt10:08:023
176499622,14cyclictest22524-21cron10:22:393
176499622,14cyclictest22524-21cron10:22:393
1744996214,17cyclictest4697-21mailstats08:53:000
176499618,33cyclictest141rcu_preempt11:37:273
1764996137,15cyclictest13206-21ssh10:58:363
1764996137,15cyclictest13206-21ssh10:58:363
1764996131,20cyclictest27937-21cut09:37:513
1764996115,29cyclictest41-21ksoftirqd/311:30:343
1764996115,29cyclictest41-21ksoftirqd/311:30:343
1764996112,26cyclictest141rcu_preempt11:22:293
1764996112,26cyclictest141rcu_preempt11:22:293
1744996124,31cyclictest0-21swapper/012:37:270
1764996039,14cyclictest8720-21sh09:07:293
176499602,44cyclictest15829-21apt-get11:57:383
176499598,31cyclictest0-21swapper/307:33:333
176499598,31cyclictest0-21swapper/307:33:333
176499598,27cyclictest32727-21kworker/3:008:48:023
176499595,14cyclictest141rcu_preempt07:42:463
176499594,27cyclictest141rcu_preempt08:52:553
1764995942,10cyclictest359-21in:imklog11:47:293
1764995929,7cyclictest141rcu_preempt09:08:033
1764995928,9cyclictest141rcu_preempt07:47:543
1764995928,9cyclictest141rcu_preempt07:47:543
1764995928,15cyclictest141rcu_preempt11:55:093
1744995937,16cyclictest0-21swapper/007:28:020
1744995914,13cyclictest4736-21awk07:13:010
1764995838,14cyclictest10247-21systemd-run07:32:283
1764995833,15cyclictest0-21swapper/307:38:063
1764995832,13cyclictest0-21swapper/308:07:503
1764995832,13cyclictest0-21swapper/308:07:503
1764995824,18cyclictest141rcu_preempt12:05:263
1764995742,10cyclictest179-21pr/ttyS011:26:273
1764995739,10cyclictest359-21in:imklog07:12:463
1764995737,13cyclictest359-21in:imklog08:37:513
1764995733,11cyclictest0-21swapper/311:42:303
1764995732,14cyclictest0-21swapper/309:22:493
1764995711,28cyclictest0-21swapper/312:08:523
1744995710,18cyclictest21896-21/usr/sbin/munin08:07:540
1744995710,18cyclictest21896-21/usr/sbin/munin08:07:540
176499563,22cyclictest0-21swapper/309:12:523
176499554,31cyclictest2068-21cstates07:08:083
1764995540,7cyclictest0-21swapper/310:31:443
1764995526,8cyclictest0-21swapper/308:02:533
174499556,19cyclictest7242-21latency_hist09:02:300
1764995434,12cyclictest0-21swapper/307:26:403
1764995434,12cyclictest0-21swapper/307:26:403
1764995416,12cyclictest141rcu_preempt07:22:273
1764995411,15cyclictest41-21ksoftirqd/307:57:493
1764995411,15cyclictest41-21ksoftirqd/307:57:493
1764995234,10cyclictest0-21swapper/310:57:273
176499523,10cyclictest0-21swapper/311:50:463
176499503,30cyclictest0-21swapper/308:46:513
176499503,30cyclictest0-21swapper/308:46:513
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional