You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-02-07 - 13:19

ARM Freescale i.MX6 Quad @996 MHz, Linux 5.16.2-rt19 (Profile)

Latency plot of system in rack #b, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rackbslot7 (updated Tue Feb 07, 2023 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3007121399,102sleep00-21swapper/019:01:580
3007121399,102sleep00-21swapper/019:01:580
303712995,13sleep30-21swapper/319:02:343
303712995,13sleep30-21swapper/319:02:343
3082499909,57cyclictest23220-21/usr/sbin/munin22:31:421
30824999068,14cyclictest29131-21unixbench_multi00:32:051
30824999068,14cyclictest29131-21unixbench_multi00:32:051
3084499843,16cyclictest7017-21runrttasks19:31:143
3084499824,15cyclictest3494-21timerwakeupswit21:56:493
3084499823,17cyclictest29697-21idleruntime-cro20:41:123
3084499823,17cyclictest29697-21idleruntime-cro20:41:123
3084499822,19cyclictest3895-21cat21:01:163
3084499822,19cyclictest3895-21cat21:01:163
3084499822,18cyclictest26941-21grep20:31:523
30834998224,48cyclictest278-21systemd-journal20:36:122
30810998252,13cyclictest19752-21fschecks_count00:16:380
30810998252,13cyclictest19752-21fschecks_count00:16:380
30810998236,31cyclictest141rcu_preempt21:06:410
3084499813,11cyclictest30594-21latency_hist00:36:133
3084499813,11cyclictest30594-21latency_hist00:36:133
3084499812,19cyclictest397-21runrttasks23:20:153
3084499812,17cyclictest4131-21grep19:21:393
3084499795,16cyclictest5474-21sort23:51:383
3084499795,12cyclictest26546-21latency_hist20:31:153
3084499788,59cyclictest2541-21taskset22:51:333
3084499783,17cyclictest16042-21awk19:56:483
3084499782,12cyclictest27073-21grep22:36:503
30844997811,12cyclictest14357-21expr19:51:583
30844997811,12cyclictest14357-21expr19:51:583
3083499783,15cyclictest15299-21grep22:16:502
3082499789,43cyclictest2536-21df_inode22:51:331
30810997857,15cyclictest4941-21kworker/u8:422:56:120
30810997835,35cyclictest0-21swapper/021:46:120
3084499774,15cyclictest14498-21uname23:11:383
3084499774,14cyclictest5377-21basename21:06:123
3084499773,9cyclictest28956-21apt-get22:41:263
30844997736,35cyclictest22651-21latency_hist23:26:143
30844997736,35cyclictest22651-21latency_hist23:26:143
3084499773,14cyclictest17825-21unixbench_multi20:01:543
3084499773,14cyclictest17825-21unixbench_multi20:01:543
3084499773,13cyclictest19766-21modprobe22:26:123
3084499773,12cyclictest11294-21ssh22:11:293
3084499772,15cyclictest23196-21latency_hist21:36:163
3081099775,14cyclictest12-21ksoftirqd/021:11:550
3084499764,16cyclictest3574-21modprobe19:21:123
3084499764,16cyclictest20257-21/usr/sbin/munin22:26:323
3084499763,16cyclictest397-21runrttasks22:33:463
3084499763,15cyclictest32004-21fschecks_time23:41:353
3084499763,14cyclictest23361-21sed00:21:553
3084499763,14cyclictest23361-21sed00:21:553
3084499763,12cyclictest0-21swapper/319:41:513
3084499763,12cyclictest0-21swapper/319:41:513
30844997558,10cyclictest6291-21unixbench_multi22:56:493
30844997556,12cyclictest5950-21fschecks_count21:06:413
3084499755,15cyclictest10443-21ssh23:05:053
3084499754,17cyclictest397-21runrttasks20:18:223
3084499753,18cyclictest19983-21cat00:16:423
3084499753,18cyclictest19983-21cat00:16:423
3084499753,16cyclictest2129-21modprobe20:56:133
3084499753,16cyclictest2129-21modprobe20:56:133
3084499753,11cyclictest397-21runrttasks21:29:523
3084499752,15cyclictest8183-21latency_hist22:06:153
3084499752,14cyclictest9723-21/usr/sbin/munin21:11:563
3084499752,10cyclictest28836-21ssh23:36:283
30810997553,14cyclictest0-21swapper/000:11:530
3084499743,20cyclictest768-21fschecks_count19:11:393
3084499743,20cyclictest768-21fschecks_count19:11:393
3084499743,19cyclictest29574-21wc21:46:373
3084499743,19cyclictest29574-21wc21:46:373
3084499743,18cyclictest20495-21kernelversion20:11:403
3084499743,15cyclictest12922-21sed19:46:503
3084499743,14cyclictest14613-21cpuspeed21:21:373
3084499743,14cyclictest14613-21cpuspeed21:21:373
3084499743,14cyclictest11069-21latency_hist22:11:143
3084499743,10cyclictest2223-21apt-get23:46:263
3084499743,10cyclictest2223-21apt-get23:46:263
3084499742,19cyclictest2320-21latency_hist21:56:143
3084499742,19cyclictest2320-21latency_hist21:56:143
30844997410,8cyclictest0-21swapper/322:16:363
30824997457,10cyclictest11768-21munin-plugin-st19:46:121
30824997457,10cyclictest11768-21munin-plugin-st19:46:121
3081099744,42cyclictest28089-21apt-get00:31:360
3081099744,42cyclictest28089-21apt-get00:31:360
307292748,40sleep230728-21/usr/sbin/munin19:06:152
307292748,40sleep230728-21/usr/sbin/munin19:06:152
3084499734,18cyclictest23212-21fschecks_time23:26:353
3084499734,14cyclictest32382-21latency_hist19:11:143
3084499733,16cyclictest9084-21ssh23:56:543
3084499733,13cyclictest13986-21cat23:11:153
3084499733,13cyclictest13986-21cat23:11:153
3084499732,17cyclictest11406-21uname00:01:463
3084499732,17cyclictest11406-21uname00:01:463
3084499732,16cyclictest27866-21latency_hist00:31:143
3083499732,54cyclictest2925-21sshd20:56:582
3083499732,54cyclictest2925-21sshd20:56:582
3083499732,15cyclictest11341-21latency_hist21:16:152
30834997321,40cyclictest0-21swapper/200:01:142
30824997343,11cyclictest141rcu_preempt23:03:581
30824997312,25cyclictest25-21ksoftirqd/122:56:401
3084499724,13cyclictest9065-21grep19:36:383
3084499724,13cyclictest278-21systemd-journal23:36:123
3084499723,17cyclictest24318-21grep21:36:503
3084499723,17cyclictest24318-21grep21:36:503
3084499723,16cyclictest31957-21ls22:46:313
3084499722,18cyclictest17627-21grep00:11:533
3084499722,17cyclictest24337-21unixbench_singl20:21:503
3084499722,11cyclictest26737-21/usr/sbin/munin21:41:423
3083499724,10cyclictest23115-21ssh23:26:342
3083499723,15cyclictest27253-21sed21:41:512
3083499722,15cyclictest28840-21latency_hist22:41:152
3083499722,12cyclictest23751-21/usr/sbin/munin21:36:402
3083499722,12cyclictest23751-21/usr/sbin/munin21:36:402
3083499722,12cyclictest21399-21unixbench_multi21:31:582
3084499715,11cyclictest16301-21idleruntime-cro00:11:113
3084499713,17cyclictest7271-21ls19:31:363
3084499713,16cyclictest11654-21sort21:16:353
3084499712,17cyclictest31389-21latency_hist20:46:143
3083499714,10cyclictest12510-21/usr/sbin/munin19:46:422
3083499713,61cyclictest5204-21munin-run19:26:122
3083499713,56cyclictest18816-21idleruntime20:06:392
3083499713,11cyclictest17091-21modprobe21:26:122
3083499713,11cyclictest17091-21modprobe21:26:122
3083499712,17cyclictest19389-21grep00:16:162
3083499712,14cyclictest10661-21fschecks_time19:41:382
3083499712,14cyclictest10661-21fschecks_time19:41:382
3083499712,12cyclictest14288-21/usr/sbin/munin00:06:442
3083499712,10cyclictest30198-21fschecks_time20:41:392
30824997140,18cyclictest141rcu_preempt21:40:551
30824997140,18cyclictest141rcu_preempt21:40:551
30810997133,12cyclictest141rcu_preempt00:26:580
30810997132,21cyclictest141rcu_preempt23:39:300
30810997123,13cyclictest141rcu_preempt00:01:530
30810997123,13cyclictest141rcu_preempt00:01:530
3084499702,13cyclictest18734-21fschecks_count20:06:373
3083499703,59cyclictest31772-21fschecks_count20:46:382
3083499703,59cyclictest29055-21cut23:36:362
3083499703,59cyclictest16709-21ssh23:16:112
3083499703,56cyclictest670-21head21:51:482
3083499703,56cyclictest670-21head21:51:482
3083499703,14cyclictest5190-21latency_hist22:56:152
3083499703,14cyclictest22694-21grep20:16:502
3083499703,10cyclictest4733-21/usr/sbin/munin21:01:582
3083499702,9cyclictest17088-21seq23:16:272
3083499702,17cyclictest28367-21tune2fs00:31:492
3083499702,17cyclictest28367-21tune2fs00:31:492
3083499702,14cyclictest32461-21grep19:11:142
3083499702,14cyclictest25894-21latency_hist22:36:142
3083499702,12cyclictest1905-21cut23:46:122
3083499702,12cyclictest15222-21latency_hist19:56:152
3083499702,12cyclictest15222-21latency_hist19:56:152
3083499702,12cyclictest14382-21cat21:21:152
30824997041,8cyclictest141rcu_preempt22:28:281
30824997034,12cyclictest0-21swapper/119:56:501
3053727038,14sleep10-21swapper/119:04:391
3053727038,14sleep10-21swapper/119:04:391
3084499693,16cyclictest646-21latency_hist20:51:283
3083499693,58cyclictest6479-21tail19:27:022
3083499693,57cyclictest8461-21latency_hist21:11:152
3083499693,55cyclictest9068-21fschecks_time19:36:382
3083499693,55cyclictest26513-21latency_hist20:31:152
3083499693,54cyclictest23799-21latency20:21:392
3083499693,14cyclictest803-21cut19:11:392
3083499693,14cyclictest803-21cut19:11:392
3083499693,13cyclictest16233-21grep19:56:502
3083499692,57cyclictest32136-21cut22:46:362
3083499692,14cyclictest18474-21/usr/sbin/munin21:26:542
30834996921,41cyclictest0-21swapper/219:16:472
3083499692,13cyclictest17062-21/usr/sbin/munin20:01:372
3083499692,13cyclictest17062-21/usr/sbin/munin20:01:372
3083499692,12cyclictest11361-21/usr/sbin/munin00:01:442
3083499692,12cyclictest11361-21/usr/sbin/munin00:01:442
30824996935,13cyclictest141rcu_preempt00:01:441
30824996935,13cyclictest141rcu_preempt00:01:441
3081099697,35cyclictest12-21ksoftirqd/023:51:370
30810996919,13cyclictest141rcu_preempt21:16:520
3083499683,13cyclictest3591-21unixbench_singl21:56:502
3083499683,13cyclictest29252-21grep20:36:512
3083499683,13cyclictest29252-21grep20:36:512
3083499683,13cyclictest27649-21latency_hist00:31:112
3083499683,10cyclictest24850-21true00:26:122
3083499683,10cyclictest24850-21true00:26:122
3083499683,10cyclictest20340-21grep22:26:352
3083499682,16cyclictest17135-21/usr/sbin/munin22:21:402
3083499682,15cyclictest5940-21ps22:56:452
3083499682,12cyclictest8905-21/usr/sbin/munin23:01:492
3083499682,11cyclictest10950-21munin-run22:11:122
30834996815,42cyclictest0-21swapper/222:41:412
30824996832,12cyclictest141rcu_preempt20:01:471
30824996832,12cyclictest141rcu_preempt20:01:471
3083499673,56cyclictest4921-21latency_hist23:51:142
3083499673,56cyclictest4921-21latency_hist23:51:142
3083499673,55cyclictest29409-21fschecks_time21:46:342
3083499673,55cyclictest29409-21fschecks_time21:46:342
3083499673,13cyclictest5674-21/usr/sbin/munin22:01:402
3083499673,11cyclictest970-21sort20:51:432
3083499673,11cyclictest970-21sort20:51:432
3083499672,57cyclictest26114-21grep23:31:382
3083499672,14cyclictest22774-21ssh23:26:152
3083499672,14cyclictest22774-21ssh23:26:152
3083499672,14cyclictest11836-21proc_pri23:06:482
3083499672,14cyclictest11836-21proc_pri23:06:482
30824996748,13cyclictest13781-21munin-plugin-st23:11:121
30824996748,13cyclictest13781-21munin-plugin-st23:11:121
30824996746,13cyclictest23409-21unixbench_singl00:21:551
30824996746,13cyclictest23409-21unixbench_singl00:21:551
30824996738,14cyclictest141rcu_preempt19:17:511
30824996735,10cyclictest141rcu_preempt19:15:211
30824996735,10cyclictest141rcu_preempt19:15:211
30824996732,12cyclictest141rcu_preempt20:40:451
30824996732,12cyclictest141rcu_preempt20:40:451
30824996731,7cyclictest141rcu_preempt20:10:561
3082499672,30cyclictest25-21ksoftirqd/121:22:551
3082499672,30cyclictest25-21ksoftirqd/121:22:551
3081099678,38cyclictest12-21ksoftirqd/023:46:270
3081099678,38cyclictest12-21ksoftirqd/023:46:270
30810996747,11cyclictest17254-21ssh21:26:160
30810996747,11cyclictest17254-21ssh21:26:160
318192662,52sleep031904-21/usr/sbin/munin23:41:310
3083499663,51cyclictest7440-21fschecks_time19:31:422
3083499663,13cyclictest20518-21latency20:11:412
30834996626,31cyclictest0-21swapper/222:11:362
3083499662,15cyclictest5340-21/usr/sbin/munin23:51:342
3083499662,10cyclictest20009-21/usr/sbin/munin00:16:442
3083499662,10cyclictest20009-21/usr/sbin/munin00:16:442
30824996636,15cyclictest141rcu_preempt23:54:401
30824996635,7cyclictest141rcu_preempt21:04:541
30824996635,13cyclictest141rcu_preempt20:56:211
30824996635,13cyclictest141rcu_preempt20:56:211
30824996631,8cyclictest141rcu_preempt00:14:321
30810996617,10cyclictest0-21swapper/022:31:490
3082499659,29cyclictest25-21ksoftirqd/122:18:421
30824996544,7cyclictest141rcu_preempt22:15:361
30824996538,12cyclictest141rcu_preempt00:30:481
30824996533,10cyclictest141rcu_preempt21:13:311
30824996510,35cyclictest25-21ksoftirqd/121:19:521
30824996510,29cyclictest25-21ksoftirqd/123:43:311
3081099658,30cyclictest12-21ksoftirqd/023:36:120
30810996539,21cyclictest12-21ksoftirqd/019:11:240
30810996539,21cyclictest12-21ksoftirqd/019:11:240
30810996520,19cyclictest141rcu_preempt22:56:490
30824996445,11cyclictest0-21swapper/123:11:501
30824996444,7cyclictest141rcu_preempt19:37:331
30824996433,25cyclictest241ktimers/122:45:081
30824996413,27cyclictest0-21swapper/121:31:371
30824996413,24cyclictest0-21swapper/120:31:001
30810996445,10cyclictest0-21swapper/021:31:570
30810996440,15cyclictest552-21in:imklog22:11:350
30810996419,23cyclictest141rcu_preempt19:26:500
137232640,4chrt13721-21ssh23:11:050
137232640,4chrt13721-21ssh23:11:050
30824996344,12cyclictest15002-21idleruntime-cro19:56:121
30824996344,12cyclictest15002-21idleruntime-cro19:56:121
30824996335,7cyclictest141rcu_preempt20:48:091
30824996318,22cyclictest552-21in:imklog21:30:531
30824996316,26cyclictest0-21swapper/121:53:561
30824996316,26cyclictest0-21swapper/121:53:561
3081099636,21cyclictest0-21swapper/021:31:110
30824996238,7cyclictest141rcu_preempt23:31:511
30824996234,9cyclictest141rcu_preempt22:04:541
30824996227,8cyclictest141rcu_preempt22:24:361
30824996226,7cyclictest141rcu_preempt20:32:361
30824996210,31cyclictest25-21ksoftirqd/123:48:161
30824996210,31cyclictest25-21ksoftirqd/123:48:161
30824996210,18cyclictest0-21swapper/121:43:091
3081099628,24cyclictest12-21ksoftirqd/023:56:530
30810996243,12cyclictest16646-21munin-plugin-st20:01:120
30810996240,16cyclictest10830-21kworker/u8:423:06:110
30810996236,19cyclictest8150irq/69-2188000.23:21:370
30810996236,19cyclictest8150irq/69-2188000.23:21:370
30810996214,8cyclictest141rcu_preempt21:46:290
30810996214,8cyclictest141rcu_preempt21:46:290
30810996210,19cyclictest12-21ksoftirqd/020:06:410
3082499616,31cyclictest25-21ksoftirqd/100:18:491
3082499616,31cyclictest25-21ksoftirqd/100:18:491
30824996139,7cyclictest141rcu_preempt21:07:421
30824996139,11cyclictest0-21swapper/119:06:481
30824996134,8cyclictest141rcu_preempt19:29:191
30824996132,9cyclictest141rcu_preempt20:43:071
30824996131,16cyclictest141rcu_preempt23:59:271
30824996130,8cyclictest141rcu_preempt20:16:271
30824996127,13cyclictest141rcu_preempt19:50:231
30824996125,19cyclictest116-21pr/ttymxc021:49:171
30824996125,19cyclictest116-21pr/ttymxc021:49:171
30824996115,25cyclictest0-21swapper/122:40:531
30824996111,23cyclictest25-21ksoftirqd/123:25:481
30824996111,23cyclictest25-21ksoftirqd/123:25:481
30810996146,9cyclictest16736-21ls23:16:110
30810996142,11cyclictest131ktimers/020:26:340
30810996133,18cyclictest0-21swapper/020:31:520
3081099613,18cyclictest32487-21unixbench_multi20:46:520
3082499607,24cyclictest25-21ksoftirqd/119:33:171
30824996031,10cyclictest141rcu_preempt23:27:011
30824996026,13cyclictest141rcu_preempt20:51:581
30824996026,13cyclictest141rcu_preempt20:51:581
30810996041,11cyclictest23132-21munin-plugin-st20:21:120
30810996037,14cyclictest14492-21fschecks_time22:16:350
30810996018,21cyclictest141rcu_preempt23:16:220
30810996013,8cyclictest0-21swapper/020:41:300
3082499599,28cyclictest25-21ksoftirqd/119:21:281
30824995934,12cyclictest141rcu_preempt00:08:201
30824995933,7cyclictest141rcu_preempt20:11:171
30824995928,16cyclictest141rcu_preempt22:47:491
30824995914,20cyclictest25-21ksoftirqd/122:06:501
30810995940,10cyclictest0-21swapper/019:41:520
30810995940,10cyclictest0-21swapper/019:41:520
30810995937,15cyclictest24886-21apt-key00:26:120
30810995937,15cyclictest24886-21apt-key00:26:120
30810995936,12cyclictest0-21swapper/020:57:040
30810995936,12cyclictest0-21swapper/020:57:040
30810995921,12cyclictest141rcu_preempt23:26:310
30810995919,21cyclictest141rcu_preempt22:06:080
30824995832,13cyclictest141rcu_preempt23:16:171
30824995827,8cyclictest141rcu_preempt23:40:561
30810995842,11cyclictest52-21kcompactd022:06:270
3081099584,18cyclictest32046-21fschecks_count22:46:350
30810995839,9cyclictest0-21swapper/021:36:500
30810995839,9cyclictest0-21swapper/021:36:500
30810995827,10cyclictest141rcu_preempt21:01:520
30810995820,8cyclictest141rcu_preempt22:39:550
30810995818,21cyclictest141rcu_preempt00:10:000
30810995812,22cyclictest12-21ksoftirqd/020:16:150
30824995736,10cyclictest0-21swapper/121:56:351
30824995716,14cyclictest25-21ksoftirqd/120:23:471
3081099574,16cyclictest23651-21cut20:21:360
3081099573,43cyclictest12992-21grep19:46:500
30810995733,14cyclictest0-21swapper/019:26:160
30810995717,9cyclictest0-21swapper/019:35:070
3081099565,25cyclictest12-21ksoftirqd/021:56:110
3081099565,25cyclictest12-21ksoftirqd/021:56:110
30810995634,7cyclictest12-21ksoftirqd/020:36:320
30810995634,7cyclictest12-21ksoftirqd/020:36:320
30810995633,13cyclictest0-21swapper/022:31:120
30810995618,12cyclictest141rcu_preempt20:06:050
30810995618,12cyclictest141rcu_preempt20:06:050
3081099556,22cyclictest5355-21diskmemload22:21:380
30810995534,11cyclictest0-21swapper/019:06:340
30810995532,15cyclictest0-21swapper/022:01:120
3081099553,14cyclictest29192-21ssh22:41:360
30810995518,8cyclictest141rcu_preempt19:51:450
30810995518,8cyclictest141rcu_preempt19:51:450
30810995432,11cyclictest0-21swapper/019:36:510
3081099534,28cyclictest2237-21apt-get19:16:240
30810995332,14cyclictest0-21swapper/020:51:390
30810995332,14cyclictest0-21swapper/020:51:390
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional