You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-02-24 - 04:10
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7 (updated Fri Feb 23, 2024 12:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1454521180,4chrt14370-21/usr/sbin/munin07:34:440
1454521180,4chrt14370-21/usr/sbin/munin07:34:440
4541210114,62sleep04172-21/usr/sbin/munin07:04:560
5698998728,11cyclictest14608-21ssh11:09:053
5698998540,36cyclictest0-21swapper/309:39:173
5690998562,14cyclictest5038-21fschecks_count11:49:452
569899833,22cyclictest31834-21grep10:40:003
569899823,24cyclictest25040-21grep09:29:583
569899823,24cyclictest25040-21grep09:29:583
569899823,18cyclictest9036-21grep08:55:003
5690998261,14cyclictest773-21unixbench_singl09:45:012
5682998230,20cyclictest12607-21cut07:29:201
5682998230,20cyclictest12607-21cut07:29:201
5698998123,12cyclictest0-21swapper/310:14:443
5698998122,12cyclictest0-21swapper/308:59:583
5698998121,50cyclictest0-21swapper/312:34:453
5698998118,11cyclictest0-21swapper/310:02:253
5698998118,11cyclictest0-21swapper/310:02:253
569899803,20cyclictest16089-21fschecks_time09:14:473
569899803,19cyclictest15914-21latency_hist07:39:203
569899803,19cyclictest15914-21latency_hist07:39:203
569899793,22cyclictest16250-21fschecks_time07:39:453
569899793,20cyclictest23391-21grep07:59:593
5698997922,49cyclictest28277-21apt-key11:34:193
5698997919,10cyclictest0-21swapper/310:49:193
5698997919,10cyclictest0-21swapper/310:49:193
5698997910,13cyclictest0-21swapper/310:34:203
5682997933,17cyclictest2927-21tail11:45:021
5682997933,17cyclictest2927-21tail11:45:021
569899783,18cyclictest25040-21unixbench_multi08:05:003
5698997822,6cyclictest0-21swapper/312:10:013
569899782,20cyclictest32371-21fschecks_count09:44:453
569899773,21cyclictest276-21systemd-journal11:04:173
569899773,20cyclictest20218-21/usr/sbin/munin12:14:543
569899773,19cyclictest5218-21grep10:50:053
5698997725,46cyclictest0-21swapper/309:50:003
5698997725,46cyclictest0-21swapper/309:50:003
5698997722,14cyclictest276-21systemd-journal08:29:183
5698997722,14cyclictest276-21systemd-journal08:29:183
5698997719,37cyclictest417-21snmpd11:34:423
5698997719,37cyclictest417-21snmpd11:34:423
5698997718,9cyclictest11148-21fschecks_time10:04:483
5698997715,13cyclictest0-21swapper/311:14:473
5682997747,23cyclictest0-21swapper/107:14:201
569899764,19cyclictest23450-21fschecks_count11:24:443
569899764,17cyclictest5845-21grep11:49:593
569899763,26cyclictest20887-21cut11:19:463
569899763,18cyclictest7786-21latency_hist09:59:183
569899763,18cyclictest7383-21grep08:49:563
569899763,18cyclictest7383-21grep08:49:563
5698997619,12cyclictest0-21swapper/311:44:183
5698997611,25cyclictest12722-21diskmemload09:24:193
5682997652,14cyclictest10768-21unixbench_multi11:00:001
569899754,18cyclictest2963-21modprobe08:39:173
569899753,24cyclictest28048-21cpuspeed10:34:423
569899753,22cyclictest26825-21cat08:10:203
5682997530,18cyclictest14486-21seq11:08:321
5675997537,28cyclictest28420-21latency_hist11:34:190
567599753,52cyclictest4130-21sshd10:49:300
569899744,21cyclictest15034-21/usr/sbin/munin11:09:423
569899743,20cyclictest15488-21systemd-run09:14:183
569899743,20cyclictest13352-21apt-key12:04:193
5698997422,9cyclictest0-21swapper/307:24:193
569899742,22cyclictest8138-21fschecks_time07:14:483
569899742,20cyclictest14650-21unixbench_multi10:10:003
569899742,20cyclictest12959-21latency_hist09:09:213
569899732,21cyclictest27816-21taskset12:29:003
569899732,21cyclictest12065-21sed07:25:003
569899732,21cyclictest12065-21sed07:25:003
5698997312,12cyclictest0-21swapper/310:24:193
5690997343,18cyclictest19268-21latency_hist10:19:202
5675997318,31cyclictest12-21ksoftirqd/010:34:370
569899729,13cyclictest0-21swapper/307:29:463
569899726,15cyclictest0-21swapper/307:09:593
569899724,22cyclictest28286-21switchtime08:15:023
569899723,18cyclictest7956-21ssh10:55:013
569899722,24cyclictest22494-21fschecks_count10:24:443
569899714,16cyclictest674-21mailstats08:29:593
569899714,15cyclictest5732-21latency_hist07:09:223
569899714,10cyclictest0-21swapper/311:47:343
569899714,10cyclictest0-21swapper/311:47:343
569899713,20cyclictest29490-21fschecks_time08:19:483
569899713,18cyclictest8349-21ntp_states11:54:543
569899713,18cyclictest8349-21ntp_states11:54:543
5682997139,25cyclictest27975-21rm10:34:381
5682997123,20cyclictest6538-21sh11:51:541
454627119,27sleep10-21swapper/107:04:571
569899709,16cyclictest0-21swapper/308:39:593
569899709,16cyclictest0-21swapper/308:39:593
569899704,21cyclictest29008-21mailstats12:29:583
5690997045,14cyclictest0-21swapper/211:00:002
5682997017,18cyclictest24798-21systemd-run10:29:181
5675997050,13cyclictest22017-21ssh10:24:200
569899693,25cyclictest5379-21ntpq08:44:553
5690996950,12cyclictest13445-21apt-key12:04:192
5690996945,15cyclictest0-21swapper/208:29:172
5690996945,15cyclictest0-21swapper/208:29:172
5682996921,18cyclictest30637-21diskmemload12:33:401
5682996917,21cyclictest13580-21dpkg12:04:201
567599699,35cyclictest14478-21ntp_states12:05:060
5675996949,11cyclictest5878-21ssh11:49:590
5675996944,15cyclictest30999-21fschecks_count10:39:460
5675996937,18cyclictest141rcu_preempt11:04:470
569899684,26cyclictest0-21swapper/312:08:363
5675996846,13cyclictest30466-21unixbench_multi09:39:590
5675996846,13cyclictest30466-21unixbench_multi09:39:590
5675996839,22cyclictest6985-21acpi10:54:340
569899677,14cyclictest0-21swapper/312:19:583
5698996746,14cyclictest22422-21unixbench_singl09:25:033
569899674,19cyclictest399-21runrttasks07:54:473
5690996746,12cyclictest0-21swapper/209:44:172
5690996746,12cyclictest0-21swapper/209:44:172
5682996727,15cyclictest12722-21diskmemload10:20:411
5675996746,7cyclictest141rcu_preempt11:00:070
5675996738,10cyclictest141rcu_preempt08:49:470
5675996738,10cyclictest141rcu_preempt08:49:470
5682996650,10cyclictest11059-21fschecks_count10:04:471
568299662,44cyclictest16754-21ssh10:14:441
5675996618,9cyclictest141rcu_preempt09:09:450
497326625,18sleep20-21swapper/207:05:102
569899652,19cyclictest29457-21/usr/sbin/munin09:39:453
569899652,19cyclictest29457-21/usr/sbin/munin09:39:453
5682996548,11cyclictest22219-21munin-plugin-st07:59:171
5682996541,15cyclictest26902-21runrttasks09:34:431
5682996534,22cyclictest22506-21apt-get07:59:381
5675996517,29cyclictest0-21swapper/012:35:140
5675996510,25cyclictest276-21systemd-journal12:24:170
5690996447,11cyclictest2993-21idleruntime-cro08:39:172
5682996447,11cyclictest28358-21latency_hist11:34:191
5682996446,12cyclictest2232-21latency_hist09:49:191
5682996443,12cyclictest0-21swapper/107:49:171
5682996442,12cyclictest25673-21apt-key08:09:201
5682996440,12cyclictest15055-21apt-get11:09:411
5682996433,22cyclictest22871-21/usr/sbin/munin12:19:481
5682996410,39cyclictest28733-21apt-get11:34:421
5682996410,39cyclictest28733-21apt-get11:34:421
5675996442,13cyclictest0-21swapper/009:39:170
5690996342,11cyclictest0-21swapper/208:19:172
5690996338,19cyclictest6731-21latency_hist10:54:192
5682996329,27cyclictest7729-21apt-get11:54:401
5682996329,27cyclictest7729-21apt-get11:54:401
567599639,36cyclictest141rcu_preempt12:10:010
5675996341,15cyclictest11230-21ntp_kernel_pll_11:59:520
5690996245,11cyclictest276-21systemd-journal07:24:202
5690996243,11cyclictest1-21systemd10:59:182
5682996236,17cyclictest417-21snmpd09:32:541
5682996236,17cyclictest417-21snmpd09:32:541
5682996234,17cyclictest0-21swapper/110:49:411
5682996233,21cyclictest18634-21ssh09:19:351
5675996245,11cyclictest9132-21unixbench_singl10:00:020
5675996245,11cyclictest9132-21unixbench_singl10:00:020
567599623,35cyclictest1555-21wc08:34:200
5675996213,34cyclictest0-21swapper/007:49:590
5675996211,28cyclictest12-21ksoftirqd/007:39:480
5690996137,13cyclictest0-21swapper/211:44:492
5690996137,13cyclictest0-21swapper/211:44:492
5682996137,15cyclictest13927-21threads09:10:001
5682996136,16cyclictest8015-21unixbench_singl10:55:021
5682996134,18cyclictest314-21/usr/sbin/munin08:29:501
5682996127,23cyclictest25526-21/usr/sbin/munin10:29:511
5682996119,16cyclictest12722-21diskmemload09:52:171
5682996119,16cyclictest12722-21diskmemload09:52:171
5682996118,16cyclictest399-21runrttasks08:22:001
5675996143,11cyclictest11110-21munin-plugin-st09:04:170
5675996122,12cyclictest0-21swapper/010:12:560
5675996113,38cyclictest0-21swapper/011:59:190
5675996113,38cyclictest0-21swapper/011:59:190
5675996111,30cyclictest12-21ksoftirqd/009:45:010
569099604,38cyclictest28777-21ssh11:34:422
569099604,38cyclictest28777-21ssh11:34:422
5690996043,11cyclictest28188-21gpgv11:34:172
5690996040,10cyclictest0-21swapper/211:54:592
5690996040,10cyclictest0-21swapper/211:54:592
5690996035,13cyclictest0-21swapper/208:49:172
5682996041,13cyclictest29763-21fschecks_time09:39:451
5682996041,13cyclictest29763-21fschecks_time09:39:451
5675996043,10cyclictest23860-21kworker/u8:208:04:170
5675996041,11cyclictest0-21swapper/009:09:170
5675996038,12cyclictest0-21swapper/012:30:010
5675996035,19cyclictest417-21snmpd09:18:550
567599603,13cyclictest28713-21/usr/sbin/munin11:34:570
567599603,13cyclictest28713-21/usr/sbin/munin11:34:570
567599602,40cyclictest0-21swapper/008:13:580
5675996017,27cyclictest0-21swapper/011:47:290
5675996017,27cyclictest0-21swapper/011:47:290
5690995943,11cyclictest26220-21threads12:25:002
5682995940,13cyclictest14660-21unixbench_multi10:10:011
5682995929,20cyclictest31436-21/usr/sbin/munin10:39:561
5682995919,33cyclictest5519-21tr08:45:001
5675995945,9cyclictest12-21ksoftirqd/009:57:420
5675995943,10cyclictest26821-21rm10:31:560
5675995942,11cyclictest20304-21ssh12:14:570
5675995941,10cyclictest3577-21cron10:49:160
5675995941,10cyclictest3577-21cron10:49:160
5675995932,13cyclictest141rcu_preempt09:49:490
5675995932,13cyclictest141rcu_preempt09:49:490
5675995930,15cyclictest0-21swapper/011:25:390
569899583,19cyclictest20577-21cron07:54:173
569099589,20cyclictest141rcu_preempt09:34:192
569099589,20cyclictest141rcu_preempt09:34:192
5690995838,14cyclictest29179-21latency_hist09:39:192
5690995837,12cyclictest0-21swapper/212:05:002
5682995837,9cyclictest21388-21apt-get09:24:381
5682995836,16cyclictest28092-21taskset12:29:191
5682995835,15cyclictest23148-21runrttasks11:24:251
5682995835,15cyclictest18076-21chrt09:18:401
5682995834,17cyclictest8423-21/usr/sbin/munin08:54:481
5682995834,15cyclictest1903-21switchtime10:45:021
5682995834,15cyclictest1903-21switchtime10:45:021
5682995833,13cyclictest0-21swapper/108:14:201
5682995832,16cyclictest14607-21fschecks_time07:34:461
5682995832,16cyclictest14607-21fschecks_time07:34:461
5682995831,20cyclictest31534-21grep12:34:451
5682995811,23cyclictest141rcu_preempt12:09:421
5675995840,11cyclictest24971-21timerwakeupswit08:05:000
5675995834,9cyclictest141rcu_preempt11:12:140
5675995826,11cyclictest141rcu_preempt10:24:480
5690995738,10cyclictest0-21swapper/212:09:412
5690995734,13cyclictest0-21swapper/208:55:002
5682995733,16cyclictest8873-21/usr/sbin/munin07:15:021
5682995733,16cyclictest417-21snmpd11:15:531
5682995729,18cyclictest7002-21/usr/sbin/munin08:49:491
5682995729,18cyclictest7002-21/usr/sbin/munin08:49:491
5682995726,23cyclictest8590-21chrt09:59:521
5682995726,23cyclictest8590-21chrt09:59:521
5682995713,18cyclictest399-21runrttasks09:06:461
567599576,27cyclictest0-21swapper/011:22:570
5675995743,9cyclictest30573-21cron08:24:170
5675995722,15cyclictest24835-21switchtime09:29:570
5675995722,15cyclictest24835-21switchtime09:29:570
5675995710,28cyclictest141rcu_preempt07:54:380
189342570,5sleep3391rcuc/307:48:583
5690995634,14cyclictest12722-21diskmemload11:10:012
5690995633,9cyclictest0-21swapper/207:29:582
5690995631,15cyclictest3520-21unixbench_singl09:50:002
5690995631,15cyclictest3520-21unixbench_singl09:50:002
5690995626,7cyclictest0-21swapper/210:39:582
5682995632,15cyclictest9834-21/usr/sbin/munin08:59:401
5682995630,19cyclictest6965-21rm09:56:591
5682995629,20cyclictest0-21swapper/107:29:361
5682995629,19cyclictest17483-21latency_hist07:44:191
5682995628,19cyclictest1700-21latency_hist11:44:191
5682995621,26cyclictest27605-21cpu08:14:441
567599568,8cyclictest141rcu_preempt08:59:210
567599565,22cyclictest12-21ksoftirqd/007:45:580
5675995644,7cyclictest0-21swapper/008:39:590
5675995644,7cyclictest0-21swapper/008:39:590
5675995622,11cyclictest141rcu_preempt10:18:380
5690995537,9cyclictest0-21swapper/211:19:202
5690995534,14cyclictest1-21systemd07:19:182
5690995532,14cyclictest276-21systemd-journal07:57:062
5690995531,13cyclictest0-21swapper/210:49:192
5690995531,13cyclictest0-21swapper/210:49:192
5690995530,12cyclictest0-21swapper/209:29:212
5690995527,15cyclictest0-21swapper/209:10:012
5682995535,9cyclictest9549-21apt-get07:19:381
5682995531,16cyclictest20196-21sed07:50:011
5682995528,20cyclictest31937-21chrt08:26:181
5682995528,20cyclictest31937-21chrt08:26:181
5682995525,23cyclictest21171-21ntp_kernel_pll_11:19:521
5682995525,22cyclictest0-21swapper/108:40:391
5682995525,22cyclictest0-21swapper/108:40:391
567599554,19cyclictest25340-21/usr/sbin/munin12:24:490
5675995539,10cyclictest347-21in:imklog09:20:480
5675995535,12cyclictest0-21swapper/009:25:030
5675995524,25cyclictest0-21swapper/008:34:580
5675995522,12cyclictest141rcu_preempt07:14:480
5690995438,10cyclictest339-21systemd-logind09:24:182
5690995429,13cyclictest0-21swapper/207:44:492
5690995428,12cyclictest0-21swapper/209:09:202
5682995432,16cyclictest14794-21timerandwakeup12:05:121
5682995431,14cyclictest0-21swapper/112:14:401
5682995428,18cyclictest2983-21munin-plugin-st08:39:171
567599545,43cyclictest0-21swapper/011:14:370
567599544,15cyclictest4883-21apt-get08:44:400
5675995433,14cyclictest12-21ksoftirqd/010:09:200
567599542,15cyclictest32172-21cron08:29:170
567599542,15cyclictest32172-21cron08:29:170
569099535,16cyclictest12722-21diskmemload12:22:032
5690995338,8cyclictest0-21swapper/208:24:172
5690995336,12cyclictest0-21swapper/211:29:172
5690995336,11cyclictest339-21systemd-logind12:34:182
5690995334,10cyclictest19424-21apt-get10:19:402
5690995330,12cyclictest0-21swapper/209:59:192
5675995335,8cyclictest0-21swapper/008:15:030
567599533,19cyclictest32174-21awk11:39:540
567599532,30cyclictest11350-21fschecks_count07:24:460
567599532,30cyclictest11350-21fschecks_count07:24:460
5690995233,12cyclictest276-21systemd-journal08:04:172
5690995233,11cyclictest0-21swapper/209:15:012
5690995230,14cyclictest0-21swapper/212:39:162
5690995230,12cyclictest0-21swapper/208:59:442
5690995226,15cyclictest0-21swapper/211:04:582
5690995211,7cyclictest141rcu_preempt10:39:172
5675995218,19cyclictest141rcu_preempt07:29:460
569099515,12cyclictest0-21swapper/208:09:472
569099514,12cyclictest141rcu_preempt10:30:012
5690995131,9cyclictest0-21swapper/210:09:192
5690995130,11cyclictest0-21swapper/211:20:002
5690995129,10cyclictest0-21swapper/210:14:212
5690995128,12cyclictest0-21swapper/207:39:572
5690995029,12cyclictest0-21swapper/207:34:562
5690995029,12cyclictest0-21swapper/207:34:562
5690995028,11cyclictest0-21swapper/209:59:412
5690995028,11cyclictest0-21swapper/209:59:412
5690995024,14cyclictest0-21swapper/207:14:202
5690994932,11cyclictest25462-21latency_hist08:09:202
5690994930,9cyclictest0-21swapper/211:39:502
5690994921,17cyclictest19666-21apt-get12:14:382
567599493,14cyclictest11002-21latency_hist07:24:200
5690994827,10cyclictest0-21swapper/208:54:212
5690994827,10cyclictest0-21swapper/208:54:212
5690994827,10cyclictest0-21swapper/207:49:562
5690994825,13cyclictest0-21swapper/208:39:462
5690994825,13cyclictest0-21swapper/208:39:462
5690994823,14cyclictest0-21swapper/208:34:202
567599484,27cyclictest12-21ksoftirqd/007:09:460
5690994710,8cyclictest0-21swapper/210:29:192
5690994625,11cyclictest0-21swapper/207:24:532
5690994625,11cyclictest0-21swapper/207:24:532
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional