You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-23 - 10:59
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7 (updated Sat Sep 23, 2023 00:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
464821116,14sleep00-21swapper/019:03:330
417621058,67sleep30-21swapper/319:03:163
5115998838,43cyclictest18663-21munin-run00:02:282
5115998838,43cyclictest18663-21munin-run00:02:282
5115998627,50cyclictest0-21swapper/220:57:512
5115998627,50cyclictest0-21swapper/220:57:512
5115998228,45cyclictest0-21swapper/222:37:582
5115997933,39cyclictest18258-21cpuspeed21:17:502
5115997922,49cyclictest0-21swapper/223:52:322
5115997911,14cyclictest0-21swapper/200:07:552
511599784,21cyclictest30683-21fschecks_time00:22:562
5115997817,55cyclictest0-21swapper/221:33:152
5124997752,15cyclictest9798-21latency_hist23:47:303
5124997710,54cyclictest5670-21kworker/u8:321:22:443
5115997730,39cyclictest0-21swapper/222:12:572
5115997722,45cyclictest0-21swapper/221:37:572
5115997722,45cyclictest0-21swapper/221:37:572
511599772,18cyclictest7179-21cpu23:42:512
511599772,18cyclictest4134-21timerandwakeup21:48:122
511599772,18cyclictest4134-21timerandwakeup21:48:122
5093997759,12cyclictest25649-21unixbench_multi00:13:080
5093997759,12cyclictest25649-21unixbench_multi00:13:080
5093997743,14cyclictest52-21kcompactd023:53:440
61742760,5sleep10-21swapper/123:40:491
511599769,12cyclictest0-21swapper/220:12:312
511599769,12cyclictest0-21swapper/220:12:312
511599768,12cyclictest0-21swapper/221:27:322
511599764,13cyclictest27983-21grep20:13:142
511599763,22cyclictest10336-21/usr/sbin/munin22:53:202
511599763,18cyclictest7251-21grep19:12:392
511599763,17cyclictest786-21cron23:32:282
511599763,17cyclictest786-21cron23:32:282
511599763,17cyclictest27897-21fschecks_count23:22:572
511599763,17cyclictest16302-21ssh22:08:312
511599763,15cyclictest21534-21fschecks_count22:18:022
511599762,21cyclictest6859-21grep20:48:132
511599762,18cyclictest21906-21cut19:57:292
511599754,22cyclictest30278-21fschecks_count22:32:562
511599753,23cyclictest2945-21fschecks_time20:37:572
511599752,18cyclictest889-21chrt00:27:502
5115997519,12cyclictest1-21systemd22:27:292
5115997518,11cyclictest0-21swapper/222:32:292
5115997517,11cyclictest0-21swapper/219:32:322
5115997517,11cyclictest0-21swapper/219:32:322
511599746,12cyclictest0-21swapper/219:43:112
511599746,12cyclictest0-21swapper/219:43:112
511599744,16cyclictest26642-21ssh21:32:312
511599743,19cyclictest4440-21timerandwakeup22:43:112
511599743,19cyclictest4440-21timerandwakeup22:43:112
511599743,19cyclictest20317-21latency_hist19:52:302
511599743,18cyclictest25772-21unixbench_singl23:18:142
511599743,17cyclictest21232-21munin-run23:12:282
511599743,17cyclictest19681-21grep00:03:082
511599743,17cyclictest14936-21wc22:07:312
511599743,16cyclictest25658-21unixbench_multi00:13:082
511599743,16cyclictest25658-21unixbench_multi00:13:082
5115997420,46cyclictest0-21swapper/222:52:312
5115997413,10cyclictest362-21rs:main2
5105997451,9cyclictest0-21swapper/100:12:591
5105997451,9cyclictest0-21swapper/100:12:591
511599734,17cyclictest5208-21latency_hist19:07:342
511599733,19cyclictest566-21fschecks_time21:42:562
511599733,19cyclictest2137-21sed23:33:102
511599733,18cyclictest13573-21cat22:58:082
511599733,17cyclictest15134-21grep21:12:312
511599733,17cyclictest15134-21grep21:12:312
511599733,16cyclictest10205-21grep19:18:322
511599732,20cyclictest22448-21fschecks_time19:57:562
511599732,18cyclictest21775-21/usr/sbin/munin23:13:342
511599732,15cyclictest17190-21taskset23:04:272
511599732,15cyclictest17190-21taskset23:04:272
5115997314,11cyclictest0-21swapper/220:47:292
5124997210,25cyclictest17951-21systemd-run22:12:293
511599728,13cyclictest0-21swapper/221:58:012
511599723,19cyclictest28619-21grep00:18:132
511599723,19cyclictest14067-21ps23:53:392
511599723,18cyclictest16632-21sed19:38:102
511599723,17cyclictest32282-21kernelversion20:28:012
511599723,17cyclictest15019-21unixbench_singl19:33:132
511599723,15cyclictest6205-21/usr/sbin/munin21:52:512
511599723,15cyclictest12178-21apt19:27:292
5115997212,11cyclictest0-21swapper/221:07:282
5093997227,19cyclictest14737-21latency_hist22:07:310
40022728,38sleep14089-21/usr/sbin/munin19:03:091
511599719,14cyclictest0-21swapper/220:52:572
511599714,13cyclictest0-21swapper/220:03:092
511599713,18cyclictest28911-21cut20:17:572
511599713,18cyclictest28911-21cut20:17:572
511599713,17cyclictest5362-21sort19:07:512
511599713,15cyclictest3726-21sort00:32:512
511599713,15cyclictest30788-21grep20:23:042
5115997110,13cyclictest0-21swapper/223:38:132
5105997156,9cyclictest30054-21fschecks_count21:37:561
5105997156,9cyclictest30054-21fschecks_count21:37:561
5105997127,8cyclictest141rcu_preempt23:33:501
5105997126,24cyclictest25-21ksoftirqd/123:57:281
5105997110,36cyclictest0-21swapper/121:22:581
5124997055,9cyclictest3774-21cron23:37:283
511599703,19cyclictest1311-21fschecks_time20:32:582
5124996951,11cyclictest23849-21idleruntime-cro22:22:293
5105996928,22cyclictest361-21in:imklog20:13:131
5093996949,13cyclictest0-21swapper/023:07:290
5093996949,13cyclictest0-21swapper/023:07:290
5105996849,12cyclictest4599-21unixbench_singl22:43:121
5105996849,12cyclictest4599-21unixbench_singl22:43:121
5105996847,12cyclictest14245-21runrttasks19:32:581
5105996821,27cyclictest25-21ksoftirqd/122:01:021
5093996825,15cyclictest21657-21cat23:12:320
5105996744,14cyclictest21549-21apt-key23:12:321
5105996727,24cyclictest25-21ksoftirqd/123:05:011
5105996727,24cyclictest25-21ksoftirqd/123:05:011
5105996725,16cyclictest361-21in:imklog20:29:091
5093996738,19cyclictest2433-21sendmail20:37:290
5124996644,13cyclictest15140-21apt-config22:07:323
5124996626,11cyclictest28698-21cpu20:17:513
5124996626,11cyclictest28698-21cpu20:17:513
5124996616,22cyclictest18236-21sh22:12:423
5105996650,10cyclictest6152-21munin-plugin-st00:37:291
5105996641,16cyclictest0-21swapper/122:37:321
5105996639,17cyclictest21002-21latency_hist21:22:331
5105996622,27cyclictest0-21swapper/121:52:421
5093996611,19cyclictest21994-21find22:18:130
5124996546,13cyclictest12855-21apt-key23:52:313
5124996543,13cyclictest12178-21diskmemload21:55:433
5105996548,10cyclictest276-21systemd-journal22:53:291
5105996543,15cyclictest361-21in:imklog21:30:371
5105996540,8cyclictest141rcu_preempt00:03:081
5105996540,16cyclictest361-21in:imklog20:02:521
5105996524,25cyclictest0-21swapper/100:00:441
5105996524,25cyclictest0-21swapper/100:00:441
5105996522,26cyclictest25-21ksoftirqd/123:28:531
5105996522,26cyclictest25-21ksoftirqd/123:28:531
509399659,34cyclictest21195-21ssh21:22:440
5093996546,12cyclictest0-21swapper/023:57:570
5093996546,12cyclictest0-21swapper/023:57:570
5093996543,13cyclictest29438-21cron21:37:290
5093996543,11cyclictest0-21swapper/023:37:310
5124996440,15cyclictest15424-21sendmail19:37:293
5124996430,25cyclictest12464-21apt-get19:27:553
5124996430,25cyclictest12464-21apt-get19:27:553
5124996415,17cyclictest1-21systemd20:28:083
5105996425,23cyclictest0-21swapper/122:14:131
5093996449,9cyclictest31321-21ssh21:39:250
5093996449,9cyclictest31321-21ssh21:39:250
5093996442,13cyclictest0-21swapper/021:52:570
5124996312,25cyclictest11442-21sendmail_mailqu21:03:053
510599634,39cyclictest9252-21cpu19:18:021
5105996327,19cyclictest25-21ksoftirqd/121:34:571
5105996325,11cyclictest141rcu_preempt22:42:301
5093996327,26cyclictest0-21swapper/023:47:520
5093996314,20cyclictest23044-21ls19:58:090
254962630,6sleep30-21swapper/300:13:073
254962630,6sleep30-21swapper/300:13:073
5124996244,13cyclictest21782-21modprobe23:12:583
5124996242,14cyclictest18131-21ntpd20:23:043
5124996218,16cyclictest3567-21sh00:32:343
5105996247,7cyclictest25-21ksoftirqd/122:07:021
5105996245,11cyclictest0-21swapper/121:47:311
5105996239,12cyclictest0-21swapper/119:48:001
5105996237,13cyclictest0-21swapper/122:17:541
5105996228,19cyclictest120-21pr/ttymxc000:18:071
5093996243,13cyclictest12-21ksoftirqd/022:52:160
5093996239,14cyclictest276-21systemd-journal22:58:290
5093996238,16cyclictest19650-21switchtime19:48:080
5093996226,27cyclictest21225-21apt-get23:12:350
5093996213,17cyclictest397-21runrttasks19:22:480
5124996145,9cyclictest9483-21latency_hist22:52:323
510599615,34cyclictest8924-21munin-run20:57:291
5105996142,11cyclictest183-21pr/ttyS022:24:291
5105996129,11cyclictest52-21kcompactd000:29:391
509399618,23cyclictest3894-21idleruntime22:42:590
509399618,23cyclictest3894-21idleruntime22:42:590
5093996145,9cyclictest31057-21ssh23:28:020
5093996145,9cyclictest31057-21ssh23:28:020
5093996138,14cyclictest31995-21ssh00:24:440
5105996024,18cyclictest25-21ksoftirqd/121:07:291
510599602,34cyclictest5878-21munin-run21:52:291
510599602,34cyclictest5878-21munin-run21:52:291
5105996023,13cyclictest141rcu_preempt22:58:011
5105996019,15cyclictest141rcu_preempt22:27:551
5105996015,28cyclictest25-21ksoftirqd/121:09:031
5105996015,28cyclictest25-21ksoftirqd/121:09:031
509399608,19cyclictest17278-21sed19:42:320
509399607,21cyclictest397-21runrttasks00:10:450
5093996046,10cyclictest131ktimers/023:37:550
5093996039,12cyclictest25420-21ssh22:23:360
512499599,19cyclictest5323-21who20:43:133
5124995943,9cyclictest21963-21runrttasks00:07:503
5124995941,12cyclictest27984-21unixbench_multi21:33:163
5124995936,12cyclictest0-21swapper/320:42:293
5124995935,13cyclictest0-21swapper/322:37:293
5124995932,15cyclictest0-21swapper/321:22:323
5124995922,15cyclictest0-21swapper/323:28:333
5124995922,15cyclictest0-21swapper/323:28:333
5124995914,26cyclictest0-21swapper/300:31:203
5124995910,21cyclictest10845-21apt19:22:483
5124995910,18cyclictest10394-21seq21:00:343
5124995910,18cyclictest10394-21seq21:00:343
5105995933,10cyclictest25-21ksoftirqd/100:08:101
510599592,30cyclictest6793-21ssh22:48:001
5105995919,22cyclictest183-21pr/ttyS020:43:081
5093995937,13cyclictest120-21pr/ttymxc019:12:350
5093995933,18cyclictest5958-21apt-get20:47:510
5093995913,16cyclictest3366-21ssh00:32:300
5124995842,10cyclictest27558-21apt-get00:17:513
5124995839,8cyclictest0-21swapper/321:42:293
5124995839,8cyclictest0-21swapper/321:42:293
5124995837,10cyclictest0-21swapper/322:47:313
5124995837,10cyclictest0-21swapper/322:47:313
5124995836,12cyclictest0-21swapper/320:02:563
5124995835,13cyclictest14727-21kworker/u8:322:27:313
5124995831,12cyclictest24088-21apt-get21:27:483
5124995817,17cyclictest24641-21sh23:17:393
5124995812,19cyclictest21306-21sh00:06:563
5105995824,9cyclictest0-21swapper/119:57:311
5105995812,19cyclictest141rcu_preempt23:52:311
5105995810,13cyclictest141rcu_preempt23:43:041
509399589,18cyclictest10044-21sed20:58:080
509399589,18cyclictest10044-21sed20:58:080
509399588,18cyclictest14464-21sed19:33:020
509399586,32cyclictest0-21swapper/022:08:030
5093995838,11cyclictest0-21swapper/022:02:290
5093995837,13cyclictest19429-21ssh21:18:430
5093995837,10cyclictest0-21swapper/022:38:020
5093995836,14cyclictest0-21swapper/023:43:090
509399581,29cyclictest12444-21/usr/sbin/munin19:28:060
509399581,29cyclictest12444-21/usr/sbin/munin19:28:060
5124995741,8cyclictest0-21swapper/322:28:093
5124995739,11cyclictest15789-21apt23:57:283
512499572,23cyclictest9987-21ssh22:53:123
510599573,29cyclictest1556-21netstat20:33:031
5105995723,12cyclictest25-21ksoftirqd/119:27:571
5105995723,12cyclictest25-21ksoftirqd/119:27:571
5093995742,9cyclictest5670-21ssh00:36:060
5093995740,12cyclictest12-21ksoftirqd/023:27:290
5093995736,13cyclictest27784-21fschecks_count00:17:580
5093995735,12cyclictest0-21swapper/020:38:110
5093995730,20cyclictest24707-21apt-get23:17:480
5093995719,27cyclictest21493-21modprobe00:07:280
5124995641,9cyclictest6817-21systemd23:42:303
5124995639,8cyclictest0-21swapper/322:57:553
5124995632,13cyclictest0-21swapper/322:42:313
5124995631,13cyclictest0-21swapper/320:02:323
5124995626,21cyclictest30397-21apt-get00:22:463
5124995611,19cyclictest5692-21ssh21:52:013
5124995611,19cyclictest5692-21ssh21:52:013
5124995611,17cyclictest12178-21diskmemload22:01:383
5105995641,9cyclictest183-21pr/ttyS000:24:371
510599564,13cyclictest30367-21ls20:22:521
5105995631,14cyclictest0-21swapper/120:47:531
5105995623,19cyclictest25-21ksoftirqd/123:20:291
5105995619,18cyclictest141rcu_preempt23:17:311
5105995610,27cyclictest0-21swapper/120:42:301
5093995641,9cyclictest13315-21/usr/sbin/munin21:08:100
5093995641,9cyclictest13315-21/usr/sbin/munin21:08:100
5093995638,11cyclictest3801-21rm21:48:050
5093995638,11cyclictest3801-21rm21:48:050
5093995632,17cyclictest26605-21idleruntime-cro21:32:290
200642560,4sleep020062-21ssh22:15:100
512499554,16cyclictest0-21swapper/323:23:103
5124995534,12cyclictest1-21systemd20:08:433
5124995534,12cyclictest1-21systemd20:08:433
5124995527,17cyclictest18611-21apt-get23:07:473
510599554,29cyclictest0-21swapper/121:13:331
5105995533,8cyclictest141rcu_preempt19:24:351
5105995532,13cyclictest0-21swapper/120:58:091
5105995532,13cyclictest0-21swapper/120:58:091
5105995520,19cyclictest120-21pr/ttymxc020:08:081
5105995520,19cyclictest120-21pr/ttymxc020:08:081
5105995510,27cyclictest0-21swapper/123:23:481
509399559,16cyclictest26174-21sed20:08:100
509399559,16cyclictest26174-21sed20:08:100
509399558,10cyclictest0-21swapper/019:53:090
5093995536,13cyclictest19859-21kworker/u8:022:31:540
5093995531,13cyclictest0-21swapper/019:43:100
5093995531,13cyclictest0-21swapper/019:43:100
5093995511,15cyclictest29393-21sed20:18:090
5093995511,15cyclictest29393-21sed20:18:090
5124995432,12cyclictest0-21swapper/321:07:583
5124995432,12cyclictest0-21swapper/321:07:583
5124995418,14cyclictest0-21swapper/323:58:563
5124995418,14cyclictest0-21swapper/323:58:563
5124995410,16cyclictest2949-21ssh21:47:303
5105995423,8cyclictest25-21ksoftirqd/119:58:021
5105995416,16cyclictest25-21ksoftirqd/119:42:521
5105995416,16cyclictest25-21ksoftirqd/119:42:521
509399544,21cyclictest31215-21sh22:33:220
5093995438,10cyclictest27284-21runrttasks20:13:000
5093995432,13cyclictest183-21pr/ttyS022:53:180
509399542,20cyclictest10633-21sed19:22:310
5124995328,13cyclictest0-21swapper/319:42:573
5124995328,13cyclictest0-21swapper/319:42:573
5105995336,11cyclictest15722-21fschecks_time22:08:191
5105995332,13cyclictest0-21swapper/119:17:301
5105995323,18cyclictest0-21swapper/120:18:111
5105995323,18cyclictest0-21swapper/120:18:111
5105995315,21cyclictest25-21ksoftirqd/119:37:481
509399538,21cyclictest12-21ksoftirqd/021:02:520
5093995335,11cyclictest131ktimers/020:23:110
509399532,40cyclictest5313-21sed20:43:120
5093995311,8cyclictest2940-21ssh21:47:290
5093995311,13cyclictest15353-21awk21:12:470
5124995237,10cyclictest0-21swapper/319:17:303
512499523,26cyclictest141rcu_preempt20:52:323
5124995231,14cyclictest21031-21ntp_kernel_err19:53:053
5124995227,16cyclictest15868-21/usr/sbin/munin21:13:023
5105995228,15cyclictest25-21ksoftirqd/119:10:221
509399529,18cyclictest12-21ksoftirqd/019:09:230
512499513,16cyclictest2053-21head20:33:123
5124995130,13cyclictest0-21swapper/319:48:093
5124995129,14cyclictest10574-21munin-plugin-st19:22:293
5124995128,11cyclictest0-21swapper/320:57:313
5124995024,18cyclictest5638-21interrupts19:07:593
509399505,17cyclictest9109-21cat20:57:310
5093995038,8cyclictest52-21kcompactd020:05:000
5124994926,12cyclictest0-21swapper/323:07:303
5124994926,12cyclictest0-21swapper/323:07:303
5124994911,23cyclictest141rcu_preempt20:13:043
5093994830,11cyclictest0-21swapper/020:28:110
512499465,14cyclictest141rcu_preempt19:38:073
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional