You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-12-12 - 04:45
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Thu Dec 12, 2024 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31991211511,65sleep132003-21cstates19:03:341
924999921,13cyclictest12218-21ssh23:03:490
92499964,29cyclictest25928-21ssh00:23:170
924999535,50cyclictest0-21swapper/021:38:510
924999525,9cyclictest0-21swapper/023:48:410
92499934,10cyclictest0-21swapper/019:18:180
92499923,15cyclictest20601-21fschecks_time00:13:360
92499923,15cyclictest20601-21fschecks_time00:13:360
924999224,11cyclictest0-21swapper/020:03:350
924999221,15cyclictest19821-21latency_hist22:23:180
924999221,11cyclictest0-21swapper/022:58:410
924999221,11cyclictest0-21swapper/022:58:410
924999123,13cyclictest10804-21ssh21:13:080
92499904,26cyclictest1098-21/usr/sbin/munin19:08:460
92499903,31cyclictest12700-21grep23:58:520
92499903,30cyclictest8105-21modprobe22:58:140
924999029,10cyclictest31290-21modprobe21:48:150
92499902,17cyclictest5180-21latency_hist22:53:150
94999893,56cyclictest20051-21apt-get22:23:292
94999893,56cyclictest20051-21apt-get22:23:292
92499893,31cyclictest5860-21grep21:58:460
924998929,9cyclictest0-21swapper/023:58:160
924998929,9cyclictest0-21swapper/023:58:160
924998910,17cyclictest0-21swapper/022:23:290
924998910,17cyclictest0-21swapper/022:23:290
92499883,28cyclictest15019-21unixbench_multi21:18:500
949998721,18cyclictest0-21swapper/223:48:352
92499873,29cyclictest23998-21sendmail-msp20:18:150
92499873,13cyclictest25685-21taskset23:27:050
6252875,12sleep20-21swapper/219:06:202
92499864,24cyclictest914-21cyclictest19:08:180
924998619,11cyclictest8150irq/69-2188000.23:43:490
924998613,18cyclictest3649-21fschecks_time20:53:380
949998534,42cyclictest0-21swapper/221:58:342
924998519,15cyclictest0-21swapper/000:33:170
92499846,28cyclictest0-21swapper/000:08:360
92499846,28cyclictest0-21swapper/000:08:360
949998316,13cyclictest21233-21timerwakeupswit00:13:492
949998316,13cyclictest21233-21timerwakeupswit00:13:492
949998123,11cyclictest0-21swapper/221:33:402
949998115,9cyclictest141rcu_preempt23:58:332
940998143,22cyclictest0-21swapper/123:23:291
924998121,10cyclictest0-21swapper/021:33:400
3207928120,28sleep30-21swapper/319:03:413
94999805,30cyclictest3551-21grep20:53:342
949998023,8cyclictest0-21swapper/222:23:182
949997925,48cyclictest0-21swapper/220:23:502
949997925,48cyclictest0-21swapper/220:23:502
949997922,14cyclictest116-21pr/ttymxc020:39:032
94999783,13cyclictest14605-21/usr/sbin/munin00:03:352
949997823,14cyclictest26747-21unixbench_multi21:38:512
949997738,13cyclictest33-21ksoftirqd/219:48:512
949997621,10cyclictest0-21swapper/220:13:172
949997618,13cyclictest9309-21unixbench_multi21:08:512
953997533,13cyclictest141rcu_preempt19:58:373
953997528,10cyclictest0-21swapper/322:58:393
953997528,10cyclictest0-21swapper/322:58:393
940997558,10cyclictest4740-21sh20:58:141
94999743,24cyclictest28834-21latency_hist22:38:182
949997424,8cyclictest0-21swapper/222:43:152
949997424,10cyclictest0-21swapper/219:18:392
949997414,16cyclictest14185-21latency_hist23:08:172
953997354,11cyclictest29101-21sshd22:38:343
949997352,14cyclictest275-21systemd-journal23:22:162
949997352,14cyclictest275-21systemd-journal23:22:162
949997339,27cyclictest33-21ksoftirqd/220:33:402
949997339,27cyclictest33-21ksoftirqd/220:33:402
94999733,22cyclictest17728-21fschecks_count00:08:362
94999733,22cyclictest17728-21fschecks_count00:08:362
949997319,12cyclictest0-21swapper/219:28:332
232562730,21sleep1231rcuc/121:33:411
953997252,12cyclictest16548-21idleruntime-cro22:18:143
953997245,17cyclictest27033-21grep00:23:483
94999723,20cyclictest394-21runrttasks19:38:352
94999723,19cyclictest466-21if_eth020:43:432
949997222,7cyclictest0-21swapper/221:28:492
949997217,11cyclictest0-21swapper/222:56:462
953997153,12cyclictest32168-21fschecks_time22:43:363
953997137,12cyclictest141rcu_preempt00:13:163
953997137,12cyclictest141rcu_preempt00:13:163
949997122,8cyclictest0-21swapper/221:43:372
940997146,17cyclictest29091-21fschecks_time00:28:321
953997027,25cyclictest41-21ksoftirqd/322:30:523
949997019,12cyclictest0-21swapper/222:58:352
949997019,12cyclictest0-21swapper/222:58:352
949997018,11cyclictest0-21swapper/223:58:152
949997018,11cyclictest0-21swapper/223:58:152
940997046,13cyclictest0-21swapper/120:43:571
940997045,15cyclictest4700-21latency_hist19:18:181
924997050,12cyclictest12005-21unixbench_singl22:08:490
924997050,12cyclictest12005-21unixbench_singl22:08:490
953996951,11cyclictest11878-21runrttasks19:38:493
953996927,22cyclictest0-21swapper/300:04:593
94999694,21cyclictest19118-21munin-plugin-st20:03:142
94999694,10cyclictest0-21swapper/219:43:422
94099697,40cyclictest2045-21apt-get22:48:281
92499698,21cyclictest24640-21/usr/sbin/munin20:18:340
953996850,11cyclictest7994-21idleruntime-cro21:08:143
953996841,16cyclictest0-21swapper/319:28:503
953996815,12cyclictest0-21swapper/323:46:133
940996842,16cyclictest11365-40logrotate23:58:161
940996842,16cyclictest11365-40logrotate23:58:161
94999678,22cyclictest0-21swapper/220:18:592
94999674,10cyclictest0-21swapper/223:14:252
94999674,10cyclictest0-21swapper/223:14:252
940996745,13cyclictest0-21swapper/123:38:141
940996745,13cyclictest0-21swapper/123:38:141
924996731,12cyclictest16971-21latency_hist23:13:140
953996649,10cyclictest31307-21latency_hist21:48:153
953996624,26cyclictest141rcu_preempt00:16:283
953996624,26cyclictest141rcu_preempt00:16:283
949996628,33cyclictest2621-21kworker/u8:022:48:452
940996645,9cyclictest0-21swapper/121:03:501
940996641,14cyclictest0-21swapper/123:13:491
940996641,14cyclictest0-21swapper/123:13:491
924996635,23cyclictest0-21swapper/021:53:170
95399657,39cyclictest41-21ksoftirqd/322:24:073
95399657,39cyclictest41-21ksoftirqd/322:24:073
953996532,13cyclictest141rcu_preempt21:48:483
953996515,28cyclictest141rcu_preempt20:38:153
953996515,28cyclictest141rcu_preempt20:38:153
940996548,9cyclictest0-21swapper/121:58:481
940996542,15cyclictest28969-21runrttasks21:43:361
953996445,10cyclictest0-21swapper/323:23:173
953996445,10cyclictest0-21swapper/323:23:173
953996437,20cyclictest14250-21grep19:48:143
953996434,7cyclictest141rcu_preempt21:18:063
953996434,7cyclictest141rcu_preempt21:18:063
924996416,20cyclictest141rcu_preempt23:18:140
924996416,20cyclictest141rcu_preempt23:18:140
95399637,36cyclictest2571-21apt-get22:48:483
95399636,33cyclictest41-21ksoftirqd/319:33:343
953996347,10cyclictest17004-21systemd-run23:13:153
949996342,12cyclictest0-21swapper/222:13:512
940996339,17cyclictest29075-21munin-plugin-st20:33:151
924996344,13cyclictest30447-21vmstat23:33:520
924996344,13cyclictest30447-21vmstat23:33:520
953996246,10cyclictest275-21systemd-journal19:18:153
953996236,16cyclictest0-21swapper/322:58:143
95399623,37cyclictest19548-21systemd-run22:23:153
949996245,11cyclictest25624-21sh22:33:142
949996237,15cyclictest23485-21runrttasks23:23:292
940996239,14cyclictest275-21systemd-journal20:28:161
940996239,14cyclictest275-21systemd-journal20:28:161
288672620,7chrt0-21swapper/300:28:273
953996142,13cyclictest19995-21ssh23:18:163
953996142,13cyclictest19995-21ssh23:18:163
953996131,9cyclictest0-21swapper/300:01:323
953996126,18cyclictest0-21swapper/319:48:343
953996112,12cyclictest141rcu_preempt23:48:353
949996141,13cyclictest179-21pr/ttyS020:13:412
949996138,12cyclictest0-21swapper/223:38:162
949996138,12cyclictest0-21swapper/223:38:162
940996140,12cyclictest0-21swapper/100:18:161
940996140,12cyclictest0-21swapper/100:18:161
940996136,16cyclictest2169-21apt-key22:48:171
924996144,10cyclictest116-21pr/ttymxc000:36:450
924996144,10cyclictest116-21pr/ttymxc000:36:450
92499614,39cyclictest141rcu_preempt20:08:490
924996142,9cyclictest0-21swapper/022:38:380
924996142,12cyclictest16772-21latency_hist22:18:170
924996134,18cyclictest0-21swapper/022:03:370
121472612,7sleep30-21swapper/323:03:493
953996038,15cyclictest16522-21modprobe21:23:143
953996029,14cyclictest41-21ksoftirqd/320:43:333
949996013,8cyclictest141rcu_preempt00:23:382
940996042,10cyclictest0-21swapper/122:08:471
940996042,10cyclictest0-21swapper/122:08:471
953995939,13cyclictest0-21swapper/320:18:143
953995935,7cyclictest0-21swapper/320:48:463
953995935,7cyclictest0-21swapper/320:48:463
953995935,14cyclictest4761-21sendmail21:58:153
953995917,17cyclictest141rcu_preempt22:33:503
953995913,28cyclictest0-21swapper/321:12:213
949995938,15cyclictest29781-21unixbench_multi00:28:452
949995938,12cyclictest0-21swapper/219:13:502
940995938,13cyclictest627-21in:imklog21:09:081
940995937,12cyclictest0-21swapper/123:53:141
940995936,13cyclictest24015-21unixbench_multi22:28:491
940995927,24cyclictest29315-21/usr/sbin/munin20:33:411
940995927,24cyclictest29315-21/usr/sbin/munin20:33:411
940995916,8cyclictest241ktimers/123:13:131
953995840,10cyclictest0-21swapper/323:23:563
95399583,48cyclictest11371-21idleruntime-cro23:58:153
95399583,48cyclictest11371-21idleruntime-cro23:58:153
95399583,27cyclictest0-21swapper/320:58:173
953995831,14cyclictest0-21swapper/320:38:423
953995830,12cyclictest275-21systemd-journal19:18:433
953995830,10cyclictest0-21swapper/320:28:463
95399582,30cyclictest24600-21cstates20:18:343
953995810,28cyclictest0-21swapper/320:11:233
949995837,12cyclictest0-21swapper/200:23:142
949995834,15cyclictest16676-21latency_hist21:23:172
940995836,15cyclictest914-21cyclictest23:43:141
940995835,11cyclictest0-21swapper/123:18:381
940995835,11cyclictest0-21swapper/123:18:381
94099583,29cyclictest15034-21ssh22:13:581
940995830,19cyclictest3713-21timerandwakeup23:43:491
940995820,20cyclictest0-21swapper/121:19:331
924995839,9cyclictest0-21swapper/022:48:140
924995839,12cyclictest2821-21sendmail_mailqu21:53:470
924995829,10cyclictest116-21pr/ttymxc021:13:520
924995829,10cyclictest116-21pr/ttymxc021:13:520
924995820,13cyclictest28588-21latency_hist22:38:180
95399574,26cyclictest141rcu_preempt21:33:143
953995736,14cyclictest627-21in:imklog22:06:173
95399573,33cyclictest32107-21ssh00:33:403
95399573,33cyclictest32107-21ssh00:33:403
953995730,9cyclictest41-21ksoftirqd/323:38:413
953995718,8cyclictest141rcu_preempt23:28:413
953995714,23cyclictest141rcu_preempt19:58:153
953995714,23cyclictest141rcu_preempt19:58:153
953995712,17cyclictest41-21ksoftirqd/320:03:253
949995734,12cyclictest0-21swapper/223:28:342
94099577,18cyclictest13436-21ssh00:00:441
940995740,8cyclictest0-21swapper/121:58:151
940995738,12cyclictest1631-21kworker/u8:100:38:141
940995738,12cyclictest1631-21kworker/u8:100:38:141
940995734,14cyclictest13378-21ntp_kernel_err19:43:421
940995716,16cyclictest18494-21sh21:25:301
953995638,11cyclictest627-21in:imklog22:00:013
953995630,17cyclictest27143-21kworker/u8:220:28:143
953995630,17cyclictest27143-21kworker/u8:220:28:143
953995616,13cyclictest141rcu_preempt21:23:483
953995614,14cyclictest141rcu_preempt21:38:373
949995640,9cyclictest2054-21idleruntime-cro22:48:142
94999562,30cyclictest13690-21munin-plugin-st21:18:152
94999562,30cyclictest13690-21munin-plugin-st21:18:152
94099566,21cyclictest11816-21/usr/sbin/munin23:03:401
940995632,15cyclictest0-21swapper/121:13:501
940995632,15cyclictest0-21swapper/121:13:501
924995632,14cyclictest12-21ksoftirqd/000:03:280
924995623,12cyclictest141rcu_preempt23:23:160
924995623,12cyclictest141rcu_preempt23:23:160
924995612,11cyclictest0-21swapper/020:48:180
95399556,33cyclictest41-21ksoftirqd/322:12:063
95399556,33cyclictest41-21ksoftirqd/322:12:063
953995537,11cyclictest0-21swapper/321:33:493
953995526,8cyclictest141rcu_preempt19:28:183
949995533,11cyclictest0-21swapper/219:33:342
94099558,13cyclictest25-21ksoftirqd/121:49:581
94099556,19cyclictest26086-21taskset22:33:321
94099554,42cyclictest22796-21apt-get00:18:281
940995539,11cyclictest18677-21unixbench_multi19:58:501
940995533,12cyclictest0-21swapper/119:08:361
94099553,29cyclictest20124-21ssh22:23:361
94099553,29cyclictest20124-21ssh22:23:361
94099553,15cyclictest28335-21cron21:43:141
924995530,19cyclictest29019-21ssh23:33:140
95399549,27cyclictest41-21ksoftirqd/320:58:513
95399549,10cyclictest141rcu_preempt19:08:493
953995436,10cyclictest0-21swapper/300:23:143
953995410,25cyclictest141rcu_preempt23:33:323
953995410,25cyclictest141rcu_preempt23:33:323
949995433,11cyclictest0-21swapper/221:23:322
949995432,11cyclictest0-21swapper/219:23:362
940995436,10cyclictest0-21swapper/120:58:501
94099542,23cyclictest394-21runrttasks20:13:181
924995431,13cyclictest0-21swapper/019:38:400
924995415,16cyclictest141rcu_preempt21:28:350
949995334,12cyclictest32187-21ntp_kernel_pll_21:48:412
940995328,13cyclictest0-21swapper/121:33:161
94099531,16cyclictest7997-21diskmemload22:22:281
92499539,27cyclictest141rcu_preempt19:43:490
92499537,38cyclictest275-21systemd-journal00:26:040
949995239,7cyclictest0-21swapper/221:58:152
949995234,11cyclictest0-21swapper/220:28:532
949995232,13cyclictest627-21in:imklog20:58:502
949995231,11cyclictest0-21swapper/220:48:482
949995231,11cyclictest0-21swapper/220:48:482
949995231,10cyclictest0-21swapper/222:08:162
949995231,10cyclictest0-21swapper/219:58:182
949995231,10cyclictest0-21swapper/219:58:182
94999522,20cyclictest7540-21grep21:03:502
949995218,9cyclictest141rcu_preempt23:38:472
949995213,12cyclictest141rcu_preempt22:10:562
949995213,12cyclictest141rcu_preempt22:10:562
94099529,14cyclictest4903-21cstates19:18:301
940995229,13cyclictest19382-21apt-get20:03:261
940995229,12cyclictest0-21swapper/119:43:171
940995226,13cyclictest0-21swapper/122:43:171
94099520,17cyclictest7997-21diskmemload23:31:071
92499524,39cyclictest2720-21latency_hist23:43:170
924995230,15cyclictest131ktimers/021:03:360
949995127,15cyclictest627-21in:imklog00:35:202
949995127,15cyclictest627-21in:imklog00:35:202
949995125,14cyclictest0-21swapper/223:08:372
940995130,11cyclictest0-21swapper/120:18:461
940995129,12cyclictest0-21swapper/100:13:141
940995129,12cyclictest0-21swapper/100:13:141
940995127,10cyclictest26050-21apt-get00:23:251
92499514,38cyclictest394-21runrttasks19:33:350
92499513,17cyclictest18011-21fschecks_time19:58:370
924995129,11cyclictest0-21swapper/022:28:310
924995127,13cyclictest0-21swapper/019:53:160
92499512,39cyclictest26114-21cpu20:23:300
92499512,39cyclictest26114-21cpu20:23:300
949995030,12cyclictest0-21swapper/223:43:312
949995028,11cyclictest0-21swapper/219:13:152
940995030,10cyclictest0-21swapper/119:33:471
940995027,16cyclictest14990-21nfs4_client19:48:431
940995025,12cyclictest0-21swapper/119:28:371
940995011,20cyclictest25-21ksoftirqd/120:48:551
940995011,20cyclictest25-21ksoftirqd/120:48:551
92499503,40cyclictest6407-21munin-run21:03:160
92499503,39cyclictest31103-21runrttasks20:38:330
924995027,11cyclictest0-21swapper/020:38:190
924995027,11cyclictest0-21swapper/020:38:190
924995027,10cyclictest179-21pr/ttyS021:28:180
924995022,8cyclictest12-21ksoftirqd/020:28:500
924995011,8cyclictest141rcu_preempt19:33:180
94999492,12cyclictest19889-21ntp_kernel_pll_20:03:412
94099499,12cyclictest7631-21taskset19:26:411
940994929,12cyclictest0-21swapper/122:58:361
940994929,12cyclictest0-21swapper/122:58:361
940994929,12cyclictest0-21swapper/119:53:501
940994929,12cyclictest0-21swapper/119:53:501
940994928,11cyclictest0-21swapper/122:53:401
92499494,15cyclictest6191-21latency_hist19:23:160
940994830,10cyclictest0-21swapper/100:08:171
940994823,13cyclictest0-21swapper/122:03:361
940994731,10cyclictest394-21runrttasks20:16:361
924994723,13cyclictest0-21swapper/020:48:430
924994723,13cyclictest0-21swapper/020:48:430
92499472,10cyclictest0-21swapper/019:58:150
92499472,10cyclictest0-21swapper/019:58:150
940994628,11cyclictest0-21swapper/120:38:371
92499467,10cyclictest0-21swapper/019:23:430
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional