You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-26 - 20:30
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7 (updated Tue Sep 26, 2023 12:44:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27677210316,56sleep127709-21munin-run07:07:161
276472885,13sleep30-21swapper/307:06:533
27840998263,13cyclictest11973-21sendmail07:57:150
2682928119,37sleep00-21swapper/007:03:000
27851997931,24cyclictest141rcu_preempt10:37:412
27851997931,24cyclictest141rcu_preempt10:37:412
27840997942,8cyclictest141rcu_preempt10:51:310
27840997942,8cyclictest141rcu_preempt10:51:310
2784099793,30cyclictest0-21swapper/009:32:570
27840997859,13cyclictest9295-21unixbench_multi09:17:560
27840997849,9cyclictest141rcu_preempt10:57:170
27851997743,9cyclictest141rcu_preempt10:27:512
27851997729,28cyclictest30227-21kworker/u8:108:07:472
27851997726,17cyclictest141rcu_preempt09:57:442
27840997729,27cyclictest141rcu_preempt09:02:340
27840997729,27cyclictest141rcu_preempt09:02:340
27847997661,9cyclictest2017-21unixbench_multi11:58:001
27840997632,25cyclictest141rcu_preempt08:27:530
2712927612,39sleep227202-21grep07:03:102
27851997555,11cyclictest8718-21idleruntime-cro07:47:162
27851997555,11cyclictest8718-21idleruntime-cro07:47:162
27851997541,7cyclictest141rcu_preempt10:03:022
27856997445,14cyclictest141rcu_preempt07:17:183
27851997441,7cyclictest141rcu_preempt10:35:202
27840997455,13cyclictest1114-21fschecks_time07:22:460
2784099743,47cyclictest2488-21diskmemload09:22:430
27840997434,11cyclictest141rcu_preempt12:16:470
27840997430,25cyclictest141rcu_preempt11:10:320
27840997429,30cyclictest4832-21rm10:09:020
27856997338,18cyclictest141rcu_preempt10:13:193
27856997335,7cyclictest141rcu_preempt08:07:583
27851997326,18cyclictest0-21swapper/211:42:022
27840997314,32cyclictest0-21swapper/011:56:410
27840997314,32cyclictest0-21swapper/011:56:410
27851997245,8cyclictest141rcu_preempt10:18:092
27851997244,8cyclictest141rcu_preempt07:27:152
27851997227,6cyclictest0-21swapper/209:08:362
27840997216,20cyclictest12-21ksoftirqd/007:32:180
27851997132,8cyclictest141rcu_preempt12:12:512
27851997128,17cyclictest0-21swapper/210:57:442
27847997154,9cyclictest1-21systemd07:12:461
27847997139,8cyclictest141rcu_preempt12:20:131
27847997139,8cyclictest141rcu_preempt12:20:131
27840997128,15cyclictest141rcu_preempt08:06:100
27840997122,17cyclictest2488-21diskmemload09:42:410
27840997115,21cyclictest12-21ksoftirqd/011:37:170
27856997039,14cyclictest52-21kcompactd011:26:273
27856997019,14cyclictest141rcu_preempt11:30:283
27851997049,15cyclictest22025-21idleruntime-cro09:42:152
27851997040,7cyclictest141rcu_preempt12:08:142
27851997037,15cyclictest141rcu_preempt07:17:482
27851997037,15cyclictest141rcu_preempt07:17:482
27851997034,9cyclictest0-21swapper/210:09:482
27847997039,7cyclictest141rcu_preempt08:05:521
27840997042,8cyclictest141rcu_preempt07:57:560
27840997016,42cyclictest131ktimers/009:37:350
27856996938,13cyclictest141rcu_preempt12:17:383
27856996938,13cyclictest141rcu_preempt12:17:383
27856996936,14cyclictest141rcu_preempt09:52:413
27851996941,8cyclictest141rcu_preempt07:32:192
27851996932,18cyclictest141rcu_preempt07:07:472
27851996932,18cyclictest141rcu_preempt07:07:472
27851996914,20cyclictest321ktimers/209:33:152
27851996911,26cyclictest33-21ksoftirqd/212:25:312
27851996911,26cyclictest33-21ksoftirqd/212:25:312
27847996943,7cyclictest141rcu_preempt12:31:301
27847996931,16cyclictest0-21swapper/112:06:001
2784099698,22cyclictest30853-21apt-get10:57:510
27840996936,7cyclictest141rcu_preempt11:43:550
27840996935,15cyclictest141rcu_preempt08:47:380
27840996929,25cyclictest141rcu_preempt10:27:090
27840996927,16cyclictest141rcu_preempt11:49:420
27840996926,22cyclictest141rcu_preempt09:54:370
27840996924,21cyclictest361-21in:imklog08:32:510
27840996923,20cyclictest141rcu_preempt08:56:410
27840996923,20cyclictest141rcu_preempt08:56:410
27856996836,13cyclictest141rcu_preempt09:05:433
27856996836,13cyclictest141rcu_preempt09:05:433
27856996834,10cyclictest52-21kcompactd010:02:503
27851996841,7cyclictest141rcu_preempt08:57:452
27851996834,8cyclictest141rcu_preempt11:32:362
27851996828,16cyclictest141rcu_preempt09:25:152
27851996825,8cyclictest141rcu_preempt08:27:472
27847996841,8cyclictest141rcu_preempt10:55:521
27840996830,12cyclictest183-21pr/ttyS008:27:170
27840996826,24cyclictest141rcu_preempt12:26:470
27840996826,24cyclictest141rcu_preempt12:26:470
251612680,17sleep1241ktimers/108:37:161
27856996751,8cyclictest391rcuc/307:22:343
27856996741,8cyclictest141rcu_preempt10:53:173
27856996734,12cyclictest141rcu_preempt11:41:143
27856996733,9cyclictest141rcu_preempt11:57:593
27856996733,8cyclictest141rcu_preempt07:52:553
27856996732,17cyclictest141rcu_preempt11:12:023
27851996736,8cyclictest141rcu_preempt12:33:432
27851996732,8cyclictest141rcu_preempt09:47:172
27851996731,16cyclictest141rcu_preempt07:52:532
27851996730,7cyclictest141rcu_preempt08:17:462
27851996729,18cyclictest141rcu_preempt11:57:412
27851996729,13cyclictest141rcu_preempt08:15:432
27851996729,13cyclictest141rcu_preempt08:15:432
27851996728,10cyclictest141rcu_preempt09:27:432
27840996744,7cyclictest141rcu_preempt10:42:180
27840996744,7cyclictest141rcu_preempt10:42:180
27840996741,12cyclictest141rcu_preempt09:08:260
27840996733,13cyclictest141rcu_preempt11:18:560
27840996725,13cyclictest141rcu_preempt12:03:380
27840996719,39cyclictest13234-21apt-get12:17:350
27840996719,39cyclictest13234-21apt-get12:17:350
27856996642,17cyclictest27090-21munin-plugin-st11:47:153
27856996637,13cyclictest141rcu_preempt10:10:553
27856996632,15cyclictest141rcu_preempt12:32:283
27856996631,8cyclictest0-21swapper/309:10:393
27856996624,16cyclictest141rcu_preempt11:16:473
27856996624,16cyclictest141rcu_preempt11:16:473
27856996620,18cyclictest141rcu_preempt08:22:163
27856996618,18cyclictest41-21ksoftirqd/307:27:543
27851996634,8cyclictest141rcu_preempt11:46:172
27851996630,14cyclictest141rcu_preempt07:59:542
27851996622,16cyclictest0-21swapper/211:10:462
27851996611,25cyclictest321ktimers/208:47:502
27840996635,8cyclictest141rcu_preempt10:03:480
27840996634,12cyclictest141rcu_preempt11:28:230
27840996630,15cyclictest52-21kcompactd007:42:470
27840996630,15cyclictest52-21kcompactd007:42:470
27840996628,17cyclictest141rcu_preempt09:31:560
27840996620,22cyclictest183-21pr/ttyS012:34:390
27840996612,23cyclictest361-21in:imklog10:32:470
27856996535,7cyclictest141rcu_preempt12:24:433
27856996535,7cyclictest141rcu_preempt12:24:433
27856996525,15cyclictest141rcu_preempt08:42:423
27856996525,15cyclictest141rcu_preempt08:42:423
27851996539,7cyclictest141rcu_preempt10:12:402
27851996535,13cyclictest141rcu_preempt09:49:122
27851996535,13cyclictest141rcu_preempt09:49:122
27851996533,14cyclictest141rcu_preempt11:56:472
27851996533,14cyclictest141rcu_preempt11:56:472
27851996531,17cyclictest141rcu_preempt10:22:572
27847996546,12cyclictest10953-21wc09:22:181
27847996538,6cyclictest141rcu_preempt10:32:001
27840996540,7cyclictest141rcu_preempt12:28:000
27840996535,12cyclictest52-21kcompactd010:27:570
27840996532,13cyclictest141rcu_preempt10:13:420
27840996529,12cyclictest141rcu_preempt07:47:210
27840996528,14cyclictest141rcu_preempt11:37:430
27840996526,10cyclictest4759-21cut07:32:540
27840996523,14cyclictest141rcu_preempt11:05:240
27840996519,7cyclictest141rcu_preempt08:20:510
27840996515,14cyclictest12-21ksoftirqd/012:07:540
96302640,10sleep3391rcuc/307:47:553
27856996438,14cyclictest141rcu_preempt09:51:273
27856996438,14cyclictest141rcu_preempt09:51:273
27856996434,12cyclictest141rcu_preempt11:17:363
27856996415,26cyclictest401ktimers/311:02:503
27851996428,19cyclictest0-21swapper/210:44:332
27851996427,8cyclictest0-21swapper/212:29:122
27851996427,10cyclictest52-21kcompactd009:55:372
27851996425,16cyclictest0-21swapper/210:54:152
27840996437,7cyclictest141rcu_preempt07:41:140
27840996435,20cyclictest31897-21fschecks_count07:17:430
27840996435,20cyclictest31897-21fschecks_count07:17:430
27840996432,13cyclictest141rcu_preempt09:02:180
27840996416,24cyclictest0-21swapper/010:17:510
27856996335,12cyclictest141rcu_preempt11:32:263
27856996326,11cyclictest27825-21cyclictest10:45:353
27856996325,8cyclictest141rcu_preempt07:42:453
27856996325,8cyclictest141rcu_preempt07:42:453
27856996324,9cyclictest141rcu_preempt09:47:183
27856996321,18cyclictest141rcu_preempt08:32:183
27851996337,7cyclictest141rcu_preempt11:02:222
27851996331,13cyclictest141rcu_preempt07:47:402
27851996327,17cyclictest141rcu_preempt09:13:332
27851996320,16cyclictest141rcu_preempt12:17:392
27851996320,16cyclictest141rcu_preempt12:17:392
27847996348,9cyclictest4762-21unixbench_multi11:07:541
27847996348,10cyclictest241ktimers/108:49:471
27847996326,8cyclictest141rcu_preempt10:40:581
27847996326,8cyclictest141rcu_preempt10:40:581
27840996340,7cyclictest141rcu_preempt11:24:460
27840996335,14cyclictest141rcu_preempt08:12:520
27840996335,14cyclictest141rcu_preempt08:12:520
27840996328,7cyclictest0-21swapper/010:47:160
27840996327,8cyclictest141rcu_preempt08:37:550
27856996244,10cyclictest401ktimers/309:57:343
27856996235,8cyclictest141rcu_preempt12:27:533
27856996235,7cyclictest141rcu_preempt09:27:523
27856996233,7cyclictest141rcu_preempt10:58:163
27856996230,6cyclictest0-21swapper/312:12:063
27856996229,8cyclictest141rcu_preempt08:52:423
27856996229,8cyclictest141rcu_preempt08:52:423
27851996235,7cyclictest141rcu_preempt12:07:062
27851996233,12cyclictest141rcu_preempt09:05:132
27851996233,12cyclictest141rcu_preempt09:05:132
27851996230,7cyclictest141rcu_preempt11:30:592
27851996229,15cyclictest141rcu_preempt07:37:562
27851996229,15cyclictest0-21swapper/211:13:202
27851996229,15cyclictest0-21swapper/211:13:202
27851996227,11cyclictest141rcu_preempt11:18:442
27847996247,9cyclictest120-21pr/ttymxc009:27:531
27840996241,7cyclictest141rcu_preempt12:02:160
27840996235,14cyclictest141rcu_preempt07:12:460
27840996234,10cyclictest141rcu_preempt09:57:500
27840996232,6cyclictest141rcu_preempt11:17:100
27840996232,6cyclictest141rcu_preempt11:17:100
27840996230,8cyclictest141rcu_preempt09:47:510
27840996230,8cyclictest141rcu_preempt09:47:510
27840996227,8cyclictest141rcu_preempt09:17:040
27840996221,15cyclictest27825-21cyclictest08:09:570
27856996135,8cyclictest141rcu_preempt12:13:003
27856996133,7cyclictest141rcu_preempt09:32:273
27856996132,8cyclictest141rcu_preempt11:52:443
27856996132,8cyclictest141rcu_preempt11:52:443
27856996129,12cyclictest52-21kcompactd009:14:513
27856996123,10cyclictest141rcu_preempt10:22:443
27856996120,9cyclictest141rcu_preempt10:29:143
27856996120,16cyclictest141rcu_preempt09:25:563
27856996119,15cyclictest141rcu_preempt12:02:513
2785199617,23cyclictest33-21ksoftirqd/208:27:182
27851996131,13cyclictest141rcu_preempt11:25:102
27851996110,20cyclictest33-21ksoftirqd/209:22:182
27847996145,10cyclictest602-21cron11:57:141
27847996145,10cyclictest602-21cron11:57:141
27847996144,10cyclictest16831-21idleruntime-cro08:12:151
27847996143,11cyclictest21104-21munin-plugin-st11:37:151
27847996134,7cyclictest141rcu_preempt12:34:331
27847996128,9cyclictest141rcu_preempt09:15:191
27847996126,8cyclictest141rcu_preempt09:42:041
27856996034,7cyclictest141rcu_preempt10:33:583
27856996033,15cyclictest141rcu_preempt10:39:453
27856996033,15cyclictest141rcu_preempt10:39:453
27856996032,7cyclictest141rcu_preempt08:47:263
27856996024,8cyclictest141rcu_preempt08:22:463
27856996022,8cyclictest141rcu_preempt08:17:153
27856996022,8cyclictest141rcu_preempt08:17:153
27856996020,14cyclictest141rcu_preempt07:10:283
27856996020,14cyclictest141rcu_preempt07:10:283
27856996017,17cyclictest41-21ksoftirqd/309:40:063
27851996025,16cyclictest141rcu_preempt11:52:162
27851996011,17cyclictest30891-21proc_pri08:52:532
27851996011,17cyclictest30891-21proc_pri08:52:532
27847996044,10cyclictest635-21systemd-run07:22:161
27847996044,10cyclictest635-21systemd-run07:22:161
27847996033,7cyclictest141rcu_preempt11:02:391
27847996014,24cyclictest241ktimers/109:48:471
27847996014,24cyclictest241ktimers/109:48:471
27840996025,9cyclictest141rcu_preempt08:43:060
27840996025,9cyclictest141rcu_preempt08:43:060
27840996017,18cyclictest141rcu_preempt07:12:180
27840996017,18cyclictest141rcu_preempt07:12:180
27856995945,9cyclictest9839-21unixbench_multi10:17:593
27856995930,8cyclictest141rcu_preempt07:17:393
27856995930,8cyclictest141rcu_preempt07:17:393
27856995927,9cyclictest141rcu_preempt10:47:513
27856995927,9cyclictest141rcu_preempt10:47:513
27856995921,16cyclictest141rcu_preempt07:37:173
27856995913,16cyclictest141rcu_preempt11:51:143
27851995926,8cyclictest141rcu_preempt07:12:472
27847995944,10cyclictest0-21swapper/110:37:151
27847995943,10cyclictest10351-21systemd-run07:52:161
27847995936,7cyclictest141rcu_preempt10:44:011
27847995927,11cyclictest183-21pr/ttyS011:31:481
27847995913,11cyclictest241ktimers/108:22:161
27847995913,10cyclictest241ktimers/110:52:161
27847995913,10cyclictest241ktimers/110:52:161
27856995821,18cyclictest141rcu_preempt09:17:553
27856995818,16cyclictest41-21ksoftirqd/307:57:463
27851995831,7cyclictest141rcu_preempt08:42:432
27851995831,7cyclictest141rcu_preempt08:42:432
27851995820,16cyclictest183-21pr/ttyS008:37:022
27851995817,18cyclictest33-21ksoftirqd/207:37:152
2784799589,14cyclictest25-21ksoftirqd/110:02:461
27847995843,10cyclictest26226-21kworker/u8:007:42:151
27847995842,10cyclictest11124-21sshd11:20:091
27847995833,7cyclictest141rcu_preempt09:33:241
27847995831,7cyclictest141rcu_preempt08:27:091
27847995828,7cyclictest141rcu_preempt09:23:431
27856995723,16cyclictest141rcu_preempt08:58:273
27851995723,10cyclictest52-21kcompactd010:50:442
27851995723,10cyclictest52-21kcompactd010:50:442
27851995717,15cyclictest141rcu_preempt08:37:422
27847995725,8cyclictest183-21pr/ttyS011:48:321
27847995725,8cyclictest141rcu_preempt10:12:511
27847995721,17cyclictest183-21pr/ttyS009:58:531
27847995715,12cyclictest31215-21tune2fs10:58:001
27856995616,21cyclictest401ktimers/308:05:523
27851995616,18cyclictest33-21ksoftirqd/208:02:442
27847995637,7cyclictest141rcu_preempt11:38:301
27847995625,8cyclictest141rcu_preempt07:44:371
27847995625,8cyclictest141rcu_preempt07:44:371
27847995624,7cyclictest141rcu_preempt10:07:241
27847995612,22cyclictest241ktimers/111:42:391
27856995522,8cyclictest141rcu_preempt08:37:513
27856995518,19cyclictest41-21ksoftirqd/307:37:503
27847995537,8cyclictest0-21swapper/109:07:151
27847995537,8cyclictest0-21swapper/109:07:151
27847995535,7cyclictest141rcu_preempt11:24:131
27847995528,7cyclictest141rcu_preempt09:52:411
27847995513,9cyclictest241ktimers/110:17:521
27856995419,15cyclictest141rcu_preempt08:32:423
27847995439,10cyclictest0-21swapper/107:32:151
27847995438,9cyclictest241ktimers/108:27:541
27847995438,9cyclictest241ktimers/107:11:301
27847995438,9cyclictest241ktimers/107:11:301
27847995438,7cyclictest0-21swapper/109:42:521
27847995437,8cyclictest0-21swapper/109:02:151
27847995435,10cyclictest11837-21fschecks_count10:22:441
27847995432,7cyclictest141rcu_preempt07:57:491
27847995416,12cyclictest6473-21apt-get11:12:341
27847995416,12cyclictest6473-21apt-get11:12:341
27847995415,14cyclictest361-21in:imklog12:12:251
27847995339,8cyclictest231rcuc/108:45:411
27847995339,8cyclictest231rcuc/108:45:411
2784799532,21cyclictest4651-21ntp_states07:32:521
2784799524,15cyclictest31011-21/usr/sbin/munin08:52:551
2784799524,15cyclictest31011-21/usr/sbin/munin08:52:551
27847995220,7cyclictest141rcu_preempt12:22:571
27847995220,7cyclictest141rcu_preempt12:22:571
27847995217,7cyclictest0-21swapper/108:12:361
27847995217,7cyclictest0-21swapper/108:12:361
27847995217,20cyclictest361-21in:imklog09:09:501
2784799514,12cyclictest25-21ksoftirqd/107:52:411
27847995136,9cyclictest241ktimers/112:07:541
27847995031,7cyclictest141rcu_preempt07:24:561
27847995021,8cyclictest141rcu_preempt08:37:491
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional