You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-15 - 14:47
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Tue Jul 15, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1758421337,16sleep00-21swapper/019:05:400
177229912835,52cyclictest0-21swapper/023:06:190
177229912835,52cyclictest0-21swapper/023:06:190
17722991273,37cyclictest356-21runrttasks19:40:580
17722991233,26cyclictest27013-21/usr/sbin/munin23:56:020
177229911731,47cyclictest0-21swapper/020:26:060
17722991172,26cyclictest22441-21ssh21:56:450
17722991172,26cyclictest22441-21ssh21:56:450
17722991152,22cyclictest2264-21sh21:21:280
17722991146,23cyclictest8422-21sh00:20:340
17722991146,23cyclictest8422-21sh00:20:340
17722991143,28cyclictest15149-21cron20:35:440
17722991143,28cyclictest15149-21cron20:35:440
1747721146,16sleep20-21swapper/219:04:152
17727991134,84cyclictest20772-21apt-get22:51:371
17727991134,84cyclictest20772-21apt-get22:51:371
17722991133,48cyclictest3418-21ssh23:16:020
17722991133,48cyclictest3418-21ssh23:16:020
17722991133,26cyclictest9625-21/usr/sbin/munin20:16:180
17722991132,28cyclictest356-21runrttasks00:35:110
17722991124,27cyclictest23002-21ssh22:53:080
17722991124,27cyclictest23002-21ssh22:53:080
177229911242,48cyclictest12151-21ssh23:30:470
17722991123,25cyclictest6289-21taskset20:06:170
17722991123,23cyclictest8350-21rm22:29:010
17722991122,28cyclictest15521-21ls20:36:040
17722991122,27cyclictest1790-21ssh00:07:510
17722991122,25cyclictest2174-21rm22:17:510
17722991117,75cyclictest29834-21cpuspeed00:01:010
17722991113,28cyclictest18052-21users20:41:240
17722991113,27cyclictest356-21runrttasks22:25:230
17722991113,27cyclictest356-21runrttasks22:25:230
17722991112,27cyclictest8054-21ssh21:32:060
17722991112,25cyclictest26442-21ssh21:06:510
17722991104,25cyclictest13868-21ssh23:32:450
17722991104,21cyclictest26326-21rm22:04:030
17722991104,21cyclictest26326-21rm22:04:030
17722991103,27cyclictest15887-21rm22:42:100
17722991103,27cyclictest14308-21sh21:43:310
17722991103,24cyclictest21239-21latency_hist22:50:490
177229911032,47cyclictest0-21swapper/023:42:290
177229911032,47cyclictest0-21swapper/023:42:290
17722991103,23cyclictest21006-21/usr/sbin/munin23:45:560
17722991093,25cyclictest19947-21/usr/sbin/munin19:11:190
177229910932,45cyclictest0-21swapper/020:05:450
17722991092,27cyclictest32661-21ssh22:15:420
17722991073,28cyclictest20058-21ssh21:53:200
17722991073,22cyclictest10768-21ssh00:24:190
177229910728,54cyclictest0-21swapper/021:18:330
17722991072,25cyclictest9672-21uname22:31:110
17722991066,24cyclictest11806-21sh21:39:350
17722991064,26cyclictest5950-21ssh21:29:190
177229910635,63cyclictest0-21swapper/020:23:390
17722991063,28cyclictest8429-21ssh23:24:280
17722991063,25cyclictest21303-21runrttasks20:51:470
17722991052,25cyclictest29425-21taskset19:40:460
17722991052,24cyclictest30381-21ssh21:15:240
17722991042,27cyclictest19925-21taskset20:49:460
17722991032,27cyclictest16517-21ssh23:36:550
1447521030,7sleep30-21swapper/322:39:533
1447521030,7sleep30-21swapper/322:39:533
17722991023,27cyclictest14454-21sh22:39:500
17722991023,27cyclictest14454-21sh22:39:500
17722991022,24cyclictest27622-21taskset19:34:100
17722991013,9cyclictest0-21swapper/020:10:580
17722991013,28cyclictest25978-21ssh22:58:170
17722991013,25cyclictest12874-21ssh00:27:050
17722991013,23cyclictest29123-21ssh22:08:460
17727991002,79cyclictest5236-21kworker/u18:1+flush-179:022:41:381
17722991003,63cyclictest356-21runrttasks21:47:460
17722991002,26cyclictest24473-21taskset19:24:570
17722991002,26cyclictest24473-21taskset19:24:570
17722991002,13cyclictest27155-21ssh23:01:010
1772299994,8cyclictest0-21swapper/019:56:470
1772299994,8cyclictest0-21swapper/019:56:470
1772299993,9cyclictest469-21ssh23:11:000
1772299992,24cyclictest22692-21taskset19:17:400
1772299989,61cyclictest23003-21seq20:57:340
1772299989,59cyclictest0-21swapper/021:02:120
1772299989,59cyclictest0-21swapper/021:02:120
1772299982,26cyclictest32368-21runrttasks19:47:520
1772299982,26cyclictest32368-21runrttasks19:47:520
1772299982,21cyclictest1685-21taskset19:54:550
1772299982,21cyclictest1685-21taskset19:54:550
1772299972,20cyclictest18472-21awk19:06:170
17729999644,15cyclictest27201-21acpi22:05:552
1772299962,22cyclictest26137-21ssh23:54:420
1772299953,28cyclictest3799-21taskset00:11:180
1772299943,25cyclictest26083-21runrttasks19:30:020
1772299943,25cyclictest26083-21runrttasks19:30:020
17727998953,8cyclictest181rcu_preempt23:24:151
1759028811,47sleep117623-21munin-run19:05:451
17727998653,11cyclictest181rcu_preempt19:26:021
17727998653,11cyclictest181rcu_preempt19:26:021
17729998563,16cyclictest5001-21sh22:22:402
17729998563,16cyclictest5001-21sh22:22:402
17727998565,14cyclictest24872-21diskmemload22:02:061
17727998565,14cyclictest24872-21diskmemload22:02:061
17727998552,9cyclictest181rcu_preempt23:56:451
17727998438,11cyclictest181rcu_preempt20:00:471
17727998438,11cyclictest181rcu_preempt20:00:471
1773499837,37cyclictest0-21swapper/323:26:343
17729998366,10cyclictest7476-21/usr/sbin/munin22:26:252
17729998363,12cyclictest17189-21ssh21:48:292
1772999835,40cyclictest39-21ksoftirqd/221:06:202
17729998264,11cyclictest9469-21/usr/sbin/munin22:31:062
17729998254,20cyclictest24872-21diskmemload21:39:542
17729998244,10cyclictest181rcu_preempt19:26:172
17729998244,10cyclictest181rcu_preempt19:26:172
17727998243,10cyclictest181rcu_preempt21:36:051
17729998162,12cyclictest0-21swapper/222:42:222
17729998161,14cyclictest24872-21diskmemload00:21:172
17727998160,13cyclictest4551-21grep22:21:261
17727998160,13cyclictest4551-21grep22:21:261
17727998155,10cyclictest181rcu_preempt23:37:161
17729998039,15cyclictest181rcu_preempt21:10:532
17729998029,16cyclictest181rcu_preempt21:58:292
17729998029,16cyclictest181rcu_preempt21:58:292
17729998012,41cyclictest39-21ksoftirqd/200:30:582
17727998055,8cyclictest181rcu_preempt23:06:141
17727998055,8cyclictest181rcu_preempt23:06:141
17727998050,10cyclictest181rcu_preempt22:27:381
17727998047,17cyclictest181rcu_preempt21:47:021
17727998036,35cyclictest20593-21sh21:54:581
1772799802,38cyclictest13373-21/usr/sbin/munin21:41:211
95782793,22sleep29651-21mailstats20:16:182
1772999796,57cyclictest26523-40ogrotate23:55:442
17729997942,15cyclictest181rcu_preempt22:40:302
17729997942,15cyclictest181rcu_preempt22:40:302
17729997936,12cyclictest181rcu_preempt23:10:472
17729997936,12cyclictest181rcu_preempt23:10:472
17727997959,14cyclictest356-21runrttasks22:56:061
17727997955,7cyclictest181rcu_preempt23:31:011
17727997945,12cyclictest181rcu_preempt20:32:291
17727997945,12cyclictest181rcu_preempt20:32:291
17727997939,17cyclictest0-21swapper/119:18:231
17727997930,36cyclictest18327-21latency19:06:111
1773499783,34cyclictest2161-21chrt00:08:563
17734997817,46cyclictest0-21swapper/322:01:103
17734997817,46cyclictest0-21swapper/322:01:103
1772999784,32cyclictest15165-21latency_hist20:35:482
1772999784,32cyclictest15165-21latency_hist20:35:482
17729997831,18cyclictest181rcu_preempt23:42:182
17729997831,18cyclictest181rcu_preempt23:42:182
17729997817,25cyclictest0-21swapper/219:46:202
17729997817,25cyclictest0-21swapper/219:46:202
17727997850,11cyclictest181rcu_preempt23:51:071
17727997845,14cyclictest181rcu_preempt21:24:041
17729997755,15cyclictest15300-21kworker/u17:4-writeback23:01:222
17729997747,8cyclictest181rcu_preempt00:20:472
17729997747,8cyclictest181rcu_preempt00:20:472
17729997741,17cyclictest181rcu_preempt23:16:362
17729997741,17cyclictest181rcu_preempt23:16:362
1772999772,50cyclictest39-21ksoftirqd/223:57:392
17727997754,17cyclictest379-21cron21:15:441
17727997748,10cyclictest181rcu_preempt20:10:201
17727997738,9cyclictest10836-21sshd23:27:201
17727997730,9cyclictest181rcu_preempt00:35:111
1772799772,55cyclictest1765-21modprobe19:55:441
1772799772,55cyclictest1765-21modprobe19:55:441
17729997649,11cyclictest181rcu_preempt00:01:092
17729997618,29cyclictest39-21ksoftirqd/221:26:212
17729997616,22cyclictest0-21swapper/223:23:462
1772799767,58cyclictest4149-21apt-get20:01:141
17727997657,14cyclictest17711-21cyclictest20:22:571
17729997552,13cyclictest24872-21diskmemload22:01:332
17729997552,13cyclictest24872-21diskmemload22:01:332
1772999753,40cyclictest1314-21grep23:11:222
17727997539,16cyclictest517-21ssh21:19:431
17727997536,10cyclictest181rcu_preempt21:31:201
1772799752,37cyclictest9396-21ssh22:31:031
1772799752,33cyclictest27395-21fschecks_count23:01:081
17729997442,13cyclictest181rcu_preempt21:31:002
17729997438,10cyclictest181rcu_preempt20:51:182
17729997435,17cyclictest181rcu_preempt00:14:342
17729997427,12cyclictest181rcu_preempt23:46:542
17729997424,12cyclictest181rcu_preempt20:48:422
17729997416,24cyclictest39-21ksoftirqd/221:22:362
17727997448,13cyclictest181rcu_preempt20:11:191
17727997440,14cyclictest181rcu_preempt00:23:241
17727997431,11cyclictest181rcu_preempt19:15:481
1772799741,35cyclictest26050-21sed21:06:231
136202740,8sleep2371rcuc/223:32:002
17729997345,19cyclictest356-21runrttasks20:15:462
17729997340,23cyclictest0-21swapper/222:16:092
1772999733,48cyclictest10869-21fschecks_time20:21:082
17729997334,7cyclictest181rcu_preempt19:06:022
17727997357,10cyclictest5931-21cron23:20:431
17727997357,10cyclictest5931-21cron23:20:431
17727997356,11cyclictest17711-21cyclictest21:26:241
17727997356,10cyclictest23791-21fschecks_time21:01:071
17727997356,10cyclictest23791-21fschecks_time21:01:071
17727997349,12cyclictest0-21swapper/120:56:091
17727997342,10cyclictest181rcu_preempt22:36:231
17727997342,10cyclictest181rcu_preempt22:36:231
17727997341,13cyclictest181rcu_preempt00:07:341
17727997334,8cyclictest181rcu_preempt20:51:221
17727997334,10cyclictest181rcu_preempt23:10:581
17727997327,13cyclictest181rcu_preempt00:26:221
17729997252,15cyclictest17711-21cyclictest22:46:022
17729997250,16cyclictest30706-21ntp_kernel_pll_22:11:172
17729997243,11cyclictest181rcu_preempt21:55:472
17729997238,12cyclictest181rcu_preempt20:30:472
17729997233,17cyclictest181rcu_preempt19:18:232
1772999722,35cyclictest39-21ksoftirqd/223:26:282
17729997217,30cyclictest381ktimers/219:11:172
17727997249,10cyclictest181rcu_preempt00:05:351
17727997240,12cyclictest181rcu_preempt21:58:191
17727997240,12cyclictest181rcu_preempt21:58:191
17727997235,11cyclictest31216-21grep22:11:261
1772799723,47cyclictest18525-21latency_hist20:45:471
1773499713,43cyclictest0-21swapper/322:51:443
1773499713,43cyclictest0-21swapper/322:51:443
17729997155,9cyclictest381ktimers/222:52:272
17729997155,9cyclictest381ktimers/222:52:272
17729997145,8cyclictest181rcu_preempt19:21:082
17729997145,8cyclictest181rcu_preempt19:21:082
17729997133,13cyclictest181rcu_preempt21:42:242
17729997132,11cyclictest181rcu_preempt21:18:502
1772999712,33cyclictest468-21ssh00:06:072
17727997153,11cyclictest20618-21ssh22:50:201
17727997147,9cyclictest181rcu_preempt00:11:071
17727997141,9cyclictest181rcu_preempt23:43:401
17727997141,9cyclictest181rcu_preempt23:43:401
17727997140,10cyclictest181rcu_preempt19:46:081
17727997140,10cyclictest181rcu_preempt19:46:081
1772799713,37cyclictest9267-21fschecks_time20:16:071
1772799712,29cyclictest29832-21fschecks_count19:41:061
17727997122,26cyclictest27926-21/usr/sbin/munin22:06:191
17729997051,13cyclictest381ktimers/200:29:592
17729997045,16cyclictest376-21in:imuxsock20:40:442
17727997045,9cyclictest181rcu_preempt20:46:001
17727997045,19cyclictest27174-21threads19:31:201
17727997045,16cyclictest28402-21kernelversion19:36:111
1772799702,37cyclictest24872-21diskmemload23:46:021
1773499698,29cyclictest0-21swapper/322:40:593
17729996949,12cyclictest22179-21fschecks_time20:56:092
17729996944,8cyclictest181rcu_preempt21:05:492
17729996944,8cyclictest181rcu_preempt21:05:492
1772999694,32cyclictest7084-21cron20:10:442
17729996940,8cyclictest0-21swapper/219:56:012
17729996940,8cyclictest0-21swapper/219:56:012
1772999693,47cyclictest16041-21cat23:36:202
17729996933,9cyclictest181rcu_preempt20:00:462
17729996933,9cyclictest181rcu_preempt20:00:462
1772999692,33cyclictest30486-21grep19:41:212
17727996947,9cyclictest181rcu_preempt22:16:171
17727996943,17cyclictest15440-21/usr/sbin/munin20:36:201
17727996929,11cyclictest181rcu_preempt19:21:211
17727996929,11cyclictest181rcu_preempt19:21:211
1772999685,41cyclictest17327-21fschecks_time20:41:092
17729996818,25cyclictest39-21ksoftirqd/222:56:242
1772999674,48cyclictest26090-21kworker/u17:2+ext4-rsv-conversion20:01:152
17727996737,11cyclictest181rcu_preempt00:16:031
17727996737,11cyclictest181rcu_preempt00:16:031
1772999666,26cyclictest39-21ksoftirqd/219:36:142
1772999653,42cyclictest356-21runrttasks19:32:432
1772799622,33cyclictest12267-21cpu20:26:021
1773499594,41cyclictest11919-21chrt21:39:563
1773499573,36cyclictest21812-21taskset23:46:193
1773499572,41cyclictest20398-21taskset19:11:133
1773499564,34cyclictest14802-21chrt20:31:253
1773499564,34cyclictest14802-21chrt20:31:253
1773499563,37cyclictest638-21chrt23:11:063
1773499563,35cyclictest5089-21chrt22:22:563
1773499563,35cyclictest5089-21chrt22:22:563
1773499553,38cyclictest26652-21taskset23:55:473
1773499552,36cyclictest13078-21chrt00:27:443
1773499543,40cyclictest1296-21chrt22:16:193
1773499543,37cyclictest17764-21chrt21:50:093
1773499543,35cyclictest3874-21taskset23:16:123
1773499543,35cyclictest3874-21taskset23:16:123
1773499543,34cyclictest15244-21taskset21:45:473
1773499543,31cyclictest6694-21taskset21:30:563
1773499535,33cyclictest1344-21chrt21:21:073
1773499533,34cyclictest9004-21taskset22:30:463
1773499533,34cyclictest18678-21chrt20:45:593
1773499532,35cyclictest29798-21chrt23:05:473
1773499532,34cyclictest6678-21chrt00:16:183
1773499532,34cyclictest6678-21chrt00:16:183
1773499523,35cyclictest28779-21chrt22:07:443
1773499523,33cyclictest4808-21chrt21:26:223
1773499523,33cyclictest18790-21chrt23:41:173
1773499523,33cyclictest18790-21chrt23:41:173
1773499523,31cyclictest28272-21taskset19:36:083
1773499522,35cyclictest19236-21chrt22:46:223
1773499522,34cyclictest17392-21chrt23:39:313
1773499522,31cyclictest7607-21chrt20:11:083
1773499513,36cyclictest28797-21chrt21:11:193
1773499513,34cyclictest23534-21chrt19:21:093
1773499513,34cyclictest23534-21chrt19:21:093
1773499513,33cyclictest1762-21chrt19:55:443
1773499513,33cyclictest1762-21chrt19:55:443
1773499513,33cyclictest14012-21chrt23:33:103
1773499512,36cyclictest20318-21chrt20:51:013
1773499512,33cyclictest9652-21chrt22:31:103
1773499503,33cyclictest27706-21chrt23:56:203
1773499502,33cyclictest21927-21taskset20:56:033
1773499502,32cyclictest6295-21chrt20:06:183
1773499502,31cyclictest30792-21chrt21:15:523
17734995012,30cyclictest0-21swapper/319:56:293
17734995012,30cyclictest0-21swapper/319:56:293
1773499493,36cyclictest18610-21chrt19:06:193
1773499493,34cyclictest10877-21chrt20:21:083
1773499492,34cyclictest31762-21chrt00:04:063
1773499492,34cyclictest19762-21chrt21:52:283
1773499483,33cyclictest6291-21chrt23:20:593
1773499482,33cyclictest31685-21chrt22:12:463
1773499482,33cyclictest25071-21chrt22:56:193
1773499482,30cyclictest4441-21chrt20:01:483
1773499482,28cyclictest29754-21taskset19:41:023
1773499474,30cyclictest22960-21chrt19:20:453
1773499473,33cyclictest900-21kworker/3:1+events00:21:133
1773499473,33cyclictest32176-21chrt19:46:253
1773499473,33cyclictest32176-21chrt19:46:253
1773499473,29cyclictest13740-21chrt20:30:483
1773499473,27cyclictest26479-21taskset19:31:023
1773499464,29cyclictest3573-21chrt00:11:113
1773499463,32cyclictest23732-21chrt22:00:383
1773499463,32cyclictest23732-21chrt22:00:383
1773499463,32cyclictest23615-21chrt21:01:013
1773499463,32cyclictest23615-21chrt21:01:013
1773499463,31cyclictest16845-21chrt00:35:463
1773499462,32cyclictest15942-21chrt20:36:153
1773499462,29cyclictest30625-21chrt23:06:163
1773499462,29cyclictest30625-21chrt23:06:163
173742464,27sleep30-21swapper/319:02:543
1773499453,31cyclictest9075-21taskset20:16:023
1773499444,28cyclictest26472-21chrt21:06:573
1773499433,31cyclictest25199-21chrt19:26:163
1773499433,31cyclictest25199-21chrt19:26:163
1773499432,29cyclictest17809-21taskset20:41:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional