You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 21:58
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Tue Sep 16, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23879214310,79sleep023912-21fschecks_time06:57:050
25095991276,93cyclictest9064-21apt-get12:21:451
250919912559,56cyclictest27941-21latency_hist07:11:310
2440121250,29sleep30-21swapper/306:57:243
25091991172,39cyclictest30079-21latency_hist12:01:300
250919911349,56cyclictest0-21swapper/011:31:280
1148221130,6sleep30-21swapper/309:24:473
25091991125,35cyclictest31953-21cpu09:01:460
25091991125,35cyclictest31953-21cpu09:01:460
250919911239,12cyclictest0-21swapper/010:46:520
25091991123,32cyclictest27966-21systemd-journal11:46:320
250919911137,11cyclictest0-21swapper/008:01:310
250919911137,11cyclictest0-21swapper/008:01:310
250919911135,16cyclictest0-21swapper/009:01:320
25091991105,42cyclictest27839-21ssh09:56:460
250919911034,20cyclictest1-21systemd09:56:290
250919911034,10cyclictest0-21swapper/011:36:500
250919910936,15cyclictest0-21swapper/009:21:310
25091991084,40cyclictest7008-21chrt11:17:060
25091991084,40cyclictest7008-21chrt11:17:060
250919910837,11cyclictest0-21swapper/007:56:280
25091991074,29cyclictest4387-21latency_hist09:11:310
25091991073,45cyclictest30667-21fschecks_count10:01:500
25091991073,34cyclictest356-21runrttasks09:46:140
25091991073,27cyclictest15087-21grep09:31:530
250919910723,25cyclictest0-21swapper/010:06:500
250919910723,25cyclictest0-21swapper/010:06:500
250919910652,48cyclictest18207-21kworker/u17:2+xprtiod09:21:560
25091991063,36cyclictest15073-21latency_hist08:11:310
25091991057,43cyclictest24391-21apt-get11:51:410
250919910535,11cyclictest0-21swapper/007:46:510
250919910535,11cyclictest0-21swapper/007:46:510
25091991053,28cyclictest30955-21grep12:02:010
250919910531,12cyclictest0-21swapper/010:55:220
250919910531,12cyclictest0-21swapper/010:55:220
250919910435,9cyclictest0-21swapper/011:10:540
25091991043,39cyclictest16401-21grep08:13:010
25091991043,35cyclictest26644-21munin-run07:06:280
25100991033,30cyclictest356-21runrttasks09:14:502
25091991034,38cyclictest6694-21ps12:16:590
25091991034,37cyclictest19283-21/usr/sbin/munin11:41:540
25091991034,37cyclictest19283-21/usr/sbin/munin11:41:540
250919910333,11cyclictest0-21swapper/009:16:310
25091991024,35cyclictest30043-21latency_hist11:01:300
25091991023,32cyclictest673-21cstates12:06:470
25091991022,43cyclictest3954-21grep12:11:580
25091991014,31cyclictest3996-21ntp_offset10:11:580
250919910141,9cyclictest0-21swapper/011:15:540
25091991013,40cyclictest14845-21fschecks_count10:31:520
25091991013,40cyclictest14845-21fschecks_count10:31:520
25091991013,37cyclictest30206-21/usr/sbin/munin11:01:500
25091991013,36cyclictest24518-21cat08:37:070
250919910126,12cyclictest0-21swapper/010:46:300
250919910126,12cyclictest0-21swapper/010:46:300
25100991002,59cyclictest14600-21apt-get10:31:412
25100991002,59cyclictest14600-21apt-get10:31:412
25091991003,38cyclictest14595-21latency_hist09:31:300
25091991003,38cyclictest14595-21latency_hist09:31:300
25091991003,36cyclictest5010-21latency_hist07:41:300
23651210013,52sleep223619-21/usr/sbin/munin06:56:462
2509199994,34cyclictest28882-21timerwakeupswit07:12:050
2509199993,36cyclictest21434-21unixbench_singl08:27:050
2509199993,36cyclictest21434-21unixbench_singl08:27:050
2509199984,35cyclictest17442-21fschecks_count10:36:510
2509199984,34cyclictest29530-21grep08:52:050
2509199984,32cyclictest11890-21/usr/sbin/munin12:26:530
2509199983,36cyclictest8980-21apt-config10:21:320
2509199983,36cyclictest2967-21unixbench_singl07:32:070
2509199974,36cyclictest17815-21grep08:17:060
2509199973,40cyclictest24961-21latency_hist09:51:300
2509199973,36cyclictest26246-21unixbench_multi08:42:050
2509199973,36cyclictest19507-21modprobe09:41:280
2509199973,33cyclictest26953-21/usr/sbin/munin08:46:480
2509199973,33cyclictest26953-21/usr/sbin/munin08:46:480
25100999652,38cyclictest11586-21idleruntime-cro12:26:272
2510099962,29cyclictest29505-21seq09:59:222
2509199964,36cyclictest356-21runrttasks10:21:480
2509199964,36cyclictest356-21runrttasks10:21:480
2509199964,36cyclictest12189-21fschecks_count08:02:440
2509199962,39cyclictest356-21runrttasks07:31:250
2509199954,31cyclictest356-21runrttasks10:31:140
2509199954,30cyclictest31208-21/usr/sbin/munin07:21:490
2510099942,32cyclictest30525-21grep07:17:042
2509599942,63cyclictest9301-21runrttasks10:22:021
2509599942,63cyclictest9301-21runrttasks10:22:021
2509199943,40cyclictest9045-21/usr/sbin/munin12:22:370
2510099933,24cyclictest356-21runrttasks09:45:252
25100999247,37cyclictest0-21swapper/208:02:442
25100999241,25cyclictest3228-21kworker/u17:0-nfsiod11:22:402
25100999228,41cyclictest0-21swapper/211:00:082
2510099913,30cyclictest25276-21/usr/sbin/munin07:02:122
2510099904,60cyclictest24937-21/usr/sbin/munin11:52:302
2509199903,35cyclictest356-21runrttasks08:34:330
2509199903,35cyclictest356-21runrttasks08:34:330
2510099892,30cyclictest25096-21/usr/sbin/munin09:52:042
2510099883,7cyclictest0-21swapper/209:06:452
2510099883,31cyclictest7173-21/usr/sbin/munin09:17:012
2510099882,34cyclictest4211-21tail11:12:072
25100998744,36cyclictest0-21swapper/211:16:502
25100998744,36cyclictest0-21swapper/211:16:502
25095998764,16cyclictest22469-21kernelversion11:46:541
2510099862,36cyclictest19361-21sh10:40:442
2510099862,33cyclictest27921-21ssh11:56:562
25100998540,34cyclictest0-21swapper/208:36:312
25100998540,34cyclictest0-21swapper/208:36:312
2510099853,37cyclictest12284-21unixbench_multi11:27:052
2510099853,35cyclictest27753-21lockfile-create07:11:282
2510099853,33cyclictest19382-21latency08:22:532
2510099852,30cyclictest27966-21systemd-journal10:17:072
25095998558,19cyclictest3712-21sh12:11:531
25095998534,25cyclictest20907-21latency08:26:541
25095998534,25cyclictest20907-21latency08:26:541
2510099843,41cyclictest12271-21/usr/sbin/munin09:26:592
2510099843,41cyclictest12271-21/usr/sbin/munin09:26:592
2510099843,38cyclictest6537-21ssh12:16:522
2510099843,30cyclictest28345-21sendmail08:51:272
2510099843,30cyclictest28345-21sendmail08:51:272
2510099842,34cyclictest32592-21/usr/sbin/munin09:02:032
2510099842,34cyclictest32592-21/usr/sbin/munin09:02:032
25095998455,20cyclictest5053-21latency_hist07:41:311
2510099834,35cyclictest6745-21latency_hist07:46:312
2510099833,37cyclictest58-21kcompactd007:51:382
2510099833,36cyclictest25723-21sed08:41:542
2510099833,35cyclictest12596-21/usr/sbin/munin12:27:032
2510099833,33cyclictest15429-21/usr/sbin/munin08:11:392
2510099832,36cyclictest431-21/usr/sbin/munin11:06:532
2510099832,32cyclictest356-21runrttasks11:38:402
2510099832,30cyclictest30840-21/usr/sbin/munin10:01:532
2510099823,8cyclictest0-21swapper/208:56:442
2510099823,8cyclictest0-21swapper/208:51:412
2510099823,7cyclictest0-21swapper/210:11:422
25100998236,39cyclictest0-21swapper/210:22:052
25100998236,39cyclictest0-21swapper/210:22:052
2510099823,33cyclictest1081-21grep07:27:002
2510099823,32cyclictest356-21runrttasks07:36:552
2510099823,32cyclictest18520-21latency_hist08:21:302
2510099822,32cyclictest3930-21ssh12:11:572
2510099822,31cyclictest7288-21/usr/sbin/munin07:46:552
2510099822,31cyclictest7288-21/usr/sbin/munin07:46:552
2510099822,29cyclictest20200-21tune2fs10:41:512
2510099822,29cyclictest20200-21tune2fs10:41:512
2510099822,23cyclictest32722-21latency_hist07:26:312
2510099813,35cyclictest15262-21grep08:11:322
2510099813,32cyclictest24156-21latency08:36:582
2510099813,32cyclictest11840-21/usr/sbin/munin10:26:542
2510099813,31cyclictest356-21runrttasks08:27:022
2510099813,31cyclictest356-21runrttasks08:27:022
25100998129,45cyclictest3169-21latency_hist10:11:302
25100998129,45cyclictest3169-21latency_hist10:11:302
2510099812,31cyclictest31314-21grep12:02:062
2510099812,25cyclictest356-21runrttasks07:11:322
25095998160,15cyclictest545-21latency_hist10:06:301
25095998152,20cyclictest580-21cpu12:06:461
2510099803,7cyclictest0-21swapper/209:21:452
2510099803,32cyclictest22900-21sh10:46:502
2510099803,32cyclictest17490-21fschecks_count09:36:512
2510099803,31cyclictest10404-21/usr/sbin/munin07:56:522
2510099803,31cyclictest10404-21/usr/sbin/munin07:56:522
2510099803,29cyclictest356-21runrttasks11:46:482
2509599807,57cyclictest24960-21apt-get11:52:221
25095998059,14cyclictest25089-21cyclictest08:37:041
25100997957,15cyclictest25089-21cyclictest11:31:492
2510099793,70cyclictest21865-21latency_hist11:46:302
2510099793,70cyclictest21865-21latency_hist11:46:302
2510099793,33cyclictest31303-21/usr/sbin/munin11:02:072
2510099793,30cyclictest3350-21modprobe07:36:282
2510099793,30cyclictest25221-21cut10:51:542
2510099793,30cyclictest25221-21cut10:51:542
2510099792,30cyclictest23568-21grep09:47:052
2510099792,30cyclictest15143-21grep09:31:542
2509599792,58cyclictest18890-21apt-get08:22:131
2510099782,31cyclictest1396-21/usr/sbin/munin12:07:052
25095997836,35cyclictest3727-21fschecks_time10:11:521
25095997814,28cyclictest14252-21modprobe10:31:271
25095997662,10cyclictest25089-21cyclictest11:27:051
25091997655,15cyclictest356-21runrttasks08:22:450
25095997514,21cyclictest21495-21runrttasks10:43:441
25095997514,21cyclictest21495-21runrttasks10:43:441
2509599743,39cyclictest11966-21sshd08:01:591
25095997353,10cyclictest0-21swapper/111:42:041
25095997353,10cyclictest0-21swapper/111:42:041
25095997352,15cyclictest25089-21cyclictest09:32:091
25095997337,27cyclictest17246-21apt-get09:36:411
2509599733,42cyclictest29931-21chrt11:01:271
2509599732,40cyclictest17643-21kernelversion10:36:571
2509199734,56cyclictest10964-21modprobe11:26:270
25095997252,14cyclictest24487-21sendmail10:51:271
25095997251,13cyclictest378-21rs:main1
2509599723,31cyclictest356-21runrttasks11:05:251
25095997213,28cyclictest0-21swapper/111:38:091
2510399713,34cyclictest0-21swapper/310:22:063
2510399713,34cyclictest0-21swapper/310:22:063
25095997147,15cyclictest10057-21kworker/u19:0+flush-179:007:41:541
2509599713,29cyclictest30047-21latency_hist08:56:301
25095997132,25cyclictest14892-21cron08:11:281
2509599712,46cyclictest31123-21latency_hist07:21:311
25091997152,14cyclictest14261-21fschecks_time11:31:500
2510399703,38cyclictest0-21swapper/311:51:413
25095997047,15cyclictest0-21swapper/111:16:571
25095997047,15cyclictest0-21swapper/111:16:571
25095997045,12cyclictest0-21swapper/109:26:311
2509599703,51cyclictest15433-21apt-get08:11:461
25095997027,28cyclictest8438-21sh10:20:531
25095996952,12cyclictest25089-21cyclictest12:17:031
25095996941,20cyclictest25214-21kernelversion10:51:541
25095996941,20cyclictest25214-21kernelversion10:51:541
25095996932,26cyclictest30224-21apt-get08:56:451
25103996812,46cyclictest0-21swapper/312:21:473
2509599689,28cyclictest0-21swapper/109:26:511
2509599689,28cyclictest0-21swapper/109:26:511
25095996848,12cyclictest0-21swapper/109:51:271
25095996835,24cyclictest25294-21apt-get07:01:581
25095996831,29cyclictest25097-21latency_hist07:01:321
25095996746,15cyclictest16930-21latency_hist10:36:301
25095996746,15cyclictest16930-21latency_hist10:36:301
25095996745,12cyclictest0-21swapper/107:11:301
2509599674,47cyclictest1835-21latency_hist07:31:311
2509599672,43cyclictest12188-21fschecks_time12:26:521
25095996647,14cyclictest25089-21cyclictest11:56:471
25095996645,15cyclictest17619-21proc_pri08:17:031
25095996644,12cyclictest0-21swapper/108:47:051
25095996644,12cyclictest0-21swapper/108:47:051
25095996641,15cyclictest14090-21cstates11:31:451
2509599663,45cyclictest751-21cpu10:06:451
2509599663,45cyclictest751-21cpu10:06:451
2509599663,43cyclictest23036-21tail08:32:031
2509599663,43cyclictest23036-21tail08:32:031
25095996624,26cyclictest28010-21apt-get07:11:401
2509599662,38cyclictest8636-21diskstats07:51:481
2509599662,36cyclictest8905-21fschecks_time11:21:561
25095996616,19cyclictest6523-21seq09:15:151
2509599653,45cyclictest356-21runrttasks11:07:581
2509599653,41cyclictest505-21sh09:02:071
2509599653,41cyclictest505-21sh09:02:071
2509199653,42cyclictest6194-21unixbench_singl07:42:050
25103996441,15cyclictest451rcuc/310:39:553
25103996416,39cyclictest0-21swapper/308:02:113
25095996441,13cyclictest0-21swapper/109:41:531
25095996438,20cyclictest25089-21cyclictest07:56:541
25095996438,20cyclictest25089-21cyclictest07:56:541
25095996438,10cyclictest1957-21apt-get09:06:451
2509599643,43cyclictest356-21runrttasks12:03:381
2509599643,41cyclictest3352-21modprobe07:36:281
25095996421,33cyclictest28117-21fschecks_time09:56:551
25091996425,27cyclictest30192-21grep07:17:000
25095996341,16cyclictest27521-21cut09:56:301
25095996339,12cyclictest0-21swapper/107:47:061
25095996339,12cyclictest0-21swapper/107:47:061
25095996337,13cyclictest0-21swapper/109:21:281
25095996232,19cyclictest3214-21apt-get11:11:451
2509599613,39cyclictest31848-21wc07:22:031
2510399603,47cyclictest22647-21taskset09:46:453
25103995844,9cyclictest451rcuc/309:39:173
25103995835,16cyclictest451rcuc/309:27:503
25103995835,16cyclictest451rcuc/309:27:503
2510399579,40cyclictest0-21swapper/308:12:023
25103995742,8cyclictest451rcuc/310:46:183
25103995742,8cyclictest451rcuc/310:46:183
25103995736,14cyclictest451rcuc/309:05:383
25103995736,14cyclictest451rcuc/309:05:383
25103995641,10cyclictest451rcuc/307:02:133
25103995640,9cyclictest451rcuc/310:59:233
25103995541,8cyclictest451rcuc/309:46:173
2510399545,33cyclictest0-21swapper/308:22:123
2510399544,37cyclictest21444-21taskset11:45:333
2510399544,37cyclictest21444-21taskset11:45:333
25103995439,9cyclictest451rcuc/311:28:483
25103995439,9cyclictest451rcuc/307:37:093
25103995438,9cyclictest451rcuc/311:02:313
2510399543,36cyclictest22056-21chrt08:31:313
2510399543,36cyclictest22056-21chrt08:31:313
2510399542,37cyclictest26127-21chrt08:42:043
25103995337,9cyclictest451rcuc/307:07:113
2510399533,37cyclictest30377-21chrt10:01:323
2510399533,36cyclictest14853-21taskset09:31:423
25103995332,14cyclictest451rcuc/309:11:383
25103995240,8cyclictest451rcuc/309:19:073
25103995239,9cyclictest451rcuc/312:04:193
2510399523,36cyclictest18519-21chrt08:21:303
2510399523,33cyclictest14451-21chrt10:31:313
2510399522,37cyclictest22218-21taskset08:31:433
2510399522,37cyclictest22218-21taskset08:31:433
25103995137,10cyclictest451rcuc/307:21:253
2510399513,35cyclictest24164-21taskset08:36:583
2510399513,35cyclictest1033-21chrt10:06:513
2510399513,35cyclictest1033-21chrt10:06:513
2510399513,34cyclictest5092-21chrt10:14:043
25103995130,15cyclictest451rcuc/310:06:153
2510399512,32cyclictest16245-21chrt11:36:053
25103995035,8cyclictest451rcuc/309:08:583
25103995035,10cyclictest451rcuc/307:34:213
2510399503,35cyclictest6584-21taskset12:16:543
2510399502,37cyclictest17187-21chrt11:36:543
2510399502,35cyclictest2751-21chrt11:11:053
2510399502,35cyclictest15630-21taskset10:32:083
2510399502,35cyclictest15630-21taskset10:32:083
2510399502,34cyclictest5314-21chrt07:41:453
2510399494,37cyclictest7141-21chrt11:17:303
2510399494,37cyclictest7141-21chrt11:17:303
25103994935,9cyclictest451rcuc/312:10:513
2510399493,37cyclictest31995-21chrt07:22:053
2510399493,32cyclictest31581-21taskset09:01:283
2510399493,31cyclictest27270-21chrt10:56:313
2510399493,31cyclictest27270-21chrt10:56:313
2510399493,31cyclictest11859-21taskset12:26:353
2510399492,35cyclictest7723-21chrt10:18:383
2510399483,35cyclictest10378-21taskset07:56:523
2510399483,35cyclictest10378-21taskset07:56:523
2510399483,34cyclictest27911-21chrt11:56:563
25103994832,11cyclictest451rcuc/307:29:113
25103994828,13cyclictest451rcuc/312:13:473
25103994735,8cyclictest451rcuc/308:06:413
2510399473,34cyclictest9786-21chrt11:22:323
2510399473,33cyclictest7291-21chrt07:46:563
2510399473,33cyclictest7291-21chrt07:46:563
2510399454,28cyclictest22915-21chrt10:46:513
2510399453,33cyclictest29058-21chrt08:51:563
25103994531,10cyclictest451rcuc/309:53:513
2510399452,31cyclictest24053-21chrt11:50:193
2510399452,27cyclictest27337-21chrt08:46:543
2510399452,27cyclictest27337-21chrt08:46:543
25103994431,8cyclictest451rcuc/307:55:563
2510399422,28cyclictest5653-21taskset11:16:283
2510399422,28cyclictest28163-21chrt07:11:483
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional