You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-15 - 07:10
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Fri May 15, 2026 00:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
141219913164,60cyclictest2374-21systemd-journal21:59:130
1376921186,15sleep00-21swapper/018:58:370
14121991105,29cyclictest28828-21chrt23:54:580
141219911011,32cyclictest352-21runrttasks22:53:330
14121991074,31cyclictest8400-21/usr/sbin/munin21:32:480
14121991073,41cyclictest15058-21apt-get20:42:140
14121991064,31cyclictest352-21runrttasks00:11:010
14121991063,30cyclictest13830-21latency21:41:360
14121991062,30cyclictest12966-21sendmail_mailtr00:21:430
14121991059,13cyclictest0-21swapper/023:41:260
14121991059,13cyclictest0-21swapper/023:41:260
14121991053,28cyclictest17560-21/usr/sbin/munin23:36:360
14121991053,28cyclictest12491-21cat23:26:450
14121991043,22cyclictest16340-21apt-get21:46:240
14121991043,22cyclictest16340-21apt-get21:46:240
14121991042,21cyclictest14106-21grep20:36:450
14121991035,37cyclictest27623-21ssh21:11:220
141219910333,63cyclictest0-21swapper/023:11:350
141219910332,64cyclictest0-21swapper/022:40:360
141219910332,64cyclictest0-21swapper/022:40:360
14121991033,23cyclictest22689-21chrt22:51:000
141219910237,55cyclictest0-21swapper/022:21:250
14121991023,32cyclictest2440-21sort23:11:110
14121991023,29cyclictest5575-21/usr/sbin/munin23:16:290
14121991023,29cyclictest5575-21/usr/sbin/munin23:16:290
14121991023,27cyclictest20462-21/usr/sbin/munin21:51:480
14121991022,24cyclictest9194-21taskset00:16:140
141219910147,47cyclictest0-21swapper/022:06:380
14121991014,29cyclictest32728-21apt-key00:01:130
14121991013,32cyclictest32082-21/usr/sbin/munin22:11:340
14121991013,30cyclictest26066-21/usr/sbin/munin22:56:340
14121991012,29cyclictest3029-21latency_hist20:06:110
141219910040,53cyclictest21265-21diskmemload23:56:340
141219910040,53cyclictest21265-21diskmemload23:56:340
14121991003,27cyclictest352-21runrttasks21:17:460
1412199993,23cyclictest7111-21cat21:31:120
14121999916,24cyclictest26292-21systemd-run23:51:090
14121999916,24cyclictest26292-21systemd-run23:51:090
1412199989,12cyclictest0-21swapper/022:21:120
1412199983,37cyclictest352-21runrttasks21:03:180
1412199983,33cyclictest16319-21/usr/sbin/munin19:06:430
1412199983,31cyclictest10232-21fschecks_time20:26:330
14121999828,63cyclictest0-21swapper/022:26:180
14121999825,8cyclictest0-21swapper/021:39:530
14121999824,12cyclictest31045-21sh23:04:140
14121999824,12cyclictest31045-21sh23:04:140
1412199982,33cyclictest24390-21latency_hist19:31:130
1412199982,32cyclictest25251-21latency_hist22:01:130
1412199982,32cyclictest25251-21latency_hist22:01:130
1412199982,32cyclictest11578-21/usr/sbin/munin20:31:320
1412199982,32cyclictest11578-21/usr/sbin/munin20:31:320
14135999714,69cyclictest7354-21apt-get21:31:442
1412199974,31cyclictest19900-21/usr/sbin/munin20:56:360
1412199973,34cyclictest32753-21grep19:56:330
1412199973,20cyclictest8349-21apt-get20:21:330
1412199973,20cyclictest1352-21/usr/sbin/munin21:21:300
1412199973,20cyclictest1352-21/usr/sbin/munin21:21:300
14121999730,56cyclictest0-21swapper/022:46:080
1412199964,21cyclictest14700-21cat23:31:260
1412199963,33cyclictest19565-21/usr/sbin/munin19:16:310
1412199963,33cyclictest18506-21grep20:51:310
1412199963,32cyclictest352-21runrttasks20:15:330
1412199963,30cyclictest17961-21/usr/sbin/munin19:11:440
1412199962,34cyclictest14023-21date22:36:120
1412199954,31cyclictest21402-21fschecks_count19:21:300
1412199953,33cyclictest8717-21cstates23:21:270
1412199953,33cyclictest24415-21/usr/sbin/munin19:31:370
1412199953,33cyclictest24415-21/usr/sbin/munin19:31:370
1412199953,33cyclictest17281-21/usr/sbin/munin20:46:420
1412199953,33cyclictest17281-21/usr/sbin/munin20:46:420
1412199953,24cyclictest6399-21cstates00:11:260
1412199952,38cyclictest8067-21latency_hist20:21:120
1412199952,38cyclictest8067-21latency_hist20:21:120
1412199952,25cyclictest15179-21/usr/sbin/munin00:26:350
1412199952,25cyclictest15179-21/usr/sbin/munin00:26:350
1412199943,29cyclictest15054-21find19:01:430
1412199943,29cyclictest15054-21find19:01:430
14121999411,26cyclictest17702-21kworker/u17:1+events_unbound21:11:090
14127999356,10cyclictest181rcu_preempt21:56:341
14127999354,11cyclictest181rcu_preempt23:33:091
1412199934,32cyclictest29230-21/usr/sbin/munin19:46:470
1412199933,33cyclictest28547-21unixbench_multi19:41:450
1412199923,34cyclictest3433-21fschecks_count20:06:300
1412199922,29cyclictest27500-21latency_hist19:41:110
1412199922,29cyclictest27500-21latency_hist19:41:110
14127999173,12cyclictest25804-21systemd-run19:36:101
14127999173,12cyclictest25804-21systemd-run19:36:101
1412199913,31cyclictest32235-21latency_hist19:56:120
14127998949,10cyclictest181rcu_preempt22:09:591
14135998775,7cyclictest0-21swapper/223:51:132
14135998768,12cyclictest2374-21systemd-journal23:18:192
14135998768,12cyclictest2374-21systemd-journal23:18:192
1413599865,65cyclictest352-21runrttasks00:01:462
14127998659,8cyclictest181rcu_preempt23:03:551
14127998659,8cyclictest181rcu_preempt23:03:551
14127998340,8cyclictest181rcu_preempt22:16:421
14135998214,24cyclictest22035-21apt-get21:56:122
14135998213,31cyclictest25137-21taskset22:54:232
14127998263,13cyclictest18190-21latency_hist00:31:121
14127998263,13cyclictest18190-21latency_hist00:31:121
14127998249,13cyclictest181rcu_preempt21:21:431
14127998249,13cyclictest181rcu_preempt21:21:431
1401528211,39sleep10-21swapper/119:01:111
14135998154,19cyclictest13075-21latency_hist20:36:112
14135998154,19cyclictest13075-21latency_hist20:36:112
14127998161,14cyclictest11335-21latency_hist23:26:111
14127998155,12cyclictest181rcu_preempt19:06:521
14127998153,11cyclictest181rcu_preempt21:17:551
14127998152,13cyclictest181rcu_preempt23:16:411
14127998152,13cyclictest181rcu_preempt23:16:411
14127998152,13cyclictest181rcu_preempt21:30:511
14127998145,12cyclictest181rcu_preempt21:46:361
14127998145,12cyclictest181rcu_preempt21:46:361
14127998130,21cyclictest0-21swapper/123:15:281
1412799811,40cyclictest31-21ksoftirqd/122:53:561
14127998047,11cyclictest181rcu_preempt23:51:231
14135997958,15cyclictest32370-21apt-key00:01:112
14135997958,15cyclictest32370-21apt-key00:01:112
14127997949,13cyclictest181rcu_preempt20:34:111
14127997949,13cyclictest181rcu_preempt20:34:111
14127997947,12cyclictest181rcu_preempt23:10:591
14127997942,13cyclictest181rcu_preempt00:06:311
14127997938,12cyclictest0-21swapper/119:21:121
14127997929,11cyclictest181rcu_preempt23:36:321
1412799792,26cyclictest32294-21systemctl00:01:091
1412799792,26cyclictest32294-21systemctl00:01:091
1413599786,62cyclictest22636-21ssh21:56:292
14135997833,34cyclictest14619-21kworker/u19:2+nfsiod23:52:082
14135997823,38cyclictest9358-21cpuspeed00:16:262
14127997845,13cyclictest181rcu_preempt22:57:121
14127997844,11cyclictest181rcu_preempt21:51:531
14127997843,15cyclictest181rcu_preempt22:21:311
14135997762,10cyclictest14108-21cyclictest00:29:512
14135997762,10cyclictest14108-21cyclictest00:29:512
14135997753,18cyclictest31788-21latency_hist23:06:112
14135997753,18cyclictest31788-21latency_hist23:06:112
1413599773,36cyclictest8403-21fschecks_time22:26:322
14127997752,9cyclictest181rcu_preempt19:01:421
14127997752,9cyclictest181rcu_preempt19:01:421
14127997748,13cyclictest181rcu_preempt22:33:501
14127997747,16cyclictest181rcu_preempt19:57:341
14127997747,13cyclictest181rcu_preempt20:06:091
14127997745,13cyclictest181rcu_preempt22:26:381
14127997743,13cyclictest181rcu_preempt19:36:221
14127997743,13cyclictest181rcu_preempt19:36:221
14127997728,14cyclictest181rcu_preempt22:41:341
14135997652,18cyclictest25544-21/usr/sbin/munin22:01:362
14135997652,18cyclictest25544-21/usr/sbin/munin22:01:362
14127997652,8cyclictest181rcu_preempt23:48:581
14127997652,8cyclictest181rcu_preempt23:48:581
14127997650,11cyclictest181rcu_preempt21:36:521
14127997649,10cyclictest181rcu_preempt19:51:121
14127997647,8cyclictest181rcu_preempt20:41:091
14127997644,10cyclictest181rcu_preempt19:51:441
14127997639,20cyclictest14108-21cyclictest21:02:261
14127997625,33cyclictest0-21swapper/100:16:211
1412799762,38cyclictest31-21ksoftirqd/119:15:311
204702750,8sleep30-21swapper/322:46:333
14135997557,11cyclictest2759-21kworker/u18:2-nfsiod22:41:192
14135997552,15cyclictest28141-21netstat21:11:382
14135997522,25cyclictest21265-21diskmemload22:26:012
14127997554,15cyclictest14058-21processes21:41:431
14127997553,8cyclictest181rcu_preempt20:46:371
14127997553,8cyclictest181rcu_preempt20:46:371
14127997548,9cyclictest181rcu_preempt21:06:281
14127997545,13cyclictest181rcu_preempt20:42:411
14127997541,17cyclictest181rcu_preempt19:26:161
14127997535,12cyclictest181rcu_preempt23:41:471
14127997535,12cyclictest181rcu_preempt23:41:471
14135997454,15cyclictest352-21runrttasks21:01:502
14135997447,21cyclictest30535-21kworker/u17:2-xprtiod21:41:482
14127997452,9cyclictest181rcu_preempt20:06:241
14127997451,17cyclictest0-21swapper/120:21:131
14127997451,17cyclictest0-21swapper/120:21:131
14127997444,11cyclictest181rcu_preempt20:30:591
14127997443,13cyclictest181rcu_preempt21:32:381
1413599734,48cyclictest5474-21grep20:11:402
14135997317,24cyclictest0-21swapper/222:38:412
14135997317,24cyclictest0-21swapper/222:38:412
14127997350,8cyclictest181rcu_preempt19:21:201
14127997349,8cyclictest6473-21ssh00:11:291
1412799733,53cyclictest3034-21latency_hist00:06:111
14127997328,8cyclictest181rcu_preempt22:46:341
14127997324,12cyclictest0-21swapper/120:52:181
14135997255,10cyclictest0-21swapper/219:16:442
14135997217,18cyclictest21207-21sh22:46:492
14127997242,12cyclictest181rcu_preempt00:21:381
14127997220,27cyclictest6230-21modprobe20:16:091
14127997220,14cyclictest0-21swapper/122:40:181
14127997220,14cyclictest0-21swapper/122:40:181
14135997153,12cyclictest4639-21sendmail20:11:092
14135997144,18cyclictest0-21swapper/222:16:332
1413599713,51cyclictest15058-21apt-get20:41:262
1413599713,48cyclictest8654-21fschecks_time20:22:262
14135997116,22cyclictest26239-21latency_hist23:51:132
14135997116,21cyclictest0-21swapper/223:34:482
14135997111,48cyclictest4445-21fschecks_time21:26:332
14127997136,14cyclictest181rcu_preempt19:42:561
14135997053,9cyclictest0-21swapper/221:06:452
14135997050,14cyclictest11467-21latency_hist20:31:122
14135997048,17cyclictest30726-21cstates21:16:262
14135997046,13cyclictest0-21swapper/223:46:122
14135997046,13cyclictest0-21swapper/223:46:122
14135997024,9cyclictest0-21swapper/219:36:452
14135997024,9cyclictest0-21swapper/219:36:452
14135997020,22cyclictest14040-21sh00:23:502
14135997017,23cyclictest0-21swapper/222:13:492
14135997012,40cyclictest0-21swapper/220:57:062
14135997011,28cyclictest12831-21ssh22:33:312
14135997010,23cyclictest19795-21taskset23:39:532
14127997048,16cyclictest28228-21sh21:11:401
14135996952,10cyclictest0-21swapper/220:46:452
14135996952,10cyclictest0-21swapper/220:46:452
1413599693,23cyclictest26066-21/usr/sbin/munin22:56:192
14127996946,10cyclictest181rcu_preempt21:00:521
14127996937,12cyclictest181rcu_preempt23:26:471
1412799693,37cyclictest25618-21ssh22:01:251
1412799693,37cyclictest25618-21ssh22:01:251
14127996927,11cyclictest181rcu_preempt20:22:321
14135996851,10cyclictest0-21swapper/220:41:092
14135996846,16cyclictest14108-21cyclictest19:26:122
1413599684,47cyclictest31087-21fschecks_count19:51:302
14135996843,16cyclictest15912-21fschecks_time19:06:152
1413599683,46cyclictest7446-21grep20:16:442
1413599683,46cyclictest7446-21grep20:16:442
14127996836,10cyclictest181rcu_preempt22:14:451
14135996749,11cyclictest0-21swapper/221:21:202
14135996749,11cyclictest0-21swapper/221:21:202
14135996745,16cyclictest7255-21sh00:11:462
1413599674,49cyclictest29480-21fschecks_time19:46:312
1414299665,36cyclictest0-21swapper/300:01:233
14135996645,15cyclictest3786-21sh00:06:362
14135996641,15cyclictest39-21ksoftirqd/219:01:342
14135996641,15cyclictest39-21ksoftirqd/219:01:342
1413599662,46cyclictest1431-21latency_hist20:01:112
14135996545,14cyclictest21265-21diskmemload21:46:402
14135996545,14cyclictest21265-21diskmemload21:46:402
14135996544,14cyclictest29019-21ssh22:06:352
14135996539,19cyclictest10544-21sh21:36:302
14135996519,11cyclictest0-21swapper/219:41:472
14135996517,8cyclictest0-21swapper/220:01:382
14135996446,10cyclictest0-21swapper/220:56:092
1413599644,35cyclictest22780-21/usr/sbin/munin19:26:402
14135996442,16cyclictest0-21swapper/219:01:132
14135996418,10cyclictest0-21swapper/223:23:512
14135996340,15cyclictest0-21swapper/223:29:022
1413599632,42cyclictest352-21runrttasks19:36:122
1413599632,42cyclictest352-21runrttasks19:36:122
1414299626,50cyclictest0-21swapper/321:31:523
14142996217,31cyclictest0-21swapper/323:51:093
14142996217,31cyclictest0-21swapper/323:51:093
1413599623,37cyclictest352-21runrttasks23:06:162
1413599613,34cyclictest2979-21ssh23:11:332
1414299593,44cyclictest30622-21taskset21:16:243
1413599593,38cyclictest19462-21latency_hist19:16:122
1414299583,42cyclictest29063-21chrt19:46:113
1414299562,40cyclictest3693-21taskset00:06:343
1414299553,38cyclictest11117-21chrt22:31:213
1414299552,43cyclictest25838-21taskset19:36:113
1414299552,43cyclictest25838-21taskset19:36:113
1414299544,38cyclictest16554-21taskset23:34:103
1414299533,37cyclictest30637-21chrt19:51:093
1414299533,36cyclictest13436-21chrt00:22:073
1414299533,34cyclictest25849-21chrt22:01:313
1414299533,34cyclictest25849-21chrt22:01:313
1414299532,39cyclictest17868-21taskset19:11:123
1414299532,36cyclictest28922-21chrt23:01:123
1414299532,35cyclictest30409-21chrt21:16:123
1414299524,37cyclictest8633-21taskset23:21:263
1414299523,39cyclictest19788-21taskset22:46:083
1414299523,38cyclictest1899-21taskset20:01:303
1414299523,37cyclictest9040-21chrt00:16:113
1414299523,36cyclictest3609-21chrt23:11:453
1414299523,33cyclictest30966-21chrt19:51:263
1414299522,39cyclictest12451-21chrt21:39:343
225682510,7sleep3451rcuc/319:26:113
1414299513,36cyclictest17641-21taskset20:46:463
1414299513,36cyclictest17641-21taskset20:46:463
1414299513,35cyclictest23968-21chrt21:05:213
1414299513,35cyclictest19587-21chrt21:51:303
1414299513,34cyclictest12832-21chrt23:27:133
1414299512,33cyclictest6409-21chrt20:16:123
1414299504,33cyclictest0-21swapper/320:41:263
1414299503,36cyclictest24167-21chrt22:51:483
1414299503,35cyclictest4108-21chrt21:26:213
1414299503,34cyclictest31576-21chrt22:11:203
1414299503,33cyclictest32589-21taskset19:56:283
1414299494,28cyclictest0-21swapper/321:56:403
1414299493,37cyclictest10534-21chrt22:30:323
1414299493,34cyclictest25126-21chrt21:06:393
1414299493,34cyclictest15155-21chrt19:01:443
1414299493,34cyclictest15155-21chrt19:01:443
1414299493,34cyclictest14622-21chrt21:42:143
1414299493,33cyclictest3565-21chrt20:06:323
1414299492,38cyclictest27165-21taskset23:51:373
1414299492,37cyclictest6218-21chrt23:16:403
1414299492,37cyclictest6218-21chrt23:16:403
1414299492,37cyclictest22295-21taskset23:43:303
1414299492,37cyclictest22295-21taskset23:43:303
1414299492,35cyclictest18171-21taskset23:36:373
1414299483,34cyclictest29198-21chrt23:01:283
1414299483,34cyclictest29198-21chrt23:01:283
1414299483,33cyclictest12364-21chrt20:31:433
1414299483,33cyclictest12364-21chrt20:31:433
1414299483,32cyclictest9874-21chrt20:26:123
1414299483,31cyclictest14243-21chrt22:36:283
1414299483,31cyclictest14243-21chrt22:36:283
1414299482,37cyclictest1548-21taskset21:21:353
1414299482,37cyclictest1548-21taskset21:21:353
1414299482,35cyclictest2315-21taskset23:11:083
1414299482,32cyclictest20339-21chrt19:16:433
1414299474,30cyclictest15648-21chrt00:26:373
1414299474,30cyclictest15648-21chrt00:26:373
1414299473,34cyclictest31423-21chrt22:11:123
1414299472,36cyclictest18601-21chrt19:11:413
1414299472,34cyclictest2482-21chrt22:16:313
139612479,20sleep30-21swapper/319:01:093
1414299463,33cyclictest7455-21chrt22:25:183
1414299463,30cyclictest17144-21chrt21:46:443
1414299463,30cyclictest17144-21chrt21:46:443
1414299463,29cyclictest20043-21chrt20:56:263
14142994532,9cyclictest451rcuc/320:29:543
14142994531,7cyclictest451rcuc/300:19:523
1414299442,31cyclictest14687-21chrt20:41:113
1414299442,30cyclictest31965-21chrt00:00:243
1414299442,30cyclictest31965-21chrt00:00:243
14142994027,9cyclictest451rcuc/319:36:463
14142994027,9cyclictest451rcuc/319:36:463
1414299402,28cyclictest18564-21taskset20:51:323
14142994018,15cyclictest229442sleep319:26:283
1414299393,25cyclictest7755-21chrt20:20:093
1414299393,25cyclictest7755-21chrt20:20:093
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional