You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-16 - 06:53
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Thu Apr 16, 2026 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2544322280,19sleep10-21swapper/118:59:361
265569910284,11cyclictest14528-21/usr/sbin/munin00:08:371
265569910284,11cyclictest14528-21/usr/sbin/munin00:08:371
26554999543,42cyclictest2115-21latency_hist21:58:090
26554999543,42cyclictest2115-21latency_hist21:58:090
26556999043,16cyclictest8468-21latency_hist23:03:091
257062899,55sleep025730-21proc_pri18:59:480
26560998866,15cyclictest26542-21cyclictest23:03:092
26560998864,13cyclictest0-21swapper/220:33:262
26556998763,17cyclictest352-21runrttasks22:33:301
26556998763,17cyclictest352-21runrttasks22:33:301
26554998768,13cyclictest277-21dbus-daemon19:08:070
26560998659,7cyclictest181rcu_preempt20:55:252
26560998659,7cyclictest181rcu_preempt20:55:252
26560998648,10cyclictest181rcu_preempt20:43:052
2655699869,46cyclictest31-21ksoftirqd/121:36:011
26556998649,8cyclictest181rcu_preempt20:28:391
26556998615,42cyclictest31-21ksoftirqd/121:42:091
26556998615,42cyclictest31-21ksoftirqd/121:42:091
2655699856,43cyclictest31-21ksoftirqd/119:35:581
2655699856,43cyclictest31-21ksoftirqd/119:35:581
26556998551,29cyclictest3453-21kworker/u19:3+events_unbound21:13:061
2655699853,59cyclictest29002-21apt-get19:09:091
2655699853,59cyclictest29002-21apt-get19:09:091
26554998569,10cyclictest2067-21munin-run22:53:050
26554998563,16cyclictest544-21cron19:43:050
26560998461,11cyclictest0-21swapper/222:13:092
26556998441,15cyclictest3030-21cpu19:28:221
26556998441,15cyclictest3030-21cpu19:28:221
26554998436,19cyclictest14749-21ssh00:08:240
26554998436,19cyclictest14749-21ssh00:08:240
26554998338,22cyclictest14909-21ssh23:13:310
26554998338,22cyclictest14909-21ssh23:13:310
26554998328,21cyclictest0-21swapper/022:35:470
26554998328,21cyclictest0-21swapper/022:35:470
26560998263,10cyclictest0-21swapper/223:28:262
2656099822,54cyclictest29002-21apt-get19:09:082
2656099822,54cyclictest29002-21apt-get19:09:082
26556998261,14cyclictest26542-21cyclictest19:08:071
26556998256,7cyclictest181rcu_preempt19:38:341
26556998245,7cyclictest181rcu_preempt23:56:421
26556998244,7cyclictest0-21swapper/123:47:501
26556998219,31cyclictest31-21ksoftirqd/122:09:261
26554998229,22cyclictest18060-21ssh23:18:360
26554998229,16cyclictest14639-21ssh22:18:320
26554998222,20cyclictest4663-21ssh23:51:050
26560998163,9cyclictest0-21swapper/219:18:232
26560998161,10cyclictest0-21swapper/223:48:212
26560998144,11cyclictest181rcu_preempt19:07:312
26556998160,14cyclictest5080-21apt-key22:58:071
26556998157,16cyclictest14581-21ssh23:13:221
26556998157,16cyclictest14581-21ssh23:13:221
26556998154,8cyclictest181rcu_preempt23:58:241
26556998147,9cyclictest181rcu_preempt20:22:301
26556998147,9cyclictest181rcu_preempt20:22:301
26556998145,8cyclictest181rcu_preempt23:23:411
26556998145,8cyclictest181rcu_preempt23:23:411
26556998142,12cyclictest0-21swapper/121:54:041
26556998142,12cyclictest0-21swapper/121:54:041
26554998158,16cyclictest846-21grep23:43:430
26554998128,24cyclictest1898-21ssh21:03:250
26560998063,11cyclictest2952-21systemd-journal22:47:482
26560998063,11cyclictest2952-21systemd-journal22:47:482
26560998062,9cyclictest0-21swapper/222:33:092
26560998061,10cyclictest0-21swapper/222:53:422
26560998010,39cyclictest39-21ksoftirqd/200:22:092
26556998058,15cyclictest26542-21cyclictest20:53:081
26556998056,8cyclictest181rcu_preempt23:48:291
26556998041,12cyclictest2908-21ssh21:05:091
26556998040,7cyclictest181rcu_preempt20:04:581
26556998026,7cyclictest0-21swapper/122:51:161
26554998065,9cyclictest17186-21ssh22:23:230
26554998056,18cyclictest2631-21fschecks_count22:53:280
26554998029,23cyclictest20689-21ssh00:18:240
26554998029,22cyclictest28398-21ssh22:42:290
26554998027,23cyclictest30062-21ssh23:38:390
26554998025,46cyclictest0-21swapper/021:03:070
2655499800,38cyclictest18154-21ssh21:29:180
2655499800,38cyclictest18154-21ssh21:29:180
26560997963,9cyclictest0-21swapper/222:03:302
26560997958,13cyclictest0-21swapper/223:08:312
26556997960,13cyclictest19676-21snmpd23:33:211
26556997956,15cyclictest31849-21kworker/u17:1+ext4-rsv-conversion20:09:231
26556997949,15cyclictest181rcu_preempt22:15:311
26556997949,15cyclictest181rcu_preempt22:15:311
26556997944,11cyclictest3177-21timerwakeupswit21:58:421
26556997942,8cyclictest0-21swapper/122:23:311
26556997911,37cyclictest31-21ksoftirqd/121:46:401
26554997954,17cyclictest0-21swapper/022:03:190
26554997934,37cyclictest29163-21ssh00:33:080
26554997928,17cyclictest13038-21ssh23:09:560
26554997925,24cyclictest20549-21ssh22:28:320
26554997925,23cyclictest8475-21ssh21:13:330
26554997919,32cyclictest0-21swapper/021:26:140
2564027913,45sleep30-21swapper/318:59:433
26560997849,7cyclictest181rcu_preempt23:56:592
26560997848,8cyclictest181rcu_preempt00:25:422
26560997815,38cyclictest39-21ksoftirqd/220:28:172
26560997814,34cyclictest39-21ksoftirqd/221:44:092
26556997861,11cyclictest28882-21latency_hist20:48:091
26556997845,9cyclictest181rcu_preempt20:26:031
26556997845,9cyclictest181rcu_preempt20:26:031
26556997843,16cyclictest0-21swapper/123:42:021
26554997840,28cyclictest27521-21/usr/sbin/munin20:43:270
26554997838,27cyclictest22128-21ssh23:25:320
26554997838,27cyclictest22128-21ssh23:25:320
26554997836,21cyclictest1298-21diskmemload22:11:580
26554997834,22cyclictest20583-21runrttasks20:20:570
26554997834,22cyclictest20583-21runrttasks20:20:570
26554997829,21cyclictest23875-21runrttasks20:31:210
26554997828,23cyclictest0-21swapper/023:33:140
26554997824,29cyclictest0-21swapper/000:07:090
26554997824,29cyclictest0-21swapper/000:07:090
26560997762,8cyclictest0-21swapper/221:18:332
26560997762,8cyclictest0-21swapper/221:18:332
26560997762,8cyclictest0-21swapper/200:28:392
26560997744,8cyclictest181rcu_preempt20:59:502
26560997742,17cyclictest31347-21kworker/u17:0-rpciod00:10:092
26560997742,17cyclictest31347-21kworker/u17:0-rpciod00:10:092
26560997737,10cyclictest27512-21kworker/u18:0-rpciod21:40:562
26560997737,10cyclictest27512-21kworker/u18:0-rpciod21:40:562
26560997729,11cyclictest181rcu_preempt00:17:392
26556997753,7cyclictest181rcu_preempt22:32:311
26556997748,9cyclictest181rcu_preempt00:04:111
26556997748,9cyclictest181rcu_preempt00:04:111
26556997743,9cyclictest181rcu_preempt23:33:011
26556997742,8cyclictest181rcu_preempt00:21:151
26556997740,20cyclictest0-21swapper/120:38:371
26556997740,10cyclictest29647-21/usr/sbin/munin22:43:371
26556997740,10cyclictest29647-21/usr/sbin/munin22:43:371
26556997738,19cyclictest26542-21cyclictest23:21:001
26556997737,8cyclictest181rcu_preempt20:56:531
26556997737,8cyclictest181rcu_preempt20:56:531
26556997737,18cyclictest0-21swapper/121:13:281
26554997745,25cyclictest0-21swapper/023:53:250
26554997737,29cyclictest24528-21ssh23:28:570
26554997730,19cyclictest2587-21runrttasks19:26:100
26554997729,36cyclictest4650-21apt-get19:33:190
26554997729,36cyclictest4650-21apt-get19:33:190
26554997726,24cyclictest2996-21/usr/sbin/munin19:28:310
26554997726,24cyclictest2996-21/usr/sbin/munin19:28:310
26554997726,23cyclictest5230-21ssh21:08:400
26554997726,22cyclictest31108-21ssh21:52:200
26554997724,23cyclictest30322-21ssh22:44:310
26554997724,23cyclictest30322-21ssh22:44:310
26560997660,8cyclictest0-21swapper/221:08:422
26560997656,15cyclictest15665-21idleruntime-cro20:08:062
26560997631,20cyclictest39-21ksoftirqd/219:16:382
26560997631,20cyclictest39-21ksoftirqd/219:16:382
26556997657,11cyclictest14233-21cstates22:18:211
26556997648,7cyclictest181rcu_preempt19:46:381
26556997644,7cyclictest181rcu_preempt22:42:501
26556997643,9cyclictest181rcu_preempt19:14:311
26556997643,9cyclictest181rcu_preempt19:14:311
26556997642,9cyclictest181rcu_preempt19:51:371
26556997641,9cyclictest26542-21cyclictest00:16:011
26556997639,17cyclictest0-21swapper/121:23:161
26554997642,26cyclictest11408-21latency_hist23:08:080
26554997639,9cyclictest12839-21ssh22:15:220
26554997639,9cyclictest12839-21ssh22:15:220
26554997629,18cyclictest12885-21kworker/u17:1-rpciod21:38:510
26554997629,18cyclictest12885-21kworker/u17:1-rpciod21:38:510
26554997627,21cyclictest7193-21ssh23:00:080
26554997626,22cyclictest10675-21ssh00:01:170
26554997622,22cyclictest12238-21ssh21:19:190
26554997622,22cyclictest12238-21ssh21:19:190
26560997557,8cyclictest0-21swapper/223:38:412
26560997557,8cyclictest0-21swapper/223:38:072
26560997556,9cyclictest0-21swapper/220:18:422
26560997556,9cyclictest0-21swapper/220:18:422
26560997549,8cyclictest181rcu_preempt20:43:302
26560997539,8cyclictest181rcu_preempt19:28:282
26560997539,8cyclictest181rcu_preempt19:28:282
2656099752,63cyclictest15175-21kworker/u19:1+flush-179:020:08:212
2656099752,38cyclictest39-21ksoftirqd/222:48:362
2656099752,35cyclictest39-21ksoftirqd/219:25:572
2655699757,29cyclictest31-21ksoftirqd/119:55:501
26556997550,14cyclictest0-21swapper/120:03:091
2655699753,25cyclictest31-21ksoftirqd/121:21:541
2655699753,25cyclictest31-21ksoftirqd/121:21:541
26554997559,10cyclictest2228-21ssh21:58:160
26554997541,24cyclictest31249-21/usr/sbin/munin19:13:370
26554997541,24cyclictest31249-21/usr/sbin/munin19:13:370
26554997536,17cyclictest30329-21taskset19:12:580
26554997536,17cyclictest30329-21taskset19:12:580
26554997535,19cyclictest951-21runrttasks19:20:480
26554997522,25cyclictest27820-21ssh21:46:240
26554997522,20cyclictest25113-21ssh00:25:160
26560997461,8cyclictest14107-21forks21:23:272
26560997453,15cyclictest20119-21sh23:23:062
26560997450,14cyclictest1886-21ssh21:03:232
26560997416,31cyclictest39-21ksoftirqd/223:44:162
26556997445,9cyclictest181rcu_preempt23:03:441
26556997440,10cyclictest5881-21ps22:03:381
26556997439,7cyclictest0-21swapper/100:26:561
26556997415,28cyclictest31-21ksoftirqd/123:08:341
26554997432,28cyclictest365-21ntpd20:03:340
26554997427,26cyclictest19093-21ssh00:14:590
26554997425,21cyclictest21173-21ssh21:34:230
26560997331,12cyclictest181rcu_preempt23:05:462
2656099732,25cyclictest39-21ksoftirqd/221:28:392
2656099732,25cyclictest39-21ksoftirqd/221:28:392
26556997345,12cyclictest181rcu_preempt20:13:401
26556997343,10cyclictest0-21swapper/121:51:571
26556997336,8cyclictest0-21swapper/119:18:321
26556997327,18cyclictest0-21swapper/121:30:571
26556997327,18cyclictest0-21swapper/121:30:571
26554997327,18cyclictest0-21swapper/019:45:500
26554997326,20cyclictest19676-21snmpd20:58:070
26554997326,20cyclictest19676-21snmpd20:58:070
26560997255,7cyclictest0-21swapper/222:23:402
26560997247,12cyclictest181rcu_preempt21:13:432
26560997244,8cyclictest181rcu_preempt19:51:422
26560997232,12cyclictest181rcu_preempt19:34:222
26560997232,12cyclictest181rcu_preempt19:34:222
26560997224,21cyclictest0-21swapper/222:42:452
26560997217,34cyclictest39-21ksoftirqd/221:33:432
26556997223,20cyclictest0-21swapper/100:28:231
26554997254,12cyclictest352-21runrttasks20:33:330
26554997234,18cyclictest12188-21runrttasks19:54:460
26554997225,21cyclictest10684-21sshd19:51:380
296222710,6sleep30-21swapper/320:48:343
26560997155,10cyclictest15173-21proc_pri23:13:382
26560997155,10cyclictest15173-21proc_pri23:13:382
26560997138,9cyclictest181rcu_preempt22:37:222
26560997138,9cyclictest181rcu_preempt22:37:222
26560997135,7cyclictest181rcu_preempt23:23:562
26560997135,7cyclictest181rcu_preempt23:23:562
26560997134,10cyclictest1298-21diskmemload00:04:232
26560997134,10cyclictest1298-21diskmemload00:04:232
2656099711,36cyclictest39-21ksoftirqd/223:58:102
2656099711,26cyclictest39-21ksoftirqd/222:13:172
2656099711,26cyclictest39-21ksoftirqd/222:13:172
26556997134,17cyclictest0-21swapper/120:38:081
26556997115,34cyclictest31-21ksoftirqd/120:58:281
2655499714,38cyclictest12924-21fschecks_count19:58:270
26554997139,20cyclictest19335-21sed20:18:090
26554997126,24cyclictest22185-21runrttasks20:25:540
26554997126,24cyclictest22185-21runrttasks20:25:540
2656099707,28cyclictest1298-21diskmemload21:53:102
26560997011,29cyclictest39-21ksoftirqd/222:01:292
26554997044,19cyclictest16325-21/usr/sbin/munin20:09:220
2655499703,35cyclictest26112-21fschecks_count20:38:260
250702709,37sleep224567-21apt-get18:58:192
26560996930,8cyclictest181rcu_preempt20:50:422
135392690,7sleep30-21swapper/319:58:413
26560996843,7cyclictest181rcu_preempt22:21:142
26560996839,9cyclictest181rcu_preempt19:43:352
26556996848,10cyclictest0-21swapper/119:23:391
2656099673,47cyclictest352-21runrttasks20:25:012
2656099673,47cyclictest352-21runrttasks20:25:012
2656099672,26cyclictest39-21ksoftirqd/219:58:372
26560996713,21cyclictest39-21ksoftirqd/221:57:522
26560996713,21cyclictest39-21ksoftirqd/221:57:522
26554996740,21cyclictest29967-21timerwakeupswit20:48:420
214222670,5sleep30-21swapper/300:18:413
26560996638,8cyclictest181rcu_preempt20:13:242
26560996635,10cyclictest181rcu_preempt19:53:172
2656099662,22cyclictest39-21ksoftirqd/219:38:332
2656599643,46cyclictest29075-21taskset19:09:123
2656599643,46cyclictest29075-21taskset19:09:123
237862620,2sleep30-21swapper/320:30:113
2656599583,39cyclictest6964-21chrt22:05:353
2656599562,41cyclictest3948-21taskset21:06:493
2656599554,37cyclictest24443-21taskset23:28:433
2656599542,39cyclictest16266-21taskset23:15:373
2656599542,39cyclictest16266-21taskset23:15:373
2656599523,35cyclictest25444-21chrt21:43:053
2656599523,35cyclictest25444-21chrt21:43:053
2656599522,33cyclictest8415-21taskset23:03:083
2656599516,28cyclictest0-21swapper/320:08:443
2656599513,33cyclictest25945-21chrt20:38:213
2656599512,36cyclictest8878-21taskset22:08:403
26565995035,10cyclictest451rcuc/319:40:113
2656599503,34cyclictest7385-21chrt23:55:183
2656599503,34cyclictest31249-21taskset22:47:113
2656599503,34cyclictest31249-21taskset22:47:113
2656599503,34cyclictest1469-21chrt22:51:253
26565995029,14cyclictest451rcuc/300:30:223
2656599493,32cyclictest26438-21chrt23:33:213
2656599493,32cyclictest20390-21chrt23:23:153
2656599493,32cyclictest20390-21chrt23:23:153
2656599493,31cyclictest11861-21chrt22:13:383
2656599493,31cyclictest11861-21chrt22:13:383
2656599492,35cyclictest4513-21taskset22:56:363
2656599492,35cyclictest15567-21chrt21:25:213
2656599492,33cyclictest1899-21chrt19:23:333
2656599492,31cyclictest20425-21chrt00:18:093
26565994913,24cyclictest0-21swapper/321:08:233
2656599484,29cyclictest8690-21taskset23:58:183
2656599483,34cyclictest24859-21chrt00:24:333
2656599483,32cyclictest21747-21chrt21:36:003
2656599483,28cyclictest1112-21taskset21:01:013
26565994829,13cyclictest451rcuc/320:20:243
26565994829,13cyclictest451rcuc/320:20:243
2656599482,36cyclictest11640-21taskset00:03:173
2656599482,36cyclictest11640-21taskset00:03:173
2656599482,35cyclictest923-21taskset21:55:143
2656599482,35cyclictest923-21taskset21:55:143
2656599482,32cyclictest23078-21taskset22:33:213
2656599482,32cyclictest23078-21taskset22:33:213
2656599473,35cyclictest5895-21taskset19:37:023
2656599473,35cyclictest5895-21taskset19:37:023
2656599473,34cyclictest19031-21chrt20:16:423
2656599473,32cyclictest4563-21taskset19:33:093
2656599473,32cyclictest4563-21taskset19:33:093
26565994733,10cyclictest451rcuc/322:27:053
2656599473,30cyclictest507-21chrt23:43:393
2656599472,34cyclictest5060-21chrt22:03:083
2656599472,33cyclictest21024-21chrt22:28:423
2656599472,32cyclictest18429-21chrt23:18:413
2656599472,32cyclictest12684-21chrt21:20:353
2656599472,32cyclictest12684-21chrt21:20:353
2656599463,29cyclictest9299-21chrt19:48:083
2656599463,29cyclictest8936-21chrt23:03:293
2656599463,29cyclictest27188-21chrt22:39:003
26565994632,10cyclictest451rcuc/320:04:373
2656599462,32cyclictest30572-21chrt21:50:483
2656599462,29cyclictest18857-21chrt21:31:173
2656599462,29cyclictest18857-21chrt21:31:173
2656599453,33cyclictest3976-21taskset23:49:063
2656599453,31cyclictest28488-21taskset19:08:103
26565994528,10cyclictest451rcuc/320:26:153
26565994528,10cyclictest451rcuc/320:26:153
2656599452,29cyclictest31346-21taskset23:41:283
2656599443,31cyclictest9247-21chrt21:14:153
26565994432,8cyclictest451rcuc/320:37:533
26565994427,12cyclictest451rcuc/319:21:033
2656599442,32cyclictest11815-21chrt23:08:283
2656599442,30cyclictest28398-21chrt21:48:043
2656599442,29cyclictest12534-21chrt19:58:083
2656599434,29cyclictest14583-21chrt00:08:193
2656599434,29cyclictest14583-21chrt00:08:193
26565994227,9cyclictest451rcuc/320:54:423
26565994227,9cyclictest451rcuc/320:54:423
2656599413,28cyclictest31149-21taskset19:13:333
2656599413,28cyclictest31149-21taskset19:13:333
2656599413,28cyclictest27683-21chrt20:43:233
2656599413,27cyclictest15663-21taskset22:19:593
2656599403,27cyclictest9446-21taskset19:48:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional