You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-14 - 03:53
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Sun Jun 14, 2026 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12891212113,74sleep112918-21/usr/sbin/munin18:54:491
13657210916,55sleep013667-21modprobe18:59:080
13834999562,8cyclictest181rcu_preempt21:35:442
13834999562,8cyclictest181rcu_preempt21:35:442
13834999459,8cyclictest181rcu_preempt21:34:042
13816999460,8cyclictest181rcu_preempt22:55:260
13834999362,21cyclictest2340-21awk00:04:072
13834999157,7cyclictest181rcu_preempt22:07:272
13834999154,8cyclictest181rcu_preempt21:40:012
13825999172,13cyclictest3574-21ssh00:04:441
13825999172,13cyclictest3574-21ssh00:04:441
13816999168,15cyclictest7931-21latency_hist20:19:110
13834999052,8cyclictest181rcu_preempt00:05:412
13834999052,8cyclictest181rcu_preempt00:05:412
13825999056,7cyclictest181rcu_preempt19:44:081
13825999056,7cyclictest181rcu_preempt19:44:081
13834998960,8cyclictest181rcu_preempt21:10:062
13834998951,8cyclictest181rcu_preempt23:28:562
13834998950,10cyclictest181rcu_preempt21:24:352
13825998968,14cyclictest1935-21kernelversion19:59:331
13825998968,14cyclictest1935-21kernelversion19:59:331
13834998856,9cyclictest181rcu_preempt00:19:562
13834998856,8cyclictest181rcu_preempt20:54:382
13834998847,7cyclictest0-21swapper/221:20:052
13840998712,61cyclictest0-21swapper/300:14:293
13840998712,61cyclictest0-21swapper/300:14:293
13834998753,8cyclictest181rcu_preempt20:49:432
13834998753,8cyclictest181rcu_preempt20:49:432
13825998766,14cyclictest8118-21latency_hist23:19:111
13825998745,12cyclictest181rcu_preempt20:58:361
13834998669,11cyclictest298-21in:imuxsock19:39:082
13834998669,11cyclictest298-21in:imuxsock19:39:082
13834998650,8cyclictest181rcu_preempt23:49:382
13834998649,9cyclictest181rcu_preempt19:19:302
13834998642,13cyclictest0-21swapper/223:05:312
13834998642,13cyclictest0-21swapper/223:05:312
13834998641,21cyclictest0-21swapper/222:42:282
13816998664,13cyclictest22398-21ssh22:49:110
1381699863,37cyclictest15-21ksoftirqd/023:17:330
13834998561,17cyclictest13800-21cyclictest20:24:432
13834998561,17cyclictest13800-21cyclictest20:24:432
13834998555,20cyclictest4150-21apt-key22:19:122
13834998555,20cyclictest4150-21apt-key22:19:122
13834998552,8cyclictest181rcu_preempt00:28:382
13834998549,7cyclictest181rcu_preempt23:34:582
13825998556,8cyclictest181rcu_preempt22:20:361
13825998554,7cyclictest181rcu_preempt21:44:331
13816998565,14cyclictest13170-21sh21:40:220
13834998444,11cyclictest25732-21unin-run19:34:092
13825998460,13cyclictest0-21swapper/119:09:301
13825998460,13cyclictest0-21swapper/119:09:301
13816998465,13cyclictest30523-21munin-plugin-st19:49:070
13816998463,11cyclictest0-21swapper/000:15:130
13816998463,11cyclictest0-21swapper/000:15:130
1384099836,43cyclictest0-21swapper/323:04:293
1384099836,43cyclictest0-21swapper/323:04:293
1384099833,34cyclictest31306-21taskset22:09:393
13840998314,35cyclictest0-21swapper/322:34:223
13834998361,12cyclictest0-21swapper/200:09:422
13834998358,13cyclictest0-21swapper/222:44:432
13834998353,20cyclictest24053-21cron19:29:072
13825998365,11cyclictest15672-21latency_hist19:04:111
13825998365,11cyclictest15672-21latency_hist19:04:111
13825998359,8cyclictest181rcu_preempt00:09:421
13816998362,11cyclictest0-21swapper/021:36:060
13816998362,11cyclictest0-21swapper/021:36:060
13816998356,18cyclictest19630-21threads21:49:440
13834998260,12cyclictest0-21swapper/219:06:012
13834998260,12cyclictest0-21swapper/219:06:012
13834998257,8cyclictest181rcu_preempt22:24:182
13834998257,19cyclictest31394-21apt-key23:04:122
13834998257,19cyclictest31394-21apt-key23:04:122
13834998253,8cyclictest181rcu_preempt21:14:252
13834998253,8cyclictest181rcu_preempt21:14:252
13834998250,8cyclictest181rcu_preempt20:14:112
13825998260,12cyclictest0-21swapper/119:24:441
13825998255,9cyclictest181rcu_preempt20:24:251
13825998255,9cyclictest181rcu_preempt20:24:251
13825998254,8cyclictest181rcu_preempt21:04:111
13825998254,8cyclictest181rcu_preempt21:04:111
13825998251,7cyclictest181rcu_preempt23:21:251
13825998251,7cyclictest181rcu_preempt20:39:431
13825998251,10cyclictest181rcu_preempt22:32:541
13825998246,22cyclictest13619-21ssh22:34:311
13816998261,11cyclictest0-21swapper/019:19:410
13816998260,16cyclictest31208-21apt-key23:04:100
13816998260,16cyclictest31208-21apt-key23:04:100
13816998257,18cyclictest19337-21latency_hist22:44:100
13816998251,22cyclictest0-21swapper/023:39:310
13816998241,11cyclictest23825-21sh21:57:340
13834998161,10cyclictest0-21swapper/221:04:502
13834998161,10cyclictest0-21swapper/221:04:502
13834998151,7cyclictest181rcu_preempt23:49:072
13834998150,7cyclictest181rcu_preempt21:00:042
13834998150,7cyclictest181rcu_preempt21:00:042
13825998158,13cyclictest0-21swapper/121:49:251
13816998159,15cyclictest413-21systemd-run21:19:090
13816998159,15cyclictest413-21systemd-run21:19:090
1384099803,29cyclictest25759-21chrt21:59:423
13834998058,12cyclictest0-21swapper/222:54:402
13834998054,7cyclictest181rcu_preempt22:53:132
13834998052,9cyclictest181rcu_preempt20:39:452
13834998036,17cyclictest0-21swapper/200:15:002
13834998036,17cyclictest0-21swapper/200:15:002
13825998064,8cyclictest0-21swapper/121:54:341
13825998060,11cyclictest0-21swapper/123:38:291
13825998052,11cyclictest181rcu_preempt00:04:021
1382599802,28cyclictest31-21ksoftirqd/123:08:581
1382599802,28cyclictest31-21ksoftirqd/123:08:581
13816998066,10cyclictest13800-21cyclictest22:14:090
13816998065,10cyclictest13800-21cyclictest00:09:300
13816998060,10cyclictest0-21swapper/022:24:110
13816998059,11cyclictest0-21swapper/000:24:430
13834997959,11cyclictest0-21swapper/219:54:442
13834997958,12cyclictest0-21swapper/221:54:472
13834997956,15cyclictest29668-21ntp_kernel_pll_19:44:362
13834997950,8cyclictest181rcu_preempt19:39:262
13834997950,8cyclictest181rcu_preempt19:39:262
13834997913,26cyclictest39-21ksoftirqd/220:29:502
13825997963,9cyclictest0-21swapper/123:39:261
13825997952,17cyclictest0-21swapper/120:19:401
13816997964,8cyclictest0-21swapper/021:24:320
13816997958,10cyclictest0-21swapper/022:35:070
13816997957,11cyclictest0-21swapper/022:29:460
13816997953,17cyclictest17556-21latency_hist20:49:100
13816997952,18cyclictest0-21swapper/021:04:490
13816997952,18cyclictest0-21swapper/021:04:490
1381699793,28cyclictest15-21ksoftirqd/023:24:460
1384099784,25cyclictest19990-21taskset23:39:103
13834997865,9cyclictest13800-21cyclictest20:24:112
13834997841,9cyclictest4367-21kworker/u17:2-rpciod23:56:022
13834997841,9cyclictest4367-21kworker/u17:2-rpciod23:56:022
13825997861,11cyclictest26458-21ntp_kernel_pll_19:34:361
13825997861,11cyclictest26458-21ntp_kernel_pll_19:34:361
13825997842,11cyclictest30160-21vmstat19:44:461
1381699788,38cyclictest15-21ksoftirqd/019:35:160
1381699788,38cyclictest15-21ksoftirqd/019:35:160
13816997858,11cyclictest0-21swapper/020:49:300
13816997858,11cyclictest0-21swapper/020:49:300
13816997856,15cyclictest0-21swapper/023:49:450
13816997852,18cyclictest5359-21sh20:09:390
13840997721,48cyclictest0-21swapper/321:09:333
13834997751,7cyclictest181rcu_preempt22:35:452
13834997742,17cyclictest0-21swapper/222:09:152
13834997715,21cyclictest39-21ksoftirqd/223:29:322
13825997758,13cyclictest22264-21latency_hist22:49:101
13825997757,14cyclictest28536-21sh21:10:501
13825997756,15cyclictest26052-21sh22:54:371
1381699776,28cyclictest15-21ksoftirqd/022:28:400
13816997762,8cyclictest0-21swapper/019:24:450
13816997758,7cyclictest181rcu_preempt23:54:480
13816997758,7cyclictest181rcu_preempt23:54:480
13816997723,29cyclictest0-21swapper/000:04:210
13816997723,29cyclictest0-21swapper/000:04:210
13834997661,8cyclictest0-21swapper/221:59:262
13834997660,9cyclictest0-21swapper/223:09:292
13834997655,12cyclictest0-21swapper/219:59:242
13834997655,12cyclictest0-21swapper/219:59:242
13834997654,15cyclictest0-21swapper/219:14:432
13834997651,10cyclictest181rcu_preempt19:50:442
13834997631,12cyclictest0-21swapper/220:44:432
13825997660,11cyclictest0-21swapper/123:09:341
13825997660,10cyclictest0-21swapper/121:29:461
13825997657,9cyclictest0-21swapper/120:54:101
13825997657,9cyclictest0-21swapper/120:54:101
13825997657,11cyclictest0-21swapper/122:04:361
13825997633,9cyclictest181rcu_preempt19:04:141
13825997633,9cyclictest181rcu_preempt19:04:141
13816997652,18cyclictest23462-21fschecks_time23:44:300
13816997649,8cyclictest0-21swapper/022:04:300
1381699764,35cyclictest15-21ksoftirqd/019:10:410
1381699764,35cyclictest15-21ksoftirqd/019:10:410
13816997637,7cyclictest181rcu_preempt19:42:460
13816997637,7cyclictest181rcu_preempt19:42:460
13834997553,12cyclictest0-21swapper/222:20:452
13834997532,19cyclictest16261-21ssh21:44:362
1382599758,24cyclictest31-21ksoftirqd/119:24:051
13825997556,13cyclictest28400-21/usr/sbin/munin22:59:291
13825997556,13cyclictest28400-21/usr/sbin/munin22:59:291
13825997553,14cyclictest0-21swapper/121:24:111
13825997552,11cyclictest0-21swapper/123:44:421
13825997549,8cyclictest181rcu_preempt22:03:201
13816997555,10cyclictest0-21swapper/023:59:460
13816997538,18cyclictest352-21runrttasks23:35:060
1384099742,34cyclictest7887-21chrt20:19:103
13834997455,9cyclictest0-21swapper/221:54:112
13834997451,9cyclictest181rcu_preempt20:14:442
13834997417,24cyclictest39-21ksoftirqd/219:13:412
13834997417,24cyclictest39-21ksoftirqd/219:13:412
13825997456,9cyclictest0-21swapper/122:49:231
13825997455,13cyclictest28596-21sh23:53:261
13825997454,10cyclictest0-21swapper/122:39:381
13825997453,7cyclictest181rcu_preempt22:27:571
13825997452,10cyclictest181rcu_preempt21:07:081
13825997452,10cyclictest181rcu_preempt21:07:081
13825997436,10cyclictest15527-21ssh21:44:101
13825997431,8cyclictest181rcu_preempt20:08:581
13816997453,14cyclictest0-21swapper/018:59:330
13816997453,14cyclictest0-21swapper/018:59:330
13816997437,18cyclictest11815-21ntp_states20:29:390
1384099733,27cyclictest13458-21taskset22:34:123
13834997353,14cyclictest13398-21latency_hist22:34:112
13834997352,11cyclictest0-21swapper/220:09:102
1383499732,33cyclictest39-21ksoftirqd/219:04:082
1383499732,33cyclictest39-21ksoftirqd/219:04:082
13825997354,7cyclictest181rcu_preempt22:14:171
13825997354,7cyclictest181rcu_preempt22:14:171
13825997353,10cyclictest0-21swapper/120:39:101
13825997353,10cyclictest0-21swapper/100:14:571
13825997353,10cyclictest0-21swapper/100:14:571
13816997344,8cyclictest181rcu_preempt20:21:220
13816997335,18cyclictest24931-21sh22:53:330
1381699733,29cyclictest15-21ksoftirqd/023:14:080
13816997330,18cyclictest0-21swapper/021:59:490
1384099728,37cyclictest0-21swapper/322:19:283
13840997217,11cyclictest0-21swapper/323:49:083
13840997215,46cyclictest0-21swapper/323:19:243
13834997255,12cyclictest0-21swapper/223:20:222
13834997254,11cyclictest0-21swapper/223:17:382
13825997255,10cyclictest0-21swapper/119:54:441
13825997249,10cyclictest181rcu_preempt23:25:541
1382599723,27cyclictest31-21ksoftirqd/100:25:531
1382599722,26cyclictest31-21ksoftirqd/121:17:441
1382599722,26cyclictest31-21ksoftirqd/121:17:441
13816997257,10cyclictest13800-21cyclictest21:11:000
13816997249,11cyclictest181rcu_preempt19:49:450
1246227217,31sleep20-21swapper/218:54:292
109832720,3sleep0211rcuc/023:24:070
1384099713,27cyclictest22155-21taskset23:42:063
13825997156,8cyclictest0-21swapper/122:14:101
13825997155,9cyclictest0-21swapper/123:29:391
13825997151,10cyclictest0-21swapper/121:29:111
13825997151,10cyclictest0-21swapper/120:44:411
13825997131,10cyclictest181rcu_preempt20:32:071
13816997156,11cyclictest13800-21cyclictest00:19:400
13816997152,9cyclictest0-21swapper/021:31:030
13816997151,10cyclictest0-21swapper/020:54:440
1381699712,26cyclictest15-21ksoftirqd/022:16:250
1381699712,26cyclictest15-21ksoftirqd/022:16:250
13834997044,7cyclictest181rcu_preempt23:40:132
13825997047,15cyclictest0-21swapper/119:49:281
13825997036,10cyclictest14468-21ssh00:24:061
13816997052,9cyclictest0-21swapper/021:49:100
13816997044,7cyclictest181rcu_preempt19:33:580
1381699703,37cyclictest15-21ksoftirqd/020:04:290
1381699703,27cyclictest15-21ksoftirqd/020:24:430
1381699703,27cyclictest15-21ksoftirqd/020:24:430
1381699703,26cyclictest15-21ksoftirqd/021:23:030
1381699702,28cyclictest15-21ksoftirqd/020:44:080
13840996914,32cyclictest0-21swapper/321:39:563
13834996952,12cyclictest13574-21switchtime20:34:412
13825996952,8cyclictest0-21swapper/120:15:291
13825996952,7cyclictest0-21swapper/121:34:241
13825996952,7cyclictest0-21swapper/121:34:241
1381699694,37cyclictest15-21ksoftirqd/020:34:550
1381699694,36cyclictest15-21ksoftirqd/019:04:200
1381699694,36cyclictest15-21ksoftirqd/019:04:200
1381699692,29cyclictest15-21ksoftirqd/019:59:290
1381699692,29cyclictest15-21ksoftirqd/019:59:290
1381699692,26cyclictest15-21ksoftirqd/019:14:420
13825996851,7cyclictest0-21swapper/119:14:401
13825996849,9cyclictest0-21swapper/119:34:111
13825996812,30cyclictest31-21ksoftirqd/123:54:331
13825996812,30cyclictest31-21ksoftirqd/123:54:331
13816996854,7cyclictest0-21swapper/023:05:330
13816996854,7cyclictest0-21swapper/023:05:330
13816996851,8cyclictest0-21swapper/019:54:430
13825996730,9cyclictest181rcu_preempt20:10:481
13816996748,7cyclictest181rcu_preempt21:00:190
13816996748,7cyclictest181rcu_preempt21:00:190
1381699663,37cyclictest15-21ksoftirqd/023:29:340
245102650,6sleep30-21swapper/321:59:103
136082649,32sleep30-21swapper/318:58:303
13840996329,17cyclictest0-21swapper/323:49:123
58272610,5sleep30-21swapper/300:09:263
1384099619,21cyclictest0-21swapper/319:04:253
1384099619,21cyclictest0-21swapper/319:04:253
1384099614,36cyclictest13393-21taskset00:21:053
1384099613,37cyclictest22500-21taskset19:24:103
1384099592,38cyclictest917-21kworker/3:1+events21:09:113
1384099592,38cyclictest917-21kworker/3:1+events21:09:113
1384099582,42cyclictest13674-21chrt20:34:423
1384099573,36cyclictest9826-21taskset20:24:293
1384099573,36cyclictest9826-21taskset20:24:293
1384099573,36cyclictest383-21chrt23:59:453
1384099573,35cyclictest25755-21taskset19:34:103
1384099572,40cyclictest10803-21taskset21:35:023
1384099572,40cyclictest10803-21taskset21:35:023
1384099564,34cyclictest14445-21chrt18:59:353
1384099564,34cyclictest14445-21chrt18:59:353
1384099563,33cyclictest16801-21taskset22:39:313
1384099553,37cyclictest18825-21taskset19:09:433
1384099553,37cyclictest18825-21taskset19:09:433
1384099553,36cyclictest4829-21chrt00:07:473
1384099553,36cyclictest4829-21chrt00:07:473
1384099553,34cyclictest32345-21chrt19:54:113
1384099552,38cyclictest7432-21chrt21:29:423
1384099544,36cyclictest16824-21taskset20:44:433
1384099543,39cyclictest7544-21taskset22:24:273
1384099543,36cyclictest4959-21chrt20:09:263
1384099543,36cyclictest15054-21chrt00:24:303
1384099543,35cyclictest8801-21chrt20:19:433
1384099543,34cyclictest3134-21chrt20:04:123
1384099543,34cyclictest3134-21chrt20:04:123
1384099542,35cyclictest3247-21taskset20:04:243
1384099533,36cyclictest26225-21chrt19:34:313
1384099533,36cyclictest26225-21chrt19:34:313
1384099533,35cyclictest15327-21taskset20:39:443
1384099533,32cyclictest27735-21taskset22:04:243
1384099532,37cyclictest31400-21chrt23:04:113
1384099532,37cyclictest31400-21chrt23:04:113
1384099532,36cyclictest2519-21taskset22:15:233
1384099532,36cyclictest2519-21taskset22:15:233
1384099532,32cyclictest28404-21chrt19:39:453
1384099532,32cyclictest28404-21chrt19:39:453
1384099523,37cyclictest31288-21taskset21:14:463
1384099523,37cyclictest31288-21taskset21:14:463
1384099523,35cyclictest3541-21taskset21:24:113
1384099522,37cyclictest20715-21chrt19:17:293
1384099522,35cyclictest25484-21taskset22:54:223
1384099514,35cyclictest5388-21taskset23:14:293
1384099513,34cyclictest21075-21taskset20:59:253
1384099513,34cyclictest21075-21taskset20:59:253
1384099513,33cyclictest20008-21taskset22:44:343
1384099513,32cyclictest22895-21chrt22:49:323
1384099513,31cyclictest15413-21chrt23:29:503
1384099512,38cyclictest2490-21chrt23:09:313
1384099512,36cyclictest18892-21taskset21:49:253
1384099512,34cyclictest15991-21chrt21:44:303
1384099502,37cyclictest12642-21chrt23:25:283
1384099493,32cyclictest32424-21chrt19:54:193
1384099492,34cyclictest24089-21chrt19:29:083
1384099482,36cyclictest20064-21chrt20:54:433
1384099474,33cyclictest31016-21chrt23:56:533
1384099474,33cyclictest31016-21chrt23:56:533
1384099472,30cyclictest4792-21chrt21:24:523
1384099464,30cyclictest29176-21chrt19:44:193
1384099463,28cyclictest12661-21chrt20:34:093
1384099392,25cyclictest17751-21taskset20:49:253
1384099392,25cyclictest17751-21taskset20:49:253
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional