You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 11:12
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Wed Nov 26, 2025 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
526921488,18sleep00-21swapper/019:04:320
526921488,18sleep00-21swapper/019:04:320
5924210911,61sleep16009-21grep19:04:541
5924210911,61sleep16009-21grep19:04:541
65909910664,34cyclictest0-21swapper/123:59:511
1106621040,9sleep20-21swapper/220:59:102
6586991032,84cyclictest86-21kswapd023:59:070
6586991008,46cyclictest2687-21apt-get20:34:210
6590999672,17cyclictest691-21cron19:54:071
659099963,71cyclictest9633-21apt-get20:54:231
6590999365,19cyclictest10210-21kworker/u17:1+events_unbound22:09:071
659099932,30cyclictest5516-21sh22:48:011
6590999233,52cyclictest0-21swapper/123:19:091
658699924,67cyclictest19957-21apt-get23:14:210
6590999162,18cyclictest9547-21latency_hist20:54:111
659099913,46cyclictest13403-40mandb23:59:091
659099903,33cyclictest14550-21grep19:29:441
6586999073,10cyclictest5834-21sh23:44:420
6586999073,10cyclictest5834-21sh23:44:420
6594998972,10cyclictest17131-21sh23:09:062
6590998970,13cyclictest540-21uptime23:34:441
6590998938,45cyclictest17759-21cpu21:14:241
659099893,32cyclictest28965-21/usr/sbin/munin21:34:371
659099893,32cyclictest28965-21/usr/sbin/munin21:34:371
659099893,31cyclictest25188-21grep22:24:391
659099892,31cyclictest31161-21grep22:34:441
659499887,64cyclictest14855-21ssh21:09:292
659099883,35cyclictest29395-21sh00:26:391
659099883,34cyclictest18477-21grep23:09:441
659099883,33cyclictest23837-21taskset19:59:361
659099883,32cyclictest31012-21grep20:19:441
659099883,32cyclictest31012-21grep20:19:441
659099882,31cyclictest29799-21grep23:29:401
6594998772,10cyclictest6575-21cyclictest20:49:442
659099874,30cyclictest8542-21chrt21:54:431
6590998738,40cyclictest0-21swapper/123:25:061
6590998738,40cyclictest0-21swapper/123:25:061
659099873,35cyclictest25874-21grep20:04:421
659099873,32cyclictest22208-21/usr/sbin/munin00:14:181
659099873,32cyclictest22208-21/usr/sbin/munin00:14:181
6590998729,52cyclictest0-21swapper/123:53:471
659099872,33cyclictest16257-21awk19:34:461
659099872,31cyclictest23594-21sh23:19:321
6594998662,13cyclictest27443-21cpuspeed22:29:232
6594998662,13cyclictest27443-21cpuspeed22:29:232
659099869,67cyclictest28428-21apt-get20:14:211
659099869,67cyclictest28428-21apt-get20:14:211
6590998638,39cyclictest0-21swapper/121:39:231
659099863,30cyclictest797-21ssh00:34:251
659099863,30cyclictest23291-21cat19:59:111
659099862,31cyclictest22044-21latency_hist00:14:091
6594998557,9cyclictest181rcu_preempt23:14:442
6590998536,41cyclictest0-21swapper/121:49:171
659099853,33cyclictest31934-21fschecks_time20:24:301
659099853,33cyclictest28199-21latency_hist20:14:111
659099853,30cyclictest6175-21/usr/sbin/munin22:49:381
659099853,30cyclictest27331-21/usr/sbin/munin22:29:271
659099853,30cyclictest27331-21/usr/sbin/munin22:29:271
659099853,30cyclictest25074-21ssh00:19:221
659099852,34cyclictest5871-21apt-key20:44:101
659099852,34cyclictest5871-21apt-key20:44:101
659099852,31cyclictest17843-21ssh00:05:131
659099844,31cyclictest9621-21ntpq22:54:361
659099844,30cyclictest18491-21/usr/sbin/munin19:44:411
659099843,36cyclictest13397-21munin-run19:29:071
659099843,33cyclictest18928-21/usr/sbin/munin22:14:391
659099843,33cyclictest12624-21munin-run21:04:081
659099843,32cyclictest28813-21latency_hist21:34:101
659099843,31cyclictest17120-21latency_hist23:09:101
659099843,31cyclictest16140-21/usr/sbin/munin22:09:271
659099843,30cyclictest355-21runrttasks22:01:281
6590998426,50cyclictest0-21swapper/122:19:251
659099842,36cyclictest11150-21taskset19:19:421
659099842,32cyclictest21655-21grep21:19:471
659099842,32cyclictest21655-21grep21:19:471
659099842,31cyclictest9305-21ntp_kernel_pll_19:14:381
6586998421,25cyclictest30483-21ssh22:34:310
6594998359,15cyclictest207-21systemd-journal22:51:182
6594998358,16cyclictest6575-21cyclictest23:09:292
6594998342,17cyclictest181rcu_preempt22:41:042
6594998317,32cyclictest39-21ksoftirqd/221:04:012
6594998316,37cyclictest39-21ksoftirqd/223:46:402
6594998316,37cyclictest39-21ksoftirqd/223:46:402
659099833,33cyclictest6834-21ps20:45:371
659099833,32cyclictest7693-21grep19:09:451
659099833,32cyclictest7693-21grep19:09:451
659099833,32cyclictest355-21runrttasks19:43:211
659099833,31cyclictest25903-21latency_hist21:29:111
659099833,31cyclictest15968-21tail21:10:101
659099833,30cyclictest2174-21sort21:44:251
6590998329,48cyclictest0-21swapper/123:39:301
659099832,32cyclictest12979-21cstates21:04:241
659099832,32cyclictest12979-21cstates21:04:241
659099832,31cyclictest823-21/usr/sbin/munin20:29:401
6586998364,12cyclictest1-21systemd00:29:080
6586998362,14cyclictest6575-21cyclictest21:18:180
6586998358,16cyclictest22293-21ssh22:19:360
6594998214,42cyclictest39-21ksoftirqd/219:55:112
659099823,34cyclictest7793-21latency_hist23:49:101
659099823,34cyclictest7793-21latency_hist23:49:101
659099823,32cyclictest3062-21latency_hist22:44:101
6586998264,11cyclictest19613-21rm22:14:390
6594998140,17cyclictest181rcu_preempt21:30:052
6594998110,37cyclictest39-21ksoftirqd/221:20:432
6594998110,37cyclictest39-21ksoftirqd/221:20:432
6590998126,47cyclictest0-21swapper/100:29:351
6586998161,12cyclictest15333-21cpuspeed19:34:260
6594998061,13cyclictest7630-21/usr/sbin/munin21:54:442
6594998056,16cyclictest6575-21cyclictest20:04:052
659099803,31cyclictest3359-21mailstats20:35:041
6590998026,47cyclictest0-21swapper/122:59:421
6590998026,47cyclictest0-21swapper/122:59:421
6586998044,8cyclictest181rcu_preempt23:19:370
6586998013,24cyclictest9868-21fschecks_count20:54:510
6594997957,15cyclictest14560-21ssh23:59:492
659499792,49cyclictest39-21ksoftirqd/200:29:232
6586997962,13cyclictest15491-21kworker/u19:2+events_unbound21:59:070
6586997961,13cyclictest6575-21cyclictest19:29:300
6586997956,15cyclictest207-21systemd-journal22:48:240
6586997952,18cyclictest9361-21apt-key20:54:100
6586997946,26cyclictest0-21swapper/021:23:380
6586997946,26cyclictest0-21swapper/021:23:380
659499789,26cyclictest39-21ksoftirqd/222:17:432
6594997846,26cyclictest22040-21fschecks_time22:19:312
6594997844,12cyclictest181rcu_preempt21:26:092
6586997860,11cyclictest292-21dbus-daemon20:34:080
658699785,51cyclictest14791-21apt-get21:09:360
6586997835,9cyclictest181rcu_preempt23:09:430
6594997747,12cyclictest181rcu_preempt21:59:592
659499772,36cyclictest39-21ksoftirqd/223:58:442
6594997657,12cyclictest355-21runrttasks23:23:282
6594997639,26cyclictest565-21sh20:29:072
659499762,50cyclictest39-21ksoftirqd/223:26:012
659499762,50cyclictest39-21ksoftirqd/223:26:012
6594997611,35cyclictest39-21ksoftirqd/200:35:482
6586997661,9cyclictest28471-21ssh22:29:530
6586997661,9cyclictest28471-21ssh22:29:530
6586997654,15cyclictest6575-21cyclictest21:44:070
658699765,35cyclictest9016-21/usr/sbin/munin22:54:300
659499757,45cyclictest39-21ksoftirqd/220:43:342
659499757,45cyclictest39-21ksoftirqd/220:43:342
659499752,24cyclictest39-21ksoftirqd/219:19:412
659499746,30cyclictest39-21ksoftirqd/221:17:372
6594997460,9cyclictest6575-21cyclictest21:52:582
6594997457,11cyclictest207-21systemd-journal20:23:182
6594997457,11cyclictest207-21systemd-journal20:23:182
6594997451,16cyclictest8595-21ssh23:49:382
659499742,48cyclictest39-21ksoftirqd/200:05:532
659499742,42cyclictest39-21ksoftirqd/221:49:092
659499741,41cyclictest39-21ksoftirqd/222:09:472
6586997448,18cyclictest31787-21latency_hist23:34:100
6594997350,8cyclictest181rcu_preempt19:29:332
6594997348,9cyclictest181rcu_preempt22:08:542
6594997327,13cyclictest181rcu_preempt20:39:112
6594997325,12cyclictest181rcu_preempt23:37:092
6594997325,11cyclictest0-21swapper/220:45:122
659499732,50cyclictest39-21ksoftirqd/222:24:352
659499732,18cyclictest4832-21munin-plugin-st23:44:062
6586997360,9cyclictest31871-21kworker/u17:0+rpciod23:49:340
6586997333,18cyclictest181rcu_preempt19:16:100
6586997327,12cyclictest181rcu_preempt21:24:280
658699732,28cyclictest28862-21latency20:14:470
658699732,28cyclictest28862-21latency20:14:470
659999724,43cyclictest0-21swapper/323:14:263
6594997251,16cyclictest16526-21kworker/u17:0+events_unbound00:29:072
6594997243,10cyclictest181rcu_preempt20:33:222
659499722,36cyclictest39-21ksoftirqd/219:51:392
6594997214,29cyclictest39-21ksoftirqd/200:23:122
6586997247,17cyclictest14239-21diskmemload00:34:260
658699724,35cyclictest16140-21/usr/sbin/munin22:09:380
658699723,45cyclictest796-21grep22:39:310
659499718,41cyclictest39-21ksoftirqd/223:31:062
6594997148,9cyclictest181rcu_preempt19:28:542
6594997141,12cyclictest181rcu_preempt23:00:432
6594997141,12cyclictest181rcu_preempt23:00:432
6594997140,8cyclictest181rcu_preempt22:46:592
6594997138,24cyclictest25490-21ntp_offset20:04:352
6594997137,12cyclictest181rcu_preempt19:45:312
6594997115,35cyclictest39-21ksoftirqd/219:38:362
658699713,51cyclictest11384-21runrttasks20:59:260
6586997110,22cyclictest12114-21awk19:24:260
546227115,37sleep30-21swapper/319:04:403
546227115,37sleep30-21swapper/319:04:403
659499708,44cyclictest39-21ksoftirqd/220:18:192
659499708,44cyclictest39-21ksoftirqd/220:18:192
659499706,43cyclictest39-21ksoftirqd/220:13:142
6594997051,11cyclictest19802-21ssh00:09:332
6594997049,15cyclictest6574-21latencyplot19:14:202
6594997048,8cyclictest181rcu_preempt21:43:392
659499704,25cyclictest39-21ksoftirqd/221:04:182
659499704,25cyclictest39-21ksoftirqd/221:04:182
659499703,34cyclictest39-21ksoftirqd/222:37:162
6586997052,12cyclictest16180-21ssh00:03:530
6586997051,12cyclictest0-21swapper/021:44:430
6586997043,22cyclictest161ktimers/021:30:320
6586997036,27cyclictest0-21swapper/000:04:360
658699703,39cyclictest6638-21latency20:45:310
658699703,36cyclictest30779-21idleruntime00:29:300
658699702,39cyclictest355-21runrttasks21:37:500
658699702,39cyclictest355-21runrttasks21:37:500
658699702,35cyclictest25177-21fschecks_count20:04:280
6586996952,11cyclictest27103-21ntp_offset20:09:400
6586996947,14cyclictest20514-21latency19:49:330
658699694,53cyclictest2089-21latency_hist23:39:100
6586996943,9cyclictest181rcu_preempt19:59:450
6586996934,13cyclictest181rcu_preempt19:54:520
6586996934,11cyclictest0-21swapper/021:59:240
658699692,32cyclictest17309-21/usr/sbin/munin19:39:330
650126910,36sleep26500-21/usr/sbin/munin19:09:112
650126910,36sleep26500-21/usr/sbin/munin19:09:112
291582690,8sleep30-21swapper/323:29:203
659499682,35cyclictest39-21ksoftirqd/219:39:362
659499682,28cyclictest39-21ksoftirqd/219:09:302
659499682,28cyclictest39-21ksoftirqd/219:09:302
6594996814,25cyclictest39-21ksoftirqd/221:36:422
6594996814,25cyclictest39-21ksoftirqd/221:36:422
658699689,10cyclictest14564-21latency_hist23:04:110
658699689,10cyclictest14564-21latency_hist23:04:110
6586996850,12cyclictest355-21runrttasks00:15:340
6586996850,12cyclictest355-21runrttasks00:15:340
6586996849,12cyclictest0-21swapper/022:04:310
6586996842,17cyclictest15057-21ssh23:04:350
658699683,31cyclictest30978-21grep20:19:430
658699683,31cyclictest30978-21grep20:19:430
658699683,31cyclictest2435-21ls23:39:250
658699683,28cyclictest27425-21modprobe00:24:060
6594996731,8cyclictest181rcu_preempt00:17:082
6594996731,8cyclictest181rcu_preempt00:17:082
6586996741,9cyclictest181rcu_preempt21:04:210
6586996741,9cyclictest181rcu_preempt21:04:210
658699673,37cyclictest5114-21fschecks_count21:49:310
6586996713,37cyclictest0-21swapper/022:51:360
6586996713,27cyclictest19634-21tune2fs00:09:290
6594996612,24cyclictest39-21ksoftirqd/222:54:362
658699666,14cyclictest58-21kcompactd019:14:070
658699666,14cyclictest58-21kcompactd019:14:070
658699663,40cyclictest15-21ksoftirqd/019:19:330
6586996633,9cyclictest181rcu_preempt20:39:250
6586996633,9cyclictest181rcu_preempt20:39:250
6586996631,26cyclictest0-21swapper/022:29:100
6586996527,13cyclictest32041-21taskset20:24:330
658699642,38cyclictest25818-21apt-get23:24:230
658699642,38cyclictest25818-21apt-get23:24:230
658699624,43cyclictest19279-21timerandwakeup19:44:420
659999593,37cyclictest6298-21taskset21:50:523
6599995914,23cyclictest0-21swapper/321:09:313
659999584,39cyclictest1587-21chrt22:39:463
6599995619,20cyclictest0-21swapper/320:34:213
659999553,38cyclictest7935-21chrt20:49:273
659999553,36cyclictest28877-21chrt21:34:113
659999542,34cyclictest20681-21chrt19:49:393
659999533,35cyclictest23783-21chrt22:22:443
659999532,34cyclictest895-21kworker/3:1+events23:44:333
659999532,34cyclictest895-21kworker/3:1+events23:44:333
659999532,34cyclictest28916-21chrt22:31:153
659999532,34cyclictest28916-21chrt22:31:153
659999523,38cyclictest15428-21taskset19:34:293
659999522,37cyclictest20199-21chrt23:14:103
659999522,35cyclictest26714-21chrt23:25:073
659999522,35cyclictest26714-21chrt23:25:073
659999513,36cyclictest4665-21chrt21:49:103
659999513,34cyclictest23852-21chrt00:16:223
659999513,34cyclictest23852-21chrt00:16:223
659999512,36cyclictest31002-21taskset20:19:433
659999512,36cyclictest31002-21taskset20:19:433
659999512,35cyclictest31959-21chrt21:39:293
659999504,34cyclictest19845-21taskset00:09:333
659999503,36cyclictest8004-21chrt23:49:233
659999503,36cyclictest12112-21taskset22:01:213
659999503,35cyclictest12003-21taskset23:55:023
659999503,34cyclictest5784-21chrt22:48:493
659999503,33cyclictest28479-21chrt20:14:373
659999503,33cyclictest28479-21chrt20:14:373
659999493,36cyclictest27531-21chrt20:09:473
659999493,34cyclictest20611-21taskset21:19:243
659999493,34cyclictest20611-21taskset21:19:243
659999493,34cyclictest10937-21chrt19:19:353
659999493,33cyclictest10120-21chrt22:54:453
659999493,32cyclictest21952-21taskset19:54:293
659999493,31cyclictest1666-21chrt20:29:433
659999492,36cyclictest18500-21chrt21:14:423
659999492,35cyclictest31451-21taskset00:29:443
659999492,34cyclictest18803-21chrt00:08:153
659999492,33cyclictest15366-21chrt22:07:413
659999483,34cyclictest29363-21chrt21:34:353
659999483,34cyclictest29363-21chrt21:34:353
659999483,34cyclictest12465-21taskset22:59:393
659999483,34cyclictest12465-21taskset22:59:393
659999483,33cyclictest6402-21chrt22:49:263
659999483,33cyclictest2713-21taskset00:37:543
659999483,33cyclictest16384-21chrt22:09:323
659999483,33cyclictest10302-21chrt21:59:103
659999482,36cyclictest17162-21chrt23:09:073
659999482,34cyclictest25465-21chrt22:24:433
659999482,32cyclictest32474-21chrt23:34:353
659999482,32cyclictest30384-21chrt00:29:113
659999474,32cyclictest12040-21chrt20:59:433
659999473,34cyclictest31061-21chrt22:34:443
659999473,32cyclictest16455-21taskset00:04:103
659999472,35cyclictest26487-21chrt20:09:113
659999472,33cyclictest9139-21chrt19:14:333
659999463,34cyclictest25595-21chrt21:28:113
659999463,32cyclictest18627-21chrt19:44:253
659999463,32cyclictest13647-21chrt21:04:413
659999463,32cyclictest13647-21chrt21:04:413
659999462,32cyclictest11181-21chrt20:59:113
659999462,30cyclictest25664-21chrt00:19:383
659999452,34cyclictest21387-21chrt22:19:043
659999452,31cyclictest13618-21chrt19:29:113
248112450,7sleep30-21swapper/323:21:013
659999443,32cyclictest4817-21chrt20:39:353
659999443,32cyclictest4817-21chrt20:39:353
659999443,30cyclictest3397-21chrt23:39:463
659999443,30cyclictest23617-21taskset19:59:313
659999423,27cyclictest6174-21chrt20:44:213
659999413,28cyclictest17504-21chrt19:39:393
659999412,28cyclictest32041-21taskset20:24:333
6599994023,11cyclictest150352sleep319:34:103
659999392,28cyclictest6776-21chrt19:09:233
659999392,28cyclictest6776-21chrt19:09:233
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional