You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 21:52
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Thu Nov 27, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2908521249,89sleep00-21swapper/007:04:260
30405991203,70cyclictest20566-21taskset12:29:340
30424991188,64cyclictest355-21runrttasks10:35:032
304059911357,49cyclictest0-21swapper/009:59:350
30405991106,76cyclictest26770-21taskset11:44:290
30405991073,70cyclictest0-21swapper/009:47:120
304059910541,57cyclictest0-21swapper/010:09:240
30424991043,49cyclictest3316-21apt-get10:04:172
30405991034,88cyclictest0-21swapper/009:19:150
304059910323,50cyclictest0-21swapper/007:56:360
304059910323,50cyclictest0-21swapper/007:56:360
30405991028,80cyclictest0-21swapper/009:39:130
30405991024,64cyclictest14437-21ssh12:19:260
30405991022,64cyclictest355-21runrttasks12:02:030
30405991022,62cyclictest13409-21taskset10:23:050
304249910115,42cyclictest15244-21apt-get11:24:162
30405991013,73cyclictest355-21runrttasks11:53:150
30405991012,64cyclictest0-21swapper/010:24:240
30405991012,64cyclictest0-21swapper/010:24:240
2895021018,61sleep229029-21sshd07:04:242
30405991004,91cyclictest0-21swapper/010:54:170
304059910047,47cyclictest25174-21ssh10:44:210
30405991002,62cyclictest24800-21taskset08:29:190
3040599994,83cyclictest0-21swapper/011:29:170
30405999923,56cyclictest0-21swapper/012:27:010
3040599982,65cyclictest12742-21taskset07:49:550
3040599982,62cyclictest7943-21taskset07:38:420
3040599982,62cyclictest7943-21taskset07:38:420
30405999817,59cyclictest17312-21sh09:30:150
30405999817,59cyclictest17312-21sh09:30:150
3040599976,55cyclictest0-21swapper/009:34:330
3040599973,58cyclictest0-21swapper/012:13:470
3040599972,90cyclictest0-21swapper/012:14:130
30405999718,61cyclictest0-21swapper/011:01:470
30405999718,56cyclictest0-21swapper/009:06:140
3040599964,60cyclictest0-21swapper/010:31:080
3040599964,60cyclictest0-21swapper/010:31:080
3040599962,57cyclictest0-21swapper/010:07:360
3042499959,67cyclictest2247-21apt-get11:59:162
3040599956,56cyclictest0-21swapper/010:51:290
3040599956,56cyclictest0-21swapper/010:51:290
3040599954,79cyclictest0-21swapper/012:34:120
3040599953,78cyclictest0-21swapper/008:49:160
30405999536,52cyclictest8388-21latency09:14:270
30405999534,53cyclictest0-21swapper/009:54:290
30416999412,71cyclictest790-21apt-get10:59:181
30405999440,46cyclictest0-21swapper/011:14:190
3040599943,56cyclictest28434-21ntp_states08:39:320
3040599943,56cyclictest28434-21ntp_states08:39:320
3040599943,56cyclictest0-21swapper/011:13:150
3040599943,56cyclictest0-21swapper/011:13:150
3040599942,80cyclictest1696-21/usr/sbin/munin11:59:020
3040599942,80cyclictest1696-21/usr/sbin/munin11:59:020
30405999424,40cyclictest0-21swapper/011:41:130
30405999418,46cyclictest0-21swapper/011:21:540
30405999416,62cyclictest15289-21ssh11:24:250
30424999374,13cyclictest22799-21/usr/sbin/munin12:34:212
3040599936,79cyclictest0-21swapper/007:39:260
3040599933,76cyclictest0-21swapper/009:24:150
3040599933,61cyclictest21452-21ssh10:36:520
30405999335,50cyclictest0-21swapper/007:19:300
30405999335,50cyclictest0-21swapper/007:19:300
3040599932,59cyclictest7581-21ssh09:13:490
3040599932,57cyclictest23742-21ssh10:40:190
3042499923,81cyclictest31796-21/usr/sbin/munin10:54:532
30405999238,47cyclictest0-21swapper/008:39:020
30405999238,47cyclictest0-21swapper/008:39:020
30405999218,42cyclictest0-21swapper/007:59:060
30405999211,48cyclictest355-21runrttasks10:14:380
3040599913,77cyclictest10104-21/usr/sbin/munin07:44:180
3040599912,59cyclictest4932-21diskmemload12:04:270
3040599902,74cyclictest0-21swapper/008:54:140
3040599893,75cyclictest3429-21apt-get07:24:110
3040599893,75cyclictest3429-21apt-get07:24:110
3040599893,58cyclictest21815-21grep11:34:400
3040599893,58cyclictest21815-21grep11:34:400
3040599893,57cyclictest19439-21taskset08:12:590
30405998932,50cyclictest0-21swapper/007:34:020
3040599892,54cyclictest29490-21/usr/sbin/munin08:44:290
2940328910,43sleep10-21swapper/107:04:401
30424998830,8cyclictest0-21swapper/211:44:172
3041699883,65cyclictest31826-21sed10:54:531
3040599883,58cyclictest17856-21runrttasks08:08:180
30424998752,13cyclictest181rcu_preempt09:50:472
30424998752,13cyclictest181rcu_preempt09:50:472
3040599872,54cyclictest1241-21taskset07:15:320
3040599872,54cyclictest1241-21taskset07:15:320
30424998662,16cyclictest13361-21unixbench_singl11:19:412
30424998650,8cyclictest181rcu_preempt08:12:372
30424998649,15cyclictest181rcu_preempt07:39:472
3040599862,76cyclictest1944-21/usr/sbin/munin08:59:390
3040599862,69cyclictest30589-21apt-get07:09:140
30405998623,53cyclictest0-21swapper/008:19:210
30405998622,58cyclictest4142-21rm11:04:190
30416998552,21cyclictest8790-21threads09:14:371
30405998516,42cyclictest27366-21/usr/sbin/munin09:49:280
30405998516,42cyclictest27366-21/usr/sbin/munin09:49:280
30424998463,15cyclictest18284-21sh09:33:272
30424998463,15cyclictest18284-21sh09:33:272
30424998436,13cyclictest181rcu_preempt09:14:282
3040599843,67cyclictest19802-21apt-get08:14:130
3042499833,52cyclictest11962-21cut07:49:272
173822830,15sleep1291rcuc/109:30:271
173822830,15sleep1291rcuc/109:30:271
30424998256,10cyclictest181rcu_preempt12:04:302
30424998250,23cyclictest24924-21fschecks_count09:44:372
3042499823,56cyclictest1347-21apt-get10:59:292
30424998235,10cyclictest181rcu_preempt09:25:362
3042499821,50cyclictest16521-21latency_hist08:04:052
30416998264,12cyclictest4932-21diskmemload12:39:041
30416998263,13cyclictest4932-21diskmemload10:12:571
30416998258,16cyclictest21044-21sshd11:34:241
30416998258,16cyclictest21044-21sshd11:34:241
30416998220,12cyclictest0-21swapper/107:09:171
3040599823,66cyclictest0-21swapper/008:24:150
30424998151,14cyclictest15345-21ssh12:20:332
30424998145,10cyclictest181rcu_preempt09:54:172
3042499813,36cyclictest30455-21latency_hist10:54:042
3042499813,36cyclictest30455-21latency_hist10:54:042
30416998152,13cyclictest181rcu_preempt09:47:321
30424998062,12cyclictest17926-21apt08:09:012
30424998050,8cyclictest181rcu_preempt10:16:242
30424998046,12cyclictest181rcu_preempt07:57:092
30424998046,12cyclictest181rcu_preempt07:57:092
30424998044,13cyclictest181rcu_preempt10:49:022
30416998064,10cyclictest8187-21apt-config07:39:041
30416998064,10cyclictest8187-21apt-config07:39:041
30416998058,15cyclictest24859-21fschecks_count08:29:231
30416998045,8cyclictest181rcu_preempt09:13:041
3041699804,54cyclictest4932-21diskmemload11:24:301
30424997945,11cyclictest181rcu_preempt10:19:322
30424997854,16cyclictest3752-21fschecks_count09:04:202
30424997836,14cyclictest181rcu_preempt11:54:242
30424997836,14cyclictest181rcu_preempt11:54:242
30424997812,12cyclictest30395-21cyclictest11:04:162
30416997854,15cyclictest4932-21diskmemload12:14:201
30416997817,37cyclictest31-21ksoftirqd/109:50:561
30416997817,37cyclictest31-21ksoftirqd/109:50:561
30424997748,10cyclictest181rcu_preempt11:43:132
30416997754,17cyclictest16469-21latency_hist08:04:041
30424997649,19cyclictest8832-21switchtime12:09:362
30424997644,9cyclictest181rcu_preempt09:38:152
30424997640,13cyclictest181rcu_preempt09:13:532
3042499762,39cyclictest31181-21cpuspeed08:49:202
3041699762,52cyclictest4087-21awk10:05:201
30424997552,15cyclictest30395-21cyclictest08:39:032
30424997552,15cyclictest30395-21cyclictest08:39:032
30424997547,11cyclictest19329-21ssh11:30:282
30424997542,12cyclictest181rcu_preempt11:14:452
30416997554,15cyclictest11951-21idleruntime-cro11:19:011
30424997454,14cyclictest309-21systemd-logind08:19:022
30424997452,16cyclictest22706-21latency_hist12:34:042
30424997449,16cyclictest29105-21cstates11:49:182
30416997452,15cyclictest316-21rs:main1
30416997452,14cyclictest22534-21sh12:34:021
3043199735,47cyclictest0-21swapper/310:54:533
30424997338,13cyclictest181rcu_preempt10:39:172
30416997357,11cyclictest207-21systemd-journal12:22:241
30416997353,13cyclictest3190-21systemd-run07:24:021
30416997353,13cyclictest3190-21systemd-run07:24:021
30416997351,14cyclictest376-21sshd11:47:231
30424997249,8cyclictest181rcu_preempt12:24:192
30424997249,16cyclictest24037-21cron09:44:012
3042499723,46cyclictest760-21wc07:14:352
3042499723,46cyclictest760-21wc07:14:352
3042499723,40cyclictest19231-21unin-run10:34:022
3042499723,40cyclictest19231-21unin-run10:34:022
30424997210,26cyclictest0-21swapper/210:24:222
30424997210,26cyclictest0-21swapper/210:24:222
30416997250,15cyclictest30395-21cyclictest11:33:321
30416997228,8cyclictest181rcu_preempt10:01:081
3041699722,29cyclictest26638-21sshd08:34:241
3041699722,29cyclictest26638-21sshd08:34:241
30424997148,15cyclictest3152-21latency_hist10:04:062
30424997147,12cyclictest181rcu_preempt08:28:192
30424997138,14cyclictest181rcu_preempt08:31:072
30424997137,13cyclictest181rcu_preempt07:09:272
3042499713,48cyclictest21030-21cut11:34:232
3042499713,48cyclictest21030-21cut11:34:232
30416997147,8cyclictest181rcu_preempt08:11:511
30416997142,20cyclictest9526-21unixbench_singl07:40:351
30416997135,11cyclictest181rcu_preempt09:34:251
30416997134,12cyclictest207-21systemd-journal09:22:061
30416997122,38cyclictest28832-21grep10:49:381
30416997122,38cyclictest28832-21grep10:49:381
3043199703,42cyclictest1467-21chrt10:59:363
30424997046,11cyclictest181rcu_preempt08:56:322
30424997045,8cyclictest181rcu_preempt07:24:132
30424997045,8cyclictest181rcu_preempt07:24:132
30424997041,13cyclictest181rcu_preempt08:45:532
3042499703,46cyclictest8326-21ssh10:14:012
30424997027,11cyclictest181rcu_preempt12:14:192
30416997056,10cyclictest30395-21cyclictest10:23:001
30416997055,9cyclictest4845-21kworker/u17:2-xprtiod12:09:011
30416997054,10cyclictest12263-21ntp_offset07:49:341
3041699705,36cyclictest31-21ksoftirqd/107:44:191
3041699702,23cyclictest31-21ksoftirqd/110:27:561
3041699702,23cyclictest31-21ksoftirqd/110:27:561
30416997019,24cyclictest31-21ksoftirqd/107:58:341
30416997019,24cyclictest31-21ksoftirqd/107:58:341
3041699701,31cyclictest31-21ksoftirqd/107:24:331
3041699701,31cyclictest31-21ksoftirqd/107:24:331
3042499694,18cyclictest7171-21ntp_states07:34:312
3042499694,18cyclictest7171-21ntp_states07:34:312
30424996940,8cyclictest181rcu_preempt11:09:332
30424996940,8cyclictest181rcu_preempt11:09:332
30424996936,13cyclictest181rcu_preempt08:59:272
30416996943,16cyclictest5103-21latency_hist09:09:051
168682690,5sleep30-21swapper/311:26:263
30424996824,35cyclictest2332-21ntpq07:19:302
30424996824,35cyclictest2332-21ntpq07:19:302
30416996849,12cyclictest0-21swapper/111:39:231
30416996845,8cyclictest181rcu_preempt10:29:181
30416996845,8cyclictest181rcu_preempt10:29:181
3041699683,39cyclictest31-21ksoftirqd/110:42:171
30416996810,26cyclictest31-21ksoftirqd/109:03:551
30424996745,9cyclictest181rcu_preempt07:44:372
30424996728,23cyclictest355-21runrttasks08:39:182
30424996728,23cyclictest355-21runrttasks08:39:182
30424996723,22cyclictest0-21swapper/209:19:302
30416996752,11cyclictest30395-21cyclictest11:09:311
30416996752,11cyclictest30395-21cyclictest11:09:311
30416996752,10cyclictest8144-21cpuspeed12:09:181
30416996751,10cyclictest30635-21kworker/u17:3-rpciod11:06:551
30416996750,12cyclictest4195-21kworker/u19:2+events_unbound08:24:011
30416996749,11cyclictest0-21swapper/111:24:041
30416996746,7cyclictest181rcu_preempt08:06:471
30416996746,13cyclictest10996-21systemd-tmpfile10:19:031
3041699672,38cyclictest31-21ksoftirqd/108:14:201
3041699672,37cyclictest31-21ksoftirqd/108:51:231
30424996633,10cyclictest181rcu_preempt08:24:022
30416996647,14cyclictest32648-21timerandwakeup11:54:471
30416996647,14cyclictest32648-21timerandwakeup11:54:471
30416996647,12cyclictest0-21swapper/108:24:321
30416996646,7cyclictest181rcu_preempt09:40:021
30416996645,8cyclictest181rcu_preempt11:51:121
30416996644,7cyclictest181rcu_preempt10:34:551
30416996638,10cyclictest181rcu_preempt12:28:511
30416996632,19cyclictest207-21systemd-journal09:25:061
30424996532,10cyclictest181rcu_preempt07:30:082
3041699658,33cyclictest31-21ksoftirqd/110:47:321
3041699655,29cyclictest31-21ksoftirqd/108:44:161
3041699654,52cyclictest2468-21apt-get11:59:501
30416996516,28cyclictest31-21ksoftirqd/109:57:151
3041699633,44cyclictest355-21runrttasks08:40:041
3041699633,44cyclictest355-21runrttasks08:40:041
3041699632,36cyclictest31-21ksoftirqd/107:16:221
3041699632,36cyclictest31-21ksoftirqd/107:16:221
30416996243,9cyclictest0-21swapper/108:54:371
3043199595,28cyclictest0-21swapper/309:44:213
3043199573,33cyclictest19869-21chrt11:32:133
3043199563,36cyclictest26097-21chrt10:44:403
3043199553,38cyclictest26176-21chrt11:44:043
3043199543,40cyclictest30906-21chrt07:09:263
3043199542,39cyclictest28735-21taskset09:50:393
3043199542,39cyclictest28735-21taskset09:50:393
3043199542,34cyclictest0-21swapper/311:59:523
3043199533,39cyclictest15383-21chrt12:20:393
3043199533,37cyclictest7736-21taskset09:14:043
3043199524,33cyclictest11070-21chrt10:19:043
30431995240,8cyclictest451rcuc/309:43:443
3043199523,36cyclictest5660-21taskset07:29:343
3043199523,36cyclictest19946-21taskset12:29:183
3043199522,35cyclictest11042-21taskset07:44:373
3043199522,33cyclictest4165-21taskset11:04:223
3043199515,32cyclictest23247-21chrt08:24:223
3043199513,39cyclictest11433-21chrt10:19:203
3043199513,38cyclictest30265-21taskset08:44:393
3043199513,37cyclictest20936-21chrt08:16:243
3043199513,37cyclictest1379-21taskset08:55:243
3043199513,35cyclictest13633-21chrt12:19:013
3043199513,34cyclictest21600-21chrt08:19:203
3043199513,34cyclictest12314-21chrt11:19:173
3043199512,36cyclictest17968-21taskset08:09:033
3043199504,34cyclictest21859-21chrt10:38:123
3043199503,35cyclictest1221-21chrt11:57:333
3043199503,35cyclictest1221-21chrt11:57:333
3043199503,34cyclictest8202-21chrt10:13:373
3043199503,34cyclictest5389-21chrt12:04:223
3043199503,34cyclictest11855-21chrt11:18:443
3043199502,37cyclictest4629-21chrt07:27:243
3043199502,37cyclictest4629-21chrt07:27:243
3043199502,37cyclictest28221-21taskset08:39:253
3043199502,37cyclictest28221-21taskset08:39:253
3043199499,24cyclictest0-21swapper/310:04:243
30431994937,8cyclictest451rcuc/312:26:533
3043199493,35cyclictest27238-21chrt11:44:413
3043199493,34cyclictest2936-21chrt10:04:043
3043199493,32cyclictest8861-21chrt07:40:123
3043199492,32cyclictest23848-21chrt10:40:433
3043199484,32cyclictest14511-21chrt10:24:273
3043199484,32cyclictest14511-21chrt10:24:273
3043199483,34cyclictest25224-21taskset12:38:213
3043199483,33cyclictest9757-21chrt12:11:313
3043199483,33cyclictest25128-21chrt08:29:283
3043199483,30cyclictest2643-21chrt08:59:333
3043199482,32cyclictest2330-21chrt07:19:303
3043199482,32cyclictest2330-21chrt07:19:303
3043199482,32cyclictest21436-21chrt09:39:043
3043199473,35cyclictest17275-21chrt10:29:293
3043199473,35cyclictest17275-21chrt10:29:293
3043199473,33cyclictest7646-21taskset11:09:363
3043199473,33cyclictest7646-21taskset11:09:363
3043199473,33cyclictest23276-21chrt11:39:023
3043199473,33cyclictest23276-21chrt11:39:023
3043199473,32cyclictest29512-21chrt10:51:333
3043199473,32cyclictest29512-21chrt10:51:333
3043199472,31cyclictest27812-21chrt08:39:043
3043199472,31cyclictest27812-21chrt08:39:043
3043199472,31cyclictest18994-21chrt08:09:363
3043199472,29cyclictest5111-21chrt09:09:053
3043199463,29cyclictest15477-21chrt07:59:273
3043199462,29cyclictest17808-21chrt09:31:553
3043199462,29cyclictest17808-21chrt09:31:553
3043199453,31cyclictest13885-21chrt09:24:293
297032458,21sleep30-21swapper/307:04:503
3043199442,28cyclictest31861-21taskset08:49:373
3043199442,27cyclictest7542-21taskset07:34:373
3043199442,27cyclictest7542-21taskset07:34:373
3043199433,27cyclictest8771-21chrt09:14:363
3043199432,32cyclictest31397-21chrt09:55:383
3043199432,25cyclictest12102-21chrt09:20:443
3043199423,25cyclictest522-21chrt07:14:283
3043199423,25cyclictest522-21chrt07:14:283
3043199412,25cyclictest13857-21chrt07:54:293
3043199412,25cyclictest13857-21chrt07:54:293
3043199403,27cyclictest28969-21chrt11:49:103
3043199392,28cyclictest11995-21taskset07:49:273
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional