You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 12:28
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Tue Nov 25, 2025 00:44:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13439211140,18sleep20-21swapper/219:09:122
135609912650,43cyclictest32259-21sh00:14:252
1220321249,88sleep00-21swapper/019:04:340
135609910128,40cyclictest0-21swapper/221:27:262
13560999580,10cyclictest30018-21ls19:59:112
13553999472,15cyclictest15965-21latency_hist20:54:141
13560999350,37cyclictest32537-21latency_hist22:24:142
13553999166,16cyclictest292-21dbus-daemon20:19:111
13545998944,8cyclictest181rcu_preempt19:39:460
13545998939,24cyclictest181rcu_preempt23:09:370
13545998939,24cyclictest181rcu_preempt23:09:370
13545998857,19cyclictest207-21systemd-journal00:03:590
1356099875,73cyclictest0-21swapper/220:49:262
13560998744,37cyclictest29363-21cron22:19:112
13560998743,37cyclictest23564-21systemctl22:09:122
13553998720,54cyclictest30535-21apt-get19:59:341
13545998713,44cyclictest15-21ksoftirqd/023:40:170
13560998641,38cyclictest0-21swapper/222:59:482
1356099863,77cyclictest0-21swapper/220:04:242
13553998665,15cyclictest30006-21apt-get19:59:161
13560998541,38cyclictest0-21swapper/221:54:452
13545998524,44cyclictest15-21ksoftirqd/022:01:090
1356099844,73cyclictest0-21swapper/219:29:262
1356099844,70cyclictest0-21swapper/220:19:282
13545998460,16cyclictest12664-21latency_hist20:44:140
13545998437,19cyclictest15-21ksoftirqd/022:10:410
13560998344,33cyclictest20662-21diskmemload23:34:482
1356099832,72cyclictest29423-21ssh00:09:242
13545998333,19cyclictest181rcu_preempt22:54:350
13560998261,15cyclictest10538-21unixbench_singl20:34:482
13560998251,24cyclictest3923-21kworker/u19:3+nfsiod21:53:332
13560998251,24cyclictest3923-21kworker/u19:3+nfsiod21:53:332
1356099824,72cyclictest0-21swapper/221:44:272
13560998236,39cyclictest0-21swapper/221:19:372
13545998259,16cyclictest2909-21latency_hist23:24:140
1356099814,71cyclictest0-21swapper/221:39:292
1356099813,72cyclictest0-21swapper/200:29:272
1356099813,70cyclictest0-21swapper/221:09:272
1356099813,69cyclictest0-21swapper/223:39:262
1356099812,68cyclictest23776-21apt-get23:04:222
13553998161,13cyclictest26712-21latency_hist21:19:131
13553998118,51cyclictest17971-21ssh22:54:351
13545998160,13cyclictest12475-21rm21:49:340
13545998160,13cyclictest12475-21rm21:49:340
1354599815,46cyclictest23329-21ssh23:59:110
1354599815,46cyclictest23329-21ssh23:59:110
13545998145,17cyclictest181rcu_preempt21:39:450
13560998063,11cyclictest17119-21systemctl19:19:112
13560998063,11cyclictest17119-21systemctl19:19:112
1356099804,65cyclictest0-21swapper/219:24:252
1356099804,65cyclictest0-21swapper/219:24:252
1356099803,66cyclictest0-21swapper/220:39:282
13560998034,39cyclictest0-21swapper/220:29:332
13545998060,13cyclictest13533-21cyclictest23:48:000
13545998054,13cyclictest181rcu_preempt00:21:590
13560997962,10cyclictest4505-21latency_hist20:19:142
13560997960,13cyclictest23479-21latency_hist23:59:142
13560997960,13cyclictest23479-21latency_hist23:59:142
1356099794,64cyclictest5485-21ssh00:24:272
13560997934,38cyclictest0-21swapper/222:30:302
13560997934,38cyclictest0-21swapper/222:30:302
13545997943,8cyclictest181rcu_preempt23:24:340
13545997941,18cyclictest181rcu_preempt20:07:040
13545997938,12cyclictest181rcu_preempt20:44:420
13545997911,30cyclictest15-21ksoftirqd/020:19:360
13560997859,13cyclictest15605-21sh22:49:452
1356099785,66cyclictest270672sleep219:49:262
1356099784,69cyclictest0-21swapper/200:19:282
1356099784,69cyclictest0-21swapper/200:04:252
1356099784,66cyclictest0-21swapper/222:39:252
13560997837,34cyclictest0-21swapper/222:24:352
1356099783,67cyclictest3313-21ssh21:34:252
13560997836,36cyclictest0-21swapper/223:19:362
13553997862,10cyclictest5179-21systemd-run00:24:121
13553997856,14cyclictest29682-21fschecks_time00:09:331
13553997851,18cyclictest1-21systemd19:13:341
13545997845,8cyclictest181rcu_preempt21:47:540
13545997844,14cyclictest181rcu_preempt21:05:090
13545997842,15cyclictest181rcu_preempt21:17:200
13560997764,8cyclictest0-21swapper/222:54:502
13560997759,14cyclictest14908-21kworker/u17:2+events_unbound19:24:112
13560997759,14cyclictest14908-21kworker/u17:2+events_unbound19:24:112
13560997733,38cyclictest24028-21sh23:59:372
1356099772,65cyclictest0-21swapper/220:59:252
1354599779,32cyclictest15-21ksoftirqd/021:20:150
13545997758,11cyclictest32116-21ssh21:28:060
13545997745,24cyclictest28601-21latency_hist19:54:150
1356099764,67cyclictest0-21swapper/222:34:272
13560997642,23cyclictest16458-21grep20:54:372
13560997642,23cyclictest16458-21grep20:54:372
13560997637,31cyclictest0-21swapper/221:14:352
1356099763,68cyclictest0-21swapper/223:29:252
1356099763,65cyclictest0-21swapper/219:44:242
13560997636,32cyclictest0-21swapper/222:02:572
13560997635,34cyclictest0-21swapper/222:13:142
13560997630,36cyclictest0-21swapper/220:24:332
1356099762,69cyclictest0-21swapper/221:29:272
1356099762,69cyclictest0-21swapper/221:29:272
13553997661,10cyclictest13533-21cyclictest23:24:471
13545997653,8cyclictest181rcu_preempt23:53:120
13545997644,16cyclictest181rcu_preempt19:45:510
13545997641,9cyclictest181rcu_preempt21:03:470
13545997635,13cyclictest181rcu_preempt23:14:410
1354599763,46cyclictest15-21ksoftirqd/022:49:280
13545997610,38cyclictest15-21ksoftirqd/022:22:070
1356099757,58cyclictest20220-21ssh23:53:522
1356099754,61cyclictest0-21swapper/221:04:252
1354599756,37cyclictest15-21ksoftirqd/021:33:110
1354599756,37cyclictest15-21ksoftirqd/021:33:110
13545997551,16cyclictest355-21runrttasks23:02:310
13545997539,13cyclictest181rcu_preempt19:09:440
13545997538,9cyclictest181rcu_preempt22:15:000
1354599753,44cyclictest15-21ksoftirqd/000:13:300
13545997510,37cyclictest15-21ksoftirqd/019:34:090
132402756,54sleep10-21swapper/119:07:161
13560997437,30cyclictest0-21swapper/223:47:042
1356099743,62cyclictest0-21swapper/219:34:272
13560997431,35cyclictest0-21swapper/223:14:302
13560997430,38cyclictest0-21swapper/223:09:312
13560997430,38cyclictest0-21swapper/223:09:312
13553997460,10cyclictest13533-21cyclictest00:19:111
13545997446,8cyclictest181rcu_preempt22:35:440
13545997446,8cyclictest181rcu_preempt19:54:220
13545997446,20cyclictest17332-21latency_hist19:19:140
13545997446,20cyclictest17332-21latency_hist19:19:140
13545997410,39cyclictest15-21ksoftirqd/000:15:210
13560997355,13cyclictest2060-21threads20:09:462
13560997345,20cyclictest30975-21latency20:00:162
1356099732,59cyclictest12851-21apt-get20:44:242
13553997353,9cyclictest0-21swapper/120:09:281
13545997348,11cyclictest181rcu_preempt22:33:150
13545997348,11cyclictest181rcu_preempt22:33:150
13545997339,14cyclictest181rcu_preempt00:37:370
13545997339,14cyclictest181rcu_preempt00:37:370
13545997336,17cyclictest181rcu_preempt20:54:160
13545997336,17cyclictest181rcu_preempt20:54:160
13545997336,16cyclictest181rcu_preempt00:08:250
13545997328,17cyclictest0-21swapper/019:36:340
1354599732,37cyclictest15-21ksoftirqd/023:04:330
1356099723,64cyclictest0-21swapper/219:39:292
13553997251,12cyclictest0-21swapper/122:34:331
1355399723,50cyclictest23285-21munin-run23:59:111
1355399723,50cyclictest23285-21munin-run23:59:111
1355399722,47cyclictest15650-21ssh21:54:431
13553997212,27cyclictest20662-21diskmemload00:39:141
13553997212,27cyclictest20662-21diskmemload00:39:141
1354599725,32cyclictest15-21ksoftirqd/019:59:330
13545997251,8cyclictest181rcu_preempt22:44:470
13545997243,7cyclictest181rcu_preempt22:07:440
13545997239,12cyclictest181rcu_preempt22:24:550
1354599722,36cyclictest15-21ksoftirqd/023:30:090
13560997145,18cyclictest12932-21unixbench_multi22:44:482
1356099713,60cyclictest14663-21unixbench_singl19:09:492
13553997152,11cyclictest0-21swapper/122:19:211
13553997146,16cyclictest27346-21ntp_kernel_pll_22:14:401
1355399713,24cyclictest20863-21fschecks_time19:29:351
1355399713,21cyclictest18249-21fschecks_time21:59:321
1354599713,32cyclictest6108-21latency_hist21:39:140
1356099704,61cyclictest0-21swapper/200:34:232
1356099704,61cyclictest0-21swapper/200:34:232
13553997053,11cyclictest13579-21switchtime20:44:471
1354599707,50cyclictest3152-21apt-get20:14:270
13545997047,9cyclictest181rcu_preempt21:57:030
13545997031,23cyclictest181rcu_preempt23:34:390
1356699694,49cyclictest0-21swapper/322:19:383
1356099693,51cyclictest3249-21ssh23:24:322
13553996951,12cyclictest20662-21diskmemload00:29:141
13545996949,14cyclictest10366-21threads20:34:460
13545996946,17cyclictest21549-21ntp_kernel_err21:09:410
13545996941,8cyclictest181rcu_preempt20:29:290
13545996941,8cyclictest181rcu_preempt20:14:130
13545996935,9cyclictest181rcu_preempt22:39:440
13553996831,18cyclictest20662-21diskmemload22:33:191
13553996831,18cyclictest20662-21diskmemload22:33:191
13553996826,27cyclictest31-21ksoftirqd/100:31:001
13545996846,15cyclictest5771-21sh00:24:350
1354599683,40cyclictest15-21ksoftirqd/020:53:550
1356699673,52cyclictest23982-21chrt19:39:313
1355399673,33cyclictest9465-21latency_hist20:34:141
1356699663,51cyclictest28453-21chrt00:07:173
1356699663,50cyclictest28505-21chrt22:16:393
1356699663,49cyclictest24569-21chrt21:14:443
1356699662,52cyclictest28079-21chrt19:50:123
13566996625,29cyclictest0-21swapper/322:55:393
1356699662,50cyclictest12711-21taskset21:49:413
1356699662,50cyclictest12711-21taskset21:49:413
13553996650,10cyclictest26425-21/usr/sbin/munin23:09:121
13553996619,31cyclictest31-21ksoftirqd/123:13:541
13553996619,31cyclictest31-21ksoftirqd/123:13:541
1354599664,22cyclictest15-21ksoftirqd/019:19:230
1354599664,22cyclictest15-21ksoftirqd/019:19:230
13545996636,13cyclictest181rcu_preempt19:27:370
13545996636,13cyclictest181rcu_preempt19:27:370
1356699653,49cyclictest32430-21taskset00:14:303
1356699653,49cyclictest31036-21taskset23:15:433
1356699653,48cyclictest4295-21chrt23:25:153
1356699653,48cyclictest22132-21chrt23:55:553
1356699653,48cyclictest22132-21chrt23:55:553
1356699653,48cyclictest15045-21chrt23:44:323
1356699652,50cyclictest6990-21taskset23:29:493
1356699652,50cyclictest15963-21taskset21:54:473
13553996552,9cyclictest13533-21cyclictest21:24:411
13553996551,10cyclictest13533-21cyclictest19:24:341
13553996551,10cyclictest13533-21cyclictest19:24:341
13553996550,9cyclictest32313-21sh23:19:121
13553996548,13cyclictest3923-21kworker/u19:3+events_unbound22:29:111
13553996542,14cyclictest25348-21latency_hist19:44:141
1355399653,44cyclictest27413-21latency21:19:381
13553996531,19cyclictest31-21ksoftirqd/123:35:071
13553996524,34cyclictest16978-21rm22:53:091
13553996521,32cyclictest27999-21grep19:49:491
13553996515,38cyclictest15318-21ntp_kernel_pll_19:14:161
13553996515,38cyclictest15318-21ntp_kernel_pll_19:14:161
13553996514,39cyclictest10168-21timerwakeupswit21:44:491
1354599653,38cyclictest15-21ksoftirqd/000:29:410
1356699643,48cyclictest2555-21taskset22:28:023
1356699643,48cyclictest17708-21chrt20:59:253
1356699643,48cyclictest11595-21chrt21:48:393
1356699643,48cyclictest11259-21taskset22:43:093
1356699642,50cyclictest21114-21chrt22:59:343
1356699642,50cyclictest12553-21taskset22:44:433
1356699642,49cyclictest9784-21taskset00:31:453
1356699642,49cyclictest6078-21chrt20:23:373
1356699642,49cyclictest23641-21taskset21:14:113
1356699642,49cyclictest19704-21taskset23:52:223
1355399643,44cyclictest355-21runrttasks21:40:181
1355399643,43cyclictest21106-21diskstats22:04:311
13553996421,10cyclictest301ktimers/123:29:421
1354599642,21cyclictest15-21ksoftirqd/020:24:480
1345926413,27sleep30-21swapper/319:09:143
1356699634,46cyclictest26084-21chrt22:13:113
1356699633,48cyclictest5466-21chrt00:24:233
1356699633,48cyclictest26712-21chrt23:09:233
1356699633,48cyclictest26712-21chrt23:09:233
1356699633,47cyclictest4553-21chrt20:19:113
1356699633,47cyclictest2457-21chrt21:32:433
1356699633,47cyclictest2457-21chrt21:32:433
1356699633,47cyclictest13831-21chrt19:09:303
1356699633,45cyclictest0-21swapper/319:29:263
1356699632,49cyclictest6408-21taskset20:24:263
1356699632,48cyclictest4945-21chrt22:31:353
1356699632,48cyclictest4945-21chrt22:31:353
1356699632,48cyclictest16321-21taskset22:51:123
13553996348,9cyclictest13134-21unixbench_singl21:49:471
13553996348,9cyclictest13134-21unixbench_singl21:49:471
13553996345,13cyclictest21467-21latency21:09:391
1355399633,41cyclictest396-21fschecks_count23:19:341
1355399632,42cyclictest20820-21ssh21:09:141
13553996319,12cyclictest181rcu_preempt19:44:471
13553996319,11cyclictest301ktimers/123:59:511
1356699624,43cyclictest0-21swapper/321:39:293
1356699623,47cyclictest7296-21taskset22:34:513
1356699623,47cyclictest11043-21taskset23:37:473
1356699623,46cyclictest5102-21chrt21:36:523
1356699623,46cyclictest22359-21chrt22:05:493
1356699622,48cyclictest20448-21chrt22:03:473
1356699622,48cyclictest1812-21taskset20:09:403
1356699622,47cyclictest29849-21chrt21:24:203
1356699622,47cyclictest29529-21chrt21:24:083
1355399629,22cyclictest6325-21latency_hist20:24:141
13553996248,9cyclictest6896-21ntp_kernel_pll_20:24:401
13553996246,11cyclictest23724-21latency_hist19:39:141
1355399623,40cyclictest26780-21fschecks_count00:04:331
1355399622,45cyclictest15817-21ssh23:44:471
13553996220,27cyclictest31-21ksoftirqd/122:59:461
13553996219,10cyclictest301ktimers/122:42:231
13553996213,27cyclictest20662-21diskmemload21:36:441
1356699613,46cyclictest19327-21chrt21:04:273
1356699613,46cyclictest19164-21chrt19:24:323
1356699613,46cyclictest19164-21chrt19:24:323
1356699613,46cyclictest11289-21chrt20:39:283
1356699613,45cyclictest2636-21chrt00:19:253
1356699613,44cyclictest14002-21chrt20:46:493
1356699613,44cyclictest0-21swapper/323:39:243
1356699612,47cyclictest1611-21chrt23:20:543
1356699612,47cyclictest10831-21taskset00:34:243
1356699612,47cyclictest10831-21taskset00:34:243
1356699612,46cyclictest10878-21chrt20:38:273
1356699612,46cyclictest0-21swapper/320:04:243
13553996142,14cyclictest9969-21kernelversion20:34:371
13553996136,10cyclictest0-21swapper/120:09:151
1355399613,42cyclictest354-21/usr/sbin/munin21:29:461
1355399613,42cyclictest354-21/usr/sbin/munin21:29:461
1355399613,39cyclictest18206-21ls23:49:371
1355399612,38cyclictest17493-21runrttasks19:19:291
1355399612,38cyclictest17493-21runrttasks19:19:291
13553996123,7cyclictest301ktimers/120:54:431
13553996123,7cyclictest301ktimers/120:54:431
1356699605,41cyclictest0-21swapper/320:49:263
1356699604,45cyclictest25652-21chrt23:07:043
1356699603,44cyclictest22984-21taskset19:34:493
1356699603,44cyclictest18503-21chrt19:21:203
1356699603,44cyclictest18503-21chrt19:21:203
1356699603,44cyclictest0-21swapper/319:14:253
1356699603,44cyclictest0-21swapper/319:14:253
1356699602,47cyclictest30543-21chrt19:59:273
1356699602,46cyclictest16055-21chrt20:54:233
1356699602,46cyclictest16055-21chrt20:54:233
13553996046,7cyclictest0-21swapper/122:09:401
1355399603,38cyclictest17682-21/usr/sbin/munin20:59:361
1356699592,46cyclictest0-21swapper/300:09:243
1356699592,45cyclictest9050-21chrt20:30:333
1356699592,45cyclictest26465-21taskset19:45:053
1355399592,31cyclictest31-21ksoftirqd/122:44:571
1356699582,44cyclictest29734-21chrt19:55:513
1356699582,44cyclictest0-21swapper/323:59:233
13553995834,13cyclictest0-21swapper/123:39:331
1355399573,23cyclictest31-21ksoftirqd/120:41:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional