You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-27 - 00:14
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System (updated Thu Mar 26, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31830211810,83sleep00-21swapper/007:00:100
31215211614,62sleep131266-21sort06:59:461
32685991143,26cyclictest4814-21runrttasks11:38:410
326859911315,23cyclictest23306-21sh10:21:120
32685991123,30cyclictest352-21runrttasks10:47:160
32685991113,35cyclictest17428-21kworker/u18:1+rpciod12:15:440
326859910952,50cyclictest1-21systemd09:56:240
326859910952,50cyclictest1-21systemd09:56:240
32685991083,35cyclictest11481-21cpuspeed11:49:420
32685991063,24cyclictest30812-21sh11:27:260
326859910538,59cyclictest30674-21latency_hist10:34:310
32685991044,28cyclictest10113-21grep09:59:550
32685991043,29cyclictest31345-21grep08:34:470
32685991042,26cyclictest20489-21timerandwakeup11:10:030
32685991034,27cyclictest8413-21apt-key09:04:320
32685991032,22cyclictest3496-21ssh12:29:540
32685991032,22cyclictest3496-21ssh12:29:540
32685991024,32cyclictest29654-21fschecks_count12:19:480
32685991024,32cyclictest29654-21fschecks_count12:19:480
32685991024,27cyclictest23046-21/usr/sbin/munin08:09:540
32685991023,29cyclictest28762-21latency_hist11:24:300
32685991013,24cyclictest21639-21ssh12:05:240
32685991012,26cyclictest25393-21/usr/sbin/munin08:15:010
326859910118,24cyclictest23428-21sed11:15:070
3252521017,64sleep232562-21awk07:04:282
32685991009,30cyclictest2138-21cmp07:09:310
32685991009,30cyclictest2138-21cmp07:09:310
326859910051,43cyclictest4041-21kworker/u17:3+rpciod10:09:290
32685991003,30cyclictest23964-21rm09:29:440
32685991003,26cyclictest352-21runrttasks10:11:060
32685991003,25cyclictest352-21runrttasks11:02:120
3268599993,38cyclictest26444-21cat08:19:450
3268599993,33cyclictest4101-21dpkg07:14:280
3268599993,28cyclictest16890-21fschecks_time11:04:480
3268599993,28cyclictest16890-21fschecks_time11:04:480
3268599993,26cyclictest5267-21cstates08:54:430
3268599993,22cyclictest3522-21apt-get09:49:390
3268599992,29cyclictest21434-21/usr/sbin/munin08:05:430
3268599984,34cyclictest29936-21/usr/sbin/munin09:39:500
3268599984,27cyclictest486-21/usr/sbin/munin09:45:060
3268599983,30cyclictest352-21runrttasks09:24:360
3268599983,30cyclictest352-21runrttasks09:24:360
3268599983,24cyclictest4132-21/usr/sbin/munin08:49:590
32696999761,12cyclictest181rcu_preempt08:59:301
3268599974,33cyclictest25672-21timerandwakeup10:25:030
3268599973,33cyclictest31996-21cstates11:29:440
3268599973,31cyclictest352-21runrttasks10:16:200
3268599973,28cyclictest352-21runrttasks07:52:270
3268599972,37cyclictest14256-21/usr/sbin/munin07:44:300
3268599972,37cyclictest14256-21/usr/sbin/munin07:44:300
32685999712,30cyclictest14608-21sh11:54:460
3268599964,36cyclictest10615-21apt-get10:54:390
3268599964,22cyclictest11720-21ssh09:09:440
3268599964,20cyclictest8660-21fschecks_count11:44:480
3268599963,33cyclictest32750-21/usr/sbin/munin08:39:550
3268599963,30cyclictest30810-21apt-get10:34:380
3268599963,24cyclictest11238-21/usr/sbin/munin07:34:470
3268599963,24cyclictest11238-21/usr/sbin/munin07:34:470
32685999612,27cyclictest17536-21sort11:59:430
32696999564,9cyclictest181rcu_preempt11:22:351
3268599954,27cyclictest2601-21mailstats08:45:010
3268599954,11cyclictest7776-21apt-get07:24:390
3268599954,11cyclictest7776-21apt-get07:24:390
3268599953,32cyclictest17910-21fschecks_count09:19:470
3268599953,32cyclictest17910-21fschecks_count09:19:470
3268599953,30cyclictest352-21runrttasks09:14:550
3268599953,22cyclictest26959-21apt-get09:34:440
3268599953,22cyclictest15094-21fschecks_time07:45:430
3268599943,30cyclictest17977-21/usr/sbin/munin07:54:520
3268599943,30cyclictest17977-21/usr/sbin/munin07:54:520
3268599943,28cyclictest32414-21/usr/sbin/munin12:25:020
3268599942,29cyclictest352-21runrttasks10:43:140
3268599942,29cyclictest352-21runrttasks10:43:140
3270699934,69cyclictest352-21runrttasks11:14:502
32696999363,13cyclictest181rcu_preempt11:16:111
32696999341,12cyclictest181rcu_preempt07:55:001
32696999341,12cyclictest181rcu_preempt07:55:001
3268599924,31cyclictest26185-21latency_hist12:14:290
3268599924,25cyclictest6579-21/usr/sbin/munin07:19:520
3268599923,31cyclictest352-21runrttasks08:31:250
3268599923,31cyclictest352-21runrttasks08:31:250
3268599923,27cyclictest10132-21apt-config10:54:290
32706999161,21cyclictest20980-21apt-key08:04:292
32685999143,42cyclictest8800-21ssh09:04:590
3268599912,32cyclictest19619-21cpu07:59:420
32685999111,25cyclictest5910-21cat11:39:530
32696999056,11cyclictest181rcu_preempt10:44:521
32696999040,12cyclictest181rcu_preempt08:29:521
32696999040,12cyclictest181rcu_preempt08:29:521
32685999077,8cyclictest0-21swapper/007:29:500
32685999077,8cyclictest0-21swapper/007:29:500
3268599904,28cyclictest352-21runrttasks07:14:580
3268599903,29cyclictest28109-21tune2fs08:24:490
32696998951,14cyclictest181rcu_preempt11:49:491
32696998940,15cyclictest181rcu_preempt09:46:321
32696998857,12cyclictest181rcu_preempt07:24:311
32696998854,11cyclictest181rcu_preempt10:29:301
32696998851,8cyclictest181rcu_preempt10:14:481
32706998764,15cyclictest2952-21systemd-journal11:33:342
32706998763,16cyclictest6028-21latency_hist07:19:322
32696998758,22cyclictest297-21systemd-logind08:44:281
32696998757,21cyclictest18350-21sh09:19:581
32696998757,21cyclictest18350-21sh09:19:581
32696998757,12cyclictest181rcu_preempt09:39:401
32696998657,11cyclictest181rcu_preempt11:43:231
32696998636,26cyclictest8130-21diskmemload12:23:271
32696998636,26cyclictest8130-21diskmemload12:23:271
32706998560,19cyclictest23983-21sh09:29:472
32706998513,56cyclictest9619-21apt-get07:29:422
32706998513,56cyclictest9619-21apt-get07:29:422
32696998554,9cyclictest181rcu_preempt12:18:321
32696998455,8cyclictest20152-21ssh12:04:291
32696998432,14cyclictest0-21swapper/109:50:031
32706998360,17cyclictest10189-21apt-key10:54:302
3269699837,66cyclictest12771-21apt-get10:04:521
32696998350,16cyclictest181rcu_preempt10:21:421
32696998343,14cyclictest181rcu_preempt10:40:041
32696998343,14cyclictest181rcu_preempt10:40:041
32696998338,14cyclictest181rcu_preempt10:50:011
32696998256,16cyclictest11089-21fschecks_time10:55:221
32696998247,13cyclictest181rcu_preempt09:56:281
32696998247,13cyclictest181rcu_preempt09:56:281
32696998221,23cyclictest23185-21ssh09:28:581
32696998221,23cyclictest23185-21ssh09:28:581
32696998221,12cyclictest181rcu_preempt07:15:161
32706998158,15cyclictest2016-21apt-key07:09:312
32706998158,15cyclictest2016-21apt-key07:09:312
32696998158,15cyclictest2952-21systemd-journal11:26:341
32696998152,9cyclictest181rcu_preempt09:14:301
32696998148,12cyclictest181rcu_preempt10:02:121
32696998146,13cyclictest181rcu_preempt09:06:471
32696998049,13cyclictest181rcu_preempt09:17:221
32696998048,13cyclictest181rcu_preempt07:49:491
3269699804,65cyclictest21452-21apt-get08:04:521
32696998046,14cyclictest181rcu_preempt12:11:481
32696998045,10cyclictest181rcu_preempt11:10:121
32696998044,11cyclictest181rcu_preempt08:16:141
32696997958,15cyclictest28728-21timerwakeupswit10:30:031
32696997948,10cyclictest181rcu_preempt12:32:421
32696997948,10cyclictest181rcu_preempt12:32:421
32696997947,13cyclictest181rcu_preempt10:13:031
32696997941,13cyclictest181rcu_preempt08:10:011
32696997937,13cyclictest181rcu_preempt07:24:591
32696997937,13cyclictest181rcu_preempt07:24:591
3271199783,50cyclictest0-21swapper/311:14:503
32696997847,16cyclictest181rcu_preempt11:45:381
32696997845,8cyclictest181rcu_preempt11:55:101
3269699783,54cyclictest23383-21apt-get09:29:421
32696997831,15cyclictest181rcu_preempt07:40:031
32696997831,15cyclictest181rcu_preempt07:40:031
32696997815,19cyclictest4717-21ssh11:38:251
32706997751,19cyclictest14280-21apt-key07:44:292
32706997751,19cyclictest14280-21apt-key07:44:292
32706997746,21cyclictest17443-21kworker/u19:2+nfsiod09:21:402
32706997746,21cyclictest17443-21kworker/u19:2+nfsiod09:21:402
32696997752,9cyclictest181rcu_preempt12:27:531
32696997747,12cyclictest181rcu_preempt09:00:031
32696997745,12cyclictest181rcu_preempt07:39:041
32696997745,12cyclictest181rcu_preempt07:39:041
32696997742,12cyclictest181rcu_preempt08:38:171
32696997736,12cyclictest181rcu_preempt08:24:501
32696997725,12cyclictest0-21swapper/111:09:271
32696997725,12cyclictest0-21swapper/111:09:271
32706997661,9cyclictest4178-21apt-key07:14:302
32696997640,13cyclictest181rcu_preempt07:59:541
32696997628,8cyclictest181rcu_preempt08:19:501
32696997628,10cyclictest181rcu_preempt07:05:021
32696997628,10cyclictest181rcu_preempt07:05:021
32706997556,13cyclictest11777-21ntp_offset07:34:552
32706997556,13cyclictest11777-21ntp_offset07:34:552
32706997549,16cyclictest9536-21apt-key07:29:322
32706997549,16cyclictest9536-21apt-key07:29:322
32706997549,15cyclictest32675-21cyclictest10:34:432
32696997547,9cyclictest181rcu_preempt10:36:131
32696997541,12cyclictest181rcu_preempt09:39:191
32696997539,13cyclictest181rcu_preempt11:31:321
32696997538,13cyclictest181rcu_preempt07:49:301
32706997456,12cyclictest14750-21cpu09:14:442
32706997453,15cyclictest12664-21apt-key10:04:312
32706997428,10cyclictest381ktimers/210:54:372
32706997355,11cyclictest4081-21sh10:44:292
32706997355,11cyclictest4081-21sh10:44:292
32706997349,16cyclictest8263-21sh09:56:462
32706997349,16cyclictest8263-21sh09:56:462
32706997348,19cyclictest14740-21fschecks_time11:54:502
3270699733,36cyclictest352-21runrttasks11:19:472
32696997346,13cyclictest181rcu_preempt11:01:581
32706997256,10cyclictest856-21timerandwakeup12:25:032
32706997255,11cyclictest12173-21ssh09:09:552
32706997254,13cyclictest31906-21threads08:35:002
32706997251,15cyclictest14164-21latency_hist11:54:292
32696997243,11cyclictest181rcu_preempt12:04:441
32696997239,11cyclictest181rcu_preempt08:49:141
32711997134,24cyclictest2636-21taskset12:29:253
32706997155,9cyclictest20551-21cstates12:04:442
32706997150,15cyclictest22509-21latency_hist11:14:302
32696997111,27cyclictest31-21ksoftirqd/107:14:301
32711997033,26cyclictest8687-21chrt09:04:403
32706997053,12cyclictest3500-21/usr/sbin/munin09:49:462
32706997046,16cyclictest5415-21cut08:54:482
32706997043,16cyclictest0-21swapper/209:29:312
32706997043,16cyclictest0-21swapper/209:29:312
32706997033,28cyclictest6931-21xargs07:20:012
32696997041,8cyclictest181rcu_preempt07:30:051
32696997041,8cyclictest181rcu_preempt07:30:051
32696997030,11cyclictest181rcu_preempt08:49:441
32711996933,23cyclictest6557-21chrt11:40:103
32711996932,25cyclictest15761-21taskset11:55:423
32711996932,23cyclictest19128-21taskset07:56:493
32711996932,23cyclictest19128-21taskset07:56:493
32711996930,25cyclictest14613-21taskset10:07:063
3270699694,46cyclictest22847-21latency_hist08:09:292
32706996934,19cyclictest39-21ksoftirqd/212:03:452
32711996835,21cyclictest27525-21chrt09:34:563
32711996832,23cyclictest5358-21chrt12:33:393
32711996832,23cyclictest5358-21chrt12:33:393
32711996830,26cyclictest19139-21taskset12:01:463
3270699684,39cyclictest29760-21fschecks_time12:19:502
3270699684,39cyclictest29760-21fschecks_time12:19:502
3270699683,47cyclictest18395-21ssh11:07:002
3270699683,47cyclictest18395-21ssh11:07:002
3270699682,46cyclictest8122-21apt-get09:04:332
32706996816,16cyclictest20158-21rm10:15:462
32711996734,21cyclictest23856-21chrt12:09:523
32711996733,21cyclictest17928-21chrt11:05:413
32711996733,21cyclictest17928-21chrt11:05:413
32711996731,24cyclictest6330-21taskset07:19:473
32711996731,23cyclictest7186-21chrt09:54:583
32711996731,23cyclictest7186-21chrt09:54:583
32711996731,23cyclictest4356-21taskset10:44:383
32711996731,23cyclictest21220-21taskset09:24:553
32711996731,23cyclictest21220-21taskset09:24:553
32711996731,23cyclictest12209-21taskset10:04:263
32711996730,23cyclictest12557-21chrt07:38:303
32711996730,23cyclictest12557-21chrt07:38:303
32706996746,14cyclictest17928-21latency_hist07:54:312
3270699674,52cyclictest2974-21apt-get12:29:392
3270699674,52cyclictest2974-21apt-get12:29:392
32706996745,16cyclictest18654-21latency_hist10:14:302
32706996723,9cyclictest381ktimers/211:39:482
32711996635,21cyclictest12003-21chrt09:09:513
32711996633,22cyclictest29756-21taskset11:25:033
32711996632,20cyclictest23087-21chrt10:20:353
32711996631,22cyclictest20146-21chrt07:59:573
32711996628,24cyclictest27927-21taskset08:24:433
32706996650,9cyclictest2953-21kworker/u17:0-rpciod09:09:242
32706996648,13cyclictest28047-21fschecks_count10:29:492
32706996646,13cyclictest0-21swapper/209:39:302
32706996614,23cyclictest2929-21seq08:45:182
32711996535,20cyclictest8149-21chrt09:04:283
32711996533,22cyclictest30888-21chrt12:21:033
32711996533,22cyclictest30888-21chrt12:21:033
32711996533,20cyclictest303-21chrt08:39:383
32711996533,20cyclictest30061-21chrt08:29:583
32711996533,20cyclictest30061-21chrt08:29:583
32711996533,19cyclictest664-21chrt07:04:493
32711996533,19cyclictest664-21chrt07:04:493
32711996533,19cyclictest21702-21chrt12:05:353
32711996532,22cyclictest13619-21chrt10:59:383
32711996532,21cyclictest9299-21chrt10:52:243
32711996532,21cyclictest5134-21chrt09:51:303
32711996531,21cyclictest8879-21chrt07:26:323
32711996531,21cyclictest8879-21chrt07:26:323
32711996531,21cyclictest26614-21chrt10:27:043
32711996531,21cyclictest13203-21chrt11:52:013
32711996530,22cyclictest22833-21chrt08:09:293
32711996530,22cyclictest14736-21chrt09:14:423
32711996528,25cyclictest10122-21chrt11:46:463
32711996527,23cyclictest3095-21taskset07:10:403
32706996548,11cyclictest4930-21ntp_kernel_err10:44:562
32706996548,10cyclictest10795-21kworker/u17:4-rpciod11:49:152
32706996547,12cyclictest297-21systemd-logind08:54:282
3270699653,46cyclictest352-21runrttasks08:25:302
3270699653,43cyclictest27658-21sh12:15:232
32706996510,22cyclictest0-21swapper/211:01:002
32711996433,20cyclictest9801-21chrt07:30:123
32711996433,20cyclictest9801-21chrt07:30:123
32711996432,20cyclictest24071-21chrt08:10:183
32711996432,20cyclictest20867-21chrt11:10:323
32711996432,19cyclictest15802-21chrt10:09:413
32711996431,21cyclictest6295-21chrt08:56:473
32711996431,21cyclictest15817-21chrt07:46:003
32711996431,21cyclictest1076-21chrt11:31:393
32711996431,20cyclictest25789-21chrt09:32:133
32711996431,20cyclictest2114-21chrt09:46:273
32711996431,20cyclictest2083-21chrt08:44:483
32711996430,22cyclictest3766-21chrt10:43:413
32711996430,22cyclictest3766-21chrt10:43:413
32711996430,21cyclictest27929-21chrt12:16:083
3270699648,48cyclictest9482-21kworker/u19:0+ext4-rsv-conversion10:05:202
32706996443,11cyclictest0-21swapper/209:40:062
32706996442,15cyclictest32675-21cyclictest07:45:462
32711996332,20cyclictest19266-21chrt10:14:523
32711996332,19cyclictest26074-21chrt11:19:483
32711996331,21cyclictest4537-21chrt07:14:333
32711996331,19cyclictest17099-21chrt07:50:013
32711996330,21cyclictest27311-21chrt08:20:143
32711996330,20cyclictest3651-21chrt11:35:243
32711996328,23cyclictest25862-21taskset08:17:143
3270699633,37cyclictest19482-21latency_hist07:59:302
3270699633,37cyclictest19482-21latency_hist07:59:302
3270699632,40cyclictest352-21runrttasks10:27:232
3270699632,29cyclictest29714-21taskset08:29:492
3270699632,29cyclictest29714-21taskset08:29:492
32711996231,20cyclictest20122-21chrt09:23:463
32711996231,20cyclictest20122-21chrt09:23:463
32711996231,19cyclictest32340-21chrt10:36:153
32711996231,19cyclictest13999-21chrt07:41:223
32711996231,19cyclictest13999-21chrt07:41:223
32711996230,21cyclictest31067-21chrt09:40:073
32711996230,19cyclictest12158-21chrt10:56:173
32711996229,21cyclictest4751-21chrt08:53:063
32711996228,22cyclictest32188-21chrt08:35:373
32706996248,9cyclictest0-21swapper/209:44:492
32706996240,16cyclictest544-21cron10:24:272
3270699623,41cyclictest26209-21wc08:19:312
32706996212,17cyclictest30754-21ssh11:27:162
32706996139,11cyclictest0-21swapper/211:34:522
3270699613,44cyclictest27743-21latency_hist08:24:302
32711996029,19cyclictest29938-21chrt10:32:593
32706996025,26cyclictest1246-21seq08:40:032
32706995942,11cyclictest23259-21fschecks_count08:09:502
3270699583,42cyclictest352-21runrttasks12:09:442
3256425614,25sleep30-21swapper/307:04:293
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional