You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-11 - 15:34
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sun Jan 11, 2026 12:44:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
305959912768,53cyclictest8886-21cstates11:54:540
29269212313,79sleep129275-21/usr/sbin/munin07:04:471
29269212313,79sleep129275-21/usr/sbin/munin07:04:471
2733821200,5sleep00-21swapper/011:31:110
30595991182,29cyclictest21632-21seq12:16:580
30595991063,31cyclictest31143-21ssh10:44:430
30595991063,31cyclictest31143-21ssh10:44:430
30595991063,27cyclictest5949-21cstates09:09:380
30612991052,30cyclictest16021-21rm11:13:192
30595991032,21cyclictest28763-21seq11:34:270
30595991032,21cyclictest28763-21seq11:34:270
305959910241,55cyclictest0-21swapper/009:59:210
305959910241,55cyclictest0-21swapper/009:59:210
30595991023,30cyclictest352-21runrttasks07:58:470
30595991023,27cyclictest14397-21cron12:04:210
30595991023,27cyclictest14397-21cron12:04:210
305959910185,10cyclictest13244-21modprobe07:54:210
305959910185,10cyclictest13244-21modprobe07:54:210
30595991013,26cyclictest544-21cron11:19:200
30595991013,13cyclictest21776-21apt-get10:29:340
305959910039,53cyclictest28482-21apt-key08:39:260
30595991003,22cyclictest18460-21/usr/sbin/munin09:29:480
30595991002,27cyclictest31672-21ssh09:51:280
30595991002,22cyclictest28350-21/usr/sbin/munin10:39:480
3059599999,80cyclictest8817-21cpu07:40:350
3059599999,80cyclictest8817-21cpu07:40:350
3059599993,32cyclictest2769-21timerandwakeup07:19:560
3059599993,32cyclictest2769-21timerandwakeup07:19:560
3059599992,24cyclictest14132-21sed11:09:470
30595999917,24cyclictest58-21kcompactd010:28:450
30595999917,24cyclictest58-21kcompactd010:28:450
3043029912,48sleep030454-21munin-run07:09:220
3043029912,48sleep030454-21munin-run07:09:220
3059599984,32cyclictest12697-21/usr/sbin/munin10:14:570
3059599984,32cyclictest12697-21/usr/sbin/munin10:14:570
3059599984,27cyclictest352-21runrttasks08:07:260
3059599983,30cyclictest2187-21modprobe12:39:200
30595999817,24cyclictest10850-21rm09:17:260
3061299973,26cyclictest9432-21sh11:01:462
30612999728,37cyclictest0-21swapper/212:05:582
30612999728,37cyclictest0-21swapper/212:05:582
3061299972,27cyclictest352-21runrttasks10:10:332
3059599973,33cyclictest31811-21latency_hist12:34:240
3059599973,33cyclictest31811-21latency_hist12:34:240
3059599973,32cyclictest325-21/usr/sbin/munin07:15:010
3059599973,24cyclictest31831-21acpi11:39:310
3059599972,32cyclictest20983-21ssh09:34:360
30612999651,35cyclictest0-21swapper/208:04:242
30612999644,10cyclictest181rcu_preempt11:54:262
30612999644,10cyclictest181rcu_preempt11:54:262
3059599964,27cyclictest24972-21latency_hist08:29:240
30612999543,45cyclictest27889-21threads09:44:582
30612999543,45cyclictest27889-21threads09:44:582
3059599953,35cyclictest3361-21sh11:44:560
3059599953,22cyclictest17165-21sh09:28:080
3059599952,29cyclictest5789-21grep07:29:540
30612999455,33cyclictest20593-21switchtime11:19:562
30612999455,33cyclictest20593-21switchtime11:19:562
30612999449,39cyclictest26535-21sh09:43:562
3059599943,39cyclictest14115-21ssh09:23:030
3059599943,39cyclictest14115-21ssh09:23:030
3059599943,19cyclictest3609-21apt-get07:24:340
3059599943,19cyclictest3609-21apt-get07:24:340
30595999429,57cyclictest0-21swapper/012:24:230
3059599942,35cyclictest27222-21sh09:44:420
3059599942,35cyclictest27222-21sh09:44:420
3059599942,27cyclictest9815-21/usr/sbin/munin10:09:380
30612999373,14cyclictest5513-21diskmemload09:27:332
3061299933,28cyclictest3830-21runrttasks09:03:402
3059599933,33cyclictest7312-21cron10:59:210
3059599933,33cyclictest20363-21sort08:15:460
3059599933,30cyclictest15748-21grep12:04:580
3059599933,30cyclictest15748-21grep12:04:580
3059599933,29cyclictest24765-21ssh10:34:310
3059599933,29cyclictest24765-21ssh10:34:310
3059599933,27cyclictest9385-21idleruntime-cro10:09:210
3059599933,27cyclictest9385-21idleruntime-cro10:09:210
3059599932,20cyclictest31386-21/usr/sbin/munin07:09:560
3061299923,24cyclictest352-21runrttasks10:52:242
3061299922,29cyclictest23069-21sh10:30:282
3059599928,9cyclictest0-21swapper/008:59:520
3059599924,25cyclictest20122-21/usr/sbin/munin11:19:440
3059599924,25cyclictest20122-21/usr/sbin/munin11:19:440
3059599923,32cyclictest25417-21chrt08:29:470
3059599923,31cyclictest15338-21grep07:59:430
3059599923,29cyclictest26430-21/usr/sbin/munin12:24:480
3059599923,29cyclictest23306-21/usr/sbin/munin11:24:490
3059599923,25cyclictest10385-21basename11:04:210
3059599923,22cyclictest8138-21modprobe11:54:210
3059599923,22cyclictest8138-21modprobe11:54:210
30595999230,51cyclictest0-21swapper/008:09:310
30595999230,51cyclictest0-21swapper/008:09:310
3059599922,21cyclictest15746-21find10:19:310
30612999153,32cyclictest20923-21mailstats08:16:022
30612999149,35cyclictest0-21swapper/208:24:442
30612999149,34cyclictest0-21swapper/207:40:532
30612999149,34cyclictest0-21swapper/207:40:532
3061299913,7cyclictest0-21swapper/212:34:322
3059599914,33cyclictest4585-21sort09:59:570
3059599914,31cyclictest18704-21cat12:09:560
30595999136,49cyclictest0-21swapper/009:04:430
3059599913,32cyclictest200-21systemd-journal09:44:210
3059599913,23cyclictest11818-21sed07:49:240
3059599913,23cyclictest11818-21sed07:49:240
3059599912,30cyclictest32387-21grep08:49:540
3059599912,30cyclictest32387-21grep08:49:540
3061299902,17cyclictest322562cut07:14:242
3059599903,29cyclictest8187-21idleruntime-cro07:39:210
3059599903,29cyclictest8187-21idleruntime-cro07:39:210
3059599903,29cyclictest29136-21runrttasks08:40:360
3059599903,28cyclictest3730-21sh10:52:440
295202908,57sleep229566-21memory07:04:562
295202908,57sleep229566-21memory07:04:562
30612998958,20cyclictest374-21sshd11:09:252
3061299893,9cyclictest0-21swapper/209:29:372
3059599894,27cyclictest31634-21latency_hist08:49:240
3059599894,27cyclictest31634-21latency_hist08:49:240
3061299884,6cyclictest0-21swapper/207:49:322
3061299884,6cyclictest0-21swapper/207:49:322
3061299883,7cyclictest0-21swapper/211:44:322
3059599883,24cyclictest11299-21/usr/sbin/munin11:04:510
30612998747,33cyclictest0-21swapper/210:45:362
30612998747,33cyclictest0-21swapper/210:45:362
30612998743,37cyclictest0-21swapper/209:49:442
3061299873,40cyclictest25946-21ssh12:24:362
30595998734,44cyclictest0-21swapper/008:19:540
30612998665,12cyclictest0-21swapper/211:59:232
3061299863,7cyclictest0-21swapper/207:34:342
3061299863,7cyclictest0-21swapper/207:34:342
3061299863,32cyclictest32112-21irqstats08:49:462
3061299863,32cyclictest32112-21irqstats08:49:462
3061299863,26cyclictest19897-21runrttasks12:12:532
30606998661,18cyclictest21084-21timerandwakeup08:16:041
30612998561,17cyclictest30582-21cyclictest09:04:432
3061299853,36cyclictest29462-21/usr/sbin/munin12:29:482
3061299853,36cyclictest29462-21/usr/sbin/munin12:29:482
3061299852,34cyclictest6333-21ssh10:04:202
3059599853,27cyclictest1615-21switchtime08:54:550
3061299843,32cyclictest30708-21date10:44:232
3061299833,7cyclictest0-21swapper/209:34:322
30612998334,42cyclictest0-21swapper/207:24:592
30612998334,42cyclictest0-21swapper/207:24:592
3061299833,32cyclictest28184-21mktemp08:39:242
3061299832,34cyclictest25608-21ssh11:29:232
3061299832,33cyclictest5954-21cstates09:09:382
3061299832,33cyclictest28981-21kernelversion08:40:322
3061299832,26cyclictest1572-21/usr/sbin/munin08:54:542
3061899823,37cyclictest4448-21taskset09:04:443
3061299823,34cyclictest23897-21grep12:19:552
3061299823,26cyclictest14487-21/usr/sbin/munin07:54:582
3061299823,24cyclictest16705-21/usr/sbin/munin11:14:532
3061299822,32cyclictest22760-21grep08:19:572
3061299813,7cyclictest0-21swapper/211:59:372
3061299813,7cyclictest0-21swapper/211:59:372
3061299813,40cyclictest606-21apt-get09:54:352
3061299813,40cyclictest606-21apt-get09:54:352
3061299813,33cyclictest31510-21modprobe08:49:212
3061299813,33cyclictest31510-21modprobe08:49:212
3061299812,34cyclictest26302-21/usr/sbin/munin11:29:462
3061299812,33cyclictest1336-21/usr/sbin/munin07:15:022
3061299812,32cyclictest5688-21/usr/sbin/munin07:29:492
3061299812,32cyclictest11644-21awk07:49:212
3061299812,32cyclictest11644-21awk07:49:212
3061299812,31cyclictest5248-21uname10:54:512
3061299812,30cyclictest10658-21taskset09:16:552
3061299812,26cyclictest16699-21/usr/sbin/munin10:19:572
30606998157,7cyclictest181rcu_preempt07:49:341
30606998157,7cyclictest181rcu_preempt07:49:341
30606998143,20cyclictest181rcu_preempt12:15:181
3061299803,36cyclictest3350-21modprobe07:24:212
3061299803,36cyclictest3350-21modprobe07:24:212
3061299803,36cyclictest12698-21/usr/sbin/munin10:14:322
3061299803,36cyclictest12698-21/usr/sbin/munin10:14:322
3061299803,31cyclictest436-21grep11:39:582
3061299803,31cyclictest18805-21/usr/sbin/munin08:09:472
3061299803,31cyclictest18805-21/usr/sbin/munin08:09:472
3061299803,28cyclictest12089-21/usr/sbin/munin09:19:412
3061299803,28cyclictest12089-21/usr/sbin/munin09:19:412
3061299803,24cyclictest25122-21/usr/sbin/munin08:29:382
3061299802,29cyclictest30741-21seq11:37:152
3061299802,29cyclictest30741-21seq11:37:152
30606998046,14cyclictest181rcu_preempt08:09:441
30606998046,14cyclictest181rcu_preempt08:09:441
2968728010,40sleep30-21swapper/307:05:053
2968728010,40sleep30-21swapper/307:05:053
3061299793,68cyclictest25227-21kernelversion10:34:462
3061299793,68cyclictest25227-21kernelversion10:34:462
3061299792,32cyclictest18810-21ssh10:24:372
3061299792,32cyclictest18810-21ssh10:24:372
30606997962,11cyclictest5513-21diskmemload10:36:551
30606997962,11cyclictest5513-21diskmemload10:36:551
3060699794,20cyclictest22351-21rm10:29:481
3061299788,26cyclictest16877-21cat08:04:392
3061299782,31cyclictest6775-21cpuspeed10:04:382
3061299782,31cyclictest6775-21cpuspeed10:04:382
30606997857,15cyclictest19437-21ssh10:24:551
30606997857,15cyclictest19437-21ssh10:24:551
30606997845,11cyclictest181rcu_preempt12:04:211
30606997845,11cyclictest181rcu_preempt12:04:211
3060699777,61cyclictest8678-21apt-get11:54:391
30606997755,15cyclictest12212-21ssh11:06:061
30606997751,19cyclictest787-21kworker/u18:0+nfsiod12:36:221
30606997751,10cyclictest1803-21ssh10:49:401
30606997749,22cyclictest32167-21switchtime10:45:371
30606997749,22cyclictest32167-21switchtime10:45:371
3061299763,24cyclictest22510-21latency_hist12:19:242
30606997657,13cyclictest2304-21kworker/u17:0-xprtiod09:20:001
30606997657,13cyclictest2304-21kworker/u17:0-xprtiod09:20:001
30606997657,13cyclictest17626-21latency_hist09:29:241
30606997651,17cyclictest5513-21diskmemload11:26:131
30606997642,9cyclictest181rcu_preempt11:44:561
3060699762,39cyclictest20196-21ssh11:19:461
3060699762,39cyclictest20196-21ssh11:19:461
30606997546,10cyclictest181rcu_preempt10:54:331
3060699754,48cyclictest29842-21cat09:49:241
3060699754,48cyclictest29842-21cat09:49:241
30606997539,11cyclictest23710-21fschecks_time08:24:441
3060699753,31cyclictest5820-21ps07:29:541
30606997532,13cyclictest181rcu_preempt09:29:501
3060699746,37cyclictest28542-21apt-get08:39:461
3060699745,47cyclictest13926-21fschecks_count11:09:431
30606997452,16cyclictest8430-21latency_hist11:54:251
30606997452,16cyclictest8430-21latency_hist11:54:251
30606997441,13cyclictest181rcu_preempt10:09:351
3060699743,39cyclictest4922-21/usr/sbin/munin09:04:561
30606997426,25cyclictest28835-21sed10:39:581
30606997332,13cyclictest181rcu_preempt12:08:271
30606997332,13cyclictest181rcu_preempt12:08:271
30606997328,13cyclictest181rcu_preempt10:14:451
30606997328,13cyclictest181rcu_preempt10:14:451
30606997328,13cyclictest0-21swapper/109:09:331
30606997248,11cyclictest181rcu_preempt11:36:571
30606997248,11cyclictest181rcu_preempt11:36:571
30606997244,19cyclictest11618-21cron07:49:211
30606997244,19cyclictest11618-21cron07:49:211
3060699724,34cyclictest30265-21fschecks_count09:49:421
30606997238,17cyclictest181rcu_preempt07:34:541
30606997238,17cyclictest181rcu_preempt07:34:541
30606997151,13cyclictest3509-21unixbench_singl08:59:591
30606997146,18cyclictest0-21swapper/112:24:591
30606997138,13cyclictest23478-21ssh09:38:531
3060699713,54cyclictest8693-21taskset07:39:361
3060699713,54cyclictest8693-21taskset07:39:361
30606997134,13cyclictest181rcu_preempt08:46:121
30606997134,13cyclictest181rcu_preempt08:46:121
30606997127,12cyclictest181rcu_preempt09:54:451
30606997127,12cyclictest181rcu_preempt09:54:451
3060699711,39cyclictest546-21tune2fs07:14:461
30606997053,11cyclictest170-21jbd2/mmcblk2p2-810:04:261
30606997053,11cyclictest170-21jbd2/mmcblk2p2-810:04:261
30606997041,12cyclictest181rcu_preempt12:22:241
30606997040,21cyclictest26397-21modprobe08:34:221
3060699703,47cyclictest8895-21cpu09:14:381
3060699703,46cyclictest352-21runrttasks10:00:241
30606997026,16cyclictest181rcu_preempt11:18:551
30606996948,12cyclictest0-21swapper/110:23:441
3060699693,43cyclictest2866-21timerwakeupswit07:19:561
3060699693,43cyclictest2866-21timerwakeupswit07:19:561
30606996934,10cyclictest181rcu_preempt10:59:541
3060699692,22cyclictest17715-21kworker/u18:2+events_unbound07:14:231
3060699692,14cyclictest1140-21fschecks_time08:54:431
30606996853,9cyclictest29169-21fschecks_count12:29:421
30606996853,9cyclictest29169-21fschecks_count12:29:421
30606996843,13cyclictest0-21swapper/108:39:221
3060699683,44cyclictest31977-21cstates11:39:381
3060699683,39cyclictest31973-21taskset08:49:431
3060699683,39cyclictest31973-21taskset08:49:431
30606996833,14cyclictest181rcu_preempt07:24:591
30606996833,14cyclictest181rcu_preempt07:24:591
3060699677,48cyclictest25766-21/usr/sbin/munin11:29:531
30606996744,15cyclictest0-21swapper/112:09:571
3060699673,30cyclictest25128-21ssh09:40:011
30606996721,12cyclictest181rcu_preempt08:04:281
3061899663,48cyclictest11023-21taskset07:44:563
3061899663,48cyclictest11023-21taskset07:44:563
3060699663,32cyclictest14086-21grep07:54:531
3061899658,38cyclictest0-21swapper/308:15:453
3061899658,28cyclictest0-21swapper/310:44:373
3061899658,28cyclictest0-21swapper/310:44:373
3061899654,38cyclictest0-21swapper/307:40:373
3061899654,38cyclictest0-21swapper/307:40:373
30618996520,30cyclictest0-21swapper/311:54:513
3060699654,39cyclictest15468-21if_eth007:59:461
30606996435,9cyclictest22572-21switchtime08:19:551
3061899634,33cyclictest0-21swapper/308:39:533
261962630,3sleep30-21swapper/311:29:453
3061899613,43cyclictest29689-21taskset10:42:053
3061899612,45cyclictest17606-21taskset12:09:233
3061899612,45cyclictest17606-21taskset12:09:233
3061899612,43cyclictest2658-21taskset10:50:003
3061899612,36cyclictest917-21kworker/3:1+events12:38:303
3061899583,40cyclictest7665-21taskset10:59:323
3061899572,40cyclictest23582-21taskset08:24:393
3061899563,36cyclictest32291-21taskset08:49:503
3061899563,36cyclictest32291-21taskset08:49:503
3061899563,36cyclictest31754-21chrt11:39:253
3061899563,36cyclictest31754-21chrt11:39:253
3061899553,34cyclictest23348-21chrt11:24:503
3061899543,34cyclictest15797-21chrt10:19:363
3061899543,33cyclictest22038-21taskset09:34:593
3061899542,35cyclictest6151-21chrt11:49:543
3061899542,35cyclictest6151-21chrt11:49:543
3061899532,39cyclictest4405-21taskset09:59:553
3061899532,36cyclictest1690-21chrt08:54:563
3061899532,35cyclictest30985-21taskset07:09:453
30618995312,29cyclictest25516-21taskset09:41:073
3061899523,36cyclictest18773-21taskset09:29:563
3061899522,37cyclictest27745-21taskset09:44:563
3061899522,37cyclictest27745-21taskset09:44:563
3061899513,36cyclictest21157-21taskset11:20:453
3061899513,36cyclictest21157-21taskset11:20:453
3061899513,35cyclictest622-21taskset07:14:473
3061899513,34cyclictest14859-21taskset07:59:213
3061899512,33cyclictest25660-21chrt12:24:233
3061899504,33cyclictest29046-21taskset12:29:373
3061899504,33cyclictest29046-21taskset12:29:373
3061899503,36cyclictest12173-21taskset11:59:473
3061899503,36cyclictest12173-21taskset11:59:473
3061899503,35cyclictest19339-21taskset10:24:513
3061899503,35cyclictest19339-21taskset10:24:513
3061899503,34cyclictest27004-21taskset12:25:003
3061899502,36cyclictest12029-21taskset09:19:393
3061899502,36cyclictest12029-21taskset09:19:393
3061899502,32cyclictest27081-21taskset08:34:473
3061899502,32cyclictest11026-21chrt10:10:373
3061899494,32cyclictest4844-21taskset10:54:423
3061899493,34cyclictest12848-21chrt07:49:583
3061899493,34cyclictest12848-21chrt07:49:583
3061899493,33cyclictest19209-21chrt12:10:583
3061899492,35cyclictest1016-21taskset09:54:483
3061899492,35cyclictest1016-21taskset09:54:483
3061899492,33cyclictest13403-21chrt11:09:213
3061899483,33cyclictest25572-21taskset08:29:503
3061899482,34cyclictest16225-21taskset09:25:313
3061899482,33cyclictest2521-21taskset07:19:493
3061899482,33cyclictest2521-21taskset07:19:493
3061899482,32cyclictest8154-21taskset07:39:213
3061899482,32cyclictest8154-21taskset07:39:213
3061899474,32cyclictest8954-21taskset10:08:133
3061899474,32cyclictest8954-21taskset10:08:133
3061899473,32cyclictest5844-21taskset09:09:343
3061899473,32cyclictest3864-21taskset09:04:073
3061899473,32cyclictest30431-21chrt09:49:463
3061899473,32cyclictest16003-21taskset07:59:583
3061899473,31cyclictest13905-21taskset11:09:423
3061899473,30cyclictest19035-21chrt11:18:173
3061899472,30cyclictest9654-21taskset09:14:573
3061899465,32cyclictest23349-21chrt10:31:153
3061899463,32cyclictest21945-21taskset08:19:373
3061899463,30cyclictest32543-21chrt11:39:533
3061899462,33cyclictest4230-21chrt07:24:553
3061899462,33cyclictest4230-21chrt07:24:553
3061899462,33cyclictest24784-21taskset10:34:313
3061899462,33cyclictest24784-21taskset10:34:313
3061899462,33cyclictest21002-21chrt12:15:153
3061899453,31cyclictest18894-21chrt08:09:503
3061899453,31cyclictest18894-21chrt08:09:503
3061899453,31cyclictest13936-21chrt10:15:193
3061899453,31cyclictest13936-21chrt10:15:193
3061899432,30cyclictest3382-21chrt11:44:563
3061899422,29cyclictest5779-21chrt07:29:543
3061899413,29cyclictest30698-21chrt08:44:523
3061899413,29cyclictest30698-21chrt08:44:523
3061899382,25cyclictest17488-21chrt08:04:563
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional