You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-11 - 22:58
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Fri Sep 11, 2026 12:44:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
286989912539,54cyclictest526-21rs:main0
286989911432,59cyclictest326-21systemd-logind08:59:280
286989911432,59cyclictest326-21systemd-logind08:59:280
27722211413,75sleep20-21swapper/207:05:032
28698991123,75cyclictest339-21ssh11:44:280
286989911021,55cyclictest7611-21/usr/sbin/munin09:15:040
28698991073,79cyclictest18464-21/usr/sbin/munin11:19:540
286989910732,53cyclictest0-21swapper/009:40:370
286989910723,47cyclictest0-21swapper/010:29:130
28698991063,79cyclictest16159-21/usr/sbin/munin09:29:550
28698991062,73cyclictest2572-21latency_hist10:54:280
28698991062,67cyclictest22042-21runrttasks11:25:050
28698991053,67cyclictest6176-21cron09:14:260
28698991053,67cyclictest6176-21cron09:14:260
28698991052,73cyclictest14472-21ssh12:07:080
28698991052,73cyclictest14472-21ssh12:07:080
28698991052,69cyclictest22255-21rm12:19:480
286989910519,55cyclictest0-21swapper/010:19:550
286989910519,55cyclictest0-21swapper/010:19:550
28698991042,75cyclictest359-21snmpd08:37:310
28698991042,75cyclictest359-21snmpd08:37:310
28698991042,64cyclictest29710-21runrttasks10:45:450
28698991042,64cyclictest29710-21runrttasks10:45:450
286989910423,61cyclictest0-21swapper/009:28:280
286989910422,63cyclictest0-21swapper/011:03:440
286989910420,66cyclictest3145-21diskmemload09:54:200
843021030,7sleep20-21swapper/211:04:122
28698991032,62cyclictest10716-21seq10:14:310
28698991032,62cyclictest0-21swapper/011:30:050
28712991024,76cyclictest4666-21apt-get07:34:402
28698991023,59cyclictest8325-21seq07:45:120
28698991023,59cyclictest8325-21seq07:45:120
28698991022,56cyclictest27431-21seq09:49:090
286989910225,60cyclictest0-21swapper/007:54:490
286989910143,40cyclictest17102-21ssh11:18:120
28698991013,63cyclictest28801-21seq08:45:250
28698991013,63cyclictest28801-21seq08:45:250
286989910132,55cyclictest0-21swapper/008:09:470
286989910124,58cyclictest0-21swapper/010:34:230
286989910124,58cyclictest0-21swapper/010:34:230
286989910120,63cyclictest3145-21diskmemload12:14:110
286989910120,63cyclictest3145-21diskmemload12:14:110
287129910061,11cyclictest181rcu_preempt10:55:072
28698991003,61cyclictest0-21swapper/010:08:050
28698991003,55cyclictest343-21runrttasks12:32:370
28698991003,55cyclictest343-21runrttasks12:32:370
28698991002,68cyclictest5532-21seq11:52:290
2869899998,55cyclictest0-21swapper/012:29:120
28698999952,41cyclictest3145-21diskmemload11:09:280
28698999941,51cyclictest0-21swapper/010:35:040
2869899993,61cyclictest20738-21seq08:20:270
2869899992,72cyclictest11432-21ssh09:22:070
2869899992,72cyclictest11432-21ssh09:22:070
2869899983,56cyclictest4577-21seq07:33:330
2869899983,56cyclictest4577-21seq07:33:330
2869899982,60cyclictest27523-21seq10:41:020
2869899982,60cyclictest27523-21seq10:41:020
28698999816,50cyclictest1359-21sshd09:02:250
2869899974,79cyclictest10015-21apt-get07:49:400
2869899973,86cyclictest0-21swapper/007:09:300
2869899973,64cyclictest4254-21runrttasks10:03:570
2869899972,61cyclictest1063-21seq09:59:180
2869899972,56cyclictest30946-21seq12:34:370
28698999719,44cyclictest1782-21/usr/sbin/munin09:04:480
28712999675,11cyclictest0-21swapper/209:16:252
2869899963,87cyclictest0-21swapper/011:59:380
2871299952,49cyclictest39-21ksoftirqd/212:13:122
2871299952,49cyclictest39-21ksoftirqd/212:13:122
2869899953,77cyclictest7783-21/usr/sbin/munin10:09:410
2869899953,77cyclictest7783-21/usr/sbin/munin10:09:410
2869899953,60cyclictest31965-21taskset07:15:530
2869899952,83cyclictest0-21swapper/009:34:400
2869899952,83cyclictest0-21swapper/009:34:400
2869899952,61cyclictest7190-21ssh11:54:570
2869899952,61cyclictest7190-21ssh11:54:570
2869899952,59cyclictest5435-21fschecks_count07:35:360
2869899952,55cyclictest0-21swapper/008:00:000
28712999474,15cyclictest28687-21cyclictest09:59:312
28712999453,14cyclictest181rcu_preempt09:14:292
28712999453,14cyclictest181rcu_preempt09:14:292
28712999441,27cyclictest0-21swapper/208:40:012
28712999441,27cyclictest0-21swapper/208:40:012
2869899943,59cyclictest343-21runrttasks07:13:210
2869899943,59cyclictest343-21runrttasks07:13:210
2869899942,57cyclictest24071-21seq08:32:010
2869899942,57cyclictest0-21swapper/012:18:400
2869899942,55cyclictest19191-21runrttasks08:16:340
28705999343,41cyclictest0-21swapper/107:51:251
2869899934,65cyclictest0-21swapper/011:38:550
2869899933,80cyclictest0-21swapper/011:09:400
2869899933,74cyclictest26176-21apt-get08:39:390
2869899933,74cyclictest26176-21apt-get08:39:390
2869899932,61cyclictest16001-21seq08:06:580
28712999264,17cyclictest26014-21sh10:39:312
28712999264,17cyclictest26014-21sh10:39:312
2871299923,56cyclictest21958-21apt-get09:39:462
2870599923,35cyclictest3457-21latency_hist11:49:291
2870599923,35cyclictest3457-21latency_hist11:49:291
2870599923,35cyclictest12306-21munin-run12:04:251
2869899923,59cyclictest1690-21seq07:24:300
2869899922,60cyclictest1270-21seq07:22:200
28712999163,22cyclictest10741-21/usr/sbin/munin10:14:482
28705999145,39cyclictest3381-21latency_hist09:09:301
2870599914,31cyclictest1680-21/usr/sbin/munin10:00:451
2869899912,59cyclictest343-21runrttasks08:50:070
2869899912,59cyclictest343-21runrttasks08:50:070
2870599902,37cyclictest17835-21latency_hist11:19:301
2869899902,75cyclictest21328-21apt-get08:24:420
28712998966,13cyclictest30020-21kworker/u17:1-xprtiod09:59:482
28712998964,17cyclictest21796-21ssh10:31:512
28712998964,17cyclictest21796-21ssh10:31:512
28705998939,40cyclictest1463-21apt-key09:59:311
2870599893,35cyclictest5662-21latency07:35:401
2870599893,28cyclictest2243-21grep07:24:551
2870599892,34cyclictest10515-21sort09:20:021
2870599892,34cyclictest10515-21sort09:20:021
2870599892,32cyclictest3544-21fschecks_time07:29:481
2870599892,32cyclictest3544-21fschecks_time07:29:481
2869899892,78cyclictest523-21apt-get11:44:350
2869899892,78cyclictest523-21apt-get11:44:350
2869899892,57cyclictest9580-21seq07:47:260
2869899892,57cyclictest9580-21seq07:47:260
2870599884,28cyclictest27791-21cpu08:44:411
2870599884,28cyclictest27791-21cpu08:44:411
2870599883,30cyclictest30232-21modprobe07:14:261
2870599883,30cyclictest30232-21modprobe07:14:261
2870599883,29cyclictest30198-21latency10:46:111
2870599883,29cyclictest30198-21latency10:46:111
28712998755,10cyclictest181rcu_preempt11:34:592
2870599874,29cyclictest27942-21/usr/sbin/munin12:30:061
2870599874,29cyclictest27942-21/usr/sbin/munin12:30:061
2870599873,34cyclictest356-21fschecks_time07:19:481
2870599873,33cyclictest32552-21fschecks_time10:49:491
28712998661,14cyclictest9920-21latency_hist07:49:292
28712998661,14cyclictest9920-21latency_hist07:49:292
28712998656,21cyclictest2537-21latency_hist10:54:282
2870599864,36cyclictest21528-21apt-key09:39:281
2870599864,36cyclictest21528-21apt-key09:39:281
2870599864,29cyclictest8221-21/usr/sbin/munin07:45:271
2870599864,29cyclictest8221-21/usr/sbin/munin07:45:271
2870599864,27cyclictest23807-21mailstats10:35:011
2870599863,35cyclictest31305-21if_err_eth012:34:491
2870599863,33cyclictest24846-21acpi09:44:371
2870599863,32cyclictest343-21runrttasks08:37:311
2870599863,32cyclictest343-21runrttasks08:37:311
2870599863,32cyclictest22740-21latency_hist10:34:281
2870599863,32cyclictest22740-21latency_hist10:34:281
2870599863,32cyclictest18832-21threads08:15:001
2870599863,32cyclictest16455-21/usr/sbin/munin08:09:581
2870599863,31cyclictest29958-21/usr/sbin/munin11:39:471
2870599863,28cyclictest10741-21/usr/sbin/munin10:14:451
2870599862,31cyclictest14855-21unixbench_singl10:20:031
2870599862,31cyclictest14855-21unixbench_singl10:20:031
28712998558,19cyclictest23313-21kworker/u17:1-rpciod12:24:562
2870599854,38cyclictest11518-21cron11:09:261
2870599854,30cyclictest18348-21latency_hist09:34:281
2870599854,30cyclictest14589-21modprobe08:04:261
2870599854,27cyclictest12730-21cstates12:04:411
2870599854,27cyclictest12730-21cstates12:04:411
2870599853,35cyclictest7567-21unixbench_singl09:15:031
2870599853,30cyclictest30595-21sh09:54:261
2870599853,30cyclictest21882-21/usr/sbin/munin12:19:501
2870599853,30cyclictest18048-21expr11:19:431
2870599852,33cyclictest16304-21latency_hist08:09:291
2870599852,32cyclictest23111-21fschecks_count08:29:481
2870599844,32cyclictest11706-21sort07:54:441
2870599844,31cyclictest5488-21latency_hist10:59:291
2870599844,25cyclictest29664-21modprobe11:39:261
2870599843,32cyclictest13543-21grep09:25:011
2870599843,29cyclictest8185-21latency_hist07:44:281
2870599843,29cyclictest8185-21latency_hist07:44:281
2870599843,29cyclictest24687-21/usr/sbin/munin11:30:061
28712998365,11cyclictest16026-21kworker/u17:1-rpciod11:19:472
28712998354,13cyclictest181rcu_preempt11:53:002
2871299833,51cyclictest39-21ksoftirqd/207:29:332
2871299833,51cyclictest39-21ksoftirqd/207:29:332
2870599834,30cyclictest343-21runrttasks11:04:281
2870599834,29cyclictest3447-21/usr/sbin/munin09:09:531
2870599834,29cyclictest3447-21/usr/sbin/munin09:09:531
2870599833,34cyclictest31686-21switchtime08:54:591
2870599833,34cyclictest31686-21switchtime08:54:591
2870599833,32cyclictest5011-21fschecks_count10:04:471
2870599833,30cyclictest26185-21cpu08:39:441
2870599833,30cyclictest26185-21cpu08:39:441
2870599833,30cyclictest11824-21/usr/sbin/munin11:09:501
2870599833,29cyclictest21734-21latency08:24:511
2870599833,29cyclictest15302-21apt-get12:09:411
2870599833,29cyclictest15302-21apt-get12:09:411
2870599833,28cyclictest343-21runrttasks12:16:211
2870599833,26cyclictest343-21runrttasks11:49:571
28705998323,8cyclictest0-21swapper/109:40:201
28712998253,20cyclictest9177-21sh11:04:482
28712998247,16cyclictest181rcu_preempt11:15:022
28712998234,12cyclictest181rcu_preempt11:57:392
28712998234,12cyclictest181rcu_preempt11:57:392
2870599827,24cyclictest28687-21cyclictest07:09:301
2870599824,32cyclictest359-21snmpd10:10:071
2870599824,32cyclictest359-21snmpd10:10:071
2870599824,32cyclictest30470-21munin-node07:14:331
2870599823,31cyclictest343-21runrttasks08:23:001
2870599823,31cyclictest17370-21cstates10:25:161
2870599823,30cyclictest27093-21grep10:40:021
2870599823,30cyclictest27093-21grep10:40:021
2870599823,27cyclictest24887-21/usr/sbin/munin12:24:591
2870599822,34cyclictest6539-21/usr/sbin/munin11:54:351
2870599822,34cyclictest6539-21/usr/sbin/munin11:54:351
28712998154,10cyclictest181rcu_preempt09:45:492
2871299812,31cyclictest39-21ksoftirqd/207:14:382
28705998127,46cyclictest0-21swapper/111:27:281
28705998125,47cyclictest30714-21cron08:54:251
28705998125,47cyclictest30714-21cron08:54:251
28712998056,17cyclictest5181-21kernelversion10:04:512
28712998055,15cyclictest24717-21ssh12:24:292
28712998048,10cyclictest181rcu_preempt11:12:092
28712998042,9cyclictest181rcu_preempt09:34:292
28712998041,10cyclictest181rcu_preempt09:24:432
28712998039,9cyclictest181rcu_preempt10:25:352
28712998038,12cyclictest181rcu_preempt11:46:582
28712998038,12cyclictest181rcu_preempt11:46:582
28712998036,12cyclictest181rcu_preempt08:44:532
28712998036,12cyclictest181rcu_preempt08:44:532
2870599804,26cyclictest334-21fschecks_count08:59:481
28712997960,13cyclictest1042-21unixbench_singl07:20:022
28712997951,10cyclictest181rcu_preempt12:14:512
28712997941,10cyclictest181rcu_preempt08:24:372
2871299792,39cyclictest16889-21apt-config10:24:302
2871299792,39cyclictest16889-21apt-config10:24:302
28712997857,7cyclictest181rcu_preempt07:49:332
28712997856,8cyclictest181rcu_preempt12:35:012
28712997854,14cyclictest0-21swapper/211:29:542
28712997851,10cyclictest181rcu_preempt10:49:292
28712997851,10cyclictest181rcu_preempt10:49:292
28712997745,22cyclictest331-21latency_hist11:44:282
28712997739,8cyclictest181rcu_preempt08:33:512
2871299772,48cyclictest39-21ksoftirqd/210:10:402
2871299772,48cyclictest39-21ksoftirqd/210:10:402
28712997651,10cyclictest181rcu_preempt08:19:532
28712997636,17cyclictest181rcu_preempt12:05:012
28712997636,17cyclictest181rcu_preempt12:05:012
2871299762,46cyclictest39-21ksoftirqd/209:36:532
2871299762,46cyclictest39-21ksoftirqd/209:36:532
2871299754,41cyclictest39-21ksoftirqd/210:34:482
28712997543,9cyclictest181rcu_preempt08:59:482
28712997538,7cyclictest181rcu_preempt12:03:542
28712997524,10cyclictest181rcu_preempt12:31:552
28712997524,10cyclictest181rcu_preempt12:31:552
28712997517,23cyclictest0-21swapper/209:22:482
28712997517,23cyclictest0-21swapper/209:22:482
28712997514,43cyclictest39-21ksoftirqd/207:55:532
28712997454,12cyclictest14690-21kworker/u17:0+flush-179:008:14:362
28712997452,7cyclictest181rcu_preempt07:09:492
28712997452,7cyclictest181rcu_preempt07:09:492
28712997443,9cyclictest181rcu_preempt08:09:432
28712997437,7cyclictest181rcu_preempt09:50:422
28712997435,9cyclictest181rcu_preempt07:26:332
28712997427,8cyclictest181rcu_preempt11:26:152
28712997421,28cyclictest0-21swapper/209:06:242
28712997341,11cyclictest181rcu_preempt07:39:492
28712997341,11cyclictest181rcu_preempt07:39:492
28712997336,17cyclictest181rcu_preempt08:55:272
28712997336,17cyclictest181rcu_preempt08:55:272
28712997140,15cyclictest181rcu_preempt08:37:572
28712997140,15cyclictest181rcu_preempt08:37:572
28712997136,12cyclictest181rcu_preempt08:04:532
28712997041,13cyclictest181rcu_preempt08:02:492
28712996833,9cyclictest181rcu_preempt08:49:522
28712996833,9cyclictest181rcu_preempt08:49:522
28716996538,18cyclictest0-21swapper/309:39:423
2871699643,42cyclictest901-21kworker/3:1+events11:19:383
2871699642,39cyclictest901-21kworker/3:1+events12:10:133
2871699642,39cyclictest901-21kworker/3:1+events12:10:133
2871699604,40cyclictest27180-21taskset12:28:163
2871699593,39cyclictest6285-21chrt11:54:273
2871699584,39cyclictest17691-21chrt11:19:293
2871699582,41cyclictest8428-21taskset11:57:113
2871699582,41cyclictest8428-21taskset11:57:113
2871699582,38cyclictest13279-21chrt12:04:573
2871699582,38cyclictest13279-21chrt12:04:573
2871699573,40cyclictest974-21taskset10:50:143
2871699572,42cyclictest24602-21taskset10:36:183
2871699572,41cyclictest10921-21chrt10:14:423
2871699564,37cyclictest28027-21chrt08:44:473
2871699564,37cyclictest28027-21chrt08:44:473
2871699563,43cyclictest0-21swapper/307:34:413
2871699563,40cyclictest32398-21chrt09:56:133
2871699563,39cyclictest21445-21chrt12:19:083
2871699563,38cyclictest4118-21taskset09:09:583
2871699563,38cyclictest4118-21taskset09:09:583
2871699563,38cyclictest3448-21chrt07:29:443
2871699563,38cyclictest3448-21chrt07:29:443
2871699558,30cyclictest29336-21chrt10:44:423
2871699558,30cyclictest29336-21chrt10:44:423
2871699554,38cyclictest12201-21taskset07:54:593
2871699553,38cyclictest3080-21taskset10:54:483
2871699553,36cyclictest7921-21taskset07:41:473
2871699553,36cyclictest7921-21taskset07:41:473
2871699553,35cyclictest8473-21chrt10:10:043
2871699553,35cyclictest8473-21chrt10:10:043
2871699552,37cyclictest901-21kworker/3:1+events09:35:363
2871699552,37cyclictest901-21kworker/3:1+events09:35:363
2871699545,32cyclictest26406-21chrt08:39:493
2871699545,32cyclictest26406-21chrt08:39:493
2871699544,36cyclictest21243-21chrt11:24:483
2871699544,35cyclictest6316-21taskset10:06:193
2871699543,35cyclictest16965-21chrt10:24:313
2871699543,35cyclictest16965-21chrt10:24:313
2871699543,34cyclictest28221-21chrt11:35:343
2871699533,38cyclictest21829-21taskset10:31:573
2871699533,38cyclictest21829-21taskset10:31:573
2871699533,37cyclictest18889-21chrt10:27:063
2871699533,34cyclictest31811-21chrt12:35:013
2871699532,38cyclictest28817-21taskset07:09:313
2871699532,34cyclictest2687-21taskset10:00:433
2871699524,35cyclictest13829-21chrt07:59:573
2871699523,37cyclictest1726-21taskset11:45:153
2871699523,37cyclictest1726-21taskset11:45:153
2871699523,33cyclictest6813-21taskset09:14:483
2871699523,33cyclictest30414-21chrt07:14:293
2871699523,33cyclictest30414-21chrt07:14:293
2871699523,33cyclictest26208-21chrt10:39:433
2871699523,33cyclictest26208-21chrt10:39:433
2871699523,33cyclictest10649-21chrt09:20:033
2871699523,33cyclictest10649-21chrt09:20:033
2871699522,36cyclictest13250-21chrt09:24:573
2871699522,33cyclictest32587-21taskset08:59:293
2871699522,33cyclictest16278-21chrt08:09:283
2871699513,34cyclictest22442-21taskset12:19:533
2871699513,32cyclictest25248-21chrt09:44:503
2871699513,32cyclictest24144-21chrt11:29:503
28716995126,13cyclictest0-21swapper/308:13:253
2871699512,37cyclictest31029-21chrt07:14:453
2871699512,36cyclictest29162-21taskset12:30:233
2871699512,36cyclictest29162-21taskset12:30:233
2871699503,33cyclictest30455-21chrt09:54:113
2871699503,32cyclictest20001-21taskset08:19:493
2871699503,31cyclictest24643-21chrt08:34:433
2871699503,31cyclictest24643-21chrt08:34:433
2871699493,34cyclictest31439-21chrt11:41:013
2871699493,32cyclictest2886-21chrt07:27:253
2871699482,34cyclictest13527-21chrt11:11:393
2871699482,33cyclictest31268-21chrt08:54:483
2871699482,33cyclictest1927-21chrt09:04:443
2871699482,32cyclictest11661-21taskset12:02:423
2871699473,30cyclictest1030-21chrt09:00:023
2871699472,34cyclictest18768-21chrt08:14:593
2871699453,31cyclictest10970-21taskset11:07:543
2871699453,30cyclictest29424-21taskset08:49:413
2871699453,30cyclictest29424-21taskset08:49:413
2871699453,29cyclictest11374-21taskset07:54:263
2871699453,26cyclictest15646-21chrt09:29:393
2871699444,29cyclictest6981-21chrt11:00:113
2871699443,30cyclictest23394-21chrt08:29:533
2871699443,30cyclictest21424-21taskset08:24:433
2871699403,27cyclictest8956-21taskset07:45:313
2871699403,27cyclictest8956-21taskset07:45:313
2871699403,25cyclictest1632-21chrt07:24:293
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional