You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-17 - 04:13
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Fri Apr 17, 2026 00:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
94619911654,55cyclictest17351-21/usr/sbin/munin21:03:270
9461991155,51cyclictest25250-21latency_hist23:58:050
9461991126,36cyclictest7059-21latency_hist23:28:050
9461991126,36cyclictest7059-21latency_hist23:28:050
94719911156,15cyclictest1728-21sh23:18:251
9461991093,27cyclictest15331-21apt-get22:48:160
9461991093,22cyclictest14686-21ntpq20:54:260
9461991092,26cyclictest19014-21cstates22:53:180
9461991083,34cyclictest23482-21cpu21:13:190
9461991083,34cyclictest23482-21cpu21:13:190
9461991073,74cyclictest29910-21modprobe20:03:220
94619910630,9cyclictest0-21swapper/021:23:380
94619910630,9cyclictest0-21swapper/021:23:380
9461991062,34cyclictest18296-21fschecks_count19:28:220
9461991062,34cyclictest18296-21fschecks_count19:28:220
94619910540,58cyclictest0-21swapper/020:13:010
94619910535,8cyclictest0-21swapper/022:38:030
9461991053,29cyclictest4554-21runrttasks21:35:090
9461991053,27cyclictest22153-21fschecks_count22:58:230
9461991053,27cyclictest22153-21fschecks_count22:58:230
94619910528,9cyclictest0-21swapper/022:33:010
9461991052,31cyclictest23080-21latency_hist21:13:050
9461991044,40cyclictest12817-21runrttasks00:31:060
9461991043,30cyclictest8483-21rm23:28:400
94619910428,9cyclictest0-21swapper/021:58:240
94619910421,15cyclictest0-21swapper/023:35:140
9461991034,29cyclictest2544-21runrttasks20:18:130
9461991033,37cyclictest31771-21cat00:08:280
9461991033,37cyclictest31771-21cat00:08:280
9461991033,23cyclictest28186-21sh00:03:040
94619910331,10cyclictest0-21swapper/021:38:300
9461991032,36cyclictest15832-21sed21:53:590
9461991032,36cyclictest15832-21sed21:53:590
9461991024,35cyclictest13262-21ntp_offset22:43:320
9461991023,33cyclictest12466-21latency_hist22:43:040
94619910227,16cyclictest0-21swapper/020:43:380
94619910217,28cyclictest19749-21sort19:33:120
94619910217,28cyclictest19749-21sort19:33:120
94619910136,58cyclictest0-21swapper/019:13:370
94619910134,8cyclictest0-21swapper/022:14:560
9461991013,34cyclictest29200-21ssh00:03:380
94619910118,28cyclictest28999-21sort19:59:400
94619910116,24cyclictest9161-21fschecks_count21:43:240
9461991004,30cyclictest11873-21/usr/sbin/munin21:48:380
9461991004,30cyclictest11873-21/usr/sbin/munin21:48:380
9461991003,35cyclictest1692-21fschecks_time23:18:240
9461991003,33cyclictest27552-21latency_hist19:58:050
9461991003,32cyclictest22149-21ssh22:03:390
9461991003,32cyclictest22149-21ssh22:03:390
94619910027,11cyclictest0-21swapper/022:23:360
94619910027,11cyclictest0-21swapper/022:23:360
94619910025,14cyclictest0-21swapper/023:38:170
94619910025,14cyclictest0-21swapper/023:38:170
94619910025,11cyclictest0-21swapper/021:33:040
9461999930,9cyclictest0-21swapper/020:48:220
946199992,36cyclictest31384-21ssh23:13:280
947799984,45cyclictest39-21ksoftirqd/221:34:312
946199984,32cyclictest30734-21ntpq22:18:320
946199983,37cyclictest9589-21latency_hist19:03:070
946199983,36cyclictest25115-21fschecks_count23:03:230
946199983,32cyclictest16427-21switchtime20:58:370
9461999826,11cyclictest0-21swapper/023:48:030
946199982,40cyclictest6610-21timerwakeupswit20:28:380
946199982,36cyclictest21977-21grep19:38:350
9461999813,27cyclictest0-21swapper/020:38:040
946199974,33cyclictest8695-21ssh00:23:370
946199973,33cyclictest24770-21cut19:48:230
946199972,37cyclictest13197-21latency_hist19:13:050
946199972,36cyclictest16366-21latency_hist19:23:040
946199972,31cyclictest26752-21taskset22:12:550
946199972,19cyclictest19632-21fschecks_time23:48:240
946199963,35cyclictest16780-21fschecks_time19:23:240
946199963,27cyclictest4132-21/usr/sbin/munin20:23:300
946199963,27cyclictest1256-21grep20:15:210
946199963,27cyclictest1256-21grep20:15:210
9461999622,20cyclictest30789-21latency_hist23:13:050
946199953,30cyclictest352-21runrttasks00:13:280
946199953,30cyclictest10350-21awk20:43:010
946199952,33cyclictest6109-21ssh00:19:010
946199952,33cyclictest6109-21ssh00:19:010
946199952,32cyclictest29229-21latency_hist21:23:040
946199942,29cyclictest22947-21/usr/sbin/munin19:43:220
946199933,28cyclictest11089-21latency_hist19:08:040
946199933,28cyclictest11089-21latency_hist19:08:040
9477998955,17cyclictest181rcu_preempt21:23:372
9477998955,17cyclictest181rcu_preempt21:23:372
9471998838,9cyclictest17059-21diskmemload22:53:491
9471998768,10cyclictest20850-21ssh22:03:021
9477998660,20cyclictest8802-21latency_hist20:38:042
9477998537,16cyclictest181rcu_preempt21:12:002
947799853,51cyclictest39-21ksoftirqd/222:28:232
9471998556,9cyclictest31-21ksoftirqd/122:36:101
947199854,56cyclictest15208-21ssh21:53:221
947199854,56cyclictest15208-21ssh21:53:221
948599842,38cyclictest26861-21chrt21:18:253
9477998452,15cyclictest181rcu_preempt21:05:132
9477998367,10cyclictest15351-21ls21:53:372
9477998367,10cyclictest15351-21ls21:53:372
9477998351,9cyclictest181rcu_preempt21:38:212
9477998342,17cyclictest181rcu_preempt22:12:042
9477998260,15cyclictest13619-21apt-key20:53:032
9477998256,18cyclictest26513-21sh23:05:092
9477998158,8cyclictest181rcu_preempt20:10:132
9477998157,14cyclictest13241-21latency_hist23:38:052
9477998132,34cyclictest7242-21cat23:28:042
9477998132,34cyclictest7242-21cat23:28:042
9471998150,8cyclictest31-21ksoftirqd/119:08:211
948599804,32cyclictest20664-21chrt21:08:243
948599803,40cyclictest6344-21taskset22:33:013
9477998061,13cyclictest11587-21/usr/sbin/munin21:48:032
9471998063,11cyclictest13974-21latency_hist20:53:061
9471998058,16cyclictest25994-21open_files23:58:331
9471998053,21cyclictest27920-21latency_hist19:58:051
9477997965,10cyclictest9452-21cyclictest23:00:132
9477997965,10cyclictest9452-21cyclictest23:00:132
9477997957,15cyclictest10050-21sendmail23:33:012
9477997953,17cyclictest27670-21apt-key19:58:042
9477997951,8cyclictest181rcu_preempt19:52:172
9477997946,11cyclictest181rcu_preempt20:47:102
9477997944,13cyclictest181rcu_preempt23:10:402
9477997942,12cyclictest181rcu_preempt20:23:052
9477997939,10cyclictest181rcu_preempt19:40:032
9477997936,14cyclictest181rcu_preempt23:46:292
9471997946,10cyclictest181rcu_preempt22:13:151
9471997944,8cyclictest181rcu_preempt21:33:341
947799789,36cyclictest39-21ksoftirqd/222:53:052
9477997837,18cyclictest181rcu_preempt19:27:312
9477997836,12cyclictest181rcu_preempt00:32:142
9477997814,27cyclictest39-21ksoftirqd/222:37:032
9471997856,16cyclictest9556-21latency_hist19:03:061
9471997837,9cyclictest181rcu_preempt21:13:011
9477997742,14cyclictest181rcu_preempt20:00:322
9477997740,14cyclictest181rcu_preempt21:15:182
9477997740,14cyclictest181rcu_preempt21:15:182
9477997737,11cyclictest181rcu_preempt20:58:362
9477997731,24cyclictest39-21ksoftirqd/220:56:132
9477997658,12cyclictest298-21in:imuxsock22:43:012
9477997655,14cyclictest11566-21kworker/u17:0-rpciod22:00:072
9477997651,9cyclictest181rcu_preempt19:03:202
9477997651,9cyclictest181rcu_preempt19:03:202
9477997651,10cyclictest181rcu_preempt19:31:532
9477997651,10cyclictest181rcu_preempt19:31:532
9477997649,16cyclictest31782-21diskmemload00:08:282
9477997649,16cyclictest31782-21diskmemload00:08:282
9477997636,17cyclictest181rcu_preempt23:39:312
9477997636,17cyclictest181rcu_preempt23:39:312
9471997661,11cyclictest9452-21cyclictest22:22:361
9471997661,10cyclictest9452-21cyclictest21:38:271
9471997650,17cyclictest20056-21latency_hist21:08:041
9477997556,10cyclictest0-21swapper/221:28:312
9477997551,13cyclictest0-21swapper/200:23:012
9477997551,13cyclictest0-21swapper/200:23:012
947799752,48cyclictest39-21ksoftirqd/200:00:002
947799752,38cyclictest39-21ksoftirqd/200:03:072
9471997561,10cyclictest9452-21cyclictest21:28:391
9471997557,11cyclictest365-21ntpd23:08:321
947799744,32cyclictest39-21ksoftirqd/221:48:252
947799744,32cyclictest39-21ksoftirqd/221:48:252
9477997439,12cyclictest181rcu_preempt00:15:282
947799742,49cyclictest39-21ksoftirqd/220:23:302
947799742,49cyclictest39-21ksoftirqd/200:27:212
9477997410,30cyclictest39-21ksoftirqd/222:13:392
9471997457,11cyclictest22062-40mandb23:53:051
9471997451,17cyclictest7067-21idleruntime-cro23:28:011
9471997451,17cyclictest7067-21idleruntime-cro23:28:011
948599734,28cyclictest30010-21taskset22:18:053
947799734,49cyclictest19919-21ntp_kernel_pll_23:48:302
947799733,31cyclictest39-21ksoftirqd/220:03:572
9477997312,36cyclictest39-21ksoftirqd/220:13:572
9477997312,36cyclictest39-21ksoftirqd/220:13:572
9471997355,12cyclictest277-21dbus-daemon23:38:021
9477997234,16cyclictest181rcu_preempt19:35:382
9477997234,16cyclictest181rcu_preempt19:35:382
947799722,39cyclictest39-21ksoftirqd/219:43:182
9471997255,12cyclictest9452-21cyclictest23:13:401
934627210,37sleep29363-21munin-run19:03:042
948599713,31cyclictest5086-21chrt23:23:323
948599713,31cyclictest5086-21chrt23:23:323
9477997147,8cyclictest181rcu_preempt19:13:352
9477997142,8cyclictest181rcu_preempt20:38:102
947799713,46cyclictest39-21ksoftirqd/223:18:292
9477997132,7cyclictest181rcu_preempt22:18:362
947799713,24cyclictest39-21ksoftirqd/219:23:052
9471997158,9cyclictest32534-21kworker/u17:3-rpciod21:47:131
9471997155,10cyclictest4623-21sshd00:18:031
9471997154,11cyclictest58-21kcompactd019:38:191
9471997154,11cyclictest18571-21sendmail22:53:011
9471997153,12cyclictest23470-21ntp_kernel_pll_19:43:291
881927121,27sleep30-21swapper/318:58:493
9477997051,14cyclictest9452-21cyclictest23:53:252
9477997046,10cyclictest181rcu_preempt22:04:512
9477997046,10cyclictest181rcu_preempt22:04:512
9477997040,9cyclictest181rcu_preempt22:53:112
947799703,48cyclictest277-21dbus-daemon19:13:012
947799703,47cyclictest6271-21ntp_kernel_pll_20:28:302
9477997013,24cyclictest39-21ksoftirqd/222:23:202
9477997013,24cyclictest39-21ksoftirqd/222:23:202
9471997056,10cyclictest9452-21cyclictest19:28:221
9471997056,10cyclictest9452-21cyclictest19:28:221
9471997053,11cyclictest5912-21ssh00:18:401
9471997053,11cyclictest5912-21ssh00:18:401
9471997043,21cyclictest10505-21latency_hist20:43:051
947199703,25cyclictest29374-21apt-get21:23:121
947199703,25cyclictest29374-21apt-get21:23:121
947199703,20cyclictest27127-21ssh21:18:331
9471997027,9cyclictest181rcu_preempt19:03:131
9471997027,9cyclictest181rcu_preempt19:03:131
947199702,25cyclictest15132-21fschecks_time19:18:221
947799692,30cyclictest39-21ksoftirqd/221:20:292
9471996952,11cyclictest277-21dbus-daemon20:13:031
9471996951,12cyclictest2374-21systemd-journal21:14:541
9471996951,12cyclictest2374-21systemd-journal21:14:541
947199693,26cyclictest28054-21apt-get19:58:151
947799683,27cyclictest39-21ksoftirqd/222:43:392
947199684,43cyclictest352-21runrttasks20:22:071
9471996844,15cyclictest13648-21fschecks_time23:38:231
9471996844,15cyclictest13648-21fschecks_time23:38:231
947199684,39cyclictest8213-21timerwakeupswit20:33:371
947199683,20cyclictest26958-21ssh23:06:241
9477996746,8cyclictest181rcu_preempt23:17:022
9471996732,12cyclictest181rcu_preempt00:28:121
244012670,6sleep30-21swapper/323:02:373
244012670,6sleep30-21swapper/323:02:373
9471996650,11cyclictest2374-21systemd-journal22:48:041
9471996648,11cyclictest7158-21systemd-run20:33:021
9471996643,12cyclictest0-21swapper/120:08:011
947199663,43cyclictest19739-21/usr/sbin/munin19:33:281
947199663,43cyclictest19739-21/usr/sbin/munin19:33:281
9471996543,8cyclictest181rcu_preempt00:09:401
9471996543,8cyclictest181rcu_preempt00:09:401
9471996541,16cyclictest0-21swapper/122:28:171
9471996540,17cyclictest12698-21switchtime21:48:371
9471996540,17cyclictest12698-21switchtime21:48:371
9471996538,20cyclictest17299-21ssh23:43:401
9471996523,8cyclictest181rcu_preempt22:03:371
9471996523,8cyclictest181rcu_preempt22:03:371
948599645,28cyclictest0-21swapper/319:58:183
9471996451,9cyclictest58-21kcompactd022:11:071
9471996446,11cyclictest277-21dbus-daemon20:28:021
948599632,30cyclictest0-21swapper/320:53:273
9485996313,37cyclictest152182sleep321:53:253
9485996313,37cyclictest152182sleep321:53:253
9471996346,10cyclictest14251-21sed19:13:381
9471996340,17cyclictest352-21runrttasks22:59:421
9471996340,17cyclictest352-21runrttasks22:59:421
9471996325,9cyclictest1345-21wc22:23:351
9471996325,9cyclictest1345-21wc22:23:351
9471996243,14cyclictest0-21swapper/123:28:381
947199623,42cyclictest28316-21/usr/sbin/munin00:03:351
947199623,41cyclictest352-21runrttasks20:44:141
947199622,42cyclictest15532-21latency_hist20:58:041
9471996144,11cyclictest0-21swapper/122:38:271
9471996139,8cyclictest181rcu_preempt20:18:021
9471996139,8cyclictest181rcu_preempt20:18:021
9471996134,19cyclictest10648-21sh00:28:041
9471996044,9cyclictest0-21swapper/123:53:311
9471996041,13cyclictest25505-21unixbench_singl19:48:381
9471996030,8cyclictest181rcu_preempt19:23:251
948599593,41cyclictest861-21chrt20:13:073
9471995830,11cyclictest181rcu_preempt20:59:241
948599573,37cyclictest19727-21taskset22:53:373
948599572,40cyclictest1621-21chrt23:18:223
948599553,41cyclictest28725-21chrt23:08:373
948599553,38cyclictest11577-21chrt22:40:563
948599543,34cyclictest8938-21taskset21:43:183
948599543,33cyclictest24928-21chrt22:08:363
948599542,35cyclictest30265-21taskset20:04:043
948599533,37cyclictest26241-21taskset21:18:053
948599533,37cyclictest26241-21taskset21:18:053
948599533,36cyclictest3936-21chrt20:23:033
948599533,34cyclictest3469-21chrt21:33:293
948599532,39cyclictest7455-21taskset21:39:543
948599532,37cyclictest9772-21chrt20:38:343
948599524,35cyclictest19205-21chrt21:58:393
948599523,37cyclictest441-21chrt22:23:043
948599523,35cyclictest22949-21chrt23:53:283
948599523,35cyclictest16651-21chrt20:58:393
948599523,32cyclictest19899-21chrt23:48:303
948599522,32cyclictest21426-21taskset19:38:193
948599514,34cyclictest12194-21taskset23:35:373
948599512,35cyclictest31610-21chrt23:13:353
948599504,34cyclictest12631-21chrt20:48:273
948599504,30cyclictest25605-21taskset23:58:233
948599503,34cyclictest25550-21taskset23:03:343
948599503,29cyclictest4329-21taskset20:23:243
948599502,34cyclictest15627-21chrt23:41:533
948599502,34cyclictest15627-21chrt23:41:533
948599502,31cyclictest6626-21chrt22:33:093
948599502,30cyclictest3066-21chrt00:13:513
948599498,27cyclictest18938-21chrt21:04:553
948599494,33cyclictest18378-21taskset19:28:233
948599494,33cyclictest18378-21taskset19:28:233
948599493,35cyclictest12266-21chrt00:29:323
948599483,33cyclictest5398-21chrt00:18:303
948599483,33cyclictest5398-21chrt00:18:303
948599483,29cyclictest1979-21chrt21:31:223
948599473,32cyclictest3365-21chrt22:28:023
948599473,32cyclictest3365-21chrt22:28:023
948599473,32cyclictest16363-21taskset19:23:043
948599473,30cyclictest29455-21chrt00:03:423
948599472,32cyclictest30684-21chrt21:24:133
948599472,32cyclictest30684-21chrt21:24:133
948599472,32cyclictest27722-21taskset19:58:043
948599472,30cyclictest24850-21chrt19:48:243
948599464,30cyclictest7512-21chrt23:28:193
948599463,30cyclictest8108-21taskset00:23:243
948599453,29cyclictest15880-21chrt22:48:133
948599453,29cyclictest12856-21chrt22:43:233
948599433,29cyclictest13401-21chrt21:49:473
948599433,29cyclictest13401-21chrt21:49:473
948599433,28cyclictest759-21chrt00:10:513
948599433,28cyclictest759-21chrt00:10:513
948599433,26cyclictest23544-21chrt22:07:193
948599433,26cyclictest23544-21chrt22:07:193
948599432,29cyclictest17623-21taskset19:25:113
948599432,28cyclictest18892-21chrt23:47:403
948599432,27cyclictest2284-21chrt20:18:013
948599432,27cyclictest2284-21chrt20:18:013
948599422,28cyclictest7398-21taskset20:33:163
948599413,29cyclictest11842-21taskset20:46:443
948599412,27cyclictest9637-21taskset19:03:163
948599412,27cyclictest9637-21taskset19:03:163
948599412,27cyclictest11816-21taskset19:08:223
948599403,25cyclictest5735-21taskset20:28:093
948599392,26cyclictest19765-21taskset19:33:173
948599392,26cyclictest19765-21taskset19:33:173
948599383,26cyclictest23999-21taskset19:44:153
948599383,24cyclictest13282-21chrt19:13:083
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional