You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-20 - 06:03
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Fri Mar 20, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32629212711,86sleep232630-21sshd19:03:402
415991193,36cyclictest2968-21latency21:50:200
415991193,36cyclictest2968-21latency21:50:200
415991183,30cyclictest15507-21grep22:10:300
4159911526,12cyclictest0-21swapper/020:55:540
4159911446,52cyclictest404-21cyclictest23:47:290
4159911243,62cyclictest1-21systemd23:43:540
4159911243,62cyclictest1-21systemd23:43:540
4159911223,12cyclictest0-21swapper/021:40:210
415991102,32cyclictest29137-21fschecks_count20:30:130
415991093,32cyclictest24021-21latency_hist23:19:570
415991073,32cyclictest23021-21ssh21:30:040
415991073,32cyclictest23021-21ssh21:30:040
415991073,23cyclictest14257-21taskset23:57:260
415991054,31cyclictest352-21runrttasks20:35:380
4159910539,60cyclictest0-21swapper/022:34:530
415991053,33cyclictest16191-21timerandwakeup23:05:470
415991052,33cyclictest15536-21/usr/sbin/munin19:50:140
415991044,30cyclictest11055-21/usr/sbin/munin21:10:250
415991043,44cyclictest21994-21latency_hist20:09:580
415991043,32cyclictest7777-21latency_hist21:04:560
415991043,32cyclictest17046-21/usr/sbin/munin21:20:290
415991042,39cyclictest12202-21latency_hist19:39:560
4159910416,9cyclictest2084-21unixbench_multi23:36:190
415991033,35cyclictest2743-21latency_hist22:44:560
415991023,33cyclictest30448-21fschecks_count23:30:140
415991023,33cyclictest30448-21fschecks_count23:30:140
415991023,33cyclictest11817-21idleruntime22:05:160
415991023,28cyclictest18636-21fschecks_time00:05:150
415991022,22cyclictest25967-21taskset00:17:290
415991022,22cyclictest25967-21taskset00:17:290
4159910221,12cyclictest15350irq/295-2188000.ethernet22:37:130
4159910221,12cyclictest15350irq/295-2188000.ethernet22:37:130
31569210213,66sleep031623-21fschecks_count19:00:200
415991019,37cyclictest20064-21kworker/u17:3+rpciod21:39:530
415991014,33cyclictest352-21runrttasks21:16:500
415991014,33cyclictest352-21runrttasks21:16:500
415991013,40cyclictest12208-40mandb23:54:550
415991013,35cyclictest23956-21latency_hist00:14:560
415991013,35cyclictest19461-21grep20:00:200
415991013,33cyclictest14773-21cat19:45:280
415991013,29cyclictest23760-21fschecks_time22:25:150
415991013,29cyclictest23760-21fschecks_time22:25:150
4159910130,8cyclictest0-21swapper/019:15:130
415991012,41cyclictest18469-21grep19:56:420
415991012,41cyclictest18469-21grep19:56:420
415991012,35cyclictest24067-21/usr/sbin/munin23:20:210
415991012,33cyclictest30032-21apt-get00:25:040
415991003,33cyclictest712-21unixbench_singl20:40:300
415991003,33cyclictest3218-21fschecks_time20:50:160
415991003,32cyclictest27630-21fschecks_count23:25:360
415991002,36cyclictest10444-21latency_hist19:34:560
415991002,35cyclictest22883-21latency_hist21:29:570
415991002,35cyclictest22883-21latency_hist21:29:570
415991002,25cyclictest16363-21timerwakeupswit00:00:290
415991002,25cyclictest16363-21timerwakeupswit00:00:290
41599993,37cyclictest19004-21/usr/sbin/munin23:10:280
41599993,33cyclictest8833-21sh19:29:530
41599993,33cyclictest8833-21sh19:29:530
41599993,30cyclictest8276-21latency_hist21:59:560
41599993,29cyclictest14777-21apt-key23:04:550
415999931,7cyclictest0-21swapper/021:05:100
415999931,10cyclictest0-21swapper/022:50:190
415999926,11cyclictest0-21swapper/019:05:040
415999917,8cyclictest126032fschecks_time19:40:150
41599984,30cyclictest23124-21cron22:24:530
41599983,35cyclictest3542-21grep19:10:320
41599983,34cyclictest22876-21kernelversion20:10:320
41599983,33cyclictest2277-21latency_hist21:49:550
41599983,32cyclictest3212-21rm22:45:150
41599983,32cyclictest27422-21fschecks_time00:20:160
41599983,32cyclictest26155-21latency20:20:200
41599983,32cyclictest26155-21latency20:20:200
41599983,30cyclictest8526-21cpu22:00:100
41599983,29cyclictest352-21runrttasks22:15:040
41599983,29cyclictest352-21runrttasks22:15:040
41599983,29cyclictest1458-21cstates20:45:110
41599982,33cyclictest6380-21awk19:20:220
41599982,30cyclictest27333-21/usr/sbin/munin20:25:150
41599963,34cyclictest11629-21sed22:59:560
41599963,32cyclictest927-21uname00:30:180
41599963,32cyclictest927-21uname00:30:180
41599942,20cyclictest24657-21/usr/sbin/munin20:15:220
324982936,16sleep10-21swapper/119:01:561
42699904,76cyclictest27278-21apt-get23:25:162
42199893,72cyclictest12154-21latency_hist23:54:561
426998560,16cyclictest2700-21sh00:33:432
426998560,16cyclictest2700-21sh00:33:432
426998450,10cyclictest181rcu_preempt21:10:592
42199847,48cyclictest31-21ksoftirqd/123:03:301
42699832,37cyclictest3883-21tail20:50:302
42199834,29cyclictest31-21ksoftirqd/121:45:211
421998322,29cyclictest31-21ksoftirqd/120:33:531
42199831,52cyclictest31-21ksoftirqd/121:50:511
42199831,52cyclictest31-21ksoftirqd/121:50:511
421998259,15cyclictest5678-21sh22:49:561
421998234,35cyclictest11135-21latency19:35:181
42199822,34cyclictest31-21ksoftirqd/122:30:141
426998159,16cyclictest26941-21latency_hist00:19:562
426998159,16cyclictest26941-21latency_hist00:19:562
426998158,14cyclictest11340-21kworker/u17:4-rpciod22:25:182
426998158,14cyclictest11340-21kworker/u17:4-rpciod22:25:182
426998150,10cyclictest181rcu_preempt22:55:082
421998158,15cyclictest28123-21mailstats23:25:491
42199812,33cyclictest31-21ksoftirqd/119:11:241
426998041,18cyclictest181rcu_preempt20:43:142
426998010,61cyclictest381ktimers/220:15:042
426997937,9cyclictest181rcu_preempt21:39:562
421997964,10cyclictest30499-21fschecks_time23:30:151
421997964,10cyclictest30499-21fschecks_time23:30:151
42199796,35cyclictest31-21ksoftirqd/123:46:431
421997954,18cyclictest1-21systemd22:19:541
421997954,18cyclictest1-21systemd22:19:541
421997951,11cyclictest181rcu_preempt00:21:521
421997950,7cyclictest181rcu_preempt23:14:201
42199792,41cyclictest31-21ksoftirqd/119:20:351
42199792,35cyclictest31-21ksoftirqd/121:24:321
421997863,9cyclictest30374-21kworker/u17:3-rpciod00:06:281
42199785,34cyclictest31-21ksoftirqd/120:54:421
421997840,18cyclictest181rcu_preempt22:58:001
42199782,40cyclictest31-21ksoftirqd/123:56:091
421997811,45cyclictest31-21ksoftirqd/123:17:551
426997760,9cyclictest22440-21ssh22:23:002
426997755,15cyclictest14479-21sh22:10:082
426997748,8cyclictest181rcu_preempt23:30:222
426997748,8cyclictest181rcu_preempt23:30:222
42199778,34cyclictest31-21ksoftirqd/119:46:081
421997749,12cyclictest181rcu_preempt19:58:121
421997749,12cyclictest181rcu_preempt19:58:121
42199774,32cyclictest31-21ksoftirqd/122:52:101
421997743,10cyclictest181rcu_preempt20:47:281
42199773,52cyclictest4836-21apt-get20:55:071
421997735,12cyclictest181rcu_preempt21:29:451
421997735,12cyclictest181rcu_preempt21:29:451
42199773,31cyclictest31-21ksoftirqd/121:10:591
42199772,48cyclictest31-21ksoftirqd/120:16:391
426997659,11cyclictest8837-21sh22:00:172
426997643,9cyclictest181rcu_preempt22:53:322
426997632,12cyclictest2545-21cpu21:50:102
426997632,12cyclictest2545-21cpu21:50:102
421997654,12cyclictest0-21swapper/120:29:531
421997652,8cyclictest181rcu_preempt21:02:551
42199763,40cyclictest31-21ksoftirqd/122:44:211
42199763,36cyclictest31-21ksoftirqd/123:40:151
42199763,36cyclictest31-21ksoftirqd/123:40:151
421997631,29cyclictest31-21ksoftirqd/122:27:251
421997631,29cyclictest31-21ksoftirqd/122:27:251
42199762,39cyclictest31-21ksoftirqd/119:25:081
42199762,39cyclictest31-21ksoftirqd/119:25:081
426997556,14cyclictest12755-21fschecks_count23:55:172
426997553,16cyclictest29233-21ssh21:40:152
42199756,45cyclictest31-21ksoftirqd/120:12:591
421997543,11cyclictest181rcu_preempt21:59:091
421997536,17cyclictest181rcu_preempt21:09:351
426997457,11cyclictest1423-21timerwakeupswit19:05:282
426997451,17cyclictest22282-21latency_hist20:09:572
426997445,8cyclictest181rcu_preempt20:58:392
426997442,15cyclictest4845-21ssh23:40:372
426997442,15cyclictest4845-21ssh23:40:372
426997428,13cyclictest181rcu_preempt21:56:192
421997458,10cyclictest19676-21snmpd23:22:441
421997455,14cyclictest19274-21fschecks_time20:00:161
421997449,9cyclictest181rcu_preempt19:17:411
421997448,11cyclictest181rcu_preempt21:30:341
421997448,11cyclictest181rcu_preempt21:30:341
42199742,37cyclictest31-21ksoftirqd/122:01:191
421997417,45cyclictest15199-21apt-get23:05:111
42699738,10cyclictest181rcu_preempt21:05:052
426997335,19cyclictest0-21swapper/223:15:202
42699732,55cyclictest20600-21ntpq21:25:222
42699732,55cyclictest20600-21ntpq21:25:222
42199736,33cyclictest31-21ksoftirqd/120:09:351
421997348,9cyclictest181rcu_preempt19:52:191
421997333,10cyclictest181rcu_preempt19:06:171
42199733,26cyclictest31-21ksoftirqd/121:37:091
42199732,27cyclictest31-21ksoftirqd/120:20:161
42199732,27cyclictest31-21ksoftirqd/120:20:161
421997317,28cyclictest31-21ksoftirqd/122:11:311
426997242,13cyclictest181rcu_preempt22:35:262
426997242,13cyclictest181rcu_preempt22:35:262
426997234,10cyclictest404-21cyclictest21:14:592
426997234,10cyclictest404-21cyclictest21:14:592
42199727,36cyclictest31-21ksoftirqd/100:26:161
42199726,37cyclictest31-21ksoftirqd/100:02:121
42199726,37cyclictest31-21ksoftirqd/100:02:121
421997249,8cyclictest181rcu_preempt20:42:021
421997242,8cyclictest181rcu_preempt00:15:241
421997242,8cyclictest181rcu_preempt00:15:241
421997229,14cyclictest31-21ksoftirqd/122:20:111
421997211,33cyclictest31-21ksoftirqd/100:34:031
421997211,33cyclictest31-21ksoftirqd/100:34:031
426997151,12cyclictest58-21kcompactd019:55:152
426997151,12cyclictest58-21kcompactd019:55:152
426997150,12cyclictest21999-21ssh00:10:282
42699714,57cyclictest12018-21apt-get23:00:112
426997144,18cyclictest19484-21sh00:05:402
426997132,9cyclictest181rcu_preempt23:50:302
42199716,38cyclictest31-21ksoftirqd/100:12:211
421997149,15cyclictest7788-21diskmemload21:40:301
421997148,8cyclictest181rcu_preempt22:05:121
421997148,8cyclictest181rcu_preempt21:15:181
421997148,8cyclictest181rcu_preempt21:15:181
421997137,7cyclictest181rcu_preempt23:36:141
426997039,9cyclictest181rcu_preempt20:10:282
426997038,7cyclictest181rcu_preempt23:11:492
426997035,8cyclictest181rcu_preempt20:44:572
42199702,44cyclictest31-21ksoftirqd/120:35:091
42199702,30cyclictest31-21ksoftirqd/119:30:091
426996954,10cyclictest1864-21switchtime23:36:172
426996943,8cyclictest181rcu_preempt22:45:112
426996938,8cyclictest181rcu_preempt21:01:172
426996930,10cyclictest181rcu_preempt21:22:402
426996910,19cyclictest352-21runrttasks00:20:312
42199693,27cyclictest31-21ksoftirqd/122:36:141
42199693,27cyclictest31-21ksoftirqd/122:36:141
426996844,9cyclictest181rcu_preempt19:10:312
426996843,11cyclictest181rcu_preempt19:15:042
426996839,8cyclictest181rcu_preempt23:46:262
42699683,38cyclictest24105-21rm23:20:052
426996832,13cyclictest181rcu_preempt23:09:382
426996829,8cyclictest181rcu_preempt22:31:242
426996818,20cyclictest32200-21ssh21:45:152
42199682,27cyclictest31-21ksoftirqd/119:40:101
426996751,9cyclictest0-21swapper/222:40:192
426996738,9cyclictest181rcu_preempt22:19:312
426996738,9cyclictest181rcu_preempt22:19:312
426996738,10cyclictest181rcu_preempt19:36:022
426996731,7cyclictest181rcu_preempt21:30:162
426996731,7cyclictest181rcu_preempt21:30:162
426996712,41cyclictest17174-21gpgv19:54:572
317102679,32sleep30-21swapper/319:00:233
426996650,10cyclictest9606-21ntp_kernel_pll_19:30:212
426996644,14cyclictest16084-21processes00:00:252
426996644,14cyclictest16084-21processes00:00:252
42699663,28cyclictest0-21swapper/200:27:352
426996546,12cyclictest0-21swapper/220:35:162
426996540,18cyclictest0-21swapper/220:24:562
426996540,18cyclictest0-21swapper/220:24:562
42699652,46cyclictest352-21runrttasks19:21:332
43199648,38cyclictest0-21swapper/323:00:343
426996431,25cyclictest14692-21switchtime19:45:272
42699642,33cyclictest14277-21latency_hist22:09:562
426996350,8cyclictest0-21swapper/220:25:162
426996328,9cyclictest8212-21switchtime19:25:262
426996328,9cyclictest8212-21switchtime19:25:262
42699623,43cyclictest352-21runrttasks19:40:512
2165226210,10sleep30-21swapper/300:10:193
426996147,10cyclictest404-21cyclictest20:00:202
42699613,40cyclictest28937-21/usr/sbin/munin20:30:172
43199595,35cyclictest0-21swapper/323:54:573
43199577,35cyclictest0-21swapper/320:55:083
43199573,40cyclictest3729-21chrt23:40:063
43199573,40cyclictest3729-21chrt23:40:063
43199573,40cyclictest17326-21chrt22:14:563
43199566,13cyclictest0-21swapper/319:55:453
43199566,13cyclictest0-21swapper/319:55:453
43199553,41cyclictest30243-21chrt00:25:143
43199547,41cyclictest0-21swapper/323:35:163
43199544,27cyclictest0-21swapper/323:25:143
257052540,7sleep30-21swapper/320:19:583
43199533,41cyclictest27572-21taskset20:25:163
43199533,38cyclictest21476-21taskset20:05:293
43199533,37cyclictest20578-21chrt22:20:113
43199533,36cyclictest30014-21chrt21:40:313
43199532,37cyclictest11781-21taskset22:05:153
431995312,26cyclictest0-21swapper/323:05:143
43199524,38cyclictest31337-21taskset22:37:513
43199524,38cyclictest31337-21taskset22:37:513
431995237,10cyclictest451rcuc/320:10:153
43199523,37cyclictest26991-21chrt23:24:563
43199523,36cyclictest11316-21taskset21:10:143
43199523,35cyclictest17121-21chrt19:54:563
79272510,7sleep30-21swapper/323:46:063
431995137,9cyclictest451rcuc/321:00:163
43199513,38cyclictest3229-21taskset21:50:283
43199513,38cyclictest3229-21taskset21:50:283
43199513,38cyclictest15641-21taskset21:16:473
43199513,38cyclictest15641-21taskset21:16:473
43199513,37cyclictest475-21taskset23:34:543
43199513,37cyclictest475-21taskset23:34:543
43199513,34cyclictest3089-21taskset20:50:113
43199513,33cyclictest15099-21chrt23:59:523
431995131,12cyclictest451rcuc/319:11:203
43199512,35cyclictest19221-21taskset00:05:283
43199503,37cyclictest29142-21chrt22:34:553
43199503,35cyclictest27205-21chrt20:24:563
43199503,35cyclictest27205-21chrt20:24:563
43199503,32cyclictest9526-21chrt21:06:253
43199503,32cyclictest4280-21taskset19:15:033
43199502,34cyclictest24182-21taskset22:25:263
43199502,34cyclictest24182-21taskset22:25:263
431994935,9cyclictest451rcuc/321:47:573
43199493,36cyclictest8424-21taskset19:25:283
43199493,36cyclictest8424-21taskset19:25:283
43199493,34cyclictest25481-21taskset21:34:223
43199493,34cyclictest25481-21taskset21:34:223
43199493,32cyclictest19113-21chrt21:23:113
43199493,30cyclictest17478-21chrt22:15:043
43199493,30cyclictest17478-21chrt22:15:043
431994932,10cyclictest451rcuc/322:59:283
431994928,13cyclictest451rcuc/300:19:153
431994928,13cyclictest451rcuc/300:19:153
43199492,37cyclictest19084-21chrt23:10:293
43199492,35cyclictest6725-21chrt21:55:513
43199492,34cyclictest2322-21taskset00:32:363
43199492,34cyclictest2322-21taskset00:32:363
43199484,33cyclictest12530-21chrt19:40:143
431994835,9cyclictest451rcuc/322:45:513
431994835,8cyclictest451rcuc/321:29:043
431994835,8cyclictest451rcuc/321:29:043
43199483,34cyclictest26302-21chrt21:35:153
43199483,32cyclictest14462-21chrt19:45:213
43199482,37cyclictest22958-21taskset23:17:353
43199482,37cyclictest17087-21taskset00:02:123
43199482,37cyclictest17087-21taskset00:02:123
43199482,33cyclictest6780-21chrt19:20:303
43199473,31cyclictest1211-21chrt20:44:563
43199472,36cyclictest9853-21taskset22:01:143
43199472,35cyclictest9222-21chrt19:30:103
43199472,35cyclictest31333-21chrt20:35:293
43199472,34cyclictest19105-21chrt20:00:113
43199472,32cyclictest28288-21chrt00:20:443
43199472,30cyclictest29100-21chrt20:30:103
43199462,33cyclictest1990-21taskset22:43:103
43199453,30cyclictest7317-21chrt22:51:333
43199433,28cyclictest1368-21chrt20:45:093
431994227,8cyclictest451rcuc/319:36:503
43199397,17cyclictest0-21swapper/319:09:573
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional