You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-21 - 22:03
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Tue Jul 21, 2026 12:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2647021187,15sleep00-21swapper/006:54:300
269349911036,11cyclictest0-21swapper/211:41:222
2646221056,17sleep20-21swapper/206:54:242
269349910236,16cyclictest27330-21cut12:21:552
269349910236,16cyclictest27330-21cut12:21:552
269349910142,8cyclictest0-21swapper/209:12:102
269349910142,8cyclictest0-21swapper/209:12:102
269349910126,12cyclictest0-21swapper/208:56:562
269349910041,51cyclictest0-21swapper/208:32:042
26934999937,14cyclictest31823-21/usr/sbin/munin09:47:342
26934999935,10cyclictest0-21swapper/210:22:082
26934999935,10cyclictest0-21swapper/210:22:082
26934999739,10cyclictest0-21swapper/209:46:362
26934999638,12cyclictest2567-21latency_hist10:46:362
26934999629,12cyclictest0-21swapper/212:07:072
26934999536,9cyclictest0-21swapper/211:01:572
26934999536,9cyclictest0-21swapper/211:01:572
26934999534,7cyclictest0-21swapper/211:17:252
2693499953,35cyclictest6063-21processes09:57:082
2693499953,35cyclictest6063-21processes09:57:082
26934999425,26cyclictest24345-21grep10:27:112
2664629416,49sleep126660-21munin-run06:56:351
2693499933,36cyclictest29388-21fschecks_count07:01:592
26934999333,11cyclictest0-21swapper/211:31:532
26934999333,11cyclictest0-21swapper/211:31:532
2693499932,34cyclictest8002-21fschecks_count09:06:562
26934999244,39cyclictest0-21swapper/207:31:372
2693499923,35cyclictest20230-21find08:12:082
2693499923,33cyclictest20496-21sort09:27:092
2693499923,30cyclictest352-21runrttasks11:54:232
2693499923,29cyclictest14541-21switchtime09:17:112
2693499914,32cyclictest17154-21grep10:16:372
26934999133,7cyclictest0-21swapper/210:36:362
26934999133,7cyclictest0-21swapper/210:36:362
2693499912,36cyclictest3217-21cut10:46:592
2693499912,36cyclictest3217-21cut10:46:592
2693499912,35cyclictest16890-21fschecks_count09:21:542
2693499912,34cyclictest23432-21cut08:22:082
2693499912,32cyclictest6093-21latency_hist11:46:362
26934999056,24cyclictest16285-21latency_hist08:01:372
2693499903,36cyclictest14979-21latency_hist12:01:362
2693499903,36cyclictest14979-21latency_hist12:01:362
2693499903,35cyclictest30649-21modprobe08:46:342
2693499903,35cyclictest30649-21modprobe08:46:342
2693499903,31cyclictest352-21runrttasks11:24:182
2693499903,31cyclictest352-21runrttasks11:24:182
2693499903,31cyclictest27585-21ps11:27:082
2693499902,34cyclictest15508-21mailstats11:07:092
26907999067,17cyclictest15383-21sh11:07:071
2693499893,32cyclictest27940-21fschecks_time08:36:572
2693499893,32cyclictest14137-21wc10:11:362
26934998931,10cyclictest0-21swapper/208:52:072
2693499892,36cyclictest9174-21kernelversion10:57:002
2693499892,36cyclictest9174-21kernelversion10:57:002
2693499892,34cyclictest18042-21mailstats10:17:082
2693499892,33cyclictest32433-21latency_hist08:51:362
2693499892,33cyclictest29672-21sed10:36:592
2693499883,32cyclictest31822-21runrttasks07:07:132
2693499883,32cyclictest27871-21wc06:58:152
2693499883,29cyclictest6557-21fschecks_count11:46:552
2693499883,29cyclictest6557-21fschecks_count11:46:552
26934998832,12cyclictest0-21swapper/210:01:512
26934998825,11cyclictest0-21swapper/209:31:512
2693499882,34cyclictest3225-21sort09:52:122
2693499882,29cyclictest20744-21latency_hist11:16:382
2693499873,35cyclictest14531-21latency_hist07:56:362
2693499873,34cyclictest5165-21kernelversion09:02:022
2693499873,34cyclictest5165-21kernelversion09:02:022
2693499873,32cyclictest5745-21/usr/sbin/munin10:52:072
26934998730,13cyclictest0-21swapper/208:16:582
2693499872,37cyclictest32655-21fschecks_count07:11:582
2693499872,37cyclictest32655-21fschecks_count07:11:582
2690799878,46cyclictest9021-21fschecks_time10:56:561
2690799878,46cyclictest9021-21fschecks_time10:56:561
2693499864,29cyclictest25597-21/usr/sbin/munin09:36:522
2693499863,36cyclictest9681-21latency_hist07:41:372
2693499863,36cyclictest352-21runrttasks08:31:142
2690799864,43cyclictest31-21ksoftirqd/112:07:021
26890998652,12cyclictest181rcu_preempt08:16:500
2693499852,33cyclictest10168-21fschecks_time07:42:192
2693499852,33cyclictest10168-21fschecks_time07:42:192
26890998554,23cyclictest12201-21sh11:56:420
26890998554,23cyclictest12201-21sh11:56:420
26890998543,7cyclictest0-21swapper/008:56:500
2693499844,36cyclictest16624-21fschecks_time08:01:572
2693499844,32cyclictest12049-21ntpq07:47:052
26907998461,16cyclictest14265-21ntp_kernel_pll_09:17:041
26890998453,10cyclictest181rcu_preempt09:51:500
2693499833,31cyclictest24803-21processes12:17:092
2693499833,31cyclictest21128-21/usr/sbin/munin12:12:122
2693499833,31cyclictest1577-21/usr/sbin/munin07:17:072
2693499833,31cyclictest1577-21/usr/sbin/munin07:17:072
26907998340,17cyclictest181rcu_preempt08:37:051
26890998362,15cyclictest11428-21sendmail_mailqu09:12:070
26890998362,15cyclictest11428-21sendmail_mailqu09:12:070
26907998265,10cyclictest25052-21timerwakeupswit12:17:131
2690799823,43cyclictest31-21ksoftirqd/108:52:001
26907998163,12cyclictest0-21swapper/111:24:491
26907998163,12cyclictest0-21swapper/111:24:491
26907998152,12cyclictest181rcu_preempt10:21:581
26907998152,12cyclictest181rcu_preempt10:21:581
2690799811,50cyclictest31-21ksoftirqd/109:56:061
26890998137,26cyclictest15-21ksoftirqd/008:42:070
26890998137,26cyclictest15-21ksoftirqd/008:42:070
2693499803,34cyclictest18056-21cpu08:06:522
2690799808,37cyclictest31-21ksoftirqd/110:35:211
2690799808,37cyclictest31-21ksoftirqd/110:35:211
26907998060,13cyclictest352-21runrttasks08:17:121
26907998060,13cyclictest23544-21ssh09:32:121
26907998047,12cyclictest181rcu_preempt09:27:071
2690799803,43cyclictest31-21ksoftirqd/110:26:381
2690799802,35cyclictest32441-21sort10:41:541
26907998012,54cyclictest20921-21apt-get11:17:061
26890998050,21cyclictest23687-21sh10:26:590
2690799796,27cyclictest31-21ksoftirqd/107:16:501
2690799796,27cyclictest31-21ksoftirqd/107:16:501
26907997961,9cyclictest0-21swapper/110:01:371
26907997961,9cyclictest0-21swapper/110:01:371
26907997949,12cyclictest181rcu_preempt08:06:071
26907997937,17cyclictest181rcu_preempt10:02:091
26890997957,15cyclictest13115-21kworker/u18:0+events_unbound09:46:330
2693499784,34cyclictest6427-21/usr/sbin/munin07:31:532
26907997848,13cyclictest181rcu_preempt08:42:241
26907997848,13cyclictest181rcu_preempt08:42:241
2690799782,49cyclictest31-21ksoftirqd/108:50:151
26890997854,18cyclictest20068-21kernelversion09:26:590
26907997747,9cyclictest181rcu_preempt10:16:581
2690799773,51cyclictest31-21ksoftirqd/111:40:431
2690799772,49cyclictest31-21ksoftirqd/112:24:191
2690799772,49cyclictest31-21ksoftirqd/112:24:191
2690799772,27cyclictest31-21ksoftirqd/112:16:011
26890997762,10cyclictest26870-21cyclictest10:01:350
26890997762,10cyclictest26870-21cyclictest10:01:350
26907997650,12cyclictest181rcu_preempt09:51:281
26907997649,11cyclictest181rcu_preempt10:15:271
26907997639,17cyclictest181rcu_preempt07:41:001
26907997631,27cyclictest31-21ksoftirqd/107:46:361
26907997631,27cyclictest31-21ksoftirqd/107:46:361
2690799762,52cyclictest28178-21unixbench_multi06:58:241
2690799762,50cyclictest31-21ksoftirqd/111:34:251
2690799762,50cyclictest31-21ksoftirqd/111:34:251
26890997647,23cyclictest277-21dbus-daemon11:06:340
26890997647,23cyclictest277-21dbus-daemon11:06:340
26890997644,17cyclictest15-21ksoftirqd/009:47:250
2690799757,45cyclictest31-21ksoftirqd/112:01:321
2690799757,45cyclictest31-21ksoftirqd/112:01:321
26907997553,14cyclictest18067-21idleruntime11:11:591
26907997549,11cyclictest181rcu_preempt11:42:111
26907997541,13cyclictest181rcu_preempt09:04:081
26907997541,13cyclictest181rcu_preempt09:04:081
2690799753,36cyclictest31-21ksoftirqd/107:01:411
26907997527,27cyclictest0-21swapper/112:06:361
26907997527,27cyclictest0-21swapper/112:06:361
26890997542,26cyclictest0-21swapper/011:17:500
2690799747,30cyclictest31-21ksoftirqd/109:40:151
26907997454,12cyclictest0-21swapper/109:42:051
26907997446,20cyclictest8212-21sh10:56:001
26907997442,23cyclictest25803-21kworker/u18:1+nfsiod10:06:561
26890997443,20cyclictest0-21swapper/012:06:360
26890997443,20cyclictest0-21swapper/012:06:360
2689099744,27cyclictest26650-21apt-get06:56:450
26890997438,31cyclictest0-21swapper/007:16:520
26890997438,31cyclictest0-21swapper/007:16:520
26907997356,8cyclictest0-21swapper/109:07:121
2690799735,36cyclictest31-21ksoftirqd/107:57:001
2690799735,34cyclictest31-21ksoftirqd/107:56:371
26907997351,16cyclictest19328-21sendmail09:26:341
26907997336,13cyclictest11608-21ssh11:55:411
26907997335,12cyclictest181rcu_preempt08:12:131
26907997312,40cyclictest31-21ksoftirqd/108:29:201
26890997359,8cyclictest15-21ksoftirqd/007:41:370
26890997349,18cyclictest23916-21latency_hist12:16:360
2690799727,45cyclictest31-21ksoftirqd/107:22:111
26907997249,16cyclictest1762-21ssh08:56:461
26907997248,15cyclictest8976-21latency_hist11:51:331
26907997248,15cyclictest8976-21latency_hist11:51:331
26907997246,10cyclictest181rcu_preempt07:32:191
2690799724,37cyclictest31-21ksoftirqd/110:46:511
2690799724,37cyclictest31-21ksoftirqd/110:46:511
26907997136,13cyclictest181rcu_preempt11:31:191
26890997148,17cyclictest30535-21cron07:06:340
2689099713,50cyclictest5354-21ntp_states09:02:060
2689099713,50cyclictest5354-21ntp_states09:02:060
2690799709,28cyclictest31-21ksoftirqd/111:02:111
2690799709,28cyclictest31-21ksoftirqd/111:02:111
26907997050,13cyclictest5770-21unixbench_singl07:27:111
26907997047,9cyclictest181rcu_preempt07:15:081
26907997047,9cyclictest181rcu_preempt07:15:081
26907997038,7cyclictest181rcu_preempt08:11:341
26907997032,12cyclictest181rcu_preempt07:50:311
26890997053,11cyclictest58-21kcompactd011:11:430
26890997050,9cyclictest15-21ksoftirqd/007:42:350
26890997050,9cyclictest15-21ksoftirqd/007:42:350
26890997050,15cyclictest9024-21latency_hist11:51:350
26890997050,15cyclictest9024-21latency_hist11:51:350
26890997047,9cyclictest15-21ksoftirqd/010:57:100
26890997047,9cyclictest15-21ksoftirqd/010:57:100
26890997020,24cyclictest27829-21cat11:27:110
26907996950,12cyclictest0-21swapper/110:37:111
26907996938,12cyclictest181rcu_preempt07:11:351
2690799692,27cyclictest31-21ksoftirqd/108:32:151
26890996954,9cyclictest19975-21rm12:09:080
26890996952,11cyclictest24094-21/usr/sbin/munin12:17:030
26890996950,13cyclictest14466-21mailstats09:17:100
2689099693,57cyclictest23528-21apt-get11:21:490
2689099693,57cyclictest23528-21apt-get11:21:490
26934996847,13cyclictest0-21swapper/212:02:122
26934996847,13cyclictest0-21swapper/212:02:122
26907996830,16cyclictest181rcu_preempt09:12:451
26907996830,16cyclictest181rcu_preempt09:12:451
26890996853,9cyclictest26222-21ntp_offset09:37:040
26890996848,11cyclictest4274-21kworker/u19:1-rpciod10:50:320
26890996848,11cyclictest4274-21kworker/u19:1-rpciod10:50:320
26890996844,16cyclictest18285-21cut08:06:580
2689099683,36cyclictest11932-21idleruntime-cro11:56:330
26907996716,26cyclictest31-21ksoftirqd/108:23:081
26890996752,9cyclictest28836-21ssh12:24:060
26890996752,9cyclictest28836-21ssh12:24:060
26890996751,10cyclictest3064-21systemd-run11:41:340
26890996747,8cyclictest15-21ksoftirqd/011:31:410
26890996747,8cyclictest15-21ksoftirqd/011:31:410
26890996747,8cyclictest15-21ksoftirqd/010:26:360
26890996747,8cyclictest15-21ksoftirqd/010:26:360
2689099673,56cyclictest27286-21grep10:32:090
2689099673,56cyclictest27286-21grep10:32:090
2689099673,54cyclictest18189-21timerandwakeup10:17:100
26890996727,24cyclictest181rcu_preempt09:22:110
301152662,8sleep030229-21sed10:37:100
26890996653,8cyclictest5393-21ssh11:44:540
26890996652,10cyclictest26870-21cyclictest08:04:410
26890996648,12cyclictest6053-21ssh10:51:570
26890996647,11cyclictest15605-21ssh11:07:100
2689099663,50cyclictest22699-21cpu09:31:510
26890996550,8cyclictest0-21swapper/010:42:050
2689099654,46cyclictest11642-21cstates07:46:520
2689099653,45cyclictest10537-21latency_hist09:11:360
26890996446,12cyclictest14124-21latency_hist10:11:360
26890996348,10cyclictest28188-21ntp_kernel_pll_08:37:040
2689099633,42cyclictest32430-21/usr/sbin/munin07:12:020
2689099633,42cyclictest32430-21/usr/sbin/munin07:12:020
2689099633,40cyclictest14774-21latency10:12:010
2689099633,34cyclictest5410-21proc_pri07:27:070
26890996316,32cyclictest20133-21grep08:12:040
26890996247,10cyclictest26870-21cyclictest10:06:040
2689099623,35cyclictest13133-21/usr/sbin/munin07:51:570
2689099622,44cyclictest15175-21latency07:57:000
26890996136,14cyclictest0-21swapper/007:07:110
2689099613,43cyclictest17460-21sshd07:33:320
2689099612,22cyclictest24559-21cstates08:26:520
26934996036,15cyclictest0-21swapper/207:22:092
2689099603,32cyclictest22807-21/usr/sbin/munin08:21:580
26890995940,12cyclictest0-21swapper/008:47:080
26954995843,10cyclictest0-21swapper/311:22:033
26954995843,10cyclictest0-21swapper/311:22:033
26890995834,20cyclictest15-21ksoftirqd/008:55:570
2695499573,40cyclictest20826-21taskset12:11:323
26890995737,11cyclictest0-21swapper/008:32:210
26890995711,10cyclictest4148-21grep07:22:110
2695499563,40cyclictest29606-21taskset11:31:353
2695499562,39cyclictest5272-21chrt09:56:383
2695499554,37cyclictest13431-21chrt09:16:363
2695499554,37cyclictest13431-21chrt09:16:363
2695499553,40cyclictest18852-21chrt08:07:123
2695499536,36cyclictest31794-21chrt09:46:403
2695499533,38cyclictest21089-21chrt09:28:073
2695499533,35cyclictest13981-21taskset11:05:263
2695499533,35cyclictest13981-21taskset11:05:263
2695499532,40cyclictest8176-21taskset10:55:533
2695499532,36cyclictest11851-21chrt10:07:003
2695499523,36cyclictest29293-21chrt07:01:543
2695499523,36cyclictest1642-21taskset08:56:373
2695499523,35cyclictest21452-21taskset11:17:343
2695499523,35cyclictest12527-21chrt11:56:573
2695499523,35cyclictest12527-21chrt11:56:573
2695499522,38cyclictest4762-21chrt07:26:373
2695499522,38cyclictest11456-21taskset11:55:153
2695499522,37cyclictest11312-21chrt11:01:203
2695499522,37cyclictest11312-21chrt11:01:203
2671525211,25sleep30-21swapper/306:56:373
2695499513,38cyclictest15850-21chrt09:20:133
2695499513,37cyclictest5678-21taskset11:45:423
2695499513,35cyclictest27973-21chrt06:58:223
2695499513,34cyclictest4190-21chrt10:48:033
2695499513,34cyclictest4190-21chrt10:48:033
2695499513,33cyclictest22793-21chrt08:21:373
2695499513,32cyclictest24461-21chrt12:17:003
2695499512,38cyclictest29934-21chrt12:26:373
2695499512,38cyclictest29934-21chrt12:26:373
2695499512,36cyclictest23362-21chrt09:32:093
2695499512,35cyclictest7592-21chrt11:47:373
2695499512,35cyclictest7592-21chrt11:47:373
102522510,6sleep30-21swapper/310:04:283
2695499504,36cyclictest5394-21chrt09:02:093
2695499504,36cyclictest5394-21chrt09:02:093
2695499503,37cyclictest10186-21taskset09:11:003
2695499503,36cyclictest28100-21taskset08:37:013
2695499503,35cyclictest5107-21chrt07:26:573
2695499503,34cyclictest22896-21taskset10:26:303
2695499503,34cyclictest22896-21taskset10:26:303
2695499503,34cyclictest17907-21chrt11:11:553
2695499503,33cyclictest3528-21chrt08:59:043
2695499503,33cyclictest32512-21taskset10:41:543
2695499503,33cyclictest31523-21chrt07:07:093
2695499503,32cyclictest16321-21taskset11:08:343
2695499502,31cyclictest26163-21chrt10:31:373
2695499494,30cyclictest31582-21chrt10:40:413
2695499493,34cyclictest19522-21chrt09:26:373
2695499493,34cyclictest14258-21chrt10:11:383
2695499493,32cyclictest2217-21chrt11:39:173
2695499493,32cyclictest21667-21chrt12:12:013
2695499493,31cyclictest25118-21chrt08:27:083
2695499492,37cyclictest18435-21chrt10:17:183
2695499484,34cyclictest5911-21taskset09:57:023
2695499484,34cyclictest5911-21taskset09:57:023
2695499483,36cyclictest9914-21chrt07:41:563
2695499483,36cyclictest9914-21chrt07:41:563
2695499483,35cyclictest28176-21taskset10:34:283
2695499483,35cyclictest28176-21taskset10:34:283
2695499483,35cyclictest13765-21chrt07:52:093
2695499483,33cyclictest7058-21chrt07:32:083
2695499483,31cyclictest22119-21chrt08:17:123
2695499482,34cyclictest28529-21taskset09:41:373
2695499473,34cyclictest29331-21chrt09:42:103
2695499473,32cyclictest9035-21chrt07:37:103
2695499473,31cyclictest17901-21chrt12:06:343
2695499473,31cyclictest17901-21chrt12:06:343
2695499472,31cyclictest2608-21taskset07:17:303
2695499472,31cyclictest2608-21taskset07:17:303
2695499463,32cyclictest30000-21chrt11:31:513
2695499463,32cyclictest30000-21chrt11:31:513
2695499452,33cyclictest11817-21taskset07:46:583
2695499433,29cyclictest30530-21taskset08:45:313
2695499433,29cyclictest30530-21taskset08:45:313
2695499423,30cyclictest27521-21chrt08:36:363
2695499423,26cyclictest16369-21taskset08:01:493
2695499403,27cyclictest19759-21taskset08:11:533
2695499393,26cyclictest31072-21taskset08:46:533
2695499393,26cyclictest15283-21taskset07:57:033
2695499392,25cyclictest32424-21taskset07:11:423
2695499392,25cyclictest32424-21taskset07:11:423
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional