You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 17:31
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Wed Mar 25, 2026 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7530991112,88cyclictest4340-21apt-get10:34:542
75309910250,42cyclictest0-21swapper/211:19:472
75259910134,18cyclictest0-21swapper/109:49:311
7530999863,25cyclictest5087-21apt-key08:34:342
7530999863,25cyclictest5087-21apt-key08:34:342
7530999841,50cyclictest13633-21latency_hist11:44:342
7525999837,9cyclictest0-21swapper/108:19:341
7525999837,9cyclictest0-21swapper/108:19:341
7530999712,13cyclictest0-21swapper/210:44:422
7530999712,13cyclictest0-21swapper/210:44:422
753099934,35cyclictest3778-21/usr/sbin/munin09:39:512
753099924,34cyclictest22646-21latency11:04:562
753099923,37cyclictest27819-21latency_hist08:04:342
753099923,35cyclictest2823-21wc08:25:072
753099923,34cyclictest8586-21latency_hist08:44:342
753099923,33cyclictest5478-21open_inodes11:30:022
753099923,33cyclictest5478-21open_inodes11:30:022
753099923,33cyclictest1680-21sh10:29:592
753099913,36cyclictest18982-21latency_hist10:59:332
753099913,36cyclictest18982-21latency_hist10:59:332
7530999127,10cyclictest0-21swapper/211:59:532
753099912,29cyclictest18417-21fschecks_count09:09:532
753099903,76cyclictest21436-21rm09:14:492
7525999042,17cyclictest181rcu_preempt11:01:361
752199903,35cyclictest15-21ksoftirqd/007:44:330
634629026,33sleep2381ktimers/206:59:592
753099893,35cyclictest24673-21/usr/sbin/munin10:14:572
753099893,34cyclictest31352-21idleruntime-cro12:14:312
753099893,34cyclictest10821-21head09:50:072
753099893,33cyclictest28843-21processes11:15:032
753099893,33cyclictest28843-21processes11:15:032
753099893,33cyclictest15444-21ssh09:04:532
7530998923,25cyclictest32571-21modprobe08:19:312
7530998923,25cyclictest32571-21modprobe08:19:312
7525998954,10cyclictest31-21ksoftirqd/108:35:151
70432897,53sleep17050-21users07:00:191
753099883,38cyclictest352-21runrttasks10:00:212
753099883,38cyclictest352-21runrttasks10:00:212
753099883,38cyclictest27833-21sh10:19:502
753099883,34cyclictest2150-21/usr/sbin/munin12:20:052
753099883,34cyclictest2150-21/usr/sbin/munin12:20:052
753099883,33cyclictest31012-21latency_hist08:14:342
753099883,33cyclictest31012-21latency_hist08:14:342
7530998828,54cyclictest0-21swapper/207:39:572
753099882,32cyclictest16330-21latency_hist07:29:352
753099882,32cyclictest16330-21latency_hist07:29:352
753099882,31cyclictest19709-21latency_hist07:39:342
753099882,31cyclictest19709-21latency_hist07:39:342
7525998847,10cyclictest7512-21cyclictest12:25:471
753099874,35cyclictest21513-21/usr/sbin/munin07:45:312
753099874,35cyclictest21513-21/usr/sbin/munin07:45:312
753099874,32cyclictest9046-21modprobe07:09:312
753099873,34cyclictest32227-21ntpq12:14:592
753099873,31cyclictest8064-21ssh11:34:522
7530998731,48cyclictest0-21swapper/208:35:212
753099872,35cyclictest643-21fschecks_time08:19:552
753099872,35cyclictest643-21fschecks_time08:19:552
753099872,33cyclictest9852-21cstates07:09:512
753099865,30cyclictest24352-21/usr/sbin/munin09:20:092
753099863,38cyclictest11931-21acpi08:54:432
753099863,33cyclictest28376-21wc08:04:572
753099863,32cyclictest7899-21ntp_kernel_pll_10:39:592
753099863,32cyclictest22351-21wc10:10:052
7530998610,27cyclictest7111-21sort09:44:552
7521998665,10cyclictest31658-21ssh11:20:110
7530998566,12cyclictest28287-21timerwakeupswit09:25:082
7530998566,12cyclictest28287-21timerwakeupswit09:25:082
753099854,31cyclictest5114-21/usr/sbin/munin12:25:432
7530998523,11cyclictest0-21swapper/211:29:342
7530998523,11cyclictest0-21swapper/211:29:342
753099852,29cyclictest26234-21/usr/sbin/munin12:04:592
7521998569,10cyclictest20833-21munin-run09:14:320
7521998554,23cyclictest0-21swapper/009:30:100
753099844,29cyclictest12725-21/usr/sbin/munin09:54:582
753099843,34cyclictest14251-21latency11:44:572
753099843,34cyclictest14251-21latency11:44:572
753099843,33cyclictest8984-21sort12:30:052
753099843,33cyclictest13839-21ntpq10:49:592
753099843,33cyclictest13373-21cut07:19:552
753099843,33cyclictest13373-21cut07:19:552
753099843,32cyclictest8794-21cstates08:44:502
753099843,32cyclictest25885-21ssh11:10:052
753099843,32cyclictest22041-21ssh11:04:342
753099843,31cyclictest25214-21taskset07:54:572
7530998428,10cyclictest0-21swapper/209:30:082
73032847,61sleep00-21swapper/007:03:320
7530998340,36cyclictest0-21swapper/210:05:272
753099833,27cyclictest30567-21/usr/sbin/munin10:25:032
753099833,27cyclictest30567-21/usr/sbin/munin10:25:032
753099832,30cyclictest11639-21cron08:54:312
753099832,29cyclictest12925-21sed07:19:342
753099832,29cyclictest12925-21sed07:19:342
753099832,23cyclictest1110-21/usr/sbin/munin09:34:532
753099824,29cyclictest16425-21/usr/sbin/munin07:30:452
753099823,36cyclictest24081-21head07:50:072
753099823,23cyclictest19719-21/usr/sbin/munin11:54:592
753099823,23cyclictest19719-21/usr/sbin/munin11:54:592
753099822,31cyclictest14124-21wc09:00:032
7521998266,9cyclictest27884-21ssh10:19:530
7525998140,32cyclictest28958-21sort12:09:531
7530998024,7cyclictest0-21swapper/211:49:472
7525998048,8cyclictest181rcu_preempt10:19:591
7525997954,15cyclictest373-21lldpd10:44:571
7525997954,15cyclictest373-21lldpd10:44:571
7521997964,10cyclictest58-21kcompactd008:47:100
7521997941,31cyclictest0-21swapper/009:15:250
7521997848,23cyclictest0-21swapper/009:25:500
7521997848,23cyclictest0-21swapper/009:25:500
7521997847,24cyclictest0-21swapper/008:34:340
7521997847,24cyclictest0-21swapper/008:34:340
7521997839,8cyclictest181rcu_preempt11:44:420
7521997839,8cyclictest181rcu_preempt11:44:420
92982777,59sleep00-21swapper/011:36:110
752599773,56cyclictest31401-21ssh11:19:491
7521997764,9cyclictest58-21kcompactd008:50:540
7521997658,12cyclictest25151-21ssh09:20:060
7521997656,14cyclictest22216-21sh11:04:440
7521997656,14cyclictest14902-21diskmemload10:49:310
7521997656,14cyclictest14902-21diskmemload10:49:310
7521997643,26cyclictest0-21swapper/012:17:480
7521997626,17cyclictest181rcu_preempt11:55:090
7521997626,17cyclictest181rcu_preempt11:55:090
7525997553,16cyclictest25440-21tune2fs11:09:531
7525997542,7cyclictest181rcu_preempt11:36:361
7521997557,12cyclictest14477-21latency_hist07:24:340
7521997557,12cyclictest14477-21latency_hist07:24:340
7525997454,15cyclictest2428-21fschecks_count12:19:531
7525997454,15cyclictest2428-21fschecks_count12:19:531
7525997453,10cyclictest0-21swapper/112:05:041
7525997452,16cyclictest1332-21fschecks_count10:29:531
7525997452,15cyclictest200-21systemd-journal09:44:271
7525997447,19cyclictest9947-21fschecks_count07:09:561
7525997411,45cyclictest21398-21apt-get09:14:551
7521997454,16cyclictest7512-21cyclictest09:05:080
7521997359,10cyclictest7512-21cyclictest12:27:530
7521997359,10cyclictest7512-21cyclictest10:39:090
7521997352,15cyclictest181rcu_preempt10:51:290
7521997350,16cyclictest23032-21latency_hist07:49:340
7521997350,16cyclictest23032-21latency_hist07:49:340
752199734,49cyclictest15-21ksoftirqd/010:14:450
7521997343,8cyclictest181rcu_preempt11:50:110
7521997343,12cyclictest181rcu_preempt11:25:010
7521997343,12cyclictest181rcu_preempt11:25:010
752199733,34cyclictest784-21modprobe10:29:310
752199733,34cyclictest784-21modprobe10:29:310
752199733,34cyclictest7324-21ssh10:39:430
233762730,10sleep1301ktimers/110:11:491
7525997251,15cyclictest30619-21cpu10:24:451
7525997251,15cyclictest30619-21cpu10:24:451
7525997234,8cyclictest181rcu_preempt07:59:351
7521997257,10cyclictest7512-21cyclictest10:09:570
7521997255,12cyclictest27542-21kworker/u18:0-rpciod09:56:100
752199725,47cyclictest25491-21fschecks_time11:09:540
7521997252,7cyclictest181rcu_preempt10:33:180
7521997251,14cyclictest20266-21sshd10:06:270
7521997244,10cyclictest181rcu_preempt07:09:510
7521997241,9cyclictest181rcu_preempt08:19:340
7521997241,9cyclictest181rcu_preempt08:19:340
7521997241,11cyclictest181rcu_preempt08:59:340
7521997239,11cyclictest181rcu_preempt09:38:030
7525997155,11cyclictest7512-21cyclictest11:44:321
7525997154,11cyclictest19389-21cron11:54:311
7525997153,12cyclictest8236-21cstates12:29:471
7521997155,10cyclictest21789-21sh11:04:170
7521997154,10cyclictest181rcu_preempt12:03:330
7521997150,15cyclictest7512-21cyclictest07:34:330
7521997149,9cyclictest181rcu_preempt12:04:380
7521997144,20cyclictest17240-21ssh10:55:070
7521997144,20cyclictest17240-21ssh10:55:070
7525997044,18cyclictest18609-21latency09:09:571
752599703,52cyclictest28197-21ssh11:14:461
752599703,52cyclictest28197-21ssh11:14:461
7521997056,10cyclictest7512-21cyclictest07:54:340
7521997053,11cyclictest1-21systemd11:18:190
7521997053,11cyclictest1-21systemd11:18:190
752199705,29cyclictest15-21ksoftirqd/008:39:460
7521997050,13cyclictest15974-21apt-key07:29:320
7521997050,13cyclictest15974-21apt-key07:29:320
7521997049,8cyclictest181rcu_preempt12:19:570
7521997049,8cyclictest181rcu_preempt12:19:570
7521997049,8cyclictest181rcu_preempt09:44:420
752199704,37cyclictest10790-21/usr/sbin/munin11:39:560
7525996954,9cyclictest845-21kworker/u17:3-rpciod10:40:081
7525996953,10cyclictest21253-21cron07:44:311
7525996949,12cyclictest19360-21kworker/u17:3-rpciod12:04:331
7521996951,12cyclictest5662-21fschecks_count08:35:050
7521996949,14cyclictest8297-21processes07:05:040
7521996948,8cyclictest181rcu_preempt09:44:340
7521996944,20cyclictest12539-21latency_hist09:54:340
752199693,37cyclictest352-21runrttasks11:32:510
752199693,37cyclictest352-21runrttasks11:32:510
7525996854,9cyclictest8427-21latency_hist08:44:331
7525996853,8cyclictest0-21swapper/110:05:081
7525996846,16cyclictest4859-21sh10:35:151
7525996844,18cyclictest20465-21mailstats11:55:041
7525996844,18cyclictest20465-21mailstats11:55:041
752599683,46cyclictest14983-21latency_hist09:04:341
752599683,25cyclictest30819-21load09:29:571
752199684,54cyclictest10771-21latency_hist12:34:330
7521996844,11cyclictest181rcu_preempt09:00:000
7521996841,10cyclictest181rcu_preempt08:00:070
7521996839,12cyclictest181rcu_preempt07:39:180
7521996839,12cyclictest181rcu_preempt07:39:180
752199683,35cyclictest352-21runrttasks12:11:230
7525996751,11cyclictest352-21runrttasks09:50:061
7525996748,13cyclictest17957-21latency_hist07:34:341
7521996748,14cyclictest7512-21cyclictest09:59:530
7521996748,14cyclictest7512-21cyclictest09:59:530
7521996746,8cyclictest181rcu_preempt08:20:090
7521996746,8cyclictest181rcu_preempt08:20:090
752199673,45cyclictest25716-21unixbench_singl07:55:070
7525996652,9cyclictest30273-21kworker/u17:2-xprtiod11:25:321
7525996652,9cyclictest30273-21kworker/u17:2-xprtiod11:25:321
7525996652,8cyclictest0-21swapper/111:45:081
7525996652,8cyclictest0-21swapper/111:45:081
7525996651,10cyclictest8208-21ntp_kernel_pll_07:05:001
7521996644,9cyclictest181rcu_preempt08:09:350
752199663,42cyclictest352-21runrttasks07:17:370
752199663,42cyclictest352-21runrttasks07:17:370
753899653,46cyclictest19827-21taskset07:39:353
753899653,46cyclictest19827-21taskset07:39:353
7525996542,14cyclictest14902-21diskmemload10:14:571
752599653,33cyclictest4809-21/usr/sbin/munin11:30:031
752599653,33cyclictest4809-21/usr/sbin/munin11:30:031
7521996541,10cyclictest181rcu_preempt08:25:030
7525996445,14cyclictest15357-21threads07:25:061
7525996445,14cyclictest15357-21threads07:25:061
7525996442,12cyclictest0-21swapper/108:59:341
752599643,43cyclictest12725-21/usr/sbin/munin09:54:471
7525996416,33cyclictest13975-21find10:50:041
752199643,30cyclictest31180-21latency_hist08:14:350
752199643,30cyclictest31180-21latency_hist08:14:350
7525996350,8cyclictest58-21kcompactd010:01:011
7525996350,8cyclictest58-21kcompactd010:01:011
752599633,41cyclictest25941-21ssh09:21:271
752599633,33cyclictest2883-21unixbench_multi08:25:071
752599633,23cyclictest32054-21/usr/sbin/munin12:14:561
7525996242,14cyclictest22866-21proc_pri11:05:031
7525996236,16cyclictest0-21swapper/109:38:321
7525996219,24cyclictest11796-21latency07:14:581
7525996219,24cyclictest11796-21latency07:14:581
753899614,41cyclictest21859-21taskset07:45:313
753899614,41cyclictest21859-21taskset07:45:313
7525996146,8cyclictest0-21swapper/108:14:321
7525996146,8cyclictest0-21swapper/108:14:321
7525996141,7cyclictest181rcu_preempt10:59:281
7525996141,7cyclictest181rcu_preempt10:59:281
7525996141,13cyclictest1198-21timerandwakeup08:20:091
7525996141,13cyclictest1198-21timerandwakeup08:20:091
7525996127,10cyclictest181rcu_preempt08:04:571
752599612,42cyclictest3584-21/usr/sbin/munin08:29:501
752599612,42cyclictest3584-21/usr/sbin/munin08:29:501
752599612,33cyclictest10788-21ntpq08:49:581
7525996113,37cyclictest14385-21wc07:24:331
7525996113,37cyclictest14385-21wc07:24:331
7525996045,8cyclictest0-21swapper/107:45:401
7525996045,8cyclictest0-21swapper/107:45:401
7525996036,17cyclictest0-21swapper/109:29:311
7525996036,17cyclictest0-21swapper/109:29:311
7538995942,12cyclictest0-21swapper/309:15:023
7525995944,9cyclictest23659-21ntp_offset07:50:001
7525995937,7cyclictest181rcu_preempt07:59:351
752599593,39cyclictest18318-21apt-get07:34:491
752599593,39cyclictest18318-21apt-get07:34:491
7525995915,36cyclictest9556-21timerwakeupswit08:45:091
753899583,44cyclictest30841-21chrt11:19:313
753899583,44cyclictest30841-21chrt11:19:313
659725813,25sleep30-21swapper/307:00:093
753899573,44cyclictest22678-21taskset11:04:563
753899573,42cyclictest15230-21chrt09:04:443
753899573,40cyclictest5305-21taskset08:34:353
753899573,40cyclictest5305-21taskset08:34:353
7525995716,29cyclictest16145-21sh09:05:081
753899554,37cyclictest0-21swapper/311:19:583
753899553,41cyclictest18688-21taskset10:58:523
753899553,41cyclictest18688-21taskset10:58:523
753899544,39cyclictest27431-21chrt10:19:333
753899544,38cyclictest4837-21chrt12:24:313
753899544,38cyclictest4837-21chrt12:24:313
753899543,41cyclictest12351-21taskset09:54:253
753899543,37cyclictest17882-21chrt07:34:333
753899543,35cyclictest2814-21chrt11:25:073
753899543,35cyclictest2814-21chrt11:25:073
753899543,35cyclictest1944-21chrt08:24:353
753899543,35cyclictest16247-21taskset07:29:353
753899543,35cyclictest16247-21taskset07:29:353
753899534,38cyclictest21328-21chrt07:44:333
753899534,37cyclictest31062-21taskset08:14:343
753899534,37cyclictest31062-21taskset08:14:343
753899534,36cyclictest6391-21chrt09:44:243
753899533,39cyclictest577-21taskset10:28:573
753899533,39cyclictest577-21taskset10:28:573
753899533,39cyclictest16873-21taskset10:00:163
753899533,39cyclictest16873-21taskset10:00:163
753899533,36cyclictest16578-21chrt11:49:343
753899533,36cyclictest16578-21chrt11:49:343
753899532,41cyclictest25494-21taskset09:20:133
753899524,36cyclictest30329-21chrt09:29:453
753899524,35cyclictest19628-21taskset10:05:073
753899524,33cyclictest12922-21chrt07:19:343
753899524,33cyclictest12922-21chrt07:19:343
753899523,40cyclictest7470-21chrt10:39:483
753899523,38cyclictest29299-21chrt08:09:203
753899523,38cyclictest14077-21taskset10:50:053
753899523,34cyclictest25619-21chrt07:55:063
753899522,39cyclictest22945-21chrt10:10:383
753899522,36cyclictest5078-21chrt12:24:353
753899514,37cyclictest8045-21taskset11:34:513
753899514,31cyclictest5333-21chrt10:35:263
753899513,34cyclictest871-21chrt08:20:013
753899513,34cyclictest27185-21chrt11:12:423
753899513,33cyclictest5650-21chrt11:30:053
753899513,33cyclictest5650-21chrt11:30:053
753899512,39cyclictest5973-21chrt08:35:123
753899512,38cyclictest1802-21taskset12:19:253
753899505,33cyclictest25669-21chrt12:04:423
753899504,36cyclictest20497-21taskset11:00:353
753899503,35cyclictest8445-21taskset08:44:313
753899503,35cyclictest14318-21taskset09:00:073
753899503,34cyclictest9148-21chrt07:09:343
753899503,32cyclictest23213-21chrt07:49:483
753899502,36cyclictest3645-21taskset10:34:123
753899502,36cyclictest19708-21chrt09:11:253
753899502,34cyclictest28790-21chrt10:20:143
7538994934,10cyclictest451rcuc/309:54:423
753899493,35cyclictest6972-21chrt09:44:533
753899493,35cyclictest26876-21taskset08:00:003
753899493,34cyclictest2260-21chrt09:36:013
753899493,32cyclictest18993-21chrt11:53:233
753899484,33cyclictest19695-21chrt11:54:373
753899484,33cyclictest19695-21chrt11:54:373
753899483,32cyclictest23440-21chrt12:00:043
753899483,31cyclictest2891-21chrt08:25:073
753899482,35cyclictest10106-21taskset08:49:333
753899482,34cyclictest11191-21chrt08:50:063
753899474,34cyclictest12033-21taskset10:46:573
753899474,34cyclictest12033-21taskset10:46:573
753899473,33cyclictest9119-21chrt12:30:063
753899473,30cyclictest13351-21chrt08:59:343
753899463,34cyclictest13261-21taskset11:43:593
753899463,31cyclictest31249-21chrt12:14:223
753899423,27cyclictest9577-21chrt07:09:423
753899413,30cyclictest27967-21taskset09:25:033
753899413,30cyclictest27967-21taskset09:25:033
753899404,26cyclictest31496-21taskset08:14:553
753899404,26cyclictest31496-21taskset08:14:553
753899383,25cyclictest13002-21taskset07:19:433
753899383,25cyclictest13002-21taskset07:19:433
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional