You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-22 - 23:11
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Tue Sep 22, 2026 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2818821089,72sleep228278-21sed07:05:382
2942121030,6sleep00-21swapper/010:44:100
522321020,9sleep20-21swapper/210:58:342
287419910159,32cyclictest0-21swapper/012:14:170
287419910159,32cyclictest0-21swapper/012:14:170
287489910082,11cyclictest731-21sendmail12:38:411
2874899993,36cyclictest8472-21chrt07:44:101
28753999769,20cyclictest2399-21apt-key10:53:462
28748999741,49cyclictest22448-21timerandwakeup12:19:151
28748999440,48cyclictest0-21swapper/111:08:411
28748999358,30cyclictest28730-21cyclictest08:54:041
2874199937,59cyclictest15-21ksoftirqd/011:09:020
28741999264,19cyclictest5790-21timerwakeupswit07:34:170
28741999165,20cyclictest24701-21latency_hist09:43:440
272412916,57sleep026760-21apt-get07:03:530
28748999043,40cyclictest10082-21fschecks_count09:19:051
28748999042,40cyclictest0-21swapper/110:58:421
2874899893,36cyclictest17762-21/usr/sbin/munin11:19:071
2874899893,36cyclictest17762-21/usr/sbin/munin11:19:071
28741998929,38cyclictest0-21swapper/011:14:180
2874899883,31cyclictest28635-21grep08:44:141
28741998857,22cyclictest21308-21netstat11:24:090
28753998760,21cyclictest11801-21apt-get11:09:022
28753998732,14cyclictest0-21swapper/211:49:092
2874899874,34cyclictest6150-21rm10:05:151
2874899874,31cyclictest3974-21sendmail_mailqu07:29:141
2874899874,31cyclictest3974-21sendmail_mailqu07:29:141
2874899873,41cyclictest25574-21processes09:44:151
2874899873,41cyclictest25574-21processes09:44:151
2874899873,33cyclictest1983-21fschecks_count09:59:031
2874899873,33cyclictest1983-21fschecks_count09:59:031
2874899873,30cyclictest31947-21sh09:55:131
28748998667,12cyclictest3322-21sh11:48:451
28748998667,12cyclictest3322-21sh11:48:451
28748998665,15cyclictest20807-21threads10:29:221
2874899864,31cyclictest26429-21/usr/sbin/munin08:38:531
2874899864,31cyclictest26429-21/usr/sbin/munin08:38:531
28748998637,41cyclictest0-21swapper/112:03:591
2874899863,37cyclictest13275-21grep11:10:261
2874899863,37cyclictest10734-21ssh10:13:571
2874899863,36cyclictest26933-21/usr/sbin/munin11:33:581
2874899863,33cyclictest10605-21grep12:00:091
2874899863,32cyclictest18885-21timerandwakeup08:14:171
2874899863,32cyclictest13006-21latency_hist07:58:451
2874899863,31cyclictest6403-21/usr/sbin/munin11:54:051
2874899863,31cyclictest4149-21ps11:49:181
2874899863,31cyclictest22890-21ssh09:39:171
2874899863,30cyclictest29010-21grep09:49:221
2874899863,29cyclictest10977-21grep07:49:181
2874199864,71cyclictest1357-21cron09:58:410
2874899854,38cyclictest19692-21switchtime09:34:171
2874899854,38cyclictest19692-21switchtime09:34:171
2874899854,32cyclictest26675-21unin-run11:33:421
2874899854,30cyclictest25537-21modprobe10:38:411
2874899854,30cyclictest25537-21modprobe10:38:411
28748998541,36cyclictest0-21swapper/111:39:171
2874899853,33cyclictest2826-21switchtime09:04:151
2874899853,32cyclictest29546-21grep10:44:141
2874899853,31cyclictest8155-21cmp07:43:441
2874899853,31cyclictest8155-21cmp07:43:441
2874899853,31cyclictest24413-21latency_hist08:33:431
2874899853,31cyclictest24413-21latency_hist08:33:431
28748998514,24cyclictest30129-21netstat08:49:101
28741998558,17cyclictest6492-21latency_hist09:13:450
28741998558,17cyclictest6492-21latency_hist09:13:450
2874899845,32cyclictest31447-21grep07:14:121
2874899844,32cyclictest8558-21switchtime10:09:201
2874899844,32cyclictest8558-21switchtime10:09:201
2874899844,32cyclictest31854-21/usr/sbin/munin10:49:241
2874899844,30cyclictest14019-21sshd07:59:171
2874899844,30cyclictest14019-21sshd07:59:171
2874899843,34cyclictest5219-21tune2fs07:34:041
2874899843,33cyclictest25494-21timerwakeupswit12:24:171
2874899843,32cyclictest719-21fschecks_time08:59:051
2874899843,32cyclictest16690-21/usr/sbin/munin10:24:031
2874899843,31cyclictest411-21fschecks_time07:19:041
2874899843,28cyclictest28223-21ps12:29:131
28748998361,16cyclictest14051-21sh09:24:361
2874899834,33cyclictest26150-21fschecks_time10:39:011
2874899833,33cyclictest25277-21wc08:34:161
2874899833,33cyclictest25277-21wc08:34:161
2874899833,33cyclictest16623-21grep12:09:191
2874899833,32cyclictest30354-21latency_hist07:13:441
2874899833,31cyclictest343-21runrttasks12:17:301
2874899833,31cyclictest343-21runrttasks12:17:301
2874899833,30cyclictest6798-21/usr/sbin/munin09:14:001
2874899833,30cyclictest21595-21fschecks_time08:24:031
2874899833,30cyclictest21595-21fschecks_time08:24:031
2874899833,30cyclictest17564-21latency_hist11:18:431
2874899833,28cyclictest16815-21sort09:29:211
2874899832,33cyclictest17933-21latency_hist08:13:441
28741998362,13cyclictest28730-21cyclictest08:34:140
28741998362,13cyclictest28730-21cyclictest08:34:140
2874899826,24cyclictest2017-21fschecks_time07:24:041
2874899826,24cyclictest2017-21fschecks_time07:24:041
2874899824,30cyclictest13998-21fschecks_count10:19:041
2874899824,30cyclictest13998-21fschecks_count10:19:041
2874899823,40cyclictest20338-21grep08:19:121
2874899823,32cyclictest6189-21ntp_kernel_pll_10:59:091
2874899823,32cyclictest6189-21ntp_kernel_pll_10:59:091
2874899823,31cyclictest16237-21modprobe08:08:411
2874899823,30cyclictest3964-21ssh09:09:001
2874899823,30cyclictest3964-21ssh09:09:001
2874899822,34cyclictest21481-21wc11:24:141
28741998248,26cyclictest28730-21cyclictest10:04:170
2874199817,54cyclictest10201-21diskstats12:00:000
28741998159,10cyclictest0-21swapper/008:23:450
28741998154,7cyclictest181rcu_preempt09:34:000
28741998154,7cyclictest181rcu_preempt09:34:000
28741998120,9cyclictest0-21swapper/009:52:040
28753998011,57cyclictest9654-21apt-get11:59:002
28741998060,10cyclictest0-21swapper/007:13:440
28741998047,7cyclictest181rcu_preempt11:56:560
28753997933,32cyclictest7929-21modprobe07:43:412
28753997933,32cyclictest7929-21modprobe07:43:412
2874899794,33cyclictest28730-21cyclictest07:08:461
28741997965,10cyclictest28730-21cyclictest11:33:580
28741997959,14cyclictest3532-21ntp_kernel_pll_10:55:060
28741997958,7cyclictest181rcu_preempt12:23:520
28741997955,15cyclictest26349-21ssh09:45:180
28741997955,15cyclictest26349-21ssh09:45:180
28753997856,12cyclictest0-21swapper/210:29:242
28741997859,13cyclictest7090-21ssh09:14:060
28741997857,15cyclictest1-21systemd07:41:110
28741997857,15cyclictest1-21systemd07:41:110
2875399779,24cyclictest0-21swapper/208:18:452
28741997762,10cyclictest28730-21cyclictest12:10:310
28741997756,13cyclictest32676-21kworker/u18:1-events_unbound09:23:430
28741997747,20cyclictest13228-21latency_hist07:58:450
2874199774,50cyclictest0-21swapper/008:57:190
28753997657,12cyclictest28730-21cyclictest12:04:152
28753997655,14cyclictest900-21threads07:19:162
28753997648,21cyclictest11385-21apt-key11:08:432
28753997629,37cyclictest30743-21cat07:13:512
28741997651,16cyclictest28730-21cyclictest11:05:560
28741997650,17cyclictest22791-21latency_hist08:28:430
28741997650,17cyclictest22791-21latency_hist08:28:430
28741997648,20cyclictest8031-21fschecks_time10:09:070
28741997648,20cyclictest8031-21fschecks_time10:09:070
28753997559,9cyclictest0-21swapper/209:39:152
28753997551,18cyclictest28730-21cyclictest10:59:542
28753997551,18cyclictest28730-21cyclictest10:59:542
28753997548,19cyclictest343-21runrttasks12:31:192
2874199757,53cyclictest3479-21diskmemload10:29:040
28741997555,14cyclictest5738-21cstates10:58:580
28741997555,14cyclictest5738-21cstates10:58:580
28741997548,16cyclictest181rcu_preempt10:36:420
28741997548,16cyclictest181rcu_preempt10:36:420
2875399741,46cyclictest0-21swapper/212:34:172
28741997459,7cyclictest0-21swapper/009:24:180
28741997455,14cyclictest28730-21cyclictest11:47:180
28741997455,14cyclictest28730-21cyclictest11:47:180
28741997453,12cyclictest0-21swapper/011:53:440
28741997448,7cyclictest181rcu_preempt12:34:100
28741997446,19cyclictest303-21latency_hist11:43:440
28741997445,19cyclictest8700-21fschecks_time07:44:170
2875399733,58cyclictest4911-21sed07:33:452
2875399733,58cyclictest4911-21sed07:33:452
2875399732,36cyclictest9604-21runrttasks09:18:442
28741997359,9cyclictest28730-21cyclictest08:29:000
28741997359,9cyclictest28730-21cyclictest08:29:000
28741997350,17cyclictest524-21in:imuxsock09:32:180
28741997343,19cyclictest0-21swapper/010:49:120
2874199733,48cyclictest15-21ksoftirqd/010:28:280
28753997251,15cyclictest28730-21cyclictest08:19:122
28741997258,8cyclictest0-21swapper/008:44:030
28741997247,12cyclictest0-21swapper/007:49:050
28741997243,20cyclictest2742-21unixbench_multi09:59:180
28741997243,20cyclictest2742-21unixbench_multi09:59:180
28741997234,8cyclictest0-21swapper/012:33:440
28741997217,29cyclictest15-21ksoftirqd/012:08:030
2876099713,54cyclictest6819-21chrt09:13:543
2875399717,31cyclictest0-21swapper/212:23:582
28753997158,7cyclictest0-21swapper/209:55:132
28753997153,8cyclictest0-21swapper/208:04:162
28753997153,12cyclictest22297-21mailstats12:19:132
2876099703,52cyclictest31574-21chrt10:48:423
2876099702,55cyclictest8448-21chrt07:43:553
2876099702,54cyclictest6432-21taskset09:13:343
2876099702,54cyclictest6432-21taskset09:13:343
28753997054,9cyclictest0-21swapper/211:29:282
2875399703,47cyclictest28902-21timerwakeupswit09:49:222
28741997048,12cyclictest0-21swapper/010:14:050
28741997045,15cyclictest26896-21ssh10:39:160
28741997037,27cyclictest0-21swapper/007:59:080
28741997037,27cyclictest0-21swapper/007:59:080
2876099693,53cyclictest12653-21chrt12:03:563
2876099693,53cyclictest11003-21chrt10:14:053
2876099692,55cyclictest23044-21chrt09:39:283
28753996953,10cyclictest7415-21ssh11:54:182
28753996951,13cyclictest21551-21cron09:38:412
28753996951,13cyclictest21551-21cron09:38:412
28753996932,9cyclictest181rcu_preempt10:03:412
28753996932,9cyclictest181rcu_preempt10:03:412
28741996955,8cyclictest0-21swapper/011:29:180
28741996953,8cyclictest0-21swapper/009:04:030
2876099683,53cyclictest16969-21taskset09:29:233
2876099683,51cyclictest20126-21taskset10:29:053
2876099682,53cyclictest22908-21chrt10:33:553
2876099682,53cyclictest22908-21chrt10:33:553
2876099682,52cyclictest10640-21taskset09:19:173
28753996851,12cyclictest27570-21sh11:34:112
28753996846,15cyclictest28730-21cyclictest09:44:052
28753996846,15cyclictest28730-21cyclictest09:44:052
28753996838,21cyclictest29407-21idleruntime-cro08:48:412
28741996847,15cyclictest17870-21cpuspeed11:18:580
28741996847,15cyclictest17870-21cpuspeed11:18:580
2876099673,52cyclictest7751-21taskset11:02:123
2876099673,52cyclictest7751-21taskset11:02:123
2876099673,51cyclictest28189-21chrt10:42:433
2876099673,51cyclictest23649-21chrt12:22:033
28753996744,13cyclictest0-21swapper/212:18:452
28753996743,14cyclictest0-21swapper/209:33:432
28753996742,17cyclictest3332-21latency_hist11:48:452
28753996742,17cyclictest3332-21latency_hist11:48:452
28753996740,19cyclictest5697-21ls10:04:192
2875399673,43cyclictest15338-21sh09:28:132
2874199673,55cyclictest18215-21cstates08:13:590
28741996729,9cyclictest181rcu_preempt08:52:220
2874199672,21cyclictest15-21ksoftirqd/010:23:190
2874199672,21cyclictest15-21ksoftirqd/010:23:190
2876099663,51cyclictest4416-21taskset10:56:163
2876099663,51cyclictest27357-21chrt11:34:053
2876099663,51cyclictest21981-21chrt11:24:373
2876099663,50cyclictest14178-21chrt09:24:573
2876099663,49cyclictest4737-21chrt11:49:503
2876099663,49cyclictest17470-21chrt12:11:293
2876099663,48cyclictest0-21swapper/308:33:543
2876099663,48cyclictest0-21swapper/308:33:543
2876099662,52cyclictest27069-21chrt09:47:183
2876099662,52cyclictest27069-21chrt09:47:183
2876099662,52cyclictest20902-21taskset09:36:573
2876099662,52cyclictest20902-21taskset09:36:573
2876099662,51cyclictest3945-21chrt10:02:263
2876099662,51cyclictest3945-21chrt10:02:263
2876099662,51cyclictest11335-21chrt12:00:563
2876099662,50cyclictest8893-21taskset10:09:313
2876099662,50cyclictest8893-21taskset10:09:313
28753996651,10cyclictest28730-21cyclictest08:53:442
2874199663,34cyclictest343-21runrttasks12:20:050
2874199662,39cyclictest15-21ksoftirqd/008:04:200
2754826614,34sleep30-21swapper/307:05:183
257492660,7sleep20-21swapper/210:38:452
257492660,7sleep20-21swapper/210:38:452
2876099654,48cyclictest0-21swapper/307:53:593
2876099653,51cyclictest832-21chrt08:59:083
2876099653,48cyclictest32121-21taskset07:16:303
2876099652,51cyclictest30703-21chrt08:49:523
2876099652,51cyclictest23984-21chrt11:28:543
2876099652,51cyclictest1663-21taskset11:44:313
2876099652,51cyclictest1663-21taskset11:44:313
2876099652,50cyclictest31108-21chrt09:53:503
2876099652,50cyclictest20066-21chrt11:22:383
2876099652,50cyclictest20066-21chrt11:22:383
2876099652,50cyclictest19768-21chrt08:18:543
2876099652,49cyclictest995-21taskset07:19:173
2876099652,49cyclictest675-21taskset10:49:483
2876099652,49cyclictest4417-21chrt07:30:103
2876099652,49cyclictest4417-21chrt07:30:103
2876099652,49cyclictest27743-21taskset08:43:163
2876099652,49cyclictest27743-21taskset08:43:163
2876099652,49cyclictest11073-21taskset11:08:113
2875399654,43cyclictest17474-21awk11:18:412
28753996543,15cyclictest27955-21kworker/u17:3+events_unbound07:58:412
28753996539,18cyclictest8692-21fschecks_time07:44:172
2875399653,43cyclictest18851-21fschecks_time12:14:022
28741996543,7cyclictest181rcu_preempt07:28:070
28741996543,7cyclictest181rcu_preempt07:28:070
28741996543,6cyclictest181rcu_preempt07:30:090
28741996543,6cyclictest181rcu_preempt07:30:090
28741996542,7cyclictest181rcu_preempt07:13:450
2874199653,44cyclictest638-21sort08:59:040
2874199652,25cyclictest15-21ksoftirqd/008:09:130
2876099643,49cyclictest28767-21chrt12:29:413
2876099643,49cyclictest18575-21chrt12:13:523
2876099643,49cyclictest18575-21chrt12:13:523
2876099643,48cyclictest10057-21chrt07:48:533
2876099642,51cyclictest13758-21chrt10:18:563
2876099642,51cyclictest13758-21chrt10:18:563
2876099642,50cyclictest5057-21taskset07:33:593
2876099642,50cyclictest17306-21chrt10:24:103
2876099642,49cyclictest4984-21chrt10:04:043
2876099642,49cyclictest29530-21taskset09:50:463
28753996446,9cyclictest0-21swapper/208:34:182
28753996446,9cyclictest0-21swapper/208:34:182
28753996446,12cyclictest17871-21cpuspeed11:18:582
28753996446,12cyclictest17871-21cpuspeed11:18:582
28753996445,13cyclictest8196-21sh10:09:112
28753996445,13cyclictest8196-21sh10:09:112
28753996437,19cyclictest3083-21cron07:28:412
28753996437,19cyclictest3083-21cron07:28:412
28753996435,20cyclictest12031-21sh10:14:572
2875399643,42cyclictest343-21runrttasks09:00:242
2875399643,40cyclictest17779-21grep10:24:192
2875399642,26cyclictest39-21ksoftirqd/207:59:062
2875399642,26cyclictest39-21ksoftirqd/207:59:062
2876099634,47cyclictest14459-21chrt08:01:333
2876099634,47cyclictest14459-21chrt08:01:333
2876099633,47cyclictest3358-21chrt09:07:263
2876099633,47cyclictest18265-21chrt08:14:003
2876099632,49cyclictest8991-21chrt11:58:293
2876099632,49cyclictest31931-21chrt11:41:473
2876099632,49cyclictest30472-21chrt12:33:503
2876099632,49cyclictest28929-21chrt07:08:563
2876099632,49cyclictest28213-21chrt08:44:013
2876099632,49cyclictest25845-21chrt12:24:513
2876099632,49cyclictest15969-21taskset11:14:323
2876099632,49cyclictest14057-21chrt11:12:333
2876099632,48cyclictest7638-21chrt07:39:583
2876099632,48cyclictest7638-21chrt07:39:583
2876099632,30cyclictest22955-21chrt08:28:503
2876099632,30cyclictest22955-21chrt08:28:503
28753996349,8cyclictest0-21swapper/209:04:172
28753996347,10cyclictest16883-21runrttasks12:09:482
28753996346,11cyclictest12555-21systemd-run09:23:422
2875399634,40cyclictest5545-21wc07:34:142
28753996343,14cyclictest16485-21latency_hist10:23:442
28753996343,14cyclictest16485-21latency_hist10:23:442
28753996343,13cyclictest28539-21latency_hist10:43:432
28753996340,13cyclictest0-21swapper/209:13:452
28753996340,13cyclictest0-21swapper/209:13:452
28753996339,10cyclictest0-21swapper/207:53:452
28753996329,25cyclictest21622-21tune2fs08:24:042
28753996329,25cyclictest21622-21tune2fs08:24:042
28741996342,13cyclictest0-21swapper/008:40:040
28741996342,13cyclictest0-21swapper/008:40:040
2876099623,47cyclictest16012-21taskset08:05:503
2876099622,48cyclictest21911-21chrt08:24:133
2876099622,48cyclictest21911-21chrt08:24:133
2875399624,32cyclictest29149-21fschecks_time10:44:042
28753996216,34cyclictest20789-21diskmemload11:23:562
28741996244,9cyclictest0-21swapper/007:18:520
2876099612,47cyclictest32033-21chrt08:54:183
2876099612,47cyclictest2867-21chrt07:25:533
2876099612,47cyclictest2867-21chrt07:25:533
2876099612,47cyclictest17651-21chrt08:11:173
28753996041,11cyclictest23360-21latency08:29:082
28753996041,11cyclictest23360-21latency08:29:082
28753996040,10cyclictest0-21swapper/208:54:202
2875399603,42cyclictest27300-21timerandwakeup08:40:142
2875399603,42cyclictest27300-21timerandwakeup08:40:142
28753995943,9cyclictest0-21swapper/211:39:012
2875399593,42cyclictest16618-21cstates08:09:012
28753995914,34cyclictest29761-21cat07:09:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional