You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-19 - 21:57
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Fri Dec 19, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19566215011,39sleep119572-21munin-run07:09:351
196919911855,56cyclictest26644-21sh10:49:330
196919911549,56cyclictest475-21modprobe07:49:330
196919911549,56cyclictest475-21modprobe07:49:330
196969910864,36cyclictest0-21swapper/111:34:441
197019910659,34cyclictest0-21swapper/210:34:372
197019910655,39cyclictest0-21swapper/211:34:362
196919910553,45cyclictest0-21swapper/010:34:580
196919910347,45cyclictest0-21swapper/008:25:080
19691991033,25cyclictest15715-21/usr/sbin/munin11:55:050
18794210310,66sleep218836-21sendmail_mailqu07:05:162
196919910250,45cyclictest0-21swapper/007:14:430
196969910058,35cyclictest0-21swapper/110:15:081
1953229918,55sleep019558-21munin-run07:09:340
19696999759,31cyclictest0-21swapper/111:00:101
19691999744,45cyclictest0-21swapper/009:24:360
19691999744,45cyclictest0-21swapper/009:24:360
19691999742,49cyclictest19681-21cyclictest09:45:100
1969199973,20cyclictest27530-21apt-get12:29:550
1969199973,20cyclictest27530-21apt-get12:29:550
19701999658,31cyclictest0-21swapper/211:05:042
19701999654,35cyclictest0-21swapper/207:15:112
19691999648,42cyclictest0-21swapper/008:54:560
1969199964,13cyclictest20837-21apt-get12:09:470
19691999618,24cyclictest21690-21seq10:33:190
19701999550,38cyclictest0-21swapper/208:45:092
1969199953,20cyclictest2091-21latency_hist07:54:360
1969199953,20cyclictest2091-21latency_hist07:54:360
19691999531,54cyclictest0-21swapper/007:34:360
19701999451,35cyclictest0-21swapper/209:10:092
19701999451,35cyclictest0-21swapper/209:10:092
1969199943,21cyclictest4307-21cstates11:19:530
1969199943,19cyclictest17244-21apt-get10:19:450
1969199942,29cyclictest27008-21cstates10:49:500
1969199932,21cyclictest17071-21apt-get11:59:470
19691999319,21cyclictest9347-21fschecks_count09:54:560
1969199925,12cyclictest14125-21apt-get11:49:450
1969199925,11cyclictest19872-21apt-get07:09:450
1969199922,21cyclictest24451-21unixbench_multi07:20:080
19701999153,32cyclictest0-21swapper/211:49:502
19696999172,13cyclictest27203-21latency_hist12:29:351
19696999167,16cyclictest19681-21cyclictest07:54:551
19691999135,51cyclictest0-21swapper/011:29:570
19691999134,49cyclictest20181-21cron10:29:330
19691999133,52cyclictest0-21swapper/012:29:360
1969199913,22cyclictest28503-21/usr/sbin/munin10:55:220
1969199913,20cyclictest29141-21apt-get12:34:480
1969199913,20cyclictest29141-21apt-get12:34:480
1969199912,25cyclictest10019-21grep08:16:060
1969199912,22cyclictest4329-21apt-get09:39:460
19691999113,21cyclictest28351-21cat07:34:450
19691999113,21cyclictest28351-21cat07:34:450
19696999053,30cyclictest0-21swapper/111:49:551
1969199905,13cyclictest7213-21apt-get08:09:460
19691999041,40cyclictest0-21swapper/010:10:090
19691999041,40cyclictest0-21swapper/010:10:090
1969199903,26cyclictest9895-21find11:35:050
1969199903,23cyclictest31768-21apt-get11:04:480
19691999026,55cyclictest0-21swapper/008:45:090
19691999015,20cyclictest25168-21grep07:24:500
19701998950,30cyclictest0-21swapper/211:35:062
19701998947,34cyclictest5371-21systemd-run08:04:352
19696998965,14cyclictest0-21swapper/109:25:061
19696998965,14cyclictest0-21swapper/109:25:061
19696998944,37cyclictest0-21swapper/107:45:091
19696998944,37cyclictest0-21swapper/107:45:091
1969199893,25cyclictest12528-21/usr/sbin/munin11:44:490
1969199893,23cyclictest2358-21apt-get07:54:440
1969199893,20cyclictest25227-21apt-get09:04:500
1969199893,20cyclictest25227-21apt-get09:04:500
1969199892,27cyclictest27126-21/usr/sbin/munin09:09:570
1969199892,27cyclictest27126-21/usr/sbin/munin09:09:570
19691998916,7cyclictest0-21swapper/008:09:330
19696998820,31cyclictest2091-21latency_hist07:54:331
19696998820,31cyclictest2091-21latency_hist07:54:331
19691998834,47cyclictest0-21swapper/009:14:500
1969199883,23cyclictest5514-21apt-get11:24:470
1969199883,23cyclictest30949-21grep07:40:150
1969199883,23cyclictest30949-21grep07:40:150
1969199882,23cyclictest11324-21/usr/sbin/munin08:20:080
1969199882,22cyclictest20376-21apt-get08:49:510
19691998815,20cyclictest12054-21seq11:42:090
19701998747,33cyclictest0-21swapper/210:34:432
19701998716,27cyclictest31060-21tr09:21:002
19701998716,27cyclictest31060-21tr09:21:002
19696998760,14cyclictest0-21swapper/107:29:551
19696998750,12cyclictest181rcu_preempt08:34:561
19696998740,39cyclictest0-21swapper/108:44:341
1969199873,26cyclictest8117-21/usr/sbin/munin09:50:040
1969199873,25cyclictest23596-21sshd08:59:440
1969199873,22cyclictest18615-21cat08:44:360
1969199873,20cyclictest1064-21/usr/sbin/munin09:29:550
1969199873,20cyclictest1064-21/usr/sbin/munin09:29:550
1969199873,19cyclictest3990-21apt-get07:59:450
19701998663,16cyclictest19681-21cyclictest12:34:352
19701998663,16cyclictest19681-21cyclictest12:34:352
19696998644,35cyclictest0-21swapper/110:49:331
1969699862,34cyclictest29957-21sed12:35:071
1969699862,34cyclictest29957-21sed12:35:071
1969199863,28cyclictest820-21/usr/sbin/munin09:29:340
1969199863,28cyclictest820-21/usr/sbin/munin09:29:340
1969199863,23cyclictest14538-21/usr/sbin/munin08:30:060
1969199862,25cyclictest2869-21fschecks_count09:34:550
1969199862,25cyclictest25127-21ls10:44:360
19701998559,18cyclictest17213-21apt-key11:59:362
19696998561,13cyclictest0-21swapper/107:19:541
19696998559,15cyclictest0-21swapper/109:34:561
19696998530,48cyclictest288-21dbus-daemon08:04:351
1969199853,30cyclictest11288-21awk10:00:000
1969199853,30cyclictest11288-21awk10:00:000
1969199853,29cyclictest24067-21/usr/sbin/munin12:20:000
1969199853,25cyclictest15579-21/usr/sbin/munin08:34:500
1969199853,23cyclictest12868-21ntpq10:05:010
1969199853,23cyclictest12868-21ntpq10:05:010
1969199853,20cyclictest2529-21apt-get11:14:480
1969199852,23cyclictest19370-21runrttasks12:04:520
19701998437,39cyclictest0-21swapper/211:24:372
19696998429,44cyclictest0-21swapper/108:49:371
1969699842,30cyclictest364-21snmpd09:41:001
1969199843,24cyclictest1431-21ntpq11:10:030
1969199843,21cyclictest22455-21/usr/sbin/munin12:14:490
1969199842,28cyclictest16390-21cat10:15:080
1969199842,22cyclictest30516-21chrt10:59:590
1970199833,24cyclictest17920-21taskset10:20:072
19701998331,9cyclictest0-21swapper/210:05:082
19701998331,9cyclictest0-21swapper/210:05:082
19696998361,16cyclictest25351-21fschecks_count07:24:551
19696998331,44cyclictest0-21swapper/108:49:451
19696998319,28cyclictest29779-21cron09:19:331
19701998236,38cyclictest0-21swapper/212:09:332
1970199823,25cyclictest22615-21/usr/sbin/munin12:14:542
1970199823,23cyclictest20995-21/usr/sbin/munin08:50:072
1970199822,30cyclictest12236-21latency_hist10:04:362
1970199822,30cyclictest12236-21latency_hist10:04:362
19696998244,32cyclictest19338-21ntp_kernel_pll_10:25:011
19696998220,28cyclictest4943-21sort11:20:101
19701998161,14cyclictest19681-21cyclictest12:19:552
19701998161,10cyclictest0-21swapper/209:50:442
19701998160,11cyclictest0-21swapper/210:40:082
19701998144,30cyclictest0-21swapper/207:40:142
19701998144,30cyclictest0-21swapper/207:40:142
19701998135,37cyclictest0-21swapper/207:24:362
19701998129,45cyclictest20455-21wc07:10:062
19701998117,26cyclictest1-21systemd09:34:342
19701998117,26cyclictest1-21systemd09:34:342
19701998112,22cyclictest26668-21latency_hist09:09:352
19701998112,22cyclictest26668-21latency_hist09:09:352
19696998159,14cyclictest25718-21ntp_kernel_pll_09:05:021
19696998159,14cyclictest25718-21ntp_kernel_pll_09:05:021
1969699813,27cyclictest11815-21unixbench_multi11:40:101
19696998128,45cyclictest0-21swapper/111:14:531
1969699812,35cyclictest19681-21cyclictest07:38:301
1969699812,35cyclictest19681-21cyclictest07:38:301
1970199807,12cyclictest0-21swapper/209:42:012
19701998058,12cyclictest0-21swapper/210:54:502
19701998056,16cyclictest0-21swapper/207:34:512
19701998056,16cyclictest0-21swapper/207:34:512
1970199804,25cyclictest28111-21seq10:52:552
19701998039,29cyclictest20901-21ls12:09:492
19701998036,37cyclictest0-21swapper/211:44:342
19701998034,39cyclictest0-21swapper/207:49:332
19701998034,39cyclictest0-21swapper/207:49:332
1970199803,25cyclictest10523-21modprobe09:59:342
1970199803,23cyclictest3009-21/usr/sbin/munin07:55:062
19701998025,48cyclictest26515-21modprobe07:29:332
19701998016,26cyclictest30282-21expr10:59:512
1969699804,25cyclictest13889-21latency_hist10:09:361
1969699804,25cyclictest13889-21latency_hist10:09:361
1969699804,23cyclictest27556-21apt-get12:29:551
1969699804,23cyclictest27556-21apt-get12:29:551
1969699803,29cyclictest13133-21timerwakeupswit08:25:081
1969699803,21cyclictest13993-21apt-get10:09:441
1969699803,21cyclictest13993-21apt-get10:09:441
19696998028,46cyclictest0-21swapper/111:59:371
19696998019,22cyclictest10026-21unixbench_multi09:55:101
19696998018,20cyclictest7590-21latency08:10:001
19696998017,24cyclictest288-21dbus-daemon12:09:341
19696998014,20cyclictest18537-21seq10:24:121
19701997954,18cyclictest16888-21sendmail08:39:342
19701997942,32cyclictest0-21swapper/210:15:082
1970199793,29cyclictest13504-21/usr/sbin/munin11:45:102
1970199793,28cyclictest8651-21cat08:14:352
1970199793,23cyclictest29588-21/usr/sbin/munin12:34:592
1970199793,23cyclictest29588-21/usr/sbin/munin12:34:592
19701997931,41cyclictest0-21swapper/212:25:062
19701997928,44cyclictest0-21swapper/211:15:082
1970199792,31cyclictest11788-21taskset08:21:282
1970199792,23cyclictest4051-21modprobe09:39:332
1969699798,11cyclictest0-21swapper/109:09:511
1969699798,11cyclictest0-21swapper/109:09:511
1969699793,30cyclictest25448-21chrt12:24:331
1969699793,25cyclictest21660-21munin-node07:14:421
19696997931,38cyclictest0-21swapper/110:30:101
19696997928,44cyclictest0-21swapper/109:50:001
1969699792,32cyclictest31956-21cut11:04:541
1969699792,31cyclictest13901-21grep11:49:351
1969699792,23cyclictest8549-21apt-get08:14:471
19696997914,20cyclictest31163-21grep09:21:021
19696997914,20cyclictest31163-21grep09:21:021
1970199783,27cyclictest6311-21/usr/sbin/munin11:25:242
1970199783,27cyclictest26675-21/usr/sbin/munin10:49:342
1970199783,26cyclictest205-21systemd-journal08:05:042
1970199783,26cyclictest14424-21ntp_kernel_pll_08:30:012
1970199783,24cyclictest17799-21/usr/sbin/munin12:00:182
19701997831,40cyclictest0-21swapper/210:10:082
19701997831,40cyclictest0-21swapper/210:10:082
19701997830,40cyclictest0-21swapper/208:54:532
19701997825,46cyclictest0-21swapper/209:45:102
1970199782,29cyclictest23619-21taskset08:59:472
19696997837,34cyclictest0-21swapper/110:35:041
1969699783,22cyclictest8302-21awk11:30:071
19696997827,43cyclictest0-21swapper/108:59:571
1969699782,30cyclictest2625-21latency_hist09:34:361
1969699782,30cyclictest2625-21latency_hist09:34:361
1969699782,30cyclictest1070-21taskset11:09:531
1970199773,25cyclictest866-21sed09:29:352
1970199773,25cyclictest866-21sed09:29:352
1970199773,24cyclictest1441-21/usr/sbin/munin11:10:032
19701997726,44cyclictest0-21swapper/208:29:362
19701997725,44cyclictest0-21swapper/207:50:082
19701997725,44cyclictest0-21swapper/207:50:082
1969699773,23cyclictest21548-21sed12:10:071
1969699773,22cyclictest20478-21/usr/sbin/munin07:10:061
19696997727,41cyclictest0-21swapper/107:39:591
19696997727,41cyclictest0-21swapper/107:39:591
1969699772,26cyclictest11382-21mailstats08:20:091
1970199763,21cyclictest26951-21fschecks_count07:29:542
19701997625,42cyclictest0-21swapper/210:29:362
1970199762,27cyclictest18130-21uptime08:40:102
1969699764,26cyclictest22076-21cstates08:54:521
1969699763,23cyclictest22874-21latency12:14:581
19696997625,44cyclictest0-21swapper/110:49:521
1969699762,23cyclictest7034-21/usr/sbin/munin11:25:401
1970199753,34cyclictest9048-21apt-get08:14:482
1970199753,24cyclictest28952-21ntp_kernel_pll_09:15:012
1969699754,22cyclictest10750-21/usr/sbin/munin09:59:501
1969699754,22cyclictest10750-21/usr/sbin/munin09:59:501
1969699753,31cyclictest23804-21cat10:39:501
1969699753,25cyclictest6547-21grep08:05:101
1969699752,25cyclictest14763-21timerwakeupswit08:30:081
1969699752,24cyclictest6821-21awk09:45:101
1969699749,12cyclictest0-21swapper/112:04:231
1969699743,21cyclictest28581-21/usr/sbin/munin10:54:501
1970799625,43cyclictest25228-21taskset09:04:453
1970799625,43cyclictest25228-21taskset09:04:453
1970799604,44cyclictest23822-21taskset08:59:553
1970799594,43cyclictest16408-21chrt08:35:103
1970799592,45cyclictest15503-21taskset10:14:363
1970799592,45cyclictest15503-21taskset10:14:363
1970799583,42cyclictest12705-21taskset10:04:573
1970799583,42cyclictest12705-21taskset10:04:573
1970799583,41cyclictest6214-21taskset08:05:063
1970799574,40cyclictest19723-21chrt08:45:103
1970799573,41cyclictest22193-21chrt07:14:583
1836525713,26sleep30-21swapper/307:04:563
1970799564,38cyclictest30001-21taskset07:39:483
1970799564,38cyclictest30001-21taskset07:39:483
1970799563,40cyclictest27190-21chrt09:09:593
1970799563,40cyclictest27190-21chrt09:09:593
1970799562,42cyclictest25505-21taskset12:24:353
1970799553,37cyclictest1490-21chrt07:50:083
1970799553,37cyclictest1490-21chrt07:50:083
1970799552,41cyclictest1410-21taskset09:29:553
1970799552,41cyclictest1410-21taskset09:29:553
1970799544,37cyclictest4767-21chrt08:00:083
1970799544,37cyclictest11220-21taskset09:59:593
1970799544,37cyclictest11220-21taskset09:59:593
1970799544,34cyclictest457-21kworker/3:1+events12:15:093
1970799543,39cyclictest20568-21taskset10:29:503
1970799543,38cyclictest3433-21chrt11:15:083
1970799543,38cyclictest25733-21taskset07:25:063
1970799543,38cyclictest24042-21taskset07:20:003
1970799543,38cyclictest10656-21taskset11:39:333
1970799535,36cyclictest19467-21taskset12:04:553
1970799534,37cyclictest4968-21chrt11:20:103
1970799534,37cyclictest27296-21chrt10:49:583
1970799534,36cyclictest21874-21chrt08:54:373
1970799534,35cyclictest32222-21chrt07:45:063
1970799534,35cyclictest32222-21chrt07:45:063
1970799533,39cyclictest16077-21chrt10:15:013
1970799533,38cyclictest17936-21chrt12:00:223
1970799533,37cyclictest9048-21taskset09:54:363
1970799533,37cyclictest30665-21chrt11:00:023
1970799533,37cyclictest29895-21chrt10:59:333
1970799533,37cyclictest27049-21chrt12:29:333
1970799533,37cyclictest11402-21chrt11:40:023
1970799533,37cyclictest11308-21chrt08:20:083
1970799524,36cyclictest8892-21chrt08:14:373
1970799524,35cyclictest28695-21taskset07:34:583
1970799524,35cyclictest28695-21taskset07:34:583
1970799523,37cyclictest22573-21taskset10:35:003
1970799523,36cyclictest12829-21chrt08:25:023
1970799523,36cyclictest12498-21taskset11:44:433
1970799515,32cyclictest19130-21taskset10:24:563
1970799514,35cyclictest29953-21taskset12:35:073
1970799514,35cyclictest29953-21taskset12:35:073
1970799514,33cyclictest14647-21chrt11:50:023
1970799513,36cyclictest27776-21chrt12:30:223
1970799513,36cyclictest27776-21chrt12:30:223
1970799513,35cyclictest3165-21taskset09:35:013
1970799513,35cyclictest21447-21chrt07:14:383
1970799513,34cyclictest8133-21chrt11:30:023
1970799513,34cyclictest5840-21chrt09:44:363
1970799512,37cyclictest31875-21chrt09:24:373
1970799512,37cyclictest31875-21chrt09:24:373
1970799512,35cyclictest6707-21chrt09:45:093
1970799506,32cyclictest18361-21taskset10:21:533
1970799504,36cyclictest0-21swapper/311:24:483
1970799503,37cyclictest17559-21taskset08:39:593
1970799503,35cyclictest9641-21chrt08:15:593
1970799503,35cyclictest32699-21chrt11:05:083
1970799503,35cyclictest17488-21chrt11:59:373
1970799503,28cyclictest457-21kworker/3:1+events10:45:053
1970799502,34cyclictest24812-21chrt10:41:443
1970799493,34cyclictest1077-21taskset11:09:543
1970799493,33cyclictest27236-21chrt07:30:003
1970799492,35cyclictest2415-21chrt07:54:503
1970799483,35cyclictest23377-21chrt08:59:353
1970799473,31cyclictest32497-21chrt09:25:053
1970799473,31cyclictest32497-21chrt09:25:053
1970799472,36cyclictest9942-21chrt09:55:093
1970799463,29cyclictest14452-21taskset08:30:043
1970799463,28cyclictest20807-21chrt12:09:373
1970799453,32cyclictest28955-21chrt09:15:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional