You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-31 - 21:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Sun May 31, 2026 12:42:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2735693996551,5cyclictest0-21swapper/111:26:011
273569399555,10cyclictest2893718-21ssh12:21:021
2735692995445,5cyclictest151ktimers/011:01:020
2735693995242,6cyclictest2838388-21idleruntime-cro10:55:001
2735692995236,8cyclictest12350irq/32-eth010:56:000
2735692995236,8cyclictest12350irq/32-eth010:56:000
2735693995125,23cyclictest2841576-21ssh11:00:001
2735693995125,23cyclictest2841576-21ssh11:00:001
2735694995039,7cyclictest1952300-21cups-browsed11:51:032
2735693994826,18cyclictest2867008-21ssh11:40:001
273569499477,7cyclictest171rcu_preempt12:30:592
2735692994739,5cyclictest1-21systemd11:25:020
2735693994638,5cyclictest2819379-21munin-plugin-st10:25:001
2735693994621,4cyclictest0-21swapper/112:11:041
2735694994536,6cyclictest2886118-21grep12:10:012
2735694994535,7cyclictest0-21swapper/209:50:492
2735693994520,3cyclictest0-21swapper/110:00:011
2735693994519,23cyclictest2902994-21kernelversion12:36:001
2735693994519,23cyclictest2902994-21kernelversion12:36:001
2735694994434,7cyclictest2854295-21basename11:20:012
2735693994436,4cyclictest0-21swapper/108:46:031
2735693994424,18cyclictest0-21swapper/109:00:001
2735693994418,4cyclictest1056-21mta-sts-daemon09:40:001
273569399441,19cyclictest2842822-21ssh11:01:011
2735692994436,5cyclictest421-21dbus-daemon12:15:010
2735692994431,9cyclictest2900174-21sshd12:31:050
2735693994318,8cyclictest2785379-21latency09:30:541
2735693994318,8cyclictest2785379-21latency09:30:541
2735692994337,4cyclictest2854266-21ssh11:20:010
2735692994331,9cyclictest2825693-21idleruntime-cro10:35:000
2735693994233,6cyclictest2822557-21systemd10:30:011
2735693994233,6cyclictest2822557-21systemd10:30:011
273569399423,15cyclictest2762040-21sshd08:36:031
2735693994219,4cyclictest2832850-21fschecks_count10:45:551
2735693994219,4cyclictest2832850-21fschecks_count10:45:551
2735693994217,7cyclictest2755985-21needreboot08:15:581
273569399421,21cyclictest1952297-21cupsd10:41:091
273569399421,21cyclictest1952297-21cupsd10:41:091
2735692994236,4cyclictest2905175-21ssh12:40:010
2735692994236,4cyclictest2905175-21ssh12:40:010
2735692994228,11cyclictest2764192-21systemd-run08:45:010
2735694994133,5cyclictest2764199-21systemd-run08:45:012
2735693994120,5cyclictest1055-21mta-sts-daemon12:07:301
2735693994120,3cyclictest1051-21mta-sts-daemon10:16:041
2735692994133,5cyclictest516-21in:imuxsock09:20:000
273569499406,29cyclictest2823935-21ssh10:31:112
2735694994032,5cyclictest2828877-21munin-run10:40:002
2735694994032,5cyclictest2768695-21systemd-run09:00:012
2735694994032,4cyclictest0-21swapper/208:20:012
2735694994030,4cyclictest0-21swapper/210:56:002
2735694994030,4cyclictest0-21swapper/210:56:002
273569499402,31cyclictest2861952-21ssh11:31:042
2735692994035,2cyclictest0-21swapper/012:25:510
2735692994033,4cyclictest2744705-21systemd-run07:40:010
2735692994027,10cyclictest2793915-21munin-plugin-st09:45:000
2735692994025,8cyclictest2816167-21cron10:20:000
273569399399,9cyclictest2771673-21diskmemload09:46:001
2735693993931,5cyclictest2782035-21fschecks_count09:25:541
2735693993931,4cyclictest0-21swapper/107:11:031
273569399392,19cyclictest1952300-21cups-browsed08:51:031
273569399392,19cyclictest1952300-21cups-browsed08:51:031
273569399392,15cyclictest261-21systemd-journal10:11:031
273569399392,15cyclictest261-21systemd-journal10:11:031
2735693993918,3cyclictest0-21swapper/111:20:551
2735692993933,3cyclictest2883770-21fschecks_count12:05:570
2735692993933,3cyclictest0-21swapper/009:05:010
2735692993928,5cyclictest2790717-21munin-run09:40:000
273569499384,26cyclictest2896968-21ssh12:26:042
273569499381,29cyclictest2858748-21ssh11:26:032
2735693993819,2cyclictest0-21swapper/110:10:001
2735693993819,2cyclictest0-21swapper/110:10:001
2735692993831,4cyclictest12350irq/32-eth008:21:030
2735692993830,5cyclictest421-21dbus-daemon08:55:010
2735692993830,5cyclictest421-21dbus-daemon08:55:010
273569499373,27cyclictest2807953-21ssh10:06:062
273569499373,27cyclictest2807953-21ssh10:06:062
2735694993731,3cyclictest0-21swapper/209:56:122
2735694993727,4cyclictest0-21swapper/207:26:032
273569399376,21cyclictest428-21mta-sts-daemon11:51:021
273569399374,8cyclictest2827015-21ssh10:36:061
2735692993731,4cyclictest2828973-21latency_hist10:40:020
273569499368,21cyclictest2811004-21ssh10:10:582
273569499368,21cyclictest2811004-21ssh10:10:582
273569499365,5cyclictest171rcu_preempt10:16:102
2735694993630,2cyclictest0-21swapper/210:50:002
2735694993630,2cyclictest0-21swapper/210:50:002
2735694993629,4cyclictest2905243-21latency_hist12:40:022
2735694993629,4cyclictest2905243-21latency_hist12:40:022
2735694993629,4cyclictest2821191-21unixbench_singl10:26:172
2735694993629,4cyclictest2821191-21unixbench_singl10:26:172
2735694993628,5cyclictest2793903-21ssh09:45:002
2735694993626,7cyclictest2766482-21needreboot08:50:592
2735694993626,7cyclictest2766482-21needreboot08:50:592
2735693993629,4cyclictest0-21swapper/107:51:031
2735693993626,5cyclictest0-21swapper/108:26:071
2735692993622,10cyclictest151ktimers/012:00:580
2735694993529,3cyclictest0-21swapper/208:25:012
2735694993528,4cyclictest1054-21mta-sts-daemon09:19:342
2735694993526,6cyclictest1056-21mta-sts-daemon07:55:582
2735694993525,6cyclictest420-21avahi-daemon11:42:252
273569399353,26cyclictest2823877-21ssh10:31:091
273569399353,25cyclictest2846050-21ssh11:06:041
2735693993527,5cyclictest2797741-21aten_r2power_po09:50:501
2735693993525,5cyclictest0-21swapper/111:16:001
2735693993524,8cyclictest2771673-21diskmemload11:56:061
2735693993522,8cyclictest2849717-21unixbench_multi11:11:131
2735692993524,9cyclictest2749118-21unixbench_singl07:51:150
273569499343,20cyclictest2772856-21sh09:11:002
2735694993427,3cyclictest0-21swapper/212:16:042
2735694993427,3cyclictest0-21swapper/209:46:012
2735694993426,5cyclictest2880557-21fschecks_count12:00:532
273569499342,28cyclictest2785662-21ssh09:31:022
273569499342,28cyclictest2785662-21ssh09:31:022
2735693993426,5cyclictest2758177-21cron08:25:011
2735693993425,6cyclictest2899649-21fschecks_time12:30:531
2735693993425,5cyclictest2781203-21cron09:25:001
273569299343,23cyclictest2833250-21ssh10:46:020
273569299343,23cyclictest2833250-21ssh10:46:020
2735692993426,6cyclictest12350irq/32-eth010:01:090
2735692993425,6cyclictest2753679-21cut08:10:020
273569299342,23cyclictest2830148-21ssh10:41:090
273569299342,23cyclictest2830148-21ssh10:41:090
2735692993422,3cyclictest0-21swapper/009:25:540
2735692993418,13cyclictest151ktimers/009:51:040
2735694993326,4cyclictest2782969-21unixbench-2d09:26:122
2735694993326,4cyclictest0-21swapper/212:00:012
2735694993325,5cyclictest2745716-21proc_pri07:41:072
2735694993325,5cyclictest2745716-21proc_pri07:41:072
2735694993324,3cyclictest0-21swapper/210:51:062
2735694993322,8cyclictest2747085-21ntp_kernel_pll_07:46:042
273569499332,24cyclictest2849272-21ssh11:11:052
2735693993320,10cyclictest695-21cron07:25:011
273569299338,20cyclictest2858660-21ssh11:25:590
2735692993324,7cyclictest2817499-21sh10:21:050
2735692993323,8cyclictest753-21ntpd10:06:020
2735692993323,8cyclictest753-21ntpd10:06:020
2735692993323,7cyclictest2752187-21/usr/sbin/munin08:05:020
2735692993323,6cyclictest2760541-21sshd08:31:030
2735692993322,5cyclictest151ktimers/007:31:020
273569299332,24cyclictest2811132-21sshd10:11:030
273569299332,24cyclictest2811132-21sshd10:11:030
273569499325,17cyclictest2871535-21ssh11:46:052
273569499323,24cyclictest2846050-21ssh11:06:042
2735693993227,2cyclictest0-21swapper/109:11:021
2735693993226,3cyclictest0-21swapper/109:41:071
2735693993224,5cyclictest2862495-21head11:31:131
2735692993223,7cyclictest2890786-21mailstats12:16:130
2735692993222,3cyclictest0-21swapper/011:06:040
2735692993220,9cyclictest12350irq/32-eth010:26:060
2735692993220,9cyclictest12350irq/32-eth010:26:060
2735692993220,9cyclictest0-21swapper/011:40:010
273569499313,22cyclictest2788798-21ssh09:36:062
2735694993124,4cyclictest2854937-21aten2.4-expect11:20:492
2735694993124,3cyclictest0-21swapper/210:01:072
2735694993122,6cyclictest659-21snmpd12:20:392
2735694993112,12cyclictest2762043-21sshd08:36:032
273569399317,18cyclictest2868255-21ssh11:41:001
2735693993123,5cyclictest2892460-21munin-run12:20:011
273569299314,23cyclictest2772857-21ssh09:11:000
2735692993123,6cyclictest2754201-21fschecks_count08:10:530
2735692993121,7cyclictest2836831-21timerandwakeup10:51:120
2735694993024,3cyclictest0-21swapper/209:21:092
2735694993023,4cyclictest0-21swapper/211:36:092
2735693993024,3cyclictest0-21swapper/108:00:011
2735693993023,4cyclictest518-21rs:main1
2735693993022,5cyclictest2805276-21unixbench_multi10:01:181
2735693993020,7cyclictest2747001-21needreboot07:45:591
2735692993024,3cyclictest0-21swapper/007:50:010
2735692993022,6cyclictest70-21kswapd012:21:030
2735692993021,6cyclictest0-21swapper/010:00:000
273569499296,17cyclictest2842959-21ssh11:01:072
2735694992923,3cyclictest0-21swapper/212:10:512
2735694992922,4cyclictest1055-21mta-sts-daemon08:06:022
2735694992922,3cyclictest0-21swapper/209:05:012
2735694992919,7cyclictest695-21cron08:30:012
2735694992916,9cyclictest421-21dbus-daemon08:31:042
2735693992924,2cyclictest0-21swapper/107:30:011
2735693992920,4cyclictest0-21swapper/108:31:041
2735692992924,3cyclictest0-21swapper/011:50:020
2735692992922,4cyclictest2787555-21idleruntime-cro09:35:010
2735692992922,4cyclictest2787555-21idleruntime-cro09:35:010
2735692992922,4cyclictest2778753-21aten_r2power_vo09:20:540
2735692992912,13cyclictest2861937-21sh11:31:040
273569499283,19cyclictest1054-21mta-sts-daemon07:16:022
273569499282,19cyclictest2830130-21ssh10:41:082
273569499282,19cyclictest2830130-21ssh10:41:082
273569499282,19cyclictest2817500-21ssh10:21:052
2735694992820,5cyclictest2771702-21systemd-run09:10:012
2735693992822,4cyclictest2735691-21cyclictest08:10:561
2735693992821,4cyclictest2872001-21unixbench_multi11:46:131
2735693992820,5cyclictest2880565-21fschecks_time12:00:531
2735693992820,5cyclictest2742053-21aten_r2power_po07:30:511
273569299285,19cyclictest1952300-21cups-browsed08:46:060
2735692992823,2cyclictest0-21swapper/012:00:010
2735692992822,3cyclictest2740713-21fschecks_count07:25:540
2735692992822,3cyclictest1055-21mta-sts-daemon07:11:120
2735692992821,4cyclictest2770485-21aten_r2power_en09:05:510
2735692992821,4cyclictest2745757-21sendmail_mailqu07:41:080
2735692992821,4cyclictest2745757-21sendmail_mailqu07:41:080
2735692992820,4cyclictest0-21swapper/011:11:020
2735692992819,6cyclictest261-21systemd-journal07:21:030
273569399272,18cyclictest2776081-21ssh09:16:001
2735693992720,4cyclictest1054-21mta-sts-daemon12:28:181
2735693992719,5cyclictest2751625-21ntp_offset08:01:041
273569299272,20cyclictest2795240-21sh09:46:050
273569299272,19cyclictest2874745-21ssh11:51:060
2735692992719,6cyclictest2870209-21cron11:45:010
2735694992621,2cyclictest0-21swapper/207:40:002
273569399269,5cyclictest62450lldpd07:43:531
273569399269,5cyclictest62450lldpd07:43:531
2735693992618,5cyclictest2738544-21unixbench_multi07:16:121
273569299266,15cyclictest2738926-21kworker/u8:6+rpciod08:26:070
2735692992616,8cyclictest0-21swapper/008:20:010
273569499251,9cyclictest2736589-21sshd07:11:032
2735694992519,3cyclictest0-21swapper/208:46:062
2735694992517,6cyclictest1056-21mta-sts-daemon07:10:022
2735694992516,5cyclictest2751338-21kernelversion08:00:572
2735693992517,5cyclictest2753676-21munin-run08:10:011
2735693992517,5cyclictest2743693-21fschecks_count07:35:541
2735694992416,5cyclictest1055-21mta-sts-daemon08:11:012
2735694992415,6cyclictest1-21systemd07:35:012
2735693992418,4cyclictest1055-21mta-sts-daemon08:40:031
273569299242,14cyclictest659-21snmpd08:56:050
2735694992317,2cyclictest0-21swapper/207:55:012
2735693992317,3cyclictest0-21swapper/107:10:011
2735692992317,3cyclictest0-21swapper/008:00:010
2735693992217,3cyclictest1056-21mta-sts-daemon09:00:361
2735692992216,4cyclictest2738196-21proc_pri07:16:070
2735694992116,2cyclictest0-21swapper/207:20:562
2735692992114,5cyclictest2761715-21gltestperf08:35:580
273569599203,7cyclictest0-21swapper/310:41:083
273569599203,7cyclictest0-21swapper/310:41:083
273569599202,8cyclictest0-21swapper/312:06:073
273569599193,9cyclictest0-21swapper/312:21:033
2735692991914,2cyclictest0-21swapper/007:10:010
273569599183,9cyclictest0-21swapper/311:50:583
273569599182,9cyclictest0-21swapper/311:26:113
273569599174,8cyclictest0-21swapper/311:11:033
273569599173,8cyclictest0-21swapper/311:56:053
273569599173,8cyclictest0-21swapper/308:46:033
273569599173,8cyclictest0-21swapper/308:36:033
273569599173,7cyclictest0-21swapper/311:31:013
273569599173,7cyclictest0-21swapper/309:36:073
273569599172,9cyclictest0-21swapper/312:31:053
273569599172,9cyclictest0-21swapper/310:06:083
273569599172,9cyclictest0-21swapper/310:06:083
273569599164,7cyclictest0-21swapper/309:55:493
273569599164,5cyclictest0-21swapper/310:31:013
273569599163,8cyclictest0-21swapper/310:16:153
273569599163,7cyclictest0-21swapper/311:36:053
273569599163,7cyclictest0-21swapper/311:16:003
273569599162,9cyclictest0-21swapper/311:01:023
273569599162,9cyclictest0-21swapper/309:16:073
273569599162,8cyclictest0-21swapper/312:26:033
273569599162,8cyclictest0-21swapper/311:41:003
273569599162,8cyclictest0-21swapper/310:51:013
273569599162,8cyclictest0-21swapper/310:26:063
273569599162,8cyclictest0-21swapper/310:26:063
273569599162,8cyclictest0-21swapper/310:11:003
273569599162,8cyclictest0-21swapper/310:11:003
273569599162,7cyclictest0-21swapper/309:21:073
273569599154,7cyclictest0-21swapper/310:01:053
273569599154,5cyclictest0-21swapper/307:20:013
273569599153,7cyclictest0-21swapper/309:26:033
273569599153,5cyclictest0-21swapper/308:21:003
273569599152,8cyclictest0-21swapper/309:41:013
273569599152,8cyclictest0-21swapper/309:11:023
273569599152,7cyclictest0-21swapper/308:16:023
273569599152,7cyclictest0-21swapper/308:01:003
273569599151,9cyclictest0-21swapper/310:46:033
273569599151,9cyclictest0-21swapper/310:46:033
273569599151,8cyclictest0-21swapper/312:11:003
273569599151,8cyclictest0-21swapper/311:21:003
273569599151,8cyclictest0-21swapper/311:06:013
273569599151,7cyclictest0-21swapper/309:31:033
273569599151,7cyclictest0-21swapper/309:31:033
273569599144,6cyclictest0-21swapper/312:01:003
273569599143,6cyclictest0-21swapper/309:01:033
273569599143,5cyclictest0-21swapper/309:06:023
273569599143,5cyclictest0-21swapper/307:36:043
273569599142,8cyclictest0-21swapper/307:25:013
273569599142,7cyclictest0-21swapper/310:56:003
273569599142,7cyclictest0-21swapper/310:56:003
273569599142,7cyclictest0-21swapper/309:46:003
273569599142,7cyclictest0-21swapper/308:06:013
273569599142,7cyclictest0-21swapper/307:11:033
273569599142,6cyclictest0-21swapper/308:41:023
273569599142,6cyclictest0-21swapper/307:56:003
273569599141,8cyclictest0-21swapper/310:21:013
273569599133,5cyclictest0-21swapper/307:46:013
273569599132,8cyclictest0-21swapper/310:36:033
273569599132,7cyclictest0-21swapper/312:15:523
273569599132,6cyclictest0-21swapper/312:35:583
273569599132,6cyclictest0-21swapper/312:35:583
273569599132,6cyclictest0-21swapper/308:56:003
273569599132,5cyclictest0-21swapper/311:50:023
273569599132,5cyclictest0-21swapper/308:26:003
273569599131,7cyclictest0-21swapper/308:51:033
273569599131,7cyclictest0-21swapper/308:51:033
273569599131,7cyclictest0-21swapper/307:26:033
273569599131,6cyclictest0-21swapper/307:51:033
273569599124,6cyclictest0-21swapper/308:30:573
273569599123,5cyclictest0-21swapper/307:41:023
273569599123,5cyclictest0-21swapper/307:41:023
273569599123,4cyclictest0-21swapper/309:50:493
273569599122,6cyclictest0-21swapper/308:11:003
273569599102,5cyclictest0-21swapper/307:30:043
27356959991,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional