You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-23 - 10:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Sun Aug 23, 2026 00:42:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2892620996254,5cyclictest2972201-21aten_r2power_vo22:20:520
2892620995426,18cyclictest12650irq/32-eth022:56:010
2892622995342,6cyclictest3017437-21systemd23:35:022
2892621995227,21cyclictest2989907-21ssh22:50:001
2892621995227,21cyclictest2989907-21ssh22:50:001
2892621995125,23cyclictest2947073-21idleruntime-cro21:40:011
2892621995125,23cyclictest2947073-21idleruntime-cro21:40:011
2892620995037,8cyclictest1121621-21cups-browsed23:26:000
289262299478,23cyclictest2979842-21ssh22:32:202
2892621994738,4cyclictest0-21swapper/122:20:581
2892620994632,5cyclictest3032680-21cron00:00:010
2892620994617,19cyclictest421-21dbus-daemon21:45:590
289262299452,33cyclictest3052276-21ssh00:31:032
289262199459,3cyclictest0-21swapper/119:16:021
289262199459,3cyclictest0-21swapper/119:16:021
2892621994534,7cyclictest0-21swapper/121:10:011
2892620994538,5cyclictest421-21dbus-daemon19:15:010
2892620994538,5cyclictest421-21dbus-daemon19:15:010
2892622994435,5cyclictest0-21swapper/221:50:502
2892621994435,6cyclictest3052755-21sed00:31:131
289262199442,7cyclictest2908536-21ntpq20:01:021
2892621994422,18cyclictest0-21swapper/123:55:021
2892622994334,6cyclictest2953199-21true21:50:012
2892621994335,3cyclictest0-21swapper/100:00:021
2892621994332,5cyclictest0-21swapper/121:25:521
289262199432,6cyclictest2902489-21ntpq19:40:591
2892621994324,15cyclictest2901633-21systemd-run19:40:011
289262199431,13cyclictest2911652-21ps20:11:061
2892622994234,5cyclictest2973276-21sed22:21:152
2892622994232,6cyclictest2970271-21unixbench_singl22:16:172
289262199428,3cyclictest0-21swapper/100:21:011
289262199422,6cyclictest2906983-21ntpq19:56:001
2892621994223,16cyclictest2966318-21if_wlan022:10:551
2892620994228,12cyclictest0-21swapper/021:31:090
289262199413,8cyclictest2893480-21ntpq19:11:011
289262199413,8cyclictest2893480-21ntpq19:11:011
2892621994133,5cyclictest2927107-21grep21:05:011
2892621994132,4cyclictest0-21swapper/123:05:591
2892621994131,6cyclictest2951153-21missed_timers21:45:561
2892621994124,14cyclictest3044951-21munin-run00:20:011
289262199412,17cyclictest2929858-21ssh21:11:041
2892621994111,3cyclictest0-21swapper/123:00:011
289262199411,12cyclictest2917459-21ntpq20:30:581
289262199411,10cyclictest2922053-21ntpq20:46:051
2892620994134,4cyclictest3035825-21sh00:05:010
2892622994033,4cyclictest3010077-21grep23:21:142
2892622994032,5cyclictest2963252-21if_eth022:05:562
2892622994032,5cyclictest1-21systemd23:45:022
289262199402,11cyclictest3040042-21ssh00:11:021
2892621994020,16cyclictest3012103-21fschecks_count23:25:531
2892621994013,24cyclictest0-21swapper/122:16:141
289262199401,14cyclictest2996659-21aten2.4-expect23:00:501
289262199401,11cyclictest665-21snmpd23:40:591
289262199401,11cyclictest2909984-21ntpq20:06:011
289262199401,11cyclictest2909984-21ntpq20:06:011
289262199401,10cyclictest2896478-21ntpq19:20:581
289262199400,12cyclictest2949620-21ssh21:43:261
2892620994032,6cyclictest3008254-21sendmail23:20:010
2892622993931,5cyclictest3057158-21munin-run00:40:012
2892622993930,6cyclictest2996015-21cron23:00:002
2892622993930,4cyclictest0-21swapper/200:16:042
2892622993929,6cyclictest2975944-21ps22:26:092
2892622993926,10cyclictest2932731-21missed_timers21:15:572
2892621993927,8cyclictest0-21swapper/122:55:021
2892621993927,8cyclictest0-21swapper/122:55:021
289262199392,6cyclictest3015657-21ntpq23:31:051
289262199391,9cyclictest3033989-21ntpq00:01:061
289262199391,9cyclictest2932878-21ssh21:16:011
289262199391,9cyclictest2923508-21ntpq20:51:031
2892621993915,20cyclictest2890084-21kworker/u8:4-rpciod19:32:351
2892620993930,6cyclictest1001-21mta-sts-daemon22:42:030
2892620993930,6cyclictest1001-21mta-sts-daemon22:42:030
2892622993830,5cyclictest3032744-21sh00:00:022
2892622993830,5cyclictest2939212-21ps21:26:062
2892622993828,6cyclictest1-21systemd23:40:012
289262199383,12cyclictest2960464-21ntpq22:00:591
289262199382,6cyclictest3055377-21ntpq00:36:051
289262199382,15cyclictest3018750-21ps23:36:051
289262199381,9cyclictest2912941-21ntpq20:15:581
289262199381,6cyclictest2957424-21ssh21:56:001
289262199381,16cyclictest2963398-21memory22:05:571
289262199381,14cyclictest2935946-21ntpq21:21:011
289262199381,12cyclictest2919028-21ntpq20:36:031
2892621993811,23cyclictest3002468-21sh23:10:201
289262199381,11cyclictest2985046-21ntpq22:41:051
289262199381,11cyclictest2985046-21ntpq22:41:051
289262199381,10cyclictest2928599-21diskmemload22:31:221
2892620993828,8cyclictest2923409-21needreboot20:50:590
2892620993828,5cyclictest0-21swapper/022:16:000
2892620993826,8cyclictest420-21avahi-daemon00:15:590
2892620993825,7cyclictest12650irq/32-eth023:11:000
2892620993824,12cyclictest2960298-21latency22:00:550
2892620993823,11cyclictest0-21swapper/022:51:140
2892620993823,11cyclictest0-21swapper/022:51:140
2892622993729,5cyclictest3036524-21sort00:05:522
2892622993729,5cyclictest2929130-21aten_r2power_cu21:10:502
289262199373,7cyclictest2898008-21ntpq19:26:001
289262199372,15cyclictest2942187-21ssh21:31:071
289262199371,9cyclictest3038811-21ssh00:09:551
289262199371,6cyclictest2955002-21users21:51:131
289262199371,11cyclictest3006391-21ntpq23:16:001
289262199371,10cyclictest3048595-21/usr/sbin/munin00:25:501
2892620993730,4cyclictest2982567-21vmstat22:36:140
289262099372,30cyclictest3009555-21ssh23:21:050
2892622993629,5cyclictest2358529-21kworker/u8:0+xprtiod19:25:002
2892622993628,5cyclictest3024306-21ssh23:45:532
2892622993628,5cyclictest2983780-21ssh22:40:012
2892622993628,5cyclictest2950121-21cron21:45:002
2892622993627,6cyclictest3013052-21unixbench_multi23:26:112
289262199363,8cyclictest3024669-21ssh23:45:581
289262199363,7cyclictest3009417-21ntpq23:21:001
289262199362,6cyclictest2915969-21ntpq20:25:591
289262199362,6cyclictest2914521-21ntpq20:21:011
289262199361,9cyclictest2925055-21ntpq20:56:051
289262199361,14cyclictest2981545-21fw_packets22:35:541
289262199361,11cyclictest2905100-21diskstats19:50:531
2892620993629,5cyclictest0-21swapper/020:10:010
2892620993629,5cyclictest0-21swapper/020:10:010
2892620993629,4cyclictest2895607-21munin-plugin-st19:20:000
2892620993629,4cyclictest2895607-21munin-plugin-st19:20:000
289262099362,10cyclictest14-21ksoftirqd/022:26:010
289262099361,30cyclictest2963549-21ssh22:06:010
2892622993523,9cyclictest2988289-21mailstats22:46:072
2892622993523,9cyclictest2988289-21mailstats22:46:072
289262199355,6cyclictest567-21gdbus20:40:491
289262199351,12cyclictest2904026-21ntpq19:46:021
289262199350,11cyclictest2975352-21diskstats22:25:541
2892620993529,4cyclictest3026784-21ls23:50:030
2892620993528,5cyclictest1121621-21cups-browsed22:46:110
2892620993528,5cyclictest1121621-21cups-browsed22:46:110
2892620993527,5cyclictest3017874-21/usr/sbin/munin23:36:110
2892620993521,11cyclictest421-21dbus-daemon22:35:020
289262099351,25cyclictest2997212-21diskmemload23:01:000
289262299343,23cyclictest3027737-21ssh23:50:592
2892622993428,3cyclictest0-21swapper/221:40:012
2892622993428,3cyclictest0-21swapper/221:40:012
2892622993425,6cyclictest421-21dbus-daemon00:01:182
289262299341,27cyclictest2985020-21rm22:41:042
289262299341,27cyclictest2985020-21rm22:41:042
2892622993411,7cyclictest62750lldpd23:09:162
2892621993426,5cyclictest2892642-21latency_hist19:10:011
289262099348,7cyclictest171rcu_preempt22:11:000
289262099345,22cyclictest428-21mta-sts-daemon21:11:000
2892620993428,3cyclictest3048014-21cron00:25:010
2892620993428,3cyclictest1003-21mta-sts-daemon00:36:120
2892620993426,6cyclictest421-21dbus-daemon00:30:410
289262099342,28cyclictest3015575-21rm23:31:020
2892620993422,10cyclictest2919726-21latency_hist20:40:020
289262299336,19cyclictest3046276-21ssh00:21:072
2892622993325,5cyclictest421-21dbus-daemon22:15:012
2892622993318,13cyclictest351ktimers/220:21:012
2892620993327,4cyclictest3023530-21grep23:45:010
2892620993327,3cyclictest0-21swapper/021:50:490
2892620993326,4cyclictest2957807-21timerandwakeup21:56:100
2892620993323,7cyclictest2945564-21threads21:36:130
2892620993323,7cyclictest2945564-21threads21:36:130
2892620993322,8cyclictest2921111-21/usr/sbin/munin20:45:020
2892622993225,4cyclictest1000-21mta-sts-daemon21:20:452
2892622993225,3cyclictest0-21swapper/223:20:012
2892622993224,5cyclictest2911999-21unixbench_multi20:11:112
2892622993224,5cyclictest2907504-21grep19:56:112
2892622993223,7cyclictest2921919-21needreboot20:46:002
2892620993227,2cyclictest0-21swapper/000:11:020
2892620993225,4cyclictest2905115-21fschecks_count19:50:530
2892620993222,7cyclictest2922007-21ntp_kernel_pll_20:46:030
2892620993222,4cyclictest1002-21mta-sts-daemon23:06:130
289262099321,25cyclictest3049285-21ssh00:26:070
2892622993125,3cyclictest0-21swapper/221:05:012
2892622993124,3cyclictest1003-21mta-sts-daemon00:12:432
289262299312,22cyclictest2960568-21ssh22:01:032
2892622993121,3cyclictest0-21swapper/223:12:472
2892622993121,3cyclictest0-21swapper/223:01:112
2892620993126,3cyclictest2934759-21munin-plugin-st21:20:010
2892620993125,4cyclictest3038962-21latency_hist00:10:020
2892622993023,4cyclictest1002-21mta-sts-daemon19:50:512
2892622993022,5cyclictest694-21cron20:55:012
2892622993022,4cyclictest0-21swapper/222:51:082
2892622993022,4cyclictest0-21swapper/222:51:082
289262299302,21cyclictest2942147-21ssh21:31:062
2892622993018,8cyclictest2957135-21sh21:55:542
2892620993024,4cyclictest2904605-21munin-plugin-st19:50:010
2892620993024,4cyclictest2902859-21timerandwakeup19:41:100
2892620993023,4cyclictest2938639-21fschecks_count21:25:520
289262299293,20cyclictest958-21sendmail-mta20:36:052
2892622992923,3cyclictest0-21swapper/219:40:012
2892622992922,4cyclictest0-21swapper/219:20:002
2892622992922,4cyclictest0-21swapper/219:20:002
2892622992922,3cyclictest0-21swapper/219:26:112
289262099299,15cyclictest151ktimers/019:21:050
289262099295,6cyclictest171rcu_preempt19:55:580
289262099292,20cyclictest2928581-21kworker/u8:5+rpciod21:41:020
2892620992920,4cyclictest1003-21mta-sts-daemon20:21:000
2892620992918,8cyclictest2901600-21grep19:40:010
2892622992822,3cyclictest0-21swapper/200:26:162
2892622992820,5cyclictest2909122-21idleruntime-cro20:05:012
2892622992820,5cyclictest2900052-21unixbench_singl19:31:142
2892620992823,2cyclictest0-21swapper/020:01:010
2892620992821,4cyclictest0-21swapper/021:10:010
2892620992821,3cyclictest0-21swapper/021:21:460
2892620992816,9cyclictest2927011-21unixbench_multi21:01:130
289262299272,19cyclictest1121621-21cups-browsed19:11:062
289262299272,19cyclictest1121621-21cups-browsed19:11:062
2892622992720,3cyclictest2925599-21basename21:00:012
2892620992717,7cyclictest2912625-21fschecks_time20:15:530
2892622992620,3cyclictest0-21swapper/220:30:012
2892620992620,4cyclictest1001-21mta-sts-daemon23:51:000
2892620992615,8cyclictest0-21swapper/020:56:120
289262099261,21cyclictest2917657-21ps20:31:060
2892622992519,3cyclictest1002-21mta-sts-daemon20:15:592
2892622992517,5cyclictest0-21swapper/221:10:012
2892620992519,4cyclictest2890084-21kworker/u8:4+xprtiod19:31:140
2892622992417,5cyclictest0-21swapper/220:31:112
2892622992414,7cyclictest2903410-21aten_r2power_en19:45:502
289262099248,10cyclictest12650irq/32-eth020:15:010
2892622992317,3cyclictest2907570-21kworker/u8:8-xprtiod20:10:002
2892622992317,3cyclictest2907570-21kworker/u8:8-xprtiod20:10:002
2892622992316,4cyclictest2921105-21cron20:45:012
2892622992315,5cyclictest2903101-21cron19:45:002
2892620992311,7cyclictest151ktimers/019:25:580
289262099221,8cyclictest2915636-21fschecks_count20:25:540
289262399203,8cyclictest0-21swapper/320:01:043
289262399202,8cyclictest0-21swapper/322:01:023
2892620992015,3cyclictest2892634-21systemd19:10:010
289262399194,8cyclictest0-21swapper/300:01:083
289262399194,6cyclictest0-21swapper/322:36:013
289262399182,8cyclictest0-21swapper/323:51:053
289262399181,8cyclictest0-21swapper/321:36:023
289262399181,8cyclictest0-21swapper/321:36:023
289262399173,8cyclictest0-21swapper/321:01:013
289262399173,8cyclictest0-21swapper/300:06:043
289262399173,7cyclictest0-21swapper/323:36:013
289262399173,6cyclictest0-21swapper/323:10:583
289262399172,9cyclictest0-21swapper/323:06:023
289262399172,8cyclictest0-21swapper/321:31:113
289262399171,8cyclictest0-21swapper/300:16:023
2892622991713,2cyclictest0-21swapper/219:10:012
289262399163,8cyclictest0-21swapper/323:21:003
289262399163,8cyclictest0-21swapper/322:51:073
289262399163,8cyclictest0-21swapper/322:51:073
289262399163,7cyclictest0-21swapper/300:31:023
289262399163,6cyclictest0-21swapper/321:16:003
289262399162,8cyclictest0-21swapper/323:25:593
289262399162,8cyclictest0-21swapper/322:45:593
289262399162,8cyclictest0-21swapper/322:45:593
289262399162,8cyclictest0-21swapper/321:51:003
289262399162,8cyclictest0-21swapper/321:26:033
289262399162,8cyclictest0-21swapper/300:11:043
289262399162,7cyclictest0-21swapper/322:21:033
289262399161,9cyclictest0-21swapper/323:41:023
289262399161,9cyclictest0-21swapper/323:01:013
289262399161,8cyclictest0-21swapper/300:36:043
289262399161,6cyclictest0-21swapper/320:31:033
289262399153,7cyclictest0-21swapper/322:41:023
289262399153,7cyclictest0-21swapper/322:41:023
289262399153,7cyclictest0-21swapper/322:26:013
289262399153,7cyclictest0-21swapper/322:16:083
289262399153,7cyclictest0-21swapper/320:15:013
289262399153,6cyclictest0-21swapper/321:55:583
289262399152,8cyclictest0-21swapper/323:55:533
289262399152,8cyclictest0-21swapper/323:46:043
289262399152,8cyclictest0-21swapper/322:06:033
289262399152,7cyclictest0-21swapper/323:30:593
289262399152,6cyclictest0-21swapper/322:10:513
289262399151,9cyclictest0-21swapper/321:40:533
289262399151,8cyclictest0-21swapper/321:46:033
289262399151,6cyclictest0-21swapper/319:46:033
289262399144,4cyclictest0-21swapper/320:06:033
289262399144,4cyclictest0-21swapper/320:06:033
289262399143,6cyclictest0-21swapper/320:15:533
289262399143,5cyclictest0-21swapper/319:31:053
289262399143,5cyclictest0-21swapper/300:25:493
289262399143,4cyclictest0-21swapper/319:51:023
289262399142,8cyclictest0-21swapper/321:10:193
289262399141,8cyclictest0-21swapper/323:15:513
289262399141,7cyclictest0-21swapper/320:46:053
289262399132,7cyclictest0-21swapper/322:56:013
289262399132,7cyclictest0-21swapper/321:20:513
289262399131,7cyclictest0-21swapper/322:30:543
289262399131,7cyclictest0-21swapper/320:25:593
289262399131,6cyclictest0-21swapper/321:05:593
289262399131,6cyclictest0-21swapper/320:56:023
289262399123,4cyclictest0-21swapper/319:11:003
289262399123,4cyclictest0-21swapper/319:11:003
289262399121,8cyclictest0-21swapper/300:20:523
289262399121,7cyclictest0-21swapper/320:21:023
289262399121,7cyclictest0-21swapper/319:21:003
289262399112,6cyclictest0-21swapper/320:36:043
289262399112,5cyclictest0-21swapper/319:15:533
289262399112,5cyclictest0-21swapper/319:15:533
289262399111,5cyclictest0-21swapper/320:51:003
289262399111,5cyclictest0-21swapper/320:41:003
289262399110,6cyclictest0-21swapper/319:26:023
289262399101,5cyclictest0-21swapper/319:35:553
28926239991,6cyclictest0-21swapper/319:55:573
28926239991,5cyclictest0-21swapper/319:40:583
28926239982,5cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional