You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-01 - 04:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Tue Sep 01, 2026 00:42:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3707952996137,5cyclictest3762421-21sendmail21:40:012
3707952994939,4cyclictest0-21swapper/223:36:082
3707951994939,6cyclictest0-21swapper/123:56:041
3707950994835,10cyclictest0-21swapper/021:36:040
370795199461,28cyclictest3869310-21ssh00:36:081
3707950994626,12cyclictest12650irq/32-eth023:46:080
3707952994425,11cyclictest3863151-21sh00:26:062
3707951994435,6cyclictest3773377-21unixbench_singl21:56:181
3707950994437,4cyclictest3820424-21kworker/u8:2+rpciod00:40:010
3707952994333,6cyclictest3716968-21systemd-run19:40:012
3707952994332,7cyclictest420-21avahi-daemon19:50:022
3707952994233,6cyclictest3865783-21ssh00:30:582
3707952994233,6cyclictest3710978-21systemd-run19:20:012
3707952994233,6cyclictest3710978-21systemd-run19:20:012
3707951994233,6cyclictest421-21dbus-daemon23:40:001
370795099426,6cyclictest171rcu_preempt21:56:080
3707952994133,5cyclictest3777736-21latency_hist22:05:022
370795299413,30cyclictest3772743-21ssh21:56:052
370795299411,34cyclictest3827733-21ssh23:26:042
370795299411,34cyclictest3827733-21ssh23:26:042
3707951994133,5cyclictest3743943-21munin-run21:10:011
370795099412,9cyclictest3829598-21kworker/u8:3+rpciod23:36:080
3707952994033,4cyclictest3823466-21munin-run23:20:012
3707952994032,5cyclictest3844833-21ssh23:55:012
3707952994027,9cyclictest3832618-21latency_hist23:35:022
3707951994033,4cyclictest3731957-21systemd-run20:30:011
3707951994033,4cyclictest3731957-21systemd-run20:30:011
3707951994032,5cyclictest3793068-21latency_hist22:30:031
3707951994032,5cyclictest3750085-21sendmail21:20:011
3707951994029,8cyclictest3761033-21timerwakeupswit21:36:161
370795099405,28cyclictest3821656-21ssh23:16:070
3707952993929,6cyclictest3787682-21grep22:20:592
3707952993928,8cyclictest3751585-21mailstats21:21:142
3707951993930,5cyclictest3721456-21systemd-run19:55:011
3707951993928,7cyclictest421-21dbus-daemon22:11:181
3707951993928,7cyclictest420-21avahi-daemon22:51:131
3707950993934,2cyclictest0-21swapper/000:26:130
3707950993929,7cyclictest3722930-21munin-plugin-st20:00:010
3707950993929,6cyclictest3790723-21fschecks_count22:25:570
3707950993928,8cyclictest3785497-21timerwakeupswit22:16:160
3707952993830,5cyclictest3849192-21sh00:01:072
3707952993830,5cyclictest3838857-21latency_hist23:45:022
3707952993830,5cyclictest3811235-21grep23:00:012
3707952993830,5cyclictest3795990-21ssh22:35:012
3707952993824,11cyclictest3857454-21timerandwakeup00:16:152
370795299382,25cyclictest3784968-21ssh22:16:042
3707951993832,3cyclictest0-21swapper/123:26:161
3707951993832,3cyclictest0-21swapper/123:26:161
3707950993823,12cyclictest3766930-21mailstats21:46:130
370795099382,28cyclictest3751416-21ssh21:21:090
3707950993820,13cyclictest3827684-21sh23:26:030
3707950993820,13cyclictest3827684-21sh23:26:030
3707950993817,15cyclictest12650irq/32-eth022:01:030
3707952993731,4cyclictest3707949-21cyclictest23:46:112
3707952993730,4cyclictest3826539-21munin-run23:25:012
3707952993730,4cyclictest3826539-21munin-run23:25:012
3707952993728,6cyclictest1-21systemd00:11:092
3707952993728,6cyclictest1-21systemd00:11:092
3707952993728,5cyclictest3743947-21diskmemload21:10:012
3707951993730,4cyclictest3786903-21latency_hist22:20:021
3707951993730,4cyclictest1000-21mta-sts-daemon22:44:011
3707951993729,5cyclictest3818268-21if_wlan023:10:591
3707951993729,5cyclictest3739427-21cron20:55:011
3707951993727,6cyclictest3816079-21unixbench_multi23:06:181
3707951993715,16cyclictest421-21dbus-daemon22:36:081
3707950993728,6cyclictest3866087-21missed_timers00:31:020
3707950993727,7cyclictest3771647-21latency_hist21:55:020
3707950993718,14cyclictest151ktimers/020:41:080
3707952993629,4cyclictest3814340-21pluginstate23:05:022
3707951993629,4cyclictest3839478-21cpuspeed23:45:551
3707951993629,4cyclictest1003-21mta-sts-daemon21:31:051
3707951993628,5cyclictest3696844-21kworker/u8:12+rpciod19:15:011
3707951993628,5cyclictest3696844-21kworker/u8:12+rpciod19:15:011
3707951993627,5cyclictest3727509-21latency_hist20:15:021
3707951993627,4cyclictest0-21swapper/121:51:061
3707951993626,6cyclictest3733453-21true20:35:011
3707950993630,4cyclictest3847894-21munin-run00:00:010
3707950993629,4cyclictest3743947-21diskmemload00:05:020
3707950993629,4cyclictest3716924-21cron19:40:010
3707950993629,3cyclictest0-21swapper/023:06:110
3707950993626,7cyclictest3853295-21aten_r2power_cu00:10:500
3707950993626,7cyclictest3853295-21aten_r2power_cu00:10:500
370795099362,28cyclictest3763650-21ssh21:41:050
370795099362,28cyclictest3763650-21ssh21:41:050
3707950993621,12cyclictest0-21swapper/023:50:570
3707952993527,5cyclictest3740799-21unixbench_multi20:56:172
3707952993526,6cyclictest3816104-21unixbench_multi23:06:182
3707952993524,7cyclictest3763527-21needreboot21:41:012
3707952993524,7cyclictest3763527-21needreboot21:41:012
3707952993522,9cyclictest3800047-21latency22:41:042
3707951993528,4cyclictest1000-21mta-sts-daemon23:04:371
3707951993528,3cyclictest0-21swapper/120:00:011
3707951993527,5cyclictest3776105-21sendmail_mailtr22:01:111
3707951993526,5cyclictest0-21swapper/121:46:121
370795099355,6cyclictest171rcu_preempt21:26:060
3707950993530,3cyclictest3802088-21sh22:45:010
370795099352,24cyclictest3837078-21sh23:41:110
3707950993510,16cyclictest12650irq/32-eth021:16:110
3707952993427,3cyclictest0-21swapper/221:26:002
370795199347,14cyclictest1001-21mta-sts-daemon20:01:111
370795199343,25cyclictest3836890-21ssh23:41:041
3707951993428,4cyclictest3707949-21cyclictest22:24:271
3707951993428,4cyclictest1000-21mta-sts-daemon19:26:091
3707951993427,4cyclictest1002-21mta-sts-daemon00:18:291
3707951993427,3cyclictest0-21swapper/119:36:041
3707951993426,5cyclictest3753375-21latency_hist21:25:031
3707951993425,6cyclictest1874569-21cups-browsed22:59:271
3707951993425,4cyclictest0-21swapper/122:06:071
3707951993420,10cyclictest421-21dbus-daemon22:32:041
3707951993410,8cyclictest62750lldpd00:28:321
3707950993427,4cyclictest3805157-21systemd-run22:50:020
370795099342,26cyclictest665-21snmpd22:06:060
370795099341,27cyclictest3824655-21ssh23:21:030
370795099341,27cyclictest3824655-21ssh23:21:030
3707952993325,5cyclictest3778334-21aten_r2power_po22:05:562
3707952993325,5cyclictest3747005-21cut21:15:012
3707952993325,4cyclictest1003-21mta-sts-daemon21:17:492
3707952993322,8cyclictest1-21systemd20:50:582
370795299332,25cyclictest3766653-21ssh21:46:042
3707951993326,5cyclictest3707949-21cyclictest21:14:581
3707951993325,5cyclictest420-21avahi-daemon23:19:161
3707951993325,5cyclictest3710978-21systemd-run19:20:011
3707951993325,5cyclictest3710978-21systemd-run19:20:011
3707951993325,4cyclictest1000-21mta-sts-daemon00:21:091
370795099333,27cyclictest0-21swapper/019:26:060
3707950993327,3cyclictest0-21swapper/000:21:090
3707950993325,5cyclictest3757962-21timerwakeupswit21:31:180
3707950993322,9cyclictest3852784-21ssh00:09:590
3707952993226,3cyclictest0-21swapper/223:15:012
3707952993225,4cyclictest3782762-21ssh22:11:502
3707952993224,5cyclictest439-21systemd-logind20:05:012
3707952993224,5cyclictest3847884-21cron00:00:012
3707952993223,6cyclictest1874569-21cups-browsed19:21:122
3707952993222,7cyclictest3769641-21needreboot21:51:022
3707952993222,7cyclictest3726747-21needreboot20:11:032
3707951993225,5cyclictest28-21ksoftirqd/120:56:111
3707951993225,4cyclictest3830632-21missed_timers23:31:021
3707951993225,3cyclictest0-21swapper/122:46:051
3707951993224,5cyclictest3727951-21fschecks_count20:15:581
3707951993223,4cyclictest0-21swapper/123:25:031
3707951993223,4cyclictest0-21swapper/123:25:031
3707951993222,7cyclictest1001-21mta-sts-daemon21:25:201
3707951993219,9cyclictest421-21dbus-daemon21:41:011
3707951993219,9cyclictest421-21dbus-daemon21:41:011
3707950993226,3cyclictest0-21swapper/019:31:080
3707950993222,7cyclictest3793964-21latency22:31:020
3707950993221,4cyclictest0-21swapper/022:36:130
3707952993124,4cyclictest3758099-21unixbench_singl21:31:192
3707952993123,5cyclictest3717469-21fschecks_count19:40:582
3707952993119,9cyclictest3859598-21cpu00:20:572
370795299311,24cyclictest3806434-21ssh22:51:092
3707951993125,3cyclictest0-21swapper/123:51:161
3707950993126,2cyclictest0-21swapper/023:00:010
3707950993124,5cyclictest3808180-21cron22:55:010
3707950993124,3cyclictest0-21swapper/019:11:080
3707950993124,3cyclictest0-21swapper/019:11:080
3707952993023,4cyclictest3725942-21munin-run20:10:012
3707952993022,5cyclictest3869808-21grep00:36:182
3707952993022,5cyclictest265-21systemd-journal22:35:032
3707952993021,6cyclictest0-21swapper/200:10:022
3707952993016,11cyclictest351ktimers/219:51:052
3707951993023,4cyclictest3712442-21cron19:25:001
3707951993022,6cyclictest0-21swapper/100:15:021
3707951993022,6cyclictest0-21swapper/100:15:021
3707951993020,7cyclictest1003-21mta-sts-daemon00:34:241
370795099309,16cyclictest665-21snmpd00:16:310
3707950993025,2cyclictest0-21swapper/020:56:030
3707950993024,3cyclictest0-21swapper/021:11:110
3707950993023,4cyclictest1002-21mta-sts-daemon23:30:230
3707950993020,7cyclictest3812212-21kernelversion23:00:590
3707950993020,7cyclictest3725318-21ntp_kernel_pll_20:06:050
370795099301,23cyclictest3788192-21ssh22:21:110
3707952992921,5cyclictest3722961-21sendmail-msp20:00:012
3707951992922,4cyclictest3730455-21munin-plugin-st20:25:011
3707951992922,4cyclictest0-21swapper/120:45:021
3707951992920,4cyclictest0-21swapper/100:00:561
3707950992924,3cyclictest0-21swapper/020:20:010
3707950992924,2cyclictest0-21swapper/020:15:010
3707950992923,3cyclictest0-21swapper/023:11:010
370795099292,21cyclictest3781928-21ssh22:11:050
3707952992821,5cyclictest3707949-21cyclictest19:10:012
3707952992821,5cyclictest3707949-21cyclictest19:10:012
3707952992820,5cyclictest3692484-21kworker/u8:1+xprtiod19:15:012
3707952992820,5cyclictest3692484-21kworker/u8:1+xprtiod19:15:012
3707952992819,6cyclictest3805150-21systemd-run22:50:012
3707952992819,5cyclictest3712978-21fschecks_time19:25:592
3707952992818,8cyclictest3714535-21if_eth019:30:582
3707951992822,3cyclictest0-21swapper/119:30:571
3707951992820,5cyclictest3718443-21sh19:45:011
3707951992820,4cyclictest3742159-21threads21:01:161
3707951992819,6cyclictest420-21avahi-daemon20:47:581
3707950992721,4cyclictest3718386-21unixbench_singl19:41:160
3707950992714,10cyclictest0-21swapper/020:01:200
370795299263,5cyclictest171rcu_preempt21:01:052
3707952992618,5cyclictest3743947-21diskmemload22:30:022
3707952992618,5cyclictest3728460-21proc_pri20:16:112
3707952992617,6cyclictest1874569-21cups-browsed20:30:112
370795099261,13cyclictest3719867-21grep19:46:160
3707952992517,5cyclictest0-21swapper/220:40:002
3707952992516,6cyclictest3736826-21basename20:45:562
3707951992517,5cyclictest439-21systemd-logind19:10:021
3707951992517,5cyclictest439-21systemd-logind19:10:021
370795099254,18cyclictest420-21avahi-daemon19:17:430
370795099254,18cyclictest420-21avahi-daemon19:17:430
3707950992520,3cyclictest151ktimers/019:21:030
3707950992519,4cyclictest3731936-21cron20:30:000
3707950992519,4cyclictest3731936-21cron20:30:000
3707950992519,3cyclictest3733456-21kworker/u8:9-xprtiod20:36:160
3707950992518,4cyclictest3721363-21head19:51:170
3707951992418,3cyclictest0-21swapper/120:35:581
3707951992412,7cyclictest3852978-21cat00:10:021
3707950992419,2cyclictest0-21swapper/020:25:010
3707950992411,6cyclictest171rcu_preempt21:06:080
3707950992410,11cyclictest567-21gdbus20:30:560
3707952992315,5cyclictest3736444-21cron20:45:012
3707951992318,2cyclictest0-21swapper/119:50:011
3707951992316,4cyclictest1001-21mta-sts-daemon20:06:141
370795099232,15cyclictest3737348-21ntpq20:46:060
3707950992319,2cyclictest0-21swapper/021:05:020
3707952992217,3cyclictest1002-21mta-sts-daemon20:26:062
3707952992217,3cyclictest1002-21mta-sts-daemon20:26:062
3707952992215,5cyclictest0-21swapper/220:21:062
370795099221,17cyclictest3738882-21ntpq20:51:070
3707950991915,2cyclictest0-21swapper/019:10:020
3707950991915,2cyclictest0-21swapper/019:10:020
370795499173,7cyclictest0-21swapper/322:41:103
370795499172,7cyclictest0-21swapper/321:21:113
370795499172,6cyclictest0-21swapper/321:41:043
370795499172,6cyclictest0-21swapper/321:41:043
370795499171,7cyclictest0-21swapper/323:16:073
370795499162,7cyclictest0-21swapper/323:56:073
370795499162,6cyclictest0-21swapper/323:21:033
370795499162,6cyclictest0-21swapper/323:21:033
370795499153,6cyclictest0-21swapper/322:01:053
370795499152,7cyclictest0-21swapper/322:36:123
370795499152,7cyclictest0-21swapper/322:06:063
370795499152,7cyclictest0-21swapper/321:56:103
370795499152,6cyclictest0-21swapper/319:31:043
370795499152,4cyclictest0-21swapper/320:01:113
370795499151,7cyclictest0-21swapper/300:01:043
370795499151,6cyclictest0-21swapper/323:41:033
370795499151,6cyclictest0-21swapper/319:46:033
370795499143,5cyclictest0-21swapper/319:16:083
370795499143,5cyclictest0-21swapper/319:16:083
370795499142,7cyclictest0-21swapper/323:45:573
370795499142,7cyclictest0-21swapper/321:46:033
370795499142,6cyclictest0-21swapper/322:21:063
370795499142,6cyclictest0-21swapper/321:51:033
370795499142,6cyclictest0-21swapper/320:55:023
370795499142,5cyclictest0-21swapper/320:31:113
370795499142,4cyclictest0-21swapper/319:36:053
370795499141,7cyclictest0-21swapper/321:26:083
370795499141,7cyclictest0-21swapper/321:06:083
370795499141,7cyclictest0-21swapper/300:16:073
370795499141,6cyclictest0-21swapper/322:26:043
370795499141,6cyclictest0-21swapper/300:21:073
370795499133,7cyclictest0-21swapper/321:10:523
370795499132,7cyclictest0-21swapper/320:25:013
370795499132,7cyclictest0-21swapper/300:30:563
370795499132,6cyclictest0-21swapper/323:11:073
370795499132,6cyclictest0-21swapper/322:16:003
370795499132,6cyclictest0-21swapper/322:11:053
370795499132,6cyclictest0-21swapper/321:30:583
370795499132,5cyclictest0-21swapper/319:21:063
370795499131,7cyclictest0-21swapper/320:36:113
370795499131,7cyclictest0-21swapper/320:16:113
370795499131,5cyclictest0-21swapper/320:06:063
370795499125,5cyclictest0-21swapper/300:26:133
370795499123,5cyclictest0-21swapper/323:26:093
370795499123,5cyclictest0-21swapper/323:26:093
370795499122,6cyclictest0-21swapper/323:31:033
370795499122,6cyclictest0-21swapper/320:25:573
370795499122,6cyclictest0-21swapper/320:25:573
370795499121,7cyclictest0-21swapper/322:46:083
370795499121,7cyclictest0-21swapper/321:16:063
370795499121,6cyclictest0-21swapper/323:05:593
370795499121,6cyclictest0-21swapper/300:36:023
370795499121,5cyclictest0-21swapper/322:35:033
370795499121,5cyclictest0-21swapper/319:56:043
370795499114,5cyclictest0-21swapper/323:01:053
370795499113,4cyclictest0-21swapper/321:01:103
370795499112,6cyclictest0-21swapper/321:36:003
370795499112,5cyclictest0-21swapper/322:55:573
370795499112,5cyclictest0-21swapper/320:15:013
370795499112,5cyclictest0-21swapper/300:11:063
370795499112,5cyclictest0-21swapper/300:11:063
370795499112,4cyclictest0-21swapper/319:51:043
370795499112,4cyclictest0-21swapper/300:06:293
370795499111,6cyclictest0-21swapper/323:36:043
370795499111,6cyclictest0-21swapper/320:56:103
370795499111,6cyclictest0-21swapper/320:41:043
370795499102,5cyclictest0-21swapper/322:50:593
370795499101,5cyclictest0-21swapper/323:50:543
370795499101,5cyclictest0-21swapper/319:25:583
370795499101,5cyclictest0-21swapper/319:11:033
370795499101,5cyclictest0-21swapper/319:11:033
37079549991,5cyclictest0-21swapper/320:46:053
37079549991,5cyclictest0-21swapper/319:40:583
37079549981,4cyclictest0-21swapper/319:10:033
37079549981,4cyclictest0-21swapper/319:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional