You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 08:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Dec 01, 2025 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
684437997413,16cyclictest131rcu_preempt23:45:020
68444799676,14cyclictest37-21ksoftirqd/322:35:023
684443996532,22cyclictest699194-21kworker/2:000:05:012
684447996427,19cyclictest37-21ksoftirqd/319:15:013
684447996427,19cyclictest37-21ksoftirqd/319:15:003
684437996434,24cyclictest12-21ksoftirqd/000:00:020
684447996314,10cyclictest37-21ksoftirqd/320:40:023
684447996217,10cyclictest37-21ksoftirqd/323:35:023
684447996217,10cyclictest37-21ksoftirqd/323:35:023
684443996153,4cyclictest30-21ksoftirqd/222:00:022
68444799608,10cyclictest37-21ksoftirqd/300:00:003
684447995840,12cyclictest701586-21systemctl00:40:003
684447995811,14cyclictest37-21ksoftirqd/321:45:023
68444799577,6cyclictest37-21ksoftirqd/320:05:013
684437995737,14cyclictest12-21ksoftirqd/022:54:590
684437995735,17cyclictest1-21systemd23:20:010
684437995735,17cyclictest1-21systemd23:20:010
684437995720,33cyclictest12-21ksoftirqd/021:20:030
68444799563,15cyclictest37-21ksoftirqd/322:00:023
68444799555,7cyclictest37-21ksoftirqd/322:30:023
68444799554,12cyclictest37-21ksoftirqd/319:45:023
684447995514,10cyclictest37-21ksoftirqd/323:55:023
684447995510,7cyclictest0-21swapper/322:15:013
684443995537,13cyclictest235-21systemd-journal23:05:002
684443995537,13cyclictest235-21systemd-journal23:05:002
684437995523,3cyclictest131rcu_preempt19:39:210
68444799547,9cyclictest37-21ksoftirqd/322:20:013
68444799546,11cyclictest37-21ksoftirqd/300:10:023
68444799541,15cyclictest687050-21grep20:00:013
68444799538,8cyclictest37-21ksoftirqd/300:20:003
68444799535,8cyclictest37-21ksoftirqd/323:25:003
68444799535,8cyclictest37-21ksoftirqd/322:40:023
684447995313,11cyclictest37-21ksoftirqd/322:25:003
684447995313,11cyclictest37-21ksoftirqd/322:25:003
684447995214,10cyclictest37-21ksoftirqd/321:25:023
684447995213,11cyclictest37-21ksoftirqd/323:40:023
684447995210,10cyclictest37-21ksoftirqd/323:05:023
684447995210,10cyclictest37-21ksoftirqd/323:05:023
68444799516,11cyclictest37-21ksoftirqd/321:00:023
68444799516,10cyclictest37-21ksoftirqd/321:30:023
68444799514,10cyclictest37-21ksoftirqd/321:35:033
68444799512,9cyclictest37-21ksoftirqd/320:25:023
68444799512,9cyclictest37-21ksoftirqd/320:25:023
68444799512,14cyclictest37-21ksoftirqd/320:54:593
68444799511,11cyclictest685619-21sed19:30:033
68444799507,9cyclictest37-21ksoftirqd/319:20:013
68444799505,16cyclictest37-21ksoftirqd/321:15:023
68444799504,9cyclictest37-21ksoftirqd/321:55:023
68444799504,11cyclictest37-21ksoftirqd/323:00:013
68444799502,12cyclictest37-21ksoftirqd/322:05:013
684447995016,7cyclictest689852-21kworker/3:121:40:003
68444799501,10cyclictest37-21ksoftirqd/320:10:023
68444799501,10cyclictest37-21ksoftirqd/320:10:023
684437995035,11cyclictest1-21systemd00:10:010
68444799496,7cyclictest37-21ksoftirqd/323:15:013
68444799494,11cyclictest37-21ksoftirqd/319:25:023
68444799493,8cyclictest37-21ksoftirqd/321:20:033
68444799493,14cyclictest37-21ksoftirqd/322:45:023
68444799492,10cyclictest37-21ksoftirqd/319:40:023
68444799491,10cyclictest685684-21unin-run19:35:003
684447994910,9cyclictest37-21ksoftirqd/300:05:003
684439994912,6cyclictest131rcu_preempt23:20:021
684439994912,6cyclictest131rcu_preempt23:20:021
684437994919,10cyclictest131rcu_preempt21:30:020
68444799487,7cyclictest37-21ksoftirqd/300:35:023
68444799485,12cyclictest684429-21cyclictest23:20:013
68444799485,12cyclictest684429-21cyclictest23:20:013
68444799484,7cyclictest37-21ksoftirqd/323:30:023
68444799483,5cyclictest37-21ksoftirqd/323:45:023
68444799482,12cyclictest37-21ksoftirqd/322:50:023
68444799482,12cyclictest37-21ksoftirqd/322:50:023
68444399481,4cyclictest131rcu_preempt22:55:012
684437994839,6cyclictest16950irq/224-800000022:10:020
684437994815,6cyclictest131rcu_preempt23:00:010
68444799476,7cyclictest37-21ksoftirqd/300:30:013
68444799472,11cyclictest37-21ksoftirqd/322:10:023
68444799471,8cyclictest37-21ksoftirqd/320:30:003
684437994740,4cyclictest16950irq/224-800000000:25:000
68444799466,5cyclictest37-21ksoftirqd/320:15:013
68444799464,6cyclictest37-21ksoftirqd/323:10:003
68444799464,12cyclictest37-21ksoftirqd/322:50:033
68444799461,7cyclictest0-21swapper/321:05:033
68444399463,3cyclictest131rcu_preempt23:30:022
684443994633,10cyclictest235-21systemd-journal21:40:012
684443994630,13cyclictest272-21dbus-daemon21:10:002
684443994630,12cyclictest272-21dbus-daemon20:35:012
68443799463,6cyclictest698137-21modprobe23:35:000
68443799463,6cyclictest698137-21modprobe23:35:000
684437994628,13cyclictest1-21systemd19:15:010
684437994628,13cyclictest1-21systemd19:15:000
68443799462,40cyclictest131rcu_preempt19:32:130
68444799457,9cyclictest37-21ksoftirqd/319:50:023
68444799457,7cyclictest37-21ksoftirqd/300:15:023
68444799457,6cyclictest37-21ksoftirqd/300:25:023
684447994531,10cyclictest272-21dbus-daemon19:55:013
68444799451,11cyclictest37-21ksoftirqd/320:20:003
684443994529,11cyclictest272-21dbus-daemon22:04:592
684443994528,11cyclictest1-21systemd00:20:002
684443994527,12cyclictest235-21systemd-journal20:45:012
684443994526,14cyclictest272-21dbus-daemon00:00:002
68443799453,3cyclictest131rcu_preempt21:40:020
68443799453,3cyclictest131rcu_preempt20:55:000
684437994531,11cyclictest272-21dbus-daemon00:30:000
68444799446,11cyclictest37-21ksoftirqd/321:10:013
684447994424,14cyclictest272-21dbus-daemon23:50:013
684447994424,14cyclictest272-21dbus-daemon23:50:013
68444399444,3cyclictest131rcu_preempt19:35:022
68444399443,4cyclictest131rcu_preempt19:39:592
684443994430,10cyclictest272-21dbus-daemon20:05:002
684443994427,13cyclictest235-21systemd-journal23:15:222
684443994427,13cyclictest235-21systemd-journal23:15:212
684437994435,6cyclictest693988-21munin-plugin-st22:15:000
68444799439,4cyclictest131rcu_preempt20:45:013
68444799435,9cyclictest37-21ksoftirqd/320:31:423
68444799432,10cyclictest37-21ksoftirqd/319:10:023
68444399434,3cyclictest131rcu_preempt22:50:002
68444399434,3cyclictest131rcu_preempt22:50:002
684443994326,13cyclictest272-21dbus-daemon00:35:012
684443994323,14cyclictest235-21systemd-journal21:00:002
684439994333,6cyclictest690347-21cron21:05:011
684439994333,5cyclictest691135-21sendmail21:20:011
684437994332,8cyclictest12-21ksoftirqd/023:10:010
684437994331,9cyclictest12-21ksoftirqd/019:19:210
68443799431,29cyclictest0-21swapper/019:22:210
68444799425,8cyclictest37-21ksoftirqd/320:50:023
684439994227,11cyclictest693808-21sed22:10:001
68443999422,4cyclictest131rcu_preempt23:05:001
68443999422,4cyclictest131rcu_preempt23:05:001
684437994229,9cyclictest235-21systemd-journal00:15:000
684437994226,13cyclictest689016-21sshd20:37:230
684437994226,13cyclictest363-21snmpd22:32:070
684437994225,12cyclictest272-21dbus-daemon20:10:010
684437994225,12cyclictest272-21dbus-daemon20:10:010
684437994225,11cyclictest272-21dbus-daemon21:55:010
684443994128,9cyclictest30-21ksoftirqd/222:20:002
684443994123,15cyclictest685050-21latency_hist19:20:022
68444399411,3cyclictest131rcu_preempt20:15:002
68443999412,3cyclictest131rcu_preempt22:55:021
684439994121,14cyclictest699460-21basename00:00:001
684437994128,10cyclictest12-21ksoftirqd/019:55:010
684437994125,13cyclictest272-21dbus-daemon23:05:010
684437994125,13cyclictest272-21dbus-daemon23:05:010
684437994125,12cyclictest701301-21sshd00:33:220
684437994122,14cyclictest694258-21cat22:20:000
68443799411,35cyclictest700507-21sshd00:17:220
684443994031,5cyclictest694771-21cron22:30:002
684443994027,9cyclictest30-21ksoftirqd/222:35:002
684443994027,10cyclictest1-21systemd21:35:012
684443994022,14cyclictest1-21systemd20:50:012
68444399401,4cyclictest131rcu_preempt20:25:032
68444399401,4cyclictest131rcu_preempt20:25:032
684439994022,13cyclictest1-21systemd20:20:011
68443999401,7cyclictest0-21swapper/121:40:001
684437994034,3cyclictest16950irq/224-800000022:00:010
684437994024,13cyclictest697066-21sshd23:10:220
684437994024,12cyclictest699158-21sshd23:53:220
684437994022,13cyclictest235-21systemd-journal20:29:590
684437994020,13cyclictest701575-21modprobe00:40:000
684437994017,18cyclictest1-21systemd00:05:000
68444799391,10cyclictest692821-21latency_hist21:50:023
684443993927,8cyclictest689057-21systemctl20:40:012
684443993926,10cyclictest30-21ksoftirqd/200:30:002
68444399392,5cyclictest131rcu_preempt23:55:012
684443993922,11cyclictest696334-21sendmail23:00:002
68443999394,3cyclictest131rcu_preempt21:35:011
684439993925,9cyclictest272-21dbus-daemon21:15:011
684439993925,11cyclictest697111-21systemd-run23:15:001
68443799399,5cyclictest131rcu_preempt21:45:010
684437993923,12cyclictest235-21systemd-journal21:50:010
684437993920,14cyclictest1-21systemd21:35:010
68443799391,3cyclictest131rcu_preempt19:30:030
684443993830,6cyclictest30-21ksoftirqd/221:25:012
684443993828,7cyclictest691088-21sshd21:15:222
684443993823,11cyclictest30-21ksoftirqd/200:40:012
684443993819,14cyclictest1-21systemd22:45:002
68443999385,3cyclictest131rcu_preempt20:15:011
68443999383,3cyclictest131rcu_preempt23:29:221
684439993826,9cyclictest306-21in:imuxsock00:18:221
684439993826,9cyclictest1-21systemd21:30:001
68443999381,4cyclictest131rcu_preempt22:29:591
68443799384,17cyclictest689463-21cat20:45:020
684437993827,8cyclictest12-21ksoftirqd/021:10:010
684437993826,9cyclictest695536-21sshd22:41:220
684437993819,14cyclictest689541-21sshd20:45:220
684443993723,11cyclictest1-21systemd23:50:012
684443993723,11cyclictest1-21systemd23:50:012
684439993729,6cyclictest23-21ksoftirqd/123:45:021
684439993727,7cyclictest23-21ksoftirqd/122:35:021
684439993726,8cyclictest688534-21sh20:30:001
684439993726,8cyclictest23-21ksoftirqd/122:05:001
684439993725,8cyclictest23-21ksoftirqd/119:19:591
684439993721,12cyclictest685327-21sed19:25:021
684439993719,12cyclictest235-21systemd-journal19:40:011
68443799373,5cyclictest687953-21sed20:15:020
68443799373,4cyclictest131rcu_preempt19:50:010
684437993728,5cyclictest306-21in:imuxsock21:20:030
68443799372,31cyclictest690368-21systemd-run21:05:010
684437993721,11cyclictest235-21systemd-journal20:20:000
68443799371,3cyclictest131rcu_preempt21:00:000
68443799371,32cyclictest408-21lldpd22:20:290
68443799371,32cyclictest408-21lldpd22:20:290
68444399364,7cyclictest131rcu_preempt19:46:212
68444399362,7cyclictest0-21swapper/222:10:012
68444399362,4cyclictest131rcu_preempt20:30:022
684443993624,8cyclictest687670-21cat20:10:032
684443993624,8cyclictest687670-21cat20:10:032
684443993623,10cyclictest1-21systemd23:45:012
684443993620,11cyclictest0-21swapper/220:55:002
68444399361,3cyclictest131rcu_preempt21:30:022
684443993611,22cyclictest1-21systemd00:25:012
68443999363,3cyclictest131rcu_preempt23:25:011
684439993625,6cyclictest306-21in:imuxsock20:00:011
684439993624,10cyclictest23-21ksoftirqd/100:05:001
684439993623,9cyclictest23-21ksoftirqd/123:55:001
684439993623,10cyclictest23-21ksoftirqd/120:37:261
68443999361,3cyclictest131rcu_preempt23:40:001
68443799363,6cyclictest695886-21latency_hist22:50:020
68443799363,6cyclictest695886-21latency_hist22:50:020
684437993628,5cyclictest12-21ksoftirqd/023:50:020
684437993628,5cyclictest12-21ksoftirqd/023:50:020
68443799362,29cyclictest0-21swapper/019:43:310
68443799361,31cyclictest0-21swapper/021:11:030
68444399354,3cyclictest131rcu_preempt00:10:012
684443993528,4cyclictest694189-21latency_hist22:15:022
684443993524,7cyclictest684613-21cron19:15:012
684443993524,7cyclictest684613-21cron19:15:002
684443993521,11cyclictest686722-21systemd-run19:55:012
684443993520,10cyclictest235-21systemd-journal19:45:012
684443993519,12cyclictest272-21dbus-daemon23:10:002
684443993519,12cyclictest272-21dbus-daemon19:30:012
68444399351,6cyclictest688016-21munin-run20:20:002
684443993516,16cyclictest272-21dbus-daemon21:45:012
684443993515,14cyclictest272-21dbus-daemon00:15:012
684439993524,8cyclictest686902-21sed19:55:021
684439993523,8cyclictest235-21systemd-journal23:10:001
684439993522,9cyclictest306-21in:imuxsock19:41:201
684439993522,9cyclictest272-21dbus-daemon22:40:021
684439993522,9cyclictest235-21systemd-journal19:50:001
68443999351,3cyclictest131rcu_preempt21:50:021
68443999351,3cyclictest131rcu_preempt19:35:011
684439993513,12cyclictest23-21ksoftirqd/123:00:011
68443799358,16cyclictest684436-21sort19:10:020
684437993528,4cyclictest695442-21latency_hist22:40:020
684437993525,7cyclictest0-21swapper/022:25:330
684437993523,9cyclictest12-21ksoftirqd/023:40:020
684437993523,9cyclictest12-21ksoftirqd/022:05:010
68443799352,10cyclictest1-21systemd23:22:220
68443799351,31cyclictest0-21swapper/020:00:040
68444399342,5cyclictest685167-21munin-run19:25:002
684443993425,6cyclictest30-21ksoftirqd/220:00:022
684443993423,7cyclictest698566-21latency_hist23:40:022
684443993420,11cyclictest30-21ksoftirqd/223:15:002
68443999346,3cyclictest131rcu_preempt21:45:001
68443999345,5cyclictest698153-21true23:35:011
68443999345,5cyclictest698153-21true23:35:011
68443999344,3cyclictest131rcu_preempt22:20:011
684439993427,4cyclictest684369-21latency_hist19:10:011
684439993427,4cyclictest412-21sshd20:33:221
684439993426,5cyclictest693246-21sh22:00:011
684439993424,8cyclictest23-21ksoftirqd/122:15:021
684439993423,8cyclictest689541-21sshd20:45:221
68443999341,7cyclictest0-21swapper/120:10:021
68443999341,7cyclictest0-21swapper/120:10:021
684437993426,5cyclictest688354-21latency_hist20:25:020
684437993426,5cyclictest688354-21latency_hist20:25:020
68443799341,3cyclictest131rcu_preempt19:55:020
684443993326,5cyclictest697640-21latency_hist23:25:012
68444399332,4cyclictest690360-21munin-run21:05:012
684443993324,7cyclictest30-21ksoftirqd/221:11:232
68444399331,8cyclictest694497-21modprobe22:20:022
68444399331,8cyclictest694497-21modprobe22:20:022
68444399331,3cyclictest131rcu_preempt22:40:022
68443999337,3cyclictest131rcu_preempt00:40:011
68443999337,3cyclictest131rcu_preempt00:25:011
68443999335,12cyclictest0-21swapper/120:25:021
68443999335,12cyclictest0-21swapper/120:25:011
68443999333,3cyclictest131rcu_preempt21:55:011
68443999332,6cyclictest0-21swapper/121:10:011
684439993326,4cyclictest684754-21latency_hist19:15:021
684439993326,4cyclictest684754-21latency_hist19:15:021
68443999332,5cyclictest131rcu_preempt21:25:001
684439993325,5cyclictest412-21sshd20:44:221
68443999332,3cyclictest131rcu_preempt22:50:021
68443999332,3cyclictest131rcu_preempt22:50:021
684439993323,7cyclictest700341-21sed00:15:021
684437993327,4cyclictest0-21swapper/023:25:200
684443993223,6cyclictest235-21systemd-journal21:55:012
68443999322,8cyclictest23-21ksoftirqd/120:58:061
684439993225,5cyclictest700016-21latency_hist00:10:021
684439993224,5cyclictest695700-21latency_hist22:45:021
684439993218,10cyclictest23-21ksoftirqd/120:00:031
68443999321,3cyclictest131rcu_preempt19:30:021
68443799322,4cyclictest0-21swapper/020:32:220
68444399312,6cyclictest692717-21munin-run21:50:022
684443993123,4cyclictest684369-21latency_hist19:10:022
68444399311,3cyclictest131rcu_preempt23:35:012
68444399311,3cyclictest131rcu_preempt23:35:012
684439993123,5cyclictest701172-21latency_hist00:30:011
684439993122,7cyclictest23-21ksoftirqd/120:52:221
684439993120,8cyclictest235-21systemd-journal23:50:001
684439993120,8cyclictest235-21systemd-journal23:50:001
68443999311,3cyclictest131rcu_preempt22:25:071
68443999311,3cyclictest131rcu_preempt22:25:071
68443999303,4cyclictest131rcu_preempt00:31:281
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional