You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 09:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sat Nov 29, 2025 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4368209910878,15cyclictest451847-40mandb00:00:021
436825998028,11cyclictest30-21ksoftirqd/200:00:012
43682599686,8cyclictest30-21ksoftirqd/219:35:012
43682599685,8cyclictest30-21ksoftirqd/200:00:032
43682599614,7cyclictest30-21ksoftirqd/223:35:012
436828995845,11cyclictest37-21ksoftirqd/320:25:003
43682599581,12cyclictest448215-21latency_hist22:50:022
436814995815,10cyclictest131rcu_preempt22:00:000
43682599575,9cyclictest30-21ksoftirqd/222:55:012
43682599567,10cyclictest30-21ksoftirqd/219:45:002
436825995640,12cyclictest452852-21cron00:20:002
436825995633,16cyclictest235-21systemd-journal20:15:012
436814995637,14cyclictest272-21dbus-daemon21:00:010
43682599556,6cyclictest30-21ksoftirqd/220:05:022
43682599555,4cyclictest131rcu_preempt19:20:012
436820995538,13cyclictest1-21systemd23:20:011
436825995431,14cyclictest272-21dbus-daemon23:25:012
436814995438,12cyclictest443525-21systemctl21:20:000
43682599536,8cyclictest30-21ksoftirqd/200:30:012
43682599533,8cyclictest30-21ksoftirqd/222:05:012
436825995333,16cyclictest448708-21run-parts23:00:002
436825995331,12cyclictest272-21dbus-daemon23:15:012
43682599532,10cyclictest30-21ksoftirqd/221:55:012
43682599532,10cyclictest30-21ksoftirqd/221:55:002
436825995311,10cyclictest30-21ksoftirqd/223:40:012
43682599522,10cyclictest447183-21latency_hist22:30:002
436825995218,12cyclictest30-21ksoftirqd/221:05:022
436825995211,7cyclictest131rcu_preempt23:45:012
43682599521,12cyclictest30-21ksoftirqd/220:00:002
436820995232,16cyclictest1-21systemd19:20:001
43682599518,8cyclictest30-21ksoftirqd/221:30:012
43682599517,8cyclictest30-21ksoftirqd/220:35:012
43682599516,9cyclictest30-21ksoftirqd/223:10:012
43682599515,8cyclictest30-21ksoftirqd/221:42:522
436825995138,9cyclictest235-21systemd-journal00:25:002
436825995138,10cyclictest1-21systemd22:35:012
43682599513,12cyclictest30-21ksoftirqd/219:40:012
43682599513,12cyclictest30-21ksoftirqd/219:40:012
436820995136,11cyclictest1-21systemd23:25:011
43682599504,11cyclictest30-21ksoftirqd/221:19:592
436825995034,11cyclictest1-21systemd21:10:002
436825995032,14cyclictest235-21systemd-journal22:10:012
436825995032,14cyclictest235-21systemd-journal22:10:002
43682599502,10cyclictest30-21ksoftirqd/221:25:022
43682599502,10cyclictest30-21ksoftirqd/221:25:022
436825995016,13cyclictest30-21ksoftirqd/221:40:012
436825995016,13cyclictest30-21ksoftirqd/221:40:002
43682599501,12cyclictest449774-21sendmail-msp23:20:012
436814995032,13cyclictest272-21dbus-daemon00:40:010
43682599496,10cyclictest30-21ksoftirqd/220:25:012
43682599495,8cyclictest30-21ksoftirqd/221:15:012
43682599493,7cyclictest30-21ksoftirqd/221:35:022
436825994934,11cyclictest448980-21munin-plugin-st23:05:012
436825994934,11cyclictest448980-21munin-plugin-st23:05:002
43682599493,11cyclictest30-21ksoftirqd/223:30:012
43682599492,9cyclictest30-21ksoftirqd/200:15:002
43682599492,13cyclictest30-21ksoftirqd/200:10:012
43682599492,13cyclictest30-21ksoftirqd/200:10:002
436825994910,5cyclictest131rcu_preempt22:35:032
436820994932,12cyclictest272-21dbus-daemon22:10:011
436820994932,12cyclictest272-21dbus-daemon22:10:001
43681499499,6cyclictest131rcu_preempt19:35:000
436814994945,2cyclictest0-21swapper/022:15:010
436814994934,11cyclictest451822-21systemctl00:00:000
43681499492,7cyclictest1-21systemd23:05:010
43681499492,7cyclictest1-21systemd23:05:000
436814994918,15cyclictest12-21ksoftirqd/021:30:000
436828994812,6cyclictest131rcu_preempt19:45:013
43682599487,9cyclictest30-21ksoftirqd/220:20:012
43682599485,9cyclictest30-21ksoftirqd/220:10:022
43682599485,9cyclictest30-21ksoftirqd/220:10:012
43682599484,14cyclictest30-21ksoftirqd/220:45:002
43682599483,6cyclictest30-21ksoftirqd/221:00:012
43682599483,12cyclictest30-21ksoftirqd/219:50:002
436825994831,11cyclictest441422-21sh20:40:002
436825994828,14cyclictest1-21systemd22:45:002
43682599482,10cyclictest30-21ksoftirqd/223:50:032
436820994830,14cyclictest272-21dbus-daemon23:00:001
436814994833,12cyclictest1-21systemd23:50:010
43682599474,8cyclictest30-21ksoftirqd/219:30:012
436825994724,15cyclictest235-21systemd-journal00:35:012
43682599471,5cyclictest0-21swapper/219:55:002
436814994730,13cyclictest272-21dbus-daemon22:55:000
43681499472,9cyclictest448045-21latency_hist22:45:020
436814994717,6cyclictest131rcu_preempt00:35:000
436814994717,17cyclictest12-21ksoftirqd/022:20:010
43682899461,5cyclictest131rcu_preempt22:00:003
43682599466,9cyclictest30-21ksoftirqd/220:55:002
43682599465,9cyclictest30-21ksoftirqd/219:10:032
43682599465,13cyclictest30-21ksoftirqd/221:50:012
43682599462,9cyclictest30-21ksoftirqd/200:37:042
436825994627,14cyclictest272-21dbus-daemon22:20:002
43682599461,12cyclictest30-21ksoftirqd/222:00:002
436820994632,10cyclictest272-21dbus-daemon19:15:001
436814994631,11cyclictest272-21dbus-daemon00:20:010
436814994627,14cyclictest440648-21sshd20:24:370
43682899452,13cyclictest131rcu_preempt00:00:013
43682599456,7cyclictest30-21ksoftirqd/220:30:012
436825994515,9cyclictest30-21ksoftirqd/222:15:022
43682099453,5cyclictest131rcu_preempt22:00:001
436820994528,13cyclictest235-21systemd-journal19:50:001
436820994525,10cyclictest23-21ksoftirqd/123:10:001
436814994535,7cyclictest19050irq/34-mmc100:04:590
436814994530,10cyclictest272-21dbus-daemon23:55:010
436814994527,13cyclictest235-21systemd-journal22:59:520
436814994526,13cyclictest272-21dbus-daemon20:50:010
43681499452,39cyclictest445295-21sshd21:50:520
43681499452,39cyclictest445295-21sshd21:50:510
436814994512,28cyclictest12-21ksoftirqd/021:41:530
43682899447,6cyclictest131rcu_preempt23:00:003
436828994432,9cyclictest37-21ksoftirqd/321:45:513
436828994419,7cyclictest131rcu_preempt23:10:013
436828994413,22cyclictest361rcuc/321:15:023
436828994410,20cyclictest131rcu_preempt22:35:013
43682599447,8cyclictest30-21ksoftirqd/220:50:022
436825994410,5cyclictest131rcu_preempt19:15:002
436820994436,5cyclictest440121-21munin-plugin-st20:15:001
436820994430,9cyclictest235-21systemd-journal19:59:591
436814994430,11cyclictest1-21systemd23:45:010
43681499442,9cyclictest440131-21systemd-run20:15:000
436814994420,19cyclictest12-21ksoftirqd/019:10:030
436814994416,17cyclictest131rcu_preempt19:30:000
43681499441,39cyclictest4429741sshd21:09:520
436814994412,6cyclictest131rcu_preempt19:19:590
436814994412,19cyclictest131rcu_preempt20:00:000
43682899431,7cyclictest131rcu_preempt00:25:013
436825994329,10cyclictest30-21ksoftirqd/223:55:012
436825994329,10cyclictest1-21systemd22:25:022
436820994327,12cyclictest272-21dbus-daemon20:20:001
436814994331,8cyclictest440906-21cron20:30:000
436814994323,16cyclictest272-21dbus-daemon21:40:010
436814994323,16cyclictest272-21dbus-daemon21:40:010
436814994316,19cyclictest12-21ksoftirqd/020:05:010
43681499431,38cyclictest442202-21latency_hist20:55:020
436814994310,6cyclictest131rcu_preempt22:40:010
436828994229,10cyclictest37-21ksoftirqd/320:40:023
436828994227,12cyclictest447684-21sendmail22:40:003
43682899422,5cyclictest131rcu_preempt19:30:013
436828994224,14cyclictest1-21systemd23:30:013
436828994223,14cyclictest1-21systemd21:20:013
436828994220,16cyclictest272-21dbus-daemon21:55:013
436828994220,16cyclictest272-21dbus-daemon21:55:003
43682899421,3cyclictest131rcu_preempt21:10:033
436820994230,8cyclictest235-21systemd-journal22:50:001
436820994228,11cyclictest235-21systemd-journal00:29:031
436820994225,12cyclictest272-21dbus-daemon23:05:011
436820994225,12cyclictest272-21dbus-daemon23:05:011
436820994223,13cyclictest272-21dbus-daemon23:35:001
43681499424,6cyclictest131rcu_preempt22:10:010
43681499424,6cyclictest131rcu_preempt22:10:010
436814994226,13cyclictest452319-21sshd00:06:040
436814994226,13cyclictest452319-21sshd00:06:040
436814994221,16cyclictest272-21dbus-daemon21:15:000
436814994216,22cyclictest12-21ksoftirqd/020:40:000
43681499421,36cyclictest437524-21sshd19:22:120
43681499421,36cyclictest437524-21sshd19:22:110
436828994128,10cyclictest439852-21sshd20:08:393
436828994128,10cyclictest439852-21sshd20:08:393
43682899412,5cyclictest131rcu_preempt23:25:003
436828994125,12cyclictest272-21dbus-daemon00:15:003
436828994122,6cyclictest37-21ksoftirqd/323:45:003
436828994122,13cyclictest272-21dbus-daemon19:20:003
43682899411,6cyclictest131rcu_preempt00:40:023
436828994112,6cyclictest131rcu_preempt23:05:013
436828994112,6cyclictest131rcu_preempt23:05:003
43682599414,11cyclictest30-21ksoftirqd/219:25:012
43682599414,11cyclictest30-21ksoftirqd/219:25:012
436820994127,10cyclictest235-21systemd-journal00:36:231
436820994124,11cyclictest306-21in:imuxsock19:20:031
436820994124,11cyclictest306-21in:imuxsock19:20:021
436820994122,13cyclictest235-21systemd-journal23:45:011
436814994134,4cyclictest16950irq/224-800000023:25:020
436814994130,9cyclictest12-21ksoftirqd/023:12:510
436814994130,8cyclictest235-21systemd-journal23:21:510
436814994127,9cyclictest306-21in:imuxsock23:40:000
436814994125,4cyclictest12-21ksoftirqd/022:35:010
436814994124,4cyclictest12-21ksoftirqd/022:20:020
43682899404,3cyclictest131rcu_preempt00:00:023
43682899403,3cyclictest131rcu_preempt21:30:013
436828994026,10cyclictest37-21ksoftirqd/300:20:003
436828994025,11cyclictest235-21systemd-journal20:00:003
436828994023,14cyclictest453719-21sed00:35:023
436828994023,12cyclictest37-21ksoftirqd/323:20:013
436828994018,7cyclictest131rcu_preempt21:00:013
43682899401,5cyclictest131rcu_preempt19:35:003
436828994015,5cyclictest131rcu_preempt22:05:013
436828994014,18cyclictest37-21ksoftirqd/319:10:013
436820994028,9cyclictest442722-21sshd21:04:521
436820994025,11cyclictest272-21dbus-daemon20:55:011
43682099401,5cyclictest131rcu_preempt19:55:031
43681499406,4cyclictest131rcu_preempt20:20:020
43681499401,5cyclictest439103-21/usr/sbin/munin19:55:010
436814994014,17cyclictest436807-21cyclictest19:10:010
436814994010,6cyclictest131rcu_preempt19:40:020
436814994010,6cyclictest131rcu_preempt19:40:010
43682899398,6cyclictest131rcu_preempt22:45:013
43682899393,3cyclictest131rcu_preempt23:40:013
436828993929,7cyclictest453377-21sshd00:28:503
43682899391,5cyclictest131rcu_preempt23:35:023
436820993929,8cyclictest23-21ksoftirqd/121:45:021
436820993929,5cyclictest437799-21sh19:29:591
436820993927,9cyclictest272-21dbus-daemon21:20:011
436820993927,8cyclictest445832-21sshd22:01:521
436820993926,10cyclictest272-21dbus-daemon20:10:011
436820993926,10cyclictest272-21dbus-daemon20:10:001
436820993925,10cyclictest23-21ksoftirqd/122:25:031
436820993921,14cyclictest451530-21sshd23:50:531
43682099391,5cyclictest131rcu_preempt00:05:001
43681499397,10cyclictest12-21ksoftirqd/019:47:380
436814993925,9cyclictest307-21in:imklog23:35:010
436814993924,11cyclictest272-21dbus-daemon22:05:000
43681499391,33cyclictest235-21systemd-journal23:20:010
43682899387,5cyclictest131rcu_preempt21:45:003
43682899386,6cyclictest131rcu_preempt21:40:013
43682899386,6cyclictest131rcu_preempt21:40:013
436828993830,6cyclictest37-21ksoftirqd/319:50:013
43682899382,7cyclictest131rcu_preempt20:55:013
436828993827,9cyclictest37-21ksoftirqd/320:35:013
436828993825,10cyclictest37-21ksoftirqd/323:50:013
436828993810,5cyclictest131rcu_preempt22:06:583
436828993810,5cyclictest131rcu_preempt22:06:583
436828993810,19cyclictest0-21swapper/320:20:013
43682099386,5cyclictest131rcu_preempt21:25:011
43682099386,5cyclictest131rcu_preempt21:25:001
436820993826,9cyclictest447920-21sshd22:42:531
43682099381,6cyclictest131rcu_preempt21:00:001
436814993830,5cyclictest19050irq/34-mmc119:44:400
43682899374,6cyclictest131rcu_preempt20:04:373
43682899373,5cyclictest131rcu_preempt19:40:013
43682899373,5cyclictest131rcu_preempt19:40:003
43682899372,3cyclictest131rcu_preempt21:05:013
436828993716,7cyclictest131rcu_preempt21:35:013
43682899371,5cyclictest131rcu_preempt23:11:523
436828993713,15cyclictest131rcu_preempt22:16:523
43682099374,3cyclictest131rcu_preempt21:40:011
43682099374,3cyclictest131rcu_preempt21:40:001
436820993725,9cyclictest450980-21sed23:40:021
436820993725,9cyclictest445046-21sshd21:45:511
436820993723,10cyclictest23-21ksoftirqd/123:30:021
436820993721,10cyclictest0-21swapper/120:50:011
436814993728,5cyclictest443792-21munin-plugin-st21:25:010
436814993728,5cyclictest443792-21munin-plugin-st21:25:000
436814993724,10cyclictest12-21ksoftirqd/000:15:010
436814993714,9cyclictest12-21ksoftirqd/021:35:000
436814993710,17cyclictest440132-21kworker/0:020:45:020
43682899368,5cyclictest131rcu_preempt20:30:023
43682899364,8cyclictest37-21ksoftirqd/321:25:013
43682899364,8cyclictest37-21ksoftirqd/321:25:003
43682899362,5cyclictest131rcu_preempt20:50:013
436828993625,8cyclictest441746-21sed20:45:023
436828993624,9cyclictest272-21dbus-daemon19:55:003
436828993620,12cyclictest37-21ksoftirqd/300:10:013
436828993620,12cyclictest37-21ksoftirqd/300:10:013
43682099366,8cyclictest0-21swapper/120:05:021
43682099363,6cyclictest131rcu_preempt23:50:011
43682099362,5cyclictest131rcu_preempt21:10:001
436820993624,8cyclictest23-21ksoftirqd/119:35:011
436820993621,11cyclictest235-21systemd-journal19:40:011
436820993621,11cyclictest235-21systemd-journal19:40:001
43681499365,5cyclictest131rcu_preempt22:30:020
436814993628,6cyclictest12-21ksoftirqd/000:25:010
436814993610,5cyclictest131rcu_preempt21:50:020
43682899354,3cyclictest131rcu_preempt19:20:033
43682899354,3cyclictest131rcu_preempt19:20:023
43682899353,5cyclictest131rcu_preempt22:15:003
436828993527,5cyclictest443012-21kworker/3:222:25:013
43682899352,5cyclictest451713-21sed23:55:023
436828993525,5cyclictest37-21ksoftirqd/322:55:003
436828993525,5cyclictest37-21ksoftirqd/320:13:373
436828993514,5cyclictest131rcu_preempt22:48:523
436820993528,4cyclictest453140-21latency_hist00:25:011
436820993527,5cyclictest441136-21sshd20:31:481
43682099352,6cyclictest0-21swapper/121:30:011
43682099352,5cyclictest131rcu_preempt22:35:011
436820993524,9cyclictest23-21ksoftirqd/100:15:021
436820993524,8cyclictest441771-21sed20:45:011
436820993518,12cyclictest446549-21wc22:15:021
436820993517,3cyclictest23-21ksoftirqd/122:55:031
436820993516,6cyclictest131rcu_preempt20:26:391
43681499358,6cyclictest131rcu_preempt20:35:010
43681499354,6cyclictest131rcu_preempt20:10:030
43681499354,6cyclictest131rcu_preempt20:10:030
436814993524,8cyclictest449219-21sshd23:09:520
43682899347,4cyclictest437194-21latency_hist19:15:023
436828993420,11cyclictest37-21ksoftirqd/322:30:013
436820993424,7cyclictest440695-21sed20:25:011
436820993424,7cyclictest23-21ksoftirqd/100:35:021
43682099342,3cyclictest131rcu_preempt00:20:001
43681499341,5cyclictest453397-21munin-run00:30:000
43681499341,5cyclictest208-21jbd2/mmcblk1p2-22:45:450
436820993324,7cyclictest23-21ksoftirqd/119:45:001
436820993323,7cyclictest23-21ksoftirqd/122:30:021
436820993322,9cyclictest23-21ksoftirqd/121:51:561
436820993322,9cyclictest23-21ksoftirqd/121:51:551
436820993317,5cyclictest131rcu_preempt22:40:021
436820993316,13cyclictest1-21systemd22:20:011
43682099331,5cyclictest131rcu_preempt00:10:011
43682099331,5cyclictest131rcu_preempt00:10:001
436814993326,4cyclictest0-21swapper/021:01:440
43682099324,8cyclictest23-21ksoftirqd/121:10:471
436820993221,7cyclictest23-21ksoftirqd/120:40:011
436820993214,5cyclictest131rcu_preempt21:35:011
436820993023,4cyclictest449458-21sshd23:12:521
436820992920,6cyclictest436822-21latency_hist19:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional