You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 14:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Sun Nov 23, 2025 00:45:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3887364997225,8cyclictest306-21in:imuxsock00:15:422
3887364997224,8cyclictest306-21in:imuxsock20:26:432
3887364997224,8cyclictest306-21in:imuxsock20:26:422
3887364997032,12cyclictest235-21systemd-journal22:40:012
3887364997022,10cyclictest306-21in:imuxsock22:50:032
3887360996427,34cyclictest23-21ksoftirqd/122:55:021
3887360996423,35cyclictest3904467-21cron00:40:001
3887355996415,14cyclictest12-21ksoftirqd/022:00:010
3887360996322,37cyclictest272-21dbus-daemon23:10:011
3887360996319,8cyclictest3894566-21sshd21:27:421
3887360996214,11cyclictest272-21dbus-daemon20:25:011
388735599615,7cyclictest12-21ksoftirqd/021:20:020
388735599615,7cyclictest12-21ksoftirqd/021:20:020
3887355996151,6cyclictest12-21ksoftirqd/021:15:000
3887360996024,32cyclictest23-21ksoftirqd/122:35:001
3887360995927,28cyclictest1-21systemd22:10:011
3887360995927,28cyclictest1-21systemd22:10:011
3887360995821,5cyclictest3892496-21latency_hist20:50:021
388736099575,14cyclictest272-21dbus-daemon21:55:011
388736099575,14cyclictest272-21dbus-daemon21:55:011
388736099573,14cyclictest3902398-40mandb00:00:031
388736099573,14cyclictest3902398-40mandb00:00:031
3887360995723,30cyclictest23-21ksoftirqd/123:20:021
3887360995723,30cyclictest23-21ksoftirqd/123:20:021
388735599575,8cyclictest12-21ksoftirqd/019:55:020
3887355995750,5cyclictest12-21ksoftirqd/021:30:020
3887355995750,5cyclictest12-21ksoftirqd/021:30:010
388735599574,13cyclictest3890926-21modprobe20:20:000
3887360995622,31cyclictest3890006-21latency_hist20:00:021
3887360995622,31cyclictest3890006-21latency_hist20:00:021
388736099562,19cyclictest235-21systemd-journal23:15:011
388736099562,16cyclictest3902374-21munin-run00:00:001
388736099562,16cyclictest3888596-21munin-plugin-st19:35:001
3887360995517,5cyclictest3893998-21latency_hist21:15:021
3887355995537,14cyclictest1-21systemd20:50:000
3887360995421,4cyclictest3894481-21latency_hist21:25:021
3887360995419,4cyclictest3897694-21kworker/u8:222:40:001
3887360995413,11cyclictest23-21ksoftirqd/100:30:001
3887360995410,11cyclictest235-21systemd-journal00:20:031
3887355995417,6cyclictest3904672-21date00:40:020
3887369995330,16cyclictest37-21ksoftirqd/322:15:023
388736099532,17cyclictest3887691-21wc19:15:021
3887360995320,12cyclictest23-21ksoftirqd/123:00:011
3887360995317,5cyclictest3896877-21kworker/u8:122:45:001
388736099531,17cyclictest3892235-21cron20:44:591
388735599535,8cyclictest12-21ksoftirqd/019:25:000
3887355995334,13cyclictest235-21systemd-journal19:15:000
3887369995229,4cyclictest131rcu_preempt21:05:023
3887360995220,13cyclictest23-21ksoftirqd/121:17:421
3887360995220,13cyclictest23-21ksoftirqd/121:17:421
3887360995214,11cyclictest272-21dbus-daemon00:15:011
3887360995213,17cyclictest131rcu_preempt00:30:021
3887360995213,17cyclictest131rcu_preempt00:30:021
388735599526,8cyclictest12-21ksoftirqd/000:15:010
388735599526,11cyclictest12-21ksoftirqd/000:35:030
388735599526,11cyclictest12-21ksoftirqd/000:35:030
3887355995243,6cyclictest12-21ksoftirqd/000:00:010
3887355995234,13cyclictest272-21dbus-daemon21:00:000
3887355995231,14cyclictest235-21systemd-journal23:05:000
388735599522,7cyclictest3895106-21munin-run21:40:000
3887355995216,5cyclictest0-21swapper/000:00:520
3887355995216,5cyclictest0-21swapper/000:00:520
3887360995135,12cyclictest1-21systemd20:05:001
3887360995125,22cyclictest3893041-21sh21:00:001
3887360995121,8cyclictest23-21ksoftirqd/121:50:021
3887360995121,8cyclictest23-21ksoftirqd/121:50:021
3887360995119,9cyclictest3894962-21date21:35:021
3887360995119,9cyclictest3894962-21date21:35:021
3887360995119,4cyclictest3891548-21latency_hist20:30:021
3887360995119,4cyclictest3891548-21latency_hist20:30:021
388735599518,6cyclictest3888327-21kworker/u8:019:40:000
388736099502,16cyclictest272-21dbus-daemon19:40:001
388735599505,5cyclictest131rcu_preempt20:30:030
388735599502,14cyclictest12-21ksoftirqd/021:55:020
388735599502,14cyclictest12-21ksoftirqd/021:55:020
3887369994931,14cyclictest235-21systemd-journal23:10:003
3887360994932,13cyclictest1-21systemd00:10:001
388736099492,18cyclictest3888105-21munin-run19:25:001
3887360994919,9cyclictest23-21ksoftirqd/122:16:531
3887360994919,7cyclictest23-21ksoftirqd/122:47:001
388735599492,8cyclictest12-21ksoftirqd/000:30:020
388735599491,42cyclictest3890182-21unin-run20:05:000
3887369994829,14cyclictest1-21systemd21:40:003
388736099483,14cyclictest3903461-21modprobe00:20:011
3887360994813,10cyclictest3887781-21cron19:17:011
388735599487,8cyclictest12-21ksoftirqd/022:50:020
388735599486,5cyclictest131rcu_preempt23:55:010
388735599485,10cyclictest12-21ksoftirqd/023:40:020
3887355994829,12cyclictest235-21systemd-journal19:35:010
388735599482,37cyclictest306-21in:imuxsock00:20:010
3887369994728,12cyclictest235-21systemd-journal20:35:003
3887360994733,10cyclictest1-21systemd22:25:011
3887360994717,7cyclictest3901318-21cron23:40:001
3887360994716,10cyclictest23-21ksoftirqd/122:05:011
3887360994715,8cyclictest3889764-21cut19:55:011
388735599471,9cyclictest3900612-21cat23:25:010
3887355994714,6cyclictest131rcu_preempt20:10:000
3887369994633,7cyclictest3889442-21cat19:50:013
3887369994629,12cyclictest272-21dbus-daemon00:00:013
3887364994633,10cyclictest30-21ksoftirqd/221:40:002
3887360994621,8cyclictest23-21ksoftirqd/123:35:021
3887360994621,8cyclictest23-21ksoftirqd/123:35:011
3887360994614,8cyclictest23-21ksoftirqd/121:10:001
3887369994526,14cyclictest272-21dbus-daemon19:15:013
3887364994531,11cyclictest235-21systemd-journal20:15:002
3887360994518,12cyclictest3889384-21cut19:50:011
388735599456,11cyclictest12-21ksoftirqd/000:10:010
388735599454,8cyclictest12-21ksoftirqd/023:20:020
388735599454,8cyclictest12-21ksoftirqd/023:20:020
388735599454,16cyclictest12-21ksoftirqd/023:12:050
3887355994538,4cyclictest16950irq/224-800000020:33:420
388735599452,11cyclictest12-21ksoftirqd/019:10:020
388735599443,8cyclictest12-21ksoftirqd/022:10:020
388735599443,8cyclictest12-21ksoftirqd/022:10:020
3887355994434,7cyclictest12-21ksoftirqd/022:30:010
3887355994433,8cyclictest235-21systemd-journal20:20:020
3887355994427,13cyclictest3900789-21sshd23:25:420
3887355994427,13cyclictest235-21systemd-journal22:02:420
3887369994336,5cyclictest361rcuc/321:10:023
3887369994329,11cyclictest37-21ksoftirqd/322:30:013
3887369994328,11cyclictest306-21in:imuxsock23:00:003
3887369994326,11cyclictest272-21dbus-daemon20:55:013
388736499433,3cyclictest131rcu_preempt23:40:012
388735599437,7cyclictest12-21ksoftirqd/023:45:020
388735599437,7cyclictest12-21ksoftirqd/023:45:020
388735599436,7cyclictest12-21ksoftirqd/021:05:020
3887355994335,5cyclictest12-21ksoftirqd/021:50:020
3887355994335,5cyclictest12-21ksoftirqd/021:50:020
3887355994335,4cyclictest16950irq/224-800000021:10:010
3887355994333,8cyclictest12-21ksoftirqd/023:00:000
388735599432,37cyclictest3894416-21sed21:25:010
3887369994234,6cyclictest37-21ksoftirqd/300:30:343
3887369994234,6cyclictest37-21ksoftirqd/300:30:343
3887369994218,20cyclictest361rcuc/300:25:023
388736499427,7cyclictest30-21ksoftirqd/200:00:122
388736499427,7cyclictest30-21ksoftirqd/200:00:122
3887364994225,12cyclictest272-21dbus-daemon19:35:002
3887364994222,14cyclictest272-21dbus-daemon19:30:012
3887364994222,14cyclictest272-21dbus-daemon19:30:002
388735599425,8cyclictest12-21ksoftirqd/019:45:010
388735599425,8cyclictest12-21ksoftirqd/019:45:000
388735599423,8cyclictest272-21dbus-daemon23:50:000
3887355994231,8cyclictest12-21ksoftirqd/022:20:020
388735599421,7cyclictest3895509-21date21:45:020
3887369994129,9cyclictest3904467-21cron00:40:003
3887369994129,7cyclictest3897946-21cron22:35:003
3887369994129,6cyclictest306-21in:imuxsock20:45:003
3887369994125,12cyclictest37-21ksoftirqd/321:45:013
3887369994125,12cyclictest272-21dbus-daemon20:50:013
3887369994120,14cyclictest272-21dbus-daemon19:45:013
3887369994120,14cyclictest272-21dbus-daemon19:45:003
3887364994130,7cyclictest3902368-21sendmail00:00:002
3887364994129,7cyclictest306-21in:imuxsock22:50:012
3887364994128,9cyclictest3900280-21cron23:20:002
3887364994128,9cyclictest3900280-21cron23:20:002
388736499412,5cyclictest131rcu_preempt20:20:002
3887364994125,12cyclictest235-21systemd-journal22:35:002
3887364994121,15cyclictest272-21dbus-daemon22:05:012
3887360994128,10cyclictest235-21systemd-journal22:29:421
388735599416,7cyclictest12-21ksoftirqd/000:25:020
388735599415,7cyclictest12-21ksoftirqd/022:35:020
3887355994130,7cyclictest16950irq/224-800000022:40:010
3887355994128,9cyclictest3891954-21cron20:40:000
3887355994128,9cyclictest3891954-21cron20:39:590
388735599411,36cyclictest0-21swapper/020:12:430
3887355994112,6cyclictest131rcu_preempt19:50:000
3887355994112,5cyclictest131rcu_preempt20:30:010
388736999409,17cyclictest3890930-21modprobe20:20:003
388736999403,24cyclictest235-21systemd-journal21:25:013
388736999402,7cyclictest131rcu_preempt22:00:003
3887369994027,10cyclictest37-21ksoftirqd/300:10:003
3887369994025,12cyclictest272-21dbus-daemon23:55:013
3887369994024,13cyclictest37-21ksoftirqd/321:30:023
3887369994024,13cyclictest37-21ksoftirqd/321:30:013
3887369994024,12cyclictest272-21dbus-daemon23:35:013
3887369994024,12cyclictest272-21dbus-daemon23:35:013
388736499407,28cyclictest3903965-21cut00:30:002
3887364994028,9cyclictest3897177-21cron22:20:002
388736499402,3cyclictest131rcu_preempt19:10:022
3887364994023,12cyclictest272-21dbus-daemon23:15:002
3887360994029,8cyclictest235-21systemd-journal21:01:421
388736099402,6cyclictest131rcu_preempt21:40:011
388735599408,8cyclictest12-21ksoftirqd/022:45:010
3887355994010,5cyclictest131rcu_preempt23:10:000
3887369993929,5cyclictest306-21in:imuxsock19:35:003
3887369993920,13cyclictest235-21systemd-journal23:20:003
3887369993920,13cyclictest235-21systemd-journal23:20:003
3887364993929,6cyclictest3896650-21cron22:10:002
3887364993929,6cyclictest3896650-21cron22:09:592
3887364993927,8cyclictest306-21in:imuxsock22:30:012
3887364993927,6cyclictest306-21in:imuxsock00:25:002
3887364993922,12cyclictest272-21dbus-daemon22:00:012
388736099396,3cyclictest131rcu_preempt22:15:011
3887360993926,10cyclictest3902281-21sed23:55:021
3887360993924,12cyclictest23-21ksoftirqd/120:20:001
3887355993918,14cyclictest272-21dbus-daemon19:20:010
388735599391,34cyclictest3888329-21sshd19:29:420
388735599391,34cyclictest3888329-21sshd19:29:410
3887369993826,8cyclictest235-21systemd-journal20:25:003
3887369993825,8cyclictest0-21swapper/321:15:003
3887369993825,10cyclictest37-21ksoftirqd/300:05:013
3887369993825,10cyclictest37-21ksoftirqd/300:05:013
3887369993823,11cyclictest37-21ksoftirqd/319:55:003
3887369993823,11cyclictest235-21systemd-journal21:30:003
3887369993821,13cyclictest272-21dbus-daemon23:40:003
388736499388,5cyclictest131rcu_preempt21:30:012
3887364993828,7cyclictest3901680-21sed23:45:022
3887364993828,7cyclictest3901680-21sed23:45:022
3887364993826,8cyclictest235-21systemd-journal23:00:002
3887364993825,9cyclictest235-21systemd-journal23:50:002
3887364993823,10cyclictest235-21systemd-journal20:00:012
3887364993823,10cyclictest235-21systemd-journal20:00:012
388736499381,33cyclictest30-21ksoftirqd/219:25:022
388736099381,3cyclictest131rcu_preempt22:00:001
388735599389,4cyclictest131rcu_preempt20:45:000
3887355993821,11cyclictest1-21systemd23:35:010
3887355993821,11cyclictest1-21systemd23:35:010
3887355993820,12cyclictest272-21dbus-daemon21:30:010
388735599381,33cyclictest0-21swapper/022:15:000
3887369993724,10cyclictest3900670-21wc23:25:013
388736999372,3cyclictest131rcu_preempt22:40:013
3887369993723,11cyclictest37-21ksoftirqd/321:55:023
3887369993723,11cyclictest37-21ksoftirqd/321:55:023
3887364993728,5cyclictest3893006-21sendmail21:00:002
3887364993727,8cyclictest30-21ksoftirqd/221:25:002
3887364993725,9cyclictest30-21ksoftirqd/219:50:002
388736499372,31cyclictest30-21ksoftirqd/219:20:012
3887364993723,10cyclictest30-21ksoftirqd/221:45:022
3887364993718,15cyclictest235-21systemd-journal19:39:592
3887364993716,15cyclictest272-21dbus-daemon20:35:012
388736099375,3cyclictest131rcu_preempt21:45:001
3887360993728,7cyclictest23-21ksoftirqd/123:45:011
3887360993728,7cyclictest23-21ksoftirqd/123:45:011
3887360993725,9cyclictest23-21ksoftirqd/120:34:411
3887360993723,10cyclictest272-21dbus-daemon20:15:001
388735599374,9cyclictest12-21ksoftirqd/022:25:020
3887355993724,10cyclictest3892721-21sshd20:51:420
388736999363,27cyclictest306-21in:imuxsock20:15:013
3887369993624,8cyclictest306-21in:imuxsock19:40:003
3887369993622,10cyclictest37-21ksoftirqd/322:55:013
388736499365,4cyclictest131rcu_preempt21:10:022
3887364993628,5cyclictest3901066-21cron23:35:002
3887364993628,5cyclictest3901066-21cron23:35:002
3887364993625,8cyclictest30-21ksoftirqd/223:55:022
3887364993623,9cyclictest272-21dbus-daemon22:25:012
3887364993622,10cyclictest235-21systemd-journal23:24:422
3887364993621,11cyclictest272-21dbus-daemon20:45:002
388736499361,3cyclictest131rcu_preempt21:50:022
388736499361,3cyclictest131rcu_preempt21:50:022
3887360993628,4cyclictest3892739-21cron20:55:001
3887360993626,4cyclictest306-21in:imuxsock19:30:011
3887360993626,4cyclictest306-21in:imuxsock19:30:011
3887369993525,8cyclictest37-21ksoftirqd/322:15:423
3887369993524,8cyclictest37-21ksoftirqd/319:26:183
3887369993524,8cyclictest37-21ksoftirqd/319:26:173
3887369993522,10cyclictest37-21ksoftirqd/320:10:003
3887369993522,10cyclictest37-21ksoftirqd/319:21:183
3887369993521,8cyclictest3899628-21wc23:05:023
3887369993521,10cyclictest3903616-21cat00:20:033
3887369993520,11cyclictest37-21ksoftirqd/323:30:003
3887369993519,12cyclictest272-21dbus-daemon22:45:003
388736499358,4cyclictest131rcu_preempt20:40:032
388736499358,4cyclictest131rcu_preempt20:40:022
3887364993525,7cyclictest3896057-21cat21:55:022
3887364993525,7cyclictest3896057-21cat21:55:012
3887364993523,9cyclictest30-21ksoftirqd/200:35:022
3887364993523,9cyclictest30-21ksoftirqd/200:35:022
3887364993519,13cyclictest3890620-21sed20:10:012
3887364993518,13cyclictest272-21dbus-daemon21:15:012
3887364993516,14cyclictest272-21dbus-daemon21:35:012
3887364993516,14cyclictest272-21dbus-daemon21:35:002
3887360993527,5cyclictest3892109-21latency_hist20:40:021
3887360993527,5cyclictest3892109-21latency_hist20:40:021
3887360993525,5cyclictest3900571-21munin-plugin-st23:25:001
3887360993511,6cyclictest131rcu_preempt20:10:011
388735599352,4cyclictest131rcu_preempt22:55:010
388736999349,5cyclictest131rcu_preempt22:05:023
388736999349,5cyclictest131rcu_preempt22:05:023
3887369993423,7cyclictest3893007-21sh21:00:003
3887369993422,9cyclictest341irq_work/320:00:023
3887369993422,9cyclictest341irq_work/320:00:023
3887369993421,9cyclictest3890316-21sed20:05:013
3887369993421,10cyclictest37-21ksoftirqd/321:50:013
3887369993421,10cyclictest37-21ksoftirqd/321:50:013
3887369993418,12cyclictest3896609-21cat22:05:023
3887369993414,16cyclictest272-21dbus-daemon22:25:013
388736499344,27cyclictest3904477-21modprobe00:40:002
3887364993424,6cyclictest3887688-21latency_hist19:15:022
3887364993415,14cyclictest272-21dbus-daemon21:20:012
3887364993415,14cyclictest272-21dbus-daemon21:20:012
388736099343,3cyclictest131rcu_preempt23:05:011
3887360993423,8cyclictest23-21ksoftirqd/119:45:021
3887360993423,8cyclictest23-21ksoftirqd/119:45:011
3887355993424,6cyclictest3889876-21cron20:00:000
3887355993424,6cyclictest3889876-21cron20:00:000
388736999334,5cyclictest131rcu_preempt23:15:023
3887369993322,8cyclictest37-21ksoftirqd/300:11:413
3887369993321,9cyclictest37-21ksoftirqd/320:30:023
3887369993321,9cyclictest37-21ksoftirqd/320:30:023
3887369993319,10cyclictest235-21systemd-journal20:37:323
3887369993319,10cyclictest235-21systemd-journal20:37:313
3887364993325,5cyclictest3903052-21latency_hist00:10:022
388736499332,4cyclictest131rcu_preempt23:10:012
3887364993322,7cyclictest1-21systemd22:15:002
3887364993320,10cyclictest235-21systemd-journal19:50:432
388736099331,11cyclictest131rcu_preempt23:50:001
3887369993225,4cyclictest3901864-21/usr/sbin/munin23:50:003
3887369993224,5cyclictest3887812-21modprobe19:20:003
3887369993219,6cyclictest306-21in:imuxsock00:30:003
3887369993217,11cyclictest0-21swapper/321:20:003
3887369993217,11cyclictest0-21swapper/321:20:003
3887364993226,4cyclictest3900827-21/usr/sbin/munin23:30:012
3887364993221,8cyclictest30-21ksoftirqd/200:15:012
3887364993220,9cyclictest0-21swapper/223:05:012
388736499321,3cyclictest131rcu_preempt21:05:012
388736099321,4cyclictest131rcu_preempt23:30:021
3887369993122,5cyclictest37-21ksoftirqd/323:45:013
3887369993122,5cyclictest37-21ksoftirqd/323:45:013
388736999312,22cyclictest3898744-21cut22:50:003
3887369993118,10cyclictest3887377-21cut19:10:023
388736499302,3cyclictest131rcu_preempt19:45:012
388736499302,3cyclictest131rcu_preempt19:45:012
3887364993018,9cyclictest30-21ksoftirqd/220:50:002
388736499301,4cyclictest131rcu_preempt22:45:012
3887364992921,5cyclictest3892815-21latency_hist20:55:022
3887364992920,4cyclictest306-21in:imuxsock20:20:422
3887364992912,12cyclictest3890307-21cat20:05:022
3887360992618,6cyclictest23-21ksoftirqd/119:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional