You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 11:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Nov 20, 2025 00:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3515991996717,16cyclictest3525330-21latency_hist22:10:022
3515991996517,14cyclictest3520508-21systemd-run20:40:002
3515991996515,13cyclictest3518336-21date19:55:022
3515991996422,13cyclictest1-21systemd00:15:022
3515991996422,13cyclictest1-21systemd00:15:012
3515991996416,14cyclictest3532256-21munin-run00:25:002
3515991996412,17cyclictest3523020-21latency_hist21:25:032
3515991996316,16cyclictest3531520-21sed00:10:022
3515991996315,15cyclictest3528591-21systemctl23:15:002
3515991996314,16cyclictest3525431-21munin-run22:15:012
3515991996314,15cyclictest3516509-21cut19:20:012
3515991996114,14cyclictest3529372-21munin-run23:30:012
3515991996014,15cyclictest3517451-21cat19:40:012
3515991995941,12cyclictest30-21ksoftirqd/223:24:592
3515991995940,14cyclictest235-21systemd-journal23:20:012
3515991995914,17cyclictest3519624-21sed20:20:012
3515987995637,14cyclictest1-21systemd22:44:201
3515996995536,14cyclictest1-21systemd20:10:003
3515996995435,14cyclictest1-21systemd20:30:013
3515996995435,14cyclictest1-21systemd20:30:003
3515991995312,10cyclictest272-21dbus-daemon23:10:012
3515987995336,13cyclictest1-21systemd23:15:001
3515991995239,9cyclictest48-21kcompactd021:56:342
3515991995236,13cyclictest3529595-21sshd23:31:262
3515991995236,12cyclictest3526692-21sshd22:38:282
3515991995233,15cyclictest3530658-21munin-run23:54:592
3515991995137,11cyclictest3523056-21sshd21:25:272
3515991995135,13cyclictest3520987-21sshd20:49:272
3515991995134,13cyclictest3521237-21sshd20:54:282
351598799512,22cyclictest3516891-21kworker/u8:019:45:021
351598799512,22cyclictest3516891-21kworker/u8:019:45:021
351598299512,15cyclictest3523092-21munin-run21:29:590
351599699507,15cyclictest131rcu_preempt19:45:033
351599699507,15cyclictest131rcu_preempt19:45:023
3515996995011,7cyclictest131rcu_preempt23:00:003
3515991995041,5cyclictest3526992-21apt-get22:44:202
3515991995037,9cyclictest3517660-21munin-run19:45:022
3515991995037,9cyclictest3517660-21munin-run19:45:012
3515991995035,12cyclictest3520716-21sshd20:40:272
3515991995035,12cyclictest3520716-21sshd20:40:262
3515991995034,13cyclictest3525635-21sshd22:15:272
3515991995029,15cyclictest3516669-21modprobe19:25:012
3515982995015,29cyclictest272-21dbus-daemon23:00:010
3515991994933,13cyclictest3526185-21sshd22:27:282
3515982994933,11cyclictest3528829-21cron23:20:000
3515996994832,12cyclictest235-21systemd-journal21:15:013
3515996994827,17cyclictest1-21systemd00:00:003
3515991994832,12cyclictest1-21systemd22:55:022
3515991994831,13cyclictest363-21snmpd23:57:432
3515987994811,12cyclictest23-21ksoftirqd/100:35:001
3515982994839,6cyclictest3515983-21latency_hist19:10:010
3515982994830,13cyclictest272-21dbus-daemon20:30:010
3515982994830,13cyclictest272-21dbus-daemon20:30:000
3515996994732,12cyclictest37-21ksoftirqd/322:55:003
3515996994723,17cyclictest272-21dbus-daemon23:40:013
3515991994736,8cyclictest3522266-21sshd21:11:042
3515991994735,9cyclictest3524363-21sshd21:54:272
3515991994734,10cyclictest3532474-21sshd00:26:282
3515991994734,10cyclictest3532474-21sshd00:26:272
3515991994734,10cyclictest3523564-21sshd21:36:082
3515991994734,10cyclictest3523564-21sshd21:36:072
3515991994733,10cyclictest3529859-21sshd23:39:272
3515991994732,12cyclictest3521471-21sshd20:55:282
3515991994732,12cyclictest3521471-21sshd20:55:272
3515991994730,12cyclictest363-21snmpd21:19:592
3515991994728,13cyclictest272-21dbus-daemon20:10:012
3515991994727,16cyclictest3527834-21true23:00:002
3515987994735,7cyclictest3522571-21modprobe21:20:001
351599699462,35cyclictest235-21systemd-journal21:55:033
3515991994634,8cyclictest48-21kcompactd022:49:152
3515991994633,9cyclictest48-21kcompactd020:28:202
3515991994633,9cyclictest48-21kcompactd020:28:192
3515991994632,10cyclictest3520220-21munin-run20:35:002
3515991994631,11cyclictest3517149-21munin-run19:35:012
3515991994628,14cyclictest363-21snmpd00:16:502
3515991994627,14cyclictest3518379-21cron20:00:012
3515987994630,12cyclictest1-21systemd00:15:011
3515987994630,12cyclictest1-21systemd00:15:001
3515987994627,15cyclictest3532241-21sshd00:24:281
3515996994528,13cyclictest37-21ksoftirqd/319:40:003
3515991994533,9cyclictest48-21kcompactd021:06:382
3515991994533,9cyclictest3524092-21sshd21:45:282
3515991994532,9cyclictest363-21snmpd19:47:442
3515991994532,10cyclictest3531177-21sshd00:03:272
3515991994531,11cyclictest3530119-21sshd23:40:282
351598299454,3cyclictest131rcu_preempt21:31:210
3515982994531,10cyclictest272-21dbus-daemon22:40:020
3515982994519,8cyclictest131rcu_preempt22:44:200
351598299451,3cyclictest131rcu_preempt19:30:010
3515996994433,8cyclictest235-21systemd-journal00:30:013
3515996994433,8cyclictest235-21systemd-journal00:30:013
3515996994425,14cyclictest272-21dbus-daemon21:40:013
3515996994425,14cyclictest272-21dbus-daemon21:40:013
3515996994419,20cyclictest361rcuc/321:30:013
3515991994434,7cyclictest3519689-21sshd20:22:282
3515991994433,8cyclictest48-21kcompactd021:01:382
3515991994431,9cyclictest3532730-21sshd00:32:272
3515991994425,15cyclictest363-21snmpd22:21:492
3515991994420,20cyclictest3533021-21systemctl00:40:002
3515987994429,11cyclictest3525965-21systemd-run22:25:011
3515987994427,13cyclictest235-21systemd-journal20:00:001
351598299448,25cyclictest151rcuc/019:50:010
3515982994432,8cyclictest3527280-21cron22:50:010
3515996994332,7cyclictest306-21in:imuxsock22:15:003
3515996994325,12cyclictest235-21systemd-journal22:35:023
351599699431,25cyclictest3530162-21kworker/u8:300:10:013
3515991994331,9cyclictest363-21snmpd19:27:202
3515991994328,11cyclictest363-21snmpd23:01:322
3515991994328,10cyclictest363-21snmpd20:00:472
3515982994325,12cyclictest272-21dbus-daemon22:25:020
351598299431,38cyclictest48-21kcompactd020:50:000
351599699426,5cyclictest131rcu_preempt21:10:013
3515996994228,8cyclictest306-21in:imuxsock20:45:013
3515996994228,8cyclictest306-21in:imuxsock20:45:013
3515996994226,13cyclictest1-21systemd22:00:003
351599699421,5cyclictest131rcu_preempt00:34:593
3515991994235,4cyclictest3519153-21sshd20:13:272
3515991994234,4cyclictest3523315-21sshd21:31:272
3515991994231,8cyclictest48-21kcompactd022:30:242
3515991994230,9cyclictest48-21kcompactd023:45:182
3515991994230,9cyclictest363-21snmpd22:01:122
3515987994231,6cyclictest3524366-21cron21:55:011
3515987994229,10cyclictest235-21systemd-journal23:15:021
3515982994235,4cyclictest16950irq/224-800000021:15:010
3515982994229,9cyclictest235-21systemd-journal21:55:010
3515982994229,10cyclictest235-21systemd-journal20:02:280
3515982994228,10cyclictest12-21ksoftirqd/023:15:020
3515982994227,11cyclictest3523623-21sendmail-msp21:40:000
3515982994227,11cyclictest3523623-21sendmail-msp21:40:000
3515982994224,5cyclictest12-21ksoftirqd/000:00:010
351598299421,36cyclictest3522807-21sshd21:21:060
3515996994127,10cyclictest3529092-21sshd23:21:273
3515991994128,10cyclictest363-21snmpd21:40:532
3515987994130,7cyclictest3523845-21sshd21:41:071
3515987994129,9cyclictest306-21in:imuxsock21:34:271
3515987994124,13cyclictest3531981-21systemctl00:20:001
3515987994121,14cyclictest272-21dbus-daemon19:40:011
3515982994128,9cyclictest12-21ksoftirqd/000:40:010
3515982994128,10cyclictest408-21lldpd19:35:470
351598299412,34cyclictest3518400-21modprobe20:00:000
3515996994029,8cyclictest3528300-21sshd23:06:263
3515996994028,7cyclictest3531187-21cron00:05:013
3515996994027,9cyclictest306-21in:imuxsock21:20:003
3515996994026,11cyclictest37-21ksoftirqd/320:20:003
3515991994024,12cyclictest3516150-21cron19:15:002
3515987994030,6cyclictest306-21in:imuxsock20:55:011
3515987994025,12cyclictest3520052-21sed20:30:021
3515987994025,12cyclictest3520052-21sed20:30:011
3515987994023,11cyclictest272-21dbus-daemon22:50:001
3515987994022,15cyclictest1-21systemd23:10:011
3515987994020,15cyclictest1-21systemd23:00:011
351598799401,7cyclictest131rcu_preempt19:25:001
351598799401,3cyclictest131rcu_preempt22:50:031
351598299403,31cyclictest0-21swapper/019:35:020
3515982994030,6cyclictest19050irq/34-mmc121:49:590
3515982994030,6cyclictest12-21ksoftirqd/023:10:000
3515982994026,10cyclictest235-21systemd-journal00:34:590
3515982994025,9cyclictest306-21in:imuxsock21:00:010
3515982994025,9cyclictest306-21in:imuxsock21:00:000
351598299401,5cyclictest3522062-21munin-run21:10:020
351598299401,4cyclictest0-21swapper/000:30:010
351598299401,4cyclictest0-21swapper/000:30:000
3515982994012,6cyclictest12-21ksoftirqd/023:42:090
3515982994012,18cyclictest1-21systemd23:04:280
351599699398,6cyclictest131rcu_preempt23:50:003
3515996993932,4cyclictest371ksoftirqd/300:20:013
3515996993929,7cyclictest37-21ksoftirqd/321:05:013
3515996993927,8cyclictest3527043-21/usr/sbin/munin22:45:013
3515996993924,12cyclictest37-21ksoftirqd/320:15:023
3515996993923,12cyclictest37-21ksoftirqd/320:35:563
3515996993921,14cyclictest272-21dbus-daemon21:45:013
3515987993930,6cyclictest23-21ksoftirqd/100:40:021
3515987993930,5cyclictest23-21ksoftirqd/100:00:011
3515987993925,11cyclictest23-21ksoftirqd/122:19:521
3515987993925,10cyclictest272-21dbus-daemon21:05:011
3515987993925,10cyclictest1-21systemd22:15:001
3515987993924,11cyclictest1-21systemd22:40:011
3515982993927,9cyclictest3522536-21sshd21:17:270
3515982993923,10cyclictest272-21dbus-daemon21:45:010
3515982993920,16cyclictest3525383-21sshd22:11:120
3515996993826,8cyclictest37-21ksoftirqd/319:15:013
3515996993824,11cyclictest37-21ksoftirqd/320:50:023
3515996993821,12cyclictest37-21ksoftirqd/323:20:003
3515996993819,13cyclictest235-21systemd-journal00:40:013
351598799383,3cyclictest131rcu_preempt19:30:011
3515987993827,7cyclictest3522294-21sh21:15:001
3515982993833,3cyclictest16950irq/224-800000023:31:230
3515982993830,5cyclictest19050irq/34-mmc120:52:510
3515982993827,9cyclictest12-21ksoftirqd/020:34:590
3515982993827,8cyclictest3521752-21sshd21:00:270
3515982993827,8cyclictest12-21ksoftirqd/022:00:020
3515982993827,6cyclictest306-21in:imuxsock23:25:010
3515982993826,8cyclictest12-21ksoftirqd/022:55:020
3515982993826,8cyclictest0-21swapper/023:30:030
3515982993825,8cyclictest1-21systemd00:25:010
3515982993825,10cyclictest12-21ksoftirqd/022:05:020
3515982993824,10cyclictest408-21lldpd23:54:520
3515982993822,11cyclictest408-21lldpd00:05:520
351598299381,32cyclictest3525323-21cat22:10:030
351598299381,32cyclictest0-21swapper/019:25:000
351599699377,7cyclictest3520333-21latency_hist20:35:013
3515996993728,6cyclictest37-21ksoftirqd/319:55:023
351599699372,7cyclictest3515976-21cyclictest19:57:203
351599699371,5cyclictest131rcu_preempt23:30:033
3515996993712,7cyclictest131rcu_preempt22:10:013
351598799374,3cyclictest131rcu_preempt21:21:271
3515987993728,6cyclictest23-21ksoftirqd/120:10:011
3515987993725,9cyclictest23-21ksoftirqd/123:40:001
3515987993725,9cyclictest23-21ksoftirqd/119:50:001
3515987993724,9cyclictest23-21ksoftirqd/120:20:001
3515987993722,11cyclictest3520780-21sed20:45:021
3515987993722,11cyclictest3520780-21sed20:45:011
351598799371,4cyclictest131rcu_preempt19:34:591
3515982993731,3cyclictest0-21swapper/020:10:000
3515982993726,8cyclictest12-21ksoftirqd/019:20:020
351598299372,31cyclictest3519176-21munin-run20:15:000
351599699365,8cyclictest37-21ksoftirqd/321:46:503
3515996993628,6cyclictest37-21ksoftirqd/321:25:023
351599699362,6cyclictest131rcu_preempt23:55:003
3515996993624,9cyclictest37-21ksoftirqd/323:35:013
3515991993621,11cyclictest3516104-21cat19:10:032
351598799365,4cyclictest131rcu_preempt22:01:271
3515987993625,8cyclictest3518110-21sshd19:51:261
3515987993624,8cyclictest272-21dbus-daemon21:59:591
3515987993621,12cyclictest23-21ksoftirqd/121:40:011
3515987993621,12cyclictest23-21ksoftirqd/121:40:001
3515987993619,12cyclictest272-21dbus-daemon23:25:011
3515987993618,14cyclictest235-21systemd-journal23:35:011
351598799361,3cyclictest131rcu_preempt23:45:001
3515987993613,6cyclictest131rcu_preempt20:15:011
3515982993628,5cyclictest19050irq/34-mmc120:25:000
3515982993626,5cyclictest0-21swapper/022:17:270
351598299362,31cyclictest3518117-21sshd19:52:270
3515982993620,13cyclictest3531691-21sshd00:14:280
3515982993620,13cyclictest3531691-21sshd00:14:270
3515982993620,12cyclictest235-21systemd-journal19:15:000
3515996993526,5cyclictest3517952-21latency_hist19:50:013
3515996993521,10cyclictest37-21ksoftirqd/320:50:583
3515996993520,10cyclictest306-21in:imuxsock23:05:013
3515996993519,12cyclictest1-21systemd22:25:023
3515996993517,13cyclictest235-21systemd-journal22:00:013
351599699351,5cyclictest131rcu_preempt21:35:013
3515987993528,4cyclictest1-21systemd22:10:001
3515987993526,7cyclictest23-21ksoftirqd/123:53:261
3515987993525,7cyclictest3520217-21modprobe20:35:001
3515987993525,6cyclictest0-21swapper/119:10:011
3515987993524,8cyclictest3518682-21pluginstate20:05:011
3515987993523,9cyclictest23-21ksoftirqd/120:40:001
351598299357,25cyclictest0-21swapper/000:17:400
3515982993529,4cyclictest0-21swapper/022:31:340
3515982993523,8cyclictest235-21systemd-journal00:05:020
3515982993520,11cyclictest344-21lldpd22:29:150
351598299351,30cyclictest0-21swapper/019:45:010
351598299351,30cyclictest0-21swapper/019:45:000
351598299351,29cyclictest3520599-21grep20:40:020
351599699347,21cyclictest3527281-21modprobe22:50:013
351599699342,6cyclictest131rcu_preempt19:20:003
3515996993426,5cyclictest3521522-21sendmail21:00:013
3515996993426,5cyclictest3521522-21sendmail21:00:003
3515996993425,5cyclictest3516173-21kworker/3:219:20:133
3515996993424,7cyclictest3531667-21sshd00:10:283
3515996993424,7cyclictest3531667-21sshd00:10:283
351598799344,3cyclictest131rcu_preempt23:00:261
351598799344,3cyclictest131rcu_preempt22:30:281
3515987993426,5cyclictest3524161-21latency_hist21:50:021
3515987993423,8cyclictest3532556-21sed00:30:021
3515987993423,8cyclictest3532556-21sed00:30:011
3515987993422,8cyclictest272-21dbus-daemon00:10:001
3515987993417,13cyclictest3522026-21cron21:10:001
351598799341,3cyclictest131rcu_preempt19:19:591
351598799341,3cyclictest131rcu_preempt19:15:011
351598299342,5cyclictest235-21systemd-journal20:18:340
351598299341,30cyclictest0-21swapper/023:48:480
351598299341,30cyclictest0-21swapper/023:35:270
351599699337,6cyclictest131rcu_preempt23:45:003
351599699334,6cyclictest131rcu_preempt23:15:013
351599699333,23cyclictest3526709-21modprobe22:40:003
351599699332,5cyclictest131rcu_preempt19:35:013
351599699332,5cyclictest131rcu_preempt19:30:013
3515996993323,6cyclictest37-21ksoftirqd/300:23:543
351598799334,4cyclictest131rcu_preempt23:30:001
3515987993326,4cyclictest3519712-21latency_hist20:25:021
351598799331,3cyclictest131rcu_preempt21:25:181
3515996993220,10cyclictest37-21ksoftirqd/320:05:003
3515987993224,5cyclictest3521672-21latency_hist21:00:031
3515987993224,5cyclictest3521672-21latency_hist21:00:021
351598799322,3cyclictest131rcu_preempt22:28:281
3515987993223,7cyclictest23-21ksoftirqd/123:45:271
3515987993222,7cyclictest23-21ksoftirqd/120:50:001
3515987993221,9cyclictest23-21ksoftirqd/100:05:011
3515982993226,4cyclictest0-21swapper/020:40:520
3515982993226,4cyclictest0-21swapper/020:40:520
351599699316,4cyclictest3515976-21cyclictest20:21:063
3515996993120,7cyclictest37-21ksoftirqd/319:10:013
3515996993112,4cyclictest37-21ksoftirqd/322:25:273
3515996993019,9cyclictest37-21ksoftirqd/322:17:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional