You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 22:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Dec 02, 2025 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
86870699791,74cyclictest0-21swapper/010:15:020
868706997874,2cyclictest0-21swapper/011:35:020
868711997738,14cyclictest1-21systemd08:20:001
868706997570,2cyclictest0-21swapper/010:50:010
868716997424,10cyclictest306-21in:imuxsock07:10:192
868716996824,10cyclictest30-21ksoftirqd/210:05:022
868716996824,10cyclictest30-21ksoftirqd/210:05:012
868706996717,45cyclictest272-21dbus-daemon07:50:000
868706996616,17cyclictest12-21ksoftirqd/008:45:010
868716996526,13cyclictest30-21ksoftirqd/207:20:022
868716996323,10cyclictest30-21ksoftirqd/212:02:102
868716996323,10cyclictest30-21ksoftirqd/212:02:102
868716996321,4cyclictest874631-21latency_hist09:05:022
868716996132,26cyclictest30-21ksoftirqd/210:30:032
868716996132,26cyclictest30-21ksoftirqd/210:30:032
868716996024,13cyclictest30-21ksoftirqd/210:24:592
868716996022,10cyclictest30-21ksoftirqd/209:15:002
86870699605,12cyclictest870483-21modprobe07:45:000
86870699601,55cyclictest0-21swapper/010:20:030
868716995921,7cyclictest30-21ksoftirqd/210:15:032
868716995920,5cyclictest874444-21latency_hist09:00:022
868716995919,10cyclictest30-21ksoftirqd/209:43:192
868706995952,4cyclictest151rcuc/009:45:020
868716995721,5cyclictest883502-21latency_hist11:55:012
868716995618,8cyclictest885070-21sshd12:23:192
868706995639,12cyclictest0-21swapper/007:20:000
868716995521,12cyclictest235-21systemd-journal12:40:002
868706995535,11cyclictest306-21in:imuxsock12:40:000
868706995531,16cyclictest235-21systemd-journal11:00:000
868716995419,4cyclictest306-21in:imuxsock08:20:012
868716995322,4cyclictest869948-21latency_hist07:35:022
868716995318,10cyclictest30-21ksoftirqd/209:35:032
868716995318,10cyclictest30-21ksoftirqd/209:35:022
86871699531,37cyclictest873043-21kworker/u8:308:50:012
868716995235,13cyclictest235-21systemd-journal11:40:002
868716995225,6cyclictest879430-21latency_hist10:35:012
868720995135,12cyclictest1-21systemd07:35:003
868711995137,11cyclictest235-21systemd-journal07:19:591
868706995125,18cyclictest1-21systemd09:40:000
868720995034,12cyclictest235-21systemd-journal09:13:203
868716995021,8cyclictest30-21ksoftirqd/207:50:022
86871199506,4cyclictest131rcu_preempt07:59:591
868706995022,19cyclictest272-21dbus-daemon07:25:000
86872099491,3cyclictest131rcu_preempt11:14:593
86871199492,3cyclictest131rcu_preempt10:40:011
86871199492,3cyclictest131rcu_preempt10:40:001
86870699499,25cyclictest882676-21modprobe11:40:010
86870699494,34cyclictest877765-21cut10:05:010
86870699494,34cyclictest877765-21cut10:05:010
868706994930,13cyclictest272-21dbus-daemon11:50:010
868706994930,13cyclictest272-21dbus-daemon11:50:000
86872099484,4cyclictest131rcu_preempt12:10:003
86872099483,4cyclictest131rcu_preempt09:37:183
868720994830,14cyclictest272-21dbus-daemon08:05:003
86871199483,4cyclictest131rcu_preempt07:15:031
868711994833,12cyclictest1-21systemd09:35:011
868711994833,12cyclictest1-21systemd09:35:001
86870699486,29cyclictest883708-21modprobe12:00:000
868706994840,2cyclictest0-21swapper/009:45:030
868706994840,2cyclictest0-21swapper/009:45:020
868706994825,5cyclictest12-21ksoftirqd/007:55:030
868706994825,15cyclictest307-21in:imklog09:55:000
868706994814,20cyclictest0-21swapper/011:10:020
86872099475,4cyclictest131rcu_preempt09:35:023
86872099475,4cyclictest131rcu_preempt09:35:023
868720994735,7cyclictest877470-21cron09:59:593
86871199471,4cyclictest131rcu_preempt12:09:591
86871199471,4cyclictest131rcu_preempt07:55:031
86871199471,3cyclictest131rcu_preempt08:40:011
86871199471,29cyclictest877444-21kworker/u8:010:20:021
868706994729,9cyclictest884041-21sh12:05:010
868706994729,9cyclictest884041-21sh12:05:000
868706994726,15cyclictest12-21ksoftirqd/007:35:010
868706994724,18cyclictest363-21snmpd11:30:000
86870699471,13cyclictest871517-21munin-run08:05:000
86872099463,4cyclictest131rcu_preempt07:40:003
86872099462,4cyclictest131rcu_preempt12:30:023
86871199463,4cyclictest131rcu_preempt07:40:011
868711994632,10cyclictest235-21systemd-journal12:11:201
868711994632,10cyclictest235-21systemd-journal10:55:011
868711994631,11cyclictest272-21dbus-daemon10:30:011
868711994631,11cyclictest272-21dbus-daemon10:30:001
86871199462,3cyclictest131rcu_preempt12:20:011
86871199462,3cyclictest131rcu_preempt12:20:001
86871199462,3cyclictest131rcu_preempt08:50:011
86871199461,4cyclictest131rcu_preempt12:40:001
868706994624,17cyclictest235-21systemd-journal10:55:010
868706994620,7cyclictest12-21ksoftirqd/011:54:160
86872099455,3cyclictest131rcu_preempt08:52:193
86872099453,4cyclictest131rcu_preempt10:22:193
86872099453,4cyclictest131rcu_preempt08:45:013
86872099453,3cyclictest131rcu_preempt08:00:003
86872099453,3cyclictest131rcu_preempt07:47:183
868716994538,5cyclictest30-21ksoftirqd/207:20:032
868716994534,7cyclictest882176-21munin-plugin-st11:30:002
86871199453,3cyclictest131rcu_preempt10:41:191
86871199453,3cyclictest131rcu_preempt09:54:591
86871199453,3cyclictest131rcu_preempt07:10:011
868706994538,4cyclictest16950irq/224-800000007:15:010
868706994530,11cyclictest885270-21latency_hist12:25:020
86870699452,5cyclictest131rcu_preempt11:45:010
868706994518,21cyclictest12-21ksoftirqd/012:15:000
86872099446,3cyclictest131rcu_preempt09:50:033
86872099446,3cyclictest131rcu_preempt09:50:023
86872099444,4cyclictest131rcu_preempt07:55:033
86872099443,4cyclictest131rcu_preempt07:20:013
86872099443,3cyclictest131rcu_preempt12:05:013
86872099443,3cyclictest131rcu_preempt12:05:003
86872099443,3cyclictest131rcu_preempt08:25:023
86872099443,3cyclictest131rcu_preempt08:25:013
868716994431,9cyclictest30-21ksoftirqd/208:45:002
868711994432,8cyclictest1-21systemd09:05:011
868711994430,10cyclictest235-21systemd-journal10:50:001
868711994428,12cyclictest1-21systemd09:50:011
868711994428,12cyclictest1-21systemd09:50:001
868706994428,12cyclictest363-21snmpd12:30:120
868706994428,12cyclictest363-21snmpd12:30:120
86872099435,3cyclictest131rcu_preempt08:19:193
86872099434,4cyclictest131rcu_preempt11:00:013
86872099434,3cyclictest131rcu_preempt12:35:023
86872099434,3cyclictest131rcu_preempt12:35:023
86872099433,4cyclictest131rcu_preempt11:51:083
86872099433,3cyclictest131rcu_preempt09:20:023
86872099433,3cyclictest131rcu_preempt09:20:013
868720994329,10cyclictest884537-21cron12:15:003
868720994312,5cyclictest131rcu_preempt09:02:013
868711994330,10cyclictest1-21systemd09:20:011
868711994330,10cyclictest1-21systemd09:20:001
868711994329,10cyclictest235-21systemd-journal09:36:191
868711994327,12cyclictest1-21systemd10:05:011
868711994327,12cyclictest1-21systemd10:05:001
86871199432,3cyclictest131rcu_preempt08:45:011
86871199432,3cyclictest131rcu_preempt07:25:031
86871199431,4cyclictest131rcu_preempt12:00:001
86871199431,4cyclictest131rcu_preempt11:25:021
86871199431,4cyclictest131rcu_preempt11:25:011
86871199431,4cyclictest131rcu_preempt07:45:001
86871199431,3cyclictest131rcu_preempt09:10:001
868706994329,9cyclictest306-21in:imuxsock09:15:000
86872099425,3cyclictest131rcu_preempt10:02:193
86872099425,3cyclictest131rcu_preempt10:02:193
86872099425,3cyclictest131rcu_preempt07:45:013
86872099424,3cyclictest131rcu_preempt10:41:203
86872099424,3cyclictest131rcu_preempt09:10:003
86872099423,4cyclictest131rcu_preempt11:40:033
86872099423,4cyclictest131rcu_preempt08:10:243
86872099422,3cyclictest131rcu_preempt10:30:013
86872099422,3cyclictest131rcu_preempt10:30:003
868720994214,14cyclictest879570-21cat10:40:013
868720994214,14cyclictest879570-21cat10:40:013
868716994231,7cyclictest30-21ksoftirqd/211:20:012
868716994228,10cyclictest1-21systemd11:45:012
868716994226,12cyclictest1-21systemd10:15:002
86871199425,4cyclictest131rcu_preempt07:50:001
868711994230,9cyclictest871483-21sshd08:00:201
86871199422,3cyclictest131rcu_preempt11:15:021
86871199421,3cyclictest131rcu_preempt12:30:021
86871199421,3cyclictest131rcu_preempt12:25:001
86871199421,3cyclictest131rcu_preempt11:40:011
86871199421,3cyclictest131rcu_preempt11:05:021
86871199421,3cyclictest131rcu_preempt09:56:191
86870699426,22cyclictest872276-21modprobe08:20:000
868706994228,8cyclictest306-21in:imuxsock12:10:000
86870699422,7cyclictest131rcu_preempt10:00:000
868706994227,10cyclictest306-21in:imuxsock11:10:010
868706994226,9cyclictest875415-21sendmail09:20:010
868706994226,9cyclictest875415-21sendmail09:20:000
86870699421,5cyclictest872819-21munin-run08:30:000
868706994210,26cyclictest235-21systemd-journal12:20:010
868706994210,26cyclictest235-21systemd-journal12:20:000
868720994132,4cyclictest306-21in:imuxsock10:15:013
868720994124,13cyclictest272-21dbus-daemon11:30:013
86872099412,3cyclictest131rcu_preempt12:16:133
86872099412,3cyclictest131rcu_preempt12:16:123
86872099411,3cyclictest131rcu_preempt12:00:023
868716994126,10cyclictest272-21dbus-daemon08:00:012
868716994124,13cyclictest272-21dbus-daemon11:50:012
868716994124,13cyclictest272-21dbus-daemon11:50:012
86871199414,3cyclictest131rcu_preempt12:35:021
86871199414,3cyclictest131rcu_preempt12:35:011
86871199413,3cyclictest131rcu_preempt11:30:191
868711994128,10cyclictest869986-21cat07:35:011
868711994128,10cyclictest23-21ksoftirqd/108:55:001
868711994127,11cyclictest1-21systemd09:30:001
868711994126,11cyclictest1-21systemd11:55:011
86871199412,4cyclictest131rcu_preempt11:20:021
86871199412,3cyclictest131rcu_preempt10:11:191
86871199412,3cyclictest131rcu_preempt08:33:191
86871199411,4cyclictest131rcu_preempt08:15:001
86870699414,24cyclictest235-21systemd-journal10:40:010
86870699414,24cyclictest235-21systemd-journal10:40:000
868706994131,7cyclictest872023-21munin-plugin-st08:14:590
86870699412,35cyclictest873051-21sshd08:34:190
868716994026,12cyclictest30-21ksoftirqd/208:55:032
868716994023,13cyclictest272-21dbus-daemon09:30:012
868711994029,8cyclictest882158-21sshd11:28:201
868711994026,9cyclictest306-21in:imuxsock07:30:011
86871199402,3cyclictest131rcu_preempt08:55:221
86871199401,3cyclictest131rcu_preempt11:45:001
86871199401,3cyclictest131rcu_preempt08:25:021
86871199401,3cyclictest131rcu_preempt08:25:021
86870699407,2cyclictest0-21swapper/011:20:000
868706994024,10cyclictest870207-21sendmail07:40:000
868720993924,11cyclictest272-21dbus-daemon08:50:013
868720993924,10cyclictest37-21ksoftirqd/311:20:013
868720993919,14cyclictest272-21dbus-daemon10:55:013
86871699395,4cyclictest131rcu_preempt08:40:002
868716993920,14cyclictest1-21systemd08:15:002
868716993916,14cyclictest30-21ksoftirqd/212:30:012
868711993927,9cyclictest1-21systemd08:30:001
86871199391,5cyclictest131rcu_preempt10:10:001
86871199391,4cyclictest131rcu_preempt12:02:101
86871199391,4cyclictest131rcu_preempt12:02:091
86871199391,3cyclictest131rcu_preempt09:40:191
868706993933,3cyclictest16950irq/224-800000008:55:320
868706993926,9cyclictest363-21snmpd10:44:520
868706993926,10cyclictest12-21ksoftirqd/008:50:000
868706993911,5cyclictest131rcu_preempt12:26:190
86872099387,4cyclictest131rcu_preempt12:20:193
86872099382,3cyclictest131rcu_preempt11:45:003
868720993823,11cyclictest37-21ksoftirqd/309:55:013
868720993823,10cyclictest272-21dbus-daemon08:30:013
868720993821,12cyclictest235-21systemd-journal09:30:003
86871699382,6cyclictest131rcu_preempt09:50:012
86871699382,6cyclictest131rcu_preempt09:50:012
868716993826,8cyclictest1-21systemd11:10:002
868716993825,10cyclictest30-21ksoftirqd/210:45:002
868716993822,12cyclictest235-21systemd-journal09:58:202
868716993822,11cyclictest881372-21cron11:15:002
86871199383,3cyclictest131rcu_preempt11:07:191
86871199383,3cyclictest131rcu_preempt09:15:021
86871199382,8cyclictest23-21ksoftirqd/108:09:221
86871199382,8cyclictest23-21ksoftirqd/108:09:211
86871199382,3cyclictest131rcu_preempt11:00:001
86871199381,3cyclictest131rcu_preempt11:50:011
86871199381,3cyclictest131rcu_preempt11:50:011
86870699387,18cyclictest873331-21munin-run08:40:010
86870699384,21cyclictest875944-21grep09:30:000
868706993826,9cyclictest12-21ksoftirqd/009:25:020
868706993825,9cyclictest869665-21sshd07:25:190
868706993825,8cyclictest876182-21cron09:35:010
868706993825,8cyclictest876182-21cron09:35:000
868706993824,9cyclictest12-21ksoftirqd/008:00:020
868706993824,10cyclictest235-21systemd-journal09:07:190
868706993811,5cyclictest131rcu_preempt08:10:020
868706993811,5cyclictest131rcu_preempt08:10:010
868720993727,6cyclictest875810-21latency_hist09:25:023
868720993725,8cyclictest235-21systemd-journal07:25:003
86872099372,4cyclictest131rcu_preempt07:30:003
86872099371,4cyclictest131rcu_preempt08:40:013
868716993725,9cyclictest30-21ksoftirqd/208:30:002
868716993725,10cyclictest30-21ksoftirqd/212:15:012
868716993719,13cyclictest272-21dbus-daemon09:20:012
868716993719,13cyclictest272-21dbus-daemon09:20:012
86871199372,3cyclictest131rcu_preempt10:32:201
86871199372,3cyclictest131rcu_preempt09:22:201
86870699377,26cyclictest12-21ksoftirqd/010:09:440
86870699374,26cyclictest880881-21munin-run11:04:590
86870699372,5cyclictest131rcu_preempt10:30:020
868720993626,7cyclictest885830-21sshd12:39:183
86872099362,3cyclictest131rcu_preempt11:50:003
86872099362,3cyclictest131rcu_preempt11:49:593
868720993621,11cyclictest37-21ksoftirqd/311:25:013
868720993621,11cyclictest37-21ksoftirqd/311:25:003
86872099361,5cyclictest131rcu_preempt10:20:013
868720993615,16cyclictest272-21dbus-daemon11:05:003
86872099361,4cyclictest131rcu_preempt09:00:013
86872099361,4cyclictest131rcu_preempt08:35:003
86871699368,6cyclictest131rcu_preempt08:00:222
868716993629,4cyclictest880343-21sh10:55:002
868716993625,6cyclictest306-21in:imuxsock11:59:592
868716993622,11cyclictest885686-21sed12:35:022
868716993622,11cyclictest885686-21sed12:35:022
868716993621,11cyclictest30-21ksoftirqd/210:10:002
86871699361,9cyclictest880607-21latency_hist11:00:002
86871199363,4cyclictest131rcu_preempt10:24:481
86870699368,25cyclictest0-21swapper/011:24:400
86870699368,25cyclictest0-21swapper/011:24:390
86870699365,23cyclictest235-21systemd-journal10:30:000
868706993625,8cyclictest12-21ksoftirqd/008:55:030
86872099353,4cyclictest131rcu_preempt10:10:003
868720993526,6cyclictest876689-21sshd09:40:193
868720993523,8cyclictest1-21systemd11:35:003
868720993522,10cyclictest871840-21sed08:10:023
868720993522,10cyclictest871840-21sed08:10:013
86871699353,13cyclictest0-21swapper/211:35:012
868716993526,6cyclictest30-21ksoftirqd/209:37:192
868716993523,10cyclictest30-21ksoftirqd/208:10:012
868716993523,10cyclictest30-21ksoftirqd/208:10:002
868716993522,9cyclictest30-21ksoftirqd/212:10:022
868716993522,10cyclictest30-21ksoftirqd/211:25:012
868716993522,10cyclictest30-21ksoftirqd/211:25:012
868716993521,10cyclictest30-21ksoftirqd/208:35:032
86871699351,5cyclictest870212-21modprobe07:40:002
86870699351,4cyclictest0-21swapper/010:35:000
86872099344,4cyclictest131rcu_preempt07:15:033
86872099343,4cyclictest131rcu_preempt07:10:013
86872099342,4cyclictest131rcu_preempt10:35:003
868720993423,8cyclictest880303-21sed10:50:023
86872099341,4cyclictest131rcu_preempt11:10:013
868716993425,5cyclictest879579-21sendmail10:40:012
868716993425,5cyclictest879579-21sendmail10:40:002
86871699342,4cyclictest131rcu_preempt08:25:032
86871699342,4cyclictest131rcu_preempt08:25:022
868716993420,10cyclictest30-21ksoftirqd/209:25:002
868716993417,14cyclictest272-21dbus-daemon07:30:002
868706993427,4cyclictest16950irq/224-800000010:20:500
86870699341,30cyclictest0-21swapper/009:00:030
86871699332,8cyclictest0-21swapper/212:20:012
86871699332,8cyclictest0-21swapper/212:20:012
868716993322,9cyclictest30-21ksoftirqd/207:44:592
868716993319,11cyclictest30-21ksoftirqd/210:50:012
868716993317,12cyclictest272-21dbus-daemon11:05:012
868706993322,6cyclictest12-21ksoftirqd/008:25:010
868706993322,6cyclictest12-21ksoftirqd/008:25:000
868716993225,4cyclictest877226-21latency_hist09:55:002
868716993224,6cyclictest30-21ksoftirqd/207:55:002
868716993221,9cyclictest30-21ksoftirqd/209:10:012
868706993018,9cyclictest868849-21cut07:10:020
86871699291,6cyclictest0-21swapper/207:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional