You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-24 - 12:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Wed Dec 24, 2025 00:45:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1403113997928,10cyclictest1406952-21munin-run20:25:000
140311399712,35cyclictest1408697-21modprobe20:55:030
140311399712,35cyclictest1408697-21modprobe20:55:020
1403125996813,12cyclictest275-21dbus-daemon22:10:012
140311399681,38cyclictest1418225-21sed00:00:030
1403128996213,8cyclictest37-21ksoftirqd/320:05:013
1403125996217,11cyclictest30-21ksoftirqd/223:40:022
1403125996213,8cyclictest30-21ksoftirqd/223:20:012
1403125996213,8cyclictest30-21ksoftirqd/223:20:002
1403113996136,4cyclictest16950irq/224-800000020:55:020
1403113996118,25cyclictest19350irq/34-mmc100:25:020
1403128996014,7cyclictest37-21ksoftirqd/322:50:023
1403128996013,9cyclictest37-21ksoftirqd/321:35:003
1403128996013,9cyclictest37-21ksoftirqd/321:35:003
1403128995817,12cyclictest37-21ksoftirqd/323:20:003
1403128995817,12cyclictest37-21ksoftirqd/323:20:003
1403128995810,13cyclictest37-21ksoftirqd/323:50:023
140312599586,9cyclictest30-21ksoftirqd/219:25:012
140312899578,10cyclictest37-21ksoftirqd/323:30:013
1403128995715,9cyclictest37-21ksoftirqd/320:00:003
1403125995717,9cyclictest30-21ksoftirqd/220:35:012
140312599567,9cyclictest30-21ksoftirqd/219:45:022
1403125995612,8cyclictest30-21ksoftirqd/221:40:012
1403125995610,9cyclictest30-21ksoftirqd/223:45:002
1403125995610,9cyclictest30-21ksoftirqd/223:45:002
1403128995514,12cyclictest37-21ksoftirqd/300:20:013
1403128995513,7cyclictest37-21ksoftirqd/319:55:003
1403128995513,10cyclictest37-21ksoftirqd/319:50:023
140312599556,6cyclictest30-21ksoftirqd/200:10:022
140312599556,6cyclictest30-21ksoftirqd/200:10:022
1403125995514,8cyclictest30-21ksoftirqd/200:30:022
140312899547,6cyclictest1411102-21systemd-run21:45:003
1403128995410,7cyclictest37-21ksoftirqd/323:40:023
140312599548,9cyclictest30-21ksoftirqd/222:55:022
140312599547,7cyclictest30-21ksoftirqd/219:40:022
140312599545,10cyclictest30-21ksoftirqd/220:00:012
140312599544,7cyclictest30-21ksoftirqd/222:40:022
140312599543,9cyclictest30-21ksoftirqd/220:50:002
1403125995411,10cyclictest30-21ksoftirqd/219:35:012
140311399542,34cyclictest131rcu_preempt00:38:220
1403128995312,10cyclictest37-21ksoftirqd/319:45:023
140312599538,10cyclictest0-21swapper/220:30:002
1403125995313,9cyclictest30-21ksoftirqd/222:00:012
140312899526,7cyclictest37-21ksoftirqd/322:10:003
1403128995212,9cyclictest37-21ksoftirqd/321:00:023
1403128995212,9cyclictest37-21ksoftirqd/321:00:023
140312599525,8cyclictest30-21ksoftirqd/223:15:012
1403125995218,9cyclictest30-21ksoftirqd/223:50:012
1403125995211,6cyclictest30-21ksoftirqd/221:45:002
1403125995210,10cyclictest0-21swapper/220:05:012
140311399523,3cyclictest131rcu_preempt21:35:020
140311399523,3cyclictest131rcu_preempt21:35:020
140312899517,11cyclictest37-21ksoftirqd/322:35:013
1403128995118,10cyclictest37-21ksoftirqd/321:30:023
140312599517,9cyclictest30-21ksoftirqd/221:10:022
140312599514,10cyclictest30-21ksoftirqd/222:25:022
1403125995113,10cyclictest30-21ksoftirqd/221:35:022
1403125995113,10cyclictest30-21ksoftirqd/221:35:022
140311399513,42cyclictest311-21rs:main0
140312899509,8cyclictest37-21ksoftirqd/319:20:003
140312899508,7cyclictest37-21ksoftirqd/321:55:023
140312899505,12cyclictest37-21ksoftirqd/322:30:023
140312899502,12cyclictest37-21ksoftirqd/322:45:013
140312599509,11cyclictest30-21ksoftirqd/223:55:022
140312599508,9cyclictest30-21ksoftirqd/221:55:022
1403113995038,8cyclictest1412011-21latency_hist22:00:010
140311399501,44cyclictest1410072-21systemd21:25:000
140312899499,9cyclictest37-21ksoftirqd/321:05:013
140312899495,8cyclictest37-21ksoftirqd/323:05:033
140312899495,8cyclictest37-21ksoftirqd/323:05:023
140312899495,8cyclictest37-21ksoftirqd/320:55:013
140312899492,14cyclictest37-21ksoftirqd/322:55:023
140312899492,12cyclictest37-21ksoftirqd/321:40:013
140312899491,7cyclictest0-21swapper/300:25:023
140312599499,8cyclictest30-21ksoftirqd/220:55:022
140312599495,8cyclictest30-21ksoftirqd/223:35:002
140312599492,9cyclictest30-21ksoftirqd/200:25:012
1403125994916,9cyclictest30-21ksoftirqd/223:00:022
1403121994911,6cyclictest131rcu_preempt19:35:001
1403113994942,4cyclictest16950irq/224-800000021:20:000
140312899486,8cyclictest37-21ksoftirqd/319:35:003
140312899486,7cyclictest37-21ksoftirqd/323:32:013
140312899484,7cyclictest1414697-21latency_hist22:55:013
140312899484,10cyclictest37-21ksoftirqd/320:30:013
1403128994830,14cyclictest275-21dbus-daemon00:00:023
140312899482,10cyclictest37-21ksoftirqd/320:20:023
140312599489,9cyclictest0-21swapper/219:48:012
140312599486,6cyclictest30-21ksoftirqd/219:30:022
140312599486,6cyclictest30-21ksoftirqd/219:30:022
140312599481,13cyclictest30-21ksoftirqd/222:50:012
1403125994810,10cyclictest0-21swapper/220:40:022
140312899478,8cyclictest37-21ksoftirqd/319:30:023
140312899477,9cyclictest1410055-21latency_hist21:25:023
140312899477,8cyclictest37-21ksoftirqd/322:20:013
140312899476,6cyclictest37-21ksoftirqd/321:50:013
140312899476,5cyclictest1413925-21munin-run22:40:003
140312899474,9cyclictest1403269-21date19:10:023
140312899473,9cyclictest37-21ksoftirqd/300:15:013
1403128994713,10cyclictest37-21ksoftirqd/322:00:013
140312599478,11cyclictest30-21ksoftirqd/220:45:012
140312599474,7cyclictest30-21ksoftirqd/222:35:012
140312599473,10cyclictest30-21ksoftirqd/223:25:012
140312599471,10cyclictest30-21ksoftirqd/221:50:022
140311399473,39cyclictest1404598-21sh19:40:000
1403113994732,11cyclictest1-21systemd20:45:000
1403113994729,14cyclictest1414679-21sshd22:53:290
140312899466,9cyclictest37-21ksoftirqd/322:05:013
140312899461,15cyclictest37-21ksoftirqd/323:15:013
140312599467,6cyclictest30-21ksoftirqd/220:15:012
140312599466,6cyclictest30-21ksoftirqd/221:30:022
140312599464,10cyclictest30-21ksoftirqd/220:25:032
1403125994631,12cyclictest1-21systemd00:05:012
140312599462,8cyclictest30-21ksoftirqd/222:30:012
140312599461,26cyclictest0-21swapper/220:10:002
140312599461,12cyclictest30-21ksoftirqd/200:40:032
1403121994630,12cyclictest238-21systemd-journal19:55:001
1403121994629,12cyclictest275-21dbus-daemon21:30:011
140311399463,39cyclictest0-21swapper/023:40:010
140312899457,8cyclictest37-21ksoftirqd/320:40:023
140312899455,8cyclictest37-21ksoftirqd/323:25:023
140312899455,6cyclictest37-21ksoftirqd/322:25:013
140312899453,37cyclictest1406160-21modprobe20:10:003
140312899452,9cyclictest37-21ksoftirqd/321:20:013
140312899452,8cyclictest37-21ksoftirqd/319:40:013
140312899451,5cyclictest0-21swapper/320:50:013
140312599459,5cyclictest131rcu_preempt22:45:012
140312599458,8cyclictest0-21swapper/219:15:012
140312599455,10cyclictest30-21ksoftirqd/223:10:012
1403125994529,11cyclictest1412892-21systemctl22:20:002
140312599452,8cyclictest30-21ksoftirqd/200:35:012
140312199453,35cyclictest275-21dbus-daemon23:55:011
1403121994530,11cyclictest238-21systemd-journal23:05:011
1403121994530,11cyclictest238-21systemd-journal23:05:011
1403113994527,13cyclictest1-21systemd23:10:000
140311399451,3cyclictest131rcu_preempt22:45:010
140312899445,8cyclictest37-21ksoftirqd/300:35:003
140312899444,7cyclictest37-21ksoftirqd/323:10:013
140312899444,10cyclictest37-21ksoftirqd/319:23:223
140312899443,8cyclictest37-21ksoftirqd/300:05:003
140312899442,7cyclictest0-21swapper/320:40:013
140312599447,7cyclictest30-21ksoftirqd/200:15:002
140312599446,6cyclictest30-21ksoftirqd/222:15:012
140312599445,9cyclictest30-21ksoftirqd/221:10:022
140312599445,9cyclictest30-21ksoftirqd/219:55:002
140312599443,8cyclictest30-21ksoftirqd/219:20:002
140312599442,8cyclictest30-21ksoftirqd/223:05:032
140312599442,8cyclictest30-21ksoftirqd/223:05:022
140312599442,8cyclictest30-21ksoftirqd/221:00:012
140312599442,8cyclictest30-21ksoftirqd/221:00:012
1403121994434,7cyclictest23-21ksoftirqd/120:50:021
1403121994429,11cyclictest275-21dbus-daemon19:40:011
1403121994426,11cyclictest1419165-21systemctl00:20:011
1403113994424,15cyclictest1406684-21systemctl20:20:000
140312899435,6cyclictest37-21ksoftirqd/323:40:213
140312899435,6cyclictest37-21ksoftirqd/323:40:213
140312899432,12cyclictest37-21ksoftirqd/320:15:013
140312599437,8cyclictest30-21ksoftirqd/221:20:012
140312599431,7cyclictest30-21ksoftirqd/200:20:002
140312599431,10cyclictest30-21ksoftirqd/221:25:012
1403121994331,8cyclictest1414681-21sshd22:53:291
1403121994331,8cyclictest1-21systemd20:30:011
1403121994328,11cyclictest275-21dbus-daemon23:15:001
1403121994322,14cyclictest275-21dbus-daemon19:45:011
1403113994327,12cyclictest275-21dbus-daemon22:20:010
140311399432,6cyclictest131rcu_preempt20:15:010
1403113994323,14cyclictest238-21systemd-journal21:40:000
140311399431,9cyclictest309-21in:imuxsock23:30:000
140311399431,38cyclictest1417541-21sed23:45:020
140311399431,38cyclictest1417541-21sed23:45:020
140312899425,11cyclictest37-21ksoftirqd/300:30:013
140312899424,7cyclictest37-21ksoftirqd/300:10:023
140312899424,7cyclictest37-21ksoftirqd/300:10:013
140312899423,9cyclictest37-21ksoftirqd/319:30:023
140312199423,8cyclictest23-21ksoftirqd/122:00:001
1403113994234,5cyclictest1409266-21munin-plugin-st21:10:000
1403113994231,8cyclictest12-21ksoftirqd/022:05:010
1403113994229,9cyclictest1410323-21systemd-run21:30:020
140311399422,34cyclictest1412651-21latency_hist22:15:010
140311399421,37cyclictest1403820-21sshd19:22:210
140312899416,9cyclictest37-21ksoftirqd/321:05:493
140312599419,6cyclictest30-21ksoftirqd/219:10:022
140312599415,5cyclictest30-21ksoftirqd/220:20:002
1403125994126,11cyclictest275-21dbus-daemon22:05:012
140312599411,5cyclictest0-21swapper/223:30:032
140312599410,9cyclictest1418328-21sed00:00:022
140312199419,25cyclictest1406953-21grep20:25:001
140312199413,3cyclictest131rcu_preempt20:15:011
1403121994133,5cyclictest210-21jbd2/mmcblk1p2-00:00:011
1403121994129,8cyclictest238-21systemd-journal00:14:591
1403121994129,8cyclictest1410800-21cron21:40:001
1403121994128,9cyclictest238-21systemd-journal21:20:001
140311399417,6cyclictest131rcu_preempt00:15:000
140311399414,9cyclictest131rcu_preempt23:32:010
140311399413,4cyclictest131rcu_preempt22:50:020
1403113994127,11cyclictest1-21systemd21:15:010
1403113994125,12cyclictest275-21dbus-daemon23:05:020
1403113994125,12cyclictest275-21dbus-daemon23:05:010
1403113994118,16cyclictest12-21ksoftirqd/023:20:000
1403113994118,16cyclictest12-21ksoftirqd/023:20:000
140311399411,36cyclictest0-21swapper/020:05:000
1403113994110,6cyclictest131rcu_preempt20:27:210
140312899405,8cyclictest37-21ksoftirqd/322:12:213
140312599404,9cyclictest0-21swapper/221:05:032
1403121994029,7cyclictest1413140-21sshd22:23:211
1403121994029,6cyclictest309-21in:imuxsock21:35:011
1403121994029,6cyclictest309-21in:imuxsock21:35:011
1403121994026,11cyclictest23-21ksoftirqd/119:25:021
1403121994026,10cyclictest1419401-21sshd00:21:211
1403121994025,12cyclictest23-21ksoftirqd/122:20:011
1403121994023,11cyclictest1418642-21sh00:10:011
1403121994023,11cyclictest1418642-21sh00:10:001
1403113994028,9cyclictest1418619-21sshd00:07:220
1403113994028,9cyclictest1418619-21sshd00:07:210
140311399402,3cyclictest131rcu_preempt23:25:020
140312899395,7cyclictest37-21ksoftirqd/323:50:023
1403128993926,9cyclictest37-21ksoftirqd/320:22:523
1403121993931,4cyclictest1408735-21sendmail21:00:001
1403121993931,4cyclictest1408735-21sendmail21:00:001
1403121993929,7cyclictest23-21ksoftirqd/121:25:021
1403121993928,8cyclictest1411069-21sshd21:43:191
1403121993925,11cyclictest1405646-21tr20:00:001
1403121993922,9cyclictest0-21swapper/121:10:001
1403121993915,16cyclictest1419646-21kworker/u8:000:30:021
140311399395,3cyclictest131rcu_preempt19:45:010
140311399395,26cyclictest1403107-21cyclictest19:10:020
140311399393,4cyclictest131rcu_preempt23:50:000
140311399393,3cyclictest131rcu_preempt21:00:480
1403113993933,4cyclictest16950irq/224-800000022:35:010
1403113993933,3cyclictest16950irq/224-800000022:40:000
1403113993931,5cyclictest1419437-21systemd-run00:25:020
1403113993929,7cyclictest19350irq/34-mmc121:55:020
1403113993910,5cyclictest131rcu_preempt21:48:190
140312899384,7cyclictest37-21ksoftirqd/300:36:323
140312199388,3cyclictest131rcu_preempt19:50:011
140312199383,3cyclictest131rcu_preempt20:05:021
1403121993828,7cyclictest1412593-21sshd22:11:221
1403121993827,5cyclictest309-21in:imuxsock20:40:001
1403121993825,10cyclictest1408490-21systemd-run20:55:001
1403121993821,12cyclictest1-21systemd20:35:011
1403113993831,4cyclictest16950irq/224-800000019:45:210
1403113993830,5cyclictest16950irq/224-800000023:00:000
1403113993827,8cyclictest12-21ksoftirqd/020:10:010
140311399382,3cyclictest131rcu_preempt23:15:030
140311399381,5cyclictest1-21systemd21:45:000
140311399381,3cyclictest131rcu_preempt19:20:000
140311399381,33cyclictest1-21systemd19:30:020
140311399381,33cyclictest1-21systemd19:30:020
140312899374,5cyclictest37-21ksoftirqd/320:35:033
1403121993727,7cyclictest23-21ksoftirqd/100:05:001
140312199372,6cyclictest131rcu_preempt23:35:011
1403121993725,8cyclictest1420187-21cron00:40:001
140311399375,3cyclictest131rcu_preempt19:15:020
140311399373,3cyclictest131rcu_preempt19:55:000
1403113993726,6cyclictest1407692-21cron20:40:000
140311399372,5cyclictest131rcu_preempt20:50:020
1403113993723,8cyclictest309-21in:imuxsock00:15:020
140312199366,14cyclictest1406668-21modprobe20:20:001
140311399364,4cyclictest131rcu_preempt00:35:010
1403113993628,5cyclictest12850irq/18-4d00000019:59:210
140311399361,5cyclictest131rcu_preempt20:35:010
1403121993526,6cyclictest1414951-21latency_hist23:00:031
1403121993524,8cyclictest1412101-21sshd22:03:211
1403121993524,7cyclictest309-21in:imuxsock22:45:021
1403121993522,9cyclictest23-21ksoftirqd/123:40:011
1403121993522,10cyclictest238-21systemd-journal21:15:001
1403121993517,14cyclictest275-21dbus-daemon22:30:021
140311399353,3cyclictest131rcu_preempt22:25:010
140311399351,4cyclictest1412373-21munin-run22:10:000
140311399351,3cyclictest131rcu_preempt19:35:000
1403121993427,4cyclictest1415229-21latency_hist23:05:031
140312199342,4cyclictest131rcu_preempt22:40:001
1403121993423,8cyclictest1420001-21sed00:35:021
1403121993421,9cyclictest23-21ksoftirqd/123:20:001
1403121993421,9cyclictest23-21ksoftirqd/123:20:001
1403121993418,12cyclictest275-21dbus-daemon19:20:001
140311399342,4cyclictest1418448-21date00:05:010
1403113993422,9cyclictest1413487-21cat22:30:020
1403121993325,6cyclictest23-21ksoftirqd/120:10:011
1403121993322,8cyclictest1395836-21kworker/1:021:55:011
1403121993321,9cyclictest1403310-21cut19:15:011
1403121993321,6cyclictest1416412-21wc23:25:011
1403121993319,11cyclictest275-21dbus-daemon20:45:011
140312199331,4cyclictest131rcu_preempt21:05:011
140312199331,3cyclictest131rcu_preempt23:45:001
140312199331,3cyclictest131rcu_preempt23:45:001
140312199331,3cyclictest131rcu_preempt19:10:021
1403121993310,18cyclictest1404096-21modprobe19:30:001
1403121993310,18cyclictest1404096-21modprobe19:30:001
140312899321,4cyclictest131rcu_preempt21:10:213
140312199324,8cyclictest23-21ksoftirqd/122:41:101
1403121993218,11cyclictest1417579-21sshd23:48:211
140312199312,3cyclictest131rcu_preempt22:10:001
1403121993122,5cyclictest1413666-21munin-plugin-st22:35:001
1403121993118,7cyclictest309-21in:imuxsock23:30:001
1403121993116,11cyclictest238-21systemd-journal21:48:211
1403128992923,4cyclictest1403051-21latency_hist19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional