You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-12 - 00:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Dec 11, 2025 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1978724999784,7cyclictest1978726-21latency_hist07:10:000
1978734997636,34cyclictest1-21systemd08:45:012
1978734997327,13cyclictest235-21systemd-journal08:30:002
1978734996556,4cyclictest30-21ksoftirqd/207:25:022
1978730996410,19cyclictest1981646-21sed08:05:021
197873499638,18cyclictest1992708-21kworker/u8:111:55:012
197873099619,17cyclictest1980505-21systemctl07:45:011
1978730996118,8cyclictest23-21ksoftirqd/110:30:031
1978730996111,16cyclictest1991844-21sed11:20:021
1978730996110,17cyclictest1994131-21latency_hist12:05:011
1978730996042,13cyclictest272-21dbus-daemon07:40:011
1978730996010,18cyclictest1991241-21cat11:10:021
1978730996010,18cyclictest1991241-21cat11:10:021
1978730995913,16cyclictest1995964-21sed12:40:011
1978724995749,6cyclictest151rcuc/010:35:020
1978739995632,17cyclictest1-21systemd10:55:013
1978739995632,17cyclictest1-21systemd10:55:013
1978730995616,9cyclictest23-21ksoftirqd/111:00:001
197873099559,15cyclictest1993285-21sh11:50:001
1978724995535,15cyclictest235-21systemd-journal12:30:010
1978724995535,15cyclictest235-21systemd-journal12:30:000
1978724995530,22cyclictest12-21ksoftirqd/012:40:010
1978739995238,9cyclictest235-21systemd-journal08:20:003
1978739995211,6cyclictest131rcu_preempt12:00:023
1978739995211,6cyclictest131rcu_preempt12:00:023
1978724995235,12cyclictest235-21systemd-journal10:55:010
1978724995235,12cyclictest235-21systemd-journal10:55:010
1978724995233,12cyclictest1990709-21latency_hist11:00:010
1978734995132,14cyclictest30-21ksoftirqd/211:20:002
1978734995129,16cyclictest235-21systemd-journal08:35:012
1978730995134,13cyclictest1992994-21sshd11:41:291
1978730995134,13cyclictest1992994-21sshd11:41:291
1978730995134,12cyclictest235-21systemd-journal09:10:571
1978730995112,9cyclictest23-21ksoftirqd/112:00:021
1978730995112,9cyclictest23-21ksoftirqd/112:00:021
1978734995033,13cyclictest272-21dbus-daemon09:45:012
1978734995033,13cyclictest272-21dbus-daemon09:45:012
1978730995020,8cyclictest1983142-21munin-run08:35:021
1978724995037,9cyclictest12-21ksoftirqd/009:25:030
1978724995032,13cyclictest235-21systemd-journal07:55:010
1978724995032,13cyclictest235-21systemd-journal07:55:010
1978724995018,28cyclictest12-21ksoftirqd/012:31:290
1978724995010,9cyclictest1982676-21ls08:25:000
197873999496,10cyclictest37-21ksoftirqd/307:20:003
197873499499,8cyclictest30-21ksoftirqd/210:40:012
197873099497,9cyclictest0-21swapper/110:50:021
1978730994934,11cyclictest1980999-21munin-run07:55:011
1978730994934,11cyclictest1980999-21munin-run07:55:011
1978724994927,15cyclictest235-21systemd-journal12:00:000
1978724994927,15cyclictest235-21systemd-journal12:00:000
1978739994818,8cyclictest37-21ksoftirqd/308:45:023
197873999481,19cyclictest1983047-21latency_hist08:30:023
1978730994833,11cyclictest1984695-21munin-run09:05:011
1978730994811,13cyclictest1982004-21kworker/u8:008:41:281
1978724994837,7cyclictest1993525-21munin-plugin-st11:55:000
197872499482,41cyclictest1989638-21grep10:40:000
1978724994822,5cyclictest1979457-21latency_hist07:25:010
1978739994737,7cyclictest37-21ksoftirqd/309:05:013
1978739994735,9cyclictest235-21systemd-journal11:40:013
1978734994726,14cyclictest272-21dbus-daemon10:00:012
1978734994726,14cyclictest272-21dbus-daemon10:00:012
1978730994734,10cyclictest1995577-21sshd12:31:291
1978730994734,10cyclictest1989884-21sshd10:42:281
1978730994733,10cyclictest1992459-21sshd11:31:121
1978730994732,11cyclictest1987054-21munin-run09:50:011
1978730994732,10cyclictest1980749-21munin-run07:50:011
1978730994731,13cyclictest1984928-21sshd09:08:281
1978730994730,12cyclictest1988343-21/usr/sbin/munin10:15:021
1978730994730,12cyclictest1988343-21/usr/sbin/munin10:15:021
1978730994723,18cyclictest1995326-21sshd12:26:301
1978730994723,18cyclictest1995326-21sshd12:26:291
1978724994736,8cyclictest12-21ksoftirqd/009:45:030
1978724994734,8cyclictest12-21ksoftirqd/011:15:010
1978724994727,5cyclictest12-21ksoftirqd/007:20:000
1978724994720,23cyclictest12-21ksoftirqd/007:45:010
197873999466,11cyclictest37-21ksoftirqd/311:35:003
1978739994628,12cyclictest235-21systemd-journal11:55:003
197873999462,7cyclictest0-21swapper/310:10:023
1978739994619,7cyclictest37-21ksoftirqd/310:15:023
1978739994619,7cyclictest37-21ksoftirqd/310:15:023
197873999461,7cyclictest0-21swapper/309:25:003
1978739994615,6cyclictest131rcu_preempt10:25:003
1978734994635,8cyclictest1984141-21sshd08:53:272
1978734994631,12cyclictest30-21ksoftirqd/208:46:292
1978730994633,10cyclictest235-21systemd-journal07:16:591
1978730994633,10cyclictest1992238-21munin-run11:30:001
1978730994633,10cyclictest1992238-21munin-run11:30:001
1978730994632,10cyclictest1981797-21munin-run08:10:021
197873099462,6cyclictest131rcu_preempt08:30:021
197873099461,2cyclictest0-21swapper/108:40:021
197873999458,11cyclictest37-21ksoftirqd/308:50:003
1978739994534,8cyclictest37-21ksoftirqd/312:10:023
1978739994533,9cyclictest37-21ksoftirqd/311:25:013
1978739994532,9cyclictest272-21dbus-daemon09:40:003
1978739994528,12cyclictest235-21systemd-journal07:20:023
1978730994537,4cyclictest1988092-21munin-run10:10:011
1978730994531,10cyclictest48-21kcompactd007:26:171
1978730994531,10cyclictest1987838-21munin-run10:05:011
1978730994531,10cyclictest1986765-21sshd09:42:281
1978730994531,10cyclictest1986765-21sshd09:42:281
1978730994530,12cyclictest1984381-21sshd08:55:281
1978730994529,12cyclictest1991459-21/usr/sbin/munin11:15:011
1978730994529,12cyclictest1986552-21munin-run09:40:011
1978724994519,21cyclictest12-21ksoftirqd/009:10:000
1978739994432,8cyclictest1985461-21cron09:20:013
1978739994426,14cyclictest1-21systemd12:20:003
1978739994410,6cyclictest131rcu_preempt09:35:003
1978734994430,9cyclictest306-21in:imuxsock09:55:002
1978734994429,11cyclictest1-21systemd10:15:012
1978734994429,11cyclictest1-21systemd10:15:012
1978730994437,4cyclictest1979974-21munin-run07:35:011
1978730994430,11cyclictest1984128-21sshd08:50:281
1978730994430,10cyclictest1994538-21sshd12:12:291
1978730994428,12cyclictest1986235-21sshd09:33:281
1978730994425,14cyclictest1979448-21cron07:25:011
197872499442,4cyclictest235-21systemd-journal08:15:000
197873999437,6cyclictest131rcu_preempt11:30:023
197873999437,6cyclictest131rcu_preempt11:30:023
1978739994334,6cyclictest37-21ksoftirqd/309:55:023
1978739994332,8cyclictest37-21ksoftirqd/311:45:293
1978739994332,8cyclictest37-21ksoftirqd/308:00:013
1978739994324,16cyclictest37-21ksoftirqd/312:05:003
1978730994330,9cyclictest1983913-21munin-run08:50:011
1978730994329,9cyclictest48-21kcompactd009:58:401
1978730994329,9cyclictest48-21kcompactd009:58:401
1978730994319,19cyclictest1981230-21sshd07:58:281
1978724994331,9cyclictest1-21systemd08:30:010
1978724994327,12cyclictest272-21dbus-daemon10:10:010
197872499432,6cyclictest131rcu_preempt11:50:000
197872499432,3cyclictest131rcu_preempt10:50:020
1978724994323,13cyclictest1-21systemd07:50:000
1978724994322,15cyclictest272-21dbus-daemon11:05:010
1978724994319,19cyclictest12-21ksoftirqd/007:40:020
197873999427,7cyclictest37-21ksoftirqd/312:40:013
197873999427,5cyclictest131rcu_preempt11:10:023
197873999427,5cyclictest131rcu_preempt11:10:023
1978739994231,8cyclictest37-21ksoftirqd/310:35:003
1978739994219,18cyclictest0-21swapper/311:05:013
1978730994231,8cyclictest1992189-21cat11:25:031
1978730994230,9cyclictest1985976-21sshd09:26:281
1978730994229,9cyclictest48-21kcompactd009:24:401
1978730994221,15cyclictest1982540-21sshd08:20:471
1978730994217,20cyclictest1990906-21modprobe11:00:291
197872499423,35cyclictest131rcu_preempt09:30:280
1978724994229,9cyclictest12-21ksoftirqd/008:10:020
1978724994228,10cyclictest1983381-21cron08:40:000
197872499421,5cyclictest131rcu_preempt10:00:000
197872499421,5cyclictest131rcu_preempt10:00:000
197872499421,37cyclictest1982271-21sshd08:17:280
197872499421,23cyclictest0-21swapper/011:18:290
1978739994131,7cyclictest37-21ksoftirqd/310:50:013
1978739994130,7cyclictest37-21ksoftirqd/310:30:013
1978739994130,7cyclictest37-21ksoftirqd/310:30:013
1978739994127,11cyclictest235-21systemd-journal12:10:023
1978739994127,10cyclictest235-21systemd-journal08:40:003
1978739994112,8cyclictest131rcu_preempt10:00:013
1978739994112,8cyclictest131rcu_preempt10:00:013
197873499414,5cyclictest0-21swapper/208:00:442
1978734994131,6cyclictest306-21in:imuxsock11:00:002
1978734994128,9cyclictest306-21in:imuxsock11:25:032
1978734994128,9cyclictest306-21in:imuxsock11:25:032
1978734994125,12cyclictest30-21ksoftirqd/208:40:012
1978730994128,10cyclictest363-21snmpd10:26:231
1978730994128,10cyclictest363-21snmpd10:26:231
1978730994128,10cyclictest363-21snmpd08:12:591
1978730994128,10cyclictest1978921-21munin-run07:15:001
1978730994125,13cyclictest363-21snmpd12:05:121
1978730994125,11cyclictest1992785-21munin-run11:40:021
1978724994133,5cyclictest0-21swapper/011:10:010
1978724994133,5cyclictest0-21swapper/011:10:010
1978724994129,9cyclictest12-21ksoftirqd/009:55:010
197872499412,5cyclictest131rcu_preempt11:40:000
1978724994124,13cyclictest272-21dbus-daemon09:05:010
197872499412,3cyclictest131rcu_preempt09:45:010
1978724994117,20cyclictest12-21ksoftirqd/009:40:010
197872499411,5cyclictest131rcu_preempt08:35:010
1978739994032,6cyclictest37-21ksoftirqd/311:18:293
1978739994028,8cyclictest37-21ksoftirqd/312:21:293
1978739994026,10cyclictest37-21ksoftirqd/307:40:013
1978739994015,20cyclictest131rcu_preempt11:15:013
1978734994028,6cyclictest1992754-21munin-run11:40:002
1978734994027,9cyclictest272-21dbus-daemon11:50:002
1978734994026,10cyclictest30-21ksoftirqd/207:30:012
197873499402,5cyclictest1991352-21date11:10:022
197873499402,5cyclictest1991352-21date11:10:022
1978730994028,9cyclictest1988550-21sshd10:16:011
1978730994027,9cyclictest363-21snmpd09:53:191
1978730994027,10cyclictest363-21snmpd10:53:061
1978730994027,10cyclictest363-21snmpd10:53:061
1978730994020,14cyclictest1994813-21cron12:20:001
1978724994033,4cyclictest16950irq/224-800000010:05:000
1978724994031,4cyclictest12-21ksoftirqd/010:30:000
1978724994031,4cyclictest12-21ksoftirqd/010:30:000
1978724994024,12cyclictest1994808-21sshd12:19:290
197872499401,3cyclictest131rcu_preempt11:45:000
197872499401,3cyclictest131rcu_preempt11:45:000
1978739993928,9cyclictest37-21ksoftirqd/307:43:283
1978739993924,12cyclictest272-21dbus-daemon08:10:003
1978739993921,12cyclictest1-21systemd08:15:003
1978734993930,7cyclictest30-21ksoftirqd/207:13:282
1978734993930,4cyclictest1980221-21munin-plugin-st07:40:012
1978734993928,8cyclictest306-21in:imuxsock12:30:022
1978734993927,9cyclictest30-21ksoftirqd/209:00:022
1978734993924,10cyclictest1979201-21tr07:20:002
1978734993923,11cyclictest235-21systemd-journal10:20:282
1978734993922,13cyclictest30-21ksoftirqd/211:57:292
1978734993922,13cyclictest30-21ksoftirqd/211:57:292
1978734993910,7cyclictest30-21ksoftirqd/210:20:002
1978730993927,9cyclictest363-21snmpd10:38:301
1978730993926,9cyclictest363-21snmpd11:52:351
1978730993924,11cyclictest363-21snmpd09:19:201
1978730993910,24cyclictest1982316-21tr08:19:591
1978724993928,7cyclictest12-21ksoftirqd/010:20:010
1978724993926,9cyclictest1986198-21sed09:30:020
1978724993924,11cyclictest272-21dbus-daemon08:45:010
1978724993916,4cyclictest12-21ksoftirqd/008:00:010
197872499391,3cyclictest0-21swapper/008:55:010
197873999385,10cyclictest37-21ksoftirqd/312:30:013
197873999385,10cyclictest37-21ksoftirqd/312:30:013
1978739993828,8cyclictest37-21ksoftirqd/307:29:283
1978739993828,7cyclictest37-21ksoftirqd/312:35:003
1978739993828,7cyclictest37-21ksoftirqd/309:06:283
1978739993828,7cyclictest1986004-21munin-plugin-st09:30:003
1978739993828,6cyclictest306-21in:imuxsock10:40:003
1978739993826,8cyclictest37-21ksoftirqd/309:49:593
197873499384,3cyclictest131rcu_preempt08:10:012
197873499383,4cyclictest131rcu_preempt11:35:012
1978734993828,8cyclictest30-21ksoftirqd/209:40:022
1978734993826,9cyclictest30-21ksoftirqd/212:10:002
1978734993826,8cyclictest235-21systemd-journal08:00:002
1978734993823,12cyclictest272-21dbus-daemon07:35:012
197873499381,7cyclictest131rcu_preempt12:20:002
1978734993812,21cyclictest1982048-21unin-run08:15:002
197873499381,11cyclictest131rcu_preempt11:15:012
1978730993826,9cyclictest1989002-21cat10:25:031
1978724993830,5cyclictest1988867-21latency_hist10:25:000
1978724993829,5cyclictest12-21ksoftirqd/011:35:000
1978724993827,8cyclictest1979943-21sshd07:33:280
1978724993825,9cyclictest1979683-21sshd07:26:280
1978724993813,6cyclictest151rcuc/009:50:010
197872499381,33cyclictest1989908-21munin-run10:45:010
197873999373,9cyclictest37-21ksoftirqd/310:16:283
1978739993728,7cyclictest37-21ksoftirqd/308:35:023
1978739993727,7cyclictest37-21ksoftirqd/309:42:283
1978739993727,7cyclictest37-21ksoftirqd/309:42:283
1978739993726,8cyclictest37-21ksoftirqd/308:25:003
1978739993722,11cyclictest37-21ksoftirqd/309:00:013
1978739993712,6cyclictest131rcu_preempt09:15:003
1978734993723,11cyclictest30-21ksoftirqd/209:13:112
1978734993723,11cyclictest235-21systemd-journal10:05:002
1978730993724,9cyclictest363-21snmpd12:22:451
1978724993726,9cyclictest12-21ksoftirqd/007:15:010
1978724993724,10cyclictest0-21swapper/011:25:030
197872499371,5cyclictest131rcu_preempt12:25:010
197872499371,4cyclictest1994562-21munin-run12:15:020
197873999364,6cyclictest131rcu_preempt10:05:013
1978739993628,6cyclictest37-21ksoftirqd/307:55:023
1978739993628,6cyclictest37-21ksoftirqd/307:55:023
1978739993622,10cyclictest37-21ksoftirqd/310:40:023
1978739993621,11cyclictest1-21systemd08:05:013
1978734993627,7cyclictest30-21ksoftirqd/209:25:592
1978734993624,9cyclictest30-21ksoftirqd/212:40:002
1978734993622,10cyclictest30-21ksoftirqd/210:55:022
1978734993622,10cyclictest30-21ksoftirqd/210:55:022
197872499365,25cyclictest281-21systemd-logind09:20:010
1978724993632,2cyclictest0-21swapper/009:00:020
1978724993630,4cyclictest16950irq/224-800000008:03:280
1978724993626,7cyclictest1994180-21sed12:05:010
1978724993626,6cyclictest12-21ksoftirqd/009:15:000
197872499361,6cyclictest1988294-21sshd10:10:280
197872499361,6cyclictest1988294-21sshd10:10:280
197873999358,5cyclictest131rcu_preempt08:55:013
1978739993528,5cyclictest37-21ksoftirqd/307:15:023
197873999351,7cyclictest0-21swapper/310:58:303
197873499353,3cyclictest131rcu_preempt11:45:022
197873499353,3cyclictest131rcu_preempt11:45:022
1978734993525,6cyclictest1991985-21munin-plugin-st11:25:002
1978734993521,10cyclictest306-21in:imuxsock08:15:022
1978724993522,10cyclictest1992218-21sshd11:28:290
1978724993522,10cyclictest1992218-21sshd11:28:290
197872499351,4cyclictest1994290-21sshd12:08:290
197873999348,5cyclictest131rcu_preempt07:32:513
1978739993425,7cyclictest37-21ksoftirqd/307:10:003
1978739993425,6cyclictest37-21ksoftirqd/307:45:013
1978739993423,7cyclictest37-21ksoftirqd/311:45:013
1978739993423,7cyclictest37-21ksoftirqd/311:45:013
197873499343,4cyclictest131rcu_preempt07:45:002
1978734993426,5cyclictest306-21in:imuxsock08:25:002
1978734993425,5cyclictest1985501-21sendmail-msp09:20:002
1978734993425,5cyclictest1980888-21latency_hist07:50:022
1978734993424,8cyclictest30-21ksoftirqd/210:50:012
1978734993423,8cyclictest1988052-21sshd10:07:292
1978734993422,9cyclictest30-21ksoftirqd/211:05:012
1978734993422,9cyclictest30-21ksoftirqd/207:55:022
1978734993422,9cyclictest30-21ksoftirqd/207:55:012
1978734993422,10cyclictest30-21ksoftirqd/210:41:282
1978734993420,11cyclictest30-21ksoftirqd/209:50:002
1978734993420,11cyclictest272-21dbus-daemon10:30:012
1978734993420,11cyclictest272-21dbus-daemon10:30:012
1978734993420,11cyclictest235-21systemd-journal10:35:002
1978730993414,16cyclictest1978663-21latency_hist07:10:011
197872499341,6cyclictest1983876-21sendmail-mta08:45:250
1978734993326,4cyclictest1986432-21latency_hist09:35:022
1978734993323,7cyclictest30-21ksoftirqd/209:25:012
1978734993321,9cyclictest1985147-21sed09:10:022
1978734993225,5cyclictest30-21ksoftirqd/212:30:022
1978734993225,5cyclictest30-21ksoftirqd/212:30:022
1978734993220,10cyclictest30-21ksoftirqd/212:24:292
197873499313,4cyclictest131rcu_preempt09:05:012
197873499311,4cyclictest131rcu_preempt12:11:292
197873499311,3cyclictest131rcu_preempt07:10:012
197873499303,3cyclictest131rcu_preempt12:05:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional