You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 19:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Fri Nov 28, 2025 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
374634997038,27cyclictest1-21systemd10:45:001
374634996829,35cyclictest383460-21latency_hist10:00:021
374634996829,35cyclictest383460-21latency_hist10:00:021
37463499662,15cyclictest381374-21munin-run09:20:011
37463499652,15cyclictest376410-21munin-run07:45:001
37463499643,18cyclictest380021-21sh08:55:001
37463499643,14cyclictest388154-21munin-run11:30:001
374644996321,26cyclictest361rcuc/310:10:013
37463499633,15cyclictest390730-21modprobe12:20:001
37463499632,16cyclictest388699-21munin-run11:40:011
37463499632,14cyclictest389190-21munin-run11:50:001
374634996228,29cyclictest1-21systemd12:25:011
37463499612,17cyclictest385581-21munin-run10:40:011
37463499612,17cyclictest385581-21munin-run10:40:011
374644996036,18cyclictest1-21systemd10:50:003
37463499603,12cyclictest380275-21modprobe09:00:001
37463499603,12cyclictest380275-21modprobe09:00:001
37463499602,16cyclictest384511-21munin-run10:20:001
37463499601,14cyclictest387584-21sshd11:16:381
37463499592,16cyclictest386206-21cat10:50:021
37464499585,9cyclictest37-21ksoftirqd/312:00:003
374644995815,14cyclictest37-21ksoftirqd/307:25:013
374634995837,15cyclictest23-21ksoftirqd/109:40:021
37463499583,16cyclictest1-21systemd11:05:001
37463499582,13cyclictest374821-21cron07:15:001
37463499582,13cyclictest374821-21cron07:15:001
374634995821,32cyclictest389536-21latency_hist11:55:021
374634995821,32cyclictest389536-21latency_hist11:55:021
37463499581,16cyclictest378197-21cron08:20:001
37464499576,9cyclictest37-21ksoftirqd/310:05:013
374634995735,17cyclictest272-21dbus-daemon12:00:001
37463499572,14cyclictest391545-21ls12:35:011
374644995620,8cyclictest37-21ksoftirqd/312:40:003
37463499562,16cyclictest383221-21cat09:55:011
37463499562,15cyclictest379232-21cron08:40:001
374634995611,9cyclictest1-21systemd09:30:011
37463499561,15cyclictest235-21systemd-journal12:15:001
374630995634,18cyclictest12-21ksoftirqd/009:35:160
37463499552,16cyclictest388543-21grep11:35:021
37463499552,15cyclictest377756-21grep08:10:011
37463499552,13cyclictest375121-21sendmail-msp07:19:591
37463499552,11cyclictest384959-21cat10:25:021
374634995518,34cyclictest374670-21latency_hist07:10:021
37463499551,19cyclictest235-21systemd-journal07:30:011
37463499551,13cyclictest383739-21munin-run10:05:001
37463099553,3cyclictest131rcu_preempt07:35:000
374634995426,24cyclictest23-21ksoftirqd/108:45:011
374634995426,24cyclictest23-21ksoftirqd/108:45:011
374644995317,12cyclictest37-21ksoftirqd/310:55:023
37463699532,47cyclictest272-21dbus-daemon08:40:012
374634995333,12cyclictest1-21systemd09:15:011
37463499533,11cyclictest389966-21sh12:05:001
374634995325,25cyclictest23-21ksoftirqd/110:35:011
374634995316,5cyclictest391787-21/usr/sbin/munin12:40:021
37464499526,6cyclictest37-21ksoftirqd/310:30:013
37464499525,10cyclictest37-21ksoftirqd/309:20:003
374634995233,14cyclictest235-21systemd-journal07:55:001
37463499522,20cyclictest376667-21cut07:50:001
37463499521,16cyclictest391283-21/usr/sbin/munin12:30:011
37464499517,8cyclictest37-21ksoftirqd/308:05:013
37464499515,6cyclictest37-21ksoftirqd/311:00:013
374644995128,17cyclictest235-21systemd-journal09:50:003
374644995118,10cyclictest37-21ksoftirqd/312:30:023
37463699512,4cyclictest131rcu_preempt10:05:012
37463499514,11cyclictest0-21swapper/107:40:001
37463499513,13cyclictest235-21systemd-journal10:15:001
37463499513,13cyclictest235-21systemd-journal10:15:001
37463499512,15cyclictest387285-21cat11:10:021
37463499512,15cyclictest387285-21cat11:10:021
37463499512,14cyclictest377441-21latency_hist08:05:001
37463499512,14cyclictest375558-21latency_hist07:25:021
374634995120,27cyclictest23-21ksoftirqd/109:50:011
37464499507,9cyclictest37-21ksoftirqd/307:30:023
374644995032,14cyclictest272-21dbus-daemon10:20:003
37464499503,10cyclictest37-21ksoftirqd/309:55:023
37464499502,13cyclictest37-21ksoftirqd/308:45:003
37464499502,13cyclictest37-21ksoftirqd/308:45:003
37463499504,14cyclictest380657-21grep09:05:011
37463499502,14cyclictest390364-21wc12:10:021
37463499502,14cyclictest390364-21wc12:10:021
374634995015,11cyclictest383958-21sshd10:05:391
374630995034,11cyclictest1-21systemd08:00:010
374630995034,11cyclictest1-21systemd08:00:010
37464499494,15cyclictest37-21ksoftirqd/307:45:013
374644994930,14cyclictest380797-21cron09:10:003
37464499492,12cyclictest37-21ksoftirqd/311:50:013
374644994918,9cyclictest37-21ksoftirqd/311:25:013
37463499492,16cyclictest377192-21lockfile-create08:00:001
37463499492,16cyclictest377192-21lockfile-create08:00:001
374634994920,25cyclictest387899-21latency_hist11:25:031
374630994935,11cyclictest235-21systemd-journal10:40:000
374630994935,11cyclictest235-21systemd-journal10:40:000
374630994929,15cyclictest1-21systemd12:30:010
37463099491,26cyclictest131rcu_preempt07:39:370
37464499484,13cyclictest37-21ksoftirqd/307:20:003
37464499481,11cyclictest37-21ksoftirqd/309:15:003
374636994833,11cyclictest272-21dbus-daemon11:00:022
374634994823,22cyclictest376062-21latency_hist07:35:021
374634994814,7cyclictest378996-21modprobe08:35:001
374634994813,5cyclictest23-21ksoftirqd/110:57:491
37464499474,7cyclictest37-21ksoftirqd/309:25:013
37464499474,7cyclictest37-21ksoftirqd/308:55:003
374644994734,10cyclictest37-21ksoftirqd/311:20:023
37464499471,14cyclictest37-21ksoftirqd/310:25:023
374636994731,12cyclictest1-21systemd12:05:002
37463499475,12cyclictest1-21systemd09:35:011
37463499472,15cyclictest388938-21munin-run11:45:001
374630994729,14cyclictest272-21dbus-daemon07:15:010
374630994729,14cyclictest272-21dbus-daemon07:15:010
37463099471,2cyclictest0-21swapper/012:25:020
37464499468,9cyclictest37-21ksoftirqd/311:55:023
37464499468,9cyclictest37-21ksoftirqd/311:55:013
37464499465,6cyclictest37-21ksoftirqd/312:25:023
37464499465,5cyclictest0-21swapper/310:45:023
374644994638,5cyclictest37-21ksoftirqd/308:10:013
374636994630,11cyclictest272-21dbus-daemon10:45:002
37463699461,13cyclictest385291-21true10:35:002
37463499465,9cyclictest306-21in:imuxsock10:55:011
37463499462,14cyclictest380832-21munin-run09:10:011
37463499462,14cyclictest377950-21cut08:15:001
374634994618,5cyclictest1-21systemd09:45:011
37463099465,32cyclictest235-21systemd-journal10:20:000
374630994615,9cyclictest12-21ksoftirqd/010:55:030
37464499457,8cyclictest37-21ksoftirqd/311:10:023
37464499457,8cyclictest37-21ksoftirqd/311:10:023
37464499456,9cyclictest37-21ksoftirqd/308:50:013
37464499455,7cyclictest37-21ksoftirqd/308:17:013
37464499454,10cyclictest37-21ksoftirqd/312:15:003
37464499454,10cyclictest37-21ksoftirqd/311:35:023
37464499451,7cyclictest0-21swapper/310:00:023
37464499451,7cyclictest0-21swapper/310:00:013
374636994527,13cyclictest272-21dbus-daemon08:35:002
374636994515,21cyclictest375664-21latency_hist07:30:022
374634994515,8cyclictest23-21ksoftirqd/108:30:021
374634994515,8cyclictest23-21ksoftirqd/108:30:021
374630994533,8cyclictest0-21swapper/008:05:000
374630994528,14cyclictest387608-21cron11:20:000
37464499446,24cyclictest380276-21modprobe09:00:003
37464499446,24cyclictest380276-21modprobe09:00:003
37464499445,9cyclictest37-21ksoftirqd/307:40:003
37464499443,7cyclictest37-21ksoftirqd/307:10:023
37464499443,12cyclictest37-21ksoftirqd/308:40:003
374636994429,11cyclictest235-21systemd-journal10:46:392
374636994427,13cyclictest235-21systemd-journal09:52:392
374634994428,11cyclictest235-21systemd-journal08:25:011
37463499441,15cyclictest381611-21modprobe09:25:001
374630994437,4cyclictest16950irq/224-800000008:45:020
374630994437,4cyclictest16950irq/224-800000008:45:020
374630994435,5cyclictest378211-21latency_hist08:20:000
374630994426,14cyclictest408-21lldpd09:05:000
37464499437,9cyclictest37-21ksoftirqd/312:10:003
37464499437,9cyclictest37-21ksoftirqd/312:10:003
37464499433,9cyclictest37-21ksoftirqd/308:35:013
374644994334,6cyclictest37-21ksoftirqd/308:15:013
37464499432,9cyclictest37-21ksoftirqd/311:05:003
37464499432,8cyclictest37-21ksoftirqd/308:29:063
37464499432,8cyclictest37-21ksoftirqd/308:29:063
374636994325,13cyclictest272-21dbus-daemon07:15:012
374636994325,13cyclictest272-21dbus-daemon07:15:002
37463499432,14cyclictest379788-21munin-run08:50:011
37463099433,3cyclictest131rcu_preempt07:20:010
37463099433,18cyclictest131rcu_preempt11:00:010
374630994330,9cyclictest307-21in:imklog11:55:030
374630994330,9cyclictest307-21in:imklog11:55:020
374630994327,10cyclictest272-21dbus-daemon09:20:010
37463099431,39cyclictest0-21swapper/011:40:010
37464499427,8cyclictest37-21ksoftirqd/312:20:013
37464499425,8cyclictest37-21ksoftirqd/308:21:393
37464499425,7cyclictest37-21ksoftirqd/310:30:033
37464499424,7cyclictest37-21ksoftirqd/311:30:023
374644994217,21cyclictest361rcuc/311:14:443
37464499421,10cyclictest381973-21sed09:30:023
374636994228,10cyclictest306-21in:imuxsock11:40:002
374636994227,11cyclictest30-21ksoftirqd/207:50:012
374636994222,14cyclictest1-21systemd11:45:002
37463699421,3cyclictest131rcu_preempt08:00:002
37463699421,3cyclictest131rcu_preempt08:00:002
374636994212,11cyclictest30-21ksoftirqd/209:40:012
37463499422,13cyclictest387347-21modprobe11:14:591
374630994235,4cyclictest16950irq/224-800000012:10:020
374630994235,4cyclictest16950irq/224-800000012:10:020
374630994231,8cyclictest235-21systemd-journal09:48:400
374630994227,11cyclictest387888-21cron11:25:000
37463099421,22cyclictest0-21swapper/010:40:020
37464499418,5cyclictest131rcu_preempt09:05:003
37464499416,6cyclictest37-21ksoftirqd/312:35:013
37464499414,8cyclictest37-21ksoftirqd/309:35:153
374644994125,12cyclictest37-21ksoftirqd/311:45:003
374636994132,5cyclictest382970-21latency_hist09:50:012
374636994128,10cyclictest30-21ksoftirqd/211:05:012
37463699412,3cyclictest131rcu_preempt11:50:002
374636994121,12cyclictest380570-21systemd-run09:05:002
37463099414,33cyclictest389701-21modprobe12:00:000
374630994130,6cyclictest376912-21munin-plugin-st07:55:000
374630994127,10cyclictest235-21systemd-journal10:30:010
37463099412,33cyclictest381192-21latency_hist09:15:020
374630994120,16cyclictest235-21systemd-journal11:10:010
374630994120,16cyclictest235-21systemd-journal11:10:010
37463099411,4cyclictest131rcu_preempt11:45:010
37464499403,8cyclictest37-21ksoftirqd/310:40:023
37464499403,8cyclictest37-21ksoftirqd/310:40:023
37464499403,7cyclictest37-21ksoftirqd/310:15:013
37464499403,7cyclictest37-21ksoftirqd/310:15:013
37464499403,10cyclictest37-21ksoftirqd/311:38:383
374644994023,13cyclictest272-21dbus-daemon07:35:003
37464499401,8cyclictest0-21swapper/307:50:373
37464499401,10cyclictest363-21snmpd12:05:003
374636994026,10cyclictest391784-21systemctl12:40:002
374636994025,12cyclictest390715-21sshd12:17:382
374636994024,12cyclictest235-21systemd-journal11:35:012
37463699401,3cyclictest131rcu_preempt08:55:002
37463099403,6cyclictest382131-21unin-run09:35:000
374630994033,4cyclictest16950irq/224-800000009:30:010
374630994032,5cyclictest16950irq/224-800000008:55:000
374630994029,8cyclictest12-21ksoftirqd/009:00:000
374630994029,8cyclictest12-21ksoftirqd/009:00:000
374630994029,6cyclictest306-21in:imuxsock08:15:010
374630994028,9cyclictest384980-21latency_hist10:25:020
374644993930,5cyclictest1-21systemd07:50:003
374644993927,9cyclictest37-21ksoftirqd/309:32:393
37463699394,29cyclictest378208-21cron08:20:002
374636993927,10cyclictest30-21ksoftirqd/210:10:002
374636993926,9cyclictest235-21systemd-journal08:05:012
374636993924,11cyclictest391023-21systemd-run12:25:012
37463699391,5cyclictest131rcu_preempt07:40:002
374630993925,9cyclictest306-21in:imuxsock12:30:020
374630993925,10cyclictest235-21systemd-journal10:10:000
374630993925,10cyclictest12-21ksoftirqd/007:10:020
374630993921,14cyclictest1-21systemd11:50:010
37464499386,7cyclictest37-21ksoftirqd/307:15:013
37464499386,7cyclictest37-21ksoftirqd/307:15:013
37464499382,12cyclictest37-21ksoftirqd/307:59:373
37464499382,12cyclictest37-21ksoftirqd/307:59:373
37463699382,4cyclictest131rcu_preempt08:15:002
374636993824,10cyclictest30-21ksoftirqd/210:40:002
374636993824,10cyclictest30-21ksoftirqd/210:40:002
374636993819,14cyclictest272-21dbus-daemon07:20:012
374630993827,8cyclictest12-21ksoftirqd/011:15:010
374630993826,7cyclictest383443-21cron10:00:000
374630993826,7cyclictest383443-21cron10:00:000
374630993825,10cyclictest381582-21sshd09:20:390
37463699373,3cyclictest131rcu_preempt12:14:382
374636993728,7cyclictest30-21ksoftirqd/209:25:002
374636993727,7cyclictest379490-21sshd08:40:072
374636993727,7cyclictest379490-21sshd08:40:072
374636993725,8cyclictest386286-21sshd10:50:282
374636993722,10cyclictest30-21ksoftirqd/211:20:012
37463099373,3cyclictest131rcu_preempt08:30:010
37463099373,3cyclictest131rcu_preempt08:30:010
374630993730,5cyclictest16950irq/224-800000008:38:390
374630993727,7cyclictest151rcuc/011:35:000
374630993725,9cyclictest12-21ksoftirqd/010:15:000
374630993725,9cyclictest12-21ksoftirqd/010:15:000
374630993723,9cyclictest235-21systemd-journal09:53:390
374630993723,9cyclictest12-21ksoftirqd/012:20:010
37463099371,33cyclictest385323-21cut10:35:010
37463699365,3cyclictest131rcu_preempt08:50:002
374636993626,8cyclictest30-21ksoftirqd/212:35:022
374636993626,8cyclictest30-21ksoftirqd/211:25:022
374636993622,11cyclictest235-21systemd-journal09:30:022
37463099362,8cyclictest1-21systemd11:28:390
374630993626,7cyclictest12-21ksoftirqd/011:05:010
374630993625,9cyclictest12-21ksoftirqd/012:40:010
37463099361,10cyclictest235-21systemd-journal08:48:350
374644993527,5cyclictest361rcuc/309:43:393
37463699352,5cyclictest384790-21systemd-run10:25:012
374636993525,8cyclictest30-21ksoftirqd/210:15:022
374636993525,8cyclictest30-21ksoftirqd/210:15:022
374636993525,6cyclictest378476-21cron08:25:002
374636993524,7cyclictest388125-21sshd11:26:382
37463699352,3cyclictest131rcu_preempt07:55:002
374636993523,8cyclictest30-21ksoftirqd/212:00:002
374636993519,10cyclictest235-21systemd-journal12:27:382
37463699351,3cyclictest131rcu_preempt09:45:002
37463099352,29cyclictest235-21systemd-journal07:25:300
374630993522,10cyclictest382757-21wc09:45:020
374630993522,10cyclictest272-21dbus-daemon12:05:010
37463699345,16cyclictest381330-21modprobe09:20:002
37463699344,3cyclictest131rcu_preempt08:10:002
374636993425,6cyclictest382316-21latency_hist09:35:022
374636993423,9cyclictest30-21ksoftirqd/211:55:022
374636993423,9cyclictest30-21ksoftirqd/211:55:012
374636993420,9cyclictest380292-21readlink09:00:002
374636993420,9cyclictest380292-21readlink09:00:002
374636993419,12cyclictest272-21dbus-daemon07:45:012
374630993429,3cyclictest0-21swapper/007:21:380
374630993418,12cyclictest235-21systemd-journal10:05:000
37463099341,31cyclictest0-21swapper/010:50:010
37463099341,30cyclictest0-21swapper/012:10:190
37463099341,30cyclictest0-21swapper/008:32:420
374636993327,4cyclictest387357-21munin-plugin-st11:15:002
374636993326,4cyclictest380811-21latency_hist09:10:022
374636993324,7cyclictest0-21swapper/210:00:022
374636993324,7cyclictest0-21swapper/210:00:022
374636993322,8cyclictest272-21dbus-daemon07:25:012
374636993320,9cyclictest1-21systemd07:35:002
374636993320,10cyclictest30-21ksoftirqd/207:10:022
374636993319,11cyclictest390190-21sshd12:05:392
374636993319,11cyclictest390190-21sshd12:05:382
374636993318,12cyclictest378708-21sshd08:25:392
374636993318,12cyclictest378708-21sshd08:25:392
374634993324,5cyclictest306-21in:imuxsock10:26:401
37463099339,21cyclictest380897-21latency_hist09:10:020
374630993327,4cyclictest0-21swapper/008:21:360
374630993317,13cyclictest376387-21sshd07:43:370
37463099331,30cyclictest0-21swapper/007:49:430
37463699324,15cyclictest384499-21modprobe10:20:002
374636993224,5cyclictest387214-21latency_hist11:10:022
374636993224,5cyclictest387214-21latency_hist11:10:022
37463699321,3cyclictest131rcu_preempt09:15:002
374630993219,10cyclictest12-21ksoftirqd/008:10:000
37463699304,3cyclictest131rcu_preempt10:25:392
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional