You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 05:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Tue Jul 01, 2025 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2667616998980,7cyclictest23-21ksoftirqd/123:20:011
266761299752,69cyclictest2669391-21pluginstate19:45:020
266761299752,69cyclictest2669391-21pluginstate19:45:010
2667612996927,38cyclictest2670678-21systemd-run20:10:010
2667623996626,36cyclictest2679327-21sed22:55:023
2667612996624,37cyclictest235-21systemd-journal00:17:470
2667612996521,40cyclictest2674813-21sshd21:28:070
2667623996421,38cyclictest272-21dbus-daemon19:25:013
2667623996326,33cyclictest2681584-21sshd23:35:463
2667623996323,36cyclictest412-21sshd20:38:473
2667623996322,36cyclictest37-21ksoftirqd/320:55:023
2667623996320,37cyclictest306-21in:imuxsock22:45:003
2667616996355,6cyclictest23-21ksoftirqd/120:40:011
2667612996336,23cyclictest19050irq/34-mmc123:15:020
266761299631,13cyclictest2672460-21sshd20:41:470
2667612996232,26cyclictest16950irq/224-800000020:00:000
2667612996225,33cyclictest2681883-21sshd23:44:470
2667612996225,33cyclictest2681883-21sshd23:44:470
2667623996123,34cyclictest2676198-21latency_hist21:55:023
2667612996114,41cyclictest308-21rs:main0
266761299611,13cyclictest363-21snmpd20:16:580
266761299602,12cyclictest2683196-21munin-run00:10:000
266761299602,12cyclictest2678986-21munin-run22:50:010
2667612996019,37cyclictest12-21ksoftirqd/019:15:010
2667612996019,37cyclictest12-21ksoftirqd/019:15:010
266761299601,56cyclictest0-21swapper/022:05:020
2667623995924,30cyclictest2682774-21sed00:00:023
2667623995924,30cyclictest2682774-21sed00:00:023
266761299592,13cyclictest2668324-21sshd19:23:450
2667616995829,21cyclictest2673787-21sendmail21:10:001
2667612995825,29cyclictest363-21snmpd20:55:580
2667612995816,36cyclictest0-21swapper/019:20:000
2667623995724,29cyclictest2669102-21latency_hist19:40:023
266762399572,18cyclictest235-21systemd-journal19:19:443
266762399572,16cyclictest2667827-21wc19:15:023
266762399572,16cyclictest2667827-21wc19:15:023
2667623995713,8cyclictest37-21ksoftirqd/323:51:473
266761299572,5cyclictest2674304-21modprobe21:20:000
266761299572,16cyclictest2680047-21sshd23:08:170
2667623995615,6cyclictest0-21swapper/300:33:353
266761699566,26cyclictest2681612-21modprobe23:40:011
2667616995634,17cyclictest235-21systemd-journal20:20:001
2667612995622,30cyclictest2673495-21sshd21:00:470
266761299562,15cyclictest2668815-21sshd19:31:440
266761299561,5cyclictest2677947-21latency_hist22:30:020
266761299561,12cyclictest363-21snmpd22:59:360
2667623995521,9cyclictest37-21ksoftirqd/322:00:013
2667623995521,9cyclictest37-21ksoftirqd/322:00:013
2667623995517,7cyclictest37-21ksoftirqd/300:26:203
2667623995516,7cyclictest37-21ksoftirqd/322:20:013
266761299552,12cyclictest2670930-21munin-run20:15:010
266761299551,13cyclictest363-21snmpd20:46:290
266761299551,13cyclictest363-21snmpd20:46:290
2667623995422,28cyclictest1-21systemd20:25:003
2667623995420,10cyclictest37-21ksoftirqd/323:25:013
266762399541,16cyclictest2667691-21cat19:10:023
2667612995431,17cyclictest2672202-21cron20:40:000
2667612995424,27cyclictest19050irq/34-mmc120:55:010
266761299542,14cyclictest2675662-21date21:45:020
266761299542,14cyclictest2675662-21date21:45:020
266761299542,13cyclictest2684760-21readlink00:40:000
266761299541,49cyclictest48-21kcompactd021:24:200
266762399535,17cyclictest2673788-21munin-run21:10:003
2667623995328,22cyclictest37-21ksoftirqd/319:50:013
2667623995322,27cyclictest2676077-21latency_hist21:50:023
2667623995318,4cyclictest412-21sshd19:55:443
2667623995314,8cyclictest2677153-21modprobe22:15:003
266761299532,17cyclictest2676363-21cron22:00:000
266761299532,17cyclictest2676363-21cron22:00:000
266761299532,13cyclictest2677463-21munin-run22:20:010
266761299532,13cyclictest2673743-21sshd21:05:470
2667612995319,29cyclictest235-21systemd-journal22:40:000
2667612995316,9cyclictest12-21ksoftirqd/000:05:000
2667612995312,6cyclictest272-21dbus-daemon21:40:000
266761299531,13cyclictest363-21snmpd23:51:530
2667623995233,16cyclictest2682150-21munin-plugin-st23:50:003
266762399522,15cyclictest2674974-21cat21:30:023
266762399522,15cyclictest2673258-21grep21:00:003
2667623995215,5cyclictest2679491-21sendmail23:00:003
2667623995215,11cyclictest37-21ksoftirqd/300:10:013
2667612995225,23cyclictest2681665-21latency_hist23:40:010
266761299522,46cyclictest2682671-40ogrotate00:00:010
266761299522,46cyclictest2682671-40ogrotate00:00:000
2667612995223,26cyclictest2667719-21latency_hist19:10:020
266761299521,13cyclictest1-21systemd20:35:020
266761299521,12cyclictest363-21snmpd23:31:280
266761299521,12cyclictest2680831-21sshd23:21:470
266762399513,16cyclictest2684750-21latency_hist00:40:003
266762399513,15cyclictest2675354-21modprobe21:40:003
266762399512,19cyclictest2674293-21cron21:20:003
2667623995115,5cyclictest235-21systemd-journal21:05:003
2667621995137,9cyclictest2668051-21cron19:20:002
2667621995135,12cyclictest2684755-21systemctl00:40:002
266761699516,30cyclictest2674295-21modprobe21:20:001
2667616995133,12cyclictest306-21in:imuxsock00:30:011
2667616995130,15cyclictest272-21dbus-daemon21:35:011
266761299512,13cyclictest2674059-21munin-run21:15:010
2667612995119,29cyclictest12-21ksoftirqd/022:25:000
266761299511,46cyclictest0-21swapper/019:37:490
266761299511,12cyclictest363-21snmpd23:47:120
266762399503,14cyclictest2682005-21latency_hist23:45:023
266762399503,14cyclictest2682005-21latency_hist23:45:023
266762399503,10cyclictest2683980-21modprobe00:25:003
2667623995027,20cyclictest37-21ksoftirqd/300:15:023
2667616995030,13cyclictest235-21systemd-journal20:35:011
2667612995022,24cyclictest408-21lldpd22:34:380
266761299502,16cyclictest2679732-21sshd23:00:460
266761299502,13cyclictest2675076-21cron21:35:010
2667612995020,26cyclictest16950irq/224-800000020:20:500
266761299501,45cyclictest0-21swapper/022:10:020
266761299501,15cyclictest0-21swapper/021:51:090
266761299501,13cyclictest363-21snmpd00:33:180
266762399492,5cyclictest131rcu_preempt21:25:013
2667616994911,9cyclictest131rcu_preempt20:50:011
2667616994911,9cyclictest131rcu_preempt20:50:011
2667612994930,13cyclictest235-21systemd-journal22:15:000
266761299491,12cyclictest363-21snmpd21:49:400
2667612994910,3cyclictest16850irq/134-800000023:19:470
266762399484,12cyclictest2670666-21munin-run20:10:003
2667623994821,9cyclictest37-21ksoftirqd/320:20:023
2667623994816,4cyclictest2679083-21latency_hist22:50:023
2667623994815,4cyclictest2681330-21kworker/u8:223:34:593
2667623994814,8cyclictest235-21systemd-journal19:35:003
2667616994828,14cyclictest235-21systemd-journal20:10:001
2667616994819,19cyclictest0-21swapper/120:00:001
266761299484,9cyclictest2683446-21munin-plugin-st00:15:000
266761299481,4cyclictest0-21swapper/019:50:080
266761299481,12cyclictest2684252-21systemd-run00:30:010
2667612994810,9cyclictest281-21systemd-logind22:55:010
266762399472,19cyclictest2672746-21munin-run20:50:003
266762399472,19cyclictest2672746-21munin-run20:50:003
2667623994719,8cyclictest37-21ksoftirqd/323:01:583
2667623994717,7cyclictest37-21ksoftirqd/323:15:013
2667621994735,8cyclictest1-21systemd00:00:012
2667621994735,8cyclictest1-21systemd00:00:012
2667616994734,8cyclictest307-21in:imklog23:58:231
2667616994734,8cyclictest307-21in:imklog23:58:221
2667616994732,11cyclictest235-21systemd-journal20:15:001
2667612994714,4cyclictest0-21swapper/019:48:440
2667621994634,9cyclictest2669108-21systemctl19:40:002
2667616994632,10cyclictest272-21dbus-daemon23:07:471
2667616994622,15cyclictest272-21dbus-daemon19:35:001
266761299461,16cyclictest0-21swapper/000:24:100
2667612994610,7cyclictest2681325-21date23:30:030
2667623994531,11cyclictest235-21systemd-journal00:02:473
266762399452,14cyclictest2669867-21cron19:55:003
2667623994519,10cyclictest37-21ksoftirqd/322:35:003
2667621994532,10cyclictest30-21ksoftirqd/220:35:022
2667621994527,12cyclictest272-21dbus-daemon19:50:002
266761699455,25cyclictest2676906-21munin-run22:10:001
2667616994526,13cyclictest306-21in:imuxsock23:30:021
2667616994526,13cyclictest306-21in:imuxsock23:30:011
2667616994525,13cyclictest272-21dbus-daemon00:40:011
2667616994525,12cyclictest2675349-21systemctl21:40:001
266761299452,12cyclictest2681208-21latency_hist23:30:020
2667623994413,9cyclictest37-21ksoftirqd/321:35:023
2667621994433,7cyclictest30-21ksoftirqd/223:35:022
2667621994432,9cyclictest2682916-21sshd00:02:472
2667621994428,12cyclictest235-21systemd-journal20:45:012
2667621994428,11cyclictest1-21systemd22:05:002
266762199441,3cyclictest131rcu_preempt19:15:022
266762199441,3cyclictest131rcu_preempt19:15:022
2667616994430,9cyclictest1-21systemd22:30:011
2667616994430,11cyclictest23-21ksoftirqd/120:25:001
2667616994429,11cyclictest235-21systemd-journal23:15:001
2667616994428,10cyclictest306-21in:imuxsock19:20:021
2667616994423,15cyclictest235-21systemd-journal00:34:591
2667616994422,14cyclictest272-21dbus-daemon00:05:011
2667616994421,16cyclictest235-21systemd-journal22:25:011
266761299441,2cyclictest0-21swapper/019:30:020
266761299441,11cyclictest272-21dbus-daemon20:30:010
2667621994328,11cyclictest30-21ksoftirqd/223:49:592
2667621994324,14cyclictest272-21dbus-daemon23:30:012
2667621994324,14cyclictest272-21dbus-daemon23:30:012
2667616994327,12cyclictest23-21ksoftirqd/123:00:001
2667616994325,11cyclictest306-21in:imuxsock19:20:011
2667616994324,13cyclictest235-21systemd-journal23:50:001
2667616994314,6cyclictest131rcu_preempt20:45:011
266762399427,9cyclictest37-21ksoftirqd/319:40:303
266762399427,9cyclictest37-21ksoftirqd/319:40:303
266762399421,4cyclictest131rcu_preempt20:45:003
2667621994230,9cyclictest306-21in:imuxsock23:40:002
2667621994229,7cyclictest306-21in:imuxsock19:35:002
2667621994227,12cyclictest2669460-21sed19:45:022
2667621994227,12cyclictest2669460-21sed19:45:022
2667621994221,16cyclictest2673769-21cron21:10:002
266761699425,25cyclictest2669088-21modprobe19:40:001
2667616994230,8cyclictest235-21systemd-journal22:37:461
2667612994218,20cyclictest0-21swapper/020:00:440
266762399412,4cyclictest131rcu_preempt21:45:023
266762399412,4cyclictest131rcu_preempt21:45:023
2667621994128,9cyclictest272-21dbus-daemon21:00:012
2667621994128,8cyclictest30-21ksoftirqd/223:55:022
2667621994128,10cyclictest30-21ksoftirqd/220:25:002
2667621994126,12cyclictest2680601-21tr23:19:592
2667621994126,10cyclictest2672224-21modprobe20:40:002
2667621994122,14cyclictest1-21systemd00:25:012
2667616994128,8cyclictest2678201-21munin-plugin-st22:34:591
2667616994127,8cyclictest306-21in:imuxsock00:20:011
2667616994126,11cyclictest23-21ksoftirqd/122:00:031
2667616994126,11cyclictest23-21ksoftirqd/122:00:021
2667616994122,13cyclictest272-21dbus-daemon23:25:001
266762399406,5cyclictest131rcu_preempt20:35:023
2667621994027,9cyclictest30-21ksoftirqd/219:55:032
2667621994021,12cyclictest0-21swapper/221:05:002
266762199401,35cyclictest30-21ksoftirqd/220:55:002
2667616994032,5cyclictest2669845-21sshd19:50:441
2667616994026,11cyclictest2669616-21cron19:50:001
2667623993931,5cyclictest2676904-21munin-plugin-st22:10:003
2667623993927,10cyclictest37-21ksoftirqd/322:25:013
2667621993929,6cyclictest271-21cron21:40:002
2667621993925,11cyclictest30-21ksoftirqd/222:20:012
2667621993925,11cyclictest30-21ksoftirqd/219:29:442
2667621993924,10cyclictest30-21ksoftirqd/220:17:472
2667621993924,10cyclictest1-21systemd23:15:012
266762199392,3cyclictest131rcu_preempt00:35:022
2667621993923,11cyclictest30-21ksoftirqd/222:25:002
2667621993922,14cyclictest2670149-21tr20:00:002
2667616993923,10cyclictest2674188-21ls21:15:031
266761699391,34cyclictest23-21ksoftirqd/121:30:031
2667623993823,10cyclictest37-21ksoftirqd/321:15:013
2667623993820,13cyclictest272-21dbus-daemon20:30:013
266762399381,6cyclictest0-21swapper/323:30:033
266762399381,6cyclictest0-21swapper/323:30:023
2667621993829,6cyclictest2679375-21cat22:55:012
2667621993827,9cyclictest30-21ksoftirqd/222:30:022
2667621993826,8cyclictest306-21in:imuxsock22:10:022
2667621993826,8cyclictest30-21ksoftirqd/200:30:022
2667621993825,9cyclictest272-21dbus-daemon20:10:012
2667621993825,9cyclictest235-21systemd-journal22:41:482
2667616993822,10cyclictest235-21systemd-journal22:19:591
2667616993822,10cyclictest235-21systemd-journal19:15:001
2667616993822,10cyclictest235-21systemd-journal19:15:001
2667616993821,12cyclictest2679764-21cron23:05:011
2667623993722,12cyclictest37-21ksoftirqd/322:00:033
2667621993727,7cyclictest30-21ksoftirqd/221:55:012
2667621993727,7cyclictest30-21ksoftirqd/220:45:022
2667621993727,7cyclictest30-21ksoftirqd/220:45:022
2667621993726,8cyclictest30-21ksoftirqd/222:40:002
2667621993724,8cyclictest306-21in:imuxsock20:12:572
266761699374,24cyclictest2678735-21munin-run22:45:001
266761699374,23cyclictest2683197-21grep00:10:001
2667616993726,8cyclictest0-21swapper/121:45:011
2667616993726,8cyclictest0-21swapper/121:45:011
2667616993722,8cyclictest2672185-21kworker/u8:021:00:001
2667616993715,14cyclictest2674581-21modprobe21:25:001
2667616993714,15cyclictest235-21systemd-journal00:15:001
266762399363,3cyclictest131rcu_preempt20:03:443
266762399363,3cyclictest131rcu_preempt00:20:013
2667623993624,6cyclictest306-21in:imuxsock22:40:003
2667621993626,8cyclictest30-21ksoftirqd/221:35:002
2667621993622,11cyclictest272-21dbus-daemon23:07:502
2667616993628,5cyclictest2681898-21munin-plugin-st23:45:011
2667616993628,5cyclictest2681898-21munin-plugin-st23:45:001
2667616993619,13cyclictest235-21systemd-journal20:55:001
266762399353,3cyclictest131rcu_preempt22:29:483
2667623993526,6cyclictest37-21ksoftirqd/323:15:173
2667621993527,6cyclictest30-21ksoftirqd/222:50:012
2667621993527,5cyclictest2679503-21sendmail23:00:012
2667621993525,5cyclictest306-21in:imuxsock23:25:002
266762199352,3cyclictest131rcu_preempt20:30:002
2667621993520,11cyclictest2683432-21sshd00:14:482
266761699355,21cyclictest2670551-21grep20:05:031
266761699355,20cyclictest2678979-21modprobe22:50:011
2667616993525,6cyclictest306-21in:imuxsock19:26:441
2667616993523,8cyclictest2684040-21date00:25:011
2667616993521,9cyclictest272-21dbus-daemon22:55:011
266761699351,5cyclictest131rcu_preempt23:34:591
266761699351,3cyclictest131rcu_preempt22:05:011
2667623993422,9cyclictest272-21dbus-daemon19:30:013
2667621993426,5cyclictest861385-21sendmail-mta21:14:202
2667621993426,4cyclictest2676352-21sshd21:58:112
2667621993426,4cyclictest2676352-21sshd21:58:102
2667621993425,7cyclictest30-21ksoftirqd/219:10:022
266762199342,27cyclictest2683198-21cut00:10:002
2667621993418,12cyclictest1-21systemd00:20:002
266762199341,4cyclictest131rcu_preempt22:10:002
266761699346,5cyclictest131rcu_preempt23:53:471
266761699342,25cyclictest2673529-21munin-plugin-st21:05:011
2667616993420,11cyclictest23-21ksoftirqd/120:30:021
2667616993417,12cyclictest235-21systemd-journal19:40:021
2667616993417,12cyclictest235-21systemd-journal19:40:021
2667623993321,9cyclictest37-21ksoftirqd/323:07:483
266761699335,24cyclictest2676051-21wc21:50:021
2667621993226,4cyclictest2678330-21latency_hist22:35:012
2667621993224,5cyclictest2670428-21latency_hist20:05:022
2667621993224,4cyclictest2674294-21cron21:20:002
2667621993217,11cyclictest235-21systemd-journal21:41:472
2667621993217,11cyclictest235-21systemd-journal21:41:472
266761699324,3cyclictest131rcu_preempt21:55:021
266762199314,4cyclictest131rcu_preempt21:30:012
2667621993124,4cyclictest2675869-21latency_hist21:50:012
2667621993124,4cyclictest2674584-21latency_hist21:25:022
2667621993118,10cyclictest30-21ksoftirqd/223:41:552
2667621993118,10cyclictest30-21ksoftirqd/223:41:552
2667621993117,11cyclictest2668441-21wc19:25:022
2667621993110,13cyclictest2679755-21cron23:05:002
2667616993122,5cyclictest23-21ksoftirqd/122:15:011
266761699311,5cyclictest131rcu_preempt19:10:011
266762399289,13cyclictest2670909-21modprobe20:15:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional