You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 06:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Nov 27, 2025 00:45:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
189697991085,3cyclictest131rcu_preempt19:55:013
189683991071,102cyclictest0-21swapper/022:30:030
189697997327,40cyclictest306-21in:imuxsock22:05:013
189683997117,31cyclictest3900308-1kworker/0:1H19:15:020
18969799703,16cyclictest189877-21munin-run19:15:003
18969799652,15cyclictest195346-21munin-run21:00:003
18969799652,14cyclictest194577-21munin-run20:45:013
189697996512,4cyclictest0-21swapper/320:02:253
189697996424,36cyclictest37-21ksoftirqd/320:49:193
189697996422,38cyclictest37-21ksoftirqd/322:15:003
18969799642,15cyclictest203400-21munin-run23:35:003
18969799642,14cyclictest190431-21munin-run19:25:003
189697996323,33cyclictest200177-21cat22:30:033
18969799632,16cyclictest202901-21true23:25:013
18969799632,16cyclictest202901-21true23:25:003
189697996321,37cyclictest196184-21latency_hist21:15:013
189683996319,10cyclictest12-21ksoftirqd/022:00:000
18968399629,15cyclictest197317-21sed21:35:010
189683996218,9cyclictest12-21ksoftirqd/020:20:000
189683996217,9cyclictest12-21ksoftirqd/020:40:000
18969799612,14cyclictest194284-21cron20:39:593
189697996043,11cyclictest1-21systemd23:59:593
189697996026,30cyclictest198460-21sendmail22:00:003
18969799602,12cyclictest235-21systemd-journal22:40:013
18969799602,12cyclictest235-21systemd-journal22:40:003
18968399608,35cyclictest0-21swapper/000:16:300
189683996028,12cyclictest0-21swapper/021:30:010
189683996028,12cyclictest0-21swapper/021:30:010
189683996015,9cyclictest12-21ksoftirqd/020:25:010
189693995916,10cyclictest30-21ksoftirqd/221:40:002
189683995925,4cyclictest197431-21munin-run21:40:000
189683995918,8cyclictest12-21ksoftirqd/021:19:590
189697995831,23cyclictest306-21in:imuxsock21:01:183
18969799582,13cyclictest199793-21munin-run22:25:013
189697995821,33cyclictest37-21ksoftirqd/321:45:023
189697995821,33cyclictest37-21ksoftirqd/321:45:013
189683995815,8cyclictest12-21ksoftirqd/020:15:010
189683995810,19cyclictest195867-21latency_hist21:10:020
18969799572,16cyclictest205283-21munin-run00:10:003
18969799572,15cyclictest201335-21munin-run22:55:023
18969799572,14cyclictest190938-21munin-run19:35:023
189697995714,10cyclictest197433-21grep21:40:013
189693995739,13cyclictest1-21systemd22:25:002
189697995623,29cyclictest205787-21latency_hist00:20:013
18969799562,16cyclictest190684-21munin-run19:30:013
18969799562,15cyclictest194044-21munin-run20:35:003
18969799562,15cyclictest194044-21munin-run20:35:003
189688995617,22cyclictest0-21swapper/120:50:031
18968399565,46cyclictest206168-21latency_hist00:25:010
18969799559,9cyclictest0-21swapper/321:10:003
18969799559,7cyclictest235-21systemd-journal00:34:173
18969799553,16cyclictest190167-21sendmail-msp19:20:003
18969799553,10cyclictest203150-21grep23:29:593
189697995521,29cyclictest37-21ksoftirqd/323:55:023
189697995511,6cyclictest0-21swapper/322:20:013
18969799545,12cyclictest272-21dbus-daemon21:50:013
18969799543,25cyclictest202625-21munin-run23:20:013
18969799537,7cyclictest0-21swapper/300:30:023
18969799532,8cyclictest193788-21modprobe20:30:003
189697995316,4cyclictest205013-21latency_hist00:05:033
189697995313,6cyclictest37-21ksoftirqd/323:35:193
189697995222,27cyclictest37-21ksoftirqd/321:50:023
18969799521,14cyclictest192266-21wc20:00:023
189697995210,8cyclictest201858-21systemd-run23:05:013
18969399524,8cyclictest30-21ksoftirqd/222:20:022
18969399522,9cyclictest30-21ksoftirqd/223:35:022
18969399521,11cyclictest30-21ksoftirqd/219:15:012
189683995214,9cyclictest12-21ksoftirqd/022:25:020
18969799515,6cyclictest201062-21modprobe22:50:013
18969799513,16cyclictest199002-21munin-run22:10:013
189697995119,9cyclictest37-21ksoftirqd/319:40:013
18969799511,12cyclictest193305-21cat20:20:013
189693995133,14cyclictest235-21systemd-journal19:50:192
189683995123,6cyclictest0-21swapper/023:20:020
189683995111,10cyclictest203413-21munin-run23:35:010
18969799507,12cyclictest272-21dbus-daemon19:50:013
189697995027,17cyclictest1-21systemd20:15:003
18969799502,18cyclictest202529-21latency_hist23:15:013
189697995014,12cyclictest197253-21sed21:35:013
189697995010,7cyclictest196739-21date21:25:023
18969399505,11cyclictest30-21ksoftirqd/219:50:022
189693995034,13cyclictest1-21systemd21:05:012
18969399502,10cyclictest30-21ksoftirqd/200:00:002
189693995013,6cyclictest131rcu_preempt21:15:022
189693995012,6cyclictest131rcu_preempt23:00:002
189697994914,4cyclictest191453-21latency_hist19:45:023
189693994910,13cyclictest30-21ksoftirqd/221:19:592
189688994934,11cyclictest272-21dbus-daemon20:40:001
189688994933,11cyclictest193243-21cron20:20:011
189688994932,13cyclictest235-21systemd-journal19:40:021
189683994930,15cyclictest200503-21sshd22:36:190
189683994930,15cyclictest200503-21sshd22:36:180
189683994928,15cyclictest235-21systemd-journal21:25:010
189683994910,8cyclictest12-21ksoftirqd/020:50:010
189697994813,7cyclictest193501-21sshd20:20:203
189697994813,5cyclictest196916-21latency_hist21:30:033
18969799481,15cyclictest196607-21latency_hist21:20:013
189693994818,21cyclictest291rcuc/219:20:022
189693994811,11cyclictest30-21ksoftirqd/222:40:002
189693994811,11cyclictest30-21ksoftirqd/222:40:002
189683994831,12cyclictest1-21systemd19:40:000
18969799473,14cyclictest196903-21sshd21:29:193
18969399478,10cyclictest30-21ksoftirqd/200:05:012
189693994735,9cyclictest235-21systemd-journal23:50:002
18969399473,13cyclictest30-21ksoftirqd/221:09:592
189683994732,11cyclictest199467-21cron22:17:000
189683994731,12cyclictest1-21systemd19:50:010
189683994731,12cyclictest1-21systemd19:15:000
189683994729,12cyclictest272-21dbus-daemon00:00:000
189683994725,17cyclictest195583-21sshd21:01:440
18969799463,10cyclictest1-21systemd20:10:003
18969399466,10cyclictest30-21ksoftirqd/220:04:202
189693994635,8cyclictest30-21ksoftirqd/220:10:002
189688994634,8cyclictest23-21ksoftirqd/120:00:001
189688994626,15cyclictest1-21systemd00:00:011
189688994619,20cyclictest272-21dbus-daemon22:55:001
189683994627,13cyclictest206824-21cron00:40:000
189683994613,28cyclictest204352-21latency_hist23:50:030
189697994514,4cyclictest0-21swapper/320:54:593
189697994513,7cyclictest37-21ksoftirqd/322:33:563
189693994530,11cyclictest272-21dbus-daemon19:40:012
18969399452,9cyclictest30-21ksoftirqd/200:40:002
189693994524,15cyclictest1-21systemd23:20:012
189688994532,9cyclictest205770-21cron00:19:591
189688994530,12cyclictest201579-21systemctl23:00:011
189683994530,11cyclictest202871-21sshd23:22:190
189683994530,11cyclictest202871-21sshd23:22:180
189683994530,11cyclictest193749-21sshd20:25:190
18969799443,14cyclictest206953-21latency_hist00:40:023
18969799442,13cyclictest202107-21modprobe23:10:003
189697994410,6cyclictest204010-21latency_hist23:45:013
18969399449,14cyclictest30-21ksoftirqd/221:40:202
18969399449,14cyclictest30-21ksoftirqd/221:40:192
189693994435,6cyclictest30-21ksoftirqd/220:40:012
189688994423,14cyclictest272-21dbus-daemon21:30:011
189688994423,14cyclictest272-21dbus-daemon21:30:001
189683994437,4cyclictest16950irq/224-800000022:15:010
189683994437,4cyclictest16950irq/224-800000020:05:010
189683994432,9cyclictest363-21snmpd21:47:050
189683994428,13cyclictest202320-21sshd23:10:180
189683994426,13cyclictest235-21systemd-journal21:00:000
18968399441,39cyclictest205485-21sshd00:10:300
189693994333,7cyclictest30-21ksoftirqd/223:27:392
189693994333,7cyclictest30-21ksoftirqd/219:25:022
189693994314,9cyclictest30-21ksoftirqd/222:30:202
189693994314,10cyclictest30-21ksoftirqd/222:45:012
189688994325,13cyclictest272-21dbus-daemon19:40:001
189688994325,13cyclictest1-21systemd22:10:001
189683994338,3cyclictest0-21swapper/022:10:010
189683994334,6cyclictest203699-21latency_hist23:40:020
189683994319,19cyclictest192181-21sshd19:56:190
18969399426,9cyclictest30-21ksoftirqd/220:55:012
18969399426,7cyclictest30-21ksoftirqd/220:20:002
189693994231,7cyclictest30-21ksoftirqd/200:30:002
189693994223,14cyclictest272-21dbus-daemon00:35:012
18969399421,7cyclictest198492-21true22:00:012
189688994232,7cyclictest235-21systemd-journal21:15:001
189688994229,10cyclictest23-21ksoftirqd/120:50:011
189688994227,11cyclictest235-21systemd-journal22:01:191
189688994226,12cyclictest1-21systemd23:35:011
189688994226,11cyclictest306-21in:imuxsock19:25:001
18968399426,6cyclictest12-21ksoftirqd/019:30:200
18968399425,29cyclictest189687-21latency_hist19:10:020
189683994236,4cyclictest16950irq/224-800000000:30:010
18968399423,35cyclictest196093-21sshd21:10:190
189683994229,10cyclictest197667-21sshd21:40:200
189683994229,10cyclictest197667-21sshd21:40:190
189683994227,11cyclictest190418-21munin-run19:25:000
189683994226,13cyclictest201043-21sshd22:47:180
189683994226,12cyclictest202072-21sshd23:06:190
18968399421,35cyclictest200792-21sshd22:42:190
189697994129,8cyclictest205495-21sshd00:13:323
18968899415,6cyclictest131rcu_preempt22:00:011
189688994130,8cyclictest200795-21sshd22:43:191
189688994128,9cyclictest203654-21sh23:40:001
18968899412,4cyclictest131rcu_preempt22:40:011
18968899412,4cyclictest131rcu_preempt22:40:001
189688994123,12cyclictest272-21dbus-daemon23:05:001
189683994128,10cyclictest195057-21sshd20:53:180
189683994126,12cyclictest204397-21sshd23:50:200
189683994125,12cyclictest194027-21sshd20:34:190
189683994125,12cyclictest194027-21sshd20:34:190
189683994124,12cyclictest206556-21sshd00:34:290
189683994115,21cyclictest201750-21latency_hist23:00:020
189697994025,10cyclictest272-21dbus-daemon23:00:013
18969399407,6cyclictest131rcu_preempt21:35:022
18969399406,11cyclictest30-21ksoftirqd/222:02:212
189693994030,6cyclictest202893-21munin-plugin-st23:25:012
189693994030,6cyclictest202893-21munin-plugin-st23:25:002
189693994028,10cyclictest30-21ksoftirqd/223:15:002
189693994024,12cyclictest272-21dbus-daemon22:55:012
18969399401,7cyclictest193538-21munin-plugin-st20:25:002
18969399401,5cyclictest0-21swapper/220:45:002
189693994013,6cyclictest131rcu_preempt23:38:192
189693994010,10cyclictest30-21ksoftirqd/219:44:032
189688994029,9cyclictest23-21ksoftirqd/123:10:011
189688994027,11cyclictest23-21ksoftirqd/119:35:011
189688994023,12cyclictest272-21dbus-daemon22:50:011
189683994029,8cyclictest12-21ksoftirqd/020:45:020
189683994028,9cyclictest191425-21sshd19:41:190
189683994027,9cyclictest198706-21sshd22:01:540
189683994026,10cyclictest205249-21sshd00:08:360
189683994026,10cyclictest198219-21munin-run21:55:010
189683994025,11cyclictest203900-21sshd23:41:190
189683994023,12cyclictest12-21ksoftirqd/020:05:190
18969399395,5cyclictest131rcu_preempt21:48:182
18969399394,5cyclictest30-21ksoftirqd/223:07:202
189693993930,6cyclictest190923-21munin-plugin-st19:35:002
189693993929,6cyclictest306-21in:imuxsock21:55:012
189693993925,10cyclictest30-21ksoftirqd/222:10:012
189693993912,5cyclictest131rcu_preempt20:50:032
189693993912,5cyclictest131rcu_preempt00:16:312
189688993929,7cyclictest204947-21sshd00:03:351
189688993928,7cyclictest306-21in:imuxsock23:30:001
189688993928,7cyclictest196641-21sshd21:21:461
189688993926,9cyclictest271-21cron20:15:001
189688993921,13cyclictest1-21systemd23:50:011
189688993920,15cyclictest235-21systemd-journal20:05:031
189688993917,18cyclictest1-21systemd21:20:011
189688993911,6cyclictest131rcu_preempt19:20:001
189683993926,10cyclictest191930-21sshd19:51:310
189683993924,11cyclictest201507-21sed22:55:030
18969399389,6cyclictest131rcu_preempt20:30:022
18969399386,6cyclictest131rcu_preempt22:30:012
189693993825,10cyclictest30-21ksoftirqd/221:00:012
189693993811,9cyclictest30-21ksoftirqd/223:45:022
18968899384,26cyclictest200287-21munin-plugin-st22:35:001
189688993830,5cyclictest0-21swapper/119:10:011
189688993824,8cyclictest197424-21latency_hist21:40:011
189688993820,12cyclictest195875-21sendmail-mta21:10:001
189683993829,5cyclictest200248-21sshd22:30:190
189683993828,7cyclictest12-21ksoftirqd/000:05:030
189683993826,9cyclictest203131-21sshd23:29:190
189683993824,11cyclictest201819-21sshd23:01:200
189697993714,4cyclictest206224-21cat00:25:013
18969399379,5cyclictest131rcu_preempt23:05:012
18969399379,5cyclictest131rcu_preempt22:10:182
18969399378,6cyclictest131rcu_preempt20:00:002
189693993726,7cyclictest30-21ksoftirqd/219:30:012
189693993724,10cyclictest30-21ksoftirqd/200:25:012
18969399371,6cyclictest204443-21systemd-run23:55:022
18969399371,4cyclictest131rcu_preempt22:50:002
189688993728,6cyclictest23-21ksoftirqd/122:20:011
189688993719,12cyclictest235-21systemd-journal21:00:001
18968899371,4cyclictest131rcu_preempt20:45:011
189683993723,9cyclictest0-21swapper/019:30:010
18969399368,5cyclictest131rcu_preempt21:26:482
18969399368,5cyclictest131rcu_preempt21:26:472
189693993627,6cyclictest30-21ksoftirqd/200:10:012
18969399361,5cyclictest131rcu_preempt20:12:012
18968899362,8cyclictest193638-21latency_hist20:25:011
189688993628,4cyclictest205435-21latency_hist00:10:011
189688993627,5cyclictest197700-21munin-plugin-st21:45:011
189688993627,5cyclictest197700-21munin-plugin-st21:45:001
189688993627,5cyclictest189875-21munin-plugin-st19:15:001
189688993626,7cyclictest202570-21sshd23:15:191
189688993623,10cyclictest23-21ksoftirqd/121:50:001
189688993621,11cyclictest272-21dbus-daemon19:30:021
189688993620,11cyclictest272-21dbus-daemon20:30:001
189688993617,14cyclictest200019-21cron22:30:001
189697993521,10cyclictest37-21ksoftirqd/322:45:003
189693993525,8cyclictest30-21ksoftirqd/200:10:312
18968899356,6cyclictest131rcu_preempt00:33:291
18968899354,4cyclictest131rcu_preempt20:05:011
189688993519,11cyclictest23-21ksoftirqd/123:55:011
18969399342,7cyclictest30-21ksoftirqd/221:25:002
18968899347,3cyclictest131rcu_preempt22:15:011
189688993427,4cyclictest197167-21latency_hist21:35:011
189688993422,8cyclictest194010-21sshd20:31:201
189688993422,8cyclictest194010-21sshd20:31:191
189688993422,10cyclictest23-21ksoftirqd/100:21:301
189688993421,10cyclictest23-21ksoftirqd/123:25:031
189688993421,10cyclictest23-21ksoftirqd/123:25:031
189688993421,10cyclictest202385-21sed23:15:011
189688993419,11cyclictest272-21dbus-daemon00:15:011
189688993418,12cyclictest1-21systemd23:45:011
18968899334,4cyclictest131rcu_preempt00:26:301
189688993324,5cyclictest198306-21grep21:55:011
189688993320,10cyclictest23-21ksoftirqd/119:55:011
18969399326,4cyclictest131rcu_preempt20:33:202
18969399326,4cyclictest131rcu_preempt20:33:192
18968899324,3cyclictest131rcu_preempt22:25:011
189688993225,4cyclictest191695-21cron19:50:001
189697993122,5cyclictest204280-21latency_hist23:50:023
18968899316,15cyclictest206819-21modprobe00:39:591
189688993019,8cyclictest195585-21sshd21:01:451
18969399294,5cyclictest131rcu_preempt19:10:012
189697992817,8cyclictest189714-21date19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional