You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 03:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Sep 18, 2025 00:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4091057997124,9cyclictest306-21in:imuxsock21:25:013
4091045996857,4cyclictest0-21swapper/020:50:010
409105799644,22cyclictest235-21systemd-journal19:15:003
4091052996415,10cyclictest30-21ksoftirqd/222:15:012
409105799632,22cyclictest4091559-21wc19:20:023
4091052996310,15cyclictest4092824-21munin-run19:45:002
4091057996223,7cyclictest4097465-21sshd21:13:263
4091057996221,12cyclictest1-21systemd22:00:013
4091057996134,22cyclictest4108156-21sendmail00:39:593
4091057996131,26cyclictest4093580-21sendmail20:00:013
409105799602,27cyclictest235-21systemd-journal19:45:003
4091057995923,5cyclictest4096137-21latency_hist20:45:033
4091057995923,5cyclictest4096137-21latency_hist20:45:023
4091057995920,13cyclictest37-21ksoftirqd/323:50:023
409105299599,14cyclictest4094112-21munin-run20:09:592
4091052995916,10cyclictest30-21ksoftirqd/220:15:012
409105799582,38cyclictest4098838-21kworker/3:122:25:033
409105799582,38cyclictest4098838-21kworker/3:122:25:023
4091052995615,8cyclictest30-21ksoftirqd/220:30:032
4091052995615,8cyclictest30-21ksoftirqd/220:30:022
4091057995518,9cyclictest4104864-21wc23:35:013
409104599559,26cyclictest19050irq/34-mmc100:00:030
4091045995526,25cyclictest4093405-21latency_hist19:55:010
409105799545,18cyclictest4102396-21kworker/u8:300:15:023
4091057995427,5cyclictest4093075-21munin-plugin-st19:50:013
409105299548,17cyclictest4100050-21cat22:00:032
409105299539,16cyclictest4107599-21sshd00:26:262
4091052995314,12cyclictest4095661-21latency_hist20:40:032
4091045995332,15cyclictest272-21dbus-daemon20:15:020
409105799523,21cyclictest235-21systemd-journal00:10:013
4091057995215,10cyclictest306-21in:imuxsock22:34:593
4091057995131,16cyclictest235-21systemd-journal21:34:593
4091057995124,6cyclictest37-21ksoftirqd/323:25:033
4091057995124,6cyclictest37-21ksoftirqd/323:25:023
4091052995135,13cyclictest4097458-21sshd21:12:092
4091052995013,8cyclictest30-21ksoftirqd/220:50:002
4091048994931,14cyclictest272-21dbus-daemon00:10:021
4091045994942,4cyclictest16950irq/224-800000021:35:000
4091052994833,11cyclictest4092275-21sshd19:34:242
4091052994832,11cyclictest235-21systemd-journal21:25:002
4091052994831,13cyclictest4096955-21sshd21:04:252
4091052994830,14cyclictest4100065-21sshd22:01:242
4091052994829,14cyclictest272-21dbus-daemon00:00:002
4091052994828,15cyclictest0-21swapper/221:45:002
4091048994832,11cyclictest272-21dbus-daemon22:30:001
4091052994735,9cyclictest30-21ksoftirqd/220:55:002
4091052994734,9cyclictest4094890-21cron20:25:002
4091052994733,10cyclictest4097709-21cron21:17:002
4091052994725,18cyclictest4102144-21sshd22:41:262
409105799466,11cyclictest131rcu_preempt20:50:013
409105299466,7cyclictest30-21ksoftirqd/200:17:012
4091052994632,11cyclictest4098247-21sshd21:26:262
4091052994632,10cyclictest4103930-21sshd23:15:242
4091052994631,12cyclictest363-21snmpd20:32:572
4091052994610,8cyclictest30-21ksoftirqd/221:10:002
4091045994629,11cyclictest235-21systemd-journal00:00:030
409105799452,4cyclictest131rcu_preempt19:35:013
4091052994532,10cyclictest363-21snmpd22:39:452
4091052994530,10cyclictest48-21kcompactd022:23:512
4091052994530,10cyclictest48-21kcompactd022:23:502
4091052994529,12cyclictest363-21snmpd23:45:262
409104899454,4cyclictest131rcu_preempt20:50:021
4091048994523,16cyclictest272-21dbus-daemon21:20:001
4091048994521,16cyclictest23-21ksoftirqd/122:00:011
4091045994531,10cyclictest1-21systemd22:35:000
4091057994430,10cyclictest235-21systemd-journal23:54:263
4091057994430,10cyclictest235-21systemd-journal23:54:253
4091057994423,14cyclictest272-21dbus-daemon21:05:003
4091052994431,10cyclictest4106311-21sshd00:03:262
4091052994431,10cyclictest235-21systemd-journal00:08:242
4091052994430,10cyclictest48-21kcompactd022:15:312
4091052994430,10cyclictest4101644-21munin-run22:35:012
4091052994425,14cyclictest272-21dbus-daemon21:35:002
4091052994423,16cyclictest4098768-21cron21:40:002
4091052994420,19cyclictest272-21dbus-daemon20:45:012
4091052994420,19cyclictest272-21dbus-daemon20:45:002
4091048994429,10cyclictest235-21systemd-journal19:20:011
4091048994428,12cyclictest23-21ksoftirqd/123:40:011
4091048994428,12cyclictest23-21ksoftirqd/123:40:001
4091048994427,13cyclictest4102953-21tr22:59:591
4091048994427,13cyclictest235-21systemd-journal19:35:021
4091048994427,12cyclictest23-21ksoftirqd/122:20:001
4091048994423,15cyclictest272-21dbus-daemon23:59:591
4091045994426,14cyclictest4107072-21sshd00:17:250
4091045994421,19cyclictest4092806-21sshd19:44:240
4091057994331,9cyclictest37-21ksoftirqd/321:10:003
4091057994324,14cyclictest1-21systemd20:05:013
4091057994321,16cyclictest1-21systemd20:35:013
4091052994330,9cyclictest4093549-21sshd19:56:562
4091052994329,11cyclictest363-21snmpd23:36:472
4091052994329,11cyclictest363-21snmpd23:36:462
4091052994324,15cyclictest363-21snmpd21:54:002
4091052994323,16cyclictest4094595-21sshd20:17:002
4091052994323,16cyclictest363-21snmpd20:02:152
4091052994317,21cyclictest4092592-21cat19:40:022
409104899434,4cyclictest131rcu_preempt19:15:001
4091048994329,12cyclictest23-21ksoftirqd/123:55:021
4091048994329,12cyclictest23-21ksoftirqd/123:55:011
4091048994329,10cyclictest306-21in:imuxsock20:35:001
4091048994327,12cyclictest272-21dbus-daemon23:25:011
4091048994327,12cyclictest272-21dbus-daemon23:25:001
4091048994327,11cyclictest306-21in:imuxsock21:05:001
4091048994318,4cyclictest23-21ksoftirqd/120:30:011
4091048994318,4cyclictest23-21ksoftirqd/120:30:001
409104599432,37cyclictest4097790-21munin-run21:20:000
4091052994230,9cyclictest4108097-21sshd00:35:242
4091052994230,9cyclictest4107851-21sshd00:31:252
4091052994230,9cyclictest4093050-21sshd19:47:252
4091052994229,9cyclictest48-21kcompactd023:09:202
4091052994229,10cyclictest4105295-21munin-run23:45:002
4091048994229,9cyclictest23-21ksoftirqd/123:20:001
4091045994228,11cyclictest272-21dbus-daemon20:55:010
4091045994227,11cyclictest4100562-21sshd22:10:250
4091045994226,12cyclictest12-21ksoftirqd/023:40:020
4091045994226,12cyclictest12-21ksoftirqd/023:40:010
4091045994225,12cyclictest235-21systemd-journal19:50:010
409104599422,36cyclictest4107612-21sshd00:29:250
409104599422,35cyclictest1-21systemd22:07:210
409105799414,3cyclictest131rcu_preempt19:25:013
4091057994128,9cyclictest4104990-21sshd23:37:383
4091057994128,9cyclictest4104990-21sshd23:37:383
4091057994124,13cyclictest235-21systemd-journal22:05:023
4091057994123,13cyclictest272-21dbus-daemon23:05:013
409105299413,5cyclictest30-21ksoftirqd/220:55:032
409105299413,5cyclictest30-21ksoftirqd/220:55:022
4091052994132,5cyclictest4092054-21munin-run19:30:012
4091052994130,7cyclictest4100312-21sshd22:05:252
4091052994128,9cyclictest48-21kcompactd019:17:172
4091052994127,9cyclictest48-21kcompactd019:54:352
4091052994124,12cyclictest272-21dbus-daemon23:00:012
4091052994114,23cyclictest4105895-21latency_hist23:55:022
4091052994114,23cyclictest4105895-21latency_hist23:55:012
4091048994128,9cyclictest23-21ksoftirqd/100:40:021
4091048994117,20cyclictest221rcuc/123:45:021
4091045994134,4cyclictest16950irq/224-800000023:05:000
409104599413,33cyclictest0-21swapper/000:35:010
4091045994129,9cyclictest235-21systemd-journal21:58:250
4091045994128,8cyclictest306-21in:imuxsock23:10:010
4091045994123,12cyclictest272-21dbus-daemon22:45:010
409104599411,10cyclictest4097652-21cat21:15:030
4091057994029,8cyclictest37-21ksoftirqd/321:00:013
4091057994029,8cyclictest37-21ksoftirqd/321:00:003
4091057994028,7cyclictest4097735-21cron21:19:593
4091057994028,10cyclictest37-21ksoftirqd/320:25:013
4091057994027,10cyclictest4100613-21systemd-run22:15:003
4091057994026,9cyclictest4107082-21sshd00:18:553
4091057994023,12cyclictest272-21dbus-daemon00:00:013
4091057994022,13cyclictest235-21systemd-journal21:44:593
4091052994028,9cyclictest363-21snmpd23:01:392
4091052994027,9cyclictest48-21kcompactd022:52:412
4091052994027,10cyclictest363-21snmpd23:21:442
4091052994027,10cyclictest363-21snmpd23:21:432
4091052994026,10cyclictest363-21snmpd23:26:442
4091052994026,10cyclictest363-21snmpd00:14:402
4091052994014,20cyclictest1-21systemd22:50:002
4091048994033,5cyclictest4091054-21latency_hist19:10:011
409104899402,29cyclictest4107015-21cat00:15:021
4091045994033,4cyclictest16950irq/224-800000019:17:250
4091045994031,6cyclictest19050irq/34-mmc123:00:000
4091045994026,10cyclictest4096666-21sshd20:57:080
4091045994026,10cyclictest4096666-21sshd20:57:070
4091045994024,12cyclictest1-21systemd20:25:010
4091045994023,12cyclictest4094635-21systemctl20:19:590
4091045994023,12cyclictest12-21ksoftirqd/022:20:010
4091045994021,15cyclictest12-21ksoftirqd/020:40:010
409104599401,33cyclictest4092678-21latency_hist19:40:010
409104599401,10cyclictest4098755-21sshd21:37:260
4091057993926,10cyclictest235-21systemd-journal22:55:253
4091057993925,11cyclictest37-21ksoftirqd/319:55:003
4091057993923,12cyclictest1-21systemd21:40:013
4091052993922,13cyclictest4091226-21sshd19:13:252
409104899395,25cyclictest4092043-21cut19:30:011
409104899394,6cyclictest131rcu_preempt21:44:591
4091048993926,10cyclictest4103861-21sed23:15:021
4091048993925,11cyclictest4097390-21sed21:10:031
4091048993922,12cyclictest235-21systemd-journal21:55:001
4091045993933,4cyclictest16950irq/224-800000019:31:260
4091045993932,4cyclictest16950irq/224-800000021:10:000
4091045993931,5cyclictest19050irq/34-mmc119:22:000
4091045993927,9cyclictest4101364-21sshd22:27:250
4091045993924,11cyclictest4108114-21sshd00:37:010
4091045993924,11cyclictest12-21ksoftirqd/020:00:010
409104599392,33cyclictest1-21systemd22:22:250
409104599392,33cyclictest1-21systemd22:22:250
4091045993923,11cyclictest235-21systemd-journal22:55:010
4091057993826,8cyclictest37-21ksoftirqd/300:25:023
4091057993825,9cyclictest272-21dbus-daemon19:30:013
4091057993824,11cyclictest37-21ksoftirqd/321:50:013
4091057993822,12cyclictest37-21ksoftirqd/322:10:003
4091052993824,9cyclictest363-21snmpd21:46:142
4091052993824,10cyclictest235-21systemd-journal23:12:572
409104899385,23cyclictest23-21ksoftirqd/119:25:011
4091048993825,6cyclictest23-21ksoftirqd/120:45:011
4091048993825,6cyclictest23-21ksoftirqd/120:45:001
4091048993825,10cyclictest272-21dbus-daemon23:35:011
4091048993824,9cyclictest1-21systemd23:05:011
4091048993823,12cyclictest23-21ksoftirqd/122:50:021
4091048993823,11cyclictest23-21ksoftirqd/100:30:241
409104599388,26cyclictest3237915-1kworker/0:2H23:50:020
409104599384,29cyclictest0-21swapper/023:27:530
4091045993828,6cyclictest4107366-21cron00:25:000
4091045993821,12cyclictest272-21dbus-daemon23:45:000
409104599381,33cyclictest0-21swapper/021:55:010
4091057993729,5cyclictest4094121-21sh20:10:003
4091057993725,9cyclictest37-21ksoftirqd/323:10:013
4091057993724,10cyclictest37-21ksoftirqd/322:50:003
4091057993723,11cyclictest4102242-21sort22:45:023
4091057993718,13cyclictest1-21systemd21:30:013
4091052993730,4cyclictest4101393-21munin-run22:30:002
4091052993728,7cyclictest30-21ksoftirqd/223:35:022
4091052993725,9cyclictest363-21snmpd19:20:562
4091052993722,12cyclictest412-21sshd00:22:242
4091052993717,16cyclictest4091153-21cat19:10:012
409104899379,6cyclictest131rcu_preempt21:50:011
4091048993729,6cyclictest23-21ksoftirqd/100:30:021
4091048993726,7cyclictest306-21in:imuxsock21:15:011
4091048993723,11cyclictest4092946-21sed19:45:031
4091048993722,11cyclictest23-21ksoftirqd/120:51:111
4091048993711,5cyclictest131rcu_preempt23:30:001
409104599373,7cyclictest4103982-21run-parts23:20:000
409104599373,3cyclictest131rcu_preempt19:10:020
4091045993726,8cyclictest12-21ksoftirqd/023:25:010
4091045993726,8cyclictest12-21ksoftirqd/023:25:000
4091045993723,11cyclictest12-21ksoftirqd/022:45:010
4091045993723,10cyclictest235-21systemd-journal20:45:010
4091045993723,10cyclictest235-21systemd-journal20:45:000
4091057993628,5cyclictest4095120-21sshd20:25:253
4091057993628,5cyclictest4095120-21sshd20:25:253
4091057993624,9cyclictest4092529-21sshd19:39:253
4091057993619,12cyclictest272-21dbus-daemon20:40:003
409105799361,4cyclictest131rcu_preempt23:20:003
409104899368,5cyclictest131rcu_preempt20:05:021
4091048993627,6cyclictest4107105-21munin-plugin-st00:20:011
4091048993625,8cyclictest4093613-21systemd-run20:00:001
4091048993622,11cyclictest23-21ksoftirqd/122:24:251
4091048993622,11cyclictest23-21ksoftirqd/122:24:241
4091048993611,4cyclictest4103657-21sed23:10:021
4091045993630,4cyclictest0-21swapper/020:05:500
4091045993626,7cyclictest0-21swapper/022:05:000
409104599362,30cyclictest235-21systemd-journal21:04:520
409104599362,30cyclictest0-21swapper/020:25:510
409104599362,30cyclictest0-21swapper/020:25:510
409104599361,31cyclictest151rcuc/023:15:010
409105799353,3cyclictest131rcu_preempt22:20:003
4091057993528,4cyclictest4101387-21sh22:30:003
4091057993527,6cyclictest37-21ksoftirqd/323:45:003
4091057993524,8cyclictest37-21ksoftirqd/320:10:033
4091057993515,16cyclictest235-21systemd-journal00:05:003
409104899353,6cyclictest131rcu_preempt20:40:011
4091048993524,9cyclictest23-21ksoftirqd/122:44:591
4091048993524,8cyclictest23-21ksoftirqd/120:15:001
4091048993523,9cyclictest23-21ksoftirqd/122:10:001
4091048993521,10cyclictest23-21ksoftirqd/119:50:011
4091048993520,11cyclictest272-21dbus-daemon19:55:011
4091045993523,9cyclictest12-21ksoftirqd/021:50:010
4091045993521,10cyclictest0-21swapper/021:45:010
409104599351,3cyclictest131rcu_preempt19:29:250
409104599351,30cyclictest0-21swapper/021:25:320
4091057993426,6cyclictest37-21ksoftirqd/320:16:053
4091057993424,8cyclictest37-21ksoftirqd/319:10:023
4091057993418,11cyclictest272-21dbus-daemon21:55:013
4091057993416,12cyclictest235-21systemd-journal00:35:013
4091048993427,4cyclictest4107386-21sh00:25:001
4091048993426,6cyclictest23-21ksoftirqd/122:40:011
4091048993425,7cyclictest23-21ksoftirqd/122:55:021
409104899342,4cyclictest131rcu_preempt21:29:591
4091048993423,8cyclictest4098509-21sshd21:33:261
4091048993422,9cyclictest4094078-21sshd20:06:261
4091048993421,9cyclictest4094863-21date20:20:021
4091045993423,8cyclictest12-21ksoftirqd/020:35:020
409104599342,10cyclictest4102089-21date22:40:030
4091045993420,10cyclictest12-21ksoftirqd/000:15:010
409104599341,30cyclictest0-21swapper/000:06:260
409104599341,29cyclictest0-21swapper/020:05:000
4091045993410,21cyclictest0-21swapper/019:14:180
4091057993325,6cyclictest37-21ksoftirqd/322:40:023
4091057993325,6cyclictest37-21ksoftirqd/320:55:023
4091057993324,5cyclictest4107707-21latency_hist00:30:013
4091057993322,8cyclictest37-21ksoftirqd/322:55:023
4091057993321,10cyclictest37-21ksoftirqd/323:15:003
4091048993326,4cyclictest4094661-21sh20:20:011
4091048993322,7cyclictest23-21ksoftirqd/122:05:011
4091048993321,9cyclictest23-21ksoftirqd/119:35:001
4091048993316,13cyclictest235-21systemd-journal22:35:011
409104599331,29cyclictest0-21swapper/023:32:050
409104599331,29cyclictest0-21swapper/021:20:550
4091048993220,9cyclictest0-21swapper/100:05:021
409104899321,6cyclictest131rcu_preempt23:50:001
4091048993213,14cyclictest23-21ksoftirqd/121:00:011
4091048993213,14cyclictest23-21ksoftirqd/121:00:011
409104599321,28cyclictest0-21swapper/023:54:250
409104599321,28cyclictest0-21swapper/023:54:240
409104899314,5cyclictest131rcu_preempt21:40:001
4091048993119,10cyclictest23-21ksoftirqd/121:25:011
409105799303,3cyclictest131rcu_preempt23:30:023
4091048992815,9cyclictest23-21ksoftirqd/122:15:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional