You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 15:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sun Nov 16, 2025 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
308059399104100,2cyclictest0-21swapper/009:25:010
308059399913,9cyclictest131rcu_preempt10:05:020
308059399681,6cyclictest0-21swapper/007:20:020
3080598996210,9cyclictest23-21ksoftirqd/109:45:001
3080598996111,13cyclictest23-21ksoftirqd/109:20:011
3080598995920,10cyclictest23-21ksoftirqd/109:50:001
308060299576,30cyclictest3082046-21kworker/u8:207:50:022
3080602995637,14cyclictest272-21dbus-daemon09:00:002
308059899566,11cyclictest23-21ksoftirqd/108:35:031
308059899566,11cyclictest23-21ksoftirqd/108:35:021
308059899562,15cyclictest23-21ksoftirqd/111:55:021
308059899562,15cyclictest23-21ksoftirqd/111:55:011
3080598995613,12cyclictest23-21ksoftirqd/111:05:031
3080598995613,12cyclictest23-21ksoftirqd/111:05:021
308059899553,7cyclictest23-21ksoftirqd/110:40:011
308060299536,5cyclictest131rcu_preempt11:00:022
3080602995322,10cyclictest30-21ksoftirqd/211:15:002
308059899533,9cyclictest23-21ksoftirqd/109:05:001
308059899531,8cyclictest3082612-21munin-run07:50:001
308059899531,10cyclictest23-21ksoftirqd/109:10:021
3080605995234,13cyclictest1-21systemd09:15:013
308060299521,10cyclictest30-21ksoftirqd/207:20:012
308059899529,8cyclictest23-21ksoftirqd/107:45:021
308059899527,7cyclictest23-21ksoftirqd/108:55:021
308059899522,14cyclictest23-21ksoftirqd/107:40:011
308059399524,41cyclictest19050irq/34-mmc108:00:030
3080605995134,12cyclictest3094556-21run-parts11:40:003
308060299519,6cyclictest131rcu_preempt08:50:022
308060299519,6cyclictest131rcu_preempt08:50:022
308060299516,7cyclictest30-21ksoftirqd/211:30:012
308060299516,11cyclictest30-21ksoftirqd/212:15:002
308060299512,8cyclictest30-21ksoftirqd/211:20:032
3080602995117,8cyclictest131rcu_preempt11:08:492
3080602995114,6cyclictest131rcu_preempt09:55:012
308059899517,10cyclictest23-21ksoftirqd/108:05:021
308059899514,12cyclictest23-21ksoftirqd/107:20:021
308059899512,9cyclictest23-21ksoftirqd/112:39:301
308059899511,15cyclictest3092535-21cut11:00:011
3080593995135,11cyclictest1-21systemd08:50:020
3080593995135,11cyclictest1-21systemd08:50:010
3080605995039,6cyclictest3093791-21munin-plugin-st11:24:593
308060299507,8cyclictest3096917-21true12:25:012
308060299506,8cyclictest30-21ksoftirqd/209:45:012
308060299505,9cyclictest30-21ksoftirqd/210:45:002
3080602995012,6cyclictest131rcu_preempt07:35:012
308059899509,9cyclictest23-21ksoftirqd/111:30:011
308059899508,12cyclictest23-21ksoftirqd/107:30:031
308059899508,12cyclictest23-21ksoftirqd/107:30:021
308059899507,6cyclictest23-21ksoftirqd/108:45:021
308059899504,12cyclictest23-21ksoftirqd/107:15:021
308059899503,15cyclictest23-21ksoftirqd/112:20:011
308059899503,15cyclictest23-21ksoftirqd/112:20:001
308059899503,11cyclictest23-21ksoftirqd/112:30:011
308059899502,12cyclictest23-21ksoftirqd/108:15:021
3080593995029,13cyclictest272-21dbus-daemon07:25:010
308060299497,9cyclictest30-21ksoftirqd/208:40:492
3080602994940,6cyclictest30-21ksoftirqd/210:05:022
3080602994925,17cyclictest235-21systemd-journal07:15:012
308060299491,9cyclictest30-21ksoftirqd/209:20:012
308059899497,10cyclictest23-21ksoftirqd/107:25:011
308059899496,7cyclictest23-21ksoftirqd/110:20:031
308059899496,7cyclictest23-21ksoftirqd/110:20:031
308059899496,11cyclictest23-21ksoftirqd/110:50:011
308059899496,11cyclictest23-21ksoftirqd/110:50:001
308059899496,11cyclictest23-21ksoftirqd/108:40:001
3080598994932,12cyclictest23-21ksoftirqd/112:00:001
308059899491,11cyclictest1-21systemd12:35:011
308059899491,11cyclictest1-21systemd12:35:001
3080593994931,15cyclictest408-21lldpd12:10:000
308060299489,5cyclictest131rcu_preempt10:20:012
308060299489,5cyclictest131rcu_preempt10:20:002
308060299488,5cyclictest131rcu_preempt07:10:022
308060299487,7cyclictest30-21ksoftirqd/207:30:012
308060299487,7cyclictest30-21ksoftirqd/207:30:002
308060299486,10cyclictest30-21ksoftirqd/212:06:492
3080602994813,8cyclictest131rcu_preempt08:20:022
308059899488,4cyclictest131rcu_preempt10:29:051
308059899487,11cyclictest23-21ksoftirqd/112:05:011
308059899485,5cyclictest23-21ksoftirqd/110:45:021
308059899484,8cyclictest23-21ksoftirqd/112:25:021
3080598994831,12cyclictest235-21systemd-journal11:20:001
308059899482,11cyclictest23-21ksoftirqd/108:20:031
308059899482,11cyclictest23-21ksoftirqd/108:20:021
3080598994811,9cyclictest23-21ksoftirqd/112:15:021
3080593994827,15cyclictest1-21systemd09:50:010
308059399482,41cyclictest3091713-21munin-run10:45:010
3080605994729,13cyclictest1-21systemd11:30:013
308060299478,6cyclictest131rcu_preempt09:25:012
308060299475,10cyclictest30-21ksoftirqd/209:10:002
308060299473,6cyclictest131rcu_preempt08:31:492
308060299473,6cyclictest131rcu_preempt08:31:482
308060299473,10cyclictest30-21ksoftirqd/210:40:002
308060299472,10cyclictest30-21ksoftirqd/211:05:022
308060299472,10cyclictest30-21ksoftirqd/211:05:012
3080602994715,10cyclictest30-21ksoftirqd/210:35:012
3080602994713,9cyclictest131rcu_preempt09:05:012
3080602994713,8cyclictest30-21ksoftirqd/207:45:022
3080602994712,5cyclictest131rcu_preempt12:20:012
3080602994712,5cyclictest131rcu_preempt12:20:012
3080602994711,6cyclictest131rcu_preempt10:50:022
3080602994711,6cyclictest131rcu_preempt10:50:012
308060299471,12cyclictest3082864-21munin-plugin-st07:55:002
3080602994710,6cyclictest131rcu_preempt10:22:492
3080602994710,5cyclictest131rcu_preempt10:54:592
308059899479,10cyclictest23-21ksoftirqd/110:15:011
308059899476,6cyclictest23-21ksoftirqd/111:40:021
308059899472,9cyclictest3089749-21latency_hist10:05:011
3080598994728,14cyclictest272-21dbus-daemon10:35:001
308059899472,13cyclictest23-21ksoftirqd/107:55:011
308059899472,12cyclictest23-21ksoftirqd/109:40:001
308059899471,9cyclictest23-21ksoftirqd/110:10:031
308059899471,8cyclictest3087220-21cat09:15:021
308059899471,5cyclictest0-21swapper/108:24:591
3080605994628,12cyclictest272-21dbus-daemon08:15:013
308060599462,12cyclictest3081053-21modprobe07:20:003
308060299468,5cyclictest131rcu_preempt11:45:022
308060299464,6cyclictest131rcu_preempt09:30:032
308060299464,6cyclictest131rcu_preempt09:30:032
3080602994614,12cyclictest30-21ksoftirqd/208:05:032
3080602994610,6cyclictest131rcu_preempt10:30:022
308059899464,11cyclictest23-21ksoftirqd/110:55:011
308059899462,11cyclictest23-21ksoftirqd/110:00:001
308059899461,4cyclictest0-21swapper/111:45:031
3080593994632,10cyclictest1-21systemd09:30:020
3080593994632,10cyclictest1-21systemd09:30:010
3080593994629,13cyclictest3092724-21sshd11:02:480
3080593994629,13cyclictest3092724-21sshd11:02:480
3080593994627,14cyclictest235-21systemd-journal08:20:010
3080593994627,14cyclictest235-21systemd-journal08:20:000
308060299458,7cyclictest131rcu_preempt11:59:592
308060299457,6cyclictest131rcu_preempt11:35:022
308060299457,6cyclictest131rcu_preempt07:25:002
308060299455,8cyclictest30-21ksoftirqd/209:15:002
3080602994536,6cyclictest3095081-21grep11:50:012
3080602994534,8cyclictest30-21ksoftirqd/212:40:032
308060299453,11cyclictest30-21ksoftirqd/212:35:012
308060299453,11cyclictest30-21ksoftirqd/212:35:002
3080602994515,6cyclictest131rcu_preempt10:15:002
3080602994514,9cyclictest131rcu_preempt10:10:002
3080602994511,5cyclictest131rcu_preempt10:00:002
308059899457,8cyclictest23-21ksoftirqd/108:00:001
308059899456,9cyclictest23-21ksoftirqd/107:10:011
308059899456,6cyclictest23-21ksoftirqd/109:55:021
308059899456,10cyclictest23-21ksoftirqd/109:00:021
308059899455,6cyclictest23-21ksoftirqd/107:35:001
308059899454,9cyclictest23-21ksoftirqd/111:10:021
308059899454,12cyclictest23-21ksoftirqd/109:25:021
308059899453,11cyclictest23-21ksoftirqd/108:50:021
308059899453,11cyclictest23-21ksoftirqd/108:50:011
308059899452,13cyclictest23-21ksoftirqd/111:45:021
308059899451,6cyclictest0-21swapper/110:25:001
3080598994511,8cyclictest0-21swapper/109:27:491
3080598994511,8cyclictest0-21swapper/109:27:481
3080593994538,4cyclictest16950irq/224-800000009:55:000
3080593994531,10cyclictest235-21systemd-journal09:07:470
308059399452,24cyclictest131rcu_preempt10:35:480
3080605994430,10cyclictest235-21systemd-journal09:25:033
3080605994430,10cyclictest235-21systemd-journal09:25:023
308060299448,6cyclictest131rcu_preempt09:50:012
308060299444,9cyclictest30-21ksoftirqd/211:39:592
308060299442,9cyclictest30-21ksoftirqd/208:00:002
308060299442,10cyclictest30-21ksoftirqd/209:40:012
3080602994410,6cyclictest131rcu_preempt08:25:022
308059899447,34cyclictest23-21ksoftirqd/109:30:021
308059899443,6cyclictest23-21ksoftirqd/111:30:021
308059899443,5cyclictest23-21ksoftirqd/112:10:011
3080593994437,4cyclictest16950irq/224-800000008:25:000
3080593994426,14cyclictest408-21lldpd12:30:010
308060599433,30cyclictest37-21ksoftirqd/308:55:013
3080605994313,21cyclictest361rcuc/309:35:023
308060299438,6cyclictest131rcu_preempt08:40:002
3080602994312,6cyclictest131rcu_preempt07:40:002
3080602994310,9cyclictest30-21ksoftirqd/212:05:032
3080602994310,6cyclictest131rcu_preempt09:35:012
3080593994328,11cyclictest235-21systemd-journal11:20:020
3080605994231,8cyclictest3089616-21sshd10:04:473
3080605994223,13cyclictest272-21dbus-daemon09:00:013
308060299428,5cyclictest131rcu_preempt08:15:022
308060299427,5cyclictest131rcu_preempt08:20:012
308059899425,9cyclictest23-21ksoftirqd/108:30:011
308059899422,10cyclictest23-21ksoftirqd/111:15:021
3080605994128,10cyclictest3083548-21sed08:05:013
3080605994127,9cyclictest37-21ksoftirqd/310:55:023
3080605994124,12cyclictest1-21systemd10:10:013
308060599412,13cyclictest3097720-21latency_hist12:40:013
308059899416,7cyclictest23-21ksoftirqd/111:25:011
308059899416,6cyclictest23-21ksoftirqd/108:10:031
308059399412,9cyclictest3095444-21latency_hist11:55:030
308059399412,9cyclictest3095444-21latency_hist11:55:020
3080593994129,7cyclictest3090138-21grep10:15:000
3080593994127,11cyclictest3086202-21sshd08:57:480
3080593994126,11cyclictest12-21ksoftirqd/007:30:000
3080593994126,11cyclictest12-21ksoftirqd/007:30:000
308059399412,34cyclictest3082049-21sshd07:36:470
3080605994029,5cyclictest306-21in:imuxsock09:50:013
3080605994025,11cyclictest235-21systemd-journal07:30:003
3080605994025,11cyclictest235-21systemd-journal07:30:003
308060299407,6cyclictest30-21ksoftirqd/208:05:152
308060299406,8cyclictest30-21ksoftirqd/212:30:002
308060299405,6cyclictest131rcu_preempt08:55:012
3080602994011,5cyclictest131rcu_preempt11:25:012
3080593994028,9cyclictest12-21ksoftirqd/011:35:010
3080593994026,10cyclictest408-21lldpd12:24:000
3080593994025,11cyclictest12-21ksoftirqd/010:35:010
3080593994018,7cyclictest131rcu_preempt08:45:000
3080605993928,9cyclictest37-21ksoftirqd/308:40:003
3080605993927,6cyclictest3082079-21sh07:40:003
3080605993921,12cyclictest272-21dbus-daemon10:50:013
3080605993921,12cyclictest272-21dbus-daemon10:50:003
3080605993917,7cyclictest131rcu_preempt11:35:023
3080602993912,5cyclictest131rcu_preempt08:30:002
3080593993932,4cyclictest16950irq/224-800000007:55:000
3080593993926,10cyclictest12-21ksoftirqd/012:05:010
3080593993924,11cyclictest235-21systemd-journal09:30:030
3080605993829,5cyclictest3084940-21munin-plugin-st08:35:003
3080605993829,5cyclictest3084940-21munin-plugin-st08:35:003
3080605993828,7cyclictest37-21ksoftirqd/311:00:013
3080605993824,11cyclictest3088715-21sed09:45:013
308060299388,5cyclictest131rcu_preempt11:55:022
308060299388,5cyclictest131rcu_preempt11:55:012
3080593993832,4cyclictest0-21swapper/007:35:010
3080593993830,5cyclictest3088325-21sendmail09:40:000
3080593993826,9cyclictest12-21ksoftirqd/007:15:000
3080593993826,9cyclictest1-21systemd11:45:010
3080593993824,10cyclictest363-21snmpd12:14:270
308059399381,4cyclictest131rcu_preempt10:59:100
308059399381,34cyclictest3083376-21sshd08:04:380
308060599379,6cyclictest131rcu_preempt09:40:003
308060599377,6cyclictest131rcu_preempt12:30:023
308060599374,27cyclictest361rcuc/307:15:023
308060599372,7cyclictest3087285-21cron09:19:593
308060599372,5cyclictest131rcu_preempt10:39:593
3080605993725,9cyclictest3084415-21sshd08:23:473
3080605993721,12cyclictest3083127-21systemctl08:00:013
3080605993720,13cyclictest131rcu_preempt08:20:033
3080605993720,13cyclictest131rcu_preempt08:20:023
3080605993720,12cyclictest3095599-21readlink12:00:003
3080605993718,5cyclictest131rcu_preempt09:09:593
3080605993716,7cyclictest131rcu_preempt10:35:013
308059399378,24cyclictest0-21swapper/009:56:520
3080593993728,6cyclictest19050irq/34-mmc110:23:540
3080593993728,6cyclictest12-21ksoftirqd/008:10:020
308059399372,6cyclictest3083909-21munin-run08:15:010
3080593993725,8cyclictest235-21systemd-journal08:35:010
3080593993725,8cyclictest235-21systemd-journal08:35:000
308059399372,4cyclictest131rcu_preempt08:30:020
3080593993724,9cyclictest12-21ksoftirqd/010:47:490
3080593993724,9cyclictest12-21ksoftirqd/010:47:480
308059399371,33cyclictest1-21systemd11:10:000
3080593993711,22cyclictest0-21swapper/011:20:530
3080593993711,21cyclictest0-21swapper/010:17:520
3080593993711,21cyclictest0-21swapper/010:17:510
3080605993628,4cyclictest3096616-21cron12:20:013
3080605993628,4cyclictest3096616-21cron12:20:013
3080605993626,6cyclictest37-21ksoftirqd/312:05:033
3080605993626,5cyclictest37-21ksoftirqd/311:45:023
3080605993625,7cyclictest37-21ksoftirqd/307:45:013
3080605993624,9cyclictest1-21systemd10:30:013
3080605993624,8cyclictest272-21dbus-daemon12:25:003
3080605993624,8cyclictest235-21systemd-journal08:45:003
3080605993622,10cyclictest37-21ksoftirqd/311:05:473
3080605993620,12cyclictest235-21systemd-journal07:55:003
3080593993626,7cyclictest3092180-21sshd10:50:160
3080593993626,7cyclictest3085181-21cron08:40:000
308059399362,5cyclictest3097401-21cron12:35:010
308059399362,5cyclictest3097401-21cron12:35:000
3080593993623,10cyclictest3094191-21sed11:30:020
3080593993622,10cyclictest306-21in:imuxsock11:49:590
3080593993619,13cyclictest1-21systemd10:10:000
3080593993619,10cyclictest0-21swapper/010:30:000
308059399361,32cyclictest0-21swapper/011:15:010
3080605993526,5cyclictest3081828-21/usr/sbin/munin07:35:003
3080605993525,7cyclictest37-21ksoftirqd/311:17:013
3080605993525,6cyclictest306-21in:imuxsock08:10:013
3080605993524,8cyclictest37-21ksoftirqd/307:50:013
3080605993524,7cyclictest3085739-21pluginstate08:50:023
3080605993524,7cyclictest3085739-21pluginstate08:50:013
3080605993523,9cyclictest37-21ksoftirqd/308:30:013
3080605993523,8cyclictest235-21systemd-journal10:25:003
308060599351,8cyclictest3097552-21grep12:35:033
308060599351,8cyclictest3097552-21grep12:35:023
308060599351,3cyclictest131rcu_preempt10:15:023
3080605993510,7cyclictest37-21ksoftirqd/309:55:003
3080605993510,6cyclictest131rcu_preempt11:49:593
3080593993525,8cyclictest12-21ksoftirqd/009:15:480
308059399351,9cyclictest1-21systemd09:02:480
308059399351,31cyclictest1-21systemd09:14:530
308060599348,4cyclictest131rcu_preempt12:10:433
3080605993426,4cyclictest412-21sshd09:01:483
308060599342,3cyclictest131rcu_preempt12:10:003
3080593993427,4cyclictest0-21swapper/011:56:040
308059399342,4cyclictest131rcu_preempt09:44:590
3080593993424,7cyclictest12-21ksoftirqd/007:45:030
3080593993420,11cyclictest12-21ksoftirqd/012:40:010
308059399341,4cyclictest3094589-21systemd-run11:40:010
308059399341,4cyclictest0-21swapper/012:19:270
308059399341,4cyclictest0-21swapper/012:19:260
308059399341,29cyclictest0-21swapper/008:51:480
308060599335,3cyclictest131rcu_preempt09:25:013
3080605993317,12cyclictest272-21dbus-daemon11:15:013
308060599331,3cyclictest131rcu_preempt10:45:023
308060599325,5cyclictest131rcu_preempt11:00:223
308060599325,5cyclictest131rcu_preempt11:00:223
308060599322,5cyclictest131rcu_preempt11:55:013
308060599322,5cyclictest131rcu_preempt11:55:003
3080605993223,6cyclictest3085486-21kworker/3:210:00:003
3080593993224,5cyclictest19050irq/34-mmc107:49:470
308060599312,5cyclictest131rcu_preempt07:25:003
3080605993113,7cyclictest131rcu_preempt10:20:013
3080605993113,7cyclictest131rcu_preempt10:20:003
308059399311,27cyclictest3080646-21sed07:10:020
308060599301,5cyclictest131rcu_preempt07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional