You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-02 - 10:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot3s.osadl.org (updated Fri Jan 02, 2026 00:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2521010991061,102cyclictest0-21swapper/000:00:020
2521017998335,43cyclictest238-21systemd-journal23:00:002
2521017997425,9cyclictest238-21systemd-journal20:31:402
2521017997322,12cyclictest238-21systemd-journal00:40:022
2521017996930,4cyclictest2521998-21sshd19:27:222
2521017996927,20cyclictest1-21systemd20:00:002
252101099676,23cyclictest2538252-21sed00:40:020
2521020996526,12cyclictest1-21systemd19:40:013
252101399636,8cyclictest23-21ksoftirqd/100:00:021
252101799624,14cyclictest2529773-21modprobe22:00:002
2521017996229,4cyclictest1-21systemd00:25:002
2521017996227,5cyclictest274-21cron00:00:002
252101099625,18cyclictest2527786-21sed21:20:010
2521017996122,9cyclictest2530823-21cat22:20:002
2521017996023,10cyclictest2533927-21systemctl23:20:002
252101399608,11cyclictest23-21ksoftirqd/100:10:011
252101099596,6cyclictest12-21ksoftirqd/023:52:560
2521010995915,8cyclictest12-21ksoftirqd/020:20:000
252102099584,12cyclictest2529774-21modprobe22:00:003
2521020995826,5cyclictest2526444-21latency_hist20:55:023
252101099584,17cyclictest2534709-21systemctl23:35:010
252101099581,20cyclictest2534075-21sed23:20:030
2521020995740,13cyclictest275-21dbus-daemon22:40:013
2521017995714,7cyclictest131rcu_preempt23:50:012
2521017995714,7cyclictest131rcu_preempt23:50:012
252101399573,10cyclictest23-21ksoftirqd/122:00:021
252101399572,11cyclictest23-21ksoftirqd/100:40:011
2521010995713,7cyclictest12-21ksoftirqd/022:10:010
2521020995615,7cyclictest0-21swapper/322:10:003
2521013995615,9cyclictest23-21ksoftirqd/121:05:021
2521013995611,12cyclictest23-21ksoftirqd/123:15:021
2521013995526,21cyclictest23-21ksoftirqd/120:35:011
2521010995527,3cyclictest0-21swapper/000:30:030
2521010995518,24cyclictest12-21ksoftirqd/023:15:010
2521010995515,10cyclictest2524290-21cat20:10:020
2521020995423,8cyclictest2534184-21cron23:25:003
2521020995423,8cyclictest2534184-21cron23:25:003
2521017995424,10cyclictest275-21dbus-daemon23:45:012
252101399545,9cyclictest23-21ksoftirqd/119:55:021
252101399545,9cyclictest23-21ksoftirqd/119:55:021
252101399543,7cyclictest23-21ksoftirqd/123:40:001
2521013995433,15cyclictest238-21systemd-journal00:15:011
252101399531,11cyclictest2532265-21wc22:45:021
2521010995315,9cyclictest275-21dbus-daemon21:10:010
2521017995211,6cyclictest131rcu_preempt23:15:002
252101399522,7cyclictest23-21ksoftirqd/122:15:011
252101399522,7cyclictest23-21ksoftirqd/122:15:011
2521013995215,13cyclictest23-21ksoftirqd/120:30:001
2521013995215,13cyclictest23-21ksoftirqd/120:30:001
252101399517,12cyclictest23-21ksoftirqd/119:20:001
252102099503,6cyclictest131rcu_preempt21:00:013
2521013995041,6cyclictest221rcuc/121:30:021
2521013995032,13cyclictest238-21systemd-journal22:20:001
252101399501,11cyclictest23-21ksoftirqd/122:40:001
252101099503,11cyclictest0-21swapper/021:00:010
252101799499,8cyclictest30-21ksoftirqd/222:14:402
252101799499,8cyclictest30-21ksoftirqd/222:14:402
252101799494,9cyclictest30-21ksoftirqd/200:01:402
252101799493,7cyclictest131rcu_preempt19:40:002
252101399495,14cyclictest23-21ksoftirqd/120:50:031
252101399495,13cyclictest23-21ksoftirqd/121:25:021
2521013994930,15cyclictest1-21systemd21:00:001
2521010994941,4cyclictest16950irq/224-800000021:45:010
2521010994926,19cyclictest2521203-21sshd19:10:400
2521010994910,7cyclictest0-21swapper/020:35:000
2521017994828,14cyclictest1-21systemd19:55:012
2521017994828,14cyclictest1-21systemd19:55:012
252101399488,10cyclictest23-21ksoftirqd/123:10:021
252101399485,7cyclictest23-21ksoftirqd/100:25:021
252101399484,8cyclictest23-21ksoftirqd/123:45:011
2521013994838,6cyclictest23-21ksoftirqd/122:10:001
252101099486,7cyclictest238-21systemd-journal19:25:000
2521010994839,5cyclictest16950irq/224-800000021:25:010
2521020994731,12cyclictest1-21systemd23:05:023
2521020994729,14cyclictest275-21dbus-daemon20:35:003
2521017994730,13cyclictest1-21systemd21:35:002
2521013994726,12cyclictest309-21in:imuxsock00:30:001
252101399471,8cyclictest2533010-21sort23:00:021
252101099478,14cyclictest2527595-21ls21:15:020
2521010994742,3cyclictest16950irq/224-800000022:15:010
2521010994742,3cyclictest16950irq/224-800000022:15:010
2521010994710,6cyclictest2522642-21sed19:40:020
252102099464,5cyclictest131rcu_preempt21:25:033
252101399466,9cyclictest23-21ksoftirqd/121:15:001
252101399466,9cyclictest23-21ksoftirqd/119:10:011
252101399466,7cyclictest23-21ksoftirqd/119:30:011
252101399466,10cyclictest23-21ksoftirqd/121:10:001
252101399465,8cyclictest23-21ksoftirqd/121:55:001
252101399465,10cyclictest23-21ksoftirqd/119:25:001
2521013994637,7cyclictest23-21ksoftirqd/120:20:001
252101399462,11cyclictest23-21ksoftirqd/120:50:021
2521013994615,6cyclictest131rcu_preempt23:35:021
252101399461,12cyclictest23-21ksoftirqd/119:35:011
2521013994610,8cyclictest23-21ksoftirqd/121:50:011
252101099469,33cyclictest0-21swapper/000:20:010
252101099469,33cyclictest0-21swapper/000:20:000
2521010994629,13cyclictest2521453-21sshd19:15:400
252102099453,4cyclictest131rcu_preempt20:30:003
252102099453,4cyclictest131rcu_preempt20:30:003
2521020994528,11cyclictest37-21ksoftirqd/300:00:013
252102099451,3cyclictest131rcu_preempt21:05:003
2521017994532,10cyclictest2525597-21sshd20:36:402
252101399451,11cyclictest23-21ksoftirqd/121:20:031
252101099456,9cyclictest0-21swapper/023:00:010
2521010994538,4cyclictest16950irq/224-800000000:25:000
252101099451,6cyclictest2526158-21/usr/sbin/munin20:50:010
2521020994431,10cyclictest2524097-21systemd-run20:10:013
2521020994424,14cyclictest275-21dbus-daemon19:35:013
252101799449,16cyclictest2525893-21kworker/2:020:55:022
252101799443,36cyclictest2527176-21cron21:09:592
252101399446,8cyclictest23-21ksoftirqd/119:40:011
252101399444,9cyclictest23-21ksoftirqd/123:50:021
252101399444,9cyclictest23-21ksoftirqd/123:50:021
252101399444,8cyclictest23-21ksoftirqd/100:20:021
252101399444,8cyclictest23-21ksoftirqd/100:20:021
252101399442,8cyclictest23-21ksoftirqd/120:10:001
252101099449,30cyclictest2534414-21sshd23:26:390
252101099445,35cyclictest0-21swapper/021:47:420
2521010994435,6cyclictest16950irq/224-800000020:45:000
2521010994430,11cyclictest2530789-21sshd22:17:400
2521010994429,10cyclictest275-21dbus-daemon19:45:010
2521010994428,11cyclictest309-21in:imuxsock22:00:000
2521010994426,14cyclictest2530071-21munin-run22:05:010
2521010994425,14cyclictest238-21systemd-journal22:30:010
252101099442,36cyclictest2523566-21munin-run20:00:000
2521010994415,26cyclictest1-21systemd00:07:590
252102099433,3cyclictest131rcu_preempt23:20:013
2521020994329,10cyclictest275-21dbus-daemon20:25:013
2521020994327,12cyclictest2525894-21cron20:45:003
252101799433,35cyclictest2534218-21wc23:25:022
252101799433,35cyclictest2534218-21wc23:25:012
252101399435,8cyclictest23-21ksoftirqd/119:15:001
252101399435,11cyclictest23-21ksoftirqd/119:46:401
252101399434,9cyclictest23-21ksoftirqd/100:05:001
2521013994335,6cyclictest221rcuc/123:05:021
2521013994330,9cyclictest238-21systemd-journal20:25:011
2521013994329,11cyclictest23-21ksoftirqd/120:45:011
2521013994328,9cyclictest23-21ksoftirqd/120:00:001
2521013994328,11cyclictest2523821-21cron20:05:001
2521010994339,2cyclictest0-21swapper/021:05:000
2521010994327,12cyclictest12-21ksoftirqd/020:55:030
2521010994326,12cyclictest2532388-21munin-run22:50:020
2521010994326,12cyclictest2532388-21munin-run22:50:010
252101099432,5cyclictest2528739-21modprobe21:40:010
252101099432,5cyclictest2528739-21modprobe21:40:000
252101099432,11cyclictest2524349-21munin-run20:15:010
2521010994311,8cyclictest12-21ksoftirqd/023:45:010
2521010994311,10cyclictest12-21ksoftirqd/019:35:020
252102099425,4cyclictest131rcu_preempt22:57:503
252102099425,10cyclictest37-21ksoftirqd/300:04:333
2521020994229,10cyclictest275-21dbus-daemon20:40:023
2521020994224,12cyclictest238-21systemd-journal19:45:003
2521020994223,14cyclictest275-21dbus-daemon21:40:023
2521020994223,14cyclictest275-21dbus-daemon21:40:013
2521020994219,17cyclictest275-21dbus-daemon00:40:013
252101799424,3cyclictest131rcu_preempt20:05:022
252101799423,6cyclictest131rcu_preempt19:25:002
2521017994232,7cyclictest30-21ksoftirqd/221:40:032
2521017994220,16cyclictest275-21dbus-daemon00:35:012
2521013994230,8cyclictest2534191-21cron23:25:011
2521013994230,8cyclictest2534191-21cron23:25:001
252101099429,10cyclictest2531305-21latency_hist22:25:030
2521010994226,12cyclictest2532099-21sshd22:42:400
2521010994220,16cyclictest275-21dbus-daemon00:15:000
252102099415,4cyclictest131rcu_preempt21:10:023
252102099415,10cyclictest37-21ksoftirqd/319:15:343
2521020994129,9cyclictest2533386-21sshd23:08:403
2521020994128,9cyclictest2532438-21ls22:50:033
2521020994128,9cyclictest2532438-21ls22:50:023
252102099412,4cyclictest131rcu_preempt22:54:593
252102099411,4cyclictest131rcu_preempt22:15:013
252102099411,4cyclictest131rcu_preempt22:15:013
2521017994127,10cyclictest1-21systemd19:35:012
252101799412,35cyclictest2535773-21sed23:55:022
252101799411,5cyclictest131rcu_preempt21:20:012
2521013994131,6cyclictest221rcuc/120:40:001
2521013994129,8cyclictest23-21ksoftirqd/122:30:411
252101399411,8cyclictest23-21ksoftirqd/123:54:401
2521013994112,5cyclictest131rcu_preempt22:25:001
2521010994133,5cyclictest2535474-21sh23:50:000
2521010994133,5cyclictest2535474-21sh23:50:000
2521010994126,11cyclictest2533121-21sshd23:00:410
252101099411,8cyclictest2536269-21munin-run00:05:010
252102099408,5cyclictest131rcu_preempt22:26:403
252102099402,4cyclictest131rcu_preempt23:35:023
2521020994019,15cyclictest238-21systemd-journal22:45:003
252101799403,4cyclictest131rcu_preempt21:50:002
252101799402,6cyclictest131rcu_preempt23:40:002
2521017994025,11cyclictest238-21systemd-journal00:20:012
2521017994025,11cyclictest238-21systemd-journal00:20:002
252101399407,6cyclictest131rcu_preempt21:45:001
252101399404,10cyclictest23-21ksoftirqd/122:27:461
2521013994028,10cyclictest23-21ksoftirqd/123:27:401
2521013994025,10cyclictest23-21ksoftirqd/121:40:021
2521013994025,10cyclictest23-21ksoftirqd/121:40:021
252101399401,8cyclictest23-21ksoftirqd/121:30:031
252101099402,6cyclictest2531859-21modprobe22:40:010
2521010994026,10cyclictest2533375-21sshd23:06:400
2521010994024,12cyclictest2528478-21sshd21:34:400
2521020993928,6cyclictest309-21in:imuxsock23:50:023
2521020993924,12cyclictest37-21ksoftirqd/319:55:023
2521020993924,12cyclictest37-21ksoftirqd/319:55:023
252102099391,3cyclictest131rcu_preempt00:15:023
252101799395,4cyclictest131rcu_preempt00:30:012
2521017993930,6cyclictest2524583-21cron20:20:002
252101799392,8cyclictest2526181-21wc20:50:022
2521017993923,12cyclictest1-21systemd19:15:012
2521017993922,13cyclictest2521514-21tr19:20:002
2521013993929,7cyclictest23-21ksoftirqd/122:50:021
2521013993929,7cyclictest23-21ksoftirqd/122:50:011
2521013993919,8cyclictest23-21ksoftirqd/123:20:021
252101099399,18cyclictest2521008-21ls19:10:010
252101099397,28cyclictest151rcuc/020:25:020
252101099395,4cyclictest151rcuc/020:40:000
2521010993925,11cyclictest2534922-21sshd23:37:390
2521010993924,11cyclictest2528215-21sshd21:27:390
2521010993923,11cyclictest2532627-21/usr/sbin/munin22:55:000
2521010993922,12cyclictest2523835-21munin-run20:05:000
252101099391,34cyclictest2523283-21sshd19:52:400
252101099391,34cyclictest2523283-21sshd19:52:400
2521020993826,8cyclictest2524321-21sshd20:14:403
252102099382,5cyclictest131rcu_preempt21:35:003
252102099382,4cyclictest131rcu_preempt21:50:003
252102099382,3cyclictest131rcu_preempt19:50:013
2521020993823,10cyclictest309-21in:imuxsock23:15:003
2521020993821,4cyclictest37-21ksoftirqd/323:25:023
252101799385,4cyclictest131rcu_preempt21:55:012
252101799383,3cyclictest131rcu_preempt21:40:022
2521017993825,9cyclictest275-21dbus-daemon00:15:002
2521017993824,9cyclictest275-21dbus-daemon21:25:012
252101399382,8cyclictest23-21ksoftirqd/122:55:001
2521013993827,7cyclictest1-21systemd19:45:001
2521010993827,8cyclictest12-21ksoftirqd/022:35:010
2521010993825,9cyclictest2529508-21sshd21:54:400
2521010993824,10cyclictest2523037-21sshd19:49:400
2521010993817,17cyclictest2537782-21sshd00:31:390
2521020993728,6cyclictest2530817-21sh22:20:003
2521020993726,8cyclictest2526170-21pluginstate20:50:013
2521020993721,12cyclictest2534940-21modprobe23:39:593
2521020993721,11cyclictest0-21swapper/320:05:003
2521020993720,13cyclictest2531282-21cut22:25:033
2521020993718,13cyclictest275-21dbus-daemon21:45:003
252102099371,4cyclictest131rcu_preempt19:25:013
252101799374,3cyclictest131rcu_preempt20:45:032
252101799373,4cyclictest131rcu_preempt22:35:012
2521017993726,6cyclictest309-21in:imuxsock23:35:002
252101799371,6cyclictest0-21swapper/222:30:032
2521010993724,9cyclictest2534180-21sshd23:24:390
2521010993724,9cyclictest2534180-21sshd23:24:390
2521010993724,9cyclictest2521994-21sshd19:26:400
2521010993723,10cyclictest1-21systemd20:30:010
2521010993723,10cyclictest1-21systemd20:30:010
252102099363,8cyclictest37-21ksoftirqd/300:06:243
2521020993623,9cyclictest309-21in:imuxsock23:45:023
2521020993623,9cyclictest309-21in:imuxsock23:45:023
2521020993620,12cyclictest275-21dbus-daemon00:25:003
2521020993620,11cyclictest2524619-21tr20:20:003
252102099361,4cyclictest131rcu_preempt21:30:003
252101799366,24cyclictest2524876-21grep20:25:002
252101799363,4cyclictest131rcu_preempt21:13:412
252101799363,3cyclictest131rcu_preempt21:30:022
252101799362,4cyclictest131rcu_preempt21:05:022
2521017993621,11cyclictest275-21dbus-daemon22:40:002
2521017993615,11cyclictest0-21swapper/221:00:012
252101399363,6cyclictest23-21ksoftirqd/122:03:401
2521013993620,10cyclictest238-21systemd-journal20:15:001
2521020993522,10cyclictest2527425-21sshd21:12:383
252101799355,6cyclictest131rcu_preempt22:09:402
252101799354,23cyclictest2533159-21munin-plugin-st23:05:002
252101799354,20cyclictest238-21systemd-journal19:50:002
2521017993525,6cyclictest309-21in:imuxsock22:25:002
252101799352,4cyclictest131rcu_preempt22:45:032
252101799352,3cyclictest131rcu_preempt19:45:012
2521013993518,11cyclictest221rcuc/100:33:451
252102099343,27cyclictest2537889-21ls00:35:013
2521020993427,4cyclictest1-21systemd00:30:013
2521020993420,11cyclictest1-21systemd22:05:003
2521020993412,6cyclictest131rcu_preempt00:16:373
2521020993412,6cyclictest131rcu_preempt00:16:363
252101799344,3cyclictest131rcu_preempt22:55:022
252101799342,5cyclictest131rcu_preempt21:45:012
252101799342,4cyclictest131rcu_preempt23:07:412
2521017993412,18cyclictest275-21dbus-daemon22:05:002
252102099337,14cyclictest2527706-21modprobe21:20:003
252102099334,3cyclictest131rcu_preempt21:55:013
252102099332,3cyclictest131rcu_preempt19:10:013
2521020993317,12cyclictest238-21systemd-journal20:00:003
252101799335,3cyclictest131rcu_preempt22:50:012
252101799335,3cyclictest131rcu_preempt22:50:012
252101799334,3cyclictest131rcu_preempt20:25:402
252101799334,3cyclictest131rcu_preempt20:25:402
252101799332,5cyclictest131rcu_preempt20:15:012
2521017993317,11cyclictest0-21swapper/220:10:002
252101799331,5cyclictest0-21swapper/200:05:412
2521020993224,5cyclictest2522126-21latency_hist19:30:023
2521020993220,6cyclictest309-21in:imuxsock22:35:003
252102099321,4cyclictest131rcu_preempt19:15:013
252101799324,3cyclictest131rcu_preempt23:27:402
252102099312,3cyclictest131rcu_preempt23:45:013
252101799315,3cyclictest131rcu_preempt19:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional