You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-14 - 16:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Mar 14, 2026 12:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
223111599506499,4cyclictest2331596-21turbostat11:55:013
223111599505497,5cyclictest2324664-21turbostat11:39:553
223110399478454,21cyclictest2344008-21turbostat12:34:550
223111599475434,26cyclictest2329864-21turbostat11:50:003
223110799469457,9cyclictest2245770-21turbostat07:54:551
223111599466457,6cyclictest2345752-21turbostat12:35:003
223111599462443,16cyclictest2321210-21turbostat11:25:003
223110799462455,4cyclictest2266576-21turbostat08:54:551
223110799462454,5cyclictest2300436-21turbostat10:29:551
223111199461454,4cyclictest2279357-21turbostat09:29:552
223111199461454,4cyclictest2252691-21turbostat08:14:552
223111599458450,5cyclictest2310819-21turbostat10:59:553
223111599458449,6cyclictest2288009-21turbostat09:50:003
223111599453446,4cyclictest2317737-21turbostat11:15:003
223111599453445,5cyclictest2310819-21turbostat10:55:003
223111599451443,5cyclictest2256147-21turbostat08:20:003
223111199449442,4cyclictest2296662-21turbostat10:19:552
223111599448439,6cyclictest2316001-21turbostat11:10:013
223111599443434,6cyclictest2240216-21turbostat07:34:593
223111599443434,6cyclictest2240216-21turbostat07:34:593
223111199443436,4cyclictest2309079-21turbostat10:54:552
223111599440431,6cyclictest2244031-21turbostat07:45:013
223111199438431,4cyclictest2276956-21turbostat09:24:552
223111199438429,6cyclictest2271762-21turbostat09:09:552
223111199438429,6cyclictest2271762-21turbostat09:09:552
223111599437428,6cyclictest2302159-21turbostat10:34:553
223111599432425,4cyclictest2247496-21turbostat07:55:003
223111599426418,5cyclictest2344008-21turbostat12:30:003
223111599425416,6cyclictest2259615-21turbostat08:30:003
223110399425418,4cyclictest2289745-21turbostat09:59:550
223110399425418,4cyclictest2289745-21turbostat09:59:550
223111599422415,4cyclictest2314277-21turbostat11:05:003
223111599422415,4cyclictest2250957-21turbostat08:09:553
223111199421414,4cyclictest2286282-21turbostat09:49:552
223111199421412,6cyclictest2303897-21turbostat10:39:552
223111599420411,6cyclictest2240216-21turbostat07:39:543
223111599419411,5cyclictest2347480-21turbostat12:40:003
223111199419410,6cyclictest2338519-21turbostat12:19:552
223111199419410,6cyclictest2338519-21turbostat12:19:552
223110399417408,6cyclictest2326397-21turbostat11:44:550
223111599416407,6cyclictest2322936-21turbostat11:30:003
223111199416409,4cyclictest2319465-21turbostat11:24:552
223110399416409,4cyclictest2230973-21turbostat07:14:550
223111599414407,4cyclictest2305622-21turbostat10:40:003
223111199414405,6cyclictest2244031-21turbostat07:49:552
223111599413406,4cyclictest2289745-21turbostat09:55:003
223111199413406,4cyclictest2322936-21turbostat11:34:552
223111599412405,4cyclictest2291474-21turbostat10:00:003
223111599412405,4cyclictest2291474-21turbostat10:00:003
223111599412403,6cyclictest2338519-21turbostat12:15:003
223111599406394,8cyclictest2340247-21turbostat12:20:013
223111599406394,8cyclictest2340247-21turbostat12:20:013
223111199406399,4cyclictest2336793-21turbostat12:14:552
223111599402395,4cyclictest2261336-21turbostat08:35:003
223111599400380,17cyclictest2266576-21turbostat08:50:003
223111599398389,6cyclictest2264848-21turbostat08:45:003
223111599396389,4cyclictest2257867-21turbostat08:25:013
223111599396389,4cyclictest2257867-21turbostat08:25:003
223111599396386,6cyclictest2302159-21turbostat10:30:003
223110399396370,23cyclictest2263061-21turbostat08:44:560
223111599392384,4cyclictest2276956-21turbostat09:20:003
223110399391384,4cyclictest2294938-21turbostat10:14:550
223111599386379,4cyclictest2270032-21turbostat09:04:553
223110399386373,10cyclictest2259615-21turbostat08:34:550
223110799385376,6cyclictest2329864-21turbostat11:54:551
223111599384377,4cyclictest2333331-21turbostat12:00:003
223111599383376,4cyclictest2319465-21turbostat11:20:003
223111599382374,5cyclictest2324664-21turbostat11:35:003
223110399382376,4cyclictest2264848-21turbostat08:49:550
223111599374368,3cyclictest2291474-21turbostat10:04:553
223111599373367,3cyclictest2234974-21turbostat07:24:553
223111599371363,5cyclictest2294938-21turbostat10:10:003
223111599367357,7cyclictest2305622-21turbostat10:44:553
223111599367357,7cyclictest2305622-21turbostat10:44:553
223111599365358,4cyclictest2273488-21turbostat09:14:563
223111599365358,4cyclictest2234974-21turbostat07:20:013
223111599364356,5cyclictest2247496-21turbostat07:59:553
223111599364356,5cyclictest2247496-21turbostat07:59:553
223110799363357,3cyclictest2293209-21turbostat10:09:551
223110399363355,5cyclictest2345752-21turbostat12:39:550
223111599360354,3cyclictest2336793-21turbostat12:10:003
223111599358352,3cyclictest2342291-21turbostat12:25:003
223110399358352,3cyclictest2307357-21turbostat10:49:560
223110799357351,3cyclictest2236712-21turbostat07:29:551
223111599356326,27cyclictest2281097-21turbostat09:30:003
223111199356348,5cyclictest2238447-21turbostat07:34:552
223111199356348,5cyclictest2238447-21turbostat07:34:552
223111599354347,4cyclictest2250957-21turbostat08:05:013
223111199354346,5cyclictest2249232-21turbostat08:04:552
223111599351343,5cyclictest2286282-21turbostat09:45:003
223111599349343,3cyclictest2298702-21turbostat10:24:563
223111199344337,4cyclictest2282818-21turbostat09:39:552
223110399343334,6cyclictest2333331-21turbostat12:04:550
223111199342334,5cyclictest2281097-21turbostat09:34:552
223111199342334,5cyclictest2281097-21turbostat09:34:552
223111599341334,4cyclictest2284548-21turbostat09:40:003
223111599340332,5cyclictest2282818-21turbostat09:35:013
223111599340332,5cyclictest2282818-21turbostat09:35:003
223111599332324,5cyclictest2268300-21turbostat08:55:003
223111599331325,3cyclictest2279357-21turbostat09:25:003
223110399327319,5cyclictest2275224-21turbostat09:19:550
223111599326319,4cyclictest2328130-21turbostat11:45:003
223110799326318,5cyclictest2288009-21turbostat09:54:551
223110799326318,5cyclictest2257867-21turbostat08:29:551
223111599325318,4cyclictest2263061-21turbostat08:40:003
223110399325318,4cyclictest2261336-21turbostat08:39:550
223111599320312,5cyclictest2254415-21turbostat08:15:003
223110799313307,3cyclictest2268300-21turbostat08:59:551
223110799312306,3cyclictest2232750-21turbostat07:19:551
223111599291284,4cyclictest2273488-21turbostat09:10:003
223111599291284,4cyclictest2273488-21turbostat09:10:003
223111599258244,10cyclictest2238447-21turbostat07:30:003
223111599198184,10cyclictest2232750-21turbostat07:15:003
223111599183167,11cyclictest2309079-21turbostat10:50:003
223111599127114,9cyclictest2296662-21turbostat10:15:003
2230888210976,23sleep00-21swapper/007:09:160
2230762210875,23sleep20-21swapper/207:07:372
2230736210875,23sleep30-21swapper/307:07:183
22311159910793,10cyclictest2335048-21turbostat12:05:003
2230601210775,22sleep10-21swapper/107:05:311
22311159910493,8cyclictest2245770-21turbostat07:50:003
22311159910491,10cyclictest2298702-21turbostat10:20:003
22311159910089,8cyclictest2270032-21turbostat09:00:003
2231111999790,4cyclictest2340247-21turbostat12:24:562
23441922960,5sleep20-21swapper/212:30:022
23408592940,6sleep00-21swapper/012:20:140
2231111999487,4cyclictest2284548-21turbostat09:44:552
23453312830,16sleep126-21ksoftirqd/112:30:291
22550992780,4sleep00-21swapper/008:15:200
23063742650,5sleep24497-21PK-Backend10:40:182
23063742650,5sleep24497-21PK-Backend10:40:182
23410592540,5sleep10-21swapper/112:20:151
22915052540,5sleep12291472-21latency_hist10:00:011
22915052540,5sleep12291472-21latency_hist10:00:011
23197842520,8sleep00-21swapper/011:20:130
2231103995028,13cyclictest13-21ksoftirqd/011:45:210
2231103994929,16cyclictest13-21ksoftirqd/008:50:200
2231103994927,15cyclictest13-21ksoftirqd/010:55:140
2231103994925,14cyclictest13-21ksoftirqd/011:40:010
2231103994833,12cyclictest13-21ksoftirqd/010:08:070
2231103994830,6cyclictest13-21ksoftirqd/009:40:010
22993702470,5sleep2331rcuc/210:20:182
2231103994725,6cyclictest13-21ksoftirqd/007:50:210
2231103994725,14cyclictest13-21ksoftirqd/010:35:230
2231103994628,6cyclictest13-21ksoftirqd/008:15:020
2231103994626,6cyclictest13-21ksoftirqd/012:05:130
2231103994622,7cyclictest13-21ksoftirqd/007:20:000
2231103994622,15cyclictest13-21ksoftirqd/009:25:290
2231103994620,13cyclictest13-21ksoftirqd/008:55:210
223110799452,17cyclictest2296771-21cat10:15:021
2231107994512,12cyclictest141rcu_preempt11:00:121
2231103994527,5cyclictest13-21ksoftirqd/009:40:130
2231103994524,6cyclictest13-21ksoftirqd/011:50:130
2231103994524,6cyclictest13-21ksoftirqd/011:00:200
2231103994524,6cyclictest13-21ksoftirqd/009:30:130
2231103994524,6cyclictest13-21ksoftirqd/009:30:130
2231103994524,6cyclictest13-21ksoftirqd/007:30:260
2231103994524,6cyclictest13-21ksoftirqd/007:30:260
2231103994523,7cyclictest13-21ksoftirqd/009:00:200
2231103994521,15cyclictest13-21ksoftirqd/010:15:120
22448872440,8sleep00-21swapper/007:45:210
2231111994425,6cyclictest34-21ksoftirqd/212:00:182
2231107994421,6cyclictest141rcu_preempt07:30:281
2231107994421,6cyclictest141rcu_preempt07:30:281
2231103994426,6cyclictest13-21ksoftirqd/009:50:130
2231103994424,6cyclictest13-21ksoftirqd/012:25:210
2231103994424,6cyclictest13-21ksoftirqd/007:35:130
2231103994424,6cyclictest13-21ksoftirqd/007:25:170
2231103994423,6cyclictest13-21ksoftirqd/012:15:020
2231103994423,6cyclictest13-21ksoftirqd/010:40:130
2231103994423,6cyclictest13-21ksoftirqd/010:40:120
2231103994423,6cyclictest13-21ksoftirqd/010:25:290
2231103994423,6cyclictest13-21ksoftirqd/010:25:010
2231103994423,6cyclictest13-21ksoftirqd/010:00:220
2231103994422,7cyclictest13-21ksoftirqd/009:10:200
2231103994422,6cyclictest13-21ksoftirqd/009:45:130
2231103994422,6cyclictest13-21ksoftirqd/007:25:020
2231103994422,5cyclictest13-21ksoftirqd/010:50:140
2231103994421,6cyclictest13-21ksoftirqd/011:55:180
2231103994421,6cyclictest13-21ksoftirqd/010:30:150
2231103994421,6cyclictest13-21ksoftirqd/009:20:270
2231103994421,5cyclictest13-21ksoftirqd/011:20:010
22385762430,15sleep2331rcuc/207:30:032
2231111994325,6cyclictest34-21ksoftirqd/211:45:012
2231111994325,6cyclictest34-21ksoftirqd/208:35:172
2231107994316,9cyclictest2276085-21missed_timers09:15:211
223110799431,21cyclictest2245197-21ls07:45:281
2231103994323,6cyclictest13-21ksoftirqd/007:41:280
2231103994322,6cyclictest13-21ksoftirqd/011:15:010
2231103994322,6cyclictest13-21ksoftirqd/011:05:290
2231103994322,6cyclictest13-21ksoftirqd/008:30:020
2231103994322,6cyclictest13-21ksoftirqd/008:20:150
2231103994322,6cyclictest13-21ksoftirqd/008:20:140
22660522420,7sleep234-21ksoftirqd/208:45:282
2231111994224,6cyclictest34-21ksoftirqd/211:55:012
223111199421,4cyclictest141rcu_preempt11:40:002
2231107994212,12cyclictest141rcu_preempt10:55:231
2231107994210,12cyclictest141rcu_preempt10:30:211
2231103994223,6cyclictest13-21ksoftirqd/011:35:000
223111199418,5cyclictest141rcu_preempt07:15:162
2231111994122,6cyclictest34-21ksoftirqd/211:05:132
2231111994121,6cyclictest34-21ksoftirqd/209:50:142
2231111994121,6cyclictest34-21ksoftirqd/207:20:282
2231111994117,13cyclictest34-21ksoftirqd/208:44:382
2231107994126,11cyclictest2279340-21cron09:25:001
223110799411,11cyclictest2287878-21sshd09:48:291
23149152400,5sleep1251rcuc/111:05:171
2231111994028,7cyclictest2321639-21tr11:25:132
2231111994021,6cyclictest34-21ksoftirqd/208:20:132
2231111994021,6cyclictest34-21ksoftirqd/208:20:132
2231111994020,6cyclictest34-21ksoftirqd/209:10:272
2231111994020,6cyclictest34-21ksoftirqd/207:10:162
223110799406,19cyclictest26-21ksoftirqd/110:20:501
2231107994026,10cyclictest2317717-21cron11:15:001
223110799401,21cyclictest2332450-21cat11:55:201
2231111993922,7cyclictest34-21ksoftirqd/210:30:172
2231111993922,6cyclictest34-21ksoftirqd/208:54:312
2231111993921,6cyclictest34-21ksoftirqd/211:49:492
2231111993921,6cyclictest34-21ksoftirqd/210:29:082
2231111993919,6cyclictest34-21ksoftirqd/208:58:122
2231111993917,18cyclictest2259987-21/usr/sbin/munin08:30:142
2231107993920,14cyclictest2088-21gnome-shell08:44:381
223110799391,8cyclictest438-20systemd-journal11:50:001
2231107993913,9cyclictest26-21ksoftirqd/109:05:301
2231107993913,9cyclictest26-21ksoftirqd/109:05:301
223110799391,22cyclictest2304659-21kernelversion10:35:211
223110799391,21cyclictest2307339-21cron10:49:561
2231111993822,5cyclictest34-21ksoftirqd/207:55:002
2231111993821,12cyclictest438-20systemd-journal08:10:002
2231111993816,13cyclictest34-21ksoftirqd/210:10:212
223110799382,16cyclictest2324929-21apt11:35:111
2231107993820,14cyclictest2254034-21users08:10:301
2231107993813,6cyclictest141rcu_preempt11:25:001
223110799381,23cyclictest2336792-21munin-plugin-st12:10:001
2231103993819,6cyclictest13-21ksoftirqd/009:10:020
2231103993819,6cyclictest13-21ksoftirqd/009:10:010
223111199372,5cyclictest141rcu_preempt12:35:262
2231111993720,6cyclictest34-21ksoftirqd/210:10:002
2231111993719,14cyclictest34-21ksoftirqd/209:00:182
223111199371,18cyclictest924-21NetworkManager07:36:492
223110799376,18cyclictest2292238-21kernelversion10:00:221
223110799372,31cyclictest2243705-40lsb_release07:41:241
223110799372,30cyclictest2236110-21python307:20:271
2231107993722,11cyclictest2266662-21sed08:50:011
2231107993721,12cyclictest1036-21gdbus09:00:221
2231107993719,14cyclictest2274293-21sed09:10:201
2231107993714,11cyclictest141rcu_preempt08:20:011
223110799371,10cyclictest0-21swapper/107:37:411
2231103993721,5cyclictest13-21ksoftirqd/008:04:380
2231103993720,5cyclictest13-21ksoftirqd/008:05:200
223111199365,11cyclictest34-21ksoftirqd/210:48:312
2231111993621,11cyclictest923-21dbus-daemon12:35:012
2231111993620,12cyclictest2243254-21ntpq07:40:242
2231111993617,12cyclictest34-21ksoftirqd/210:00:192
2231111993613,14cyclictest34-21ksoftirqd/211:00:232
223110799362,19cyclictest2259847-21/usr/sbin/munin08:30:271
223110799362,14cyclictest0-21swapper/111:15:141
2231107993620,12cyclictest2347188-21sshd12:36:291
2231107993620,12cyclictest2298430-40apt-check10:17:231
2231107993616,15cyclictest2256771-21df08:20:151
2231107993616,15cyclictest2256771-21df08:20:151
223110799361,30cyclictest438-20systemd-journal09:34:551
223110799361,30cyclictest438-20systemd-journal09:34:551
2231107993612,8cyclictest356-21jbd2/sdb2-809:25:131
223110799361,18cyclictest2286269-21cron09:45:011
2231103993620,12cyclictest2339583-21grep12:15:240
2231103993620,12cyclictest2339583-21grep12:15:240
223111199356,6cyclictest141rcu_preempt09:55:242
223111199356,6cyclictest141rcu_preempt09:55:242
223111199352,10cyclictest0-21swapper/212:10:022
2231111993517,14cyclictest34-21ksoftirqd/211:59:402
223110799352,16cyclictest2307355-21sh10:45:011
223110799352,16cyclictest2307355-21sh10:45:011
223110799351,30cyclictest4497-21PK-Backend12:12:321
223110799351,30cyclictest4497-21PK-Backend11:30:191
223110799351,17cyclictest2343421-21python312:25:291
2231107993510,6cyclictest141rcu_preempt10:50:141
2231103993518,5cyclictest13-21ksoftirqd/007:55:160
2231103993518,5cyclictest13-21ksoftirqd/007:55:160
2231111993418,11cyclictest438-20systemd-journal08:18:292
2231111993417,12cyclictest2276963-21munin-run09:20:002
223111199341,4cyclictest141rcu_preempt10:55:272
223110799341,16cyclictest0-21swapper/109:40:011
223110799341,15cyclictest4497-21PK-Backend08:05:241
2231111993318,11cyclictest2258225-21cpuspeed08:25:132
223111199331,29cyclictest4497-21PK-Backend11:15:002
223110799339,8cyclictest141rcu_preempt11:40:211
223110799334,12cyclictest141rcu_preempt07:55:131
223110799334,12cyclictest141rcu_preempt07:55:121
2231107993323,5cyclictest26-21ksoftirqd/111:28:351
223110799332,12cyclictest141rcu_preempt08:35:311
2231107993319,10cyclictest2231276-21awk07:10:101
223110799331,4cyclictest4497-21PK-Backend08:00:471
223110799331,18cyclictest2333999-21idleruntime12:00:201
2231111993216,11cyclictest2247955-21cstates07:55:152
2231111993216,11cyclictest2247955-21cstates07:55:152
223111199321,28cyclictest0-21swapper/211:16:342
223110799327,12cyclictest141rcu_preempt12:15:131
223110799327,12cyclictest141rcu_preempt12:15:131
2231103993215,13cyclictest13-21ksoftirqd/011:30:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional