You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-09 - 07:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #f, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackfslot7s.osadl.org (updated Sun Nov 09, 2025 00:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
208232699311514,3097cyclictest2129138-21kworker/1:3+events21:40:181
208232699272211,2707cyclictest2087110-21kworker/1:2+events19:30:211
208233199271913,2702cyclictest2093818-21kworker/2:2+events19:50:212
208233199263713,2620cyclictest2151216-21kworker/2:0+events23:05:172
208233399256813,2550cyclictest2106734-21kworker/3:0+events20:40:133
208233199252112,2505cyclictest2135310-21kworker/2:1+events21:55:152
208233199250510,2491cyclictest2172000-21kworker/2:1+events23:51:472
208233199249212,2476cyclictest2165408-21kworker/2:0+events23:48:022
20823339924909,2477cyclictest2122707-21kworker/3:1+events21:15:143
208233199248912,2473cyclictest2125639-21kworker/2:2+events22:08:122
208233199248212,2467cyclictest2125639-21kworker/2:2+events21:53:432
208233199247712,2462cyclictest2117671-21kworker/2:2+events20:58:062
208232699247712,2463cyclictest2102008-21kworker/1:2+events20:20:031
208233199247611,2462cyclictest2087228-21kworker/2:2+events19:23:332
208233199247211,2457cyclictest2125639-21kworker/2:2+events21:45:162
208233199245611,2442cyclictest2141594-21kworker/2:1+events22:23:002
20823319924549,2442cyclictest2156108-21kworker/2:2+events23:00:192
208232699245112,2435cyclictest2133606-21kworker/1:1+events21:52:571
208233199241512,2399cyclictest2119350-21kworker/2:1+events21:05:182
208233199241411,2400cyclictest2103716-21kworker/2:2+events20:25:492
208232699240911,2395cyclictest2127455-21kworker/1:0+events21:30:511
208233399240812,2393cyclictest2100044-21kworker/3:1+events20:07:353
208232699240511,2390cyclictest2113219-21kworker/1:1+events21:00:181
208232699240112,2386cyclictest2168704-21kworker/1:2+events23:44:381
208232699239812,2383cyclictest2162545-21kworker/1:1+events23:34:591
20823339923978,2384cyclictest2145136-21kworker/3:0+events22:40:123
208232699239712,2382cyclictest2181501-21kworker/1:2+events00:18:431
208233399239612,2382cyclictest2121080-21kworker/3:0+events21:10:173
208232699239611,2382cyclictest2120926-21kworker/1:2+events21:10:391
208233399239512,2379cyclictest2128785-21kworker/3:2+events21:50:133
208233399239412,2378cyclictest2180022-21kworker/3:1+events00:20:383
20823339923885,2380cyclictest2130414-21kworker/3:0+events21:41:023
208233399238711,2373cyclictest2148384-21kworker/3:1+events22:53:263
208233399238512,2370cyclictest2159460-21kworker/3:2+events23:33:073
208233199238212,2367cyclictest2127242-21kworker/2:1+events21:35:212
208233399238111,2366cyclictest2180022-21kworker/3:1+events00:13:283
208232699237511,2361cyclictest2138681-21kworker/1:2+events22:20:421
208233199237211,2358cyclictest2087228-21kworker/2:2+events19:35:392
208233399236911,2355cyclictest2090339-21kworker/3:0+events19:33:023
208232299236712,2353cyclictest2096790-21kworker/0:2+events21:31:590
20823269923569,2343cyclictest2096821-21kworker/1:0+events19:56:281
208233399235512,2339cyclictest2157698-21kworker/3:0+events23:06:033
208233199235312,2338cyclictest2159398-21kworker/2:2+events23:23:362
208233399235212,2337cyclictest2114585-21kworker/3:2+events21:07:123
20823339923489,2336cyclictest2140132-21kworker/3:1+events22:09:203
208233399234712,2331cyclictest2143399-21kworker/3:2+events22:20:213
208233199234312,2328cyclictest2117671-21kworker/2:2+events21:04:072
208233399233411,2320cyclictest2148384-21kworker/3:1+events22:58:483
208233199233412,2319cyclictest2121136-21kworker/2:0+events21:17:312
208232299232612,2311cyclictest2085543-21kworker/0:1+events19:19:460
208232299231912,2304cyclictest2130443-21kworker/0:0+events22:05:340
208233399231812,2304cyclictest2145136-21kworker/3:0+events22:25:173
208233199231712,2302cyclictest2145018-21kworker/2:2+events22:30:342
208233399230612,2291cyclictest2085911-21kworker/3:3+events19:24:403
208233399230410,2291cyclictest2145136-21kworker/3:0+events22:32:313
208233399230212,2286cyclictest2148384-21kworker/3:1+events23:03:203
208233199229410,2281cyclictest2093818-21kworker/2:2+events19:48:312
208232699228811,2273cyclictest2159276-21kworker/1:2+events23:10:151
208233399228712,2272cyclictest2135367-21kworker/3:1+events21:55:383
208232699228510,2272cyclictest2133606-21kworker/1:1+events22:00:581
208233199228010,2266cyclictest2125639-21kworker/2:2+events22:00:112
208233399227510,2262cyclictest2133731-21kworker/3:0+events21:49:253
20823269922739,2261cyclictest2108342-21kworker/1:0+events20:33:051
208233199227011,2255cyclictest2087228-21kworker/2:2+events19:34:102
208232299226811,2254cyclictest2167193-21kworker/0:0+events23:45:220
20823319922679,2255cyclictest2125639-21kworker/2:2+events21:29:002
208232699225812,2243cyclictest2186366-21kworker/1:0+events00:39:411
208233399225710,2244cyclictest2170155-21kworker/3:0+events23:49:553
208232699225711,2242cyclictest2100344-21kworker/1:1+events20:12:541
20823319922559,2243cyclictest2159398-21kworker/2:2+events23:17:062
20823319922555,2247cyclictest2141594-21kworker/2:1+events22:16:112
208233199225511,2241cyclictest2173627-21kworker/2:0+events23:55:572
20823319922538,2243cyclictest2175312-21kworker/2:3+events00:00:102
208233199225310,2241cyclictest2175312-21kworker/2:3+events00:10:052
208233199225310,2240cyclictest2125639-21kworker/2:2+events22:10:302
208233199225211,2238cyclictest2152877-21kworker/2:2+events22:50:472
208233199225211,2238cyclictest2087228-21kworker/2:2+events19:28:502
208233199225210,2239cyclictest2141594-21kworker/2:1+events22:26:272
20823319922519,2238cyclictest2103716-21kworker/2:2+events20:37:132
208233199224610,2233cyclictest2165408-21kworker/2:0+events23:34:382
20823269922319,2218cyclictest2159276-21kworker/1:2+events23:09:071
20823229922245,2216cyclictest2082057-21kworker/0:0+events19:26:320
208233399222112,2206cyclictest2124406-21kworker/3:2+events21:22:543
208233399221410,2200cyclictest2106734-21kworker/3:0+events20:45:503
208233399221211,2198cyclictest2180022-21kworker/3:1+events00:15:223
208232699220711,2193cyclictest2144841-21kworker/1:1+events22:41:441
20823269922059,2193cyclictest2154709-21kworker/1:0+events23:02:361
208232699220510,2192cyclictest2125823-21kworker/1:2+events21:37:591
208232699220411,2189cyclictest2148168-21kworker/1:2+events22:55:431
208232699220410,2191cyclictest2151428-21kworker/1:0+events22:48:321
208232699220310,2190cyclictest2090369-21kworker/1:1+events19:39:041
208233399220210,2189cyclictest2159460-21kworker/3:2+events23:38:483
208232699220210,2189cyclictest2102008-21kworker/1:2+events20:17:021
20823269921998,2187cyclictest2144841-21kworker/1:1+events22:37:541
20823269921997,2189cyclictest2168704-21kworker/1:2+events23:54:041
208232699219911,2185cyclictest2148168-21kworker/1:2+events22:34:031
208232699219811,2184cyclictest2183125-21kworker/1:0+events00:25:561
20823269921978,2185cyclictest2096821-21kworker/1:0+events20:09:041
208232699219711,2183cyclictest2160935-21kworker/1:0+events23:22:271
208232699219711,2183cyclictest2146442-21kworker/1:0+events22:28:421
208232699219711,2183cyclictest2108342-21kworker/1:0+events20:40:571
208232699219710,2184cyclictest2183125-21kworker/1:0+events00:24:041
208232699219710,2184cyclictest2160935-21kworker/1:0+events23:17:531
208232699219611,2181cyclictest2168704-21kworker/1:2+events23:57:051
208232699219611,2181cyclictest2090369-21kworker/1:1+events19:53:461
208232699219511,2181cyclictest2113219-21kworker/1:1+events21:08:221
208232699219510,2181cyclictest2080078-21kworker/1:1+events19:25:461
20823269921949,2182cyclictest2135199-21kworker/1:2+events21:56:471
208232699219411,2180cyclictest2125823-21kworker/1:2+events21:29:231
208232699219410,2181cyclictest2173568-21kworker/1:1+events00:02:471
20823269921937,2184cyclictest2100344-21kworker/1:1+events20:25:031
20823269921886,2179cyclictest2141913-21kworker/1:0+events22:15:031
208233399218711,2172cyclictest2159460-21kworker/3:2+events23:40:223
20823269921879,2174cyclictest2090369-21kworker/1:1+events19:44:431
20823269921856,2175cyclictest2164262-21kworker/1:2+events23:26:131
208232699218411,2170cyclictest2162545-21kworker/1:1+events23:35:441
20823269921809,2168cyclictest2184762-21kworker/1:2+events00:31:181
20823319921796,2171cyclictest2162671-21kworker/2:1+events23:25:062
208232699217710,2164cyclictest2124166-21kworker/1:1+events21:23:391
208233399217510,2161cyclictest2140132-21kworker/3:1+events22:17:433
20823269921755,2166cyclictest2153026-21kworker/1:1+events22:51:321
208233399217411,2160cyclictest2173357-21kworker/3:1+events00:00:323
208232699217411,2160cyclictest2138681-21kworker/1:2+events22:13:541
20823319921739,2161cyclictest2100133-21kworker/2:2+events20:03:002
208232699217110,2158cyclictest2114824-21kworker/1:0+events20:54:371
20823339921707,2161cyclictest2173357-21kworker/3:1+events00:05:043
208233399217011,2156cyclictest2106734-21kworker/3:0+events20:30:483
208233399216911,2155cyclictest2116188-21kworker/3:0+events20:59:573
208233399216811,2154cyclictest2185005-21kworker/3:1+events00:38:563
208233399216810,2154cyclictest2173357-21kworker/3:1+events23:53:403
208233399216810,2154cyclictest2128785-21kworker/3:2+events21:36:523
20823339921678,2156cyclictest2159460-21kworker/3:2+events23:15:573
208233399216610,2153cyclictest2185005-21kworker/3:1+events00:32:513
20823339921657,2155cyclictest2083774-21kworker/3:2+events19:25:243
208233399216411,2150cyclictest2181680-21kworker/3:2+events00:27:513
208233399216411,2150cyclictest2108494-21kworker/3:2+events20:36:053
20823319921627,2153cyclictest2184915-21kworker/2:2+events00:30:082
20823339921619,2148cyclictest2162239-21kworker/3:1+events23:24:433
20823339921599,2147cyclictest2145136-21kworker/3:0+events22:35:113
208233399215911,2145cyclictest2116188-21kworker/3:0+events21:01:063
208233399215610,2143cyclictest2145136-21kworker/3:0+events22:49:423
208233399215511,2141cyclictest2100044-21kworker/3:1+events20:16:403
208232299215412,2139cyclictest2159210-21kworker/0:1+events23:08:440
20823339921399,2128cyclictest2093621-21kworker/3:2+events19:45:043
20823269921379,2125cyclictest2124166-21kworker/1:1+events21:18:401
20823269921376,2128cyclictest2136790-21kworker/1:0+events22:05:571
208232299213011,2115cyclictest2173596-21kworker/0:0+events23:56:420
20823339921289,2115cyclictest2100044-21kworker/3:1+events20:02:353
208233199212411,2110cyclictest2165408-21kworker/2:0+events23:38:252
208232299211911,2105cyclictest2094510-21kworker/0:1+events19:53:250
208232299211811,2103cyclictest2096790-21kworker/0:2+events20:26:110
208233199211710,2104cyclictest2159398-21kworker/2:2+events23:14:472
208232299211711,2103cyclictest2130443-21kworker/0:0+events22:12:240
20823319921169,2103cyclictest2098183-21kworker/2:1+events19:58:432
208232299211611,2102cyclictest2145079-21kworker/0:1+events22:24:540
208232299211611,2102cyclictest2141792-21kworker/0:1+events22:15:260
208232299211411,2100cyclictest2180356-21kworker/0:2+events00:16:500
208232299211411,2100cyclictest2173596-21kworker/0:0+events00:07:010
208232299211311,2099cyclictest2130443-21kworker/0:0+events21:46:000
208233199211210,2099cyclictest2092121-21kworker/2:0+events20:07:572
208233199211110,2098cyclictest2111662-21kworker/2:0+events20:41:412
208232699210911,2095cyclictest2093591-21kworker/1:0+events19:47:431
208233199210810,2095cyclictest2127242-21kworker/2:1+events21:41:262
208232699210810,2095cyclictest2111573-21kworker/1:2+events20:47:021
20823319921079,2095cyclictest2152877-21kworker/2:2+events22:48:562
208232699210711,2093cyclictest2116481-21kworker/1:2+events20:59:361
208232299210710,2095cyclictest2118078-21kworker/0:0+events21:05:410
208232299210510,2092cyclictest2096790-21kworker/0:2+events20:06:480
208233199210411,2089cyclictest2151216-21kworker/2:0+events22:58:252
20823229921048,2092cyclictest2096790-21kworker/0:2+events20:54:140
208232299210411,2090cyclictest2103661-21kworker/0:0+events20:18:330
20823319921039,2090cyclictest2186518-21kworker/2:1+events00:35:322
20823319921029,2090cyclictest2083565-21kworker/2:1+events19:44:202
208233199210210,2089cyclictest2177934-21kworker/2:1+events00:06:142
208232299210211,2088cyclictest2145079-21kworker/0:1+events22:45:340
208232299210211,2088cyclictest2118078-21kworker/0:0+events21:20:350
20823319920998,2088cyclictest2101770-21kworker/2:1+events20:11:442
20823319920987,2088cyclictest2151216-21kworker/2:0+events22:42:312
20823319920985,2090cyclictest2113277-21kworker/2:1+events20:51:352
20823319920976,2088cyclictest2149557-21kworker/2:1+events22:36:212
208233199209710,2084cyclictest2168767-21kworker/2:2+events23:41:552
208232299209710,2083cyclictest2096790-21kworker/0:2+events20:00:190
20823319920969,2084cyclictest2125639-21kworker/2:2+events21:30:302
208233199208910,2075cyclictest2183240-21kworker/2:0+events00:28:382
208233199208211,2068cyclictest2085573-21kworker/2:0+events19:19:012
208232699208010,2067cyclictest2133606-21kworker/1:1+events21:49:031
20823229920776,2068cyclictest2136912-21kworker/0:1+events22:03:390
208232299207211,2058cyclictest2157609-21kworker/0:2+events23:11:460
208233399206910,2056cyclictest2106734-21kworker/3:0+events20:23:293
208232299206911,2055cyclictest2160876-21kworker/0:0+events23:23:580
208233399206610,2053cyclictest2127152-21kworker/3:1+events21:33:053
208233399206510,2052cyclictest2116188-21kworker/3:0+events20:52:213
20823269920617,2051cyclictest2080078-21kworker/1:1+events19:23:111
208233399205610,2042cyclictest2170155-21kworker/3:0+events23:57:523
20823319920499,2037cyclictest2103716-21kworker/2:2+events20:15:302
20823229920457,2035cyclictest2170186-21kworker/0:2+events23:42:170
20823269920449,2032cyclictest2176825-21kworker/1:0+events00:05:511
20823319920416,2032cyclictest2075424-21kworker/2:2+events19:11:262
20823269920417,2031cyclictest2109947-21kworker/1:1+events20:36:511
20823269920399,2027cyclictest2096821-21kworker/1:0+events20:04:551
208232299203911,2025cyclictest2162512-21kworker/0:2+events23:30:240
20823269920382,2032cyclictest2085481-21kworker/1:0+events19:18:151
20823269920327,2022cyclictest2170336-21kworker/1:1+events23:45:451
20823319920309,2018cyclictest2103716-21kworker/2:2+events20:33:492
208232699203010,2016cyclictest2178741-21kworker/1:1+events00:11:361
20823319920278,2016cyclictest2113277-21kworker/2:1+events20:47:232
20823229920256,2016cyclictest2088734-21kworker/0:2+events19:42:240
208233399202310,2009cyclictest2081772-21kworker/3:0+events19:14:063
20823339920197,2009cyclictest2140132-21kworker/3:1+events22:12:013
208233399201410,2001cyclictest2095255-21kworker/3:1+events19:51:543
20823339920129,2000cyclictest2083774-21kworker/3:2+events19:36:463
208233399201110,1998cyclictest2100044-21kworker/3:1+events20:10:593
20823339920098,1998cyclictest2127152-21kworker/3:1+events21:26:193
20823339920083,2001cyclictest2157698-21kworker/3:0+events23:12:103
20823339920074,2000cyclictest2133731-21kworker/3:0+events22:03:163
20823339920048,1993cyclictest2106734-21kworker/3:0+events20:25:263
20823339920043,1998cyclictest2097017-21kworker/3:0+events19:59:073
208233399200210,1989cyclictest2090339-21kworker/3:0+events19:40:553
20823319919848,1973cyclictest2103716-21kworker/2:2+events20:20:482
208232299197010,1957cyclictest2092283-21kworker/0:0+events19:45:250
20823229919649,1952cyclictest2105292-21kworker/0:1+events20:24:160
20823229919628,1951cyclictest2096790-21kworker/0:2+events21:04:310
208232299196210,1949cyclictest2145079-21kworker/0:1+events22:31:450
208232299196110,1948cyclictest2108219-21kworker/0:0+events20:35:420
20823269919608,1949cyclictest2082278-21kworker/1:3+events19:14:281
20823229919607,1950cyclictest2130443-21kworker/0:0+events21:54:510
20823229919587,1950cyclictest2152730-21kworker/0:2+events22:50:040
20823229919587,1948cyclictest2160876-21kworker/0:0+events23:19:240
208232299195310,1940cyclictest2173596-21kworker/0:0+events00:36:400
20823319919508,1939cyclictest2177934-21kworker/2:1+events00:17:142
208232299195010,1937cyclictest2165467-21kworker/0:1+events23:36:310
20823229919488,1937cyclictest2090464-21kworker/0:1+events19:36:240
20823229919475,1939cyclictest2162512-21kworker/0:2+events23:26:360
20823229919457,1935cyclictest2088734-21kworker/0:2+events19:33:250
208233199194410,1931cyclictest2183240-21kworker/2:0+events00:23:412
20823229919434,1936cyclictest2082057-21kworker/0:0+events19:13:440
20823229919409,1928cyclictest2096790-21kworker/0:2+events20:13:390
20823229919376,1928cyclictest2127599-21kworker/0:1+events21:37:150
20823229919364,1928cyclictest2096790-21kworker/0:2+events20:43:120
20823229919339,1921cyclictest2130443-21kworker/0:0+events21:59:020
208232299192610,1913cyclictest2096790-21kworker/0:2+events20:48:550
20823229919257,1915cyclictest2116364-21kworker/0:1+events20:56:100
20823229919199,1907cyclictest2170186-21kworker/0:2+events23:54:260
20823229918825,1874cyclictest2154619-21kworker/0:0+events23:02:130
20823229918819,1869cyclictest2143431-21kworker/0:2+events22:28:190
20823339918619,1849cyclictest2081772-21kworker/3:0+events19:15:583
20823319918005,1792cyclictest2121136-21kworker/2:0+events21:21:202
20818152145102,29sleep00-21swapper/019:08:260
20818282140106,29sleep10-21swapper/119:08:341
20819212139106,28sleep20-21swapper/219:09:472
20816422137105,27sleep30-21swapper/319:06:173
208232299222,14cyclictest2086729-21ntpq19:20:210
208232299202,7cyclictest2183066-21cat00:20:200
208233399190,17cyclictest2164454-21chrt23:25:123
208232299199,7cyclictest2098538-21dpkg20:00:010
208232299194,4cyclictest0-21swapper/000:10:160
208232299186,9cyclictest2110122-21idleruntime-cro20:34:590
2082322991810,6cyclictest2122015-21ntpq21:10:190
208232299178,6cyclictest2151539-21perf22:44:590
208232299174,4cyclictest2185869-21ntpq00:30:190
208232299172,7cyclictest2149123-21ntpq22:35:180
2082322991712,2cyclictest2156264-21munin-run23:00:000
208232299166,8cyclictest2183964-21hddtemp_smartct00:25:160
208232299162,6cyclictest2131538-21ntpq21:40:200
2082331991511,3cyclictest2106989-21kworker/u16:1+events_unbound21:10:202
208232299154,6cyclictest2176073-21ntpq00:00:170
208232299141,11cyclictest2124366-21munin-run21:19:590
208232299140,9cyclictest2126420-21df_inode21:25:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional