You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 20:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #f, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackfslot7s.osadl.org (updated Tue Nov 25, 2025 00:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
127778999332716,3306cyclictest1301845-21kworker/3:2+events20:35:173
127778999296014,2941cyclictest1356028-21kworker/3:1+events23:10:003
127778699284914,2830cyclictest1359445-21kworker/2:2+events23:30:142
127778499281013,2793cyclictest1294338-21kworker/1:0+events20:00:181
127778099272013,2704cyclictest1284158-21kworker/0:1+events19:39:590
127778099268613,2669cyclictest1356865-21kworker/0:0+events23:20:210
127778999266313,2646cyclictest1342729-21kworker/3:0+events22:40:203
127778099249011,2475cyclictest1324533-21kworker/0:1+events21:44:170
127778699248810,2474cyclictest1294490-21kworker/2:2+events20:01:502
127778099248411,2470cyclictest1287619-21kworker/0:2+events19:48:030
127778099247910,2466cyclictest1292652-21kworker/0:1+events19:59:550
127778099243212,2417cyclictest1364187-21kworker/0:2+events00:00:170
12777809924294,2421cyclictest1324533-21kworker/0:1+events21:30:130
127778699240210,2389cyclictest1330652-21kworker/2:2+events21:50:182
127778699239810,2383cyclictest1282828-21kworker/2:1+events20:05:122
127778699239411,2379cyclictest1365759-21kworker/2:2+events23:52:202
12777849923729,2360cyclictest1296057-21kworker/1:1+events20:09:421
127778699235910,2345cyclictest1341742-21kworker/2:1+events23:04:182
127778999233011,2316cyclictest1287815-21kworker/3:1+events19:46:313
12777899923288,2317cyclictest1292536-21kworker/3:2+events20:03:413
12777899923279,2315cyclictest1304117-21kworker/3:1+events20:30:453
127778999232614,2309cyclictest1329020-21kworker/3:2+events22:12:343
127778699231512,2300cyclictest1359445-21kworker/2:2+events23:25:172
127778999231412,2299cyclictest1346372-21kworker/3:2+events22:50:133
127778699230110,2287cyclictest1306039-21kworker/2:1+events20:44:522
127778999229510,2282cyclictest1312706-21kworker/3:2+events21:12:033
127778099229511,2281cyclictest1349478-21kworker/0:1+events22:52:100
127778699228910,2276cyclictest1332435-21kworker/2:1+events22:09:512
127778699228111,2267cyclictest1357805-21kworker/2:1+events23:22:142
127778699227410,2261cyclictest1376706-21kworker/2:0+events00:13:082
127778099227111,2257cyclictest1319779-21kworker/0:0+events21:17:440
127778699226911,2255cyclictest1306039-21kworker/2:1+events20:51:402
127778699226910,2255cyclictest1372282-21kworker/2:3+events00:05:562
127778699226411,2250cyclictest1341742-21kworker/2:1+events22:31:552
127778099226210,2249cyclictest1316045-21kworker/0:0+events21:06:360
127778099226010,2247cyclictest1267489-21kworker/0:1+events19:19:290
12777809922599,2246cyclictest1294674-21kworker/0:0+events20:07:050
127778099225910,2245cyclictest1346401-21kworker/0:2+events22:39:100
127778699225710,2244cyclictest1294490-21kworker/2:2+events19:58:202
127778699225411,2240cyclictest1301276-21kworker/2:0+events20:20:512
127778099225311,2240cyclictest1352891-21kworker/0:2+events23:05:020
127778699225210,2239cyclictest1306039-21kworker/2:1+events21:02:002
127778699225010,2237cyclictest1328930-21kworker/2:1+events21:41:582
127778699225010,2237cyclictest1326873-21kworker/2:0+events21:59:282
127778699224910,2236cyclictest1297774-21kworker/2:2+events20:12:002
127778699224711,2233cyclictest1281122-21kworker/2:0+events19:31:202
127778699223711,2223cyclictest1299639-21kworker/2:1+events20:18:312
127778699223711,2223cyclictest1299639-21kworker/2:1+events20:18:312
127778099223511,2220cyclictest1292652-21kworker/0:1+events19:53:450
12777809922319,2219cyclictest1292652-21kworker/0:1+events20:21:130
127778699222011,2205cyclictest1322865-21kworker/2:1+events21:34:012
127778099221910,2206cyclictest1309681-21kworker/0:0+events20:55:290
127778699218212,2167cyclictest1281122-21kworker/2:0+events19:24:542
127778699217910,2166cyclictest1355874-21kworker/2:0+events23:15:472
12777809921746,2165cyclictest1375374-21kworker/0:0+events00:36:280
12777809921667,2155cyclictest1302575-21kworker/0:2+events20:28:510
12777809921664,2159cyclictest1356865-21kworker/0:0+events23:16:110
12777809921658,2154cyclictest1364187-21kworker/0:2+events23:39:030
127778999216111,2147cyclictest1373714-21kworker/3:1+events00:18:303
12777869921610,2158cyclictest1345009-21kworker/2:2+events22:36:082
12777869921595,2150cyclictest1336847-21kworker/2:0+events22:29:402
12777869921595,2150cyclictest1336847-21kworker/2:0+events22:29:402
127778999214211,2127cyclictest1354392-21kworker/3:0+events23:14:383
127778999214011,2126cyclictest1312706-21kworker/3:2+events21:04:423
127778999213911,2124cyclictest1352712-21kworker/3:1+events23:01:363
127778999213910,2126cyclictest1360827-21kworker/3:1+events23:41:443
127778999213811,2124cyclictest1297652-21kworker/3:1+events20:12:243
12777899921379,2125cyclictest1292536-21kworker/3:2+events20:09:213
12777899921322,2127cyclictest1309441-21kworker/3:2+events20:42:113
12777899921319,2119cyclictest1327397-21kworker/3:1+events21:46:093
127778999213111,2117cyclictest1348116-21kworker/3:1+events22:48:403
127778099213010,2116cyclictest1359229-21kworker/0:1+events23:31:460
127778999212911,2115cyclictest1358442-21kworker/3:2+events23:30:373
127778999212911,2115cyclictest1277434-21kworker/3:0+events19:16:273
127778999212811,2114cyclictest1373714-21kworker/3:1+events00:24:403
127778999212711,2113cyclictest1281170-21kworker/3:2+events19:39:373
127778999212710,2114cyclictest1325740-21kworker/3:0+events21:41:353
127778999212610,2113cyclictest1281170-21kworker/3:2+events19:26:243
127778999212511,2111cyclictest1321380-21kworker/3:2+events21:22:183
127778699212310,2109cyclictest1361391-21kworker/2:0+events23:43:372
127778999212210,2109cyclictest1277434-21kworker/3:0+events19:21:253
12777809921229,2110cyclictest1349478-21kworker/0:1+events22:46:500
127778699212110,2108cyclictest1314597-21kworker/2:0+events21:18:072
127778699212010,2107cyclictest1378307-21kworker/2:2+events00:37:392
127778699211910,2106cyclictest1306039-21kworker/2:1+events20:47:092
127778699211710,2104cyclictest1354243-21kworker/2:2+events23:06:572
127778099211710,2104cyclictest1282497-21kworker/0:0+events19:32:270
127778699211510,2102cyclictest1382975-21kworker/2:0+events00:33:002
127778699211510,2102cyclictest1382975-21kworker/2:0+events00:32:592
127778499211511,2101cyclictest1324196-21kworker/1:2+events21:42:441
12777809921159,2103cyclictest1309681-21kworker/0:0+events20:47:530
12777869921139,2101cyclictest1282828-21kworker/2:1+events19:28:192
127778499211310,2100cyclictest1306010-21kworker/1:0+events20:45:371
127778699211210,2099cyclictest1326873-21kworker/2:0+events21:36:152
127778099211210,2099cyclictest1332252-21kworker/0:1+events21:54:080
127778099211210,2099cyclictest1292652-21kworker/0:1+events20:11:380
127778499211011,2096cyclictest1376766-21kworker/1:0+events00:25:491
12777869921098,2098cyclictest1349655-21kworker/2:0+events22:55:352
127778499210911,2095cyclictest1346434-21kworker/1:2+events22:49:031
127778099210910,2096cyclictest1294674-21kworker/0:0+events20:01:040
127778099210811,2094cyclictest1365790-21kworker/0:1+events23:49:180
127778699210710,2093cyclictest1380212-21kworker/2:1+events00:29:342
12777809921073,2101cyclictest1356865-21kworker/0:0+events23:11:100
127778999210610,2092cyclictest1354392-21kworker/3:0+events23:18:063
127778699210610,2092cyclictest1370285-21kworker/2:0+events23:58:442
127778699210610,2092cyclictest1341742-21kworker/2:1+events22:48:182
12777809921065,2098cyclictest1338421-21kworker/0:1+events22:13:190
12777869921056,2096cyclictest1281122-21kworker/2:0+events19:36:592
12777809921059,2093cyclictest1279247-21kworker/0:2+events19:22:590
127778699210410,2091cyclictest1332435-21kworker/2:1+events22:00:372
12777809921047,2093cyclictest1376974-21kworker/0:2+events00:15:490
127778099210410,2091cyclictest1322751-21kworker/0:2+events21:23:480
12777809921039,2091cyclictest1302575-21kworker/0:2+events20:31:080
12777809921037,2093cyclictest1302575-21kworker/0:2+events20:44:300
127778099210310,2090cyclictest1324533-21kworker/0:1+events21:39:190
127778099210310,2090cyclictest1284158-21kworker/0:1+events19:28:420
127778499210211,2088cyclictest1359324-21kworker/1:2+events23:18:291
127778099210210,2089cyclictest1335812-21kworker/0:0+events22:16:250
127778099210210,2089cyclictest1302575-21kworker/0:2+events20:38:200
127778099210010,2087cyclictest1381641-21kworker/0:2+events00:31:300
127778099210010,2087cyclictest1381641-21kworker/0:2+events00:31:300
127778699209910,2086cyclictest1281122-21kworker/2:0+events19:19:522
12777809920988,2087cyclictest1335812-21kworker/0:0+events22:40:440
12777809920983,2092cyclictest1335812-21kworker/0:0+events22:25:510
12777809920983,2092cyclictest1335812-21kworker/0:0+events22:25:510
12777849920978,2086cyclictest1368862-21kworker/1:0+events23:51:121
12777869920968,2085cyclictest1376706-21kworker/2:0+events00:18:072
127778099209610,2083cyclictest1312313-21kworker/0:1+events20:53:580
12777809920957,2085cyclictest1379819-21kworker/0:1+events00:21:580
127778699209411,2080cyclictest1306039-21kworker/2:1+events20:58:312
12777809920939,2080cyclictest1319779-21kworker/0:0+events21:27:120
12777809920919,2078cyclictest1335812-21kworker/0:0+events22:06:020
12777869920906,2080cyclictest1288383-21kworker/2:2+events19:41:082
12777899920858,2074cyclictest1365632-21kworker/3:2+events23:56:523
127778999208510,2071cyclictest1297652-21kworker/3:1+events20:18:103
127778999208510,2071cyclictest1297652-21kworker/3:1+events20:18:093
127778499208510,2072cyclictest1378464-21kworker/1:1+events00:23:531
127778099208510,2073cyclictest1364187-21kworker/0:2+events23:55:010
12777809920847,2074cyclictest1332252-21kworker/0:1+events21:55:370
12777809920839,2071cyclictest1351141-21kworker/0:0+events22:59:220
127778999207910,2066cyclictest1312706-21kworker/3:2+events20:59:183
127778099207810,2065cyclictest1344228-21kworker/0:1+events22:30:500
12777869920775,2069cyclictest1341742-21kworker/2:1+events22:51:232
127778999207610,2063cyclictest1316168-21kworker/3:1+events21:17:213
12777809920752,2070cyclictest1372494-21kworker/0:1+events00:14:160
127778099207510,2062cyclictest1372494-21kworker/0:1+events00:09:430
12777809920736,2064cyclictest1332252-21kworker/0:1+events22:02:310
127778699207210,2059cyclictest1363973-21kworker/2:1+events23:35:332
127778099207110,2059cyclictest1379819-21kworker/0:1+events00:25:040
127778099207010,2057cyclictest1362517-21kworker/0:0+events23:29:260
12777849920688,2057cyclictest1361091-21kworker/1:1+events23:33:181
127778499206611,2052cyclictest1289745-21kworker/1:2+events19:55:381
12777869920658,2054cyclictest1316213-21kworker/2:2+events21:05:512
12777869920596,2050cyclictest1297774-21kworker/2:2+events20:25:032
12777849920599,2047cyclictest1381730-21kworker/1:2+events00:36:521
127778999205810,2044cyclictest1312706-21kworker/3:2+events21:09:203
12777849920568,2045cyclictest1301066-21kworker/1:2+events20:31:301
12777849920554,2048cyclictest1351321-21kworker/1:0+events22:56:431
12777899920542,2049cyclictest1287815-21kworker/3:1+events19:44:593
12777849920549,2042cyclictest1338636-21kworker/1:1+events22:33:501
127778499205310,2040cyclictest1367234-21kworker/1:2+events23:42:531
12777869920518,2039cyclictest1336847-21kworker/2:0+events22:17:102
127778999205011,2036cyclictest1333888-21kworker/3:0+events22:06:483
127778499205011,2036cyclictest1353076-21kworker/1:1+events23:08:281
127778099205010,2037cyclictest1330531-21kworker/0:2+events21:46:550
12777809920498,2038cyclictest1341712-21kworker/0:2+events22:24:190
127778499204810,2035cyclictest1297835-21kworker/1:0+events20:17:241
127778499204810,2035cyclictest1297835-21kworker/1:0+events20:17:241
12777869920468,2035cyclictest1376706-21kworker/2:0+events00:21:122
127778499204610,2034cyclictest1338636-21kworker/1:1+events22:25:051
127778499204610,2034cyclictest1338636-21kworker/1:1+events22:25:051
12777869920459,2033cyclictest1322865-21kworker/2:1+events21:25:412
12777809920449,2033cyclictest1364187-21kworker/0:2+events23:50:270
127778499204311,2029cyclictest1321277-21kworker/1:1+events21:22:421
12777849920426,2035cyclictest1324196-21kworker/1:2+events21:45:021
12777809920424,2035cyclictest1312313-21kworker/0:1+events21:01:370
12777869920418,2030cyclictest1370285-21kworker/2:0+events00:02:322
12777869920418,2030cyclictest1277420-21kworker/2:2+events19:13:262
12777869920413,2035cyclictest1336847-21kworker/2:0+events22:23:122
127778699204110,2028cyclictest1306039-21kworker/2:1+events20:39:062
127778499204010,2027cyclictest1368862-21kworker/1:0+events00:01:481
12777869920399,2027cyclictest1306039-21kworker/2:1+events20:32:592
12777809920376,2028cyclictest1312313-21kworker/0:1+events21:11:400
12777869920329,2020cyclictest1314597-21kworker/2:0+events21:14:402
12777809920329,2020cyclictest1300944-21kworker/0:0+events20:17:470
12777809920329,2020cyclictest1300944-21kworker/0:0+events20:17:470
12777869920319,2019cyclictest1368687-21kworker/2:1+events23:49:432
12777809920318,2020cyclictest1365790-21kworker/0:1+events23:44:010
127778499202710,2013cyclictest1373547-21kworker/1:3+events00:16:111
127778999201410,2001cyclictest1333888-21kworker/3:0+events21:59:513
12777809920149,2002cyclictest1352891-21kworker/0:2+events23:01:580
127778999201110,1998cyclictest1329020-21kworker/3:2+events22:22:283
12777809920099,1997cyclictest1267489-21kworker/0:1+events19:14:560
12777849920056,1996cyclictest1306010-21kworker/1:0+events20:42:561
12777899920049,1992cyclictest1380132-21kworker/3:0+events00:27:413
12777869920037,1992cyclictest1355874-21kworker/2:0+events23:13:052
12777899919979,1986cyclictest1381788-21kworker/3:2+events00:35:203
127778499199411,1980cyclictest1353076-21kworker/1:1+events23:02:221
12777899919907,1980cyclictest1365632-21kworker/3:2+events23:52:443
127778999198510,1972cyclictest1311100-21kworker/3:1+events20:48:163
12777899919846,1975cyclictest1299670-21kworker/3:0+events20:20:273
127778999198410,1970cyclictest1329020-21kworker/3:2+events22:02:103
127778999198310,1970cyclictest1307551-21kworker/3:0+events20:52:503
12777899919825,1974cyclictest1373714-21kworker/3:1+events00:05:133
127778999198210,1969cyclictest1373714-21kworker/3:1+events00:10:543
12777899919809,1969cyclictest1360827-21kworker/3:1+events23:35:113
12777899919803,1974cyclictest1342729-21kworker/3:0+events22:31:113
12777899919799,1967cyclictest1373714-21kworker/3:1+events00:34:103
12777899919799,1967cyclictest1373714-21kworker/3:1+events00:34:103
12777899919795,1971cyclictest1292536-21kworker/3:2+events19:59:073
12777899919778,1966cyclictest1301845-21kworker/3:2+events20:26:583
12777869919777,1967cyclictest1345009-21kworker/2:2+events22:44:072
12777869919766,1967cyclictest1314597-21kworker/2:0+events21:20:472
12777899919739,1962cyclictest1365632-21kworker/3:2+events23:45:103
12777899919735,1965cyclictest1333888-21kworker/3:0+events22:16:473
12777899919724,1965cyclictest1325740-21kworker/3:0+events21:31:423
12777899919719,1959cyclictest1360827-21kworker/3:1+events23:29:513
127778999197110,1959cyclictest1360827-21kworker/3:1+events23:22:373
127778999197110,1958cyclictest1346372-21kworker/3:2+events22:58:383
12777899919707,1960cyclictest1323790-21kworker/3:1+events21:29:043
127778999197010,1957cyclictest1342729-21kworker/3:0+events22:39:343
12777899919687,1958cyclictest1372064-21kworker/3:0+events00:00:383
12777899919685,1961cyclictest1321380-21kworker/3:2+events21:35:073
12777849919685,1960cyclictest1281017-21kworker/1:2+events19:25:161
127778999196710,1954cyclictest1330619-21kworker/3:0+events21:50:423
12777849919669,1954cyclictest1338636-21kworker/1:1+events22:15:131
12777899919644,1957cyclictest1279160-21kworker/3:1+events19:14:343
12777849919649,1952cyclictest1330712-21kworker/1:1+events22:02:551
12777849919635,1954cyclictest1302923-21kworker/1:1+events20:26:341
127778499196210,1949cyclictest1282940-21kworker/1:1+events19:42:401
12777849919619,1949cyclictest1344770-21kworker/1:0+events22:39:571
12777849919607,1950cyclictest1346434-21kworker/1:2+events22:41:061
12777849919603,1954cyclictest1312942-21kworker/1:0+events20:58:541
127778699195911,1945cyclictest1288383-21kworker/2:2+events19:49:112
12777849919595,1951cyclictest1337031-21kworker/1:0+events22:10:141
12777849919587,1948cyclictest1281017-21kworker/1:2+events19:32:491
12777849919584,1951cyclictest1335389-21kworker/1:2+events22:08:421
12777849919576,1948cyclictest1324196-21kworker/1:2+events21:36:391
127778499195510,1941cyclictest1306010-21kworker/1:0+events20:36:481
12777849919539,1941cyclictest1324196-21kworker/1:2+events21:51:031
127778499195310,1940cyclictest1274404-21kworker/1:0+events19:17:121
12777849919519,1939cyclictest1297835-21kworker/1:0+events20:10:511
12777869919503,1944cyclictest1292508-21kworker/2:0+events19:54:312
12777849919486,1939cyclictest1309561-21kworker/1:2+events21:16:111
12777849919479,1935cyclictest1373547-21kworker/1:3+events00:12:251
12777849919479,1935cyclictest1316135-21kworker/1:1+events21:06:151
12777849919476,1938cyclictest1309561-21kworker/1:2+events21:13:121
12777849919474,1940cyclictest1330712-21kworker/1:1+events21:56:231
12777849919468,1935cyclictest1361091-21kworker/1:1+events23:39:481
127778999194210,1929cyclictest1329020-21kworker/3:2+events22:26:583
127778999194210,1929cyclictest1329020-21kworker/3:2+events22:26:583
127778499194210,1929cyclictest1353076-21kworker/1:1+events23:12:421
127778499194110,1928cyclictest1359324-21kworker/1:2+events23:26:501
12777869919394,1932cyclictest1326873-21kworker/2:0+events21:48:022
12777849919228,1911cyclictest1375135-21kworker/1:1+events00:09:231
127778499192210,1909cyclictest1381730-21kworker/1:2+events00:31:541
127778499192210,1909cyclictest1381730-21kworker/1:2+events00:31:531
12777849919188,1907cyclictest1297835-21kworker/1:0+events20:21:581
127778499191710,1904cyclictest1319582-21kworker/1:0+events21:29:271
12777899919168,1905cyclictest1281170-21kworker/3:2+events19:34:423
127778999191610,1903cyclictest1292536-21kworker/3:2+events19:51:253
12777849919129,1900cyclictest1349715-21kworker/1:1+events22:53:411
12777849919029,1890cyclictest1371912-21kworker/1:2+events23:59:291
12777849918979,1885cyclictest1311195-21kworker/1:1+events20:50:521
12777849918979,1885cyclictest1289745-21kworker/1:2+events19:50:211
12777849918949,1883cyclictest1357625-21kworker/1:0+events23:21:521
12777849918916,1882cyclictest1282940-21kworker/1:1+events19:39:141
12777849918827,1872cyclictest1338636-21kworker/1:1+events22:22:051
12777849918827,1872cyclictest1309561-21kworker/1:2+events21:02:471
12777869918809,1868cyclictest1332435-21kworker/2:1+events22:10:372
12777849918807,1870cyclictest1282940-21kworker/1:1+events19:47:401
12777849918489,1836cyclictest1276028-21kworker/1:1+events19:10:231
12777849918159,1803cyclictest1361091-21kworker/1:1+events23:47:461
12775682141111,25sleep00-21swapper/019:09:290
12774382140109,26sleep10-21swapper/119:07:501
12773862139109,25sleep20-21swapper/219:07:152
12762832139105,27sleep30-21swapper/319:05:123
1277780991810,7cyclictest0-21swapper/019:40:170
127778499170,6cyclictest1326084-21munin-run21:35:001
127778499150,4cyclictest1282857-21cron19:25:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional