You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 08:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #f, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackfslot7s.osadl.org (updated Tue Jul 01, 2025 00:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
99690499280513,2788cyclictest1022672-21kworker/0:0+events20:35:140
99690499274911,2733cyclictest1096686-21kworker/0:2+events00:30:130
99691299271213,2695cyclictest1036745-21kworker/3:2+events21:35:183
99690499265313,2636cyclictest1038494-21kworker/0:1+events21:30:190
99690999263312,2617cyclictest1073580-21kworker/2:2+events23:10:142
9969049925860,2582cyclictest1062396-21kworker/0:1+events22:40:230
99690499254512,2529cyclictest1007160-21kworker/0:1+events19:52:240
99690999251511,2500cyclictest1013031-21kworker/2:2+events19:59:202
99690499250012,2485cyclictest1017819-21kworker/0:0+events20:19:190
99690499249812,2483cyclictest1038494-21kworker/0:1+events21:17:240
99690499249412,2478cyclictest1019432-21kworker/0:1+events20:24:410
99690499248912,2474cyclictest1001806-21kworker/0:2+events19:32:140
99690499248911,2474cyclictest1021073-21kworker/0:2+events20:27:370
99690499248712,2471cyclictest1038494-21kworker/0:1+events21:26:560
99690499248711,2472cyclictest1028750-21kworker/0:0+events20:54:390
99690499248612,2471cyclictest1031959-21kworker/0:2+events20:58:280
99690499248612,2470cyclictest1011343-21kworker/0:2+events20:11:010
99690499248612,2470cyclictest1011343-21kworker/0:2+events19:57:020
99690499248611,2472cyclictest1028750-21kworker/0:0+events20:49:200
99690499248611,2472cyclictest1028750-21kworker/0:0+events20:49:200
99690499248412,2469cyclictest1083404-21kworker/0:0+events23:40:310
99690499248311,2470cyclictest1038494-21kworker/0:1+events21:20:040
99690499248212,2466cyclictest1030352-21kworker/0:1+events21:01:060
99690499248112,2466cyclictest1012969-21kworker/0:0+events20:01:570
99690499247812,2463cyclictest1009764-21kworker/0:0+events19:48:380
9969129924725,2463cyclictest1017697-21kworker/3:0+events20:17:023
99690999246812,2454cyclictest1014683-21kworker/2:1+events20:20:042
99690499244611,2432cyclictest1012969-21kworker/0:0+events20:05:200
99690999242810,2414cyclictest1021045-21kworker/2:0+events20:33:462
99691299242312,2408cyclictest1022702-21kworker/3:1+events20:32:133
99690999241312,2397cyclictest996408-21kworker/2:1+events19:30:192
99690999240912,2395cyclictest996408-21kworker/2:1+events19:20:272
99690999240910,2396cyclictest1057590-21kworker/2:2+events22:42:362
99691299240712,2391cyclictest1036745-21kworker/3:2+events21:18:093
9969129924019,2388cyclictest1085033-21kworker/3:1+events23:43:153
99690999239712,2382cyclictest1027359-21kworker/2:1+events20:54:162
99690999239712,2381cyclictest1038556-21kworker/2:2+events21:19:412
99691299239612,2381cyclictest1001578-21kworker/3:1+events19:43:413
99690999239312,2377cyclictest1068781-21kworker/2:0+events23:06:242
99691299239110,2378cyclictest1021015-21kworker/3:2+events20:24:173
99690999239112,2375cyclictest1019393-21kworker/2:2+events20:18:552
99691299239012,2375cyclictest1078586-21kworker/3:0+events23:35:543
99691299239011,2375cyclictest1025555-21kworker/3:0+events20:47:033
99691299239011,2375cyclictest1025555-21kworker/3:0+events20:47:033
99691299238912,2373cyclictest1100338-21kworker/3:1+events00:34:273
99691299238912,2373cyclictest1001578-21kworker/3:1+events19:21:573
99691299238911,2376cyclictest1017697-21kworker/3:0+events20:25:033
99690999238912,2373cyclictest1032168-21kworker/2:2+events21:06:422
9969129923889,2375cyclictest1025555-21kworker/3:0+events20:39:013
99691299238812,2372cyclictest1088246-21kworker/3:2+events23:55:523
99691299238812,2372cyclictest1064000-21kworker/3:0+events22:48:143
99690999238712,2372cyclictest1014683-21kworker/2:1+events20:04:132
99690999238612,2371cyclictest1052721-21kworker/2:0+events22:26:162
99690999238511,2371cyclictest1027359-21kworker/2:1+events20:45:552
99690999238511,2371cyclictest1027359-21kworker/2:1+events20:45:552
99690999238211,2368cyclictest1030564-21kworker/2:0+events20:57:212
99690999236712,2352cyclictest1081764-21kworker/2:1+events23:42:062
99690499236611,2352cyclictest1083404-21kworker/0:0+events23:45:120
99691299236212,2347cyclictest1001578-21kworker/3:1+events19:47:553
9969049923619,2348cyclictest1062396-21kworker/0:1+events22:39:140
99690499235211,2337cyclictest998152-21kworker/0:2+events19:17:480
9969099923469,2333cyclictest1052721-21kworker/2:0+events22:17:312
99690699233012,2315cyclictest1004968-21kworker/1:1+events19:39:501
99691299232312,2307cyclictest1036745-21kworker/3:2+events21:40:183
99691299232212,2307cyclictest1049557-21kworker/3:0+events21:58:093
99690699232112,2306cyclictest1008573-21kworker/1:0+events20:13:371
99690999230910,2295cyclictest1089364-21kworker/2:1+events00:03:532
99690699230812,2293cyclictest990169-21kworker/1:0+events19:22:191
9969129922899,2277cyclictest1011636-21kworker/3:1+events20:00:483
9969069922889,2276cyclictest1056045-21kworker/1:1+events22:14:051
9969099922879,2274cyclictest1013031-21kworker/2:2+events20:08:192
99691299228210,2268cyclictest1043451-21kworker/3:0+events21:34:323
99691299227910,2265cyclictest1054627-21kworker/3:0+events22:13:423
99691299227510,2263cyclictest1006450-21kworker/3:2+events19:55:063
99690499227212,2257cyclictest1046564-21kworker/0:1+events21:43:420
99690499226810,2254cyclictest1043326-21kworker/0:2+events21:39:320
9969049922669,2253cyclictest1062396-21kworker/0:1+events22:52:000
99690499226611,2252cyclictest1072308-21kworker/0:2+events23:06:470
99690499226610,2253cyclictest1048166-21kworker/0:0+events21:51:420
99690499226610,2253cyclictest1043326-21kworker/0:2+events21:59:410
99690499226610,2253cyclictest1001806-21kworker/0:2+events19:36:010
9969049922659,2252cyclictest1043326-21kworker/0:2+events22:09:380
99690499226511,2251cyclictest1093193-21kworker/0:1+events00:24:080
99690499226510,2251cyclictest1054683-21kworker/0:0+events22:19:260
99690499226411,2250cyclictest1022672-21kworker/0:0+events20:30:160
99690999226210,2248cyclictest1068781-21kworker/2:0+events22:57:432
99690499226211,2248cyclictest1068871-21kworker/0:0+events22:57:190
99690499226210,2249cyclictest1027141-21kworker/0:2+events20:41:420
99690499226111,2247cyclictest1068871-21kworker/0:0+events23:01:320
99690499226111,2246cyclictest1072308-21kworker/0:2+events23:12:320
99690499226110,2247cyclictest1043326-21kworker/0:2+events21:48:170
99690499226011,2246cyclictest1076466-21kworker/0:1+events23:17:240
99690499226011,2245cyclictest1064060-21kworker/0:2+events22:48:360
99690499225910,2245cyclictest1088365-21kworker/0:0+events00:05:450
99690499225810,2245cyclictest1096686-21kworker/0:2+events00:26:270
99690499225810,2245cyclictest1089963-21kworker/0:2+events00:13:520
99690499225710,2244cyclictest1096686-21kworker/0:2+events00:37:540
99690499225311,2239cyclictest1031959-21kworker/0:2+events21:10:550
99690499224611,2232cyclictest1059128-21kworker/0:1+events22:24:470
99690999223112,2215cyclictest1036610-21kworker/2:0+events21:31:522
99691299222611,2212cyclictest1028813-21kworker/3:1+events21:12:243
9969099922269,2214cyclictest1091263-21kworker/2:2+events00:11:572
99690499222410,2211cyclictest1060708-21kworker/0:0+events22:27:260
99690699222210,2209cyclictest1086640-21kworker/1:2+events00:07:421
99690499222011,2206cyclictest1095035-21kworker/0:0+events00:17:190
99690499222010,2206cyclictest1001806-21kworker/0:2+events19:25:240
99690999221510,2202cyclictest1025732-21kworker/2:2+events20:42:282
99690699221111,2196cyclictest1035206-21kworker/1:0+events21:15:281
99691299219811,2184cyclictest1004847-21kworker/3:0+events19:34:323
99691299219810,2185cyclictest996471-21kworker/3:2+events19:26:563
99690999219710,2184cyclictest1068781-21kworker/2:0+events23:04:542
99691299219611,2181cyclictest999954-21kworker/3:0+events19:18:333
99691299219611,2181cyclictest1091589-21kworker/3:2+events00:12:213
99691299219611,2181cyclictest1033461-21kworker/3:0+events21:02:123
99690999219511,2181cyclictest1061782-21kworker/2:1+events22:34:382
9969129921899,2176cyclictest1006450-21kworker/3:2+events19:53:113
99691299218911,2175cyclictest1064000-21kworker/3:0+events22:44:293
99690999218911,2174cyclictest1003358-21kworker/2:2+events19:29:342
99690999218910,2176cyclictest1100369-21kworker/2:1+events00:39:252
9969049921898,2178cyclictest1089963-21kworker/0:2+events00:03:290
9969099921889,2175cyclictest1036610-21kworker/2:0+events21:12:452
9969099921878,2175cyclictest1046594-21kworker/2:2+events21:57:022
99690999218710,2174cyclictest1075179-21kworker/2:1+events23:24:422
9969129921868,2174cyclictest1056261-21kworker/3:1+events22:22:293
99691299218511,2170cyclictest1021015-21kworker/3:2+events20:44:443
99690999218511,2171cyclictest1052721-21kworker/2:0+events22:13:202
99691299218411,2169cyclictest1060640-21kworker/3:2+events22:32:003
9969099921848,2172cyclictest1093561-21kworker/2:1+events00:25:412
9969129921839,2171cyclictest1073313-21kworker/3:1+events23:09:513
99691299218210,2169cyclictest1078586-21kworker/3:0+events23:27:483
9969099921829,2170cyclictest1057590-21kworker/2:2+events22:46:462
9969099921829,2170cyclictest1046594-21kworker/2:2+events22:05:252
99690999218210,2169cyclictest1081764-21kworker/2:1+events23:33:372
99691299218111,2166cyclictest1033461-21kworker/3:0+events21:08:153
99691299218110,2168cyclictest1065677-21kworker/3:2+events22:50:073
9969099921818,2170cyclictest1083371-21kworker/2:0+events23:39:452
99690999218110,2167cyclictest1041786-21kworker/2:1+events21:29:112
99691299218011,2166cyclictest1092223-21kworker/3:1+events00:19:583
99690999217911,2164cyclictest1095227-21kworker/2:0+events00:22:392
99691299217711,2163cyclictest1051306-21kworker/3:1+events22:08:293
99690999217411,2159cyclictest1098452-21kworker/2:0+events00:31:202
99691299217311,2160cyclictest1080226-21kworker/3:1+events23:30:083
9969099921729,2159cyclictest1032168-21kworker/2:2+events21:00:232
99690999217211,2158cyclictest1075179-21kworker/2:1+events23:17:472
9969129921716,2162cyclictest1011636-21kworker/3:1+events20:10:373
9969099921719,2159cyclictest998122-21kworker/2:0+events19:14:472
99690999217111,2156cyclictest1008158-21kworker/2:2+events19:41:462
9969099921699,2157cyclictest1021045-21kworker/2:0+events20:25:262
99690999216911,2154cyclictest1008158-21kworker/2:2+events19:49:462
99691299216811,2154cyclictest1001578-21kworker/3:1+events19:37:553
99691299216811,2153cyclictest1091589-21kworker/3:2+events00:06:093
99690999216711,2152cyclictest996408-21kworker/2:1+events19:37:322
99690999216710,2154cyclictest1085001-21kworker/2:2+events23:53:372
99690999216710,2152cyclictest1076788-21kworker/2:0+events23:28:592
99690999216611,2152cyclictest1061782-21kworker/2:1+events22:36:322
99690999216511,2150cyclictest1017786-21kworker/2:0+events20:14:242
99690999216410,2151cyclictest1021045-21kworker/2:0+events20:38:162
9969049921599,2147cyclictest1051412-21kworker/0:0+events22:04:160
99691299215811,2144cyclictest1078586-21kworker/3:0+events00:04:583
99691299215610,2143cyclictest1025555-21kworker/3:0+events20:51:583
99690999215510,2141cyclictest1044967-21kworker/2:0+events21:42:352
9969049921559,2143cyclictest1053053-21kworker/0:1+events22:14:520
99690499214810,2134cyclictest1078346-21kworker/0:0+events23:29:210
9969129921463,2140cyclictest1074972-21kworker/3:0+events23:15:063
99690999213210,2119cyclictest1036610-21kworker/2:0+events21:21:552
99690699212910,2116cyclictest1071715-21kworker/1:0+events23:15:521
99690699211410,2101cyclictest1041637-21kworker/1:1+events21:49:031
9969049921127,2102cyclictest1086674-21kworker/0:1+events23:50:590
99690499211210,2100cyclictest1088365-21kworker/0:0+events23:56:390
99690499210810,2095cyclictest1078346-21kworker/0:0+events23:23:120
99690499210810,2094cyclictest1033853-21kworker/0:3+events21:07:050
99691299210610,2092cyclictest1036745-21kworker/3:2+events21:28:283
99690699210610,2093cyclictest1068970-21kworker/1:2+events23:00:461
99690699210510,2092cyclictest1022437-21kworker/1:1+events20:39:471
9969049921057,2095cyclictest988418-21kworker/0:1+events19:11:220
9969049921049,2092cyclictest988418-21kworker/0:1+events19:20:500
99690499210410,2090cyclictest1076466-21kworker/0:1+events23:38:340
99691299209310,2080cyclictest1065677-21kworker/3:2+events22:59:383
9969069920909,2078cyclictest1032019-21kworker/1:2+events20:58:521
99690999208910,2076cyclictest1041786-21kworker/2:1+events21:37:582
99690699208811,2074cyclictest1060914-21kworker/1:1+events22:31:361
99690699208711,2073cyclictest1029019-21kworker/1:1+events21:07:521
99690699208610,2073cyclictest1024402-21kworker/1:0+events20:40:341
9969129920839,2071cyclictest1078586-21kworker/3:0+events23:51:223
99690499208210,2069cyclictest1081781-21kworker/0:2+events23:32:290
99690699208110,2068cyclictest1066949-21kworker/1:0+events22:50:311
9969099920808,2068cyclictest1089364-21kworker/2:1+events23:58:092
9969129920789,2066cyclictest1091589-21kworker/3:2+events00:26:053
9969129920788,2067cyclictest1036745-21kworker/3:2+events21:24:173
99691299207311,2058cyclictest1053022-21kworker/3:2+events22:16:463
9969099920714,2064cyclictest1052721-21kworker/2:0+events22:03:302
99690699207011,2056cyclictest1037018-21kworker/1:2+events21:25:471
99690999206910,2056cyclictest1046594-21kworker/2:2+events21:53:362
99691299206811,2054cyclictest1011636-21kworker/3:1+events20:09:503
9969099920679,2055cyclictest1091263-21kworker/2:2+events00:16:552
99691299206610,2053cyclictest1096819-21kworker/3:0+events00:35:583
9969129920658,2054cyclictest1058828-21kworker/3:0+events22:25:553
99690699206310,2050cyclictest998032-21kworker/1:1+events19:15:311
99690699205710,2044cyclictest990169-21kworker/1:0+events19:34:091
9969049920559,2043cyclictest1060708-21kworker/0:0+events22:31:140
9969099920465,2038cyclictest1091949-21kworker/2:0+events00:08:282
99690999204111,2027cyclictest1005298-21kworker/2:0+events19:51:382
9969099920409,2028cyclictest1057590-21kworker/2:2+events22:22:072
9969129920377,2027cyclictest1074972-21kworker/3:0+events23:12:533
9969129920336,2023cyclictest1078586-21kworker/3:0+events23:46:453
99691299203210,2020cyclictest1076664-21kworker/3:2+events23:20:293
9969099920318,2020cyclictest1085001-21kworker/2:2+events23:46:222
99690699203111,2017cyclictest1017669-21kworker/1:2+events20:26:311
9969129920308,2019cyclictest1045201-21kworker/3:1+events21:47:323
99691299203010,2017cyclictest1049557-21kworker/3:0+events22:00:263
9969099920308,2019cyclictest1067039-21kworker/2:1+events22:53:532
99691299202910,2016cyclictest1030710-21kworker/3:2+events20:58:063
99690999202910,2016cyclictest1046594-21kworker/2:2+events21:49:482
9969129920287,2018cyclictest1065677-21kworker/3:2+events23:00:263
9969129920286,2019cyclictest1036745-21kworker/3:2+events21:50:573
99691299202610,2014cyclictest1062306-21kworker/3:1+events22:38:293
9969099920269,2014cyclictest996408-21kworker/2:1+events19:18:112
99690699202610,2012cyclictest996278-21kworker/1:2+events19:10:131
9969049920033,1997cyclictest1007160-21kworker/0:1+events19:44:270
9969129919937,1983cyclictest994813-21kworker/3:1+events19:12:083
99690699198310,1971cyclictest1011602-21kworker/1:1+events20:01:111
9969069919817,1971cyclictest1060914-21kworker/1:1+events22:55:251
9969129919749,1962cyclictest1096819-21kworker/3:0+events00:23:463
9969069919627,1951cyclictest1086640-21kworker/1:2+events23:54:001
9969069919616,1952cyclictest1024402-21kworker/1:0+events20:49:411
9969069919616,1952cyclictest1024402-21kworker/1:0+events20:49:411
9969069919527,1941cyclictest1053108-21kworker/1:2+events22:29:231
99690699195210,1939cyclictest1076448-21kworker/1:1+events23:47:541
9969069919499,1937cyclictest1001667-21kworker/1:2+events19:27:171
9969069919497,1939cyclictest1092039-21kworker/1:1+events00:17:421
99690699194910,1936cyclictest1006597-21kworker/1:2+events19:45:361
99690699194810,1935cyclictest1045057-21kworker/1:0+events22:06:571
99690699194610,1933cyclictest1088306-21kworker/1:0+events00:03:061
99690699192810,1915cyclictest1008573-21kworker/1:0+events19:54:411
9969069919099,1896cyclictest1045057-21kworker/1:0+events21:44:051
9969069918838,1872cyclictest1060914-21kworker/1:1+events22:45:371
9969069918818,1870cyclictest1041637-21kworker/1:1+events21:39:081
9964842156110,26sleep30-21swapper/319:07:583
9966292143111,26sleep20-21swapper/219:09:492
9964912143108,28sleep00-21swapper/019:08:030
9965762142108,29sleep10-21swapper/119:09:081
99690699292,22cyclictest1051495-21awk22:00:001
99690699271,24cyclictest0-21swapper/122:45:001
99690699230,22cyclictest0-21swapper/119:55:171
99690699230,16cyclictest0-21swapper/100:10:091
99690699220,21cyclictest1030415-21seq20:52:131
99690699220,21cyclictest0-21swapper/122:40:001
99690699210,19cyclictest798-21runrttasks23:05:121
99690699205,4cyclictest89133-21systemd-journal00:40:001
99690699203,14cyclictest1080258-21awk23:30:001
99690699201,16cyclictest1083651-21cron23:40:011
99690699201,16cyclictest1049898-21munin-run21:55:001
99690699201,15cyclictest1019564-21munin-run20:20:001
99690699200,7cyclictest1042340-21seq21:30:151
99690699200,17cyclictest1037333-21cat21:15:011
99690699190,18cyclictest1099323-21sh00:30:151
99690699190,17cyclictest1016188-21chrt20:07:231
99690699186,5cyclictest1090019-10run-parts00:00:011
99690699181,16cyclictest1507-21gnome-shell23:24:081
99690699181,13cyclictest1023203-21df20:30:161
996906991810,5cyclictest1007125-21gpg-connect-age19:40:011
99690699180,17cyclictest1057144-21/usr/sbin/munin22:15:221
99690699176,9cyclictest1058642-21ntpq22:20:221
99690699171,14cyclictest1051742-21/usr/sbin/munin22:00:141
996906991711,4cyclictest1033602-21runrttasks21:02:231
99690699171,13cyclictest1097765-21ntpq00:25:171
99690699170,15cyclictest1084475-21ntpq23:40:181
996906991614,1cyclictest1075228-21sh23:15:001
99690699161,12cyclictest1039111-21apt-get21:20:151
99690699161,12cyclictest1021170-21idleruntime-cro20:25:001
99690699160,14cyclictest1080711-21gpg-connect-age23:30:001
99690699150,5cyclictest1096916-21seq00:24:051
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional