You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 14:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot4s.osadl.org (updated Sun Nov 30, 2025 12:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1851505996452,9cyclictest282-21systemd-journal11:20:000
1851505996452,9cyclictest282-21systemd-journal11:19:590
1851505996452,9cyclictest2111957-21apt-get11:30:010
1851505996452,9cyclictest2111957-21apt-get11:30:000
1851507996356,5cyclictest1851682-21kworker/u8:3+flush-8:007:36:261
1851507996256,3cyclictest1918243-21kworker/u8:1+events_unbound09:31:251
1851507996256,3cyclictest1918243-21kworker/u8:1+events_unbound09:31:251
19828592610,3sleep11982858-21apt09:20:001
1851507995750,4cyclictest1958265-21kworker/u8:0+events_unbound10:31:291
1851507995652,3cyclictest1923242-21kworker/u8:3+flush-8:008:26:261
1851507995550,3cyclictest1918243-21kworker/u8:1+flush-8:009:01:251
1851507995549,4cyclictest2018046-21kworker/u8:1+events_unbound10:06:201
1851507995448,4cyclictest1958265-21kworker/u8:0+events_unbound10:21:151
20654032530,2sleep3586-21kworker/3:2+mm_percpu_wq10:41:163
21218072520,4chrt0-21swapper/011:36:310
1851507995245,5cyclictest2096864-21kworker/u8:2+flush-8:012:11:231
1851507995245,5cyclictest1943122-21latency_hist08:40:011
1851507995145,4cyclictest1883384-21kworker/u8:2+flush-8:007:51:251
1851507995144,5cyclictest1958265-21kworker/u8:0+events_unbound10:16:091
1851505995146,3cyclictest1973148-21kworker/u8:2+flush-8:009:26:140
1851505995146,3cyclictest1973148-21kworker/u8:2+flush-8:009:26:130
1851507995043,5cyclictest2116882-21kworker/u8:3+events_unbound11:36:251
1851507995043,5cyclictest2096864-21kworker/u8:2+events_unbound12:01:131
1851507995043,5cyclictest1883384-21kworker/u8:2+flush-8:008:46:211
1851507995040,7cyclictest1998263-21/usr/sbin/munin09:36:211
1851505995042,5cyclictest2072364-21latency_hist10:50:010
1851507994944,4cyclictest1877911-21kworker/u8:1+events_unbound07:56:151
1851507994943,4cyclictest2156481-21kworker/u8:1+flush-8:012:21:191
1851507994943,4cyclictest1799523-21kworker/u8:2+flush-8:007:16:221
1851507994943,3cyclictest2111650-21kworker/u8:0+events_unbound11:46:101
1851507994942,4cyclictest282-21systemd-journal10:46:071
1851505994943,4cyclictest1927969-21apt-get08:25:010
1851507994843,4cyclictest2096864-21kworker/u8:2+flush-8:012:26:261
1851507994842,5cyclictest2018046-21kworker/u8:1+flush-8:010:11:221
1851507994743,3cyclictest2052795-21kworker/u8:1+flush-8:010:36:251
1851507994743,3cyclictest1958265-21kworker/u8:0+events_unbound10:51:111
1851507994741,5cyclictest1832130-21kworker/u8:0+events_unbound08:06:201
1851507994741,4cyclictest1883384-21kworker/u8:2+flush-8:007:41:291
18788652460,2chrt0-21swapper/007:36:130
1851507994642,3cyclictest2052795-21kworker/u8:1+events_unbound11:06:101
1851507994640,4cyclictest2052795-21kworker/u8:1+flush-8:011:11:181
1851507994638,6cyclictest1972034-21ntpq09:06:191
1851507994540,3cyclictest1851682-21kworker/u8:3+events_unbound07:26:131
1851507994539,4cyclictest1851682-21kworker/u8:3+events_unbound07:31:211
1851507994438,4cyclictest2081734-21kworker/u8:3+flush-8:011:01:271
1851507994437,5cyclictest2047299-21kworker/u8:3+events_unbound10:26:101
1851505994439,4cyclictest1918243-21kworker/u8:1+flush-8:009:06:230
1851507994339,3cyclictest1958265-21kworker/u8:0+flush-8:010:01:271
1851507994338,4cyclictest2052795-21kworker/u8:1+events_unbound10:41:091
1851507994338,3cyclictest1918243-21kworker/u8:1+flush-8:008:51:221
1851507994337,4cyclictest1958265-21kworker/u8:0+events_unbound11:16:191
1851507994337,4cyclictest1958265-21kworker/u8:0+events_unbound11:16:191
1851505994338,3cyclictest2018046-21kworker/u8:1+flush-8:010:06:280
1851507994237,4cyclictest1883384-21kworker/u8:2+flush-8:007:46:081
1851507994236,4cyclictest1973148-21kworker/u8:2+flush-8:010:56:121
1851507994235,5cyclictest1883384-21kworker/u8:2+events_unbound08:11:111
1851507994234,5cyclictest2136978-21latency_hist11:55:011
1851507994138,2cyclictest1973148-21kworker/u8:2+flush-8:009:26:121
1851507994138,2cyclictest1973148-21kworker/u8:2+flush-8:009:26:121
1851507994136,4cyclictest1958265-21kworker/u8:0+flush-8:009:56:231
1851507994134,5cyclictest1918243-21kworker/u8:1+events_unbound08:21:231
1851507994131,8cyclictest2150957-21ntpq12:06:201
1851505994136,4cyclictest1799523-21kworker/u8:2+events_unbound07:10:010
1851505994133,5cyclictest2002911-21latency_hist09:40:000
1851505994035,4cyclictest1851682-21kworker/u8:3+events_unbound07:31:140
19671542390,2sleep0201rcuc/009:01:210
1851510993924,13cyclictest1983232-21aten2_r5power_c09:21:072
1851510993924,13cyclictest1983232-21aten2_r5power_c09:21:062
1851510993920,17cyclictest1952135-21needreboot08:46:192
1851507993933,4cyclictest1832130-21kworker/u8:0+flush-8:008:16:171
1851507993931,6cyclictest2176540-21latency_hist12:35:011
1851505993930,7cyclictest2091365-21ntpq11:06:320
1851510993827,8cyclictest282-21systemd-journal10:35:002
1851507993834,3cyclictest2096864-21kworker/u8:2+flush-8:011:21:121
1851507993832,4cyclictest1918243-21kworker/u8:1+flush-8:008:56:121
1851507993829,6cyclictest282-21systemd-journal08:01:141
1851507993827,8cyclictest2111106-21ntpq11:26:201
1851507993827,8cyclictest2111106-21ntpq11:26:201
1851505993829,7cyclictest2131231-21ntpq11:46:200
1851507993731,4cyclictest2096864-21kworker/u8:2+events_unbound11:41:291
1851507993730,5cyclictest1799523-21kworker/u8:2+events_unbound07:21:161
1851507993730,5cyclictest1799523-21kworker/u8:2+events_unbound07:21:161
1851505993727,7cyclictest2062478-21latency_hist10:40:010
1851510993622,12cyclictest2072808-21aten2_r5power_v10:51:122
1851510993622,12cyclictest1918400-21aten2_r5power_e08:16:102
1851510993622,12cyclictest1908534-21aten2_r5power_p08:06:092
1851507993630,5cyclictest1883384-21kworker/u8:2+events_unbound08:31:211
1851505993631,4cyclictest2047299-21kworker/u8:3+events_unbound10:26:170
1851505993629,5cyclictest1893915-21diskstats07:51:210
1851505993626,8cyclictest2077899-21diskstats10:56:110
1851505993625,9cyclictest1958598-21diskstats08:56:110
1851507993531,3cyclictest2136706-21kworker/u8:3+flush-8:011:56:161
1851507993530,4cyclictest1851682-21kworker/u8:3+flush-8:007:11:161
1851507993528,5cyclictest1918243-21kworker/u8:1+flush-8:009:11:201
1851507993528,5cyclictest1918243-21kworker/u8:1+events_unbound09:21:161
1851507993528,5cyclictest1918243-21kworker/u8:1+events_unbound09:21:161
1851505993530,3cyclictest1856383-21latency_hist07:15:010
1851505993526,7cyclictest2038220-21diskstats10:16:110
1851510993424,8cyclictest1928396-21cp08:25:002
1851510993419,13cyclictest2091816-21unixbench_multi11:06:382
1851507993430,3cyclictest1958265-21kworker/u8:0+flush-8:009:46:111
185151099332,24cyclictest1933515-21apt-get08:31:082
1851510993320,11cyclictest2152269-21aten2_r5power_v12:11:112
185151099331,25cyclictest2112528-21apt-get11:31:062
1851505993329,3cyclictest1958265-21kworker/u8:0+events_unbound09:46:090
1851505993323,8cyclictest282-21systemd-journal09:45:000
1851510993220,10cyclictest2137315-21sshd11:56:072
1851510993217,13cyclictest0-21swapper/212:30:002
1851507993227,3cyclictest2157324-21tr12:16:101
1851507993226,4cyclictest2096864-21kworker/u8:2+flush-8:011:31:221
1851507993225,5cyclictest1948121-21latency_hist08:45:011
1851507993223,7cyclictest2181482-21latency_hist12:40:001
185126523219,7sleep30-21swapper/307:08:593
185151099312,23cyclictest2122491-21apt-get11:41:102
1851510993115,14cyclictest2172046-21aten2_r5power_p12:31:082
185151099311,25cyclictest2082601-21/usr/sbin/munin11:01:062
1851505993126,3cyclictest2161716-21sh12:19:590
1851505993126,3cyclictest1918243-21kworker/u8:1+flush-8:009:16:280
1851505993125,4cyclictest1883384-21kworker/u8:2+flush-8:008:11:240
1851505993122,7cyclictest2096864-21kworker/u8:2+flush-8:011:21:220
1851505993122,7cyclictest1987810-21latency_hist09:25:010
1851505993122,7cyclictest1987810-21latency_hist09:25:000
185150599311,24cyclictest490116-21kworker/0:0+events08:16:090
185151099303,24cyclictest1991940-21grep09:26:192
185151099303,24cyclictest1991940-21grep09:26:192
185151099302,21cyclictest2142289-21apt-get12:01:092
185151099301,24cyclictest2080982-21kthreadcore10:56:172
1851507993026,3cyclictest2007537-21kworker/u8:3+events_unbound09:41:281
1851505993022,6cyclictest1953055-21latency_hist08:50:010
185151099299,18cyclictest2122131-21cron11:40:012
185151099299,17cyclictest1907195-21grep08:01:182
185151099293,21cyclictest1872399-21ntpq07:26:222
1851510992925,2cyclictest2067819-21aten2_r5power_e10:46:102
1851510992925,2cyclictest1943506-21aten2_r5power_c08:41:092
1851510992919,8cyclictest1888411-21cat07:45:012
1851510992916,11cyclictest0-21swapper/210:21:132
185151099291,25cyclictest2013342-21/usr/sbin/munin09:51:092
185151099291,23cyclictest2008634-21/usr/sbin/munin09:46:112
185151099291,23cyclictest1888608-21apt-get07:45:032
1851510992910,17cyclictest2025836-21tr10:01:192
185151099290,25cyclictest2058267-21/usr/sbin/munin10:36:122
185151099290,16cyclictest1974651-21grep09:11:142
1851505992926,1cyclictest2018046-21kworker/u8:1+events_unbound10:11:090
1851505992925,2cyclictest282-21systemd-journal10:45:000
1851505992925,2cyclictest2167006-21runrttasks12:26:070
1851505992924,3cyclictest1832130-21kworker/u8:0+events_unbound08:10:010
1851505992920,6cyclictest2146868-21latency_hist12:05:010
185151099289,16cyclictest2160197-21tr12:16:172
185151099289,15cyclictest2177811-21tr12:36:152
185151099288,17cyclictest2108487-21tr11:26:152
185151099288,17cyclictest2108487-21tr11:26:152
185151099285,21cyclictest2101654-21grep11:16:302
185151099285,21cyclictest2101654-21grep11:16:302
1851510992824,2cyclictest2162383-21fschecks_time12:21:092
185151099281,24cyclictest1902559-21/usr/sbin/munin07:56:252
185151099281,23cyclictest2042254-21awk10:16:282
185151099281,21cyclictest1968305-21apt-get09:06:072
185150799288,12cyclictest303-21systemd-udevd09:51:071
185150599284,23cyclictest2072381-21kworker/0:2+events11:11:300
1851505992822,5cyclictest1851682-21kworker/u8:3+flush-8:007:15:050
1851510992722,3cyclictest1852021-21fschecks_count07:11:102
1851510992713,12cyclictest0-21swapper/208:11:212
185151099271,23cyclictest2028384-21/usr/sbin/munin10:06:132
185151099271,23cyclictest1943081-21chrt08:39:402
185151099271,21cyclictest2032940-21/usr/sbin/munin10:11:062
1851510992710,15cyclictest0-21swapper/207:53:132
185151099270,25cyclictest2136697-21chrt11:51:342
185151099270,25cyclictest1933067-21seq08:28:422
185151099270,24cyclictest2067432-21munin-run10:45:002
185151099270,23cyclictest2127424-21seq11:46:062
185150599276,19cyclictest1967995-21kworker/0:1+events09:12:100
1851505992723,3cyclictest1893309-21cron07:50:000
1851505992721,4cyclictest1923249-21kworker/u8:4+events_unbound08:36:100
1851505992720,5cyclictest1958265-21kworker/u8:0+flush-8:010:01:230
21472052260,7sleep02147314-21/usr/sbin/munin12:06:100
185151099264,18cyclictest1963294-21runrttasks09:00:442
185151099262,21cyclictest2007125-21ps09:41:222
1851510992613,11cyclictest0-21swapper/211:21:182
185151099261,22cyclictest1978143-21chrt09:15:392
185151099261,21cyclictest1873432-21sshd07:30:062
185151099261,20cyclictest1998293-21sshd09:36:072
185151099260,22cyclictest1958393-21/usr/sbin/munin08:56:092
185150599263,22cyclictest1997994-21kworker/0:2+events09:56:290
1851505992623,2cyclictest2165754-21needreboot12:21:170
1851505992623,2cyclictest1868868-21cut07:26:120
1851505992622,3cyclictest2081734-21kworker/u8:3+events_unbound11:01:310
1851505992616,8cyclictest2180660-21ntpq12:36:220
185151099256,17cyclictest1866726-21tr07:21:212
185151099256,17cyclictest1866726-21tr07:21:212
1851510992513,10cyclictest0-21swapper/208:51:262
185151099251,22cyclictest2052384-21chrt10:28:152
185151099251,22cyclictest1862277-21/usr/sbin/munin07:16:182
185151099250,23cyclictest2096591-21/usr/sbin/munin11:11:322
185151099250,23cyclictest2022509-21chrt09:57:092
185151099250,22cyclictest2147695-21/usr/sbin/munin12:06:132
185151099250,22cyclictest1993274-21taskset09:30:232
185151099250,22cyclictest1993274-21taskset09:30:232
185151099250,21cyclictest1878396-21taskset07:35:282
1851505992521,3cyclictest1994072-21irqrtprio09:31:130
1851505992521,3cyclictest1994072-21irqrtprio09:31:130
1851505992521,3cyclictest0-21swapper/012:15:000
1851505992517,5cyclictest1868270-21latency_hist07:25:010
1851505992517,5cyclictest1868270-21latency_hist07:25:010
1851505992517,1cyclictest0-21swapper/008:41:070
185150599251,23cyclictest1997994-21kworker/0:2+events09:51:100
185150599242,17cyclictest2117178-21kworker/0:0+events12:30:260
1851505992420,3cyclictest2113122-21sort11:31:140
1851505992418,4cyclictest1-21systemd07:45:000
1851505992416,6cyclictest2047578-21latency_hist10:25:010
1851505992317,4cyclictest1923242-21kworker/u8:3+flush-8:008:26:250
1851505992317,4cyclictest1832130-21kworker/u8:0+flush-8:008:01:190
1851505992314,6cyclictest2127090-21latency_hist11:45:010
185150599231,21cyclictest490116-21kworker/0:0+events08:51:140
1851505992220,1cyclictest0-21swapper/010:31:290
1851505992218,3cyclictest2141946-21/usr/sbin/munin12:00:010
1851505992217,3cyclictest1923242-21kworker/u8:3+flush-8:008:31:120
1851505992217,3cyclictest0-21swapper/007:56:170
185150599221,20cyclictest2072381-21kworker/0:2+events10:52:020
185150599220,21cyclictest2117178-21kworker/0:0+events11:51:140
18510722214,11sleep10-21swapper/107:06:291
18465942214,10sleep20-21swapper/207:05:022
21617032132,6sleep30-21swapper/312:20:013
21119502132,7sleep30-21swapper/311:30:013
21119502132,7sleep30-21swapper/311:30:013
20197422132,7sleep30-21swapper/309:56:193
20029712132,6sleep30-21swapper/309:40:013
19279532130,8sleep30-21swapper/308:25:013
19039682132,7sleep30-21swapper/308:01:133
21723822122,6sleep30-21swapper/312:31:133
21625912122,6sleep30-21swapper/312:21:143
21542272122,6sleep30-21swapper/312:11:183
21509512122,6sleep30-21swapper/312:06:213
21462722122,6sleep30-21swapper/312:01:273
21328572120,8sleep30-21swapper/311:51:113
21314742122,6sleep30-21swapper/311:46:263
21249622122,6sleep30-21swapper/311:41:213
21215862122,6sleep30-21swapper/311:36:303
20978662122,6sleep30-21swapper/311:16:163
20978662122,6sleep30-21swapper/311:16:163
20872092122,6sleep30-21swapper/311:05:013
20821672122,6sleep30-21swapper/310:59:063
20700012122,6sleep30-21swapper/310:46:193
20577982122,6sleep30-21swapper/310:36:073
20521812122,6sleep30-21swapper/310:26:283
20328622122,6sleep30-21swapper/310:10:073
20090652122,6sleep30-21swapper/309:46:153
20022032122,6sleep30-21swapper/309:36:243
19942372122,6sleep30-21swapper/309:31:143
19942372122,6sleep30-21swapper/309:31:133
19832062122,6sleep30-21swapper/309:21:073
19832062122,6sleep30-21swapper/309:21:073
19819592122,6sleep30-21swapper/309:16:223
19633532122,6sleep30-21swapper/309:01:093
19590602122,6sleep30-21swapper/308:56:153
19573892122,6sleep30-21swapper/308:51:253
19377692122,6sleep30-21swapper/308:31:273
19285392122,6sleep30-21swapper/308:26:093
19149032122,6sleep30-21swapper/308:11:163
18933312122,5sleep30-21swapper/307:50:013
18834872122,6sleep30-21swapper/307:41:113
18734312122,6sleep30-21swapper/307:30:073
18684012122,5sleep30-21swapper/307:25:013
18648152122,6sleep30-21swapper/307:21:203
18648152122,6sleep30-21swapper/307:21:203
18545132122,6sleep30-21swapper/307:11:173
21769612112,5sleep30-21swapper/312:36:103
21710372112,6sleep30-21swapper/312:26:273
21373162112,5sleep30-21swapper/311:56:093
21151062112,6sleep30-21swapper/311:31:173
21024612112,5sleep30-21swapper/311:21:103
20928322112,5sleep30-21swapper/311:11:193
20727822112,5sleep30-21swapper/310:51:123
20525552112,5sleep30-21swapper/310:30:013
20429792112,6sleep30-21swapper/310:21:113
20378992112,6sleep30-21swapper/310:15:333
20276562111,5sleep30-21swapper/310:05:003
20230662112,5sleep30-21swapper/310:01:083
20171452112,5sleep30-21swapper/309:51:253
19881722112,5sleep30-21swapper/309:26:093
19881722112,5sleep30-21swapper/309:26:093
19769982112,5sleep30-21swapper/309:11:203
19723112112,5sleep30-21swapper/309:06:263
19483502112,5sleep30-21swapper/308:45:073
19475202112,5sleep30-21swapper/308:41:253
19385012112,6sleep30-21swapper/308:36:093
19186872112,5sleep30-21swapper/308:16:143
19084202112,5sleep30-21swapper/308:06:063
18994712112,5sleep30-21swapper/307:56:163
18887182112,5sleep30-21swapper/307:46:083
18787612112,5sleep30-21swapper/307:36:133
18591022112,5sleep30-21swapper/307:16:153
20824952102,5sleep30-21swapper/311:00:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional