You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-07 - 12:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Wed Jan 07, 2026 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2033656212054,61sleep30-21swapper/319:07:483
21096442930,1sleep1321rcuc/120:20:111
203360929259,10sleep10-21swapper/119:07:111
203370427964,9sleep20-21swapper/219:08:242
203379627853,19sleep00-21swapper/019:09:330
2033958993322,7cyclictest2362131-21turbostat00:20:003
203395899212,13cyclictest2119985-21taskset20:30:123
2033958992118,1cyclictest0-21swapper/320:50:003
2033958992017,1cyclictest0-21swapper/322:30:123
2033958992017,1cyclictest0-21swapper/320:20:183
2033958992017,1cyclictest0-21swapper/300:35:213
2033958992013,3cyclictest2167334-21sleep21:15:003
203395899201,15cyclictest2377896-21sleep00:35:003
203395899201,15cyclictest2067255-21taskset19:38:573
203395899201,14cyclictest0-21swapper/323:10:113
203395899197,9cyclictest2314325-21sleep23:35:013
2033958991916,1cyclictest0-21swapper/321:55:163
2033958991916,1cyclictest0-21swapper/321:40:173
2033958991916,1cyclictest0-21swapper/321:30:003
2033958991916,1cyclictest0-21swapper/320:40:163
2033958991916,1cyclictest0-21swapper/320:25:273
2033958991915,2cyclictest0-21swapper/323:35:153
203395899191,13cyclictest0-21swapper/322:20:123
203395899191,13cyclictest0-21swapper/322:00:133
21098282181,4sleep22110037-21hddtemp_smartct20:20:142
2033958991815,1cyclictest0-21swapper/323:40:173
2033958991815,1cyclictest0-21swapper/323:20:133
2033958991815,1cyclictest0-21swapper/322:35:153
2033958991815,1cyclictest0-21swapper/319:10:163
2033958991814,2cyclictest0-21swapper/322:30:013
2033958991814,2cyclictest0-21swapper/319:15:223
203395899181,13cyclictest0-21swapper/319:20:133
20882512170,10sleep12088400-21sed20:00:001
203395899177,7cyclictest2129979-21taskset20:36:573
2033958991714,1cyclictest0-21swapper/323:50:113
2033958991714,1cyclictest0-21swapper/322:05:163
2033958991713,2cyclictest0-21swapper/323:00:173
2033958991713,2cyclictest0-21swapper/300:20:173
203395899170,13cyclictest0-21swapper/323:25:203
203395899170,12cyclictest0-21swapper/320:55:123
203395899170,11cyclictest2233742-21taskset22:15:163
21995662161,10sleep12200085-21tr21:45:131
21681132161,7sleep12168622-21kthreadcore21:15:161
203395899161,9cyclictest2262210-21taskset22:45:033
203395899161,9cyclictest2188482-21taskset21:35:093
203395899161,9cyclictest2072844-21taskset19:45:023
2033958991613,1cyclictest0-21swapper/323:05:153
2033958991613,1cyclictest0-21swapper/321:05:163
2033958991613,1cyclictest0-21swapper/320:10:203
2033958991613,1cyclictest0-21swapper/300:05:173
2033958991612,2cyclictest0-21swapper/321:55:013
2033958991612,2cyclictest0-21swapper/321:55:003
2033958991611,3cyclictest0-21swapper/321:45:103
203395899161,12cyclictest2093208-21taskset20:00:583
203395899161,12cyclictest2056880-21turbostat19:30:003
203395899161,10cyclictest2372538-21taskset00:28:283
203395899161,10cyclictest2296797-21taskset23:15:163
203395899161,10cyclictest2272734-21taskset22:55:133
203395899161,10cyclictest2170329-21taskset21:15:163
203395899161,10cyclictest2057150-21taskset19:30:113
203395899160,13cyclictest0-21swapper/323:45:193
203395899160,13cyclictest0-21swapper/322:10:183
203395899160,13cyclictest0-21swapper/320:50:183
203395899160,10cyclictest2153835-21taskset21:00:173
23361382151,4sleep02336261-21perl23:55:160
23316972151,9sleep12333402-21cut23:50:151
21838392151,9sleep22184784-21grep21:30:152
21619442158,2sleep2171rcu_preempt21:08:472
20341272151,4sleep22034146-21apt-get19:10:112
203395899151,9cyclictest2272096-21taskset22:50:323
203395899151,9cyclictest2083291-21taskset19:55:043
2033958991512,1cyclictest0-21swapper/323:55:183
2033958991512,1cyclictest0-21swapper/300:00:153
203395899151,11cyclictest0-21swapper/322:40:103
203395899151,10cyclictest2356897-21turbostat00:15:003
203395899150,12cyclictest0-21swapper/321:20:173
203395899150,12cyclictest0-21swapper/319:50:163
203395899150,12cyclictest0-21swapper/319:40:143
203395899150,11cyclictest21847862sleep321:30:163
23575322141,9sleep12357702-21cat00:15:171
23451732141,9sleep12345222-21ldconfig00:00:211
23200642141,7sleep02320083-21apt-get23:40:110
22560092141,4sleep12256225-21sed22:35:241
22137502141,8sleep02213938-21cat21:55:270
22135722140,6sleep12213699-21systemctl21:55:231
21345262141,9sleep22134585-21ld20:40:212
20525922141,4sleep22054244-21grep19:25:182
203395899142,9cyclictest2098329-21chrt20:05:263
203395899141,10cyclictest15850irq/127-ahci[0000:00:17.0]20:15:133
23133822131,8sleep02313438-21pmu-power23:30:220
22527582131,8sleep02254460-21cut22:35:140
22000862131,8sleep20-21swapper/221:45:152
21082792121,5sleep22108328-21ntpq20:15:202
20732402121,8sleep22073409-21/usr/sbin/munin19:45:142
20676432122,6sleep134-21ksoftirqd/119:40:121
20523892121,6sleep12052476-21kthreadcore19:25:171
23644652111,6sleep22366186-21wc00:20:182
23577032111,6sleep22219742-21kworker/2:0+events00:15:182
23291522114,3sleep22329293-21ntpq23:45:192
23080942111,5sleep02308142-21ntpq23:25:210
22609922111,5sleep02261046-21ntpq22:40:190
22134212111,5sleep22213461-21ntpq21:55:202
21830112111,5sleep22183031-21munin-run21:30:002
20885352112,4sleep02088554-21apt-get20:00:120
23717262100,1sleep00-21swapper/000:25:210
23716192101,4sleep22371670-21ntpq00:25:202
23519132101,4sleep22351932-21apt-get00:10:122
23195042101,5sleep01-21systemd23:39:590
23145532101,4sleep12314572-21apt-get23:35:111
22453592101,4sleep22245568-21users22:25:282
22354732101,4sleep12235545-21/usr/sbin/munin22:20:011
22251942101,4sleep22225213-21apt-get22:10:112
22080062101,5sleep12208186-21uname21:50:181
22080062101,5sleep12208186-21uname21:50:181
21942712101,6sleep02194354-21kernelversion21:40:150
21885692101,4sleep22188772-21diskstats21:35:132
21827502100,1sleep00-21swapper/021:26:450
21728502100,2sleep20-21swapper/221:20:132
21662612101,4sleep22166309-21ntpq21:10:192
21576382100,7sleep00-21swapper/021:05:170
21508752101,5sleep02151119-21users20:55:270
20833222101,4sleep22083580-21df_inode19:55:122
20609442101,4sleep12061085-21ntpq19:30:181
20466822101,3sleep10-21swapper/119:20:121
20419182101,5sleep12042847-21kthreadcore19:15:151
20413512101,5sleep02041355-21find19:15:100
20368042101,4sleep12037950-21memory19:10:181
2367953291,4sleep12368166-21hddtemp_smartct00:25:151
2362408291,3sleep02362427-21apt-get00:20:120
2356201293,3sleep02356425-21grep00:10:260
2341331291,4sleep02341443-21/usr/sbin/munin00:00:110
2334902291,5sleep20-21swapper/223:50:242
2320086291,4sleep12320321-21ls23:40:121
2314575291,3sleep22314831-21df23:35:122
2300382291,4sleep22302153-21cut23:20:182
2293635291,3sleep12293653-21apt-get23:15:111
2287234291,4sleep02287317-21grep23:05:210
2281907291,3sleep12281949-21ntpq23:00:221
2273217291,4sleep22273300-21tr22:55:162
2247025291,4sleep22247533-21grep22:30:152
2245248291,4sleep12245357-21grep22:25:261
2240038291,4sleep22240157-21cat22:20:252
2194355291,4sleep12195851-21tr21:40:161
2188224291,4sleep12188247-21munin-plugin-st21:34:591
2176733291,4sleep12176781-21ntpq21:20:201
2164080291,4sleep02165875-21cut21:10:170
2147071292,4sleep22148552-21kthreadcore20:55:162
2141730291,4sleep02141836-21irqstats20:50:130
2135734291,3sleep22135753-21apt-get20:45:102
2130523291,4sleep02130776-21df20:40:110
2118656291,4sleep12118704-21ntpq20:25:211
2104835291,4sleep12105168-21grep20:15:151
2078541291,4sleep12078657-21irqstats19:50:151
2074190291,4sleep02075932-21kthreadcore19:45:160
2071662291,4sleep22071710-21/usr/sbin/munin19:40:212
2067621291,3sleep02067640-21apt-get19:40:120
2066350291,3sleep12066395-21ntpq19:35:191
2050637291,4sleep22050686-21ntpq19:20:212
2349199281,3sleep12350491-21memory00:05:191
2346652280,4sleep22346155-21apt-get00:05:022
2330569280,2sleep00-21swapper/023:50:090
2324981280,1sleep20-21swapper/223:44:272
2313135281,3sleep12313276-21ntpq23:30:191
2303599280,1sleep10-21swapper/123:21:581
2246464282,3sleep12033953-21cyclictest22:30:021
2245580280,1sleep00-21swapper/022:25:290
2240564280,1sleep10-21swapper/122:22:571
2235757280,4sleep00-21swapper/022:20:110
2230495281,3sleep22230756-21df_inode22:15:132
2219350281,3sleep12219352-21modprobe22:00:571
2215016281,3sleep02215224-21gltestperf22:00:150
2146999280,5sleep10-21swapper/120:55:161
2142988281,3sleep22144790-21kthreadcore20:50:152
2141837281,3sleep12142987-21grep20:50:141
2129120281,3sleep02129209-21ntpq20:35:180
2125787281,3sleep12126716-21cat20:35:151
2124646280,1sleep00-21swapper/020:31:020
2120264281,3sleep10-21swapper/120:30:141
2119963281,2sleep20-21swapper/220:30:122
2119700283,2sleep2421rcuc/220:29:592
2088557281,3sleep10-21swapper/120:00:131
2067035280,1sleep20-21swapper/219:36:132
2062362280,3sleep02062381-21apt-get19:35:100
2050742280,1sleep00-21swapper/019:20:230
2042850281,3sleep22044608-21kthreadcore19:15:172
2035099281,3sleep02036803-21tr19:10:170
2382413271,2sleep02378131-21/usr/sbin/munin00:35:270
2377824270,1sleep10-21swapper/100:34:271
2377549270,1sleep20-21swapper/200:31:002
2356564272,3sleep12033953-21cyclictest00:10:591
2339751271,2sleep12339840-21ntpq23:55:201
2330201270,1sleep10-21swapper/123:48:381
2309869270,4sleep22310440-21cut23:30:162
2308888270,1sleep20-21swapper/223:27:552
2298333270,1sleep20-21swapper/223:16:532
2293191270,1sleep00-21swapper/023:12:430
2293057270,1sleep20-21swapper/223:11:012
2288065270,1sleep10-21swapper/123:09:021
2287415270,4sleep20-21swapper/223:05:232
2277330270,1sleep00-21swapper/022:55:510
2260801270,2sleep20-21swapper/222:40:172
2246757270,3sleep02246808-21sort22:30:130
2234372271,2sleep00-21swapper/022:15:190
2229596270,1sleep10-21swapper/122:10:261
2224729270,1sleep00-21swapper/022:07:260
2204257271,2sleep20-21swapper/221:50:112
2204257271,2sleep20-21swapper/221:50:112
2189057271,2sleep10-21swapper/121:35:151
2172384270,1sleep00-21swapper/021:17:590
2162309270,1sleep10-21swapper/121:10:101
2156773271,3sleep02156925-21latency_hist21:05:000
2126717271,3sleep20-21swapper/220:35:162
2114234270,1sleep00-21swapper/020:21:480
2109132270,1sleep00-21swapper/020:18:270
2098116272,2sleep22098328-21sed20:05:262
2093789270,2sleep00-21swapper/020:05:120
2077553270,5sleep10-21swapper/119:46:341
2057410270,3sleep0221rcuc/019:30:140
2057128270,2sleep20-21swapper/219:30:112
2056687270,1sleep00-21swapper/019:27:450
2378700261,2sleep20-21swapper/200:35:182
2377716260,1sleep00-21swapper/000:33:060
2366418260,2sleep10-21swapper/100:20:211
2361803260,4sleep00-21swapper/000:16:040
2341152260,1sleep20-21swapper/200:00:092
2336418260,2sleep22337284-21kthreadcore23:55:172
2303712260,1sleep00-21swapper/023:23:200
2293324260,1sleep10-21swapper/123:14:241
2282582260,4sleep00-21swapper/023:00:570
2282216261,2sleep20-21swapper/223:00:262
2277428261,3sleep1770-21lldpd22:57:041
2268016261,2sleep2421rcuc/222:50:152
2263248261,2sleep00-21swapper/022:45:160
2262499260,2sleep10-21swapper/122:45:131
2261863260,1sleep10-21swapper/122:43:491
2229933260,1sleep00-21swapper/022:11:300
2224829260,1sleep10-21swapper/122:08:431
2223835261,2sleep20-21swapper/222:05:182
2215297260,2sleep20-21swapper/222:00:172
2208955260,4sleep00-21swapper/021:51:350
2208955260,4sleep00-21swapper/021:51:340
2198557260,1sleep20-21swapper/221:42:262
2192435260,2sleep00-21swapper/021:35:180
2188118260,4sleep00-21swapper/021:33:420
2177737260,1sleep00-21swapper/021:24:320
2168625261,2sleep20-21swapper/221:15:172
2156926262,1sleep10-21swapper/121:05:021
2155775260,2sleep20-21swapper/221:00:212
2151793261,2sleep10-21swapper/121:00:121
2139566260,2sleep12139679-21strings20:45:171
2130777261,2sleep12130810-21fschecks_count20:40:121
2094897261,2sleep10-21swapper/120:05:171
2078658261,2sleep20-21swapper/219:50:152
2347521250,2sleep00-21swapper/000:05:180
2326406250,2sleep00-21swapper/023:45:180
2267740250,2sleep00-21swapper/022:50:140
2266036250,1sleep20-21swapper/222:45:182
2256608252,1sleep20-21swapper/222:38:502
2198762250,1sleep00-21swapper/021:45:000
2182117250,1sleep10-21swapper/121:25:211
2119458251,2sleep00-21swapper/020:26:570
2099068250,1sleep20-21swapper/220:10:122
2088771250,1sleep20-21swapper/220:00:132
2087121250,1sleep00-21swapper/019:55:180
2082903250,3sleep00-21swapper/019:53:190
2378122240,1sleep10-21swapper/100:35:081
2303913240,1sleep10-21swapper/123:25:011
2294044240,1sleep00-21swapper/023:15:140
2267454240,1sleep10-21swapper/122:50:101
2135475240,1sleep00-21swapper/020:45:010
2099362240,1sleep10-21swapper/120:10:151
2098859240,1sleep00-21swapper/020:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional