You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-05 - 02:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Mon Jan 05, 2026 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2209304211458,51sleep30-21swapper/319:06:063
220943528555,9sleep20-21swapper/219:07:452
220957228254,20sleep00-21swapper/019:09:310
220876827456,10sleep10-21swapper/119:05:231
2209720992320,1cyclictest0-21swapper/320:39:413
2209720992320,1cyclictest0-21swapper/319:10:273
2209720992319,1cyclictest0-21swapper/322:50:003
220972099231,18cyclictest2269610-21taskset20:05:013
221530822211,7sleep12215108-21munin-node19:15:011
2209720992219,1cyclictest0-21swapper/300:09:593
2209720992118,1cyclictest0-21swapper/322:40:263
2209720992017,1cyclictest0-21swapper/320:44:593
2209720992017,1cyclictest0-21swapper/320:25:123
2209720992016,2cyclictest0-21swapper/323:30:103
2209720992016,2cyclictest0-21swapper/322:20:123
220972099199,7cyclictest2338366-21taskset21:10:133
220972099195,7cyclictest2495204-21taskset23:35:483
2209720991917,1cyclictest0-21swapper/319:30:173
2209720991916,1cyclictest0-21swapper/323:40:243
2209720991916,1cyclictest0-21swapper/323:10:213
2209720991916,1cyclictest0-21swapper/322:10:183
2209720991916,1cyclictest0-21swapper/320:45:153
2209720991916,1cyclictest0-21swapper/319:15:153
2209720991915,2cyclictest0-21swapper/322:35:143
2209720991915,2cyclictest0-21swapper/321:55:293
220972099191,14cyclictest0-21swapper/320:50:123
220972099191,13cyclictest2537914-21chrt00:20:133
220972099191,13cyclictest2369211-21turbostat21:40:013
220972099186,9cyclictest2390171-21turbostat22:04:553
2209720991815,1cyclictest0-21swapper/323:15:123
2209720991815,1cyclictest0-21swapper/320:00:273
2209720991815,1cyclictest0-21swapper/300:25:183
2209720991814,2cyclictest0-21swapper/321:20:013
2209720991814,2cyclictest0-21swapper/320:20:013
220972099181,13cyclictest0-21swapper/320:55:263
220972099181,13cyclictest0-21swapper/300:35:143
220972099181,10cyclictest2451784-21taskset22:55:163
23420142171,10sleep22342061-21/usr/sbin/munin21:10:222
2209720991714,1cyclictest0-21swapper/319:45:113
2209720991713,2cyclictest0-21swapper/323:25:003
2209720991713,2cyclictest0-21swapper/320:10:183
220972099171,10cyclictest2396011-21taskset22:05:133
220972099170,14cyclictest0-21swapper/321:40:133
24638142161,11sleep02463925-21sed23:10:010
24329402161,9sleep22433149-21hddtemp_smartct22:40:172
24174172161,10sleep12417486-21irqstats22:25:171
22497882163,8sleep02251484-21cut19:45:150
220972099161,9cyclictest2453606-21taskset23:00:123
220972099161,9cyclictest2243393-21chrt19:40:123
2209720991613,1cyclictest0-21swapper/322:25:133
2209720991612,2cyclictest0-21swapper/300:33:243
2209720991611,3cyclictest0-21swapper/323:45:123
220972099161,13cyclictest0-21swapper/322:35:013
220972099161,13cyclictest0-21swapper/321:05:203
220972099161,13cyclictest0-21swapper/319:25:013
220972099161,12cyclictest0-21swapper/321:00:153
220972099161,12cyclictest0-21swapper/321:00:153
220972099161,12cyclictest0-21swapper/319:55:223
220972099161,11cyclictest2443089-21chrt22:50:113
220972099161,10cyclictest2295468-21sleep20:34:553
220972099160,13cyclictest0-21swapper/322:15:233
220972099160,12cyclictest0-21swapper/319:35:203
25501132151,9sleep22551886-21tr00:30:192
22640832151,4sleep22264116-21munin-run20:00:002
220972099151,9cyclictest2533113-21taskset00:15:153
220972099151,9cyclictest2527362-21taskset00:10:113
220972099151,9cyclictest2516363-21taskset23:59:133
220972099151,9cyclictest2463590-21taskset23:07:103
220972099151,9cyclictest2253900-21taskset19:50:063
220972099151,9cyclictest2232538-21taskset19:28:093
2209720991512,1cyclictest0-21swapper/321:50:223
2209720991512,1cyclictest0-21swapper/321:30:183
2209720991512,1cyclictest0-21swapper/321:25:173
2209720991512,1cyclictest0-21swapper/320:20:143
2209720991512,1cyclictest0-21swapper/300:00:193
220972099151,11cyclictest2353452-21sleep21:25:003
220972099150,12cyclictest0-21swapper/323:50:183
25027152140,3sleep20-21swapper/223:45:172
24378802140,2sleep10-21swapper/122:45:131
24155612141,8sleep12415650-21ntpq22:20:201
220972099141,9cyclictest2480035-21taskset23:25:023
220972099140,11cyclictest0-21swapper/321:45:123
25157352131,8sleep12515786-21sed23:55:261
25069422131,7sleep02508563-21kthreadcore23:50:170
24260972131,3sleep02426133-21/usr/sbin/munin22:30:200
24069382131,8sleep22407557-21cut22:15:152
23659112131,7sleep12367668-21kthreadcore21:35:181
23158002131,8sleep22315914-21/usr/sbin/munin20:45:212
22644412131,8sleep02264655-21df_inode20:00:130
22635452131,7sleep02263714-21users19:55:290
25489322121,6sleep12548965-21kthreadcore00:30:171
25481002121,5sleep02548148-21munin-run00:30:010
25207652121,7sleep02520811-21ntp_kernel_pll_00:00:230
25065212121,6sleep12506543-21fschecks_count23:50:141
24972992121,7sleep02497856-21kthreadcore23:40:190
24593792121,6sleep22459422-21kthreadcore23:05:182
24488702121,6sleep12450018-21kthreadcore22:55:151
23796672121,7sleep02379879-21latency_hist21:50:000
23647732125,3sleep02365909-21cut21:35:170
23366962121,6sleep22336745-21ntpq21:05:262
22696232121,5sleep12269642-21apt-get20:05:121
22544642123,5sleep12254995-21cut19:50:201
22476442127,2sleep00-21swapper/019:40:250
22368002122,5sleep02236889-21ntpq19:30:190
22335152121,7sleep12233551-21kthreadcore19:30:171
24222732111,5sleep12422441-21diskstats22:30:141
24174872114,4sleep22417522-21kthreadcore22:25:182
24048692111,5sleep22404910-21ntpq22:10:232
23733362111,5sleep12373424-21ntpq21:40:201
23393202111,7sleep12341029-21cut21:10:171
22917312111,6sleep22293428-21kthreadcore20:25:172
22399192111,6sleep12241702-21kthreadcore19:35:181
25538372101,4sleep12553341-21apt-get00:35:031
25225392101,4sleep02522608-21irqstats00:05:160
24958212101,4sleep22495840-21apt-get23:40:132
24840052101,4sleep22484047-21ntpq23:25:222
24732772101,4sleep02473324-21ntpq23:15:220
24436062100,6sleep22443930-21kthreadcore22:50:142
23838232101,4sleep02383909-21ntpq21:50:210
23749682101,5sleep22374990-21/usr/sbin/munin21:45:142
23592062101,4sleep22359329-21perl21:30:152
23476062101,4sleep22347791-21users21:15:282
23432612101,5sleep02343474-21df_inode21:15:130
23385222101,5sleep02339319-21grep21:10:160
23362452101,5sleep02336641-21grep21:05:240
22997352101,4sleep22299780-21ntpq20:30:212
22844432101,4sleep22284628-21users20:15:312
22421072101,4sleep02242156-21ntpq19:35:210
22280182101,4sleep22228120-21perl19:25:162
2537892291,2sleep20-21swapper/200:20:132
2537429290,1sleep20-21swapper/200:17:472
2534786291,4sleep12536425-21cat00:15:191
2531453291,3sleep22531585-21apt-get00:10:232
2531394291,4sleep12531451-21grep00:10:221
2511821291,4sleep22512010-21if_enp1s023:55:152
2467991291,4sleep02468042-21ntpq23:10:250
2390989291,4sleep22391660-21cat22:00:172
2390839291,4sleep12390988-21if_err_enp1s022:00:161
2389197291,4sleep12389256-21ld21:55:221
2381388291,4sleep22383129-21cut21:50:192
2373425291,4sleep22373472-21ntpq21:40:212
2349224291,4sleep02350865-21tr21:20:190
2296265291,4sleep12297212-21grep20:30:171
2295117291,3sleep02295118-21sshd20:26:040
2274193290,1sleep00-21swapper/020:05:310
2260069291,4sleep12261705-21cut19:55:181
2226474291,3sleep02226506-21ntpq19:20:220
2544057281,3sleep22545724-21kthreadcore00:25:182
2522609281,3sleep12523683-21grep00:05:161
2511500280,1sleep00-21swapper/023:55:140
2505741280,1sleep10-21swapper/123:46:561
2479518281,3sleep12479587-21/usr/sbin/munin23:25:011
2469285280,2sleep22469304-21apt-get23:15:122
2458556281,3sleep12458568-21latency_hist23:05:001
2453866281,3sleep02453968-21perl23:00:150
2453299281,3sleep22453535-21grep23:00:012
2452826280,1sleep00-21swapper/022:55:260
2429279281,3sleep12431064-21kthreadcore22:35:191
2412209281,3sleep22413360-21kthreadcore22:20:172
2410657281,3sleep00-21swapper/022:15:260
2404824281,3sleep12404862-21ntpq22:10:221
2396644280,4sleep10-21swapper/122:05:161
2395921281,3sleep22396010-21cut22:05:132
2385766281,3sleep02387231-21ps21:55:170
2380495281,3sleep12380533-21kthreadcore21:50:181
2358550280,1sleep10-21swapper/121:28:411
2353682281,2sleep02353701-21apt-get21:25:110
2327836281,3sleep22328025-21if_enp1s021:00:162
2327836281,3sleep22328025-21if_enp1s021:00:152
2326579280,1sleep20-21swapper/220:55:282
2316949281,2sleep00-21swapper/020:50:120
2315755280,1sleep10-21swapper/120:45:191
2311645280,3sleep00-21swapper/020:45:020
2310269280,1sleep20-21swapper/220:40:232
2285578281,3sleep22285790-21smartctl20:20:152
2275369281,3sleep22275966-21kthreadcore20:10:172
2270374281,3sleep22271644-21kthreadcore20:05:172
2232684281,4sleep12232877-21cat19:30:011
2226508281,2sleep12226552-21ntpq19:20:231
2216813281,3sleep22217165-21ntp_192.168.11419:15:022
2210643281,3sleep02209906-21/usr/sbin/munin19:10:160
2555637271,2sleep20-21swapper/200:35:202
2553870270,2sleep00-21swapper/000:35:140
2548360270,2sleep00-21swapper/000:30:120
2517026270,3sleep22517158-21head00:00:172
2501030270,2sleep02501049-21apt-get23:45:110
2494587271,2sleep00-21swapper/023:35:210
2490562271,1sleep22490581-21apt-get23:35:122
2487190271,2sleep02488978-21kthreadcore23:30:170
2482227271,2sleep00-21swapper/023:25:190
2478563270,3sleep02478613-21ld23:20:230
2474569270,2sleep20-21swapper/223:20:122
2471171271,3sleep12472955-21kthreadcore23:15:191
2464358270,3sleep22464380-21fschecks_count23:10:182
2454133271,2sleep20-21swapper/223:00:162
2446882271,3sleep12447021-21awk22:50:171
2432672271,3sleep02432833-21cstates22:40:140
2432116270,1sleep20-21swapper/222:36:522
2400440270,1sleep00-21swapper/022:07:480
2395328270,1sleep00-21swapper/022:04:230
2369772270,4sleep00-21swapper/021:40:150
2332723270,2sleep10-21swapper/121:05:161
2328347270,3sleep02329737-21grep21:00:180
2328347270,3sleep02329737-21grep21:00:170
2326215271,2sleep02326261-21ntpq20:55:200
2321253270,2sleep22321324-21find20:50:262
2320964270,2sleep10-21swapper/120:50:211
2305879270,3sleep22306087-21latency_hist20:40:002
2305582270,1sleep10-21swapper/120:36:151
2300198271,2sleep00-21swapper/020:30:280
2290497270,1sleep10-21swapper/120:25:121
2287324271,3sleep12289009-21latency20:20:181
2269328270,1sleep10-21swapper/120:04:351
2244938271,2sleep10-21swapper/119:40:191
2543127260,2sleep12543148-21http00:25:131
2542842260,4sleep10-21swapper/100:24:451
2533209261,2sleep00-21swapper/000:15:180
2527507261,1sleep00-21swapper/000:10:130
2523684261,2sleep22524121-21kthreadcore00:05:172
2518132260,2sleep10-21swapper/100:00:201
2500081260,2sleep10-21swapper/123:40:281
2485305260,1sleep20-21swapper/223:30:112
2480357261,2sleep10-21swapper/123:25:171
2464769261,2sleep10-21swapper/123:10:221
2443349261,2sleep00-21swapper/022:50:120
2438462261,2sleep20-21swapper/222:45:182
2437874260,4sleep00-21swapper/022:45:080
2427940261,2sleep00-21swapper/022:35:180
2425375261,2sleep20-21swapper/222:30:192
2421816260,4sleep00-21swapper/022:29:040
2406353260,1sleep10-21swapper/122:15:031
2388946260,2sleep22389094-21strings21:55:192
2375296261,2sleep10-21swapper/121:45:161
2362969260,2sleep10-21swapper/121:30:221
2358938260,3sleep00-21swapper/021:30:130
2355323261,2sleep20-21swapper/221:25:182
2353417261,3sleep2800-21snmpd21:24:562
2332119260,4sleep10-21swapper/121:00:441
2332119260,4sleep10-21swapper/121:00:431
2311136260,2sleep12311282-21store20:45:001
2307477261,2sleep00-21swapper/020:40:180
2289872260,1sleep00-21swapper/020:20:390
2275099261,2sleep02274814-21/usr/sbin/munin20:10:160
2269319260,4sleep20-21swapper/220:04:282
2258856260,4sleep00-21swapper/019:54:560
2253347260,1sleep20-21swapper/219:46:142
2247462261,2sleep20-21swapper/219:40:242
2238145260,1sleep20-21swapper/219:35:022
2234491261,2sleep20-21swapper/219:30:182
2227726260,1sleep00-21swapper/019:25:070
2222814261,2sleep20-21swapper/219:20:162
2214957261,2sleep00-21swapper/019:15:010
2214273261,2sleep20-21swapper/219:10:262
2541909250,1sleep00-21swapper/000:20:220
2494852250,1sleep10-21swapper/123:35:261
2488981250,1sleep12489250-21strings23:30:181
2462673250,1sleep10-21swapper/123:05:211
2434471251,1sleep10-21swapper/122:40:191
2416649250,1sleep00-21swapper/022:24:550
2352492250,1sleep10-21swapper/121:20:241
2326263250,1sleep10-21swapper/120:55:211
2301129250,1sleep00-21swapper/020:35:120
2280308251,1sleep00-21swapper/020:15:170
2278607250,1sleep10-21swapper/120:10:201
2253936250,2sleep20-21swapper/219:50:182
2251485250,1sleep10-21swapper/119:45:161
2214850250,1sleep10-21swapper/119:13:421
2506568240,1sleep20-21swapper/223:50:152
2401203240,1sleep00-21swapper/022:10:170
2364492240,1sleep20-21swapper/221:35:162
2343475240,1sleep10-21swapper/121:15:151
2280024240,1sleep10-21swapper/120:15:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional