You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-20 - 17:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Sat Dec 20, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
329559321940,2sleep23295551-31find07:37:422
3261882212669,52sleep30-21swapper/307:06:343
325723628555,9sleep10-21swapper/107:05:071
326198428259,6sleep00-21swapper/007:07:510
326184327858,10sleep20-21swapper/207:06:042
3262288992925,2cyclictest0-21swapper/311:40:003
3262288992925,2cyclictest0-21swapper/311:40:003
3262288992623,1cyclictest0-21swapper/308:40:003
3262288992623,1cyclictest0-21swapper/307:37:573
3262288992421,1cyclictest0-21swapper/308:15:003
34693592231,11sleep23469557-21cat10:25:002
3262288992320,1cyclictest0-21swapper/312:35:243
3262288992320,1cyclictest0-21swapper/311:30:133
3262288992320,1cyclictest0-21swapper/308:50:003
326228899221,17cyclictest3416629-21taskset09:31:583
326228899217,7cyclictest3401138-21taskset09:19:543
3262288992118,1cyclictest0-21swapper/310:05:183
3262288992017,1cyclictest0-21swapper/310:35:173
32665102191,9sleep23266558-21ntpq07:10:212
3262288991915,2cyclictest0-21swapper/307:30:243
326228899191,15cyclictest0-21swapper/307:25:183
326228899191,14cyclictest0-21swapper/310:00:123
326228899190,16cyclictest0-21swapper/311:55:263
35842112181,12sleep23584260-21ntpq12:10:212
3262288991816,1cyclictest0-21swapper/307:20:143
3262288991816,1cyclictest0-21swapper/307:20:143
3262288991815,1cyclictest0-21swapper/312:25:163
3262288991815,1cyclictest0-21swapper/310:30:183
3262288991815,1cyclictest0-21swapper/310:30:183
3262288991815,1cyclictest0-21swapper/309:40:203
3262288991815,1cyclictest0-21swapper/307:50:013
326228899181,13cyclictest3437767-21turbostat09:55:003
326228899181,13cyclictest3364531-21turbostat08:45:003
326228899181,13cyclictest0-21swapper/311:45:133
326228899181,12cyclictest3327585-21turbostat08:10:003
326228899181,12cyclictest3301124-21chrt07:45:003
326228899181,10cyclictest3406548-21taskset09:25:003
33377022171,10sleep23337919-21/usr/sbin/munin08:15:262
326228899173,6cyclictest3306865-21taskset07:50:013
3262288991715,1cyclictest0-21swapper/308:25:183
326228899171,13cyclictest0-21swapper/312:00:263
326228899171,13cyclictest0-21swapper/310:55:143
326228899171,12cyclictest3469642-21chrt10:25:123
326228899171,12cyclictest3411650-21turbostat09:30:003
326228899171,10cyclictest3495282-21taskset10:46:493
326228899171,10cyclictest3442523-21taskset09:55:263
326228899171,10cyclictest3393362-21taskset09:10:173
34316472161,10sleep03431712-21ls09:45:220
326228899165,6cyclictest3385739-21taskset09:05:123
326228899162,9cyclictest3511540-21taskset11:05:123
326228899161,9cyclictest3317386-21taskset08:00:073
3262288991613,1cyclictest0-21swapper/310:10:173
3262288991612,2cyclictest0-21swapper/308:35:003
3262288991612,2cyclictest0-21swapper/308:15:143
326228899161,12cyclictest0-21swapper/311:00:243
326228899161,12cyclictest0-21swapper/307:10:333
326228899161,11cyclictest3579931-21turbostat12:10:003
326228899161,10cyclictest3526048-21taskset11:15:223
326228899160,13cyclictest0-21swapper/312:30:143
326228899160,13cyclictest0-21swapper/311:10:123
326228899160,13cyclictest0-21swapper/310:40:163
326228899160,13cyclictest0-21swapper/310:15:263
326228899160,13cyclictest0-21swapper/307:55:133
326228899160,12cyclictest0-21swapper/312:20:163
326228899160,12cyclictest0-21swapper/311:50:153
326228899160,12cyclictest0-21swapper/311:20:213
34158842151,4sleep23415934-21ntpq09:30:212
33967412151,9sleep03398025-21grep09:15:160
33330532151,9sleep03333284-21date08:15:000
33226002151,4sleep13322619-21apt-get08:05:111
326228899158,5cyclictest0-21swapper/312:10:133
326228899152,9cyclictest3585240-21sleep12:19:553
326228899151,9cyclictest3343450-21taskset08:23:483
3262288991513,1cyclictest0-21swapper/307:15:013
3262288991512,1cyclictest0-21swapper/309:45:173
3262288991511,2cyclictest0-21swapper/309:35:213
3262288991511,2cyclictest0-21swapper/308:55:173
326228899151,11cyclictest3469385-21sleep10:25:003
326228899150,9cyclictest3537079-21taskset11:25:483
326228899150,13cyclictest0-21swapper/311:40:273
326228899150,13cyclictest0-21swapper/310:50:113
326228899150,10cyclictest3370015-21taskset08:50:133
35754332141,2sleep00-21swapper/012:05:140
34486092141,8sleep03448777-21diskstats10:05:140
34423062141,9sleep23442520-21grep09:55:262
33434582140,1sleep10-21swapper/108:23:551
326228899140,11cyclictest0-21swapper/309:00:143
35426662131,8sleep03542782-21cat11:35:000
36105462121,7sleep03610652-21processes12:35:240
35965042120,8sleep23596575-21/usr/sbin/munin12:25:162
34595692121,7sleep03459723-21/usr/sbin/munin10:15:160
34063982121,7sleep23406547-21latency_hist09:25:002
33690442121,7sleep03369191-21timerwakeupswit08:45:260
33353652121,7sleep13337081-21latency08:15:171
33226222121,7sleep23322736-21cpuspeed_turbos08:05:122
35636252111,5sleep03563649-21munin-run11:55:000
35531792111,5sleep23553228-21munin-run11:45:002
35219852112,4sleep03522004-21apt-get11:15:120
35062652111,5sleep00-21swapper/011:00:130
35007662111,5sleep03500788-21munin-run10:55:000
34908712111,4sleep03491084-21hddtemp_smartct10:45:140
34399242111,6sleep13441719-21kthreadcore09:55:171
34325472111,4sleep13432564-21munin-run09:50:001
34118982111,1sleep13411917-21apt-get09:30:121
33980272111,6sleep13398215-21kthreadcore09:15:171
33490472111,4sleep23349066-21apt-get08:30:132
33338872111,3sleep03335361-21kthreadcore08:15:160
33319612110,1sleep10-21swapper/108:10:201
33228912111,5sleep03323100-21hddtemp_smartct08:05:140
33208452110,2sleep23321293-21strings08:00:192
32955972110,5sleep13295551-31find07:37:461
36069312100,2sleep1171rcu_preempt12:35:171
35960862101,4sleep03596293-21diskstats12:25:140
35521592101,4sleep03552197-21ntpq11:40:210
35429292102,3sleep13542948-21apt-get11:35:131
35429292102,3sleep13542948-21apt-get11:35:131
35376702101,4sleep13537689-21apt-get11:30:121
35374082101,5sleep13537605-21sed11:30:001
35259432101,4sleep13525993-21ntpq11:15:211
35062872101,4sleep13506543-21df11:00:141
34957942100,5sleep13495813-21apt-get10:50:121
34788792101,4sleep23478929-21ntpq10:30:212
34788792101,4sleep23478929-21ntpq10:30:212
34696202101,4sleep23469639-21apt-get10:25:122
34650002101,5sleep13466458-21sort10:20:181
34631762101,5sleep23463235-21ntpq10:15:222
34537162101,4sleep13453975-21df_inode10:10:121
34267572101,4sleep03426974-21users09:40:290
34224142101,5sleep23422660-21expr09:40:152
34223922101,3sleep13422411-21apt-get09:40:141
34172152101,4sleep03417418-21diskstats09:35:120
33909812100,6sleep23391137-21cstates09:10:132
33702922101,6sleep13370502-21idleruntime08:50:161
33280512101,4sleep23327561-21apt-get08:10:012
33214022101,5sleep13321459-21/usr/sbin/munin08:00:221
33069002101,5sleep23307056-21/usr/sbin/munin07:50:132
33010892101,5sleep23301112-21munin-run07:45:002
32955962101,4sleep03295503-31sort07:37:450
32800202101,4sleep23280069-21munin-run07:25:002
32800202101,4sleep23280069-21munin-run07:25:002
32760532101,5sleep10-21swapper/107:20:181
32760532101,5sleep10-21swapper/107:20:181
32674772101,5sleep03267609-21date07:15:000
3606169291,4sleep03606358-21sed12:35:000
3600933291,2sleep13262279-21cyclictest12:29:441
3591278291,4sleep23591349-21irqstats12:20:192
3590861291,3sleep03591067-21diskstats12:20:170
3574905291,3sleep13575161-21df12:05:121
3573608291,3sleep23573656-21ntpq12:00:192
3536441291,3sleep03536490-21ntpq11:25:220
3520683291,4sleep03520732-21sed11:10:190
3506544291,4sleep23506619-21fschecks_time11:00:152
3504959291,4sleep13505001-21ntpq10:55:211
3494494291,3sleep13494545-21ntpq10:45:191
3484104291,4sleep13484159-21ld10:35:211
3479854291,4sleep03479921-21latency_hist10:35:000
3479854291,4sleep03479921-21latency_hist10:35:000
3474858291,3sleep13474877-21apt-get10:30:121
3474858291,3sleep13474877-21apt-get10:30:121
3459160291,4sleep13459178-21apt-get10:15:131
3457620291,4sleep03457667-21ntpq10:10:190
3417629291,3sleep23417700-21irqstats09:35:142
3415991291,4sleep03416106-21gzip09:30:230
3410592291,3sleep23410640-21ntpq09:25:202
3390959291,2sleep10-21swapper/109:10:121
3364781291,3sleep23364800-21apt-get08:45:122
3354605291,4sleep03354761-21head08:35:150
3347848291,3sleep03347901-21ntp_states08:25:220
3344308291,3sleep23344379-21irqstats08:25:162
3338597291,4sleep03338781-21expr08:20:130
3280797291,4sleep13280914-21cut07:25:191
3275027290,4sleep03275046-21apt-get07:20:130
3275027290,4sleep03275046-21apt-get07:20:130
3569642280,2sleep10-21swapper/112:00:021
3564722280,4sleep00-21swapper/011:55:160
3564140281,2sleep20-21swapper/211:55:022
3563332280,1sleep20-21swapper/211:51:182
3562666281,3sleep13562720-21ntpq11:50:241
3558391281,3sleep13558440-21unin-run11:50:001
3553941281,3sleep23554011-21irqstats11:45:162
3531203280,3sleep03531257-21ntpq11:20:210
3501049281,3sleep23501223-21cstates10:55:142
3489327281,3sleep13489378-21ntpq10:40:221
3480114280,3sleep03480133-21apt-get10:35:120
3448487280,2sleep20-21swapper/210:05:122
3448073281,3sleep0800-21snmpd10:03:060
3447134281,3sleep23447222-21ntpq10:00:192
3438593281,3sleep03439923-21tr09:55:160
3433096281,3sleep23433202-21perl09:50:172
3432815280,3sleep03432834-21apt-get09:50:150
3408514281,3sleep03410309-21kthreadcore09:25:180
3385097280,1sleep20-21swapper/209:00:272
3380745280,4sleep03380906-21grep09:00:130
3369993280,3sleep23370012-21apt-get08:50:132
3354054281,3sleep03354102-21munin-run08:35:000
3348782280,3sleep13348833-21munin-run08:30:001
3311629280,1sleep10-21swapper/107:51:561
3280307281,3sleep23280480-21expr07:25:162
3274524281,3sleep03274525-21sshd07:17:170
3269923280,1sleep10-21swapper/107:15:131
3262452280,2sleep10-21swapper/107:10:121
3607006271,3sleep20-21swapper/212:35:182
3594373270,2sleep13594700-21strings12:20:221
3586081271,2sleep10-21swapper/112:15:171
3569155271,3sleep00-21swapper/012:00:010
3565529270,3sleep13567266-21kthreadcore11:55:171
3558044270,1sleep00-21swapper/011:45:340
3548814271,2sleep10-21swapper/111:40:181
3546770271,3sleep03546910-21/usr/sbin/munin11:35:200
3546770271,3sleep03546910-21/usr/sbin/munin11:35:200
3541698271,2sleep20-21swapper/211:30:212
3464412271,2sleep00-21swapper/010:20:130
3458298270,1sleep20-21swapper/210:10:532
3453335270,1sleep10-21swapper/110:08:441
3436797270,2sleep10-21swapper/109:50:241
3427599270,2sleep20-21swapper/209:45:122
3406160270,1sleep10-21swapper/109:21:591
3405420271,2sleep00-21swapper/009:20:210
3399847270,2sleep23400093-21strings09:15:182
3380458270,4sleep10-21swapper/109:00:101
3379231271,2sleep23379268-21ntpq08:55:222
3375220271,3sleep10-21swapper/108:55:131
3374598270,1sleep00-21swapper/008:50:390
3369732271,3sleep13369761-21idleruntime-cro08:50:001
3363832271,3sleep03363935-21/usr/sbin/munin08:40:250
3359570270,2sleep20-21swapper/208:40:112
3359199270,1sleep20-21swapper/208:38:552
3312134270,2sleep00-21swapper/007:55:120
3310713270,1sleep00-21swapper/007:50:190
3306477271,3sleep23306479-21modprobe07:48:172
3300122272,1sleep00-21swapper/007:40:200
3290157270,1sleep20-21swapper/207:31:062
3605154260,2sleep20-21swapper/212:30:212
3589559261,2sleep20-21swapper/212:15:222
3585469260,2sleep00-21swapper/012:15:120
3581117261,2sleep10-21swapper/112:10:161
3580165260,2sleep00-21swapper/012:10:110
3579649260,1sleep20-21swapper/212:06:452
3542920260,1sleep20-21swapper/211:35:062
3542920260,1sleep20-21swapper/211:35:062
3537103260,1sleep20-21swapper/211:26:082
3531823260,1sleep20-21swapper/211:20:512
3531497261,2sleep10-21swapper/111:20:261
3526809260,1sleep20-21swapper/211:18:392
3518541261,2sleep20-21swapper/211:10:172
3515336260,3sleep10-21swapper/111:05:191
3511518260,1sleep20-21swapper/211:05:122
3505449260,3sleep00-21swapper/010:55:280
3496367261,2sleep20-21swapper/210:50:172
3490147260,1sleep20-21swapper/210:42:482
3482159261,2sleep20-21swapper/210:35:182
3448205260,4sleep10-21swapper/110:04:471
3420794260,2sleep13421054-21strings09:35:171
3395633260,1sleep00-21swapper/009:11:320
3390609260,1sleep20-21swapper/209:09:012
3380158260,1sleep00-21swapper/008:59:490
3364307260,1sleep10-21swapper/108:42:191
3354762261,2sleep10-21swapper/108:35:171
3354014260,1sleep10-21swapper/108:34:331
3306445260,1sleep00-21swapper/007:47:510
3300770260,1sleep10-21swapper/107:40:591
3268341261,2sleep20-21swapper/207:15:022
3521704250,3sleep10-21swapper/111:14:451
3491162250,2sleep20-21swapper/210:45:162
3490071250,3sleep00-21swapper/010:41:500
3474595250,2sleep10-21swapper/110:30:001
3410312250,1sleep10-21swapper/109:25:191
3390336250,3sleep00-21swapper/009:05:300
3386011250,1sleep10-21swapper/109:05:151
3343018250,1sleep20-21swapper/208:20:272
3321622250,1sleep00-21swapper/008:00:250
3312738250,2sleep10-21swapper/107:55:171
3305394250,1sleep10-21swapper/107:45:191
3289782250,1sleep00-21swapper/007:30:270
3284940250,3sleep00-21swapper/007:26:190
3601518240,1sleep10-21swapper/112:30:161
3511794240,1sleep00-21swapper/011:05:140
3469896240,1sleep00-21swapper/010:25:140
3312312240,1sleep20-21swapper/207:55:142
3285791240,1sleep10-21swapper/107:30:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional