You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-26 - 07:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Fri Dec 26, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3757828212365,54sleep30-21swapper/319:07:233
37579022122114,5sleep10-21swapper/119:08:181
375773328871,10sleep20-21swapper/219:06:102
375785127456,12sleep00-21swapper/019:07:370
40187562510,4sleep244-21ksoftirqd/223:15:012
37751482510,2sleep1321rcuc/119:20:241
40018852280,5sleep1321rcuc/122:55:281
3758170992822,3cyclictest3897543-21turbostat21:20:003
3758170992623,1cyclictest0-21swapper/320:20:003
40413002230,4sleep2421rcuc/223:35:172
3758170992320,1cyclictest0-21swapper/321:10:293
375817099227,4cyclictest4055307-21sleep23:50:013
3758170992219,1cyclictest0-21swapper/300:20:003
375817099211,9cyclictest3960720-21turbostat22:20:003
40447732206,9sleep04044842-21systemctl23:40:010
38176122201,8sleep13817666-21ld20:00:201
375817099204,4cyclictest3986932-21turbostat22:45:013
3758170992017,1cyclictest0-21swapper/320:30:193
3758170992017,1cyclictest0-21swapper/319:15:173
3758170992017,1cyclictest0-21swapper/300:25:153
375817099201,15cyclictest3898140-21chrt21:20:133
375817099201,14cyclictest3950471-21chrt22:10:123
375817099200,10cyclictest0-21swapper/319:27:163
40561062191,14sleep04057018-21cut23:50:190
39400032191,4sleep13940262-21df_inode22:00:131
38407582191,5sleep00-21swapper/020:25:170
375817099192,10cyclictest4013395-21taskset23:10:003
375817099192,10cyclictest3916913-21taskset21:35:173
3758170991916,1cyclictest0-21swapper/323:35:173
3758170991915,2cyclictest0-21swapper/319:45:123
375817099191,14cyclictest4018630-21turbostat23:15:003
375817099191,14cyclictest0-21swapper/323:15:123
375817099191,14cyclictest0-21swapper/320:20:113
375817099191,14cyclictest0-21swapper/319:40:133
375817099191,12cyclictest3887096-21taskset21:09:563
375817099191,12cyclictest3887096-21taskset21:09:563
39018182181,11sleep13901867-21ntpq21:20:211
375817099188,6cyclictest3991931-21taskset22:47:433
3758170991815,1cyclictest0-21swapper/323:55:263
3758170991814,2cyclictest0-21swapper/322:35:193
3758170991814,2cyclictest0-21swapper/319:10:263
375817099181,13cyclictest0-21swapper/319:30:193
375817099180,14cyclictest0-21swapper/320:55:233
40437692171,11sleep14043812-21ntpq23:35:211
39716992171,12sleep03971819-21tune2fs22:30:140
375817099174,9cyclictest3904999-21taskset21:25:193
375817099171,9cyclictest3824934-21taskset20:10:153
3758170991714,1cyclictest0-21swapper/321:55:173
3758170991714,1cyclictest0-21swapper/321:45:183
3758170991714,1cyclictest0-21swapper/320:40:133
375817099171,13cyclictest0-21swapper/322:05:133
375817099171,13cyclictest0-21swapper/320:45:163
375817099171,13cyclictest0-21swapper/319:35:133
375817099171,13cyclictest0-21swapper/300:20:213
375817099171,11cyclictest3908307-21taskset21:30:113
375817099171,10cyclictest3845112-21taskset20:29:003
3758170991710,4cyclictest0-21swapper/300:05:163
375817099170,14cyclictest0-21swapper/323:20:143
375817099170,13cyclictest0-21swapper/321:50:203
375817099170,13cyclictest0-21swapper/300:00:203
40493752161,10sleep24045057-21/usr/sbin/munin23:40:262
39387242161,10sleep23938773-21ntpq21:55:212
38460082161,6sleep03846062-21kthreadcore20:30:180
37909142161,9sleep13791114-21users19:35:281
375817099162,9cyclictest3976228-21taskset22:33:013
375817099161,9cyclictest4081220-21taskset00:10:283
375817099161,9cyclictest4039242-21taskset23:31:333
375817099161,9cyclictest3997170-21taskset22:53:053
3758170991613,1cyclictest0-21swapper/321:00:233
3758170991612,2cyclictest0-21swapper/320:55:003
375817099161,11cyclictest4103079-21chrt00:35:143
3758170991610,3cyclictest0-21swapper/323:40:123
375817099160,13cyclictest0-21swapper/323:25:153
375817099160,13cyclictest0-21swapper/320:05:193
375817099160,13cyclictest0-21swapper/319:55:233
41029632150,1sleep20-21swapper/200:35:022
40552752151,9sleep24055299-21perf23:50:012
40152222151,9sleep14016934-21grep23:10:201
39609892151,10sleep13961148-21cstates22:20:131
39279142151,4sleep03927955-21ntpq21:45:200
38725372151,9sleep03874164-21grep20:55:180
37761922151,9sleep13776394-21diskstats19:25:141
375817099151,9cyclictest4102775-21taskset00:34:463
375817099151,9cyclictest4002340-21taskset22:57:333
375817099151,9cyclictest3965567-21taskset22:20:343
375817099151,9cyclictest3923679-21taskset21:44:423
375817099151,9cyclictest3813639-21taskset20:00:063
3758170991511,2cyclictest0-21swapper/319:50:213
375817099150,13cyclictest0-21swapper/322:00:263
375817099150,12cyclictest0-21swapper/320:35:223
375817099150,11cyclictest0-21swapper/323:00:153
375817099150,11cyclictest0-21swapper/319:20:223
375817099141,8cyclictest3971139-21taskset22:29:523
40028422131,8sleep24002821-21/usr/sbin/munin23:00:152
375817099138,3cyclictest4055538-21chrt23:50:133
39871852121,7sleep13987164-21/usr/sbin/munin22:45:151
38940542121,7sleep13895769-21cut21:15:201
38611952120,8sleep23861214-21apt-get20:45:152
40966232111,5sleep04096673-21ntpq00:25:210
39352472111,6sleep13935674-21tr21:55:151
39033422111,5sleep03903551-21hddtemp_smartct21:25:170
38983512111,6sleep03898735-21kthreadcore21:20:160
38363542111,6sleep13838066-21kthreadcore20:20:171
38301682111,7sleep03830268-21sed20:15:190
41025902100,1sleep10-21swapper/100:32:281
40766412101,4sleep04076900-21df_inode00:10:150
40708472100,1sleep00-21swapper/000:02:090
40598242101,5sleep24060028-21sed23:50:282
40342672101,4sleep24034318-21munin-run23:30:012
40107842101,4sleep14011912-21memory23:05:191
40028202101,4sleep14002839-21apt-get23:00:141
39925412101,4sleep13992664-21diskstats22:50:161
39858792101,4sleep23985919-21ntpq22:40:192
39766482101,4sleep13976667-21apt-get22:35:121
39648722101,4sleep03964929-21ntpq22:20:200
39151712102,3sleep23915541-21kthreadcore21:35:172
39139952101,4sleep03914064-21irqstats21:35:140
39122092101,4sleep23912255-21ntpq21:30:202
38966222101,5sleep23896703-21awk21:15:252
38913042101,4sleep13891342-21ntpq21:10:231
38768962101,4sleep13876915-21apt-get21:00:131
38679842101,5sleep03869689-21grep20:50:180
38598872104,1sleep00-21swapper/020:40:200
38285102101,2sleep20-21swapper/220:10:232
38081102101,5sleep03808254-21/usr/sbin/munin19:55:110
38075552101,6sleep2800-21snmpd19:51:312
37858452100,1sleep10-21swapper/119:30:271
4097844291,3sleep04097863-21apt-get00:30:110
4097585291,4sleep24097806-21sed00:30:012
4091330291,4sleep04091383-21ld00:20:210
4082092291,3sleep24081599-21apt-get00:15:022
4049081290,1sleep10-21swapper/123:40:211
4038507290,3sleep04038550-21ntpq23:30:220
4023834291,4sleep04023854-21latency_hist23:20:010
4013071291,4sleep24013394-21gpgconf23:10:012
4009134291,4sleep04010783-21cut23:05:180
4003206291,4sleep04003376-21cat23:00:160
3997275291,3sleep03997278-21sshd22:54:270
3976670291,4sleep23976926-21df22:35:122
3971150291,4sleep2248-21systemd-journal22:29:592
3950983291,4sleep23951303-21tr22:10:162
3928914291,4sleep23929163-21apt-key21:50:002
3918754291,3sleep03918773-21apt-get21:40:120
3903624291,4sleep23904998-21cut21:25:192
3898093291,4sleep23898139-21fschecks_count21:20:132
3875693291,4sleep13875743-21open_inodes20:55:231
3866011290,1sleep00-21swapper/020:48:210
3854721291,3sleep13854766-21ntpq20:35:241
3845491291,4sleep13845742-21df_inode20:30:151
3843164290,2sleep20-21swapper/220:25:192
3839935291,4sleep03840077-21latency_hist20:25:010
3823201290,1sleep10-21swapper/120:05:261
3802818291,3sleep13802837-21apt-get19:50:101
3797262291,2sleep20-21swapper/219:45:032
3796580290,1sleep00-21swapper/019:42:590
3795741291,3sleep23795789-21ntpq19:40:222
3790791290,5sleep03790913-21sed19:35:270
3785261291,3sleep23785309-21ntpq19:30:202
3767677290,1sleep00-21swapper/019:15:190
4106492281,3sleep14106977-21needreboot00:35:201
4102521280,1sleep20-21swapper/200:31:332
4082125280,3sleep14082144-21apt-get00:15:131
4075314281,4sleep24075368-21ld00:05:242
4071506281,3sleep00-21swapper/000:05:180
4060805280,1sleep20-21swapper/223:55:022
4057019281,3sleep14058734-21cut23:50:201
4054693281,3sleep04054910-21users23:45:300
4035825281,3sleep10-21swapper/123:30:191
4029043281,3sleep24029044-21sshd23:24:282
4028105280,4sleep00-21swapper/023:20:200
4022843281,3sleep14022895-21ld23:15:211
3997584281,2sleep23997603-21apt-get22:55:142
3955705281,3sleep23955791-21tr22:15:132
3945215281,2sleep00-21swapper/022:05:120
3934439281,3sleep13934530-21latency_hist21:55:001
3930002281,3sleep23930570-21kthreadcore21:50:162
3925588281,3sleep13927308-21kthreadcore21:45:181
3886903280,1sleep20-21swapper/221:07:322
3886903280,1sleep20-21swapper/221:07:312
3880711281,3sleep03880851-21/usr/sbin/munin21:00:200
3856532281,3sleep13857843-21grep20:40:171
3841477281,3sleep13843163-21grep20:25:181
3802793280,3sleep03802299-21apt-get19:50:020
3780172281,3sleep23780284-21ps19:25:232
3776128281,4sleep03776191-21cpuspeed_turbos19:25:130
3774831281,2sleep03774877-21ntpq19:20:190
3762690281,3sleep00-21swapper/019:10:260
4104762270,3sleep04106490-21kthreadcore00:35:190
4091384270,3sleep14091435-21ntpq00:20:221
4069916272,2sleep20-21swapper/200:00:182
4060818270,2sleep04060837-21apt-get23:55:120
4049743270,1sleep00-21swapper/023:41:250
4037514270,3sleep2421rcuc/223:30:212
3987163270,2sleep03987182-21apt-get22:45:140
3981361270,4sleep00-21swapper/022:36:280
3975662271,3sleep23975674-21run-parts22:30:252
3967416271,2sleep00-21swapper/022:25:170
3959910270,4sleep10-21swapper/122:15:231
3955329270,1sleep00-21swapper/022:13:510
3950170270,1sleep20-21swapper/222:09:482
3939982270,3sleep03940000-21apt-get22:00:120
3935026271,3sleep03935234-21if_enp1s021:55:140
3923461270,1sleep20-21swapper/221:41:592
3914065271,2sleep10-21swapper/121:35:161
3897317270,1sleep00-21swapper/021:17:200
3885520271,3sleep10-21swapper/121:05:181
3885520271,3sleep10-21swapper/121:05:171
3880897270,4sleep20-21swapper/221:00:222
3839614270,1sleep20-21swapper/220:20:572
3834601270,1sleep20-21swapper/220:19:222
3826396271,2sleep10-21swapper/120:10:191
3823801270,1sleep20-21swapper/220:08:522
3819749271,3sleep03821370-21cut20:05:170
3818588270,1sleep00-21swapper/020:04:450
3813056270,1sleep20-21swapper/219:59:382
3786062270,1sleep00-21swapper/019:32:250
3762989273,2sleep20-21swapper/219:10:562
4097344260,4sleep10-21swapper/100:26:591
4087569260,2sleep20-21swapper/200:20:152
4082451260,3sleep00-21swapper/000:15:160
4081325260,1sleep20-21swapper/200:11:412
4076248260,1sleep10-21swapper/100:08:401
4070382260,2sleep10-21swapper/100:00:261
4061906261,2sleep10-21swapper/123:55:171
4034045260,1sleep00-21swapper/023:27:140
4026252261,2sleep10-21swapper/123:20:171
4023626260,1sleep20-21swapper/223:17:262
4018275260,1sleep00-21swapper/023:10:380
3995982260,2sleep23996273-21strings22:50:212
3989264261,2sleep20-21swapper/222:45:192
3986822260,4sleep10-21swapper/122:44:021
3966164260,4sleep10-21swapper/122:25:011
3946102261,2sleep10-21swapper/122:05:171
3923412260,1sleep10-21swapper/121:41:211
3906979260,2sleep10-21swapper/121:25:221
3889263261,2sleep20-21swapper/221:10:192
3887371260,1sleep00-21swapper/021:10:140
3883778260,2sleep00-21swapper/021:05:170
3883778260,2sleep00-21swapper/021:05:170
3876415260,1sleep20-21swapper/220:57:342
3871067260,4sleep10-21swapper/120:51:071
3870248260,1sleep20-21swapper/220:50:202
3864338260,2sleep10-21swapper/120:45:211
3857845261,2sleep20-21swapper/220:40:182
3854097261,2sleep20-21swapper/220:35:212
3851274260,3sleep03852349-21kthreadcore20:35:180
3849754261,2sleep20-21swapper/220:30:282
3829681261,2sleep10-21swapper/120:15:161
3825004261,2sleep00-21swapper/020:10:180
3813614260,2sleep20-21swapper/220:00:012
3801120260,2sleep13801233-21strings19:45:181
3791767260,2sleep10-21swapper/119:40:121
3790461260,1sleep20-21swapper/219:35:212
3771131261,2sleep20-21swapper/219:20:142
3770452260,5sleep10-21swapper/119:18:321
3765545260,2sleep20-21swapper/219:15:122
4079423250,1sleep10-21swapper/100:10:201
4001588250,1sleep00-21swapper/022:55:240
3986831250,3sleep00-21swapper/022:44:090
3960683250,1sleep00-21swapper/022:20:000
3954926250,1sleep10-21swapper/122:10:271
3942709250,1sleep20-21swapper/222:00:182
3932242250,2sleep00-21swapper/021:50:180
3912361250,2sleep10-21swapper/121:30:231
3812142250,1sleep10-21swapper/119:55:201
3796739250,1sleep00-21swapper/019:45:010
3758331250,1sleep10-21swapper/119:10:111
4050572240,1sleep10-21swapper/123:45:151
4029529240,1sleep10-21swapper/123:25:151
3971412240,1sleep10-21swapper/122:30:111
3961150240,1sleep20-21swapper/222:20:142
3908557240,1sleep00-21swapper/021:30:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional