You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-24 - 14:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Wed Dec 24, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
292872121283,120sleep30-21swapper/307:06:463
292411028554,9sleep10-21swapper/107:05:121
292891028156,19sleep00-21swapper/007:09:110
292867627964,9sleep20-21swapper/207:06:142
31589322690,5sleep13063517-21kworker/1:0+events10:45:111
32227562300,2sleep20-21swapper/211:45:172
32227562300,2sleep20-21swapper/211:45:162
2929098992521,2cyclictest0-21swapper/307:40:013
2929098992219,1cyclictest0-21swapper/307:25:133
2929098992212,7cyclictest3163830-21turbostat10:50:013
2929098992118,1cyclictest0-21swapper/307:24:593
292909899211,15cyclictest0-21swapper/309:35:163
292909899208,8cyclictest2957536-21taskset07:33:223
292909899207,7cyclictest2968101-21taskset07:42:563
2929098992017,1cyclictest0-21swapper/310:29:593
2929098992017,1cyclictest0-21swapper/309:25:133
2929098992016,2cyclictest0-21swapper/312:39:593
2929098991917,1cyclictest0-21swapper/309:10:123
2929098991917,1cyclictest0-21swapper/308:05:123
2929098991916,1cyclictest0-21swapper/311:45:283
2929098991916,1cyclictest0-21swapper/311:45:273
2929098991916,1cyclictest0-21swapper/310:00:193
2929098991915,2cyclictest0-21swapper/310:24:593
292909899191,11cyclictest3153189-21taskset10:40:003
29848112181,5sleep02984914-21perl08:00:150
292909899182,11cyclictest3262875-21taskset12:20:203
2929098991815,1cyclictest0-21swapper/311:35:163
2929098991815,1cyclictest0-21swapper/310:50:173
2929098991815,1cyclictest0-21swapper/309:40:173
2929098991815,1cyclictest0-21swapper/307:45:173
292909899181,4cyclictest3100427-21sleep09:50:013
2929098991814,2cyclictest0-21swapper/312:10:263
2929098991814,2cyclictest0-21swapper/311:29:593
2929098991814,2cyclictest0-21swapper/311:15:173
292909899181,13cyclictest3174615-21chrt11:00:123
292909899181,12cyclictest3005505-21sleep08:19:593
32684792171,10sleep03268664-21users12:25:270
31693962171,4sleep03169599-21diskstats10:55:130
292909899178,5cyclictest2978775-21turbostat07:55:003
2929098991714,1cyclictest0-21swapper/311:40:123
2929098991714,1cyclictest0-21swapper/311:30:183
2929098991713,2cyclictest0-21swapper/310:40:183
2929098991713,2cyclictest0-21swapper/310:40:173
2929098991713,2cyclictest0-21swapper/310:30:253
292909899171,13cyclictest0-21swapper/310:10:233
292909899171,13cyclictest0-21swapper/308:25:123
292909899171,12cyclictest2999967-21sleep08:15:003
292909899171,11cyclictest0-21swapper/311:10:123
292909899171,10cyclictest3016231-21chrt08:30:033
29298372161,10sleep22930749-21tr07:10:132
292909899166,7cyclictest3111149-21turbostat10:00:013
292909899161,9cyclictest3179812-21taskset11:05:023
292909899161,9cyclictest3116661-21taskset10:05:013
292909899161,9cyclictest3037465-21taskset08:50:073
2929098991614,1cyclictest0-21swapper/307:15:173
2929098991612,2cyclictest0-21swapper/312:05:183
292909899161,12cyclictest3068688-21turbostat09:20:003
292909899161,12cyclictest0-21swapper/311:50:103
292909899161,12cyclictest0-21swapper/309:30:223
292909899161,12cyclictest0-21swapper/307:55:123
292909899161,11cyclictest3274326-21chrt12:35:003
292909899161,10cyclictest2934118-21taskset07:13:023
292909899160,9cyclictest3053251-21taskset09:05:013
292909899160,13cyclictest0-21swapper/309:20:223
292909899160,13cyclictest0-21swapper/308:40:163
292909899160,13cyclictest0-21swapper/308:35:153
30518772151,9sleep23051998-21ntpq09:00:202
29635832151,10sleep02963757-21expr07:40:140
292909899151,9cyclictest3047382-21taskset08:55:393
2929098991512,1cyclictest0-21swapper/312:15:113
2929098991512,1cyclictest0-21swapper/312:00:163
2929098991512,1cyclictest0-21swapper/312:00:013
292909899151,10cyclictest3008107-21chrt08:20:173
292909899150,13cyclictest0-21swapper/310:15:223
292909899150,13cyclictest0-21swapper/309:00:153
292909899150,12cyclictest0-21swapper/310:55:233
292909899150,12cyclictest0-21swapper/309:50:013
292909899150,11cyclictest2984551-21taskset08:00:123
292909899150,11cyclictest0-21swapper/311:20:113
31210822140,9sleep23121274-21users10:05:252
30809632141,8sleep23082655-21tr09:30:182
30634522141,9sleep03063682-21cat09:15:010
29883602141,9sleep12988501-21ntpq08:00:201
29628852140,9sleep02962839-31find07:38:330
292909899141,9cyclictest3264357-21taskset12:25:133
292909899141,8cyclictest3037209-21taskset08:49:583
32705922131,8sleep13272280-21tr12:30:181
32575432131,7sleep23257593-21ntpq12:15:192
31660342131,7sleep23167816-21kthreadcore10:50:182
31523472131,8sleep23152450-21timerandwakeup10:35:272
31361962131,2sleep10-21swapper/110:20:201
30966282131,8sleep00-21swapper/009:45:150
30327922130,8sleep13033431-21tr08:45:161
29903282130,8sleep12990963-21cut08:05:161
29539252131,8sleep22955611-21cut07:30:172
32785862120,1sleep10-21swapper/112:35:221
32593222121,7sleep13259392-21irqstats12:20:161
31259002121,6sleep13125947-21ntpq10:10:221
31009142121,4sleep23100406-21apt-get09:50:022
30890842121,3sleep13089171-21processes09:35:271
30704712121,7sleep03072192-21grep09:20:180
30267662121,6sleep23026797-21munin-run08:40:002
31856572111,6sleep03185690-21kthreadcore11:10:170
30482952111,6sleep13048396-21fschecks_time09:00:151
29685292112,4sleep12968548-21apt-get07:45:111
29628122111,6sleep22962842-40gst-plugin-scan07:38:272
29622052110,7sleep12962257-21cat07:35:261
29565142111,5sleep02956655-21ntpq07:30:200
29426802111,6sleep02943054-21kthreadcore07:20:170
29331772111,5sleep12933266-21ntpq07:10:161
32722812101,5sleep23273151-21cat12:30:192
32646252100,6sleep13265066-21kthreadcore12:25:161
32365202101,3sleep03236568-21pool-udisksd11:55:210
32325922101,4sleep13232848-21df11:55:141
32313612101,4sleep03231415-21ntpq11:50:200
32209232101,4sleep23220976-21ntp_states11:40:212
31588472101,2sleep0171rcu_preempt10:45:110
31573872101,5sleep03157459-21grep10:40:220
31573872101,5sleep03157459-21grep10:40:220
31572842101,4sleep23157331-21ntpq10:40:212
31572842101,4sleep23157331-21ntpq10:40:202
31166942100,6sleep03116893-21cstates10:05:110
31105492101,4sleep23110766-21users09:55:272
30739222101,5sleep13074119-21find09:25:001
30642942101,5sleep03065747-21ps09:15:180
30572622101,4sleep03057312-21ntpq09:05:200
30312932101,5sleep03031552-21/usr/sbin/munin08:40:260
30202822101,4sleep03020335-21ntp_states08:30:240
29909642101,5sleep22992653-21grep08:05:172
29807792101,5sleep12982554-21cut07:55:181
29514622101,5sleep22951511-21ntpq07:25:202
29430572101,4sleep12944708-21cut07:20:181
3265070290,6sleep23266718-21kthreadcore12:25:172
3262819291,3sleep23262867-21ntpq12:20:212
3241742291,3sleep13241790-21ntpq12:00:191
3227646291,4sleep13227860-21smartctl11:50:131
3216869291,4sleep13190100-21kworker/1:0+events11:40:121
3212104291,4sleep03213732-21cut11:35:170
3204838291,3sleep23204894-21pool-udisksd11:25:212
3196183291,4sleep23197615-21cut11:20:172
3183808291,4sleep03183854-21ntpq11:05:190
3174593291,3sleep20-21swapper/211:00:122
3172736291,4sleep13173240-21grep10:55:181
3153317291,4sleep13153325-21/usr/sbin/munin10:40:121
3153317291,4sleep13153325-21/usr/sbin/munin10:40:111
3152054291,3sleep13152105-21ntpq10:35:221
3148039291,3sleep00-21swapper/010:35:130
3142805291,3sleep13142824-21apt-get10:30:141
3132380291,3sleep23132399-21apt-get10:20:132
3111701291,4sleep23111772-21fschecks_time10:00:162
3110197291,4sleep13110250-21ntp_states09:55:221
3100934291,3sleep03100953-21apt-get09:50:130
3100442291,4sleep1248-21systemd-journal09:50:011
3099484290,1sleep10-21swapper/109:45:201
3090700291,4sleep13090816-21irqstats09:40:151
3084972291,3sleep23084991-21apt-get09:35:162
3074459291,3sleep13074478-21apt-get09:25:131
3043258291,4sleep13043814-21cut08:55:181
3019995290,1sleep10-21swapper/108:30:211
2989517291,5sleep20-21swapper/208:05:022
2979037291,5sleep00-21swapper/007:55:130
2974069291,4sleep12974193-21perl07:50:141
2963863292,3sleep20-21swapper/207:40:162
2951275290,1sleep00-21swapper/007:25:180
2947438290,4sleep12947457-21apt-get07:25:111
2946491290,1sleep20-21swapper/207:20:272
2937494291,4sleep02939176-21grep07:15:160
2936863290,6sleep22937075-21if_enp1s007:15:152
2934326291,3sleep12934819-21ntpq07:15:011
3273565280,1sleep00-21swapper/012:30:250
3232849280,1sleep20-21swapper/211:55:162
3222120280,2sleep10-21swapper/111:45:131
3222120280,2sleep10-21swapper/111:45:121
3195001280,1sleep00-21swapper/011:16:010
3169764281,3sleep23169879-21irqstats10:55:152
3143801281,3sleep20-21swapper/210:30:192
3128254281,3sleep03129930-21kthreadcore10:15:170
3111970281,3sleep03112603-21kthreadcore10:00:190
3110148283,2sleep00-21swapper/009:55:210
3094187281,3sleep23094235-21ntpq09:40:202
3089281281,4sleep00-21swapper/009:35:290
3049406283,2sleep03051094-21kthreadcore09:00:180
3046757281,3sleep23046805-21ntpq08:55:232
3042369280,1sleep20-21swapper/208:54:062
3042314281,3sleep13042315-21sshd08:53:251
3027065281,3sleep23027094-21grep08:40:132
3005750281,2sleep13005769-21apt-get08:20:121
3004581281,3sleep23004702-21gzip08:15:232
2999939281,3sleep03000095-21systemd08:15:010
2996474281,3sleep22998198-21kthreadcore08:10:192
2983843280,1sleep20-21swapper/207:58:272
2973339280,1sleep00-21swapper/007:47:390
3275706271,2sleep00-21swapper/012:35:170
3263658270,1sleep00-21swapper/012:23:000
3258208270,1sleep00-21swapper/012:15:490
3252613271,3sleep20-21swapper/212:10:262
3227352270,2sleep20-21swapper/211:50:102
3226803270,1sleep00-21swapper/011:46:190
3226803270,1sleep00-21swapper/011:46:190
3210838270,1sleep10-21swapper/111:32:351
3210025271,2sleep23210067-21ntpq11:30:232
3205659270,1sleep10-21swapper/111:28:251
3205653270,1sleep00-21swapper/011:28:200
3195601270,2sleep10-21swapper/111:20:121
3186812271,2sleep10-21swapper/111:10:181
3184690270,1sleep20-21swapper/211:08:222
3169026270,1sleep00-21swapper/010:54:460
3142799270,1sleep00-21swapper/010:30:080
3139642270,3sleep1321rcuc/110:25:161
3138193271,3sleep03139640-21kthreadcore10:25:150
3137316270,1sleep00-21swapper/010:24:360
3132028270,1sleep20-21swapper/210:19:122
3123326270,3sleep23125037-21cut10:10:182
3118828271,2sleep10-21swapper/110:05:161
3099254270,2sleep20-21swapper/209:45:182
3084694270,3sleep03084695-21sshd09:34:530
3084460271,4sleep1800-21snmpd09:31:531
3078499270,2sleep20-21swapper/209:25:222
3038044271,2sleep00-21swapper/008:50:160
3030879270,3sleep13030968-21ntpq08:40:191
3025895271,2sleep10-21swapper/108:35:221
3016261270,2sleep20-21swapper/208:30:152
3011007271,2sleep10-21swapper/108:25:121
3005772271,2sleep23005928-21cstates08:20:132
3005408270,1sleep10-21swapper/108:19:181
2973224270,1sleep20-21swapper/207:46:132
3279385260,4sleep20-21swapper/212:38:322
3253560261,1sleep10-21swapper/112:15:101
3249104261,2sleep10-21swapper/112:10:181
3248329260,1sleep00-21swapper/012:10:130
3247829260,1sleep20-21swapper/212:06:582
3247718260,4sleep00-21swapper/012:05:310
3243049260,1sleep10-21swapper/112:05:121
3242561260,4sleep20-21swapper/212:02:092
3242461260,1sleep00-21swapper/012:00:510
3211593261,2sleep20-21swapper/211:35:152
3206345260,3sleep0221rcuc/011:30:170
3195600260,4sleep00-21swapper/011:20:110
3195122260,1sleep10-21swapper/111:17:311
3188558260,2sleep23188983-21strings11:10:182
3184649260,1sleep10-21swapper/111:07:491
3178402260,2sleep13178515-21strings11:00:191
3168704260,4sleep10-21swapper/110:50:371
3163755260,1sleep20-21swapper/210:49:192
3142208260,1sleep20-21swapper/210:25:492
3131927260,4sleep10-21swapper/110:17:541
3067717261,1sleep10-21swapper/109:15:231
3063282260,1sleep20-21swapper/209:12:542
3037001260,1sleep00-21swapper/008:47:240
3015923260,4sleep00-21swapper/008:29:190
3015915260,4sleep20-21swapper/208:29:132
3005191260,2sleep0800-21snmpd08:16:330
2999918260,4sleep10-21swapper/108:14:461
2978683260,1sleep20-21swapper/207:54:062
2978065261,2sleep00-21swapper/007:50:250
2967996260,1sleep10-21swapper/107:41:361
2953253261,2sleep10-21swapper/107:30:161
2933974260,1sleep00-21swapper/007:11:120
3216847250,1sleep00-21swapper/011:40:100
3215982250,3sleep10-21swapper/111:36:061
3190851250,2sleep20-21swapper/211:15:152
3126631250,1sleep00-21swapper/010:11:330
3116125250,3sleep10-21swapper/110:01:241
3094970251,2sleep00-21swapper/009:42:180
3078313250,1sleep00-21swapper/009:25:190
3060548250,2sleep10-21swapper/109:10:171
3058093250,3sleep20-21swapper/209:08:132
3057544250,1sleep10-21swapper/109:05:251
3046570250,1sleep00-21swapper/008:55:220
3033432250,2sleep23035098-21cut08:45:172
3010218250,2sleep03010383-21vmstat08:20:260
2994452250,1sleep00-21swapper/008:06:310
3174819240,1sleep00-21swapper/011:00:150
3068950240,1sleep20-21swapper/209:20:122
3064002240,1sleep20-21swapper/209:15:162
3021264240,1sleep00-21swapper/008:35:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional