You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-08 - 15:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Thu Jan 08, 2026 12:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2806437211957,57sleep30-21swapper/307:05:073
281122428360,6sleep00-21swapper/007:08:420
281113728065,9sleep10-21swapper/107:07:331
281111827964,9sleep20-21swapper/207:07:182
30076232540,2sleep00-21swapper/010:20:150
2811458992724,1cyclictest0-21swapper/308:20:003
2811458992320,1cyclictest0-21swapper/312:10:013
281145899214,13cyclictest3042429-21taskset10:55:013
2811458992118,1cyclictest0-21swapper/310:50:193
2811458992118,1cyclictest0-21swapper/308:55:003
281145899201,9cyclictest2865859-21taskset07:55:213
2811458992017,1cyclictest0-21swapper/312:15:173
2811458992017,1cyclictest0-21swapper/311:30:003
2811458992017,1cyclictest0-21swapper/311:15:183
281145899201,14cyclictest0-21swapper/311:45:113
2811458991916,1cyclictest0-21swapper/312:35:173
2811458991916,1cyclictest0-21swapper/310:10:173
2811458991915,2cyclictest0-21swapper/310:45:233
2811458991915,2cyclictest0-21swapper/308:25:153
281145899191,14cyclictest2978322-21turbostat09:50:003
281145899191,14cyclictest2956483-21chrt09:25:383
281145899191,14cyclictest0-21swapper/308:00:143
281145899191,13cyclictest0-21swapper/311:50:103
281145899190,12cyclictest0-21swapper/308:35:183
281145899187,8cyclictest2984177-21turbostat10:04:553
2811458991816,1cyclictest0-21swapper/309:05:163
2811458991816,1cyclictest0-21swapper/309:05:003
2811458991815,1cyclictest0-21swapper/310:35:143
2811458991814,2cyclictest0-21swapper/309:40:003
281145899181,13cyclictest0-21swapper/312:25:103
281145899181,10cyclictest2821529-21taskset07:15:163
30632162171,10sleep13063427-21users11:15:241
29307912171,5sleep12930810-21apt-get09:05:091
28821212171,5sleep12882162-21ntpq08:15:201
281145899172,11cyclictest2860251-21taskset07:50:203
281145899171,5cyclictest3085375-21sleep11:40:013
2811458991715,1cyclictest0-21swapper/311:05:133
2811458991714,1cyclictest0-21swapper/311:40:153
2811458991714,1cyclictest0-21swapper/311:24:593
2811458991714,1cyclictest0-21swapper/307:45:163
2811458991713,2cyclictest0-21swapper/309:10:203
281145899171,13cyclictest0-21swapper/309:40:213
281145899171,13cyclictest0-21swapper/307:25:113
281145899171,12cyclictest3106930-21chrt12:00:113
281145899171,12cyclictest2914893-21turbostat08:50:003
281145899171,12cyclictest0-21swapper/307:40:113
30192292161,10sleep23020079-21sort10:30:162
28988632161,10sleep22899105-21base6408:35:002
281145899167,6cyclictest2872664-21turbostat08:10:003
281145899162,9cyclictest3117929-21taskset12:10:133
2811458991613,1cyclictest0-21swapper/312:30:163
2811458991613,1cyclictest0-21swapper/309:30:163
2811458991612,2cyclictest0-21swapper/310:30:003
281145899161,11cyclictest2984199-21sleep10:00:003
281145899161,10cyclictest2888315-21chrt08:24:143
281145899160,14cyclictest0-21swapper/310:15:113
281145899160,13cyclictest0-21swapper/310:30:103
281145899160,13cyclictest0-21swapper/309:20:313
281145899160,13cyclictest0-21swapper/308:55:213
281145899160,13cyclictest0-21swapper/307:30:173
31419402151,10sleep23142855-21missed_timers12:30:172
28285592151,9sleep22824528-21/usr/sbin/munin07:20:222
281145899152,9cyclictest3071162-21taskset11:25:153
281145899152,9cyclictest3005792-21taskset10:20:103
281145899151,9cyclictest2994663-21taskset10:09:203
281145899151,9cyclictest2994663-21taskset10:09:193
281145899151,9cyclictest2877591-21taskset08:10:593
281145899151,9cyclictest2829358-21taskset07:23:153
281145899151,9cyclictest2816442-21taskset07:12:203
281145899151,11cyclictest0-21swapper/310:40:183
2811458991511,1cyclictest0-21swapper/309:50:133
281145899151,10cyclictest3058318-21chrt11:12:093
281145899151,10cyclictest2840788-21chrt07:35:123
281145899150,13cyclictest0-21swapper/312:20:123
281145899150,12cyclictest0-21swapper/311:55:163
281145899150,12cyclictest0-21swapper/311:00:153
281145899150,12cyclictest0-21swapper/309:15:153
30696312141,9sleep13069642-21/usr/sbin/munin11:25:101
29505782140,7sleep02950630-21/usr/sbin/munin09:20:200
29199172140,1sleep00-21swapper/008:52:550
283977621410,2sleep22746992-21kworker/2:0+events_freezable07:32:172
281145899141,10cyclictest2893887-21chrt08:30:113
281145899140,11cyclictest0-21swapper/308:40:183
30842992131,7sleep03084350-21ntpq11:35:190
30824352130,9sleep23084136-21sed11:35:182
29057502131,7sleep02907435-21kthreadcore08:40:160
28658112138,2sleep20-21swapper/207:55:212
30696432121,7sleep23069680-21cpu11:25:112
30575272121,7sleep03057575-21ntpq11:10:180
30470962120,7sleep03047150-21ntpq11:00:160
30325192121,6sleep03032555-21kthreadcore10:45:130
29852542121,7sleep02986907-21kthreadcore10:00:110
29209202120,7sleep22922160-21grep08:55:172
28891722121,7sleep22889363-21kthreadcore08:25:152
28405332121,6sleep12840599-21tune2fs07:35:121
31216252111,5sleep13121671-21ntpq12:10:201
30643882111,2sleep00-21swapper/011:20:120
30335472111,6sleep13035273-21kthreadcore10:45:141
29999752111,5sleep12999994-21munin-run10:15:001
29308132111,6sleep22931063-21cstates09:05:102
29138302111,4sleep12913878-21ntpq08:45:181
29087252111,5sleep12908781-21ntp_states08:40:211
28768882111,4sleep22876934-21ntpq08:10:152
28767472111,5sleep12876887-21ntpq08:10:141
28192062112,4sleep12819225-21apt-get07:15:121
28156542111,5sleep22815703-21ntpq07:10:202
31440162101,5sleep13144018-21cron12:35:011
30809482101,6sleep13082434-21kthreadcore11:35:171
30754272101,4sleep23075636-21hddtemp_smartct11:30:122
30414582101,5sleep03041462-21mailstats10:50:210
30412102101,4sleep13041259-21ntpq10:50:181
30254402101,4sleep03025481-21ntpq10:35:190
30062702101,4sleep23006340-21irqstats10:20:142
30007362101,5sleep13000814-21uname10:15:121
29681512101,4sleep12968367-21diskstats09:40:131
29360772100,6sleep22936221-21/usr/sbin/munin09:10:122
29034412101,4sleep12903495-21ntpq08:35:201
28781912101,4sleep22878447-21df08:15:132
28617642101,4sleep12861977-21df_inode07:55:121
28499962101,5sleep12850165-21grep07:40:241
28451282101,3sleep22845033-31frcode07:36:412
3121572291,4sleep03121623-21ntpq12:10:190
3106908290,3sleep20-21swapper/212:00:112
3101394291,4sleep13101635-21sed11:55:001
3085894290,3sleep23085913-21apt-get11:40:102
3064332291,3sleep23063816-21apt-get11:20:012
3063803291,4sleep03063868-21https11:20:000
3058803291,3sleep23058822-21apt-get11:15:092
3057852290,2sleep20-21swapper/211:10:232
3054661291,4sleep13056326-21kthreadcore11:10:141
3052364291,4sleep13052421-21gcc11:05:221
3049291291,4sleep03050953-21grep11:05:170
3043397291,4sleep23043608-21smartctl11:00:092
3032236291,4sleep23032385-21gltestperf10:45:112
3016278291,3sleep03016296-21apt-get10:30:090
3009401291,4sleep13009685-21/usr/sbin/munin10:20:171
2982539291,3sleep22982586-21ntpq09:50:202
2972075291,3sleep02972124-21ntpq09:40:190
2929850291,4sleep22930072-21grep09:00:262
2925276291,4sleep02925295-21munin-run08:59:590
2915688291,4sleep12916177-21cat08:50:151
2862254291,4sleep02862928-21grep07:55:150
2860149291,3sleep12860196-21ntpq07:50:181
2849727291,4sleep02849775-21ntpq07:40:190
2849574291,4sleep22849671-21ntpq07:40:182
2836693291,4sleep02838451-21grep07:30:170
2830580291,4sleep00-21swapper/007:25:170
2811628291,2sleep10-21swapper/107:10:111
3144276281,2sleep03144295-21apt-get12:35:120
3139590282,3sleep03140209-21kthreadcore12:30:140
3129063281,3sleep23130102-21kthreadcore12:20:152
3128215281,3sleep01-21systemd12:20:010
3123476281,3sleep20-21swapper/212:15:132
3122492281,3sleep2800-21snmpd12:13:332
3117105281,3sleep23117255-21apt-config12:10:002
3075710283,2sleep0221rcuc/011:30:140
3075282281,3sleep13075426-21diskstats11:30:111
3073915280,4sleep03074162-21unixbench_multi11:25:260
3069022280,1sleep10-21swapper/111:20:341
3031678280,1sleep10-21swapper/110:44:531
3027276281,3sleep23027580-21kthreadcore10:40:142
3011042281,2sleep23011061-21apt-get10:25:112
2989699281,2sleep22989718-21apt-get10:05:112
2989699281,2sleep22989718-21apt-get10:05:102
2982491281,3sleep12982537-21ntpq09:50:181
2963323281,3sleep02963828-21kthreadcore09:35:150
2958189281,3sleep22959832-21kthreadcore09:30:162
2957503280,2sleep10-21swapper/109:30:111
2956511280,5sleep12956507-21apt-get09:25:421
2956459281,2sleep20-21swapper/209:25:382
2920348280,3sleep12920367-21apt-get08:55:111
2919116281,3sleep22919165-21ntpq08:50:202
2910224280,4sleep00-21swapper/008:45:120
2905079280,1sleep20-21swapper/208:40:142
2895063281,3sleep00-21swapper/008:30:180
2887416281,2sleep00-21swapper/008:20:200
2875885281,3sleep02872907-21/usr/sbin/munin08:10:140
2872366280,1sleep20-21swapper/208:06:272
2841078281,3sleep02842698-21kthreadcore07:35:150
2833850281,3sleep22833963-21rt-features07:25:222
2830042282,3sleep134-21ksoftirqd/107:25:141
2829033280,1sleep00-21swapper/007:20:270
2823076281,3sleep02823189-21strings07:15:190
3148574270,3sleep23148645-21find12:35:262
3134338271,3sleep23135604-21kthreadcore12:25:162
3133297270,1sleep00-21swapper/012:22:210
3128906270,1sleep10-21swapper/112:20:141
3116926270,1sleep00-21swapper/012:07:460
3110975271,2sleep13111088-21ps12:00:221
3101183270,1sleep20-21swapper/211:52:212
3096992271,3sleep03097027-21kthreadcore11:50:150
3091155270,1sleep20-21swapper/211:45:032
3042761270,1sleep20-21swapper/210:59:142
3031594270,1sleep00-21swapper/010:43:480
3025483271,2sleep13025530-21ntpq10:35:201
3004931270,1sleep00-21swapper/010:16:320
3000816271,3sleep20-21swapper/210:15:122
2983883270,1sleep10-21swapper/109:56:331
2978229270,1sleep10-21swapper/109:49:031
2963831271,2sleep10-21swapper/109:35:161
2963054271,2sleep20-21swapper/209:35:132
2956503270,2sleep00-21swapper/009:25:410
2951270270,1sleep20-21swapper/209:21:352
2941305270,1sleep00-21swapper/009:15:070
2936055270,2sleep10-21swapper/109:10:111
2914564270,1sleep20-21swapper/208:46:032
2898829270,5sleep10-21swapper/108:34:401
2893421270,1sleep10-21swapper/108:27:471
2871766270,1sleep20-21swapper/208:01:572
2856182270,3sleep02856201-21apt-get07:50:090
2850948270,2sleep20-21swapper/207:45:112
2825840270,3sleep12827525-21cut07:20:171
3148911260,1sleep10-21swapper/112:35:331
3117039260,4sleep10-21swapper/112:09:101
3105935261,2sleep10-21swapper/111:55:241
3095861260,1sleep10-21swapper/111:46:111
3086175260,2sleep00-21swapper/011:40:120
3085338260,3sleep10-21swapper/111:40:011
3053225260,4sleep20-21swapper/211:08:562
3048022260,4sleep10-21swapper/111:04:211
3042434260,1sleep00-21swapper/010:55:030
3041881260,1sleep20-21swapper/210:51:132
3026318260,4sleep20-21swapper/210:38:142
3017539261,2sleep10-21swapper/110:30:151
3015933260,4sleep10-21swapper/110:28:591
3015924260,1sleep00-21swapper/010:28:540
2999804260,3sleep00-21swapper/010:12:550
2998975260,1sleep20-21swapper/210:10:202
2994444260,1sleep00-21swapper/010:06:340
2994444260,1sleep00-21swapper/010:06:340
2989144260,4sleep10-21swapper/110:01:171
2984067260,4sleep20-21swapper/209:58:542
2983306260,1sleep00-21swapper/009:52:220
2978223260,1sleep00-21swapper/009:48:580
2977945260,3sleep20-21swapper/209:45:272
2943166261,2sleep10-21swapper/109:15:151
2940949262,2sleep0151ktimers/009:13:470
2935603260,1sleep00-21swapper/009:07:420
2930174260,4sleep10-21swapper/109:00:291
2925582261,2sleep00-21swapper/009:00:120
2899969260,2sleep02901233-21kthreadcore08:35:140
2888068260,4sleep10-21swapper/108:21:051
2887249260,2sleep20-21swapper/208:20:172
2872473260,1sleep00-21swapper/008:07:500
2871455261,2sleep10-21swapper/108:00:281
2860198260,2sleep22860243-21ntpq07:50:202
2851224261,2sleep00-21swapper/007:45:130
2819913261,2sleep20-21swapper/207:15:162
2815971261,2sleep00-21swapper/007:10:260
3137435250,1sleep00-21swapper/012:25:180
3134266250,1sleep10-21swapper/112:25:151
3123145250,1sleep10-21swapper/112:15:051
3109194250,1sleep00-21swapper/012:00:170
3101976250,1sleep20-21swapper/211:55:142
3101674250,1sleep00-21swapper/011:55:110
3091457250,2sleep00-21swapper/011:45:130
2989992250,1sleep10-21swapper/110:05:141
2989992250,1sleep10-21swapper/110:05:131
2972165251,1sleep20-21swapper/209:40:222
2951337252,1sleep10-21swapper/109:22:261
2945597250,1sleep20-21swapper/209:15:242
2872335250,1sleep10-21swapper/108:06:021
2869989250,1sleep00-21swapper/008:00:190
2984737240,1sleep20-21swapper/210:00:092
2984098240,2sleep00-21swapper/009:59:160
2957478240,1sleep00-21swapper/009:30:020
2899394240,1sleep20-21swapper/208:35:092
2888920240,1sleep00-21swapper/008:25:130
2878504240,1sleep00-21swapper/008:15:150
2851247240,1sleep10-21swapper/107:45:141
2835290240,1sleep10-21swapper/107:30:151
3042435230,1sleep10-21swapper/110:55:041
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional