You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-15 - 08:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Mon Dec 15, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2530087212357,61sleep30-21swapper/319:07:173
253010529056,9sleep10-21swapper/119:07:321
253015928756,10sleep20-21swapper/219:08:132
252998227253,13sleep00-21swapper/019:05:590
2530418992825,1cyclictest0-21swapper/320:04:593
2530418992421,1cyclictest0-21swapper/323:55:013
2530418992118,1cyclictest0-21swapper/320:10:163
2530418992118,1cyclictest0-21swapper/319:25:013
2530418992017,1cyclictest0-21swapper/323:40:013
2530418992016,2cyclictest0-21swapper/321:50:173
253041899192,11cyclictest2658497-21taskset21:05:203
253041899192,11cyclictest2658497-21taskset21:05:203
2530418991916,1cyclictest0-21swapper/320:30:213
2530418991915,2cyclictest0-21swapper/300:10:163
253041899191,14cyclictest0-21swapper/319:25:123
253041899191,13cyclictest0-21swapper/320:50:213
27848362181,12sleep12784878-21ntpq23:05:191
253041899189,6cyclictest2696097-21turbostat21:45:013
253041899188,6cyclictest2738293-21taskset22:24:323
253041899188,2cyclictest26701512sleep321:20:073
2530418991815,1cyclictest0-21swapper/323:00:163
2530418991815,1cyclictest0-21swapper/320:40:163
2530418991815,1cyclictest0-21swapper/319:10:193
2530418991815,1cyclictest0-21swapper/300:05:193
253041899181,14cyclictest0-21swapper/322:15:133
253041899181,14cyclictest0-21swapper/300:20:233
253041899181,13cyclictest2664657-21turbostat21:15:003
253041899181,13cyclictest2612443-21chrt20:25:123
253041899181,13cyclictest0-21swapper/323:55:223
253041899181,13cyclictest0-21swapper/320:15:113
253041899181,12cyclictest2775296-21taskset22:59:553
253041899181,12cyclictest0-21swapper/320:55:113
25760082171,8sleep12577502-21kthreadcore19:50:161
25429282171,8sleep22542929-21cat19:19:552
2530418991714,1cyclictest0-21swapper/323:15:163
2530418991714,1cyclictest0-21swapper/322:45:213
2530418991714,1cyclictest0-21swapper/300:35:183
253041899171,3cyclictest0-21swapper/322:50:133
253041899171,14cyclictest0-21swapper/319:40:153
253041899171,13cyclictest2786086-21taskset23:10:013
253041899171,13cyclictest0-21swapper/323:45:133
253041899171,12cyclictest2839070-21chrt00:00:113
253041899171,12cyclictest0-21swapper/322:00:113
253041899171,11cyclictest2559293-21chrt19:34:383
253041899171,10cyclictest2801360-21taskset23:22:443
253041899171,10cyclictest2711490-21taskset21:55:383
253041899171,10cyclictest2579795-21taskset19:50:243
2530418991710,3cyclictest26275852sleep320:35:553
253041899170,10cyclictest2758603-21taskset22:40:193
27743292160,1sleep00-21swapper/022:55:200
2530418991613,1cyclictest0-21swapper/319:45:173
2530418991612,2cyclictest0-21swapper/323:40:023
253041899161,13cyclictest0-21swapper/300:25:163
253041899161,12cyclictest0-21swapper/300:15:263
253041899161,11cyclictest2564623-21sleep19:40:003
253041899161,11cyclictest0-21swapper/322:30:123
253041899161,10cyclictest2612156-21taskset20:24:563
253041899161,10cyclictest2591407-21chrt20:05:083
253041899161,10cyclictest2537885-21taskset19:15:063
253041899160,13cyclictest0-21swapper/321:45:163
253041899160,13cyclictest0-21swapper/321:15:193
253041899160,13cyclictest0-21swapper/300:30:193
253041899160,12cyclictest0-21swapper/322:35:143
253041899160,12cyclictest0-21swapper/321:35:133
28536442151,9sleep02853677-21ntpq00:10:190
28332542151,9sleep12833359-21cut23:55:011
27496502151,10sleep22749672-21/usr/sbin/munin22:35:142
27287052154,6sleep12728738-21kthreadcore22:15:171
26823542151,5sleep12684148-21grep21:30:161
25865132151,9sleep22588073-21tr20:00:152
25846702151,9sleep02580662-21/usr/sbin/munin19:55:200
25490722151,10sleep02549196-21perl19:25:140
253041899152,8cyclictest2722909-21taskset22:10:133
253041899152,8cyclictest2722909-21taskset22:10:133
2530418991513,1cyclictest0-21swapper/322:10:003
2530418991511,2cyclictest0-21swapper/321:00:253
253041899151,11cyclictest2743597-21turbostat22:30:003
253041899151,10cyclictest0-21swapper/321:30:113
253041899150,12cyclictest0-21swapper/323:25:233
253041899150,12cyclictest0-21swapper/321:25:143
253041899150,11cyclictest0-21swapper/323:05:113
253041899150,10cyclictest25806612sleep319:55:113
27643672140,3sleep02764370-21grep22:45:250
25654442146,4sleep22566735-21cut19:40:142
253041899141,9cyclictest2633777-21chrt20:45:123
2530418991411,2cyclictest0-21swapper/323:30:243
25853822131,8sleep22585383-21sshd19:56:552
25527922131,7sleep22552848-21ntp_kernel_pll_19:25:202
25346272131,9sleep12534673-21ntpq19:10:161
27952982121,5sleep02795345-21ntpq23:15:200
26811762121,3sleep02682353-21cut21:30:150
26534472121,7sleep22653458-21mailstats21:00:232
26235132121,7sleep02624640-21kthreadcore20:35:160
25487862121,7sleep12548788-21acpi19:25:111
25344852122,5sleep02534626-21ntpq19:10:150
28617242111,6sleep10-21swapper/100:20:181
28386402110,1sleep00-21swapper/000:00:010
28239022111,6sleep02825602-21kthreadcore23:45:170
27694602111,4sleep02769676-21users22:50:260
27021542111,6sleep22703306-21kthreadcore21:50:172
26916992111,6sleep22692441-21kthreadcore21:40:152
26665322111,6sleep12668319-21kthreadcore21:15:161
28748762101,4sleep22874931-21ntpq00:30:212
28283502101,4sleep22828606-21df23:50:132
28055262100,7sleep22805707-21cut23:25:202
27968702101,4sleep02796973-21perl23:20:130
27875632101,4sleep22789331-21cut23:10:162
27808172101,4sleep02780836-21apt-get23:05:100
27756092101,4sleep02775862-21df23:00:130
27638022101,4sleep12763843-21ntpq22:45:191
27547252101,5sleep02754941-21/usr/sbin/munin22:40:130
27174982101,4sleep02717640-21diskstats22:05:140
27170892101,4sleep22717138-21munin-run22:05:002
27003052101,4sleep12700353-21ntpq21:45:191
26704462101,4sleep12670655-21hddtemp_smartct21:20:141
26636732101,4sleep02663722-21ntpq21:10:190
26441752101,5sleep22644321-21head20:55:142
26426792101,5sleep22642736-21ntpq20:50:212
26425342101,4sleep02642623-21ntpq20:50:190
26339852101,5sleep02634535-21cut20:45:140
26229692101,4sleep12623228-21df_inode20:35:121
26217022101,4sleep02621751-21ntpq20:30:190
25913912101,3sleep02590882-21apt-get20:05:020
25704342101,5sleep02570644-21/usr/sbin/munin19:45:140
25472722101,4sleep12547323-21ntpq19:20:221
2865679291,4sleep02865698-21apt-get00:25:100
2860429291,3sleep02860448-21apt-get00:20:130
2855770291,4sleep02856939-21grep00:15:160
2850479291,4sleep12852146-21cut00:10:161
2845934290,1sleep00-21swapper/000:05:180
2818368291,4sleep22819453-21cut23:40:152
2769062291,4sleep22769108-21ntpq22:50:212
2764813291,4sleep22765012-21cat22:50:012
2755117291,4sleep22755221-21irqstats22:40:152
2744856291,4sleep22746500-21grep22:30:172
2742706291,4sleep22742830-21find22:25:202
2737353290,4sleep00-21swapper/022:20:210
2728218291,3sleep22728421-21diskstats22:15:132
2705986291,3sleep02706231-21users21:50:270
2696347291,3sleep02696366-21apt-get21:45:100
2690852290,1sleep00-21swapper/021:39:570
2679355291,4sleep22679406-21ntpq21:25:192
2674075291,4sleep22674132-21ntpq21:20:192
2659102291,1sleep20-21swapper/221:06:002
2659102291,1sleep20-21swapper/221:06:002
2612421291,2sleep20-21swapper/220:25:122
2597202291,5sleep134-21ksoftirqd/120:10:161
2580264290,1sleep00-21swapper/019:53:160
2574162291,4sleep22574217-21ntpq19:45:202
2555642291,4sleep12557376-21tr19:30:151
2554892291,4sleep02555641-21cut19:30:150
2530614291,3sleep22530867-21df19:10:092
2866578281,3sleep12868271-21kthreadcore00:25:151
2864733281,4sleep22864935-21grep00:20:272
2856940281,3sleep12858752-21kthreadcore00:15:171
2839048281,2sleep20-21swapper/200:00:112
2818251281,3sleep12818367-21irqstats23:40:141
2813092281,3sleep02814748-21grep23:35:160
2802373281,3sleep02803815-21kthreadcore23:25:180
2795408281,3sleep22791332-21/usr/sbin/munin23:15:222
2786106281,3sleep12786121-21http23:10:121
2769899280,1sleep10-21swapper/122:53:091
2749374281,2sleep00-21swapper/022:35:120
2722346281,3sleep1553-21dbus-daemon22:10:001
2711767280,1sleep20-21swapper/221:59:112
2711146282,3sleep12711384-21unixbench_multi21:55:251
2668985281,3sleep22669098-21ps21:15:222
2665474281,3sleep02665510-21kthreadcore21:15:160
2653480281,3sleep02653696-21sed21:00:250
2647893281,2sleep02647943-21ntpq20:55:200
2638644280,2sleep10-21swapper/120:50:121
2637408280,3sleep12637457-21ntpq20:45:191
2632233280,3sleep02632287-21ntpq20:40:210
2607988281,3sleep02609674-21kthreadcore20:20:170
2602655281,3sleep02604304-21tr20:15:150
2597795280,5sleep20-21swapper/220:10:172
2591920281,3sleep22592118-21cat20:05:162
2543222281,2sleep02543241-21apt-get19:20:120
2880951270,1sleep20-21swapper/200:38:312
2870870270,2sleep10-21swapper/100:30:121
2855487271,3sleep22855684-21gltestperf00:15:142
2839347271,3sleep12839558-21idleruntime00:00:141
2838200270,4sleep20-21swapper/223:56:422
2833514270,2sleep02833533-21apt-get23:55:120
2823350271,3sleep20-21swapper/223:45:162
2817242271,2sleep02817755-21gpgv23:40:010
2807579271,3sleep10-21swapper/123:30:161
2796588270,2sleep20-21swapper/223:20:112
2791110270,1sleep10-21swapper/123:15:011
2744642271,2sleep10-21swapper/122:30:161
2728133271,2sleep02728152-21apt-get22:15:120
2727676270,1sleep00-21swapper/022:12:320
2727676270,1sleep00-21swapper/022:12:320
2723273270,3sleep10-21swapper/122:10:161
2723273270,3sleep10-21swapper/122:10:151
2722888270,2sleep20-21swapper/222:10:132
2722888270,2sleep20-21swapper/222:10:132
2721577270,1sleep20-21swapper/222:05:242
2706232270,1sleep10-21swapper/121:50:281
2680894271,3sleep22681075-21head21:30:132
2675379271,2sleep12675398-21apt-get21:25:111
2664463270,1sleep10-21swapper/121:12:411
2654939271,3sleep12655056-21kthreadcore21:05:151
2654939271,3sleep12655056-21kthreadcore21:05:151
2633715271,3sleep20-21swapper/220:45:112
2633046270,1sleep20-21swapper/220:43:202
2632836270,1sleep10-21swapper/120:40:381
2627779270,1sleep20-21swapper/220:38:222
2622580270,1sleep10-21swapper/120:33:411
2617987271,2sleep22618201-21smartctl20:30:132
2612147270,1sleep10-21swapper/120:24:511
2606805270,1sleep20-21swapper/220:19:152
2601525270,1sleep00-21swapper/020:13:520
2585909270,2sleep10-21swapper/120:00:101
2580173270,4sleep20-21swapper/219:52:052
2569193271,3sleep02569339-21sort19:40:250
2563663271,3sleep22563714-21/usr/sbin/munin19:35:222
2560195270,3sleep12561821-21cut19:35:171
2545311270,3sleep22547038-21ls19:20:192
2874686260,2sleep00-21swapper/000:30:190
2854457260,1sleep20-21swapper/200:12:252
2844389261,2sleep20-21swapper/200:05:122
2834088260,2sleep10-21swapper/123:55:161
2812232260,1sleep20-21swapper/223:35:022
2808643261,2sleep20-21swapper/223:30:172
2796974261,2sleep12535780-21munin-node23:20:141
2786098261,2sleep00-21swapper/023:10:110
2782790260,2sleep20-21swapper/223:05:162
2780531260,4sleep20-21swapper/223:04:402
2750194261,2sleep10-21swapper/122:35:171
2738609260,3sleep00-21swapper/022:25:100
2735404261,2sleep10-21swapper/122:20:181
2714248260,3sleep00-21swapper/022:00:160
2698156261,2sleep20-21swapper/221:45:162
2694989260,2sleep10-21swapper/121:40:181
2694198260,3sleep0221rcuc/021:40:170
2687051261,2sleep10-21swapper/121:35:171
2663777261,2sleep2421rcuc/221:10:212
2654016260,1sleep10-21swapper/121:03:031
2617109260,1sleep10-21swapper/120:25:541
2606668260,1sleep10-21swapper/120:17:301
2596053260,1sleep10-21swapper/120:05:431
2575156261,2sleep10-21swapper/119:50:001
2560092261,1sleep00-21swapper/019:35:160
2559254260,1sleep20-21swapper/219:34:102
2535684261,2sleep10-21swapper/119:15:011
2879853250,1sleep10-21swapper/100:35:211
2869692250,1sleep20-21swapper/200:25:202
2849432250,1sleep10-21swapper/100:10:001
2832098250,1sleep00-21swapper/023:50:190
2816968250,3sleep10-21swapper/123:36:331
2801978250,1sleep10-21swapper/123:25:151
2754942250,2sleep10-21swapper/122:40:141
2748557250,1sleep00-21swapper/022:30:270
2738753250,1sleep10-21swapper/122:25:111
2712411250,1sleep10-21swapper/122:00:141
2707753250,1sleep00-21swapper/021:55:160
2690789250,3sleep20-21swapper/221:39:112
2674780250,3sleep00-21swapper/021:20:410
2654726251,1sleep00-21swapper/021:05:140
2654726251,1sleep00-21swapper/021:05:130
2645462250,2sleep10-21swapper/120:55:171
2876270240,1sleep00-21swapper/000:35:160
2823041240,1sleep10-21swapper/123:45:071
2807010240,1sleep00-21swapper/023:30:110
2775863240,1sleep10-21swapper/123:00:131
2770600240,1sleep20-21swapper/222:55:142
2770178240,1sleep10-21swapper/122:55:011
2733660240,1sleep20-21swapper/222:20:152
2675669240,1sleep00-21swapper/021:25:140
2612701240,1sleep00-21swapper/020:25:140
2607452240,1sleep20-21swapper/220:20:152
2585701240,1sleep00-21swapper/020:00:010
2580941240,1sleep10-21swapper/119:55:131
2564872240,1sleep10-21swapper/119:40:091
2538171240,1sleep00-21swapper/019:15:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional