Add, as root, to your /etc/apt/sources.list (remember to do a backup of this file) the following line:

# deb http://www.texware.it/ubuntu feisty/
      

Then execute:

# wget -q http://www.texware.it/ubuntu/feisty/BBA3222D.gpg -O- | sudo apt-key add -
# sudo apt-get update
# sudo apt-get install linux-realtime
      

The provided packages include realtime-enabled kernels as well as the cyclictest utility (see below).

Thanks to Alessio Igor Bogani for making these kernels available. Additional information is given here.

Download the mainstream kernel sources and the corresonding realtime-preempt patch, for example linux-2.6.21.6 and patch-2.6.21.6-rt21:

# cd /usr/src/kernels
# wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.21.6.tar.bz2
# wget http://www.kernel.org/pub/linux/kernel/projects/rt/older/patch-2.6.21.6-rt21

Unpack the kernel:

# tar -jxvf linux-2.6.21.6.tar.bz2

Rename the kernel directory:

# mv linux-2.6.21.6 linux-2.6.21.6-rt21

Apply the patch:

# cd linux-2.6.21.6-rt21
# patch -p1 <../patch-2.6.21.6-rt21

Copy the configuration file as provided by your distribution to the file .config:

# cp /boot/config-`uname -r` .config

Invoke the text based kernel configuration menu:

# make menuconfig

The preemption model is selected along with the processor type and features:

Processor type and features  --->
    Preemption Mode (Complete Preemption (Real-Time))  --->

Debug, trace and diagnostic tools are part of the "kernel hacking" settings, for example:

Kernel hacking  --->
    [*] Wakeup latency timing
    [*] Latency tracing
    [*] wakeup latency histogram

Make sure that stack overflow checking is disabled since this may produce large latencies:

Kernel hacking  --->
    [ ] Check for stack overflows

The new kernel is then compiled, linked and installed as usual.

# make
# make modules_install install

When the system is rebooted, the newly provided kernel becomes part of the boot menu and can be selected.

The kernel global variable kernel_preemption is now logical true,

# cat /proc/sys/kernel/kernel_preemption
1

or something went completely wrong, otherwise.

The built-in latency trace can be used as an overall test to ensure that the system behaves correctly. This function, however, must be initialized before meaningful values can be obtained:

# echo 0 >/proc/sys/kernel/preempt_max_latency

If the kernel configuration item wakeup latency histogram is enabled, the latency trace is enabled implicitly at boot time. The worst-case latency that was observed since latency tracing was enabled is available in the latency_trace entry of the /proc file system:

# cat /proc/latency_trace
preemption latency trace v1.1.5 on 2.6.19-1.rt10.0001
--------------------------------------------------------------------
 latency: 39 us, #2/2, CPU#0 | (M:rt VP:0, KP:0, SP:1 HP:1 #P:1)
    -----------------
    | task: posix_cpu_timer-3 (uid:0 nice:0 policy:1 rt_prio:99)
    -----------------
 
                 _------=> CPU#
                / _-----=> irqs-off
               | / _----=> need-resched
               || / _---=> hardirq/softirq
               ||| / _--=> preempt-depth
               |||| /
               |||||     delay
   cmd     pid ||||| time  |   caller
      \   /    |||||   \   |   /
   <...>-3     0...1   39us : __schedule (__schedule)

As a role of thumb, the maximum (worst-case) latency amounts to approximately 10⁵ / clock frequency on an idle system. For example, a maximum latency of about 100 microseconds can be expected in a 1 GHz-CPU. If the value displayed is off by more than one order of magnitude, something is not working correctly.

The histograms of the preemption latencies (one per CPU) are available here:

# ls /proc/latency_hist/wakeup_latency/CPU?

This configuration feature is always available in the prebuilt Ubuntu kernels; in the locally created kernels it is only available, if configured as shown above. The granularity of the histograms amounts to one microsecond:

# grep -v " 0$" /proc/latency_hist/wakeup_latency/CPU0
#Minimum latency: 0 microseconds.
#Average latency: 7 microseconds.
#Maximum latency: 39 microseconds.
#Total samples: 7069336
#There are 0 samples greater or equal than 10240 microseconds
#usecs           samples
    0             249884
    1             120023
    2             338781
    3             197834
    4             210872
    5             150366
    6              45870
    7            1053204
    8             564637
    9             273756
   10             190432
   11             483611
   12             328509
   13              44716
   14              72925
   15              59304
   16              28927
   17              10836
   18               2821
   19                543
   20                110
   21                 82
   22                 63
   23                 61
   24                 32
   25                 21
   26                  9
   27                  4
   28                  3
   29                  1
   30                  3
   31                  2
   32                  4
   33                  2
   39                  1

In addition, Thomas Gleixner made available the test tool cyclictest, that allows to better and more spcifically determine the realtime capabilities of a given system. The sources can be downloaded from here. If you installed the Ubuntu apt packages (see above), the tool cyclictest is available immediately. The git repository contains other useful test programs for real-time systems.

Download, unpack and compile the tool

# git-clone git://git.kernel.org/pub/scm/linux/kernel/git/tglx/rt-tests.git rt-tests
# cd rt-tests/cyclictest
# make

To run three test threads with priorities of 99, 98 and 97, type

# ./cyclictest -n -t3 -p99

On a non-realtime system, you may see something like

T: 0 ( 3431) P:99 I:1000 C: 100000 Min:      5 Act:   10 Avg:   14 Max:   39242
T: 1 ( 3432) P:98 I:1500 C:  66934 Min:      4 Act:   10 Avg:   17 Max:   39661
T: 2 ( 3433) P:97 I:2000 C:  50291 Min:      5 Act:    8 Avg:   20 Max:   38884

The rightmost column contains the most important result, i.e. the worst-case latency of 39.661 milliseconds. On a realtime-enabled system, the result may look like

T: 0 ( 3407) P:99 I:1000 C: 100000 Min:      7 Act:   10 Avg:   10 Max:      18
T: 1 ( 3408) P:98 I:1500 C:  67043 Min:      7 Act:    8 Avg:   10 Max:      22
T: 2 ( 3409) P:97 I:2000 C:  50372 Min:      7 Act:   10 Avg:    9 Max:      26

and, thus, indicate a worst-case latency of 26 microseconds.

If - as in the above example - a low worst-case latency is measured, and this is the case even under a system load that is equivalent to the load expected under production conditions, everything is alright. Of course, the measurement must last suffciently long, preferably 24 hours or more. The particular kernel in combination with the hardware tested can then probably be regarded as real-time capable.

What, however, if the latency is higher than acceptable? Then, the famous "latency fighting" begins. For this purpose, the cyclictest tool provides the -b option that causes a function tracing to be written to /proc/latency_trace, if a specified latency threshold was exceeded, for example:

# ./cyclictest -q -t 3 -n -p 99 -b 100

This causes the program to abort execution, if a latency value exceeds 100 microseconds; the culprit can then be found in the trace output at /proc/latency_trace. The function that was executed just before a latency of more than 100 microseconds was detected is marked with an exclamation mark such as

qemu-30047 2D.h3 742805us : __activate_task+0x42/0x68 <cyclicte-426> (199 1)
qemu-30047 2D.h3 742806us : __trace_start_sched_wakeup+0x40/0x161 <cyclicte-426> (0 -1)
qemu-30047 2DNh3 742806us!: try_to_wake_up+0x422/0x460 <cyclicte-426> (199 -5)   
qemu-30047 2DN.1 742939us : __sched_text_start+0xf3/0xdcd (c064e442 0)

The first column indicates the process responsilble for triggering the latency.

If the trace output is not obvious, it can be submitted to the OSADL Latency Fight Support Service at <latency-fightersªosadl.org> In addition to the output of cat /proc/latency_trace, the output of lspci and the .config file that was used to build the kernel in question must be submitted. We are sure you understand that OSADL members will be served first, but we promise to do our best to help everybody to successfully fight against kernel latencies.