Subject: [PATCH 10/16] FIX [2/2] slub: Tid must be retrieved from the percpu area of the current processor
From: Sebastian Andrzej Siewior <[email-masked]>
To: Steven Rostedt <[email-masked]>
Cc: [email-masked], [email-masked], Carsten Emde <[email-masked]>, Christoph Lameter <[email-masked]>, Pekka Enberg <[email-masked]>, Thomas Gleixner <[email-masked]>, Sebastian Andrzej Siewior <[email-masked]>
Date: Wed, 13 Feb 2013 17:12:05 +0100
List-Id: <linux-rt-users.vger.kernel.org>

From: Christoph Lameter <[email-masked]>

As Steven Rostedt has pointer out: Rescheduling could occur on a differnet processor
after the determination of the per cpu pointer and before the tid is retrieved.
This could result in allocation from the wrong node in slab_alloc.

The effect is much more severe in slab_free() where we could free to the freelist
of the wrong page.

The window for something like that occurring is pretty small but it is possible.

Signed-off-by: Christoph Lameter <[email-masked]>
Cc: Steven Rostedt <[email-masked]>
Cc: Pekka Enberg <[email-masked]>
Signed-off-by: Thomas Gleixner <[email-masked]>
Signed-off-by: Sebastian Andrzej Siewior <[email-masked]>
---
 mm/slub.c |   12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

Index: linux-3.2.35-rt53/mm/slub.c
===================================================================
--- linux-3.2.35-rt53.orig/mm/slub.c
+++ linux-3.2.35-rt53/mm/slub.c
@@ -2286,13 +2286,18 @@ static __always_inline void *slab_alloc(
 		return NULL;
 
 redo:
-
 	/*
 	 * Must read kmem_cache cpu data via this cpu ptr. Preemption is
 	 * enabled. We may switch back and forth between cpus while
 	 * reading from one cpu area. That does not matter as long
 	 * as we end up on the original cpu again when doing the cmpxchg.
+	 *
+	 * Preemption is disabled for the retrieval of the tid because that
+	 * must occur from the current processor. We cannot allow rescheduling
+	 * on a different processor between the determination of the pointer
+	 * and the retrieval of the tid.
 	 */
+	preempt_disable();
 	c = __this_cpu_ptr(s->cpu_slab);
 
 	/*
@@ -2302,7 +2307,7 @@ redo:
 	 * linked list in between.
 	 */
 	tid = c->tid;
-	barrier();
+	preempt_enable();
 
 	object = c->freelist;
 	if (unlikely(!object || !node_match(c, node)))
@@ -2544,10 +2549,11 @@ redo:
 	 * data is retrieved via this pointer. If we are on the same cpu
 	 * during the cmpxchg then the free will succedd.
 	 */
+	preempt_disable();
 	c = __this_cpu_ptr(s->cpu_slab);
 
 	tid = c->tid;
-	barrier();
+	preempt_enable();
 
 	if (likely(page == c->page)) {
 		set_freepointer(s, object, c->freelist);