Index: linux-4.4.25-rt35/Documentation/hwlat_detector.txt =================================================================== @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +Introduction: +------------- + +The module hwlat_detector is a special purpose kernel module that is used to +detect large system latencies induced by the behavior of certain underlying +hardware or firmware, independent of Linux itself. The code was developed +originally to detect SMIs (System Management Interrupts) on x86 systems, +however there is nothing x86 specific about this patchset. It was +originally written for use by the "RT" patch since the Real Time +kernel is highly latency sensitive. + +SMIs are usually not serviced by the Linux kernel, which typically does not +even know that they are occuring. SMIs are instead are set up by BIOS code +and are serviced by BIOS code, usually for "critical" events such as +management of thermal sensors and fans. Sometimes though, SMIs are used for +other tasks and those tasks can spend an inordinate amount of time in the +handler (sometimes measured in milliseconds). Obviously this is a problem if +you are trying to keep event service latencies down in the microsecond range. + +The hardware latency detector works by hogging all of the cpus for configurable +amounts of time (by calling stop_machine()), polling the CPU Time Stamp Counter +for some period, then looking for gaps in the TSC data. Any gap indicates a +time when the polling was interrupted and since the machine is stopped and +interrupts turned off the only thing that could do that would be an SMI. + +Note that the SMI detector should *NEVER* be used in a production environment. +It is intended to be run manually to determine if the hardware platform has a +problem with long system firmware service routines. + +Usage: +------ + +Loading the module hwlat_detector passing the parameter "enabled=1" (or by +setting the "enable" entry in "hwlat_detector" debugfs toggled on) is the only +step required to start the hwlat_detector. It is possible to redefine the +threshold in microseconds (us) above which latency spikes will be taken +into account (parameter "threshold="). + +Example: + + # modprobe hwlat_detector enabled=1 threshold=100 + +After the module is loaded, it creates a directory named "hwlat_detector" under +the debugfs mountpoint, "/debug/hwlat_detector" for this text. It is necessary +to have debugfs mounted, which might be on /sys/debug on your system. + +The /debug/hwlat_detector interface contains the following files: + +count - number of latency spikes observed since last reset +enable - a global enable/disable toggle (0/1), resets count +max - maximum hardware latency actually observed (usecs) +sample - a pipe from which to read current raw sample data + in the format <timestamp> <latency observed usecs> + (can be opened O_NONBLOCK for a single sample) +threshold - minimum latency value to be considered (usecs) +width - time period to sample with CPUs held (usecs) + must be less than the total window size (enforced) +window - total period of sampling, width being inside (usecs) + +By default we will set width to 500,000 and window to 1,000,000, meaning that +we will sample every 1,000,000 usecs (1s) for 500,000 usecs (0.5s). If we +observe any latencies that exceed the threshold (initially 100 usecs), +then we write to a global sample ring buffer of 8K samples, which is +consumed by reading from the "sample" (pipe) debugfs file interface. Index: linux-4.4.25-rt35/Documentation/kernel-parameters.txt =================================================================== --- linux-4.4.25-rt35.orig/Documentation/kernel-parameters.txt +++ linux-4.4.25-rt35/Documentation/kernel-parameters.txt @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1639 @ bytes respectively. Such letter suffixes ip= [IP_PNP] See Documentation/filesystems/nfs/nfsroot.txt. + irqaffinity= [SMP] Set the default irq affinity mask + Format: + <cpu number>,...,<cpu number> + or + <cpu number>-<cpu number> + (must be a positive range in ascending order) + or a mixture + <cpu number>,...,<cpu number>-<cpu number> + irqfixup [HW] When an interrupt is not handled search all handlers for it. Intended to get systems with badly broken Index: linux-4.4.25-rt35/Documentation/sysrq.txt =================================================================== --- linux-4.4.25-rt35.orig/Documentation/sysrq.txt +++ linux-4.4.25-rt35/Documentation/sysrq.txt @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:62 @ On PowerPC - Press 'ALT - Print Screen ( On other - If you know of the key combos for other architectures, please let me know so I can add them to this section. -On all - write a character to /proc/sysrq-trigger. e.g.: - +On all - write a character to /proc/sysrq-trigger, e.g.: echo t > /proc/sysrq-trigger +On all - Enable network SysRq by writing a cookie to icmp_echo_sysrq, e.g. + echo 0x01020304 >/proc/sys/net/ipv4/icmp_echo_sysrq + Send an ICMP echo request with this pattern plus the particular + SysRq command key. Example: + # ping -c1 -s57 -p0102030468 + will trigger the SysRq-H (help) command. + + * What are the 'command' keys? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 'b' - Will immediately reboot the system without syncing or unmounting Index: linux-4.4.25-rt35/Documentation/trace/histograms.txt =================================================================== --- /dev/null +++ linux-4.4.25-rt35/Documentation/trace/histograms.txt @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ + Using the Linux Kernel Latency Histograms + + +This document gives a short explanation how to enable, configure and use +latency histograms. Latency histograms are primarily relevant in the +context of real-time enabled kernels (CONFIG_PREEMPT/CONFIG_PREEMPT_RT) +and are used in the quality management of the Linux real-time +capabilities. + + +* Purpose of latency histograms + +A latency histogram continuously accumulates the frequencies of latency +data. There are two types of histograms +- potential sources of latencies +- effective latencies + + +* Potential sources of latencies + +Potential sources of latencies are code segments where interrupts, +preemption or both are disabled (aka critical sections). To create +histograms of potential sources of latency, the kernel stores the time +stamp at the start of a critical section, determines the time elapsed +when the end of the section is reached, and increments the frequency +counter of that latency value - irrespective of whether any concurrently +running process is affected by latency or not. +- Configuration items (in the Kernel hacking/Tracers submenu) + CONFIG_INTERRUPT_OFF_LATENCY + CONFIG_PREEMPT_OFF_LATENCY + + +* Effective latencies + +Effective latencies are actually occuring during wakeup of a process. To +determine effective latencies, the kernel stores the time stamp when a +process is scheduled to be woken up, and determines the duration of the +wakeup time shortly before control is passed over to this process. Note +that the apparent latency in user space may be somewhat longer, since the +process may be interrupted after control is passed over to it but before +the execution in user space takes place. Simply measuring the interval +between enqueuing and wakeup may also not appropriate in cases when a +process is scheduled as a result of a timer expiration. The timer may have +missed its deadline, e.g. due to disabled interrupts, but this latency +would not be registered. Therefore, the offsets of missed timers are +recorded in a separate histogram. If both wakeup latency and missed timer +offsets are configured and enabled, a third histogram may be enabled that +records the overall latency as a sum of the timer latency, if any, and the +wakeup latency. This histogram is called "timerandwakeup". +- Configuration items (in the Kernel hacking/Tracers submenu) + CONFIG_WAKEUP_LATENCY + CONFIG_MISSED_TIMER_OFSETS + + +* Usage + +The interface to the administration of the latency histograms is located +in the debugfs file system. To mount it, either enter + +mount -t sysfs nodev /sys +mount -t debugfs nodev /sys/kernel/debug + +from shell command line level, or add + +nodev /sys sysfs defaults 0 0 +nodev /sys/kernel/debug debugfs defaults 0 0 + +to the file /etc/fstab. All latency histogram related files are then +available in the directory /sys/kernel/debug/tracing/latency_hist. A +particular histogram type is enabled by writing non-zero to the related +variable in the /sys/kernel/debug/tracing/latency_hist/enable directory. +Select "preemptirqsoff" for the histograms of potential sources of +latencies and "wakeup" for histograms of effective latencies etc. The +histogram data - one per CPU - are available in the files + +/sys/kernel/debug/tracing/latency_hist/preemptoff/CPUx +/sys/kernel/debug/tracing/latency_hist/irqsoff/CPUx +/sys/kernel/debug/tracing/latency_hist/preemptirqsoff/CPUx +/sys/kernel/debug/tracing/latency_hist/wakeup/CPUx +/sys/kernel/debug/tracing/latency_hist/wakeup/sharedprio/CPUx +/sys/kernel/debug/tracing/latency_hist/missed_timer_offsets/CPUx +/sys/kernel/debug/tracing/latency_hist/timerandwakeup/CPUx + +The histograms are reset by writing non-zero to the file "reset" in a +particular latency directory. To reset all latency data, use + +#!/bin/sh + +TRACINGDIR=/sys/kernel/debug/tracing +HISTDIR=$TRACINGDIR/latency_hist + +if test -d $HISTDIR +then + cd $HISTDIR + for i in `find . | grep /reset$` + do + echo 1 >$i + done +fi + + +* Data format + +Latency data are stored with a resolution of one microsecond. The +maximum latency is 10,240 microseconds. The data are only valid, if the +overflow register is empty. Every output line contains the latency in +microseconds in the first row and the number of samples in the second +row. To display only lines with a positive latency count, use, for +example, + +grep -v " 0$" /sys/kernel/debug/tracing/latency_hist/preemptoff/CPU0 + +#Minimum latency: 0 microseconds. +#Average latency: 0 microseconds. +#Maximum latency: 25 microseconds. +#Total samples: 3104770694 +#There are 0 samples greater or equal than 10240 microseconds +#usecs samples + 0 2984486876 + 1 49843506 + 2 58219047 + 3 5348126 + 4 2187960 + 5 3388262 + 6 959289 + 7 208294 + 8 40420 + 9 4485 + 10 14918 + 11 18340 + 12 25052 + 13 19455 + 14 5602 + 15 969 + 16 47 + 17 18 + 18 14 + 19 1 + 20 3 + 21 2 + 22 5 + 23 2 + 25 1 + + +* Wakeup latency of a selected process + +To only collect wakeup latency data of a particular process, write the +PID of the requested process to + +/sys/kernel/debug/tracing/latency_hist/wakeup/pid + +PIDs are not considered, if this variable is set to 0. + + +* Details of the process with the highest wakeup latency so far + +Selected data of the process that suffered from the highest wakeup +latency that occurred in a particular CPU are available in the file + +/sys/kernel/debug/tracing/latency_hist/wakeup/max_latency-CPUx. + +In addition, other relevant system data at the time when the +latency occurred are given. + +The format of the data is (all in one line): +<PID> <Priority> <Latency> (<Timeroffset>) <Command> \ +<- <PID> <Priority> <Command> <Timestamp> + +The value of <Timeroffset> is only relevant in the combined timer +and wakeup latency recording. In the wakeup recording, it is +always 0, in the missed_timer_offsets recording, it is the same +as <Latency>. + +When retrospectively searching for the origin of a latency and +tracing was not enabled, it may be helpful to know the name and +some basic data of the task that (finally) was switching to the +late real-tlme task. In addition to the victim's data, also the +data of the possible culprit are therefore displayed after the +"<-" symbol. + +Finally, the timestamp of the time when the latency occurred +in <seconds>.<microseconds> after the most recent system boot +is provided. + +These data are also reset when the wakeup histogram is reset. Index: linux-4.4.25-rt35/Makefile =================================================================== --- linux-4.4.25-rt35.orig/Makefile +++ linux-4.4.25-rt35/Makefile @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:788 @ KBUILD_CFLAGS += $(call cc-option,-Wer # Prohibit date/time macros, which would make the build non-deterministic KBUILD_CFLAGS += $(call cc-option,-Werror=date-time) +# enforce correct pointer usage +KBUILD_CFLAGS += $(call cc-option,-Werror=incompatible-pointer-types) + # use the deterministic mode of AR if available KBUILD_ARFLAGS := $(call ar-option,D) Index: linux-4.4.25-rt35/arch/Kconfig =================================================================== --- linux-4.4.25-rt35.orig/arch/Kconfig +++ linux-4.4.25-rt35/arch/Kconfig @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:12 @ config OPROFILE tristate "OProfile system profiling" depends on PROFILING depends on HAVE_OPROFILE + depends on !PREEMPT_RT_FULL select RING_BUFFER select RING_BUFFER_ALLOW_SWAP help @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:56 @ config KPROBES config JUMP_LABEL bool "Optimize very unlikely/likely branches" depends on HAVE_ARCH_JUMP_LABEL + depends on (!INTERRUPT_OFF_HIST && !PREEMPT_OFF_HIST && !WAKEUP_LATENCY_HIST && !MISSED_TIMER_OFFSETS_HIST) help This option enables a transparent branch optimization that makes certain almost-always-true or almost-always-false branch Index: linux-4.4.25-rt35/arch/arm/Kconfig =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/Kconfig +++ linux-4.4.25-rt35/arch/arm/Kconfig @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:36 @ config ARM select HARDIRQS_SW_RESEND select HAVE_ARCH_AUDITSYSCALL if (AEABI && !OABI_COMPAT) select HAVE_ARCH_BITREVERSE if (CPU_32v7M || CPU_32v7) && !CPU_32v6 - select HAVE_ARCH_JUMP_LABEL if !XIP_KERNEL && !CPU_ENDIAN_BE32 + select HAVE_ARCH_JUMP_LABEL if !XIP_KERNEL && !CPU_ENDIAN_BE32 && !PREEMPT_RT_BASE select HAVE_ARCH_KGDB if !CPU_ENDIAN_BE32 select HAVE_ARCH_SECCOMP_FILTER if (AEABI && !OABI_COMPAT) select HAVE_ARCH_TRACEHOOK @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:71 @ config ARM select HAVE_PERF_EVENTS select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP + select HAVE_PREEMPT_LAZY select HAVE_RCU_TABLE_FREE if (SMP && ARM_LPAE) select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_SYSCALL_TRACEPOINTS Index: linux-4.4.25-rt35/arch/arm/include/asm/switch_to.h =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/include/asm/switch_to.h +++ linux-4.4.25-rt35/arch/arm/include/asm/switch_to.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:6 @ #include <linux/thread_info.h> +#if defined CONFIG_PREEMPT_RT_FULL && defined CONFIG_HIGHMEM +void switch_kmaps(struct task_struct *prev_p, struct task_struct *next_p); +#else +static inline void +switch_kmaps(struct task_struct *prev_p, struct task_struct *next_p) { } +#endif + /* * For v7 SMP cores running a preemptible kernel we may be pre-empted * during a TLB maintenance operation, so execute an inner-shareable dsb @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:35 @ extern struct task_struct *__switch_to(s #define switch_to(prev,next,last) \ do { \ __complete_pending_tlbi(); \ + switch_kmaps(prev, next); \ last = __switch_to(prev,task_thread_info(prev), task_thread_info(next)); \ } while (0) Index: linux-4.4.25-rt35/arch/arm/include/asm/thread_info.h =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/include/asm/thread_info.h +++ linux-4.4.25-rt35/arch/arm/include/asm/thread_info.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:52 @ struct cpu_context_save { struct thread_info { unsigned long flags; /* low level flags */ int preempt_count; /* 0 => preemptable, <0 => bug */ + int preempt_lazy_count; /* 0 => preemptable, <0 => bug */ mm_segment_t addr_limit; /* address limit */ struct task_struct *task; /* main task structure */ __u32 cpu; /* cpu */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:146 @ extern int vfp_restore_user_hwstate(stru #define TIF_SYSCALL_TRACE 4 /* syscall trace active */ #define TIF_SYSCALL_AUDIT 5 /* syscall auditing active */ #define TIF_SYSCALL_TRACEPOINT 6 /* syscall tracepoint instrumentation */ -#define TIF_SECCOMP 7 /* seccomp syscall filtering active */ +#define TIF_SECCOMP 8 /* seccomp syscall filtering active */ +#define TIF_NEED_RESCHED_LAZY 7 #define TIF_NOHZ 12 /* in adaptive nohz mode */ #define TIF_USING_IWMMXT 17 @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:157 @ extern int vfp_restore_user_hwstate(stru #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) +#define _TIF_NEED_RESCHED_LAZY (1 << TIF_NEED_RESCHED_LAZY) #define _TIF_UPROBE (1 << TIF_UPROBE) #define _TIF_SYSCALL_TRACE (1 << TIF_SYSCALL_TRACE) #define _TIF_SYSCALL_AUDIT (1 << TIF_SYSCALL_AUDIT) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:173 @ extern int vfp_restore_user_hwstate(stru * Change these and you break ASM code in entry-common.S */ #define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ - _TIF_NOTIFY_RESUME | _TIF_UPROBE) + _TIF_NOTIFY_RESUME | _TIF_UPROBE | \ + _TIF_NEED_RESCHED_LAZY) #endif /* __KERNEL__ */ #endif /* __ASM_ARM_THREAD_INFO_H */ Index: linux-4.4.25-rt35/arch/arm/kernel/asm-offsets.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/kernel/asm-offsets.c +++ linux-4.4.25-rt35/arch/arm/kernel/asm-offsets.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:68 @ int main(void) BLANK(); DEFINE(TI_FLAGS, offsetof(struct thread_info, flags)); DEFINE(TI_PREEMPT, offsetof(struct thread_info, preempt_count)); + DEFINE(TI_PREEMPT_LAZY, offsetof(struct thread_info, preempt_lazy_count)); DEFINE(TI_ADDR_LIMIT, offsetof(struct thread_info, addr_limit)); DEFINE(TI_TASK, offsetof(struct thread_info, task)); DEFINE(TI_CPU, offsetof(struct thread_info, cpu)); Index: linux-4.4.25-rt35/arch/arm/kernel/entry-armv.S =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/kernel/entry-armv.S +++ linux-4.4.25-rt35/arch/arm/kernel/entry-armv.S @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:218 @ __irq_svc: #ifdef CONFIG_PREEMPT get_thread_info tsk ldr r8, [tsk, #TI_PREEMPT] @ get preempt count - ldr r0, [tsk, #TI_FLAGS] @ get flags teq r8, #0 @ if preempt count != 0 + bne 1f @ return from exeption + ldr r0, [tsk, #TI_FLAGS] @ get flags + tst r0, #_TIF_NEED_RESCHED @ if NEED_RESCHED is set + blne svc_preempt @ preempt! + + ldr r8, [tsk, #TI_PREEMPT_LAZY] @ get preempt lazy count + teq r8, #0 @ if preempt lazy count != 0 movne r0, #0 @ force flags to 0 - tst r0, #_TIF_NEED_RESCHED + tst r0, #_TIF_NEED_RESCHED_LAZY blne svc_preempt +1: #endif svc_exit r5, irq = 1 @ return from exception @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:244 @ svc_preempt: 1: bl preempt_schedule_irq @ irq en/disable is done inside ldr r0, [tsk, #TI_FLAGS] @ get new tasks TI_FLAGS tst r0, #_TIF_NEED_RESCHED + bne 1b + tst r0, #_TIF_NEED_RESCHED_LAZY reteq r8 @ go again - b 1b + ldr r0, [tsk, #TI_PREEMPT_LAZY] @ get preempt lazy count + teq r0, #0 @ if preempt lazy count != 0 + beq 1b + ret r8 @ go again + #endif __und_fault: Index: linux-4.4.25-rt35/arch/arm/kernel/entry-common.S =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/kernel/entry-common.S +++ linux-4.4.25-rt35/arch/arm/kernel/entry-common.S @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:39 @ ret_fast_syscall: UNWIND(.cantunwind ) disable_irq_notrace @ disable interrupts ldr r1, [tsk, #TI_FLAGS] @ re-check for syscall tracing - tst r1, #_TIF_SYSCALL_WORK | _TIF_WORK_MASK + tst r1, #((_TIF_SYSCALL_WORK | _TIF_WORK_MASK) & ~_TIF_SECCOMP) + bne fast_work_pending + tst r1, #_TIF_SECCOMP bne fast_work_pending /* perform architecture specific actions before user return */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:67 @ ret_fast_syscall: str r0, [sp, #S_R0 + S_OFF]! @ save returned r0 disable_irq_notrace @ disable interrupts ldr r1, [tsk, #TI_FLAGS] @ re-check for syscall tracing - tst r1, #_TIF_SYSCALL_WORK | _TIF_WORK_MASK + tst r1, #((_TIF_SYSCALL_WORK | _TIF_WORK_MASK) & ~_TIF_SECCOMP) + bne do_slower_path + tst r1, #_TIF_SECCOMP beq no_work_pending +do_slower_path: UNWIND(.fnend ) ENDPROC(ret_fast_syscall) Index: linux-4.4.25-rt35/arch/arm/kernel/process.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/kernel/process.c +++ linux-4.4.25-rt35/arch/arm/kernel/process.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:322 @ unsigned long arch_randomize_brk(struct } #ifdef CONFIG_MMU +/* + * CONFIG_SPLIT_PTLOCK_CPUS results in a page->ptl lock. If the lock is not + * initialized by pgtable_page_ctor() then a coredump of the vector page will + * fail. + */ +static int __init vectors_user_mapping_init_page(void) +{ + struct page *page; + unsigned long addr = 0xffff0000; + pgd_t *pgd; + pud_t *pud; + pmd_t *pmd; + + pgd = pgd_offset_k(addr); + pud = pud_offset(pgd, addr); + pmd = pmd_offset(pud, addr); + page = pmd_page(*(pmd)); + + pgtable_page_ctor(page); + + return 0; +} +late_initcall(vectors_user_mapping_init_page); + #ifdef CONFIG_KUSER_HELPERS /* * The vectors page is always readable from user space for the Index: linux-4.4.25-rt35/arch/arm/kernel/signal.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/kernel/signal.c +++ linux-4.4.25-rt35/arch/arm/kernel/signal.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:575 @ do_work_pending(struct pt_regs *regs, un */ trace_hardirqs_off(); do { - if (likely(thread_flags & _TIF_NEED_RESCHED)) { + if (likely(thread_flags & (_TIF_NEED_RESCHED | + _TIF_NEED_RESCHED_LAZY))) { schedule(); } else { if (unlikely(!user_mode(regs))) Index: linux-4.4.25-rt35/arch/arm/kernel/smp.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/kernel/smp.c +++ linux-4.4.25-rt35/arch/arm/kernel/smp.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:233 @ int __cpu_disable(void) flush_cache_louis(); local_flush_tlb_all(); - clear_tasks_mm_cpumask(cpu); - return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:248 @ void __cpu_die(unsigned int cpu) pr_err("CPU%u: cpu didn't die\n", cpu); return; } + + clear_tasks_mm_cpumask(cpu); + pr_notice("CPU%u: shutdown\n", cpu); /* Index: linux-4.4.25-rt35/arch/arm/kernel/unwind.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/kernel/unwind.c +++ linux-4.4.25-rt35/arch/arm/kernel/unwind.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:96 @ extern const struct unwind_idx __start_u static const struct unwind_idx *__origin_unwind_idx; extern const struct unwind_idx __stop_unwind_idx[]; -static DEFINE_SPINLOCK(unwind_lock); +static DEFINE_RAW_SPINLOCK(unwind_lock); static LIST_HEAD(unwind_tables); /* Convert a prel31 symbol to an absolute address */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:204 @ static const struct unwind_idx *unwind_f /* module unwind tables */ struct unwind_table *table; - spin_lock_irqsave(&unwind_lock, flags); + raw_spin_lock_irqsave(&unwind_lock, flags); list_for_each_entry(table, &unwind_tables, list) { if (addr >= table->begin_addr && addr < table->end_addr) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:216 @ static const struct unwind_idx *unwind_f break; } } - spin_unlock_irqrestore(&unwind_lock, flags); + raw_spin_unlock_irqrestore(&unwind_lock, flags); } pr_debug("%s: idx = %p\n", __func__, idx); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:532 @ struct unwind_table *unwind_table_add(un tab->begin_addr = text_addr; tab->end_addr = text_addr + text_size; - spin_lock_irqsave(&unwind_lock, flags); + raw_spin_lock_irqsave(&unwind_lock, flags); list_add_tail(&tab->list, &unwind_tables); - spin_unlock_irqrestore(&unwind_lock, flags); + raw_spin_unlock_irqrestore(&unwind_lock, flags); return tab; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:546 @ void unwind_table_del(struct unwind_tabl if (!tab) return; - spin_lock_irqsave(&unwind_lock, flags); + raw_spin_lock_irqsave(&unwind_lock, flags); list_del(&tab->list); - spin_unlock_irqrestore(&unwind_lock, flags); + raw_spin_unlock_irqrestore(&unwind_lock, flags); kfree(tab); } Index: linux-4.4.25-rt35/arch/arm/kvm/arm.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/kvm/arm.c +++ linux-4.4.25-rt35/arch/arm/kvm/arm.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:499 @ static void kvm_arm_resume_guest(struct struct kvm_vcpu *vcpu; kvm_for_each_vcpu(i, vcpu, kvm) { - wait_queue_head_t *wq = kvm_arch_vcpu_wq(vcpu); + struct swait_queue_head *wq = kvm_arch_vcpu_wq(vcpu); vcpu->arch.pause = false; - wake_up_interruptible(wq); + swake_up(wq); } } static void vcpu_sleep(struct kvm_vcpu *vcpu) { - wait_queue_head_t *wq = kvm_arch_vcpu_wq(vcpu); + struct swait_queue_head *wq = kvm_arch_vcpu_wq(vcpu); - wait_event_interruptible(*wq, ((!vcpu->arch.power_off) && + swait_event_interruptible(*wq, ((!vcpu->arch.power_off) && (!vcpu->arch.pause))); } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:569 @ int kvm_arch_vcpu_ioctl_run(struct kvm_v * involves poking the GIC, which must be done in a * non-preemptible context. */ - preempt_disable(); + migrate_disable(); kvm_timer_flush_hwstate(vcpu); kvm_vgic_flush_hwstate(vcpu); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:588 @ int kvm_arch_vcpu_ioctl_run(struct kvm_v local_irq_enable(); kvm_timer_sync_hwstate(vcpu); kvm_vgic_sync_hwstate(vcpu); - preempt_enable(); + migrate_enable(); continue; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:642 @ int kvm_arch_vcpu_ioctl_run(struct kvm_v kvm_vgic_sync_hwstate(vcpu); - preempt_enable(); + migrate_enable(); ret = handle_exit(vcpu, run, ret); } Index: linux-4.4.25-rt35/arch/arm/kvm/psci.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/kvm/psci.c +++ linux-4.4.25-rt35/arch/arm/kvm/psci.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:73 @ static unsigned long kvm_psci_vcpu_on(st { struct kvm *kvm = source_vcpu->kvm; struct kvm_vcpu *vcpu = NULL; - wait_queue_head_t *wq; + struct swait_queue_head *wq; unsigned long cpu_id; unsigned long context_id; phys_addr_t target_pc; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:122 @ static unsigned long kvm_psci_vcpu_on(st smp_mb(); /* Make sure the above is visible */ wq = kvm_arch_vcpu_wq(vcpu); - wake_up_interruptible(wq); + swake_up(wq); return PSCI_RET_SUCCESS; } Index: linux-4.4.25-rt35/arch/arm/mach-at91/Kconfig =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mach-at91/Kconfig +++ linux-4.4.25-rt35/arch/arm/mach-at91/Kconfig @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:102 @ config HAVE_AT91_USB_CLK config COMMON_CLK_AT91 bool select COMMON_CLK + select MFD_SYSCON config HAVE_AT91_SMD bool Index: linux-4.4.25-rt35/arch/arm/mach-at91/at91rm9200.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mach-at91/at91rm9200.c +++ linux-4.4.25-rt35/arch/arm/mach-at91/at91rm9200.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:15 @ #include <linux/of_platform.h> #include <asm/mach/arch.h> -#include <asm/system_misc.h> #include "generic.h" #include "soc.h" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:35 @ static void __init at91rm9200_dt_device_ of_platform_populate(NULL, of_default_bus_match_table, NULL, soc_dev); - arm_pm_idle = at91rm9200_idle; at91rm9200_pm_init(); } Index: linux-4.4.25-rt35/arch/arm/mach-at91/at91sam9.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mach-at91/at91sam9.c +++ linux-4.4.25-rt35/arch/arm/mach-at91/at91sam9.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:65 @ static void __init at91sam9_common_init( soc_dev = soc_device_to_device(soc); of_platform_populate(NULL, of_default_bus_match_table, NULL, soc_dev); - - arm_pm_idle = at91sam9_idle; } static void __init at91sam9_dt_device_init(void) Index: linux-4.4.25-rt35/arch/arm/mach-at91/generic.h =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mach-at91/generic.h +++ linux-4.4.25-rt35/arch/arm/mach-at91/generic.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:14 @ #ifndef _AT91_GENERIC_H #define _AT91_GENERIC_H -#include <linux/of.h> -#include <linux/reboot.h> - - /* Map io */ -extern void __init at91_map_io(void); -extern void __init at91_alt_map_io(void); - -/* idle */ -extern void at91rm9200_idle(void); -extern void at91sam9_idle(void); - #ifdef CONFIG_PM extern void __init at91rm9200_pm_init(void); extern void __init at91sam9260_pm_init(void); extern void __init at91sam9g45_pm_init(void); extern void __init at91sam9x5_pm_init(void); +extern void __init sama5_pm_init(void); #else static inline void __init at91rm9200_pm_init(void) { } static inline void __init at91sam9260_pm_init(void) { } static inline void __init at91sam9g45_pm_init(void) { } static inline void __init at91sam9x5_pm_init(void) { } +static inline void __init sama5_pm_init(void) { } #endif #endif /* _AT91_GENERIC_H */ Index: linux-4.4.25-rt35/arch/arm/mach-at91/pm.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mach-at91/pm.c +++ linux-4.4.25-rt35/arch/arm/mach-at91/pm.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:34 @ #include <asm/mach/irq.h> #include <asm/fncpy.h> #include <asm/cacheflush.h> +#include <asm/system_misc.h> #include "generic.h" #include "pm.h" +static void __iomem *pmc; + /* * FIXME: this is needed to communicate between the pinctrl driver and * the PM implementation in the machine. Possibly part of the PM @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:93 @ static int at91_pm_verify_clocks(void) unsigned long scsr; int i; - scsr = at91_pmc_read(AT91_PMC_SCSR); + scsr = readl(pmc + AT91_PMC_SCSR); /* USB must not be using PLLB */ if ((scsr & at91_pm_data.uhp_udp_mask) != 0) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:107 @ static int at91_pm_verify_clocks(void) if ((scsr & (AT91_PMC_PCK0 << i)) == 0) continue; - - css = at91_pmc_read(AT91_PMC_PCKR(i)) & AT91_PMC_CSS; + css = readl(pmc + AT91_PMC_PCKR(i)) & AT91_PMC_CSS; if (css != AT91_PMC_CSS_SLOW) { pr_err("AT91: PM - Suspend-to-RAM with PCK%d src %d\n", i, css); return 0; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:150 @ static void at91_pm_suspend(suspend_stat flush_cache_all(); outer_disable(); - at91_suspend_sram_fn(at91_pmc_base, at91_ramc_base[0], - at91_ramc_base[1], pm_data); + at91_suspend_sram_fn(pmc, at91_ramc_base[0], + at91_ramc_base[1], pm_data); outer_resume(); } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:358 @ static __init void at91_dt_ramc(void) at91_pm_set_standby(standby); } +void at91rm9200_idle(void) +{ + /* + * Disable the processor clock. The processor will be automatically + * re-enabled by an interrupt or by a reset. + */ + writel(AT91_PMC_PCK, pmc + AT91_PMC_SCDR); +} + +void at91sam9_idle(void) +{ + writel(AT91_PMC_PCK, pmc + AT91_PMC_SCDR); + cpu_do_idle(); +} + static void __init at91_pm_sram_init(void) { struct gen_pool *sram_pool; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:419 @ static void __init at91_pm_sram_init(voi &at91_pm_suspend_in_sram, at91_pm_suspend_in_sram_sz); } -static void __init at91_pm_init(void) +static const struct of_device_id atmel_pmc_ids[] __initconst = { + { .compatible = "atmel,at91rm9200-pmc" }, + { .compatible = "atmel,at91sam9260-pmc" }, + { .compatible = "atmel,at91sam9g45-pmc" }, + { .compatible = "atmel,at91sam9n12-pmc" }, + { .compatible = "atmel,at91sam9x5-pmc" }, + { .compatible = "atmel,sama5d3-pmc" }, + { .compatible = "atmel,sama5d2-pmc" }, + { /* sentinel */ }, +}; + +static void __init at91_pm_init(void (*pm_idle)(void)) { - at91_pm_sram_init(); + struct device_node *pmc_np; if (at91_cpuidle_device.dev.platform_data) platform_device_register(&at91_cpuidle_device); + pmc_np = of_find_matching_node(NULL, atmel_pmc_ids); + pmc = of_iomap(pmc_np, 0); + if (!pmc) { + pr_err("AT91: PM not supported, PMC not found\n"); + return; + } + + if (pm_idle) + arm_pm_idle = pm_idle; + + at91_pm_sram_init(); + if (at91_suspend_sram_fn) suspend_set_ops(&at91_pm_ops); else @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:467 @ void __init at91rm9200_pm_init(void) at91_pm_data.uhp_udp_mask = AT91RM9200_PMC_UHP | AT91RM9200_PMC_UDP; at91_pm_data.memctrl = AT91_MEMCTRL_MC; - at91_pm_init(); + at91_pm_init(at91rm9200_idle); } void __init at91sam9260_pm_init(void) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:475 @ void __init at91sam9260_pm_init(void) at91_dt_ramc(); at91_pm_data.memctrl = AT91_MEMCTRL_SDRAMC; at91_pm_data.uhp_udp_mask = AT91SAM926x_PMC_UHP | AT91SAM926x_PMC_UDP; - return at91_pm_init(); + at91_pm_init(at91sam9_idle); } void __init at91sam9g45_pm_init(void) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:483 @ void __init at91sam9g45_pm_init(void) at91_dt_ramc(); at91_pm_data.uhp_udp_mask = AT91SAM926x_PMC_UHP; at91_pm_data.memctrl = AT91_MEMCTRL_DDRSDR; - return at91_pm_init(); + at91_pm_init(at91sam9_idle); } void __init at91sam9x5_pm_init(void) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:491 @ void __init at91sam9x5_pm_init(void) at91_dt_ramc(); at91_pm_data.uhp_udp_mask = AT91SAM926x_PMC_UHP | AT91SAM926x_PMC_UDP; at91_pm_data.memctrl = AT91_MEMCTRL_DDRSDR; - return at91_pm_init(); + at91_pm_init(at91sam9_idle); +} + +void __init sama5_pm_init(void) +{ + at91_dt_ramc(); + at91_pm_data.uhp_udp_mask = AT91SAM926x_PMC_UHP | AT91SAM926x_PMC_UDP; + at91_pm_data.memctrl = AT91_MEMCTRL_DDRSDR; + at91_pm_init(NULL); } Index: linux-4.4.25-rt35/arch/arm/mach-at91/sama5.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mach-at91/sama5.c +++ linux-4.4.25-rt35/arch/arm/mach-at91/sama5.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:54 @ static void __init sama5_dt_device_init( soc_dev = soc_device_to_device(soc); of_platform_populate(NULL, of_default_bus_match_table, NULL, soc_dev); - at91sam9x5_pm_init(); + sama5_pm_init(); } static const char *const sama5_dt_board_compat[] __initconst = { Index: linux-4.4.25-rt35/arch/arm/mach-exynos/platsmp.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mach-exynos/platsmp.c +++ linux-4.4.25-rt35/arch/arm/mach-exynos/platsmp.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:233 @ static void __iomem *scu_base_addr(void) return (void __iomem *)(S5P_VA_SCU); } -static DEFINE_SPINLOCK(boot_lock); +static DEFINE_RAW_SPINLOCK(boot_lock); static void exynos_secondary_init(unsigned int cpu) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:246 @ static void exynos_secondary_init(unsign /* * Synchronise with the boot thread. */ - spin_lock(&boot_lock); - spin_unlock(&boot_lock); + raw_spin_lock(&boot_lock); + raw_spin_unlock(&boot_lock); } int exynos_set_boot_addr(u32 core_id, unsigned long boot_addr) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:311 @ static int exynos_boot_secondary(unsigne * Set synchronisation state between this boot processor * and the secondary one */ - spin_lock(&boot_lock); + raw_spin_lock(&boot_lock); /* * The secondary processor is waiting to be released from @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:338 @ static int exynos_boot_secondary(unsigne if (timeout == 0) { printk(KERN_ERR "cpu1 power enable failed"); - spin_unlock(&boot_lock); + raw_spin_unlock(&boot_lock); return -ETIMEDOUT; } } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:384 @ static int exynos_boot_secondary(unsigne * calibrations, then wait for it to finish */ fail: - spin_unlock(&boot_lock); + raw_spin_unlock(&boot_lock); return pen_release != -1 ? ret : 0; } Index: linux-4.4.25-rt35/arch/arm/mach-hisi/platmcpm.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mach-hisi/platmcpm.c +++ linux-4.4.25-rt35/arch/arm/mach-hisi/platmcpm.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:64 @ static void __iomem *sysctrl, *fabric; static int hip04_cpu_table[HIP04_MAX_CLUSTERS][HIP04_MAX_CPUS_PER_CLUSTER]; -static DEFINE_SPINLOCK(boot_lock); +static DEFINE_RAW_SPINLOCK(boot_lock); static u32 fabric_phys_addr; /* * [0]: bootwrapper physical address @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:116 @ static int hip04_boot_secondary(unsigned if (cluster >= HIP04_MAX_CLUSTERS || cpu >= HIP04_MAX_CPUS_PER_CLUSTER) return -EINVAL; - spin_lock_irq(&boot_lock); + raw_spin_lock_irq(&boot_lock); if (hip04_cpu_table[cluster][cpu]) goto out; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:150 @ static int hip04_boot_secondary(unsigned out: hip04_cpu_table[cluster][cpu]++; - spin_unlock_irq(&boot_lock); + raw_spin_unlock_irq(&boot_lock); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:165 @ static void hip04_cpu_die(unsigned int l cpu = MPIDR_AFFINITY_LEVEL(mpidr, 0); cluster = MPIDR_AFFINITY_LEVEL(mpidr, 1); - spin_lock(&boot_lock); + raw_spin_lock(&boot_lock); hip04_cpu_table[cluster][cpu]--; if (hip04_cpu_table[cluster][cpu] == 1) { /* A power_up request went ahead of us. */ - spin_unlock(&boot_lock); + raw_spin_unlock(&boot_lock); return; } else if (hip04_cpu_table[cluster][cpu] > 1) { pr_err("Cluster %d CPU%d boots multiple times\n", cluster, cpu); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:177 @ static void hip04_cpu_die(unsigned int l } last_man = hip04_cluster_is_down(cluster); - spin_unlock(&boot_lock); + raw_spin_unlock(&boot_lock); if (last_man) { /* Since it's Cortex A15, disable L2 prefetching. */ asm volatile( @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:206 @ static int hip04_cpu_kill(unsigned int l cpu >= HIP04_MAX_CPUS_PER_CLUSTER); count = TIMEOUT_MSEC / POLL_MSEC; - spin_lock_irq(&boot_lock); + raw_spin_lock_irq(&boot_lock); for (tries = 0; tries < count; tries++) { if (hip04_cpu_table[cluster][cpu]) goto err; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:214 @ static int hip04_cpu_kill(unsigned int l data = readl_relaxed(sysctrl + SC_CPU_RESET_STATUS(cluster)); if (data & CORE_WFI_STATUS(cpu)) break; - spin_unlock_irq(&boot_lock); + raw_spin_unlock_irq(&boot_lock); /* Wait for clean L2 when the whole cluster is down. */ msleep(POLL_MSEC); - spin_lock_irq(&boot_lock); + raw_spin_lock_irq(&boot_lock); } if (tries >= count) goto err; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:234 @ static int hip04_cpu_kill(unsigned int l goto err; if (hip04_cluster_is_down(cluster)) hip04_set_snoop_filter(cluster, 0); - spin_unlock_irq(&boot_lock); + raw_spin_unlock_irq(&boot_lock); return 1; err: - spin_unlock_irq(&boot_lock); + raw_spin_unlock_irq(&boot_lock); return 0; } #endif Index: linux-4.4.25-rt35/arch/arm/mach-imx/Kconfig =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mach-imx/Kconfig +++ linux-4.4.25-rt35/arch/arm/mach-imx/Kconfig @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:527 @ config SOC_IMX6Q bool "i.MX6 Quad/DualLite support" select ARM_ERRATA_764369 if SMP select HAVE_ARM_SCU if SMP - select HAVE_ARM_TWD if SMP + select HAVE_ARM_TWD select PCI_DOMAINS if PCI select PINCTRL_IMX6Q select SOC_IMX6 Index: linux-4.4.25-rt35/arch/arm/mach-omap2/omap-smp.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mach-omap2/omap-smp.c +++ linux-4.4.25-rt35/arch/arm/mach-omap2/omap-smp.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:46 @ /* SCU base address */ static void __iomem *scu_base; -static DEFINE_SPINLOCK(boot_lock); +static DEFINE_RAW_SPINLOCK(boot_lock); void __iomem *omap4_get_scu_base(void) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:77 @ static void omap4_secondary_init(unsigne /* * Synchronise with the boot thread. */ - spin_lock(&boot_lock); - spin_unlock(&boot_lock); + raw_spin_lock(&boot_lock); + raw_spin_unlock(&boot_lock); } static int omap4_boot_secondary(unsigned int cpu, struct task_struct *idle) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:92 @ static int omap4_boot_secondary(unsigned * Set synchronisation state between this boot processor * and the secondary one */ - spin_lock(&boot_lock); + raw_spin_lock(&boot_lock); /* * Update the AuxCoreBoot0 with boot state for secondary core. @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:169 @ static int omap4_boot_secondary(unsigned * Now the secondary core is starting up let it run its * calibrations, then wait for it to finish */ - spin_unlock(&boot_lock); + raw_spin_unlock(&boot_lock); return 0; } Index: linux-4.4.25-rt35/arch/arm/mach-prima2/platsmp.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mach-prima2/platsmp.c +++ linux-4.4.25-rt35/arch/arm/mach-prima2/platsmp.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:25 @ static void __iomem *clk_base; -static DEFINE_SPINLOCK(boot_lock); +static DEFINE_RAW_SPINLOCK(boot_lock); static void sirfsoc_secondary_init(unsigned int cpu) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:39 @ static void sirfsoc_secondary_init(unsig /* * Synchronise with the boot thread. */ - spin_lock(&boot_lock); - spin_unlock(&boot_lock); + raw_spin_lock(&boot_lock); + raw_spin_unlock(&boot_lock); } static const struct of_device_id clk_ids[] = { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:78 @ static int sirfsoc_boot_secondary(unsign /* make sure write buffer is drained */ mb(); - spin_lock(&boot_lock); + raw_spin_lock(&boot_lock); /* * The secondary processor is waiting to be released from @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:110 @ static int sirfsoc_boot_secondary(unsign * now the secondary core is starting up let it run its * calibrations, then wait for it to finish */ - spin_unlock(&boot_lock); + raw_spin_unlock(&boot_lock); return pen_release != -1 ? -ENOSYS : 0; } Index: linux-4.4.25-rt35/arch/arm/mach-qcom/platsmp.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mach-qcom/platsmp.c +++ linux-4.4.25-rt35/arch/arm/mach-qcom/platsmp.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:49 @ extern void secondary_startup_arm(void); -static DEFINE_SPINLOCK(boot_lock); +static DEFINE_RAW_SPINLOCK(boot_lock); #ifdef CONFIG_HOTPLUG_CPU static void qcom_cpu_die(unsigned int cpu) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:63 @ static void qcom_secondary_init(unsigned /* * Synchronise with the boot thread. */ - spin_lock(&boot_lock); - spin_unlock(&boot_lock); + raw_spin_lock(&boot_lock); + raw_spin_unlock(&boot_lock); } static int scss_release_secondary(unsigned int cpu) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:287 @ static int qcom_boot_secondary(unsigned * set synchronisation state between this boot processor * and the secondary one */ - spin_lock(&boot_lock); + raw_spin_lock(&boot_lock); /* * Send the secondary CPU a soft interrupt, thereby causing @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:300 @ static int qcom_boot_secondary(unsigned * now the secondary core is starting up let it run its * calibrations, then wait for it to finish */ - spin_unlock(&boot_lock); + raw_spin_unlock(&boot_lock); return ret; } Index: linux-4.4.25-rt35/arch/arm/mach-spear/platsmp.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mach-spear/platsmp.c +++ linux-4.4.25-rt35/arch/arm/mach-spear/platsmp.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:35 @ static void write_pen_release(int val) sync_cache_w(&pen_release); } -static DEFINE_SPINLOCK(boot_lock); +static DEFINE_RAW_SPINLOCK(boot_lock); static void __iomem *scu_base = IOMEM(VA_SCU_BASE); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:50 @ static void spear13xx_secondary_init(uns /* * Synchronise with the boot thread. */ - spin_lock(&boot_lock); - spin_unlock(&boot_lock); + raw_spin_lock(&boot_lock); + raw_spin_unlock(&boot_lock); } static int spear13xx_boot_secondary(unsigned int cpu, struct task_struct *idle) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:62 @ static int spear13xx_boot_secondary(unsi * set synchronisation state between this boot processor * and the secondary one */ - spin_lock(&boot_lock); + raw_spin_lock(&boot_lock); /* * The secondary processor is waiting to be released from @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:87 @ static int spear13xx_boot_secondary(unsi * now the secondary core is starting up let it run its * calibrations, then wait for it to finish */ - spin_unlock(&boot_lock); + raw_spin_unlock(&boot_lock); return pen_release != -1 ? -ENOSYS : 0; } Index: linux-4.4.25-rt35/arch/arm/mach-sti/platsmp.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mach-sti/platsmp.c +++ linux-4.4.25-rt35/arch/arm/mach-sti/platsmp.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:38 @ static void write_pen_release(int val) sync_cache_w(&pen_release); } -static DEFINE_SPINLOCK(boot_lock); +static DEFINE_RAW_SPINLOCK(boot_lock); static void sti_secondary_init(unsigned int cpu) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:51 @ static void sti_secondary_init(unsigned /* * Synchronise with the boot thread. */ - spin_lock(&boot_lock); - spin_unlock(&boot_lock); + raw_spin_lock(&boot_lock); + raw_spin_unlock(&boot_lock); } static int sti_boot_secondary(unsigned int cpu, struct task_struct *idle) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:63 @ static int sti_boot_secondary(unsigned i * set synchronisation state between this boot processor * and the secondary one */ - spin_lock(&boot_lock); + raw_spin_lock(&boot_lock); /* * The secondary processor is waiting to be released from @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:94 @ static int sti_boot_secondary(unsigned i * now the secondary core is starting up let it run its * calibrations, then wait for it to finish */ - spin_unlock(&boot_lock); + raw_spin_unlock(&boot_lock); return pen_release != -1 ? -ENOSYS : 0; } Index: linux-4.4.25-rt35/arch/arm/mm/fault.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mm/fault.c +++ linux-4.4.25-rt35/arch/arm/mm/fault.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:433 @ do_translation_fault(unsigned long addr, if (addr < TASK_SIZE) return do_page_fault(addr, fsr, regs); + if (interrupts_enabled(regs)) + local_irq_enable(); + if (user_mode(regs)) goto bad_area; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:503 @ do_translation_fault(unsigned long addr, static int do_sect_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) { + if (interrupts_enabled(regs)) + local_irq_enable(); + do_bad_area(addr, fsr, regs); return 0; } Index: linux-4.4.25-rt35/arch/arm/mm/highmem.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/mm/highmem.c +++ linux-4.4.25-rt35/arch/arm/mm/highmem.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:37 @ static inline pte_t get_fixmap_pte(unsig return *ptep; } +static unsigned int fixmap_idx(int type) +{ + return FIX_KMAP_BEGIN + type + KM_TYPE_NR * smp_processor_id(); +} + void *kmap(struct page *page) { might_sleep(); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:62 @ EXPORT_SYMBOL(kunmap); void *kmap_atomic(struct page *page) { + pte_t pte = mk_pte(page, kmap_prot); unsigned int idx; unsigned long vaddr; void *kmap; int type; - preempt_disable(); + preempt_disable_nort(); pagefault_disable(); if (!PageHighMem(page)) return page_address(page); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:88 @ void *kmap_atomic(struct page *page) type = kmap_atomic_idx_push(); - idx = FIX_KMAP_BEGIN + type + KM_TYPE_NR * smp_processor_id(); + idx = fixmap_idx(type); vaddr = __fix_to_virt(idx); #ifdef CONFIG_DEBUG_HIGHMEM /* @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:102 @ void *kmap_atomic(struct page *page) * in place, so the contained TLB flush ensures the TLB is updated * with the new mapping. */ - set_fixmap_pte(idx, mk_pte(page, kmap_prot)); +#ifdef CONFIG_PREEMPT_RT_FULL + current->kmap_pte[type] = pte; +#endif + set_fixmap_pte(idx, pte); return (void *)vaddr; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:118 @ void __kunmap_atomic(void *kvaddr) if (kvaddr >= (void *)FIXADDR_START) { type = kmap_atomic_idx(); - idx = FIX_KMAP_BEGIN + type + KM_TYPE_NR * smp_processor_id(); + idx = fixmap_idx(type); if (cache_is_vivt()) __cpuc_flush_dcache_area((void *)vaddr, PAGE_SIZE); +#ifdef CONFIG_PREEMPT_RT_FULL + current->kmap_pte[type] = __pte(0); +#endif #ifdef CONFIG_DEBUG_HIGHMEM BUG_ON(vaddr != __fix_to_virt(idx)); - set_fixmap_pte(idx, __pte(0)); #else (void) idx; /* to kill a warning */ #endif + set_fixmap_pte(idx, __pte(0)); kmap_atomic_idx_pop(); } else if (vaddr >= PKMAP_ADDR(0) && vaddr < PKMAP_ADDR(LAST_PKMAP)) { /* this address was obtained through kmap_high_get() */ kunmap_high(pte_page(pkmap_page_table[PKMAP_NR(vaddr)])); } pagefault_enable(); - preempt_enable(); + preempt_enable_nort(); } EXPORT_SYMBOL(__kunmap_atomic); void *kmap_atomic_pfn(unsigned long pfn) { + pte_t pte = pfn_pte(pfn, kmap_prot); unsigned long vaddr; int idx, type; struct page *page = pfn_to_page(pfn); - preempt_disable(); + preempt_disable_nort(); pagefault_disable(); if (!PageHighMem(page)) return page_address(page); type = kmap_atomic_idx_push(); - idx = FIX_KMAP_BEGIN + type + KM_TYPE_NR * smp_processor_id(); + idx = fixmap_idx(type); vaddr = __fix_to_virt(idx); #ifdef CONFIG_DEBUG_HIGHMEM BUG_ON(!pte_none(get_fixmap_pte(vaddr))); #endif - set_fixmap_pte(idx, pfn_pte(pfn, kmap_prot)); +#ifdef CONFIG_PREEMPT_RT_FULL + current->kmap_pte[type] = pte; +#endif + set_fixmap_pte(idx, pte); return (void *)vaddr; } +#if defined CONFIG_PREEMPT_RT_FULL +void switch_kmaps(struct task_struct *prev_p, struct task_struct *next_p) +{ + int i; + + /* + * Clear @prev's kmap_atomic mappings + */ + for (i = 0; i < prev_p->kmap_idx; i++) { + int idx = fixmap_idx(i); + + set_fixmap_pte(idx, __pte(0)); + } + /* + * Restore @next_p's kmap_atomic mappings + */ + for (i = 0; i < next_p->kmap_idx; i++) { + int idx = fixmap_idx(i); + + if (!pte_none(next_p->kmap_pte[i])) + set_fixmap_pte(idx, next_p->kmap_pte[i]); + } +} +#endif Index: linux-4.4.25-rt35/arch/arm/plat-versatile/platsmp.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm/plat-versatile/platsmp.c +++ linux-4.4.25-rt35/arch/arm/plat-versatile/platsmp.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:33 @ static void write_pen_release(int val) sync_cache_w(&pen_release); } -static DEFINE_SPINLOCK(boot_lock); +static DEFINE_RAW_SPINLOCK(boot_lock); void versatile_secondary_init(unsigned int cpu) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:46 @ void versatile_secondary_init(unsigned i /* * Synchronise with the boot thread. */ - spin_lock(&boot_lock); - spin_unlock(&boot_lock); + raw_spin_lock(&boot_lock); + raw_spin_unlock(&boot_lock); } int versatile_boot_secondary(unsigned int cpu, struct task_struct *idle) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:58 @ int versatile_boot_secondary(unsigned in * Set synchronisation state between this boot processor * and the secondary one */ - spin_lock(&boot_lock); + raw_spin_lock(&boot_lock); /* * This is really belt and braces; we hold unintended secondary @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:88 @ int versatile_boot_secondary(unsigned in * now the secondary core is starting up let it run its * calibrations, then wait for it to finish */ - spin_unlock(&boot_lock); + raw_spin_unlock(&boot_lock); return pen_release != -1 ? -ENOSYS : 0; } Index: linux-4.4.25-rt35/arch/arm64/Kconfig =================================================================== --- linux-4.4.25-rt35.orig/arch/arm64/Kconfig +++ linux-4.4.25-rt35/arch/arm64/Kconfig @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:79 @ config ARM64 select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP select HAVE_RCU_TABLE_FREE + select HAVE_PREEMPT_LAZY select HAVE_SYSCALL_TRACEPOINTS select IOMMU_DMA if IOMMU_SUPPORT select IRQ_DOMAIN @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:586 @ config XEN_DOM0 config XEN bool "Xen guest support on ARM64" - depends on ARM64 && OF + depends on ARM64 && OF && !PREEMPT_RT_FULL select SWIOTLB_XEN help Say Y if you want to run Linux in a Virtual Machine on Xen on ARM64. Index: linux-4.4.25-rt35/arch/arm64/include/asm/thread_info.h =================================================================== --- linux-4.4.25-rt35.orig/arch/arm64/include/asm/thread_info.h +++ linux-4.4.25-rt35/arch/arm64/include/asm/thread_info.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:52 @ struct thread_info { mm_segment_t addr_limit; /* address limit */ struct task_struct *task; /* main task structure */ int preempt_count; /* 0 => preemptable, <0 => bug */ + int preempt_lazy_count; /* 0 => preemptable, <0 => bug */ int cpu; /* cpu */ }; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:107 @ static inline struct thread_info *curren #define TIF_NEED_RESCHED 1 #define TIF_NOTIFY_RESUME 2 /* callback before returning to user */ #define TIF_FOREIGN_FPSTATE 3 /* CPU's FP state is not current's */ +#define TIF_NEED_RESCHED_LAZY 4 #define TIF_NOHZ 7 #define TIF_SYSCALL_TRACE 8 #define TIF_SYSCALL_AUDIT 9 @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:123 @ static inline struct thread_info *curren #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_FOREIGN_FPSTATE (1 << TIF_FOREIGN_FPSTATE) +#define _TIF_NEED_RESCHED_LAZY (1 << TIF_NEED_RESCHED_LAZY) #define _TIF_NOHZ (1 << TIF_NOHZ) #define _TIF_SYSCALL_TRACE (1 << TIF_SYSCALL_TRACE) #define _TIF_SYSCALL_AUDIT (1 << TIF_SYSCALL_AUDIT) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:132 @ static inline struct thread_info *curren #define _TIF_32BIT (1 << TIF_32BIT) #define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ - _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE) + _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \ + _TIF_NEED_RESCHED_LAZY) #define _TIF_SYSCALL_WORK (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \ _TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP | \ Index: linux-4.4.25-rt35/arch/arm64/kernel/asm-offsets.c =================================================================== --- linux-4.4.25-rt35.orig/arch/arm64/kernel/asm-offsets.c +++ linux-4.4.25-rt35/arch/arm64/kernel/asm-offsets.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:38 @ int main(void) BLANK(); DEFINE(TI_FLAGS, offsetof(struct thread_info, flags)); DEFINE(TI_PREEMPT, offsetof(struct thread_info, preempt_count)); + DEFINE(TI_PREEMPT_LAZY, offsetof(struct thread_info, preempt_lazy_count)); DEFINE(TI_ADDR_LIMIT, offsetof(struct thread_info, addr_limit)); DEFINE(TI_TASK, offsetof(struct thread_info, task)); DEFINE(TI_CPU, offsetof(struct thread_info, cpu)); Index: linux-4.4.25-rt35/arch/arm64/kernel/entry.S =================================================================== --- linux-4.4.25-rt35.orig/arch/arm64/kernel/entry.S +++ linux-4.4.25-rt35/arch/arm64/kernel/entry.S @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:379 @ el1_irq: #ifdef CONFIG_PREEMPT get_thread_info tsk ldr w24, [tsk, #TI_PREEMPT] // get preempt count - cbnz w24, 1f // preempt count != 0 + cbnz w24, 2f // preempt count != 0 ldr x0, [tsk, #TI_FLAGS] // get flags - tbz x0, #TIF_NEED_RESCHED, 1f // needs rescheduling? - bl el1_preempt + tbnz x0, #TIF_NEED_RESCHED, 1f // needs rescheduling? + + ldr w24, [tsk, #TI_PREEMPT_LAZY] // get preempt lazy count + cbnz w24, 2f // preempt lazy count != 0 + tbz x0, #TIF_NEED_RESCHED_LAZY, 2f // needs rescheduling? 1: + bl el1_preempt +2: #endif #ifdef CONFIG_TRACE_IRQFLAGS bl trace_hardirqs_on @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:402 @ el1_preempt: 1: bl preempt_schedule_irq // irq en/disable is done inside ldr x0, [tsk, #TI_FLAGS] // get new tasks TI_FLAGS tbnz x0, #TIF_NEED_RESCHED, 1b // needs rescheduling? + tbnz x0, #TIF_NEED_RESCHED_LAZY, 1b // needs rescheduling? ret x24 #endif @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:647 @ ret_fast_syscall_trace: */ work_pending: tbnz x1, #TIF_NEED_RESCHED, work_resched + tbnz x1, #TIF_NEED_RESCHED_LAZY, work_resched /* TIF_SIGPENDING, TIF_NOTIFY_RESUME or TIF_FOREIGN_FPSTATE case */ ldr x2, [sp, #S_PSTATE] mov x0, sp // 'regs' Index: linux-4.4.25-rt35/arch/mips/Kconfig =================================================================== --- linux-4.4.25-rt35.orig/arch/mips/Kconfig +++ linux-4.4.25-rt35/arch/mips/Kconfig @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2413 @ config CPU_R4400_WORKAROUNDS # config HIGHMEM bool "High Memory Support" - depends on 32BIT && CPU_SUPPORTS_HIGHMEM && SYS_SUPPORTS_HIGHMEM && !CPU_MIPS32_3_5_EVA + depends on 32BIT && CPU_SUPPORTS_HIGHMEM && SYS_SUPPORTS_HIGHMEM && !CPU_MIPS32_3_5_EVA && !PREEMPT_RT_FULL config CPU_SUPPORTS_HIGHMEM bool Index: linux-4.4.25-rt35/arch/mips/kvm/mips.c =================================================================== --- linux-4.4.25-rt35.orig/arch/mips/kvm/mips.c +++ linux-4.4.25-rt35/arch/mips/kvm/mips.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:457 @ int kvm_vcpu_ioctl_interrupt(struct kvm_ dvcpu->arch.wait = 0; - if (waitqueue_active(&dvcpu->wq)) - wake_up_interruptible(&dvcpu->wq); + if (swait_active(&dvcpu->wq)) + swake_up(&dvcpu->wq); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1186 @ static void kvm_mips_comparecount_func(u kvm_mips_callbacks->queue_timer_int(vcpu); vcpu->arch.wait = 0; - if (waitqueue_active(&vcpu->wq)) - wake_up_interruptible(&vcpu->wq); + if (swait_active(&vcpu->wq)) + swake_up(&vcpu->wq); } /* low level hrtimer wake routine */ Index: linux-4.4.25-rt35/arch/powerpc/Kconfig =================================================================== --- linux-4.4.25-rt35.orig/arch/powerpc/Kconfig +++ linux-4.4.25-rt35/arch/powerpc/Kconfig @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:63 @ config LOCKDEP_SUPPORT config RWSEM_GENERIC_SPINLOCK bool + default y if PREEMPT_RT_FULL config RWSEM_XCHGADD_ALGORITHM bool - default y + default y if !PREEMPT_RT_FULL config GENERIC_LOCKBREAK bool @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:145 @ config PPC select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST select GENERIC_STRNCPY_FROM_USER select GENERIC_STRNLEN_USER + select HAVE_PREEMPT_LAZY select HAVE_MOD_ARCH_SPECIFIC select MODULES_USE_ELF_RELA select CLONE_BACKWARDS @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:324 @ menu "Kernel options" config HIGHMEM bool "High memory support" - depends on PPC32 + depends on PPC32 && !PREEMPT_RT_FULL source kernel/Kconfig.hz source kernel/Kconfig.preempt Index: linux-4.4.25-rt35/arch/powerpc/include/asm/kvm_host.h =================================================================== --- linux-4.4.25-rt35.orig/arch/powerpc/include/asm/kvm_host.h +++ linux-4.4.25-rt35/arch/powerpc/include/asm/kvm_host.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:289 @ struct kvmppc_vcore { struct list_head runnable_threads; struct list_head preempt_list; spinlock_t lock; - wait_queue_head_t wq; + struct swait_queue_head wq; spinlock_t stoltb_lock; /* protects stolen_tb and preempt_tb */ u64 stolen_tb; u64 preempt_tb; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:629 @ struct kvm_vcpu_arch { u8 prodded; u32 last_inst; - wait_queue_head_t *wqp; + struct swait_queue_head *wqp; struct kvmppc_vcore *vcore; int ret; int trap; Index: linux-4.4.25-rt35/arch/powerpc/include/asm/thread_info.h =================================================================== --- linux-4.4.25-rt35.orig/arch/powerpc/include/asm/thread_info.h +++ linux-4.4.25-rt35/arch/powerpc/include/asm/thread_info.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:45 @ struct thread_info { int cpu; /* cpu we're on */ int preempt_count; /* 0 => preemptable, <0 => BUG */ + int preempt_lazy_count; /* 0 => preemptable, + <0 => BUG */ unsigned long local_flags; /* private flags for thread */ /* low level flags - has atomic operations done on it */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:87 @ static inline struct thread_info *curren #define TIF_SYSCALL_TRACE 0 /* syscall trace active */ #define TIF_SIGPENDING 1 /* signal pending */ #define TIF_NEED_RESCHED 2 /* rescheduling necessary */ -#define TIF_POLLING_NRFLAG 3 /* true if poll_idle() is polling - TIF_NEED_RESCHED */ +#define TIF_NEED_RESCHED_LAZY 3 /* lazy rescheduling necessary */ #define TIF_32BIT 4 /* 32 bit binary */ #define TIF_RESTORE_TM 5 /* need to restore TM FP/VEC/VSX */ #define TIF_SYSCALL_AUDIT 7 /* syscall auditing active */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:105 @ static inline struct thread_info *curren #if defined(CONFIG_PPC64) #define TIF_ELF2ABI 18 /* function descriptors must die! */ #endif +#define TIF_POLLING_NRFLAG 19 /* true if poll_idle() is polling + TIF_NEED_RESCHED */ /* as above, but as bit values */ #define _TIF_SYSCALL_TRACE (1<<TIF_SYSCALL_TRACE) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:125 @ static inline struct thread_info *curren #define _TIF_SYSCALL_TRACEPOINT (1<<TIF_SYSCALL_TRACEPOINT) #define _TIF_EMULATE_STACK_STORE (1<<TIF_EMULATE_STACK_STORE) #define _TIF_NOHZ (1<<TIF_NOHZ) +#define _TIF_NEED_RESCHED_LAZY (1<<TIF_NEED_RESCHED_LAZY) #define _TIF_SYSCALL_DOTRACE (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \ _TIF_SECCOMP | _TIF_SYSCALL_TRACEPOINT | \ _TIF_NOHZ) #define _TIF_USER_WORK_MASK (_TIF_SIGPENDING | _TIF_NEED_RESCHED | \ _TIF_NOTIFY_RESUME | _TIF_UPROBE | \ - _TIF_RESTORE_TM) + _TIF_RESTORE_TM | _TIF_NEED_RESCHED_LAZY) #define _TIF_PERSYSCALL_MASK (_TIF_RESTOREALL|_TIF_NOERROR) +#define _TIF_NEED_RESCHED_MASK (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY) /* Bits in local_flags */ /* Don't move TLF_NAPPING without adjusting the code in entry_32.S */ Index: linux-4.4.25-rt35/arch/powerpc/kernel/asm-offsets.c =================================================================== --- linux-4.4.25-rt35.orig/arch/powerpc/kernel/asm-offsets.c +++ linux-4.4.25-rt35/arch/powerpc/kernel/asm-offsets.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:163 @ int main(void) DEFINE(TI_FLAGS, offsetof(struct thread_info, flags)); DEFINE(TI_LOCAL_FLAGS, offsetof(struct thread_info, local_flags)); DEFINE(TI_PREEMPT, offsetof(struct thread_info, preempt_count)); + DEFINE(TI_PREEMPT_LAZY, offsetof(struct thread_info, preempt_lazy_count)); DEFINE(TI_TASK, offsetof(struct thread_info, task)); DEFINE(TI_CPU, offsetof(struct thread_info, cpu)); Index: linux-4.4.25-rt35/arch/powerpc/kernel/entry_32.S =================================================================== --- linux-4.4.25-rt35.orig/arch/powerpc/kernel/entry_32.S +++ linux-4.4.25-rt35/arch/powerpc/kernel/entry_32.S @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:821 @ resume_kernel: cmpwi 0,r0,0 /* if non-zero, just restore regs and return */ bne restore andi. r8,r8,_TIF_NEED_RESCHED + bne+ 1f + lwz r0,TI_PREEMPT_LAZY(r9) + cmpwi 0,r0,0 /* if non-zero, just restore regs and return */ + bne restore + lwz r0,TI_FLAGS(r9) + andi. r0,r0,_TIF_NEED_RESCHED_LAZY beq+ restore +1: lwz r3,_MSR(r1) andi. r0,r3,MSR_EE /* interrupts off? */ beq restore /* don't schedule if so */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:839 @ resume_kernel: */ bl trace_hardirqs_off #endif -1: bl preempt_schedule_irq +2: bl preempt_schedule_irq CURRENT_THREAD_INFO(r9, r1) lwz r3,TI_FLAGS(r9) - andi. r0,r3,_TIF_NEED_RESCHED - bne- 1b + andi. r0,r3,_TIF_NEED_RESCHED_MASK + bne- 2b #ifdef CONFIG_TRACE_IRQFLAGS /* And now, to properly rebalance the above, we tell lockdep they * are being turned back on, which will happen when we return @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1164 @ global_dbcr0: #endif /* !(CONFIG_4xx || CONFIG_BOOKE) */ do_work: /* r10 contains MSR_KERNEL here */ - andi. r0,r9,_TIF_NEED_RESCHED + andi. r0,r9,_TIF_NEED_RESCHED_MASK beq do_user_signal do_resched: /* r10 contains MSR_KERNEL here */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1185 @ recheck: MTMSRD(r10) /* disable interrupts */ CURRENT_THREAD_INFO(r9, r1) lwz r9,TI_FLAGS(r9) - andi. r0,r9,_TIF_NEED_RESCHED + andi. r0,r9,_TIF_NEED_RESCHED_MASK bne- do_resched andi. r0,r9,_TIF_USER_WORK_MASK beq restore_user Index: linux-4.4.25-rt35/arch/powerpc/kernel/entry_64.S =================================================================== --- linux-4.4.25-rt35.orig/arch/powerpc/kernel/entry_64.S +++ linux-4.4.25-rt35/arch/powerpc/kernel/entry_64.S @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:686 @ _GLOBAL(ret_from_except_lite) #else beq restore #endif -1: andi. r0,r4,_TIF_NEED_RESCHED +1: andi. r0,r4,_TIF_NEED_RESCHED_MASK beq 2f bl restore_interrupts SCHEDULE_USER @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:748 @ resume_kernel: #ifdef CONFIG_PREEMPT /* Check if we need to preempt */ + lwz r8,TI_PREEMPT(r9) + cmpwi 0,r8,0 /* if non-zero, just restore regs and return */ + bne restore andi. r0,r4,_TIF_NEED_RESCHED + bne+ check_count + + andi. r0,r4,_TIF_NEED_RESCHED_LAZY beq+ restore + lwz r8,TI_PREEMPT_LAZY(r9) + /* Check that preempt_count() == 0 and interrupts are enabled */ - lwz r8,TI_PREEMPT(r9) +check_count: cmpwi cr1,r8,0 ld r0,SOFTE(r1) cmpdi r0,0 @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:776 @ resume_kernel: /* Re-test flags and eventually loop */ CURRENT_THREAD_INFO(r9, r1) ld r4,TI_FLAGS(r9) - andi. r0,r4,_TIF_NEED_RESCHED + andi. r0,r4,_TIF_NEED_RESCHED_MASK bne 1b /* Index: linux-4.4.25-rt35/arch/powerpc/kernel/irq.c =================================================================== --- linux-4.4.25-rt35.orig/arch/powerpc/kernel/irq.c +++ linux-4.4.25-rt35/arch/powerpc/kernel/irq.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:617 @ void irq_ctx_init(void) } } +#ifndef CONFIG_PREEMPT_RT_FULL void do_softirq_own_stack(void) { struct thread_info *curtp, *irqtp; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:635 @ void do_softirq_own_stack(void) if (irqtp->flags) set_bits(irqtp->flags, &curtp->flags); } +#endif irq_hw_number_t virq_to_hw(unsigned int virq) { Index: linux-4.4.25-rt35/arch/powerpc/kernel/misc_32.S =================================================================== --- linux-4.4.25-rt35.orig/arch/powerpc/kernel/misc_32.S +++ linux-4.4.25-rt35/arch/powerpc/kernel/misc_32.S @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:43 @ * We store the saved ksp_limit in the unused part * of the STACK_FRAME_OVERHEAD */ +#ifndef CONFIG_PREEMPT_RT_FULL _GLOBAL(call_do_softirq) mflr r0 stw r0,4(r1) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:60 @ _GLOBAL(call_do_softirq) stw r10,THREAD+KSP_LIMIT(r2) mtlr r0 blr +#endif /* * void call_do_irq(struct pt_regs *regs, struct thread_info *irqtp); Index: linux-4.4.25-rt35/arch/powerpc/kernel/misc_64.S =================================================================== --- linux-4.4.25-rt35.orig/arch/powerpc/kernel/misc_64.S +++ linux-4.4.25-rt35/arch/powerpc/kernel/misc_64.S @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:33 @ .text +#ifndef CONFIG_PREEMPT_RT_FULL _GLOBAL(call_do_softirq) mflr r0 std r0,16(r1) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:44 @ _GLOBAL(call_do_softirq) ld r0,16(r1) mtlr r0 blr +#endif _GLOBAL(call_do_irq) mflr r0 Index: linux-4.4.25-rt35/arch/powerpc/kvm/Kconfig =================================================================== --- linux-4.4.25-rt35.orig/arch/powerpc/kvm/Kconfig +++ linux-4.4.25-rt35/arch/powerpc/kvm/Kconfig @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:175 @ config KVM_E500MC config KVM_MPIC bool "KVM in-kernel MPIC emulation" depends on KVM && E500 + depends on !PREEMPT_RT_FULL select HAVE_KVM_IRQCHIP select HAVE_KVM_IRQFD select HAVE_KVM_IRQ_ROUTING Index: linux-4.4.25-rt35/arch/powerpc/kvm/book3s_hv.c =================================================================== --- linux-4.4.25-rt35.orig/arch/powerpc/kvm/book3s_hv.c +++ linux-4.4.25-rt35/arch/powerpc/kvm/book3s_hv.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:117 @ static bool kvmppc_ipi_thread(int cpu) static void kvmppc_fast_vcpu_kick_hv(struct kvm_vcpu *vcpu) { int cpu; - wait_queue_head_t *wqp; + struct swait_queue_head *wqp; wqp = kvm_arch_vcpu_wq(vcpu); - if (waitqueue_active(wqp)) { - wake_up_interruptible(wqp); + if (swait_active(wqp)) { + swake_up(wqp); ++vcpu->stat.halt_wakeup; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:710 @ int kvmppc_pseries_do_hcall(struct kvm_v tvcpu->arch.prodded = 1; smp_mb(); if (vcpu->arch.ceded) { - if (waitqueue_active(&vcpu->wq)) { - wake_up_interruptible(&vcpu->wq); + if (swait_active(&vcpu->wq)) { + swake_up(&vcpu->wq); vcpu->stat.halt_wakeup++; } } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1450 @ static struct kvmppc_vcore *kvmppc_vcore INIT_LIST_HEAD(&vcore->runnable_threads); spin_lock_init(&vcore->lock); spin_lock_init(&vcore->stoltb_lock); - init_waitqueue_head(&vcore->wq); + init_swait_queue_head(&vcore->wq); vcore->preempt_tb = TB_NIL; vcore->lpcr = kvm->arch.lpcr; vcore->first_vcpuid = core * threads_per_subcore; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2522 @ static void kvmppc_vcore_blocked(struct { struct kvm_vcpu *vcpu; int do_sleep = 1; + DECLARE_SWAITQUEUE(wait); - DEFINE_WAIT(wait); - - prepare_to_wait(&vc->wq, &wait, TASK_INTERRUPTIBLE); + prepare_to_swait(&vc->wq, &wait, TASK_INTERRUPTIBLE); /* * Check one last time for pending exceptions and ceded state after @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2538 @ static void kvmppc_vcore_blocked(struct } if (!do_sleep) { - finish_wait(&vc->wq, &wait); + finish_swait(&vc->wq, &wait); return; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2546 @ static void kvmppc_vcore_blocked(struct trace_kvmppc_vcore_blocked(vc, 0); spin_unlock(&vc->lock); schedule(); - finish_wait(&vc->wq, &wait); + finish_swait(&vc->wq, &wait); spin_lock(&vc->lock); vc->vcore_state = VCORE_INACTIVE; trace_kvmppc_vcore_blocked(vc, 1); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2602 @ static int kvmppc_run_vcpu(struct kvm_ru kvmppc_start_thread(vcpu, vc); trace_kvm_guest_enter(vcpu); } else if (vc->vcore_state == VCORE_SLEEPING) { - wake_up(&vc->wq); + swake_up(&vc->wq); } } Index: linux-4.4.25-rt35/arch/powerpc/platforms/ps3/device-init.c =================================================================== --- linux-4.4.25-rt35.orig/arch/powerpc/platforms/ps3/device-init.c +++ linux-4.4.25-rt35/arch/powerpc/platforms/ps3/device-init.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:755 @ static int ps3_notification_read_write(s } pr_debug("%s:%u: notification %s issued\n", __func__, __LINE__, op); - res = wait_event_interruptible(dev->done.wait, + res = swait_event_interruptible(dev->done.wait, dev->done.done || kthread_should_stop()); if (kthread_should_stop()) res = -EINTR; Index: linux-4.4.25-rt35/arch/s390/include/asm/kvm_host.h =================================================================== --- linux-4.4.25-rt35.orig/arch/s390/include/asm/kvm_host.h +++ linux-4.4.25-rt35/arch/s390/include/asm/kvm_host.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:430 @ struct kvm_s390_irq_payload { struct kvm_s390_local_interrupt { spinlock_t lock; struct kvm_s390_float_interrupt *float_int; - wait_queue_head_t *wq; + struct swait_queue_head *wq; atomic_t *cpuflags; DECLARE_BITMAP(sigp_emerg_pending, KVM_MAX_VCPUS); struct kvm_s390_irq_payload irq; Index: linux-4.4.25-rt35/arch/s390/kvm/interrupt.c =================================================================== --- linux-4.4.25-rt35.orig/arch/s390/kvm/interrupt.c +++ linux-4.4.25-rt35/arch/s390/kvm/interrupt.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:871 @ no_timer: void kvm_s390_vcpu_wakeup(struct kvm_vcpu *vcpu) { - if (waitqueue_active(&vcpu->wq)) { + if (swait_active(&vcpu->wq)) { /* * The vcpu gave up the cpu voluntarily, mark it as a good * yield-candidate. */ vcpu->preempted = true; - wake_up_interruptible(&vcpu->wq); + swake_up(&vcpu->wq); vcpu->stat.halt_wakeup++; } } Index: linux-4.4.25-rt35/arch/sh/kernel/irq.c =================================================================== --- linux-4.4.25-rt35.orig/arch/sh/kernel/irq.c +++ linux-4.4.25-rt35/arch/sh/kernel/irq.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:150 @ void irq_ctx_exit(int cpu) hardirq_ctx[cpu] = NULL; } +#ifndef CONFIG_PREEMPT_RT_FULL void do_softirq_own_stack(void) { struct thread_info *curctx; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:178 @ void do_softirq_own_stack(void) "r5", "r6", "r7", "r8", "r9", "r15", "t", "pr" ); } +#endif #else static inline void handle_one_irq(unsigned int irq) { Index: linux-4.4.25-rt35/arch/sparc/Kconfig =================================================================== --- linux-4.4.25-rt35.orig/arch/sparc/Kconfig +++ linux-4.4.25-rt35/arch/sparc/Kconfig @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:192 @ config NR_CPUS source kernel/Kconfig.hz config RWSEM_GENERIC_SPINLOCK - bool - default y if SPARC32 + def_bool PREEMPT_RT_FULL config RWSEM_XCHGADD_ALGORITHM - bool - default y if SPARC64 + def_bool !RWSEM_GENERIC_SPINLOCK && !PREEMPT_RT_FULL config GENERIC_HWEIGHT bool Index: linux-4.4.25-rt35/arch/sparc/kernel/irq_64.c =================================================================== --- linux-4.4.25-rt35.orig/arch/sparc/kernel/irq_64.c +++ linux-4.4.25-rt35/arch/sparc/kernel/irq_64.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:857 @ void __irq_entry handler_irq(int pil, st set_irq_regs(old_regs); } +#ifndef CONFIG_PREEMPT_RT_FULL void do_softirq_own_stack(void) { void *orig_sp, *sp = softirq_stack[smp_processor_id()]; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:872 @ void do_softirq_own_stack(void) __asm__ __volatile__("mov %0, %%sp" : : "r" (orig_sp)); } +#endif #ifdef CONFIG_HOTPLUG_CPU void fixup_irqs(void) Index: linux-4.4.25-rt35/arch/x86/Kconfig =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/Kconfig +++ linux-4.4.25-rt35/arch/x86/Kconfig @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:20 @ config X86_64 ### Arch settings config X86 def_bool y + select HAVE_PREEMPT_LAZY select ACPI_LEGACY_TABLES_LOOKUP if ACPI select ACPI_SYSTEM_POWER_STATES_SUPPORT if ACPI select ANON_INODES @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:216 @ config ARCH_MAY_HAVE_PC_FDC def_bool y depends on ISA_DMA_API +config RWSEM_GENERIC_SPINLOCK + def_bool PREEMPT_RT_FULL + config RWSEM_XCHGADD_ALGORITHM - def_bool y + def_bool !RWSEM_GENERIC_SPINLOCK && !PREEMPT_RT_FULL config GENERIC_CALIBRATE_DELAY def_bool y @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:855 @ config IOMMU_HELPER config MAXSMP bool "Enable Maximum number of SMP Processors and NUMA Nodes" depends on X86_64 && SMP && DEBUG_KERNEL - select CPUMASK_OFFSTACK + select CPUMASK_OFFSTACK if !PREEMPT_RT_FULL ---help--- Enable maximum number of CPUS and NUMA Nodes for this architecture. If unsure, say N. Index: linux-4.4.25-rt35/arch/x86/crypto/aesni-intel_glue.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/crypto/aesni-intel_glue.c +++ linux-4.4.25-rt35/arch/x86/crypto/aesni-intel_glue.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:386 @ static int ecb_encrypt(struct blkcipher_ err = blkcipher_walk_virt(desc, &walk); desc->flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP; - kernel_fpu_begin(); while ((nbytes = walk.nbytes)) { + kernel_fpu_begin(); aesni_ecb_enc(ctx, walk.dst.virt.addr, walk.src.virt.addr, - nbytes & AES_BLOCK_MASK); + nbytes & AES_BLOCK_MASK); + kernel_fpu_end(); nbytes &= AES_BLOCK_SIZE - 1; err = blkcipher_walk_done(desc, &walk, nbytes); } - kernel_fpu_end(); return err; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:410 @ static int ecb_decrypt(struct blkcipher_ err = blkcipher_walk_virt(desc, &walk); desc->flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP; - kernel_fpu_begin(); while ((nbytes = walk.nbytes)) { + kernel_fpu_begin(); aesni_ecb_dec(ctx, walk.dst.virt.addr, walk.src.virt.addr, nbytes & AES_BLOCK_MASK); + kernel_fpu_end(); nbytes &= AES_BLOCK_SIZE - 1; err = blkcipher_walk_done(desc, &walk, nbytes); } - kernel_fpu_end(); return err; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:434 @ static int cbc_encrypt(struct blkcipher_ err = blkcipher_walk_virt(desc, &walk); desc->flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP; - kernel_fpu_begin(); while ((nbytes = walk.nbytes)) { + kernel_fpu_begin(); aesni_cbc_enc(ctx, walk.dst.virt.addr, walk.src.virt.addr, nbytes & AES_BLOCK_MASK, walk.iv); + kernel_fpu_end(); nbytes &= AES_BLOCK_SIZE - 1; err = blkcipher_walk_done(desc, &walk, nbytes); } - kernel_fpu_end(); return err; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:458 @ static int cbc_decrypt(struct blkcipher_ err = blkcipher_walk_virt(desc, &walk); desc->flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP; - kernel_fpu_begin(); while ((nbytes = walk.nbytes)) { + kernel_fpu_begin(); aesni_cbc_dec(ctx, walk.dst.virt.addr, walk.src.virt.addr, nbytes & AES_BLOCK_MASK, walk.iv); + kernel_fpu_end(); nbytes &= AES_BLOCK_SIZE - 1; err = blkcipher_walk_done(desc, &walk, nbytes); } - kernel_fpu_end(); return err; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:517 @ static int ctr_crypt(struct blkcipher_de err = blkcipher_walk_virt_block(desc, &walk, AES_BLOCK_SIZE); desc->flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP; - kernel_fpu_begin(); while ((nbytes = walk.nbytes) >= AES_BLOCK_SIZE) { + kernel_fpu_begin(); aesni_ctr_enc_tfm(ctx, walk.dst.virt.addr, walk.src.virt.addr, nbytes & AES_BLOCK_MASK, walk.iv); + kernel_fpu_end(); nbytes &= AES_BLOCK_SIZE - 1; err = blkcipher_walk_done(desc, &walk, nbytes); } if (walk.nbytes) { + kernel_fpu_begin(); ctr_crypt_final(ctx, &walk); + kernel_fpu_end(); err = blkcipher_walk_done(desc, &walk, 0); } - kernel_fpu_end(); return err; } Index: linux-4.4.25-rt35/arch/x86/crypto/cast5_avx_glue.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/crypto/cast5_avx_glue.c +++ linux-4.4.25-rt35/arch/x86/crypto/cast5_avx_glue.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:62 @ static inline void cast5_fpu_end(bool fp static int ecb_crypt(struct blkcipher_desc *desc, struct blkcipher_walk *walk, bool enc) { - bool fpu_enabled = false; + bool fpu_enabled; struct cast5_ctx *ctx = crypto_blkcipher_ctx(desc->tfm); const unsigned int bsize = CAST5_BLOCK_SIZE; unsigned int nbytes; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:78 @ static int ecb_crypt(struct blkcipher_de u8 *wsrc = walk->src.virt.addr; u8 *wdst = walk->dst.virt.addr; - fpu_enabled = cast5_fpu_begin(fpu_enabled, nbytes); + fpu_enabled = cast5_fpu_begin(false, nbytes); /* Process multi-block batch */ if (nbytes >= bsize * CAST5_PARALLEL_BLOCKS) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:106 @ static int ecb_crypt(struct blkcipher_de } while (nbytes >= bsize); done: + cast5_fpu_end(fpu_enabled); err = blkcipher_walk_done(desc, walk, nbytes); } - - cast5_fpu_end(fpu_enabled); return err; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:229 @ done: static int cbc_decrypt(struct blkcipher_desc *desc, struct scatterlist *dst, struct scatterlist *src, unsigned int nbytes) { - bool fpu_enabled = false; + bool fpu_enabled; struct blkcipher_walk walk; int err; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:238 @ static int cbc_decrypt(struct blkcipher_ desc->flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP; while ((nbytes = walk.nbytes)) { - fpu_enabled = cast5_fpu_begin(fpu_enabled, nbytes); + fpu_enabled = cast5_fpu_begin(false, nbytes); nbytes = __cbc_decrypt(desc, &walk); + cast5_fpu_end(fpu_enabled); err = blkcipher_walk_done(desc, &walk, nbytes); } - - cast5_fpu_end(fpu_enabled); return err; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:312 @ done: static int ctr_crypt(struct blkcipher_desc *desc, struct scatterlist *dst, struct scatterlist *src, unsigned int nbytes) { - bool fpu_enabled = false; + bool fpu_enabled; struct blkcipher_walk walk; int err; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:321 @ static int ctr_crypt(struct blkcipher_de desc->flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP; while ((nbytes = walk.nbytes) >= CAST5_BLOCK_SIZE) { - fpu_enabled = cast5_fpu_begin(fpu_enabled, nbytes); + fpu_enabled = cast5_fpu_begin(false, nbytes); nbytes = __ctr_crypt(desc, &walk); + cast5_fpu_end(fpu_enabled); err = blkcipher_walk_done(desc, &walk, nbytes); } - cast5_fpu_end(fpu_enabled); - if (walk.nbytes) { ctr_crypt_final(desc, &walk); err = blkcipher_walk_done(desc, &walk, 0); Index: linux-4.4.25-rt35/arch/x86/crypto/glue_helper.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/crypto/glue_helper.c +++ linux-4.4.25-rt35/arch/x86/crypto/glue_helper.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:42 @ static int __glue_ecb_crypt_128bit(const void *ctx = crypto_blkcipher_ctx(desc->tfm); const unsigned int bsize = 128 / 8; unsigned int nbytes, i, func_bytes; - bool fpu_enabled = false; + bool fpu_enabled; int err; err = blkcipher_walk_virt(desc, walk); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:52 @ static int __glue_ecb_crypt_128bit(const u8 *wdst = walk->dst.virt.addr; fpu_enabled = glue_fpu_begin(bsize, gctx->fpu_blocks_limit, - desc, fpu_enabled, nbytes); + desc, false, nbytes); for (i = 0; i < gctx->num_funcs; i++) { func_bytes = bsize * gctx->funcs[i].num_blocks; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:74 @ static int __glue_ecb_crypt_128bit(const } done: + glue_fpu_end(fpu_enabled); err = blkcipher_walk_done(desc, walk, nbytes); } - glue_fpu_end(fpu_enabled); return err; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:197 @ int glue_cbc_decrypt_128bit(const struct struct scatterlist *src, unsigned int nbytes) { const unsigned int bsize = 128 / 8; - bool fpu_enabled = false; + bool fpu_enabled; struct blkcipher_walk walk; int err; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:206 @ int glue_cbc_decrypt_128bit(const struct while ((nbytes = walk.nbytes)) { fpu_enabled = glue_fpu_begin(bsize, gctx->fpu_blocks_limit, - desc, fpu_enabled, nbytes); + desc, false, nbytes); nbytes = __glue_cbc_decrypt_128bit(gctx, desc, &walk); + glue_fpu_end(fpu_enabled); err = blkcipher_walk_done(desc, &walk, nbytes); } - glue_fpu_end(fpu_enabled); return err; } EXPORT_SYMBOL_GPL(glue_cbc_decrypt_128bit); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:280 @ int glue_ctr_crypt_128bit(const struct c struct scatterlist *src, unsigned int nbytes) { const unsigned int bsize = 128 / 8; - bool fpu_enabled = false; + bool fpu_enabled; struct blkcipher_walk walk; int err; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:289 @ int glue_ctr_crypt_128bit(const struct c while ((nbytes = walk.nbytes) >= bsize) { fpu_enabled = glue_fpu_begin(bsize, gctx->fpu_blocks_limit, - desc, fpu_enabled, nbytes); + desc, false, nbytes); nbytes = __glue_ctr_crypt_128bit(gctx, desc, &walk); + glue_fpu_end(fpu_enabled); err = blkcipher_walk_done(desc, &walk, nbytes); } - glue_fpu_end(fpu_enabled); - if (walk.nbytes) { glue_ctr_crypt_final_128bit( gctx->funcs[gctx->num_funcs - 1].fn_u.ctr, desc, &walk); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:349 @ int glue_xts_crypt_128bit(const struct c void *tweak_ctx, void *crypt_ctx) { const unsigned int bsize = 128 / 8; - bool fpu_enabled = false; + bool fpu_enabled; struct blkcipher_walk walk; int err; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:362 @ int glue_xts_crypt_128bit(const struct c /* set minimum length to bsize, for tweak_fn */ fpu_enabled = glue_fpu_begin(bsize, gctx->fpu_blocks_limit, - desc, fpu_enabled, + desc, false, nbytes < bsize ? bsize : nbytes); - /* calculate first value of T */ tweak_fn(tweak_ctx, walk.iv, walk.iv); + glue_fpu_end(fpu_enabled); while (nbytes) { + fpu_enabled = glue_fpu_begin(bsize, gctx->fpu_blocks_limit, + desc, false, nbytes); nbytes = __glue_xts_crypt_128bit(gctx, crypt_ctx, desc, &walk); + glue_fpu_end(fpu_enabled); err = blkcipher_walk_done(desc, &walk, nbytes); nbytes = walk.nbytes; } - - glue_fpu_end(fpu_enabled); - return err; } EXPORT_SYMBOL_GPL(glue_xts_crypt_128bit); Index: linux-4.4.25-rt35/arch/x86/entry/common.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/entry/common.c +++ linux-4.4.25-rt35/arch/x86/entry/common.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:223 @ long syscall_trace_enter(struct pt_regs #define EXIT_TO_USERMODE_LOOP_FLAGS \ (_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_UPROBE | \ - _TIF_NEED_RESCHED | _TIF_USER_RETURN_NOTIFY) + _TIF_NEED_RESCHED_MASK | _TIF_USER_RETURN_NOTIFY) static void exit_to_usermode_loop(struct pt_regs *regs, u32 cached_flags) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:239 @ static void exit_to_usermode_loop(struct /* We have work to do. */ local_irq_enable(); - if (cached_flags & _TIF_NEED_RESCHED) + if (cached_flags & _TIF_NEED_RESCHED_MASK) schedule(); +#ifdef ARCH_RT_DELAYS_SIGNAL_SEND + if (unlikely(current->forced_info.si_signo)) { + struct task_struct *t = current; + force_sig_info(t->forced_info.si_signo, &t->forced_info, t); + t->forced_info.si_signo = 0; + } +#endif if (cached_flags & _TIF_UPROBE) uprobe_notify_resume(regs); Index: linux-4.4.25-rt35/arch/x86/entry/entry_32.S =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/entry/entry_32.S +++ linux-4.4.25-rt35/arch/x86/entry/entry_32.S @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:281 @ END(ret_from_exception) ENTRY(resume_kernel) DISABLE_INTERRUPTS(CLBR_ANY) need_resched: + # preempt count == 0 + NEED_RS set? cmpl $0, PER_CPU_VAR(__preempt_count) +#ifndef CONFIG_PREEMPT_LAZY jnz restore_all +#else + jz test_int_off + + # atleast preempt count == 0 ? + cmpl $_PREEMPT_ENABLED,PER_CPU_VAR(__preempt_count) + jne restore_all + + cmpl $0,TI_preempt_lazy_count(%ebp) # non-zero preempt_lazy_count ? + jnz restore_all + + testl $_TIF_NEED_RESCHED_LAZY, TI_flags(%ebp) + jz restore_all +test_int_off: +#endif testl $X86_EFLAGS_IF, PT_EFLAGS(%esp) # interrupts off (exception path) ? jz restore_all call preempt_schedule_irq Index: linux-4.4.25-rt35/arch/x86/entry/entry_64.S =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/entry/entry_64.S +++ linux-4.4.25-rt35/arch/x86/entry/entry_64.S @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:582 @ retint_kernel: bt $9, EFLAGS(%rsp) /* were interrupts off? */ jnc 1f 0: cmpl $0, PER_CPU_VAR(__preempt_count) +#ifndef CONFIG_PREEMPT_LAZY jnz 1f +#else + jz do_preempt_schedule_irq + + # atleast preempt count == 0 ? + cmpl $_PREEMPT_ENABLED,PER_CPU_VAR(__preempt_count) + jnz 1f + + GET_THREAD_INFO(%rcx) + cmpl $0, TI_preempt_lazy_count(%rcx) + jnz 1f + + bt $TIF_NEED_RESCHED_LAZY,TI_flags(%rcx) + jnc 1f +do_preempt_schedule_irq: +#endif call preempt_schedule_irq jmp 0b 1: @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:886 @ bad_gs: jmp 2b .previous +#ifndef CONFIG_PREEMPT_RT_FULL /* Call softirq on interrupt stack. Interrupts are off. */ ENTRY(do_softirq_own_stack) pushq %rbp @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:899 @ ENTRY(do_softirq_own_stack) decl PER_CPU_VAR(irq_count) ret END(do_softirq_own_stack) +#endif #ifdef CONFIG_XEN idtentry xen_hypervisor_callback xen_do_hypervisor_callback has_error_code=0 Index: linux-4.4.25-rt35/arch/x86/include/asm/preempt.h =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/include/asm/preempt.h +++ linux-4.4.25-rt35/arch/x86/include/asm/preempt.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:82 @ static __always_inline void __preempt_co * a decrement which hits zero means we have no preempt_count and should * reschedule. */ -static __always_inline bool __preempt_count_dec_and_test(void) +static __always_inline bool ____preempt_count_dec_and_test(void) { GEN_UNARY_RMWcc("decl", __preempt_count, __percpu_arg(0), "e"); } +static __always_inline bool __preempt_count_dec_and_test(void) +{ + if (____preempt_count_dec_and_test()) + return true; +#ifdef CONFIG_PREEMPT_LAZY + if (current_thread_info()->preempt_lazy_count) + return false; + return test_thread_flag(TIF_NEED_RESCHED_LAZY); +#else + return false; +#endif +} + /* * Returns true when we need to resched and can (barring IRQ state). */ static __always_inline bool should_resched(int preempt_offset) { +#ifdef CONFIG_PREEMPT_LAZY + u32 tmp; + + tmp = raw_cpu_read_4(__preempt_count); + if (tmp == preempt_offset) + return true; + + /* preempt count == 0 ? */ + tmp &= ~PREEMPT_NEED_RESCHED; + if (tmp) + return false; + if (current_thread_info()->preempt_lazy_count) + return false; + return test_thread_flag(TIF_NEED_RESCHED_LAZY); +#else return unlikely(raw_cpu_read_4(__preempt_count) == preempt_offset); +#endif } #ifdef CONFIG_PREEMPT Index: linux-4.4.25-rt35/arch/x86/include/asm/signal.h =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/include/asm/signal.h +++ linux-4.4.25-rt35/arch/x86/include/asm/signal.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:26 @ typedef struct { unsigned long sig[_NSIG_WORDS]; } sigset_t; +/* + * Because some traps use the IST stack, we must keep preemption + * disabled while calling do_trap(), but do_trap() may call + * force_sig_info() which will grab the signal spin_locks for the + * task, which in PREEMPT_RT_FULL are mutexes. By defining + * ARCH_RT_DELAYS_SIGNAL_SEND the force_sig_info() will set + * TIF_NOTIFY_RESUME and set up the signal to be sent on exit of the + * trap. + */ +#if defined(CONFIG_PREEMPT_RT_FULL) +#define ARCH_RT_DELAYS_SIGNAL_SEND +#endif + #ifndef CONFIG_COMPAT typedef sigset_t compat_sigset_t; #endif Index: linux-4.4.25-rt35/arch/x86/include/asm/stackprotector.h =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/include/asm/stackprotector.h +++ linux-4.4.25-rt35/arch/x86/include/asm/stackprotector.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:62 @ */ static __always_inline void boot_init_stack_canary(void) { - u64 canary; + u64 uninitialized_var(canary); u64 tsc; #ifdef CONFIG_X86_64 @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:73 @ static __always_inline void boot_init_st * of randomness. The TSC only matters for very early init, * there it already has some randomness on most systems. Later * on during the bootup the random pool has true entropy too. + * + * For preempt-rt we need to weaken the randomness a bit, as + * we can't call into the random generator from atomic context + * due to locking constraints. We just leave canary + * uninitialized and use the TSC based randomness on top of it. */ +#ifndef CONFIG_PREEMPT_RT_FULL get_random_bytes(&canary, sizeof(canary)); +#endif tsc = rdtsc(); canary += tsc + (tsc << 32UL); Index: linux-4.4.25-rt35/arch/x86/include/asm/thread_info.h =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/include/asm/thread_info.h +++ linux-4.4.25-rt35/arch/x86/include/asm/thread_info.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:61 @ struct thread_info { __u32 status; /* thread synchronous flags */ __u32 cpu; /* current CPU */ mm_segment_t addr_limit; + int preempt_lazy_count; /* 0 => lazy preemptable + <0 => BUG */ unsigned int sig_on_uaccess_error:1; unsigned int uaccess_err:1; /* uaccess failed */ }; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:100 @ struct thread_info { #define TIF_SYSCALL_EMU 6 /* syscall emulation active */ #define TIF_SYSCALL_AUDIT 7 /* syscall auditing active */ #define TIF_SECCOMP 8 /* secure computing */ +#define TIF_NEED_RESCHED_LAZY 9 /* lazy rescheduling necessary */ #define TIF_USER_RETURN_NOTIFY 11 /* notify kernel of userspace return */ #define TIF_UPROBE 12 /* breakpointed or singlestepping */ #define TIF_NOTSC 16 /* TSC is not accessible in userland */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:125 @ struct thread_info { #define _TIF_SYSCALL_EMU (1 << TIF_SYSCALL_EMU) #define _TIF_SYSCALL_AUDIT (1 << TIF_SYSCALL_AUDIT) #define _TIF_SECCOMP (1 << TIF_SECCOMP) +#define _TIF_NEED_RESCHED_LAZY (1 << TIF_NEED_RESCHED_LAZY) #define _TIF_USER_RETURN_NOTIFY (1 << TIF_USER_RETURN_NOTIFY) #define _TIF_UPROBE (1 << TIF_UPROBE) #define _TIF_NOTSC (1 << TIF_NOTSC) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:159 @ struct thread_info { #define _TIF_WORK_CTXSW_PREV (_TIF_WORK_CTXSW|_TIF_USER_RETURN_NOTIFY) #define _TIF_WORK_CTXSW_NEXT (_TIF_WORK_CTXSW) +#define _TIF_NEED_RESCHED_MASK (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY) + #define STACK_WARN (THREAD_SIZE/8) /* Index: linux-4.4.25-rt35/arch/x86/include/asm/uv/uv_bau.h =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/include/asm/uv/uv_bau.h +++ linux-4.4.25-rt35/arch/x86/include/asm/uv/uv_bau.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:618 @ struct bau_control { cycles_t send_message; cycles_t period_end; cycles_t period_time; - spinlock_t uvhub_lock; - spinlock_t queue_lock; - spinlock_t disable_lock; + raw_spinlock_t uvhub_lock; + raw_spinlock_t queue_lock; + raw_spinlock_t disable_lock; /* tunables */ int max_concurr; int max_concurr_const; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:779 @ static inline int atom_asr(short i, stru * to be lowered below the current 'v'. atomic_add_unless can only stop * on equal. */ -static inline int atomic_inc_unless_ge(spinlock_t *lock, atomic_t *v, int u) +static inline int atomic_inc_unless_ge(raw_spinlock_t *lock, atomic_t *v, int u) { - spin_lock(lock); + raw_spin_lock(lock); if (atomic_read(v) >= u) { - spin_unlock(lock); + raw_spin_unlock(lock); return 0; } atomic_inc(v); - spin_unlock(lock); + raw_spin_unlock(lock); return 1; } Index: linux-4.4.25-rt35/arch/x86/include/asm/uv/uv_hub.h =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/include/asm/uv/uv_hub.h +++ linux-4.4.25-rt35/arch/x86/include/asm/uv/uv_hub.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:495 @ struct uv_blade_info { unsigned short nr_online_cpus; unsigned short pnode; short memory_nid; - spinlock_t nmi_lock; /* obsolete, see uv_hub_nmi */ + raw_spinlock_t nmi_lock; /* obsolete, see uv_hub_nmi */ unsigned long nmi_count; /* obsolete, see uv_hub_nmi */ }; extern struct uv_blade_info *uv_blade_info; Index: linux-4.4.25-rt35/arch/x86/kernel/acpi/boot.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kernel/acpi/boot.c +++ linux-4.4.25-rt35/arch/x86/kernel/acpi/boot.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:90 @ static u64 acpi_lapic_addr __initdata = * ->ioapic_mutex * ->ioapic_lock */ +#ifdef CONFIG_X86_IO_APIC static DEFINE_MUTEX(acpi_ioapic_lock); +#endif /* -------------------------------------------------------------------------- Boot-time Configuration Index: linux-4.4.25-rt35/arch/x86/kernel/apic/io_apic.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kernel/apic/io_apic.c +++ linux-4.4.25-rt35/arch/x86/kernel/apic/io_apic.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1714 @ static bool io_apic_level_ack_pending(st static inline bool ioapic_irqd_mask(struct irq_data *data) { /* If we are moving the irq we need to mask it */ - if (unlikely(irqd_is_setaffinity_pending(data))) { + if (unlikely(irqd_is_setaffinity_pending(data) && + !irqd_irq_inprogress(data))) { mask_ioapic_irq(data); return true; } Index: linux-4.4.25-rt35/arch/x86/kernel/apic/x2apic_uv_x.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kernel/apic/x2apic_uv_x.c +++ linux-4.4.25-rt35/arch/x86/kernel/apic/x2apic_uv_x.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:950 @ void __init uv_system_init(void) uv_blade_info[blade].pnode = pnode; uv_blade_info[blade].nr_possible_cpus = 0; uv_blade_info[blade].nr_online_cpus = 0; - spin_lock_init(&uv_blade_info[blade].nmi_lock); + raw_spin_lock_init(&uv_blade_info[blade].nmi_lock); min_pnode = min(pnode, min_pnode); max_pnode = max(pnode, max_pnode); blade++; Index: linux-4.4.25-rt35/arch/x86/kernel/asm-offsets.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kernel/asm-offsets.c +++ linux-4.4.25-rt35/arch/x86/kernel/asm-offsets.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:35 @ void common(void) { OFFSET(TI_flags, thread_info, flags); OFFSET(TI_status, thread_info, status); OFFSET(TI_addr_limit, thread_info, addr_limit); + OFFSET(TI_preempt_lazy_count, thread_info, preempt_lazy_count); BLANK(); OFFSET(crypto_tfm_ctx_offset, crypto_tfm, __crt_ctx); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:93 @ void common(void) { BLANK(); DEFINE(PTREGS_SIZE, sizeof(struct pt_regs)); + DEFINE(_PREEMPT_ENABLED, PREEMPT_ENABLED); } Index: linux-4.4.25-rt35/arch/x86/kernel/cpu/mcheck/mce.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kernel/cpu/mcheck/mce.c +++ linux-4.4.25-rt35/arch/x86/kernel/cpu/mcheck/mce.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:44 @ #include <linux/debugfs.h> #include <linux/irq_work.h> #include <linux/export.h> +#include <linux/jiffies.h> +#include <linux/swork.h> #include <asm/processor.h> #include <asm/traps.h> @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1241 @ void mce_log_therm_throt_event(__u64 sta static unsigned long check_interval = INITIAL_CHECK_INTERVAL; static DEFINE_PER_CPU(unsigned long, mce_next_interval); /* in jiffies */ -static DEFINE_PER_CPU(struct timer_list, mce_timer); +static DEFINE_PER_CPU(struct hrtimer, mce_timer); static unsigned long mce_adjust_timer_default(unsigned long interval) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1250 @ static unsigned long mce_adjust_timer_de static unsigned long (*mce_adjust_timer)(unsigned long interval) = mce_adjust_timer_default; -static void __restart_timer(struct timer_list *t, unsigned long interval) +static enum hrtimer_restart __restart_timer(struct hrtimer *timer, unsigned long interval) { - unsigned long when = jiffies + interval; - unsigned long flags; - - local_irq_save(flags); - - if (timer_pending(t)) { - if (time_before(when, t->expires)) - mod_timer_pinned(t, when); - } else { - t->expires = round_jiffies(when); - add_timer_on(t, smp_processor_id()); - } - - local_irq_restore(flags); + if (!interval) + return HRTIMER_NORESTART; + hrtimer_forward_now(timer, ns_to_ktime(jiffies_to_nsecs(interval))); + return HRTIMER_RESTART; } -static void mce_timer_fn(unsigned long data) +static enum hrtimer_restart mce_timer_fn(struct hrtimer *timer) { - struct timer_list *t = this_cpu_ptr(&mce_timer); - int cpu = smp_processor_id(); unsigned long iv; - WARN_ON(cpu != data); - iv = __this_cpu_read(mce_next_interval); if (mce_available(this_cpu_ptr(&cpu_info))) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1284 @ static void mce_timer_fn(unsigned long d done: __this_cpu_write(mce_next_interval, iv); - __restart_timer(t, iv); + return __restart_timer(timer, iv); } /* @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1292 @ done: */ void mce_timer_kick(unsigned long interval) { - struct timer_list *t = this_cpu_ptr(&mce_timer); + struct hrtimer *t = this_cpu_ptr(&mce_timer); unsigned long iv = __this_cpu_read(mce_next_interval); __restart_timer(t, interval); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1307 @ static void mce_timer_delete_all(void) int cpu; for_each_online_cpu(cpu) - del_timer_sync(&per_cpu(mce_timer, cpu)); + hrtimer_cancel(&per_cpu(mce_timer, cpu)); } static void mce_do_trigger(struct work_struct *work) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1317 @ static void mce_do_trigger(struct work_s static DECLARE_WORK(mce_trigger_work, mce_do_trigger); +static void __mce_notify_work(struct swork_event *event) +{ + /* Not more than two messages every minute */ + static DEFINE_RATELIMIT_STATE(ratelimit, 60*HZ, 2); + + /* wake processes polling /dev/mcelog */ + wake_up_interruptible(&mce_chrdev_wait); + + /* + * There is no risk of missing notifications because + * work_pending is always cleared before the function is + * executed. + */ + if (mce_helper[0] && !work_pending(&mce_trigger_work)) + schedule_work(&mce_trigger_work); + + if (__ratelimit(&ratelimit)) + pr_info(HW_ERR "Machine check events logged\n"); +} + +#ifdef CONFIG_PREEMPT_RT_FULL +static bool notify_work_ready __read_mostly; +static struct swork_event notify_work; + +static int mce_notify_work_init(void) +{ + int err; + + err = swork_get(); + if (err) + return err; + + INIT_SWORK(¬ify_work, __mce_notify_work); + notify_work_ready = true; + return 0; +} + +static void mce_notify_work(void) +{ + if (notify_work_ready) + swork_queue(¬ify_work); +} +#else +static void mce_notify_work(void) +{ + __mce_notify_work(NULL); +} +static inline int mce_notify_work_init(void) { return 0; } +#endif + /* * Notify the user(s) about new machine check events. * Can be called from interrupt context, but not from machine check/NMI @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1374 @ static DECLARE_WORK(mce_trigger_work, mc */ int mce_notify_irq(void) { - /* Not more than two messages every minute */ - static DEFINE_RATELIMIT_STATE(ratelimit, 60*HZ, 2); - if (test_and_clear_bit(0, &mce_need_notify)) { - /* wake processes polling /dev/mcelog */ - wake_up_interruptible(&mce_chrdev_wait); - - if (mce_helper[0]) - schedule_work(&mce_trigger_work); - - if (__ratelimit(&ratelimit)) - pr_info(HW_ERR "Machine check events logged\n"); - + mce_notify_work(); return 1; } return 0; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1669 @ static void __mcheck_cpu_clear_vendor(st } } -static void mce_start_timer(unsigned int cpu, struct timer_list *t) +static void mce_start_timer(unsigned int cpu, struct hrtimer *t) { unsigned long iv = check_interval * HZ; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1678 @ static void mce_start_timer(unsigned int per_cpu(mce_next_interval, cpu) = iv; - t->expires = round_jiffies(jiffies + iv); - add_timer_on(t, cpu); + hrtimer_start_range_ns(t, ns_to_ktime(jiffies_to_usecs(iv) * 1000ULL), + 0, HRTIMER_MODE_REL_PINNED); } static void __mcheck_cpu_init_timer(void) { - struct timer_list *t = this_cpu_ptr(&mce_timer); + struct hrtimer *t = this_cpu_ptr(&mce_timer); unsigned int cpu = smp_processor_id(); - setup_timer(t, mce_timer_fn, cpu); + hrtimer_init(t, CLOCK_MONOTONIC, HRTIMER_MODE_REL); + t->function = mce_timer_fn; mce_start_timer(cpu, t); } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2407 @ static void mce_disable_cpu(void *h) if (!mce_available(raw_cpu_ptr(&cpu_info))) return; + hrtimer_cancel(this_cpu_ptr(&mce_timer)); + if (!(action & CPU_TASKS_FROZEN)) cmci_clear(); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2431 @ static void mce_reenable_cpu(void *h) if (b->init) wrmsrl(MSR_IA32_MCx_CTL(i), b->ctl); } + __mcheck_cpu_init_timer(); } /* Get notified when a cpu comes on/off. Be hotplug friendly. */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2439 @ static int mce_cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu) { unsigned int cpu = (unsigned long)hcpu; - struct timer_list *t = &per_cpu(mce_timer, cpu); switch (action & ~CPU_TASKS_FROZEN) { case CPU_ONLINE: @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2458 @ mce_cpu_callback(struct notifier_block * break; case CPU_DOWN_PREPARE: smp_call_function_single(cpu, mce_disable_cpu, &action, 1); - del_timer_sync(t); break; case CPU_DOWN_FAILED: smp_call_function_single(cpu, mce_reenable_cpu, &action, 1); - mce_start_timer(cpu, t); break; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2499 @ static __init int mcheck_init_device(voi goto err_out; } + err = mce_notify_work_init(); + if (err) + goto err_out; + if (!zalloc_cpumask_var(&mce_device_initialized, GFP_KERNEL)) { err = -ENOMEM; goto err_out; Index: linux-4.4.25-rt35/arch/x86/kernel/cpu/perf_event_intel_rapl.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kernel/cpu/perf_event_intel_rapl.c +++ linux-4.4.25-rt35/arch/x86/kernel/cpu/perf_event_intel_rapl.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:120 @ static struct perf_pmu_events_attr event }; struct rapl_pmu { - spinlock_t lock; + raw_spinlock_t lock; int n_active; /* number of active events */ struct list_head active_list; struct pmu *pmu; /* pointer to rapl_pmu_class */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:223 @ static enum hrtimer_restart rapl_hrtimer if (!pmu->n_active) return HRTIMER_NORESTART; - spin_lock_irqsave(&pmu->lock, flags); + raw_spin_lock_irqsave(&pmu->lock, flags); list_for_each_entry(event, &pmu->active_list, active_entry) { rapl_event_update(event); } - spin_unlock_irqrestore(&pmu->lock, flags); + raw_spin_unlock_irqrestore(&pmu->lock, flags); hrtimer_forward_now(hrtimer, pmu->timer_interval); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:266 @ static void rapl_pmu_event_start(struct struct rapl_pmu *pmu = __this_cpu_read(rapl_pmu); unsigned long flags; - spin_lock_irqsave(&pmu->lock, flags); + raw_spin_lock_irqsave(&pmu->lock, flags); __rapl_pmu_event_start(pmu, event); - spin_unlock_irqrestore(&pmu->lock, flags); + raw_spin_unlock_irqrestore(&pmu->lock, flags); } static void rapl_pmu_event_stop(struct perf_event *event, int mode) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:277 @ static void rapl_pmu_event_stop(struct p struct hw_perf_event *hwc = &event->hw; unsigned long flags; - spin_lock_irqsave(&pmu->lock, flags); + raw_spin_lock_irqsave(&pmu->lock, flags); /* mark event as deactivated and stopped */ if (!(hwc->state & PERF_HES_STOPPED)) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:302 @ static void rapl_pmu_event_stop(struct p hwc->state |= PERF_HES_UPTODATE; } - spin_unlock_irqrestore(&pmu->lock, flags); + raw_spin_unlock_irqrestore(&pmu->lock, flags); } static int rapl_pmu_event_add(struct perf_event *event, int mode) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:311 @ static int rapl_pmu_event_add(struct per struct hw_perf_event *hwc = &event->hw; unsigned long flags; - spin_lock_irqsave(&pmu->lock, flags); + raw_spin_lock_irqsave(&pmu->lock, flags); hwc->state = PERF_HES_UPTODATE | PERF_HES_STOPPED; if (mode & PERF_EF_START) __rapl_pmu_event_start(pmu, event); - spin_unlock_irqrestore(&pmu->lock, flags); + raw_spin_unlock_irqrestore(&pmu->lock, flags); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:606 @ static int rapl_cpu_prepare(int cpu) pmu = kzalloc_node(sizeof(*pmu), GFP_KERNEL, cpu_to_node(cpu)); if (!pmu) return -1; - spin_lock_init(&pmu->lock); + raw_spin_lock_init(&pmu->lock); INIT_LIST_HEAD(&pmu->active_list); Index: linux-4.4.25-rt35/arch/x86/kernel/dumpstack_32.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kernel/dumpstack_32.c +++ linux-4.4.25-rt35/arch/x86/kernel/dumpstack_32.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:45 @ void dump_trace(struct task_struct *task unsigned long *stack, unsigned long bp, const struct stacktrace_ops *ops, void *data) { - const unsigned cpu = get_cpu(); + const unsigned cpu = get_cpu_light(); int graph = 0; u32 *prev_esp; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:89 @ void dump_trace(struct task_struct *task break; touch_nmi_watchdog(); } - put_cpu(); + put_cpu_light(); } EXPORT_SYMBOL(dump_trace); Index: linux-4.4.25-rt35/arch/x86/kernel/dumpstack_64.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kernel/dumpstack_64.c +++ linux-4.4.25-rt35/arch/x86/kernel/dumpstack_64.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:155 @ void dump_trace(struct task_struct *task unsigned long *stack, unsigned long bp, const struct stacktrace_ops *ops, void *data) { - const unsigned cpu = get_cpu(); + const unsigned cpu = get_cpu_light(); struct thread_info *tinfo; unsigned long *irq_stack = (unsigned long *)per_cpu(irq_stack_ptr, cpu); unsigned long dummy; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:244 @ void dump_trace(struct task_struct *task * This handles the process stack: */ bp = ops->walk_stack(tinfo, stack, bp, ops, data, NULL, &graph); - put_cpu(); + put_cpu_light(); } EXPORT_SYMBOL(dump_trace); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:258 @ show_stack_log_lvl(struct task_struct *t int cpu; int i; - preempt_disable(); + migrate_disable(); cpu = smp_processor_id(); irq_stack_end = (unsigned long *)(per_cpu(irq_stack_ptr, cpu)); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:294 @ show_stack_log_lvl(struct task_struct *t pr_cont(" %016lx", *stack++); touch_nmi_watchdog(); } - preempt_enable(); + migrate_enable(); pr_cont("\n"); show_trace_log_lvl(task, regs, sp, bp, log_lvl); Index: linux-4.4.25-rt35/arch/x86/kernel/irq_32.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kernel/irq_32.c +++ linux-4.4.25-rt35/arch/x86/kernel/irq_32.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:131 @ void irq_ctx_init(int cpu) cpu, per_cpu(hardirq_stack, cpu), per_cpu(softirq_stack, cpu)); } +#ifndef CONFIG_PREEMPT_RT_FULL void do_softirq_own_stack(void) { struct thread_info *curstk; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:150 @ void do_softirq_own_stack(void) call_on_stack(__do_softirq, isp); } +#endif bool handle_irq(struct irq_desc *desc, struct pt_regs *regs) { Index: linux-4.4.25-rt35/arch/x86/kernel/kvm.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kernel/kvm.c +++ linux-4.4.25-rt35/arch/x86/kernel/kvm.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:39 @ #include <linux/kprobes.h> #include <linux/debugfs.h> #include <linux/nmi.h> +#include <linux/swait.h> #include <asm/timer.h> #include <asm/cpu.h> #include <asm/traps.h> @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:95 @ static void kvm_io_delay(void) struct kvm_task_sleep_node { struct hlist_node link; - wait_queue_head_t wq; + struct swait_queue_head wq; u32 token; int cpu; bool halted; }; static struct kvm_task_sleep_head { - spinlock_t lock; + raw_spinlock_t lock; struct hlist_head list; } async_pf_sleepers[KVM_TASK_SLEEP_HASHSIZE]; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:126 @ void kvm_async_pf_task_wait(u32 token) u32 key = hash_32(token, KVM_TASK_SLEEP_HASHBITS); struct kvm_task_sleep_head *b = &async_pf_sleepers[key]; struct kvm_task_sleep_node n, *e; - DEFINE_WAIT(wait); + DECLARE_SWAITQUEUE(wait); rcu_irq_enter(); - spin_lock(&b->lock); + raw_spin_lock(&b->lock); e = _find_apf_task(b, token); if (e) { /* dummy entry exist -> wake up was delivered ahead of PF */ hlist_del(&e->link); kfree(e); - spin_unlock(&b->lock); + raw_spin_unlock(&b->lock); rcu_irq_exit(); return; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:145 @ void kvm_async_pf_task_wait(u32 token) n.token = token; n.cpu = smp_processor_id(); n.halted = is_idle_task(current) || preempt_count() > 1; - init_waitqueue_head(&n.wq); + init_swait_queue_head(&n.wq); hlist_add_head(&n.link, &b->list); - spin_unlock(&b->lock); + raw_spin_unlock(&b->lock); for (;;) { if (!n.halted) - prepare_to_wait(&n.wq, &wait, TASK_UNINTERRUPTIBLE); + prepare_to_swait(&n.wq, &wait, TASK_UNINTERRUPTIBLE); if (hlist_unhashed(&n.link)) break; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:170 @ void kvm_async_pf_task_wait(u32 token) } } if (!n.halted) - finish_wait(&n.wq, &wait); + finish_swait(&n.wq, &wait); rcu_irq_exit(); return; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:182 @ static void apf_task_wake_one(struct kvm hlist_del_init(&n->link); if (n->halted) smp_send_reschedule(n->cpu); - else if (waitqueue_active(&n->wq)) - wake_up(&n->wq); + else if (swait_active(&n->wq)) + swake_up(&n->wq); } static void apf_task_wake_all(void) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:193 @ static void apf_task_wake_all(void) for (i = 0; i < KVM_TASK_SLEEP_HASHSIZE; i++) { struct hlist_node *p, *next; struct kvm_task_sleep_head *b = &async_pf_sleepers[i]; - spin_lock(&b->lock); + raw_spin_lock(&b->lock); hlist_for_each_safe(p, next, &b->list) { struct kvm_task_sleep_node *n = hlist_entry(p, typeof(*n), link); if (n->cpu == smp_processor_id()) apf_task_wake_one(n); } - spin_unlock(&b->lock); + raw_spin_unlock(&b->lock); } } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:216 @ void kvm_async_pf_task_wake(u32 token) } again: - spin_lock(&b->lock); + raw_spin_lock(&b->lock); n = _find_apf_task(b, token); if (!n) { /* @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:229 @ again: * Allocation failed! Busy wait while other cpu * handles async PF. */ - spin_unlock(&b->lock); + raw_spin_unlock(&b->lock); cpu_relax(); goto again; } n->token = token; n->cpu = smp_processor_id(); - init_waitqueue_head(&n->wq); + init_swait_queue_head(&n->wq); hlist_add_head(&n->link, &b->list); } else apf_task_wake_one(n); - spin_unlock(&b->lock); + raw_spin_unlock(&b->lock); return; } EXPORT_SYMBOL_GPL(kvm_async_pf_task_wake); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:490 @ void __init kvm_guest_init(void) paravirt_ops_setup(); register_reboot_notifier(&kvm_pv_reboot_nb); for (i = 0; i < KVM_TASK_SLEEP_HASHSIZE; i++) - spin_lock_init(&async_pf_sleepers[i].lock); + raw_spin_lock_init(&async_pf_sleepers[i].lock); if (kvm_para_has_feature(KVM_FEATURE_ASYNC_PF)) x86_init.irqs.trap_init = kvm_apf_trap_init; Index: linux-4.4.25-rt35/arch/x86/kernel/nmi.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kernel/nmi.c +++ linux-4.4.25-rt35/arch/x86/kernel/nmi.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:234 @ pci_serr_error(unsigned char reason, str #endif if (panic_on_unrecovered_nmi) - panic("NMI: Not continuing"); + nmi_panic(regs, "NMI: Not continuing"); pr_emerg("Dazed and confused, but trying to continue\n"); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:258 @ io_check_error(unsigned char reason, str reason, smp_processor_id()); show_regs(regs); - if (panic_on_io_nmi) - panic("NMI IOCK error: Not continuing"); + if (panic_on_io_nmi) { + nmi_panic(regs, "NMI IOCK error: Not continuing"); + + /* + * If we end up here, it means we have received an NMI while + * processing panic(). Simply return without delaying and + * re-enabling NMIs. + */ + return; + } /* Re-enable the IOCK line, wait for a few seconds */ reason = (reason & NMI_REASON_CLEAR_MASK) | NMI_REASON_CLEAR_IOCHK; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:308 @ unknown_nmi_error(unsigned char reason, pr_emerg("Do you have a strange power saving mode enabled?\n"); if (unknown_nmi_panic || panic_on_unrecovered_nmi) - panic("NMI: Not continuing"); + nmi_panic(regs, "NMI: Not continuing"); pr_emerg("Dazed and confused, but trying to continue\n"); } Index: linux-4.4.25-rt35/arch/x86/kernel/process_32.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kernel/process_32.c +++ linux-4.4.25-rt35/arch/x86/kernel/process_32.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:38 @ #include <linux/uaccess.h> #include <linux/io.h> #include <linux/kdebug.h> +#include <linux/highmem.h> #include <asm/pgtable.h> #include <asm/ldt.h> @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:214 @ start_thread(struct pt_regs *regs, unsig } EXPORT_SYMBOL_GPL(start_thread); +#ifdef CONFIG_PREEMPT_RT_FULL +static void switch_kmaps(struct task_struct *prev_p, struct task_struct *next_p) +{ + int i; + + /* + * Clear @prev's kmap_atomic mappings + */ + for (i = 0; i < prev_p->kmap_idx; i++) { + int idx = i + KM_TYPE_NR * smp_processor_id(); + pte_t *ptep = kmap_pte - idx; + + kpte_clear_flush(ptep, __fix_to_virt(FIX_KMAP_BEGIN + idx)); + } + /* + * Restore @next_p's kmap_atomic mappings + */ + for (i = 0; i < next_p->kmap_idx; i++) { + int idx = i + KM_TYPE_NR * smp_processor_id(); + + if (!pte_none(next_p->kmap_pte[i])) + set_pte(kmap_pte - idx, next_p->kmap_pte[i]); + } +} +#else +static inline void +switch_kmaps(struct task_struct *prev_p, struct task_struct *next_p) { } +#endif + /* * switch_to(x,y) should switch tasks from x to y. @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:319 @ __switch_to(struct task_struct *prev_p, task_thread_info(next_p)->flags & _TIF_WORK_CTXSW_NEXT)) __switch_to_xtra(prev_p, next_p, tss); + switch_kmaps(prev_p, next_p); + /* * Leave lazy mode, flushing any hypercalls made here. * This must be done before restoring TLS segments so Index: linux-4.4.25-rt35/arch/x86/kernel/reboot.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kernel/reboot.c +++ linux-4.4.25-rt35/arch/x86/kernel/reboot.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:729 @ static int crashing_cpu; static nmi_shootdown_cb shootdown_callback; static atomic_t waiting_for_crash_ipi; +static int crash_ipi_issued; static int crash_nmi_callback(unsigned int val, struct pt_regs *regs) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:792 @ void nmi_shootdown_cpus(nmi_shootdown_cb smp_send_nmi_allbutself(); + /* Kick CPUs looping in NMI context. */ + WRITE_ONCE(crash_ipi_issued, 1); + msecs = 1000; /* Wait at most a second for the other cpus to stop */ while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) { mdelay(1); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:803 @ void nmi_shootdown_cpus(nmi_shootdown_cb /* Leave the nmi callback set */ } + +/* Override the weak function in kernel/panic.c */ +void nmi_panic_self_stop(struct pt_regs *regs) +{ + while (1) { + /* + * Wait for the crash dumping IPI to be issued, and then + * call its callback directly. + */ + if (READ_ONCE(crash_ipi_issued)) + crash_nmi_callback(0, regs); /* Don't return */ + + cpu_relax(); + } +} + #else /* !CONFIG_SMP */ void nmi_shootdown_cpus(nmi_shootdown_cb callback) { Index: linux-4.4.25-rt35/arch/x86/kvm/lapic.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kvm/lapic.c +++ linux-4.4.25-rt35/arch/x86/kvm/lapic.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1198 @ static void apic_update_lvtt(struct kvm_ static void apic_timer_expired(struct kvm_lapic *apic) { struct kvm_vcpu *vcpu = apic->vcpu; - wait_queue_head_t *q = &vcpu->wq; + struct swait_queue_head *q = &vcpu->wq; struct kvm_timer *ktimer = &apic->lapic_timer; if (atomic_read(&apic->lapic_timer.pending)) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1207 @ static void apic_timer_expired(struct kv atomic_inc(&apic->lapic_timer.pending); kvm_set_pending_timer(vcpu); - if (waitqueue_active(q)) - wake_up_interruptible(q); + if (swait_active(q)) + swake_up(q); if (apic_lvtt_tscdeadline(apic)) ktimer->expired_tscdeadline = ktimer->tscdeadline; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1804 @ int kvm_create_lapic(struct kvm_vcpu *vc hrtimer_init(&apic->lapic_timer.timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS); apic->lapic_timer.timer.function = apic_timer_fn; + apic->lapic_timer.timer.irqsafe = 1; /* * APIC is created enabled. This will prevent kvm_lapic_set_base from Index: linux-4.4.25-rt35/arch/x86/kvm/x86.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/kvm/x86.c +++ linux-4.4.25-rt35/arch/x86/kvm/x86.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:5807 @ int kvm_arch_init(void *opaque) goto out; } +#ifdef CONFIG_PREEMPT_RT_FULL + if (!boot_cpu_has(X86_FEATURE_CONSTANT_TSC)) { + printk(KERN_ERR "RT requires X86_FEATURE_CONSTANT_TSC\n"); + return -EOPNOTSUPP; + } +#endif + r = kvm_mmu_module_init(); if (r) goto out_free_percpu; Index: linux-4.4.25-rt35/arch/x86/mm/highmem_32.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/mm/highmem_32.c +++ linux-4.4.25-rt35/arch/x86/mm/highmem_32.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:35 @ EXPORT_SYMBOL(kunmap); */ void *kmap_atomic_prot(struct page *page, pgprot_t prot) { + pte_t pte = mk_pte(page, prot); unsigned long vaddr; int idx, type; - preempt_disable(); + preempt_disable_nort(); pagefault_disable(); if (!PageHighMem(page)) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:49 @ void *kmap_atomic_prot(struct page *page idx = type + KM_TYPE_NR*smp_processor_id(); vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx); BUG_ON(!pte_none(*(kmap_pte-idx))); - set_pte(kmap_pte-idx, mk_pte(page, prot)); +#ifdef CONFIG_PREEMPT_RT_FULL + current->kmap_pte[type] = pte; +#endif + set_pte(kmap_pte-idx, pte); arch_flush_lazy_mmu_mode(); return (void *)vaddr; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:95 @ void __kunmap_atomic(void *kvaddr) * is a bad idea also, in case the page changes cacheability * attributes or becomes a protected page in a hypervisor. */ +#ifdef CONFIG_PREEMPT_RT_FULL + current->kmap_pte[type] = __pte(0); +#endif kpte_clear_flush(kmap_pte-idx, vaddr); kmap_atomic_idx_pop(); arch_flush_lazy_mmu_mode(); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:110 @ void __kunmap_atomic(void *kvaddr) #endif pagefault_enable(); - preempt_enable(); + preempt_enable_nort(); } EXPORT_SYMBOL(__kunmap_atomic); Index: linux-4.4.25-rt35/arch/x86/mm/iomap_32.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/mm/iomap_32.c +++ linux-4.4.25-rt35/arch/x86/mm/iomap_32.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:59 @ EXPORT_SYMBOL_GPL(iomap_free); void *kmap_atomic_prot_pfn(unsigned long pfn, pgprot_t prot) { + pte_t pte = pfn_pte(pfn, prot); unsigned long vaddr; int idx, type; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:69 @ void *kmap_atomic_prot_pfn(unsigned long type = kmap_atomic_idx_push(); idx = type + KM_TYPE_NR * smp_processor_id(); vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx); - set_pte(kmap_pte - idx, pfn_pte(pfn, prot)); + WARN_ON(!pte_none(*(kmap_pte - idx))); + +#ifdef CONFIG_PREEMPT_RT_FULL + current->kmap_pte[type] = pte; +#endif + set_pte(kmap_pte - idx, pte); arch_flush_lazy_mmu_mode(); return (void *)vaddr; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:122 @ iounmap_atomic(void __iomem *kvaddr) * is a bad idea also, in case the page changes cacheability * attributes or becomes a protected page in a hypervisor. */ +#ifdef CONFIG_PREEMPT_RT_FULL + current->kmap_pte[type] = __pte(0); +#endif kpte_clear_flush(kmap_pte-idx, vaddr); kmap_atomic_idx_pop(); } Index: linux-4.4.25-rt35/arch/x86/platform/uv/tlb_uv.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/platform/uv/tlb_uv.c +++ linux-4.4.25-rt35/arch/x86/platform/uv/tlb_uv.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:717 @ static void destination_plugged(struct b quiesce_local_uvhub(hmaster); - spin_lock(&hmaster->queue_lock); + raw_spin_lock(&hmaster->queue_lock); reset_with_ipi(&bau_desc->distribution, bcp); - spin_unlock(&hmaster->queue_lock); + raw_spin_unlock(&hmaster->queue_lock); end_uvhub_quiesce(hmaster); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:739 @ static void destination_timeout(struct b quiesce_local_uvhub(hmaster); - spin_lock(&hmaster->queue_lock); + raw_spin_lock(&hmaster->queue_lock); reset_with_ipi(&bau_desc->distribution, bcp); - spin_unlock(&hmaster->queue_lock); + raw_spin_unlock(&hmaster->queue_lock); end_uvhub_quiesce(hmaster); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:762 @ static void disable_for_period(struct ba cycles_t tm1; hmaster = bcp->uvhub_master; - spin_lock(&hmaster->disable_lock); + raw_spin_lock(&hmaster->disable_lock); if (!bcp->baudisabled) { stat->s_bau_disabled++; tm1 = get_cycles(); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:775 @ static void disable_for_period(struct ba } } } - spin_unlock(&hmaster->disable_lock); + raw_spin_unlock(&hmaster->disable_lock); } static void count_max_concurr(int stat, struct bau_control *bcp, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:838 @ static void record_send_stats(cycles_t t */ static void uv1_throttle(struct bau_control *hmaster, struct ptc_stats *stat) { - spinlock_t *lock = &hmaster->uvhub_lock; + raw_spinlock_t *lock = &hmaster->uvhub_lock; atomic_t *v; v = &hmaster->active_descriptor_count; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:971 @ static int check_enable(struct bau_contr struct bau_control *hmaster; hmaster = bcp->uvhub_master; - spin_lock(&hmaster->disable_lock); + raw_spin_lock(&hmaster->disable_lock); if (bcp->baudisabled && (get_cycles() >= bcp->set_bau_on_time)) { stat->s_bau_reenabled++; for_each_present_cpu(tcpu) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:983 @ static int check_enable(struct bau_contr tbcp->period_giveups = 0; } } - spin_unlock(&hmaster->disable_lock); + raw_spin_unlock(&hmaster->disable_lock); return 0; } - spin_unlock(&hmaster->disable_lock); + raw_spin_unlock(&hmaster->disable_lock); return -1; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1904 @ static void __init init_per_cpu_tunables bcp->cong_reps = congested_reps; bcp->disabled_period = sec_2_cycles(disabled_period); bcp->giveup_limit = giveup_limit; - spin_lock_init(&bcp->queue_lock); - spin_lock_init(&bcp->uvhub_lock); - spin_lock_init(&bcp->disable_lock); + raw_spin_lock_init(&bcp->queue_lock); + raw_spin_lock_init(&bcp->uvhub_lock); + raw_spin_lock_init(&bcp->disable_lock); } } Index: linux-4.4.25-rt35/arch/x86/platform/uv/uv_time.c =================================================================== --- linux-4.4.25-rt35.orig/arch/x86/platform/uv/uv_time.c +++ linux-4.4.25-rt35/arch/x86/platform/uv/uv_time.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:60 @ static DEFINE_PER_CPU(struct clock_event /* There is one of these allocated per node */ struct uv_rtc_timer_head { - spinlock_t lock; + raw_spinlock_t lock; /* next cpu waiting for timer, local node relative: */ int next_cpu; /* number of cpus on this node: */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:180 @ static __init int uv_rtc_allocate_timers uv_rtc_deallocate_timers(); return -ENOMEM; } - spin_lock_init(&head->lock); + raw_spin_lock_init(&head->lock); head->ncpus = uv_blade_nr_possible_cpus(bid); head->next_cpu = -1; blade_info[bid] = head; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:234 @ static int uv_rtc_set_timer(int cpu, u64 unsigned long flags; int next_cpu; - spin_lock_irqsave(&head->lock, flags); + raw_spin_lock_irqsave(&head->lock, flags); next_cpu = head->next_cpu; *t = expires; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:246 @ static int uv_rtc_set_timer(int cpu, u64 if (uv_setup_intr(cpu, expires)) { *t = ULLONG_MAX; uv_rtc_find_next_timer(head, pnode); - spin_unlock_irqrestore(&head->lock, flags); + raw_spin_unlock_irqrestore(&head->lock, flags); return -ETIME; } } - spin_unlock_irqrestore(&head->lock, flags); + raw_spin_unlock_irqrestore(&head->lock, flags); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:270 @ static int uv_rtc_unset_timer(int cpu, i unsigned long flags; int rc = 0; - spin_lock_irqsave(&head->lock, flags); + raw_spin_lock_irqsave(&head->lock, flags); if ((head->next_cpu == bcpu && uv_read_rtc(NULL) >= *t) || force) rc = 1; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:282 @ static int uv_rtc_unset_timer(int cpu, i uv_rtc_find_next_timer(head, pnode); } - spin_unlock_irqrestore(&head->lock, flags); + raw_spin_unlock_irqrestore(&head->lock, flags); return rc; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:302 @ static int uv_rtc_unset_timer(int cpu, i static cycle_t uv_read_rtc(struct clocksource *cs) { unsigned long offset; + cycle_t cycles; + preempt_disable(); if (uv_get_min_hub_revision_id() == 1) offset = 0; else offset = (uv_blade_processor_id() * L1_CACHE_BYTES) % PAGE_SIZE; - return (cycle_t)uv_read_local_mmr(UVH_RTC | offset); + cycles = (cycle_t)uv_read_local_mmr(UVH_RTC | offset); + preempt_enable(); + + return cycles; } /* Index: linux-4.4.25-rt35/block/blk-core.c =================================================================== --- linux-4.4.25-rt35.orig/block/blk-core.c +++ linux-4.4.25-rt35/block/blk-core.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:128 @ void blk_rq_init(struct request_queue *q INIT_LIST_HEAD(&rq->queuelist); INIT_LIST_HEAD(&rq->timeout_list); +#ifdef CONFIG_PREEMPT_RT_FULL + INIT_WORK(&rq->work, __blk_mq_complete_request_remote_work); +#endif rq->cpu = -1; rq->q = q; rq->__sector = (sector_t) -1; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:239 @ EXPORT_SYMBOL(blk_start_queue_async); **/ void blk_start_queue(struct request_queue *q) { - WARN_ON(!irqs_disabled()); + WARN_ON_NONRT(!irqs_disabled()); queue_flag_clear(QUEUE_FLAG_STOPPED, q); __blk_run_queue(q); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:665 @ int blk_queue_enter(struct request_queue if (!gfpflags_allow_blocking(gfp)) return -EBUSY; - ret = wait_event_interruptible(q->mq_freeze_wq, + ret = swait_event_interruptible(q->mq_freeze_wq, !atomic_read(&q->mq_freeze_depth) || blk_queue_dying(q)); if (blk_queue_dying(q)) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:685 @ static void blk_queue_usage_counter_rele struct request_queue *q = container_of(ref, struct request_queue, q_usage_counter); - wake_up_all(&q->mq_freeze_wq); + swake_up_all(&q->mq_freeze_wq); } struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:747 @ struct request_queue *blk_alloc_queue_no q->bypass_depth = 1; __set_bit(QUEUE_FLAG_BYPASS, &q->queue_flags); - init_waitqueue_head(&q->mq_freeze_wq); + init_swait_queue_head(&q->mq_freeze_wq); /* * Init percpu_ref in atomic mode so that it's faster to shutdown. @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:3206 @ static void queue_unplugged(struct reque blk_run_queue_async(q); else __blk_run_queue(q); - spin_unlock(q->queue_lock); + spin_unlock_irq(q->queue_lock); } static void flush_plug_callbacks(struct blk_plug *plug, bool from_schedule) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:3254 @ EXPORT_SYMBOL(blk_check_plugged); void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule) { struct request_queue *q; - unsigned long flags; struct request *rq; LIST_HEAD(list); unsigned int depth; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:3273 @ void blk_flush_plug_list(struct blk_plug q = NULL; depth = 0; - /* - * Save and disable interrupts here, to avoid doing it for every - * queue lock we have to take. - */ - local_irq_save(flags); while (!list_empty(&list)) { rq = list_entry_rq(list.next); list_del_init(&rq->queuelist); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:3285 @ void blk_flush_plug_list(struct blk_plug queue_unplugged(q, depth, from_schedule); q = rq->q; depth = 0; - spin_lock(q->queue_lock); + spin_lock_irq(q->queue_lock); } /* @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:3312 @ void blk_flush_plug_list(struct blk_plug */ if (q) queue_unplugged(q, depth, from_schedule); - - local_irq_restore(flags); } void blk_finish_plug(struct blk_plug *plug) Index: linux-4.4.25-rt35/block/blk-ioc.c =================================================================== --- linux-4.4.25-rt35.orig/block/blk-ioc.c +++ linux-4.4.25-rt35/block/blk-ioc.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:10 @ #include <linux/bio.h> #include <linux/blkdev.h> #include <linux/slab.h> +#include <linux/delay.h> #include "blk.h" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:113 @ static void ioc_release_fn(struct work_s spin_unlock(q->queue_lock); } else { spin_unlock_irqrestore(&ioc->lock, flags); - cpu_relax(); + cpu_chill(); spin_lock_irqsave_nested(&ioc->lock, flags, 1); } } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:191 @ retry: spin_unlock(icq->q->queue_lock); } else { spin_unlock_irqrestore(&ioc->lock, flags); - cpu_relax(); + cpu_chill(); goto retry; } } Index: linux-4.4.25-rt35/block/blk-iopoll.c =================================================================== --- linux-4.4.25-rt35.orig/block/blk-iopoll.c +++ linux-4.4.25-rt35/block/blk-iopoll.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:38 @ void blk_iopoll_sched(struct blk_iopoll list_add_tail(&iop->list, this_cpu_ptr(&blk_cpu_iopoll)); __raise_softirq_irqoff(BLOCK_IOPOLL_SOFTIRQ); local_irq_restore(flags); + preempt_check_resched_rt(); } EXPORT_SYMBOL(blk_iopoll_sched); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:136 @ static void blk_iopoll_softirq(struct so __raise_softirq_irqoff(BLOCK_IOPOLL_SOFTIRQ); local_irq_enable(); + preempt_check_resched_rt(); } /** @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:206 @ static int blk_iopoll_cpu_notify(struct this_cpu_ptr(&blk_cpu_iopoll)); __raise_softirq_irqoff(BLOCK_IOPOLL_SOFTIRQ); local_irq_enable(); + preempt_check_resched_rt(); } return NOTIFY_OK; Index: linux-4.4.25-rt35/block/blk-mq-cpu.c =================================================================== --- linux-4.4.25-rt35.orig/block/blk-mq-cpu.c +++ linux-4.4.25-rt35/block/blk-mq-cpu.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:19 @ #include "blk-mq.h" static LIST_HEAD(blk_mq_cpu_notify_list); -static DEFINE_RAW_SPINLOCK(blk_mq_cpu_notify_lock); +static DEFINE_SPINLOCK(blk_mq_cpu_notify_lock); static int blk_mq_main_cpu_notify(struct notifier_block *self, unsigned long action, void *hcpu) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:28 @ static int blk_mq_main_cpu_notify(struct struct blk_mq_cpu_notifier *notify; int ret = NOTIFY_OK; - raw_spin_lock(&blk_mq_cpu_notify_lock); + if (action != CPU_POST_DEAD) + return NOTIFY_OK; + + spin_lock(&blk_mq_cpu_notify_lock); list_for_each_entry(notify, &blk_mq_cpu_notify_list, list) { ret = notify->notify(notify->data, action, cpu); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:39 @ static int blk_mq_main_cpu_notify(struct break; } - raw_spin_unlock(&blk_mq_cpu_notify_lock); + spin_unlock(&blk_mq_cpu_notify_lock); return ret; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:47 @ void blk_mq_register_cpu_notifier(struct { BUG_ON(!notifier->notify); - raw_spin_lock(&blk_mq_cpu_notify_lock); + spin_lock(&blk_mq_cpu_notify_lock); list_add_tail(¬ifier->list, &blk_mq_cpu_notify_list); - raw_spin_unlock(&blk_mq_cpu_notify_lock); + spin_unlock(&blk_mq_cpu_notify_lock); } void blk_mq_unregister_cpu_notifier(struct blk_mq_cpu_notifier *notifier) { - raw_spin_lock(&blk_mq_cpu_notify_lock); + spin_lock(&blk_mq_cpu_notify_lock); list_del(¬ifier->list); - raw_spin_unlock(&blk_mq_cpu_notify_lock); + spin_unlock(&blk_mq_cpu_notify_lock); } void blk_mq_init_cpu_notifier(struct blk_mq_cpu_notifier *notifier, Index: linux-4.4.25-rt35/block/blk-mq.c =================================================================== --- linux-4.4.25-rt35.orig/block/blk-mq.c +++ linux-4.4.25-rt35/block/blk-mq.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:95 @ EXPORT_SYMBOL_GPL(blk_mq_freeze_queue_st static void blk_mq_freeze_queue_wait(struct request_queue *q) { - wait_event(q->mq_freeze_wq, percpu_ref_is_zero(&q->q_usage_counter)); + swait_event(q->mq_freeze_wq, percpu_ref_is_zero(&q->q_usage_counter)); } /* @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:133 @ void blk_mq_unfreeze_queue(struct reques WARN_ON_ONCE(freeze_depth < 0); if (!freeze_depth) { percpu_ref_reinit(&q->q_usage_counter); - wake_up_all(&q->mq_freeze_wq); + swake_up_all(&q->mq_freeze_wq); } } EXPORT_SYMBOL_GPL(blk_mq_unfreeze_queue); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:152 @ void blk_mq_wake_waiters(struct request_ * dying, we need to ensure that processes currently waiting on * the queue are notified as well. */ - wake_up_all(&q->mq_freeze_wq); + swake_up_all(&q->mq_freeze_wq); } bool blk_mq_can_queue(struct blk_mq_hw_ctx *hctx) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:199 @ static void blk_mq_rq_ctx_init(struct re rq->resid_len = 0; rq->sense = NULL; +#ifdef CONFIG_PREEMPT_RT_FULL + INIT_WORK(&rq->work, __blk_mq_complete_request_remote_work); +#endif INIT_LIST_HEAD(&rq->timeout_list); rq->timeout = 0; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:331 @ void blk_mq_end_request(struct request * } EXPORT_SYMBOL(blk_mq_end_request); +#ifdef CONFIG_PREEMPT_RT_FULL + +void __blk_mq_complete_request_remote_work(struct work_struct *work) +{ + struct request *rq = container_of(work, struct request, work); + + rq->q->softirq_done_fn(rq); +} + +#else + static void __blk_mq_complete_request_remote(void *data) { struct request *rq = data; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:349 @ static void __blk_mq_complete_request_re rq->q->softirq_done_fn(rq); } +#endif + static void blk_mq_ipi_complete_request(struct request *rq) { struct blk_mq_ctx *ctx = rq->mq_ctx; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:362 @ static void blk_mq_ipi_complete_request( return; } - cpu = get_cpu(); + cpu = get_cpu_light(); if (!test_bit(QUEUE_FLAG_SAME_FORCE, &rq->q->queue_flags)) shared = cpus_share_cache(cpu, ctx->cpu); if (cpu != ctx->cpu && !shared && cpu_online(ctx->cpu)) { +#ifdef CONFIG_PREEMPT_RT_FULL + schedule_work_on(ctx->cpu, &rq->work); +#else rq->csd.func = __blk_mq_complete_request_remote; rq->csd.info = rq; rq->csd.flags = 0; smp_call_function_single_async(ctx->cpu, &rq->csd); +#endif } else { rq->q->softirq_done_fn(rq); } - put_cpu(); + put_cpu_light(); } static void __blk_mq_complete_request(struct request *rq) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:887 @ void blk_mq_run_hw_queue(struct blk_mq_h return; if (!async) { - int cpu = get_cpu(); + int cpu = get_cpu_light(); if (cpumask_test_cpu(cpu, hctx->cpumask)) { __blk_mq_run_hw_queue(hctx); - put_cpu(); + put_cpu_light(); return; } - put_cpu(); + put_cpu_light(); } kblockd_schedule_delayed_work_on(blk_mq_hctx_next_cpu(hctx), @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1642 @ static int blk_mq_hctx_notify(void *data { struct blk_mq_hw_ctx *hctx = data; - if (action == CPU_DEAD || action == CPU_DEAD_FROZEN) + if (action == CPU_POST_DEAD) return blk_mq_hctx_cpu_offline(hctx, cpu); /* Index: linux-4.4.25-rt35/block/blk-mq.h =================================================================== --- linux-4.4.25-rt35.orig/block/blk-mq.h +++ linux-4.4.25-rt35/block/blk-mq.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:77 @ struct blk_align_bitmap { static inline struct blk_mq_ctx *__blk_mq_get_ctx(struct request_queue *q, unsigned int cpu) { - return per_cpu_ptr(q->queue_ctx, cpu); + struct blk_mq_ctx *ctx; + + ctx = per_cpu_ptr(q->queue_ctx, cpu); + return ctx; } /* @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:91 @ static inline struct blk_mq_ctx *__blk_m */ static inline struct blk_mq_ctx *blk_mq_get_ctx(struct request_queue *q) { - return __blk_mq_get_ctx(q, get_cpu()); + return __blk_mq_get_ctx(q, get_cpu_light()); } static inline void blk_mq_put_ctx(struct blk_mq_ctx *ctx) { - put_cpu(); + put_cpu_light(); } struct blk_mq_alloc_data { Index: linux-4.4.25-rt35/block/blk-softirq.c =================================================================== --- linux-4.4.25-rt35.orig/block/blk-softirq.c +++ linux-4.4.25-rt35/block/blk-softirq.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:54 @ static void trigger_softirq(void *data) raise_softirq_irqoff(BLOCK_SOFTIRQ); local_irq_restore(flags); + preempt_check_resched_rt(); } /* @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:97 @ static int blk_cpu_notify(struct notifie this_cpu_ptr(&blk_cpu_done)); raise_softirq_irqoff(BLOCK_SOFTIRQ); local_irq_enable(); + preempt_check_resched_rt(); } return NOTIFY_OK; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:155 @ do_local: goto do_local; local_irq_restore(flags); + preempt_check_resched_rt(); } /** Index: linux-4.4.25-rt35/block/bounce.c =================================================================== --- linux-4.4.25-rt35.orig/block/bounce.c +++ linux-4.4.25-rt35/block/bounce.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:58 @ static void bounce_copy_vec(struct bio_v unsigned long flags; unsigned char *vto; - local_irq_save(flags); + local_irq_save_nort(flags); vto = kmap_atomic(to->bv_page); memcpy(vto + to->bv_offset, vfrom, to->bv_len); kunmap_atomic(vto); - local_irq_restore(flags); + local_irq_restore_nort(flags); } #else /* CONFIG_HIGHMEM */ Index: linux-4.4.25-rt35/crypto/algapi.c =================================================================== --- linux-4.4.25-rt35.orig/crypto/algapi.c +++ linux-4.4.25-rt35/crypto/algapi.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:722 @ EXPORT_SYMBOL_GPL(crypto_spawn_tfm2); int crypto_register_notifier(struct notifier_block *nb) { - return blocking_notifier_chain_register(&crypto_chain, nb); + return srcu_notifier_chain_register(&crypto_chain, nb); } EXPORT_SYMBOL_GPL(crypto_register_notifier); int crypto_unregister_notifier(struct notifier_block *nb) { - return blocking_notifier_chain_unregister(&crypto_chain, nb); + return srcu_notifier_chain_unregister(&crypto_chain, nb); } EXPORT_SYMBOL_GPL(crypto_unregister_notifier); Index: linux-4.4.25-rt35/crypto/api.c =================================================================== --- linux-4.4.25-rt35.orig/crypto/api.c +++ linux-4.4.25-rt35/crypto/api.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:34 @ EXPORT_SYMBOL_GPL(crypto_alg_list); DECLARE_RWSEM(crypto_alg_sem); EXPORT_SYMBOL_GPL(crypto_alg_sem); -BLOCKING_NOTIFIER_HEAD(crypto_chain); +SRCU_NOTIFIER_HEAD(crypto_chain); EXPORT_SYMBOL_GPL(crypto_chain); static struct crypto_alg *crypto_larval_wait(struct crypto_alg *alg); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:239 @ int crypto_probing_notify(unsigned long { int ok; - ok = blocking_notifier_call_chain(&crypto_chain, val, v); + ok = srcu_notifier_call_chain(&crypto_chain, val, v); if (ok == NOTIFY_DONE) { request_module("cryptomgr"); - ok = blocking_notifier_call_chain(&crypto_chain, val, v); + ok = srcu_notifier_call_chain(&crypto_chain, val, v); } return ok; Index: linux-4.4.25-rt35/crypto/internal.h =================================================================== --- linux-4.4.25-rt35.orig/crypto/internal.h +++ linux-4.4.25-rt35/crypto/internal.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:50 @ struct crypto_larval { extern struct list_head crypto_alg_list; extern struct rw_semaphore crypto_alg_sem; -extern struct blocking_notifier_head crypto_chain; +extern struct srcu_notifier_head crypto_chain; #ifdef CONFIG_PROC_FS void __init crypto_init_proc(void); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:146 @ static inline int crypto_is_moribund(str static inline void crypto_notify(unsigned long val, void *v) { - blocking_notifier_call_chain(&crypto_chain, val, v); + srcu_notifier_call_chain(&crypto_chain, val, v); } #endif /* _CRYPTO_INTERNAL_H */ Index: linux-4.4.25-rt35/drivers/acpi/acpica/acglobal.h =================================================================== --- linux-4.4.25-rt35.orig/drivers/acpi/acpica/acglobal.h +++ linux-4.4.25-rt35/drivers/acpi/acpica/acglobal.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:119 @ ACPI_GLOBAL(u8, acpi_gbl_global_lock_pen * interrupt level */ ACPI_GLOBAL(acpi_spinlock, acpi_gbl_gpe_lock); /* For GPE data structs and registers */ -ACPI_GLOBAL(acpi_spinlock, acpi_gbl_hardware_lock); /* For ACPI H/W except GPE registers */ +ACPI_GLOBAL(acpi_raw_spinlock, acpi_gbl_hardware_lock); /* For ACPI H/W except GPE registers */ ACPI_GLOBAL(acpi_spinlock, acpi_gbl_reference_count_lock); /* Mutex for _OSI support */ Index: linux-4.4.25-rt35/drivers/acpi/acpica/hwregs.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/acpi/acpica/hwregs.c +++ linux-4.4.25-rt35/drivers/acpi/acpica/hwregs.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:272 @ acpi_status acpi_hw_clear_acpi_status(vo ACPI_BITMASK_ALL_FIXED_STATUS, ACPI_FORMAT_UINT64(acpi_gbl_xpm1a_status.address))); - lock_flags = acpi_os_acquire_lock(acpi_gbl_hardware_lock); + raw_spin_lock_irqsave(acpi_gbl_hardware_lock, lock_flags); /* Clear the fixed events in PM1 A/B */ status = acpi_hw_register_write(ACPI_REGISTER_PM1_STATUS, ACPI_BITMASK_ALL_FIXED_STATUS); - acpi_os_release_lock(acpi_gbl_hardware_lock, lock_flags); + raw_spin_unlock_irqrestore(acpi_gbl_hardware_lock, lock_flags); if (ACPI_FAILURE(status)) { goto exit; Index: linux-4.4.25-rt35/drivers/acpi/acpica/hwxface.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/acpi/acpica/hwxface.c +++ linux-4.4.25-rt35/drivers/acpi/acpica/hwxface.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:377 @ acpi_status acpi_write_bit_register(u32 return_ACPI_STATUS(AE_BAD_PARAMETER); } - lock_flags = acpi_os_acquire_lock(acpi_gbl_hardware_lock); + raw_spin_lock_irqsave(acpi_gbl_hardware_lock, lock_flags); /* * At this point, we know that the parent register is one of the @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:438 @ acpi_status acpi_write_bit_register(u32 unlock_and_exit: - acpi_os_release_lock(acpi_gbl_hardware_lock, lock_flags); + raw_spin_unlock_irqrestore(acpi_gbl_hardware_lock, lock_flags); return_ACPI_STATUS(status); } Index: linux-4.4.25-rt35/drivers/acpi/acpica/utmutex.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/acpi/acpica/utmutex.c +++ linux-4.4.25-rt35/drivers/acpi/acpica/utmutex.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:91 @ acpi_status acpi_ut_mutex_initialize(voi return_ACPI_STATUS (status); } - status = acpi_os_create_lock (&acpi_gbl_hardware_lock); + status = acpi_os_create_raw_lock (&acpi_gbl_hardware_lock); if (ACPI_FAILURE (status)) { return_ACPI_STATUS (status); } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:159 @ void acpi_ut_mutex_terminate(void) /* Delete the spinlocks */ acpi_os_delete_lock(acpi_gbl_gpe_lock); - acpi_os_delete_lock(acpi_gbl_hardware_lock); + acpi_os_delete_raw_lock(acpi_gbl_hardware_lock); acpi_os_delete_lock(acpi_gbl_reference_count_lock); /* Delete the reader/writer lock */ Index: linux-4.4.25-rt35/drivers/ata/libata-sff.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/ata/libata-sff.c +++ linux-4.4.25-rt35/drivers/ata/libata-sff.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:681 @ unsigned int ata_sff_data_xfer_noirq(str unsigned long flags; unsigned int consumed; - local_irq_save(flags); + local_irq_save_nort(flags); consumed = ata_sff_data_xfer32(dev, buf, buflen, rw); - local_irq_restore(flags); + local_irq_restore_nort(flags); return consumed; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:722 @ static void ata_pio_sector(struct ata_qu unsigned long flags; /* FIXME: use a bounce buffer */ - local_irq_save(flags); + local_irq_save_nort(flags); buf = kmap_atomic(page); /* do the actual data transfer */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:730 @ static void ata_pio_sector(struct ata_qu do_write); kunmap_atomic(buf); - local_irq_restore(flags); + local_irq_restore_nort(flags); } else { buf = page_address(page); ap->ops->sff_data_xfer(qc->dev, buf + offset, qc->sect_size, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:867 @ next_sg: unsigned long flags; /* FIXME: use bounce buffer */ - local_irq_save(flags); + local_irq_save_nort(flags); buf = kmap_atomic(page); /* do the actual data transfer */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:875 @ next_sg: count, rw); kunmap_atomic(buf); - local_irq_restore(flags); + local_irq_restore_nort(flags); } else { buf = page_address(page); consumed = ap->ops->sff_data_xfer(dev, buf + offset, Index: linux-4.4.25-rt35/drivers/block/zram/zram_drv.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/block/zram/zram_drv.c +++ linux-4.4.25-rt35/drivers/block/zram/zram_drv.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:523 @ static struct zram_meta *zram_meta_alloc goto out_error; } + zram_meta_init_table_locks(meta, disksize); + return meta; out_error: @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:573 @ static int zram_decompress_page(struct z unsigned long handle; size_t size; - bit_spin_lock(ZRAM_ACCESS, &meta->table[index].value); + zram_lock_table(&meta->table[index]); handle = meta->table[index].handle; size = zram_get_obj_size(meta, index); if (!handle || zram_test_flag(meta, index, ZRAM_ZERO)) { - bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value); + zram_unlock_table(&meta->table[index]); clear_page(mem); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:589 @ static int zram_decompress_page(struct z else ret = zcomp_decompress(zram->comp, cmem, size, mem); zs_unmap_object(meta->mem_pool, handle); - bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value); + zram_unlock_table(&meta->table[index]); /* Should NEVER happen. Return bio error if it does. */ if (unlikely(ret)) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:609 @ static int zram_bvec_read(struct zram *z struct zram_meta *meta = zram->meta; page = bvec->bv_page; - bit_spin_lock(ZRAM_ACCESS, &meta->table[index].value); + zram_lock_table(&meta->table[index]); if (unlikely(!meta->table[index].handle) || zram_test_flag(meta, index, ZRAM_ZERO)) { - bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value); + zram_unlock_table(&meta->table[index]); handle_zero_page(bvec); return 0; } - bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value); + zram_unlock_table(&meta->table[index]); if (is_partial_io(bvec)) /* Use a temporary buffer to decompress the page */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:694 @ static int zram_bvec_write(struct zram * if (user_mem) kunmap_atomic(user_mem); /* Free memory associated with this sector now. */ - bit_spin_lock(ZRAM_ACCESS, &meta->table[index].value); + zram_lock_table(&meta->table[index]); zram_free_page(zram, index); zram_set_flag(meta, index, ZRAM_ZERO); - bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value); + zram_unlock_table(&meta->table[index]); atomic64_inc(&zram->stats.zero_pages); ret = 0; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:757 @ static int zram_bvec_write(struct zram * * Free memory associated with this sector * before overwriting unused sectors. */ - bit_spin_lock(ZRAM_ACCESS, &meta->table[index].value); + zram_lock_table(&meta->table[index]); zram_free_page(zram, index); meta->table[index].handle = handle; zram_set_obj_size(meta, index, clen); - bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value); + zram_unlock_table(&meta->table[index]); /* Update stats */ atomic64_add(clen, &zram->stats.compr_data_size); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:805 @ static void zram_bio_discard(struct zram } while (n >= PAGE_SIZE) { - bit_spin_lock(ZRAM_ACCESS, &meta->table[index].value); + zram_lock_table(&meta->table[index]); zram_free_page(zram, index); - bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value); + zram_unlock_table(&meta->table[index]); atomic64_inc(&zram->stats.notify_free); index++; n -= PAGE_SIZE; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:933 @ static void zram_slot_free_notify(struct zram = bdev->bd_disk->private_data; meta = zram->meta; - bit_spin_lock(ZRAM_ACCESS, &meta->table[index].value); + zram_lock_table(&meta->table[index]); zram_free_page(zram, index); - bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value); + zram_unlock_table(&meta->table[index]); atomic64_inc(&zram->stats.notify_free); } Index: linux-4.4.25-rt35/drivers/block/zram/zram_drv.h =================================================================== --- linux-4.4.25-rt35.orig/drivers/block/zram/zram_drv.h +++ linux-4.4.25-rt35/drivers/block/zram/zram_drv.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:75 @ enum zram_pageflags { struct zram_table_entry { unsigned long handle; unsigned long value; +#ifdef CONFIG_PREEMPT_RT_BASE + spinlock_t lock; +#endif }; struct zram_stats { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:125 @ struct zram { */ bool claim; /* Protected by bdev->bd_mutex */ }; + +#ifndef CONFIG_PREEMPT_RT_BASE +static inline void zram_lock_table(struct zram_table_entry *table) +{ + bit_spin_lock(ZRAM_ACCESS, &table->value); +} + +static inline void zram_unlock_table(struct zram_table_entry *table) +{ + bit_spin_unlock(ZRAM_ACCESS, &table->value); +} + +static inline void zram_meta_init_table_locks(struct zram_meta *meta, u64 disksize) { } +#else /* CONFIG_PREEMPT_RT_BASE */ +static inline void zram_lock_table(struct zram_table_entry *table) +{ + spin_lock(&table->lock); + __set_bit(ZRAM_ACCESS, &table->value); +} + +static inline void zram_unlock_table(struct zram_table_entry *table) +{ + __clear_bit(ZRAM_ACCESS, &table->value); + spin_unlock(&table->lock); +} + +static inline void zram_meta_init_table_locks(struct zram_meta *meta, u64 disksize) +{ + size_t num_pages = disksize >> PAGE_SHIFT; + size_t index; + + for (index = 0; index < num_pages; index++) { + spinlock_t *lock = &meta->table[index].lock; + spin_lock_init(lock); + } +} +#endif /* CONFIG_PREEMPT_RT_BASE */ + #endif Index: linux-4.4.25-rt35/drivers/char/random.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/char/random.c +++ linux-4.4.25-rt35/drivers/char/random.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:802 @ static void add_timer_randomness(struct } sample; long delta, delta2, delta3; - preempt_disable(); - sample.jiffies = jiffies; sample.cycles = random_get_entropy(); sample.num = num; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:842 @ static void add_timer_randomness(struct */ credit_entropy_bits(r, min_t(int, fls(delta>>1), 11)); } - preempt_enable(); } void add_input_randomness(unsigned int type, unsigned int code, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:894 @ static __u32 get_reg(struct fast_pool *f return *(ptr + f->reg_idx++); } -void add_interrupt_randomness(int irq, int irq_flags) +void add_interrupt_randomness(int irq, int irq_flags, __u64 ip) { struct entropy_store *r; struct fast_pool *fast_pool = this_cpu_ptr(&irq_randomness); - struct pt_regs *regs = get_irq_regs(); unsigned long now = jiffies; cycles_t cycles = random_get_entropy(); __u32 c_high, j_high; - __u64 ip; unsigned long seed; int credit = 0; if (cycles == 0) - cycles = get_reg(fast_pool, regs); + cycles = get_reg(fast_pool, NULL); c_high = (sizeof(cycles) > 4) ? cycles >> 32 : 0; j_high = (sizeof(now) > 4) ? now >> 32 : 0; fast_pool->pool[0] ^= cycles ^ j_high ^ irq; fast_pool->pool[1] ^= now ^ c_high; - ip = regs ? instruction_pointer(regs) : _RET_IP_; + if (!ip) + ip = _RET_IP_; fast_pool->pool[2] ^= ip; fast_pool->pool[3] ^= (sizeof(ip) > 4) ? ip >> 32 : - get_reg(fast_pool, regs); + get_reg(fast_pool, NULL); fast_mix(fast_pool); add_interrupt_bench(cycles); Index: linux-4.4.25-rt35/drivers/clk/at91/clk-generated.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/clk-generated.c +++ linux-4.4.25-rt35/drivers/clk/at91/clk-generated.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:18 @ #include <linux/clkdev.h> #include <linux/clk/at91_pmc.h> #include <linux/of.h> -#include <linux/of_address.h> -#include <linux/io.h> +#include <linux/mfd/syscon.h> +#include <linux/regmap.h> #include "pmc.h" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:31 @ struct clk_generated { struct clk_hw hw; - struct at91_pmc *pmc; + struct regmap *regmap; struct clk_range range; + spinlock_t *lock; u32 id; u32 gckdiv; u8 parent_id; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:45 @ struct clk_generated { static int clk_generated_enable(struct clk_hw *hw) { struct clk_generated *gck = to_clk_generated(hw); - struct at91_pmc *pmc = gck->pmc; - u32 tmp; + unsigned long flags; pr_debug("GCLK: %s, gckdiv = %d, parent id = %d\n", __func__, gck->gckdiv, gck->parent_id); - pmc_lock(pmc); - pmc_write(pmc, AT91_PMC_PCR, (gck->id & AT91_PMC_PCR_PID_MASK)); - tmp = pmc_read(pmc, AT91_PMC_PCR) & - ~(AT91_PMC_PCR_GCKDIV_MASK | AT91_PMC_PCR_GCKCSS_MASK); - pmc_write(pmc, AT91_PMC_PCR, tmp | AT91_PMC_PCR_GCKCSS(gck->parent_id) - | AT91_PMC_PCR_CMD - | AT91_PMC_PCR_GCKDIV(gck->gckdiv) - | AT91_PMC_PCR_GCKEN); - pmc_unlock(pmc); + spin_lock_irqsave(gck->lock, flags); + regmap_write(gck->regmap, AT91_PMC_PCR, + (gck->id & AT91_PMC_PCR_PID_MASK)); + regmap_update_bits(gck->regmap, AT91_PMC_PCR, + AT91_PMC_PCR_GCKDIV_MASK | AT91_PMC_PCR_GCKCSS_MASK | + AT91_PMC_PCR_CMD | AT91_PMC_PCR_GCKEN, + AT91_PMC_PCR_GCKCSS(gck->parent_id) | + AT91_PMC_PCR_CMD | + AT91_PMC_PCR_GCKDIV(gck->gckdiv) | + AT91_PMC_PCR_GCKEN); + spin_unlock_irqrestore(gck->lock, flags); return 0; } static void clk_generated_disable(struct clk_hw *hw) { struct clk_generated *gck = to_clk_generated(hw); - struct at91_pmc *pmc = gck->pmc; - u32 tmp; + unsigned long flags; - pmc_lock(pmc); - pmc_write(pmc, AT91_PMC_PCR, (gck->id & AT91_PMC_PCR_PID_MASK)); - tmp = pmc_read(pmc, AT91_PMC_PCR) & ~AT91_PMC_PCR_GCKEN; - pmc_write(pmc, AT91_PMC_PCR, tmp | AT91_PMC_PCR_CMD); - pmc_unlock(pmc); + spin_lock_irqsave(gck->lock, flags); + regmap_write(gck->regmap, AT91_PMC_PCR, + (gck->id & AT91_PMC_PCR_PID_MASK)); + regmap_update_bits(gck->regmap, AT91_PMC_PCR, + AT91_PMC_PCR_CMD | AT91_PMC_PCR_GCKEN, + AT91_PMC_PCR_CMD); + spin_unlock_irqrestore(gck->lock, flags); } static int clk_generated_is_enabled(struct clk_hw *hw) { struct clk_generated *gck = to_clk_generated(hw); - struct at91_pmc *pmc = gck->pmc; - int ret; + unsigned long flags; + unsigned int status; - pmc_lock(pmc); - pmc_write(pmc, AT91_PMC_PCR, (gck->id & AT91_PMC_PCR_PID_MASK)); - ret = !!(pmc_read(pmc, AT91_PMC_PCR) & AT91_PMC_PCR_GCKEN); - pmc_unlock(pmc); + spin_lock_irqsave(gck->lock, flags); + regmap_write(gck->regmap, AT91_PMC_PCR, + (gck->id & AT91_PMC_PCR_PID_MASK)); + regmap_read(gck->regmap, AT91_PMC_PCR, &status); + spin_unlock_irqrestore(gck->lock, flags); - return ret; + return status & AT91_PMC_PCR_GCKEN ? 1 : 0; } static unsigned long @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:221 @ static const struct clk_ops generated_op */ static void clk_generated_startup(struct clk_generated *gck) { - struct at91_pmc *pmc = gck->pmc; u32 tmp; + unsigned long flags; - pmc_lock(pmc); - pmc_write(pmc, AT91_PMC_PCR, (gck->id & AT91_PMC_PCR_PID_MASK)); - tmp = pmc_read(pmc, AT91_PMC_PCR); - pmc_unlock(pmc); + spin_lock_irqsave(gck->lock, flags); + regmap_write(gck->regmap, AT91_PMC_PCR, + (gck->id & AT91_PMC_PCR_PID_MASK)); + regmap_read(gck->regmap, AT91_PMC_PCR, &tmp); + spin_unlock_irqrestore(gck->lock, flags); gck->parent_id = (tmp & AT91_PMC_PCR_GCKCSS_MASK) >> AT91_PMC_PCR_GCKCSS_OFFSET; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:237 @ static void clk_generated_startup(struct } static struct clk * __init -at91_clk_register_generated(struct at91_pmc *pmc, const char *name, - const char **parent_names, u8 num_parents, +at91_clk_register_generated(struct regmap *regmap, spinlock_t *lock, const char + *name, const char **parent_names, u8 num_parents, u8 id, const struct clk_range *range) { struct clk_generated *gck; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:257 @ at91_clk_register_generated(struct at91_ gck->id = id; gck->hw.init = &init; - gck->pmc = pmc; + gck->regmap = regmap; + gck->lock = lock; gck->range = *range; clk = clk_register(NULL, &gck->hw); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:270 @ at91_clk_register_generated(struct at91_ return clk; } -void __init of_sama5d2_clk_generated_setup(struct device_node *np, - struct at91_pmc *pmc) +void __init of_sama5d2_clk_generated_setup(struct device_node *np) { int num; u32 id; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:280 @ void __init of_sama5d2_clk_generated_set const char *parent_names[GENERATED_SOURCE_MAX]; struct device_node *gcknp; struct clk_range range = CLK_RANGE(0, 0); + struct regmap *regmap; num_parents = of_clk_get_parent_count(np); if (num_parents <= 0 || num_parents > GENERATED_SOURCE_MAX) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:292 @ void __init of_sama5d2_clk_generated_set if (!num || num > PERIPHERAL_MAX) return; + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) + return; + for_each_child_of_node(np, gcknp) { if (of_property_read_u32(gcknp, "reg", &id)) continue; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:309 @ void __init of_sama5d2_clk_generated_set of_at91_get_clk_range(gcknp, "atmel,clk-output-range", &range); - clk = at91_clk_register_generated(pmc, name, parent_names, - num_parents, id, &range); + clk = at91_clk_register_generated(regmap, &pmc_pcr_lock, name, + parent_names, num_parents, + id, &range); if (IS_ERR(clk)) continue; of_clk_add_provider(gcknp, of_clk_src_simple_get, clk); } } +CLK_OF_DECLARE(of_sama5d2_clk_generated_setup, "atmel,sama5d2-clk-generated", + of_sama5d2_clk_generated_setup); Index: linux-4.4.25-rt35/drivers/clk/at91/clk-h32mx.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/clk-h32mx.c +++ linux-4.4.25-rt35/drivers/clk/at91/clk-h32mx.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:18 @ #include <linux/clk-provider.h> #include <linux/clkdev.h> #include <linux/clk/at91_pmc.h> -#include <linux/delay.h> #include <linux/of.h> -#include <linux/of_address.h> -#include <linux/of_irq.h> -#include <linux/io.h> -#include <linux/interrupt.h> -#include <linux/irq.h> -#include <linux/sched.h> -#include <linux/wait.h> +#include <linux/regmap.h> +#include <linux/mfd/syscon.h> #include "pmc.h" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:28 @ struct clk_sama5d4_h32mx { struct clk_hw hw; - struct at91_pmc *pmc; + struct regmap *regmap; }; #define to_clk_sama5d4_h32mx(hw) container_of(hw, struct clk_sama5d4_h32mx, hw) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:37 @ static unsigned long clk_sama5d4_h32mx_r unsigned long parent_rate) { struct clk_sama5d4_h32mx *h32mxclk = to_clk_sama5d4_h32mx(hw); + unsigned int mckr; - if (pmc_read(h32mxclk->pmc, AT91_PMC_MCKR) & AT91_PMC_H32MXDIV) + regmap_read(h32mxclk->regmap, AT91_PMC_MCKR, &mckr); + if (mckr & AT91_PMC_H32MXDIV) return parent_rate / 2; if (parent_rate > H32MX_MAX_FREQ) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:69 @ static int clk_sama5d4_h32mx_set_rate(st unsigned long parent_rate) { struct clk_sama5d4_h32mx *h32mxclk = to_clk_sama5d4_h32mx(hw); - struct at91_pmc *pmc = h32mxclk->pmc; - u32 tmp; + u32 mckr = 0; if (parent_rate != rate && (parent_rate / 2) != rate) return -EINVAL; - pmc_lock(pmc); - tmp = pmc_read(pmc, AT91_PMC_MCKR) & ~AT91_PMC_H32MXDIV; if ((parent_rate / 2) == rate) - tmp |= AT91_PMC_H32MXDIV; - pmc_write(pmc, AT91_PMC_MCKR, tmp); - pmc_unlock(pmc); + mckr = AT91_PMC_H32MXDIV; + + regmap_update_bits(h32mxclk->regmap, AT91_PMC_MCKR, + AT91_PMC_H32MXDIV, mckr); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:89 @ static const struct clk_ops h32mx_ops = .set_rate = clk_sama5d4_h32mx_set_rate, }; -void __init of_sama5d4_clk_h32mx_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_sama5d4_clk_h32mx_setup(struct device_node *np) { struct clk_sama5d4_h32mx *h32mxclk; struct clk_init_data init; const char *parent_name; + struct regmap *regmap; struct clk *clk; + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) + return; + h32mxclk = kzalloc(sizeof(*h32mxclk), GFP_KERNEL); if (!h32mxclk) return; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:114 @ void __init of_sama5d4_clk_h32mx_setup(s init.flags = CLK_SET_RATE_GATE; h32mxclk->hw.init = &init; - h32mxclk->pmc = pmc; + h32mxclk->regmap = regmap; clk = clk_register(NULL, &h32mxclk->hw); if (IS_ERR(clk)) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:124 @ void __init of_sama5d4_clk_h32mx_setup(s of_clk_add_provider(np, of_clk_src_simple_get, clk); } +CLK_OF_DECLARE(of_sama5d4_clk_h32mx_setup, "atmel,sama5d4-clk-h32mx", + of_sama5d4_clk_h32mx_setup); Index: linux-4.4.25-rt35/drivers/clk/at91/clk-main.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/clk-main.c +++ linux-4.4.25-rt35/drivers/clk/at91/clk-main.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:16 @ #include <linux/clk/at91_pmc.h> #include <linux/delay.h> #include <linux/of.h> -#include <linux/of_address.h> -#include <linux/of_irq.h> -#include <linux/io.h> -#include <linux/interrupt.h> -#include <linux/irq.h> -#include <linux/sched.h> -#include <linux/wait.h> +#include <linux/mfd/syscon.h> +#include <linux/regmap.h> #include "pmc.h" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:32 @ struct clk_main_osc { struct clk_hw hw; - struct at91_pmc *pmc; - unsigned int irq; - wait_queue_head_t wait; + struct regmap *regmap; }; #define to_clk_main_osc(hw) container_of(hw, struct clk_main_osc, hw) struct clk_main_rc_osc { struct clk_hw hw; - struct at91_pmc *pmc; - unsigned int irq; - wait_queue_head_t wait; + struct regmap *regmap; unsigned long frequency; unsigned long accuracy; }; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:48 @ struct clk_main_rc_osc { struct clk_rm9200_main { struct clk_hw hw; - struct at91_pmc *pmc; + struct regmap *regmap; }; #define to_clk_rm9200_main(hw) container_of(hw, struct clk_rm9200_main, hw) struct clk_sam9x5_main { struct clk_hw hw; - struct at91_pmc *pmc; - unsigned int irq; - wait_queue_head_t wait; + struct regmap *regmap; u8 parent; }; #define to_clk_sam9x5_main(hw) container_of(hw, struct clk_sam9x5_main, hw) -static irqreturn_t clk_main_osc_irq_handler(int irq, void *dev_id) +static inline bool clk_main_osc_ready(struct regmap *regmap) { - struct clk_main_osc *osc = dev_id; + unsigned int status; - wake_up(&osc->wait); - disable_irq_nosync(osc->irq); + regmap_read(regmap, AT91_PMC_SR, &status); - return IRQ_HANDLED; + return status & AT91_PMC_MOSCS; } static int clk_main_osc_prepare(struct clk_hw *hw) { struct clk_main_osc *osc = to_clk_main_osc(hw); - struct at91_pmc *pmc = osc->pmc; + struct regmap *regmap = osc->regmap; u32 tmp; - tmp = pmc_read(pmc, AT91_CKGR_MOR) & ~MOR_KEY_MASK; + regmap_read(regmap, AT91_CKGR_MOR, &tmp); + tmp &= ~MOR_KEY_MASK; + if (tmp & AT91_PMC_OSCBYPASS) return 0; if (!(tmp & AT91_PMC_MOSCEN)) { tmp |= AT91_PMC_MOSCEN | AT91_PMC_KEY; - pmc_write(pmc, AT91_CKGR_MOR, tmp); + regmap_write(regmap, AT91_CKGR_MOR, tmp); } - while (!(pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_MOSCS)) { - enable_irq(osc->irq); - wait_event(osc->wait, - pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_MOSCS); - } + while (!clk_main_osc_ready(regmap)) + cpu_relax(); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:96 @ static int clk_main_osc_prepare(struct c static void clk_main_osc_unprepare(struct clk_hw *hw) { struct clk_main_osc *osc = to_clk_main_osc(hw); - struct at91_pmc *pmc = osc->pmc; - u32 tmp = pmc_read(pmc, AT91_CKGR_MOR); + struct regmap *regmap = osc->regmap; + u32 tmp; + regmap_read(regmap, AT91_CKGR_MOR, &tmp); if (tmp & AT91_PMC_OSCBYPASS) return; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:107 @ static void clk_main_osc_unprepare(struc return; tmp &= ~(AT91_PMC_KEY | AT91_PMC_MOSCEN); - pmc_write(pmc, AT91_CKGR_MOR, tmp | AT91_PMC_KEY); + regmap_write(regmap, AT91_CKGR_MOR, tmp | AT91_PMC_KEY); } static int clk_main_osc_is_prepared(struct clk_hw *hw) { struct clk_main_osc *osc = to_clk_main_osc(hw); - struct at91_pmc *pmc = osc->pmc; - u32 tmp = pmc_read(pmc, AT91_CKGR_MOR); + struct regmap *regmap = osc->regmap; + u32 tmp, status; + regmap_read(regmap, AT91_CKGR_MOR, &tmp); if (tmp & AT91_PMC_OSCBYPASS) return 1; - return !!((pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_MOSCS) && - (pmc_read(pmc, AT91_CKGR_MOR) & AT91_PMC_MOSCEN)); + regmap_read(regmap, AT91_PMC_SR, &status); + + return (status & AT91_PMC_MOSCS) && (tmp & AT91_PMC_MOSCEN); } static const struct clk_ops main_osc_ops = { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:132 @ static const struct clk_ops main_osc_ops }; static struct clk * __init -at91_clk_register_main_osc(struct at91_pmc *pmc, - unsigned int irq, +at91_clk_register_main_osc(struct regmap *regmap, const char *name, const char *parent_name, bool bypass) { - int ret; struct clk_main_osc *osc; struct clk *clk = NULL; struct clk_init_data init; - if (!pmc || !irq || !name || !parent_name) + if (!name || !parent_name) return ERR_PTR(-EINVAL); osc = kzalloc(sizeof(*osc), GFP_KERNEL); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:155 @ at91_clk_register_main_osc(struct at91_p init.flags = CLK_IGNORE_UNUSED; osc->hw.init = &init; - osc->pmc = pmc; - osc->irq = irq; - - init_waitqueue_head(&osc->wait); - irq_set_status_flags(osc->irq, IRQ_NOAUTOEN); - ret = request_irq(osc->irq, clk_main_osc_irq_handler, - IRQF_TRIGGER_HIGH, name, osc); - if (ret) { - kfree(osc); - return ERR_PTR(ret); - } + osc->regmap = regmap; if (bypass) - pmc_write(pmc, AT91_CKGR_MOR, - (pmc_read(pmc, AT91_CKGR_MOR) & - ~(MOR_KEY_MASK | AT91_PMC_MOSCEN)) | - AT91_PMC_OSCBYPASS | AT91_PMC_KEY); + regmap_update_bits(regmap, + AT91_CKGR_MOR, MOR_KEY_MASK | + AT91_PMC_MOSCEN, + AT91_PMC_OSCBYPASS | AT91_PMC_KEY); clk = clk_register(NULL, &osc->hw); - if (IS_ERR(clk)) { - free_irq(irq, osc); + if (IS_ERR(clk)) kfree(osc); - } return clk; } -void __init of_at91rm9200_clk_main_osc_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91rm9200_clk_main_osc_setup(struct device_node *np) { struct clk *clk; - unsigned int irq; const char *name = np->name; const char *parent_name; + struct regmap *regmap; bool bypass; of_property_read_string(np, "clock-output-names", &name); bypass = of_property_read_bool(np, "atmel,osc-bypass"); parent_name = of_clk_get_parent_name(np, 0); - irq = irq_of_parse_and_map(np, 0); - if (!irq) + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) return; - clk = at91_clk_register_main_osc(pmc, irq, name, parent_name, bypass); + clk = at91_clk_register_main_osc(regmap, name, parent_name, bypass); if (IS_ERR(clk)) return; of_clk_add_provider(np, of_clk_src_simple_get, clk); } +CLK_OF_DECLARE(at91rm9200_clk_main_osc, "atmel,at91rm9200-clk-main-osc", + of_at91rm9200_clk_main_osc_setup); -static irqreturn_t clk_main_rc_osc_irq_handler(int irq, void *dev_id) +static bool clk_main_rc_osc_ready(struct regmap *regmap) { - struct clk_main_rc_osc *osc = dev_id; + unsigned int status; - wake_up(&osc->wait); - disable_irq_nosync(osc->irq); + regmap_read(regmap, AT91_PMC_SR, &status); - return IRQ_HANDLED; + return status & AT91_PMC_MOSCRCS; } static int clk_main_rc_osc_prepare(struct clk_hw *hw) { struct clk_main_rc_osc *osc = to_clk_main_rc_osc(hw); - struct at91_pmc *pmc = osc->pmc; - u32 tmp; + struct regmap *regmap = osc->regmap; + unsigned int mor; - tmp = pmc_read(pmc, AT91_CKGR_MOR) & ~MOR_KEY_MASK; + regmap_read(regmap, AT91_CKGR_MOR, &mor); - if (!(tmp & AT91_PMC_MOSCRCEN)) { - tmp |= AT91_PMC_MOSCRCEN | AT91_PMC_KEY; - pmc_write(pmc, AT91_CKGR_MOR, tmp); - } + if (!(mor & AT91_PMC_MOSCRCEN)) + regmap_update_bits(regmap, AT91_CKGR_MOR, + MOR_KEY_MASK | AT91_PMC_MOSCRCEN, + AT91_PMC_MOSCRCEN | AT91_PMC_KEY); - while (!(pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_MOSCRCS)) { - enable_irq(osc->irq); - wait_event(osc->wait, - pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_MOSCRCS); - } + while (!clk_main_rc_osc_ready(regmap)) + cpu_relax(); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:226 @ static int clk_main_rc_osc_prepare(struc static void clk_main_rc_osc_unprepare(struct clk_hw *hw) { struct clk_main_rc_osc *osc = to_clk_main_rc_osc(hw); - struct at91_pmc *pmc = osc->pmc; - u32 tmp = pmc_read(pmc, AT91_CKGR_MOR); + struct regmap *regmap = osc->regmap; + unsigned int mor; - if (!(tmp & AT91_PMC_MOSCRCEN)) + regmap_read(regmap, AT91_CKGR_MOR, &mor); + + if (!(mor & AT91_PMC_MOSCRCEN)) return; - tmp &= ~(MOR_KEY_MASK | AT91_PMC_MOSCRCEN); - pmc_write(pmc, AT91_CKGR_MOR, tmp | AT91_PMC_KEY); + regmap_update_bits(regmap, AT91_CKGR_MOR, + MOR_KEY_MASK | AT91_PMC_MOSCRCEN, AT91_PMC_KEY); } static int clk_main_rc_osc_is_prepared(struct clk_hw *hw) { struct clk_main_rc_osc *osc = to_clk_main_rc_osc(hw); - struct at91_pmc *pmc = osc->pmc; + struct regmap *regmap = osc->regmap; + unsigned int mor, status; + + regmap_read(regmap, AT91_CKGR_MOR, &mor); + regmap_read(regmap, AT91_PMC_SR, &status); - return !!((pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_MOSCRCS) && - (pmc_read(pmc, AT91_CKGR_MOR) & AT91_PMC_MOSCRCEN)); + return (mor & AT91_PMC_MOSCRCEN) && (status & AT91_PMC_MOSCRCS); } static unsigned long clk_main_rc_osc_recalc_rate(struct clk_hw *hw, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:275 @ static const struct clk_ops main_rc_osc_ }; static struct clk * __init -at91_clk_register_main_rc_osc(struct at91_pmc *pmc, - unsigned int irq, +at91_clk_register_main_rc_osc(struct regmap *regmap, const char *name, u32 frequency, u32 accuracy) { - int ret; struct clk_main_rc_osc *osc; struct clk *clk = NULL; struct clk_init_data init; - if (!pmc || !irq || !name || !frequency) + if (!name || !frequency) return ERR_PTR(-EINVAL); osc = kzalloc(sizeof(*osc), GFP_KERNEL); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:297 @ at91_clk_register_main_rc_osc(struct at9 init.flags = CLK_IS_ROOT | CLK_IGNORE_UNUSED; osc->hw.init = &init; - osc->pmc = pmc; - osc->irq = irq; + osc->regmap = regmap; osc->frequency = frequency; osc->accuracy = accuracy; - init_waitqueue_head(&osc->wait); - irq_set_status_flags(osc->irq, IRQ_NOAUTOEN); - ret = request_irq(osc->irq, clk_main_rc_osc_irq_handler, - IRQF_TRIGGER_HIGH, name, osc); - if (ret) - return ERR_PTR(ret); - clk = clk_register(NULL, &osc->hw); - if (IS_ERR(clk)) { - free_irq(irq, osc); + if (IS_ERR(clk)) kfree(osc); - } return clk; } -void __init of_at91sam9x5_clk_main_rc_osc_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91sam9x5_clk_main_rc_osc_setup(struct device_node *np) { struct clk *clk; - unsigned int irq; u32 frequency = 0; u32 accuracy = 0; const char *name = np->name; + struct regmap *regmap; of_property_read_string(np, "clock-output-names", &name); of_property_read_u32(np, "clock-frequency", &frequency); of_property_read_u32(np, "clock-accuracy", &accuracy); - irq = irq_of_parse_and_map(np, 0); - if (!irq) + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) return; - clk = at91_clk_register_main_rc_osc(pmc, irq, name, frequency, - accuracy); + clk = at91_clk_register_main_rc_osc(regmap, name, frequency, accuracy); if (IS_ERR(clk)) return; of_clk_add_provider(np, of_clk_src_simple_get, clk); } +CLK_OF_DECLARE(at91sam9x5_clk_main_rc_osc, "atmel,at91sam9x5-clk-main-rc-osc", + of_at91sam9x5_clk_main_rc_osc_setup); -static int clk_main_probe_frequency(struct at91_pmc *pmc) +static int clk_main_probe_frequency(struct regmap *regmap) { unsigned long prep_time, timeout; - u32 tmp; + unsigned int mcfr; timeout = jiffies + usecs_to_jiffies(MAINFRDY_TIMEOUT); do { prep_time = jiffies; - tmp = pmc_read(pmc, AT91_CKGR_MCFR); - if (tmp & AT91_PMC_MAINRDY) + regmap_read(regmap, AT91_CKGR_MCFR, &mcfr); + if (mcfr & AT91_PMC_MAINRDY) return 0; usleep_range(MAINF_LOOP_MIN_WAIT, MAINF_LOOP_MAX_WAIT); } while (time_before(prep_time, timeout)); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:351 @ static int clk_main_probe_frequency(stru return -ETIMEDOUT; } -static unsigned long clk_main_recalc_rate(struct at91_pmc *pmc, +static unsigned long clk_main_recalc_rate(struct regmap *regmap, unsigned long parent_rate) { - u32 tmp; + unsigned int mcfr; if (parent_rate) return parent_rate; pr_warn("Main crystal frequency not set, using approximate value\n"); - tmp = pmc_read(pmc, AT91_CKGR_MCFR); - if (!(tmp & AT91_PMC_MAINRDY)) + regmap_read(regmap, AT91_CKGR_MCFR, &mcfr); + if (!(mcfr & AT91_PMC_MAINRDY)) return 0; - return ((tmp & AT91_PMC_MAINF) * SLOW_CLOCK_FREQ) / MAINF_DIV; + return ((mcfr & AT91_PMC_MAINF) * SLOW_CLOCK_FREQ) / MAINF_DIV; } static int clk_rm9200_main_prepare(struct clk_hw *hw) { struct clk_rm9200_main *clkmain = to_clk_rm9200_main(hw); - return clk_main_probe_frequency(clkmain->pmc); + return clk_main_probe_frequency(clkmain->regmap); } static int clk_rm9200_main_is_prepared(struct clk_hw *hw) { struct clk_rm9200_main *clkmain = to_clk_rm9200_main(hw); + unsigned int status; + + regmap_read(clkmain->regmap, AT91_CKGR_MCFR, &status); - return !!(pmc_read(clkmain->pmc, AT91_CKGR_MCFR) & AT91_PMC_MAINRDY); + return status & AT91_PMC_MAINRDY ? 1 : 0; } static unsigned long clk_rm9200_main_recalc_rate(struct clk_hw *hw, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:389 @ static unsigned long clk_rm9200_main_rec { struct clk_rm9200_main *clkmain = to_clk_rm9200_main(hw); - return clk_main_recalc_rate(clkmain->pmc, parent_rate); + return clk_main_recalc_rate(clkmain->regmap, parent_rate); } static const struct clk_ops rm9200_main_ops = { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:399 @ static const struct clk_ops rm9200_main_ }; static struct clk * __init -at91_clk_register_rm9200_main(struct at91_pmc *pmc, +at91_clk_register_rm9200_main(struct regmap *regmap, const char *name, const char *parent_name) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:407 @ at91_clk_register_rm9200_main(struct at9 struct clk *clk = NULL; struct clk_init_data init; - if (!pmc || !name) + if (!name) return ERR_PTR(-EINVAL); if (!parent_name) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:424 @ at91_clk_register_rm9200_main(struct at9 init.flags = 0; clkmain->hw.init = &init; - clkmain->pmc = pmc; + clkmain->regmap = regmap; clk = clk_register(NULL, &clkmain->hw); if (IS_ERR(clk)) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:433 @ at91_clk_register_rm9200_main(struct at9 return clk; } -void __init of_at91rm9200_clk_main_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91rm9200_clk_main_setup(struct device_node *np) { struct clk *clk; const char *parent_name; const char *name = np->name; + struct regmap *regmap; parent_name = of_clk_get_parent_name(np, 0); of_property_read_string(np, "clock-output-names", &name); - clk = at91_clk_register_rm9200_main(pmc, name, parent_name); + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) + return; + + clk = at91_clk_register_rm9200_main(regmap, name, parent_name); if (IS_ERR(clk)) return; of_clk_add_provider(np, of_clk_src_simple_get, clk); } +CLK_OF_DECLARE(at91rm9200_clk_main, "atmel,at91rm9200-clk-main", + of_at91rm9200_clk_main_setup); -static irqreturn_t clk_sam9x5_main_irq_handler(int irq, void *dev_id) +static inline bool clk_sam9x5_main_ready(struct regmap *regmap) { - struct clk_sam9x5_main *clkmain = dev_id; + unsigned int status; - wake_up(&clkmain->wait); - disable_irq_nosync(clkmain->irq); + regmap_read(regmap, AT91_PMC_SR, &status); - return IRQ_HANDLED; + return status & AT91_PMC_MOSCSELS ? 1 : 0; } static int clk_sam9x5_main_prepare(struct clk_hw *hw) { struct clk_sam9x5_main *clkmain = to_clk_sam9x5_main(hw); - struct at91_pmc *pmc = clkmain->pmc; + struct regmap *regmap = clkmain->regmap; - while (!(pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_MOSCSELS)) { - enable_irq(clkmain->irq); - wait_event(clkmain->wait, - pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_MOSCSELS); - } + while (!clk_sam9x5_main_ready(regmap)) + cpu_relax(); - return clk_main_probe_frequency(pmc); + return clk_main_probe_frequency(regmap); } static int clk_sam9x5_main_is_prepared(struct clk_hw *hw) { struct clk_sam9x5_main *clkmain = to_clk_sam9x5_main(hw); - return !!(pmc_read(clkmain->pmc, AT91_PMC_SR) & AT91_PMC_MOSCSELS); + return clk_sam9x5_main_ready(clkmain->regmap); } static unsigned long clk_sam9x5_main_recalc_rate(struct clk_hw *hw, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:488 @ static unsigned long clk_sam9x5_main_rec { struct clk_sam9x5_main *clkmain = to_clk_sam9x5_main(hw); - return clk_main_recalc_rate(clkmain->pmc, parent_rate); + return clk_main_recalc_rate(clkmain->regmap, parent_rate); } static int clk_sam9x5_main_set_parent(struct clk_hw *hw, u8 index) { struct clk_sam9x5_main *clkmain = to_clk_sam9x5_main(hw); - struct at91_pmc *pmc = clkmain->pmc; - u32 tmp; + struct regmap *regmap = clkmain->regmap; + unsigned int tmp; if (index > 1) return -EINVAL; - tmp = pmc_read(pmc, AT91_CKGR_MOR) & ~MOR_KEY_MASK; + regmap_read(regmap, AT91_CKGR_MOR, &tmp); + tmp &= ~MOR_KEY_MASK; if (index && !(tmp & AT91_PMC_MOSCSEL)) - pmc_write(pmc, AT91_CKGR_MOR, tmp | AT91_PMC_MOSCSEL); + regmap_write(regmap, AT91_CKGR_MOR, tmp | AT91_PMC_MOSCSEL); else if (!index && (tmp & AT91_PMC_MOSCSEL)) - pmc_write(pmc, AT91_CKGR_MOR, tmp & ~AT91_PMC_MOSCSEL); + regmap_write(regmap, AT91_CKGR_MOR, tmp & ~AT91_PMC_MOSCSEL); - while (!(pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_MOSCSELS)) { - enable_irq(clkmain->irq); - wait_event(clkmain->wait, - pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_MOSCSELS); - } + while (!clk_sam9x5_main_ready(regmap)) + cpu_relax(); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:517 @ static int clk_sam9x5_main_set_parent(st static u8 clk_sam9x5_main_get_parent(struct clk_hw *hw) { struct clk_sam9x5_main *clkmain = to_clk_sam9x5_main(hw); + unsigned int status; + + regmap_read(clkmain->regmap, AT91_CKGR_MOR, &status); - return !!(pmc_read(clkmain->pmc, AT91_CKGR_MOR) & AT91_PMC_MOSCEN); + return status & AT91_PMC_MOSCEN ? 1 : 0; } static const struct clk_ops sam9x5_main_ops = { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:533 @ static const struct clk_ops sam9x5_main_ }; static struct clk * __init -at91_clk_register_sam9x5_main(struct at91_pmc *pmc, - unsigned int irq, +at91_clk_register_sam9x5_main(struct regmap *regmap, const char *name, const char **parent_names, int num_parents) { - int ret; struct clk_sam9x5_main *clkmain; struct clk *clk = NULL; struct clk_init_data init; + unsigned int status; - if (!pmc || !irq || !name) + if (!name) return ERR_PTR(-EINVAL); if (!parent_names || !num_parents) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:560 @ at91_clk_register_sam9x5_main(struct at9 init.flags = CLK_SET_PARENT_GATE; clkmain->hw.init = &init; - clkmain->pmc = pmc; - clkmain->irq = irq; - clkmain->parent = !!(pmc_read(clkmain->pmc, AT91_CKGR_MOR) & - AT91_PMC_MOSCEN); - init_waitqueue_head(&clkmain->wait); - irq_set_status_flags(clkmain->irq, IRQ_NOAUTOEN); - ret = request_irq(clkmain->irq, clk_sam9x5_main_irq_handler, - IRQF_TRIGGER_HIGH, name, clkmain); - if (ret) - return ERR_PTR(ret); + clkmain->regmap = regmap; + regmap_read(clkmain->regmap, AT91_CKGR_MOR, &status); + clkmain->parent = status & AT91_PMC_MOSCEN ? 1 : 0; clk = clk_register(NULL, &clkmain->hw); - if (IS_ERR(clk)) { - free_irq(clkmain->irq, clkmain); + if (IS_ERR(clk)) kfree(clkmain); - } return clk; } -void __init of_at91sam9x5_clk_main_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91sam9x5_clk_main_setup(struct device_node *np) { struct clk *clk; const char *parent_names[2]; int num_parents; - unsigned int irq; const char *name = np->name; + struct regmap *regmap; num_parents = of_clk_get_parent_count(np); if (num_parents <= 0 || num_parents > 2) return; of_clk_parent_fill(np, parent_names, num_parents); + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) + return; of_property_read_string(np, "clock-output-names", &name); - irq = irq_of_parse_and_map(np, 0); - if (!irq) - return; - - clk = at91_clk_register_sam9x5_main(pmc, irq, name, parent_names, + clk = at91_clk_register_sam9x5_main(regmap, name, parent_names, num_parents); if (IS_ERR(clk)) return; of_clk_add_provider(np, of_clk_src_simple_get, clk); } +CLK_OF_DECLARE(at91sam9x5_clk_main, "atmel,at91sam9x5-clk-main", + of_at91sam9x5_clk_main_setup); Index: linux-4.4.25-rt35/drivers/clk/at91/clk-master.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/clk-master.c +++ linux-4.4.25-rt35/drivers/clk/at91/clk-master.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:15 @ #include <linux/clkdev.h> #include <linux/clk/at91_pmc.h> #include <linux/of.h> -#include <linux/of_address.h> -#include <linux/of_irq.h> -#include <linux/io.h> -#include <linux/wait.h> -#include <linux/sched.h> -#include <linux/interrupt.h> -#include <linux/irq.h> +#include <linux/mfd/syscon.h> +#include <linux/regmap.h> #include "pmc.h" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:42 @ struct clk_master_layout { struct clk_master { struct clk_hw hw; - struct at91_pmc *pmc; - unsigned int irq; - wait_queue_head_t wait; + struct regmap *regmap; const struct clk_master_layout *layout; const struct clk_master_characteristics *characteristics; }; -static irqreturn_t clk_master_irq_handler(int irq, void *dev_id) +static inline bool clk_master_ready(struct regmap *regmap) { - struct clk_master *master = (struct clk_master *)dev_id; + unsigned int status; - wake_up(&master->wait); - disable_irq_nosync(master->irq); + regmap_read(regmap, AT91_PMC_SR, &status); - return IRQ_HANDLED; + return status & AT91_PMC_MCKRDY ? 1 : 0; } + static int clk_master_prepare(struct clk_hw *hw) { struct clk_master *master = to_clk_master(hw); - struct at91_pmc *pmc = master->pmc; - while (!(pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_MCKRDY)) { - enable_irq(master->irq); - wait_event(master->wait, - pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_MCKRDY); - } + while (!clk_master_ready(master->regmap)) + cpu_relax(); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:70 @ static int clk_master_is_prepared(struct { struct clk_master *master = to_clk_master(hw); - return !!(pmc_read(master->pmc, AT91_PMC_SR) & AT91_PMC_MCKRDY); + return clk_master_ready(master->regmap); } static unsigned long clk_master_recalc_rate(struct clk_hw *hw, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:80 @ static unsigned long clk_master_recalc_r u8 div; unsigned long rate = parent_rate; struct clk_master *master = to_clk_master(hw); - struct at91_pmc *pmc = master->pmc; const struct clk_master_layout *layout = master->layout; const struct clk_master_characteristics *characteristics = master->characteristics; - u32 tmp; + unsigned int mckr; - pmc_lock(pmc); - tmp = pmc_read(pmc, AT91_PMC_MCKR) & layout->mask; - pmc_unlock(pmc); + regmap_read(master->regmap, AT91_PMC_MCKR, &mckr); + mckr &= layout->mask; - pres = (tmp >> layout->pres_shift) & MASTER_PRES_MASK; - div = (tmp >> MASTER_DIV_SHIFT) & MASTER_DIV_MASK; + pres = (mckr >> layout->pres_shift) & MASTER_PRES_MASK; + div = (mckr >> MASTER_DIV_SHIFT) & MASTER_DIV_MASK; if (characteristics->have_div3_pres && pres == MASTER_PRES_MAX) rate /= 3; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:109 @ static unsigned long clk_master_recalc_r static u8 clk_master_get_parent(struct clk_hw *hw) { struct clk_master *master = to_clk_master(hw); - struct at91_pmc *pmc = master->pmc; + unsigned int mckr; - return pmc_read(pmc, AT91_PMC_MCKR) & AT91_PMC_CSS; + regmap_read(master->regmap, AT91_PMC_MCKR, &mckr); + + return mckr & AT91_PMC_CSS; } static const struct clk_ops master_ops = { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:124 @ static const struct clk_ops master_ops = }; static struct clk * __init -at91_clk_register_master(struct at91_pmc *pmc, unsigned int irq, +at91_clk_register_master(struct regmap *regmap, const char *name, int num_parents, const char **parent_names, const struct clk_master_layout *layout, const struct clk_master_characteristics *characteristics) { - int ret; struct clk_master *master; struct clk *clk = NULL; struct clk_init_data init; - if (!pmc || !irq || !name || !num_parents || !parent_names) + if (!name || !num_parents || !parent_names) return ERR_PTR(-EINVAL); master = kzalloc(sizeof(*master), GFP_KERNEL); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:150 @ at91_clk_register_master(struct at91_pmc master->hw.init = &init; master->layout = layout; master->characteristics = characteristics; - master->pmc = pmc; - master->irq = irq; - init_waitqueue_head(&master->wait); - irq_set_status_flags(master->irq, IRQ_NOAUTOEN); - ret = request_irq(master->irq, clk_master_irq_handler, - IRQF_TRIGGER_HIGH, "clk-master", master); - if (ret) { - kfree(master); - return ERR_PTR(ret); - } + master->regmap = regmap; clk = clk_register(NULL, &master->hw); if (IS_ERR(clk)) { - free_irq(master->irq, master); kfree(master); } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:198 @ out_free_characteristics: } static void __init -of_at91_clk_master_setup(struct device_node *np, struct at91_pmc *pmc, +of_at91_clk_master_setup(struct device_node *np, const struct clk_master_layout *layout) { struct clk *clk; int num_parents; - unsigned int irq; const char *parent_names[MASTER_SOURCE_MAX]; const char *name = np->name; struct clk_master_characteristics *characteristics; + struct regmap *regmap; num_parents = of_clk_get_parent_count(np); if (num_parents <= 0 || num_parents > MASTER_SOURCE_MAX) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:220 @ of_at91_clk_master_setup(struct device_n if (!characteristics) return; - irq = irq_of_parse_and_map(np, 0); - if (!irq) - goto out_free_characteristics; + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) + return; - clk = at91_clk_register_master(pmc, irq, name, num_parents, + clk = at91_clk_register_master(regmap, name, num_parents, parent_names, layout, characteristics); if (IS_ERR(clk)) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:237 @ out_free_characteristics: kfree(characteristics); } -void __init of_at91rm9200_clk_master_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91rm9200_clk_master_setup(struct device_node *np) { - of_at91_clk_master_setup(np, pmc, &at91rm9200_master_layout); + of_at91_clk_master_setup(np, &at91rm9200_master_layout); } +CLK_OF_DECLARE(at91rm9200_clk_master, "atmel,at91rm9200-clk-master", + of_at91rm9200_clk_master_setup); -void __init of_at91sam9x5_clk_master_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91sam9x5_clk_master_setup(struct device_node *np) { - of_at91_clk_master_setup(np, pmc, &at91sam9x5_master_layout); + of_at91_clk_master_setup(np, &at91sam9x5_master_layout); } +CLK_OF_DECLARE(at91sam9x5_clk_master, "atmel,at91sam9x5-clk-master", + of_at91sam9x5_clk_master_setup); Index: linux-4.4.25-rt35/drivers/clk/at91/clk-peripheral.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/clk-peripheral.c +++ linux-4.4.25-rt35/drivers/clk/at91/clk-peripheral.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:15 @ #include <linux/clkdev.h> #include <linux/clk/at91_pmc.h> #include <linux/of.h> -#include <linux/of_address.h> -#include <linux/io.h> +#include <linux/mfd/syscon.h> +#include <linux/regmap.h> #include "pmc.h" +DEFINE_SPINLOCK(pmc_pcr_lock); + #define PERIPHERAL_MAX 64 #define PERIPHERAL_AT91RM9200 0 @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:38 @ struct clk_peripheral { struct clk_hw hw; - struct at91_pmc *pmc; + struct regmap *regmap; u32 id; }; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:46 @ struct clk_peripheral { struct clk_sam9x5_peripheral { struct clk_hw hw; - struct at91_pmc *pmc; + struct regmap *regmap; struct clk_range range; + spinlock_t *lock; u32 id; u32 div; bool auto_div; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:60 @ struct clk_sam9x5_peripheral { static int clk_peripheral_enable(struct clk_hw *hw) { struct clk_peripheral *periph = to_clk_peripheral(hw); - struct at91_pmc *pmc = periph->pmc; int offset = AT91_PMC_PCER; u32 id = periph->id; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:67 @ static int clk_peripheral_enable(struct return 0; if (id > PERIPHERAL_ID_MAX) offset = AT91_PMC_PCER1; - pmc_write(pmc, offset, PERIPHERAL_MASK(id)); + regmap_write(periph->regmap, offset, PERIPHERAL_MASK(id)); + return 0; } static void clk_peripheral_disable(struct clk_hw *hw) { struct clk_peripheral *periph = to_clk_peripheral(hw); - struct at91_pmc *pmc = periph->pmc; int offset = AT91_PMC_PCDR; u32 id = periph->id; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:82 @ static void clk_peripheral_disable(struc return; if (id > PERIPHERAL_ID_MAX) offset = AT91_PMC_PCDR1; - pmc_write(pmc, offset, PERIPHERAL_MASK(id)); + regmap_write(periph->regmap, offset, PERIPHERAL_MASK(id)); } static int clk_peripheral_is_enabled(struct clk_hw *hw) { struct clk_peripheral *periph = to_clk_peripheral(hw); - struct at91_pmc *pmc = periph->pmc; int offset = AT91_PMC_PCSR; + unsigned int status; u32 id = periph->id; if (id < PERIPHERAL_ID_MIN) return 1; if (id > PERIPHERAL_ID_MAX) offset = AT91_PMC_PCSR1; - return !!(pmc_read(pmc, offset) & PERIPHERAL_MASK(id)); + regmap_read(periph->regmap, offset, &status); + + return status & PERIPHERAL_MASK(id) ? 1 : 0; } static const struct clk_ops peripheral_ops = { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:108 @ static const struct clk_ops peripheral_o }; static struct clk * __init -at91_clk_register_peripheral(struct at91_pmc *pmc, const char *name, +at91_clk_register_peripheral(struct regmap *regmap, const char *name, const char *parent_name, u32 id) { struct clk_peripheral *periph; struct clk *clk = NULL; struct clk_init_data init; - if (!pmc || !name || !parent_name || id > PERIPHERAL_ID_MAX) + if (!name || !parent_name || id > PERIPHERAL_ID_MAX) return ERR_PTR(-EINVAL); periph = kzalloc(sizeof(*periph), GFP_KERNEL); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:130 @ at91_clk_register_peripheral(struct at91 periph->id = id; periph->hw.init = &init; - periph->pmc = pmc; + periph->regmap = regmap; clk = clk_register(NULL, &periph->hw); if (IS_ERR(clk)) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:167 @ static void clk_sam9x5_peripheral_autodi static int clk_sam9x5_peripheral_enable(struct clk_hw *hw) { struct clk_sam9x5_peripheral *periph = to_clk_sam9x5_peripheral(hw); - struct at91_pmc *pmc = periph->pmc; - u32 tmp; + unsigned long flags; if (periph->id < PERIPHERAL_ID_MIN) return 0; - pmc_lock(pmc); - pmc_write(pmc, AT91_PMC_PCR, (periph->id & AT91_PMC_PCR_PID_MASK)); - tmp = pmc_read(pmc, AT91_PMC_PCR) & ~AT91_PMC_PCR_DIV_MASK; - pmc_write(pmc, AT91_PMC_PCR, tmp | AT91_PMC_PCR_DIV(periph->div) - | AT91_PMC_PCR_CMD - | AT91_PMC_PCR_EN); - pmc_unlock(pmc); + spin_lock_irqsave(periph->lock, flags); + regmap_write(periph->regmap, AT91_PMC_PCR, + (periph->id & AT91_PMC_PCR_PID_MASK)); + regmap_update_bits(periph->regmap, AT91_PMC_PCR, + AT91_PMC_PCR_DIV_MASK | AT91_PMC_PCR_CMD | + AT91_PMC_PCR_EN, + AT91_PMC_PCR_DIV(periph->div) | + AT91_PMC_PCR_CMD | + AT91_PMC_PCR_EN); + spin_unlock_irqrestore(periph->lock, flags); + return 0; } static void clk_sam9x5_peripheral_disable(struct clk_hw *hw) { struct clk_sam9x5_peripheral *periph = to_clk_sam9x5_peripheral(hw); - struct at91_pmc *pmc = periph->pmc; - u32 tmp; + unsigned long flags; if (periph->id < PERIPHERAL_ID_MIN) return; - pmc_lock(pmc); - pmc_write(pmc, AT91_PMC_PCR, (periph->id & AT91_PMC_PCR_PID_MASK)); - tmp = pmc_read(pmc, AT91_PMC_PCR) & ~AT91_PMC_PCR_EN; - pmc_write(pmc, AT91_PMC_PCR, tmp | AT91_PMC_PCR_CMD); - pmc_unlock(pmc); + spin_lock_irqsave(periph->lock, flags); + regmap_write(periph->regmap, AT91_PMC_PCR, + (periph->id & AT91_PMC_PCR_PID_MASK)); + regmap_update_bits(periph->regmap, AT91_PMC_PCR, + AT91_PMC_PCR_EN | AT91_PMC_PCR_CMD, + AT91_PMC_PCR_CMD); + spin_unlock_irqrestore(periph->lock, flags); } static int clk_sam9x5_peripheral_is_enabled(struct clk_hw *hw) { struct clk_sam9x5_peripheral *periph = to_clk_sam9x5_peripheral(hw); - struct at91_pmc *pmc = periph->pmc; - int ret; + unsigned long flags; + unsigned int status; if (periph->id < PERIPHERAL_ID_MIN) return 1; - pmc_lock(pmc); - pmc_write(pmc, AT91_PMC_PCR, (periph->id & AT91_PMC_PCR_PID_MASK)); - ret = !!(pmc_read(pmc, AT91_PMC_PCR) & AT91_PMC_PCR_EN); - pmc_unlock(pmc); + spin_lock_irqsave(periph->lock, flags); + regmap_write(periph->regmap, AT91_PMC_PCR, + (periph->id & AT91_PMC_PCR_PID_MASK)); + regmap_read(periph->regmap, AT91_PMC_PCR, &status); + spin_unlock_irqrestore(periph->lock, flags); - return ret; + return status & AT91_PMC_PCR_EN ? 1 : 0; } static unsigned long @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:226 @ clk_sam9x5_peripheral_recalc_rate(struct unsigned long parent_rate) { struct clk_sam9x5_peripheral *periph = to_clk_sam9x5_peripheral(hw); - struct at91_pmc *pmc = periph->pmc; - u32 tmp; + unsigned long flags; + unsigned int status; if (periph->id < PERIPHERAL_ID_MIN) return parent_rate; - pmc_lock(pmc); - pmc_write(pmc, AT91_PMC_PCR, (periph->id & AT91_PMC_PCR_PID_MASK)); - tmp = pmc_read(pmc, AT91_PMC_PCR); - pmc_unlock(pmc); + spin_lock_irqsave(periph->lock, flags); + regmap_write(periph->regmap, AT91_PMC_PCR, + (periph->id & AT91_PMC_PCR_PID_MASK)); + regmap_read(periph->regmap, AT91_PMC_PCR, &status); + spin_unlock_irqrestore(periph->lock, flags); - if (tmp & AT91_PMC_PCR_EN) { - periph->div = PERIPHERAL_RSHIFT(tmp); + if (status & AT91_PMC_PCR_EN) { + periph->div = PERIPHERAL_RSHIFT(status); periph->auto_div = false; } else { clk_sam9x5_peripheral_autodiv(periph); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:331 @ static const struct clk_ops sam9x5_perip }; static struct clk * __init -at91_clk_register_sam9x5_peripheral(struct at91_pmc *pmc, const char *name, - const char *parent_name, u32 id, - const struct clk_range *range) +at91_clk_register_sam9x5_peripheral(struct regmap *regmap, spinlock_t *lock, + const char *name, const char *parent_name, + u32 id, const struct clk_range *range) { struct clk_sam9x5_peripheral *periph; struct clk *clk = NULL; struct clk_init_data init; - if (!pmc || !name || !parent_name) + if (!name || !parent_name) return ERR_PTR(-EINVAL); periph = kzalloc(sizeof(*periph), GFP_KERNEL); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:355 @ at91_clk_register_sam9x5_peripheral(stru periph->id = id; periph->hw.init = &init; periph->div = 0; - periph->pmc = pmc; + periph->regmap = regmap; + periph->lock = lock; periph->auto_div = true; periph->range = *range; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:370 @ at91_clk_register_sam9x5_peripheral(stru } static void __init -of_at91_clk_periph_setup(struct device_node *np, struct at91_pmc *pmc, u8 type) +of_at91_clk_periph_setup(struct device_node *np, u8 type) { int num; u32 id; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:378 @ of_at91_clk_periph_setup(struct device_n const char *parent_name; const char *name; struct device_node *periphclknp; + struct regmap *regmap; parent_name = of_clk_get_parent_name(np, 0); if (!parent_name) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:388 @ of_at91_clk_periph_setup(struct device_n if (!num || num > PERIPHERAL_MAX) return; + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) + return; + for_each_child_of_node(np, periphclknp) { if (of_property_read_u32(periphclknp, "reg", &id)) continue; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:403 @ of_at91_clk_periph_setup(struct device_n name = periphclknp->name; if (type == PERIPHERAL_AT91RM9200) { - clk = at91_clk_register_peripheral(pmc, name, + clk = at91_clk_register_peripheral(regmap, name, parent_name, id); } else { struct clk_range range = CLK_RANGE(0, 0); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:412 @ of_at91_clk_periph_setup(struct device_n "atmel,clk-output-range", &range); - clk = at91_clk_register_sam9x5_peripheral(pmc, name, + clk = at91_clk_register_sam9x5_peripheral(regmap, + &pmc_pcr_lock, + name, parent_name, id, &range); } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:426 @ of_at91_clk_periph_setup(struct device_n } } -void __init of_at91rm9200_clk_periph_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91rm9200_clk_periph_setup(struct device_node *np) { - of_at91_clk_periph_setup(np, pmc, PERIPHERAL_AT91RM9200); + of_at91_clk_periph_setup(np, PERIPHERAL_AT91RM9200); } +CLK_OF_DECLARE(at91rm9200_clk_periph, "atmel,at91rm9200-clk-peripheral", + of_at91rm9200_clk_periph_setup); -void __init of_at91sam9x5_clk_periph_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91sam9x5_clk_periph_setup(struct device_node *np) { - of_at91_clk_periph_setup(np, pmc, PERIPHERAL_AT91SAM9X5); + of_at91_clk_periph_setup(np, PERIPHERAL_AT91SAM9X5); } +CLK_OF_DECLARE(at91sam9x5_clk_periph, "atmel,at91sam9x5-clk-peripheral", + of_at91sam9x5_clk_periph_setup); Index: linux-4.4.25-rt35/drivers/clk/at91/clk-pll.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/clk-pll.c +++ linux-4.4.25-rt35/drivers/clk/at91/clk-pll.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:15 @ #include <linux/clkdev.h> #include <linux/clk/at91_pmc.h> #include <linux/of.h> -#include <linux/of_address.h> -#include <linux/of_irq.h> -#include <linux/io.h> -#include <linux/kernel.h> -#include <linux/wait.h> -#include <linux/sched.h> -#include <linux/interrupt.h> -#include <linux/irq.h> +#include <linux/mfd/syscon.h> +#include <linux/regmap.h> #include "pmc.h" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:55 @ struct clk_pll_layout { struct clk_pll { struct clk_hw hw; - struct at91_pmc *pmc; - unsigned int irq; - wait_queue_head_t wait; + struct regmap *regmap; u8 id; u8 div; u8 range; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:64 @ struct clk_pll { const struct clk_pll_characteristics *characteristics; }; -static irqreturn_t clk_pll_irq_handler(int irq, void *dev_id) +static inline bool clk_pll_ready(struct regmap *regmap, int id) { - struct clk_pll *pll = (struct clk_pll *)dev_id; + unsigned int status; - wake_up(&pll->wait); - disable_irq_nosync(pll->irq); + regmap_read(regmap, AT91_PMC_SR, &status); - return IRQ_HANDLED; + return status & PLL_STATUS_MASK(id) ? 1 : 0; } static int clk_pll_prepare(struct clk_hw *hw) { struct clk_pll *pll = to_clk_pll(hw); - struct at91_pmc *pmc = pll->pmc; + struct regmap *regmap = pll->regmap; const struct clk_pll_layout *layout = pll->layout; const struct clk_pll_characteristics *characteristics = pll->characteristics; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:84 @ static int clk_pll_prepare(struct clk_hw u32 mask = PLL_STATUS_MASK(id); int offset = PLL_REG(id); u8 out = 0; - u32 pllr, icpr; + unsigned int pllr; + unsigned int status; u8 div; u16 mul; - pllr = pmc_read(pmc, offset); + regmap_read(regmap, offset, &pllr); div = PLL_DIV(pllr); mul = PLL_MUL(pllr, layout); - if ((pmc_read(pmc, AT91_PMC_SR) & mask) && + regmap_read(regmap, AT91_PMC_SR, &status); + if ((status & mask) && (div == pll->div && mul == pll->mul)) return 0; if (characteristics->out) out = characteristics->out[pll->range]; - if (characteristics->icpll) { - icpr = pmc_read(pmc, AT91_PMC_PLLICPR) & ~PLL_ICPR_MASK(id); - icpr |= (characteristics->icpll[pll->range] << - PLL_ICPR_SHIFT(id)); - pmc_write(pmc, AT91_PMC_PLLICPR, icpr); - } - pllr &= ~layout->pllr_mask; - pllr |= layout->pllr_mask & - (pll->div | (PLL_MAX_COUNT << PLL_COUNT_SHIFT) | - (out << PLL_OUT_SHIFT) | - ((pll->mul & layout->mul_mask) << layout->mul_shift)); - pmc_write(pmc, offset, pllr); - - while (!(pmc_read(pmc, AT91_PMC_SR) & mask)) { - enable_irq(pll->irq); - wait_event(pll->wait, - pmc_read(pmc, AT91_PMC_SR) & mask); - } + if (characteristics->icpll) + regmap_update_bits(regmap, AT91_PMC_PLLICPR, PLL_ICPR_MASK(id), + characteristics->icpll[pll->range] << PLL_ICPR_SHIFT(id)); + + regmap_update_bits(regmap, offset, layout->pllr_mask, + pll->div | (PLL_MAX_COUNT << PLL_COUNT_SHIFT) | + (out << PLL_OUT_SHIFT) | + ((pll->mul & layout->mul_mask) << layout->mul_shift)); + + while (!clk_pll_ready(regmap, pll->id)) + cpu_relax(); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:119 @ static int clk_pll_prepare(struct clk_hw static int clk_pll_is_prepared(struct clk_hw *hw) { struct clk_pll *pll = to_clk_pll(hw); - struct at91_pmc *pmc = pll->pmc; - return !!(pmc_read(pmc, AT91_PMC_SR) & - PLL_STATUS_MASK(pll->id)); + return clk_pll_ready(pll->regmap, pll->id); } static void clk_pll_unprepare(struct clk_hw *hw) { struct clk_pll *pll = to_clk_pll(hw); - struct at91_pmc *pmc = pll->pmc; - const struct clk_pll_layout *layout = pll->layout; - int offset = PLL_REG(pll->id); - u32 tmp = pmc_read(pmc, offset) & ~(layout->pllr_mask); + unsigned int mask = pll->layout->pllr_mask; - pmc_write(pmc, offset, tmp); + regmap_update_bits(pll->regmap, PLL_REG(pll->id), mask, ~mask); } static unsigned long clk_pll_recalc_rate(struct clk_hw *hw, unsigned long parent_rate) { struct clk_pll *pll = to_clk_pll(hw); + unsigned int pllr; + u16 mul; + u8 div; + + regmap_read(pll->regmap, PLL_REG(pll->id), &pllr); + + div = PLL_DIV(pllr); + mul = PLL_MUL(pllr, pll->layout); - if (!pll->div || !pll->mul) + if (!div || !mul) return 0; - return (parent_rate / pll->div) * (pll->mul + 1); + return (parent_rate / div) * (mul + 1); } static long clk_pll_get_best_div_mul(struct clk_pll *pll, unsigned long rate, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:300 @ static const struct clk_ops pll_ops = { }; static struct clk * __init -at91_clk_register_pll(struct at91_pmc *pmc, unsigned int irq, const char *name, +at91_clk_register_pll(struct regmap *regmap, const char *name, const char *parent_name, u8 id, const struct clk_pll_layout *layout, const struct clk_pll_characteristics *characteristics) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:308 @ at91_clk_register_pll(struct at91_pmc *p struct clk_pll *pll; struct clk *clk = NULL; struct clk_init_data init; - int ret; int offset = PLL_REG(id); - u32 tmp; + unsigned int pllr; if (id > PLL_MAX_ID) return ERR_PTR(-EINVAL); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:328 @ at91_clk_register_pll(struct at91_pmc *p pll->hw.init = &init; pll->layout = layout; pll->characteristics = characteristics; - pll->pmc = pmc; - pll->irq = irq; - tmp = pmc_read(pmc, offset) & layout->pllr_mask; - pll->div = PLL_DIV(tmp); - pll->mul = PLL_MUL(tmp, layout); - init_waitqueue_head(&pll->wait); - irq_set_status_flags(pll->irq, IRQ_NOAUTOEN); - ret = request_irq(pll->irq, clk_pll_irq_handler, IRQF_TRIGGER_HIGH, - id ? "clk-pllb" : "clk-plla", pll); - if (ret) { - kfree(pll); - return ERR_PTR(ret); - } + pll->regmap = regmap; + regmap_read(regmap, offset, &pllr); + pll->div = PLL_DIV(pllr); + pll->mul = PLL_MUL(pllr, layout); clk = clk_register(NULL, &pll->hw); if (IS_ERR(clk)) { - free_irq(pll->irq, pll); kfree(pll); } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:464 @ out_free_characteristics: } static void __init -of_at91_clk_pll_setup(struct device_node *np, struct at91_pmc *pmc, +of_at91_clk_pll_setup(struct device_node *np, const struct clk_pll_layout *layout) { u32 id; - unsigned int irq; struct clk *clk; + struct regmap *regmap; const char *parent_name; const char *name = np->name; struct clk_pll_characteristics *characteristics; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:481 @ of_at91_clk_pll_setup(struct device_node of_property_read_string(np, "clock-output-names", &name); - characteristics = of_at91_clk_pll_get_characteristics(np); - if (!characteristics) + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) return; - irq = irq_of_parse_and_map(np, 0); - if (!irq) + characteristics = of_at91_clk_pll_get_characteristics(np); + if (!characteristics) return; - clk = at91_clk_register_pll(pmc, irq, name, parent_name, id, layout, + clk = at91_clk_register_pll(regmap, name, parent_name, id, layout, characteristics); if (IS_ERR(clk)) goto out_free_characteristics; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:501 @ out_free_characteristics: kfree(characteristics); } -void __init of_at91rm9200_clk_pll_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91rm9200_clk_pll_setup(struct device_node *np) { - of_at91_clk_pll_setup(np, pmc, &at91rm9200_pll_layout); + of_at91_clk_pll_setup(np, &at91rm9200_pll_layout); } +CLK_OF_DECLARE(at91rm9200_clk_pll, "atmel,at91rm9200-clk-pll", + of_at91rm9200_clk_pll_setup); -void __init of_at91sam9g45_clk_pll_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91sam9g45_clk_pll_setup(struct device_node *np) { - of_at91_clk_pll_setup(np, pmc, &at91sam9g45_pll_layout); + of_at91_clk_pll_setup(np, &at91sam9g45_pll_layout); } +CLK_OF_DECLARE(at91sam9g45_clk_pll, "atmel,at91sam9g45-clk-pll", + of_at91sam9g45_clk_pll_setup); -void __init of_at91sam9g20_clk_pllb_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91sam9g20_clk_pllb_setup(struct device_node *np) { - of_at91_clk_pll_setup(np, pmc, &at91sam9g20_pllb_layout); + of_at91_clk_pll_setup(np, &at91sam9g20_pllb_layout); } +CLK_OF_DECLARE(at91sam9g20_clk_pllb, "atmel,at91sam9g20-clk-pllb", + of_at91sam9g20_clk_pllb_setup); -void __init of_sama5d3_clk_pll_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_sama5d3_clk_pll_setup(struct device_node *np) { - of_at91_clk_pll_setup(np, pmc, &sama5d3_pll_layout); + of_at91_clk_pll_setup(np, &sama5d3_pll_layout); } +CLK_OF_DECLARE(sama5d3_clk_pll, "atmel,sama5d3-clk-pll", + of_sama5d3_clk_pll_setup); Index: linux-4.4.25-rt35/drivers/clk/at91/clk-plldiv.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/clk-plldiv.c +++ linux-4.4.25-rt35/drivers/clk/at91/clk-plldiv.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:15 @ #include <linux/clkdev.h> #include <linux/clk/at91_pmc.h> #include <linux/of.h> -#include <linux/of_address.h> -#include <linux/io.h> +#include <linux/mfd/syscon.h> +#include <linux/regmap.h> #include "pmc.h" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:24 @ struct clk_plldiv { struct clk_hw hw; - struct at91_pmc *pmc; + struct regmap *regmap; }; static unsigned long clk_plldiv_recalc_rate(struct clk_hw *hw, unsigned long parent_rate) { struct clk_plldiv *plldiv = to_clk_plldiv(hw); - struct at91_pmc *pmc = plldiv->pmc; + unsigned int mckr; - if (pmc_read(pmc, AT91_PMC_MCKR) & AT91_PMC_PLLADIV2) + regmap_read(plldiv->regmap, AT91_PMC_MCKR, &mckr); + + if (mckr & AT91_PMC_PLLADIV2) return parent_rate / 2; return parent_rate; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:62 @ static int clk_plldiv_set_rate(struct cl unsigned long parent_rate) { struct clk_plldiv *plldiv = to_clk_plldiv(hw); - struct at91_pmc *pmc = plldiv->pmc; - u32 tmp; - if (parent_rate != rate && (parent_rate / 2) != rate) + if ((parent_rate != rate) && (parent_rate / 2 != rate)) return -EINVAL; - pmc_lock(pmc); - tmp = pmc_read(pmc, AT91_PMC_MCKR) & ~AT91_PMC_PLLADIV2; - if ((parent_rate / 2) == rate) - tmp |= AT91_PMC_PLLADIV2; - pmc_write(pmc, AT91_PMC_MCKR, tmp); - pmc_unlock(pmc); + regmap_update_bits(plldiv->regmap, AT91_PMC_MCKR, AT91_PMC_PLLADIV2, + parent_rate != rate ? AT91_PMC_PLLADIV2 : 0); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:79 @ static const struct clk_ops plldiv_ops = }; static struct clk * __init -at91_clk_register_plldiv(struct at91_pmc *pmc, const char *name, +at91_clk_register_plldiv(struct regmap *regmap, const char *name, const char *parent_name) { struct clk_plldiv *plldiv; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:97 @ at91_clk_register_plldiv(struct at91_pmc init.flags = CLK_SET_RATE_GATE; plldiv->hw.init = &init; - plldiv->pmc = pmc; + plldiv->regmap = regmap; clk = clk_register(NULL, &plldiv->hw); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:108 @ at91_clk_register_plldiv(struct at91_pmc } static void __init -of_at91_clk_plldiv_setup(struct device_node *np, struct at91_pmc *pmc) +of_at91sam9x5_clk_plldiv_setup(struct device_node *np) { struct clk *clk; const char *parent_name; const char *name = np->name; + struct regmap *regmap; parent_name = of_clk_get_parent_name(np, 0); of_property_read_string(np, "clock-output-names", &name); - clk = at91_clk_register_plldiv(pmc, name, parent_name); + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) + return; + clk = at91_clk_register_plldiv(regmap, name, parent_name); if (IS_ERR(clk)) return; of_clk_add_provider(np, of_clk_src_simple_get, clk); return; } - -void __init of_at91sam9x5_clk_plldiv_setup(struct device_node *np, - struct at91_pmc *pmc) -{ - of_at91_clk_plldiv_setup(np, pmc); -} +CLK_OF_DECLARE(at91sam9x5_clk_plldiv, "atmel,at91sam9x5-clk-plldiv", + of_at91sam9x5_clk_plldiv_setup); Index: linux-4.4.25-rt35/drivers/clk/at91/clk-programmable.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/clk-programmable.c +++ linux-4.4.25-rt35/drivers/clk/at91/clk-programmable.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:15 @ #include <linux/clkdev.h> #include <linux/clk/at91_pmc.h> #include <linux/of.h> -#include <linux/of_address.h> -#include <linux/io.h> -#include <linux/wait.h> -#include <linux/sched.h> +#include <linux/mfd/syscon.h> +#include <linux/regmap.h> #include "pmc.h" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:25 @ #define PROG_STATUS_MASK(id) (1 << ((id) + 8)) #define PROG_PRES_MASK 0x7 +#define PROG_PRES(layout, pckr) ((pckr >> layout->pres_shift) & PROG_PRES_MASK) #define PROG_MAX_RM9200_CSS 3 struct clk_programmable_layout { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:36 @ struct clk_programmable_layout { struct clk_programmable { struct clk_hw hw; - struct at91_pmc *pmc; + struct regmap *regmap; u8 id; const struct clk_programmable_layout *layout; }; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:46 @ struct clk_programmable { static unsigned long clk_programmable_recalc_rate(struct clk_hw *hw, unsigned long parent_rate) { - u32 pres; struct clk_programmable *prog = to_clk_programmable(hw); - struct at91_pmc *pmc = prog->pmc; - const struct clk_programmable_layout *layout = prog->layout; + unsigned int pckr; + + regmap_read(prog->regmap, AT91_PMC_PCKR(prog->id), &pckr); - pres = (pmc_read(pmc, AT91_PMC_PCKR(prog->id)) >> layout->pres_shift) & - PROG_PRES_MASK; - return parent_rate >> pres; + return parent_rate >> PROG_PRES(prog->layout, pckr); } static int clk_programmable_determine_rate(struct clk_hw *hw, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:101 @ static int clk_programmable_set_parent(s { struct clk_programmable *prog = to_clk_programmable(hw); const struct clk_programmable_layout *layout = prog->layout; - struct at91_pmc *pmc = prog->pmc; - u32 tmp = pmc_read(pmc, AT91_PMC_PCKR(prog->id)) & ~layout->css_mask; + unsigned int mask = layout->css_mask; + unsigned int pckr = 0; if (layout->have_slck_mck) - tmp &= AT91_PMC_CSSMCK_MCK; + mask |= AT91_PMC_CSSMCK_MCK; if (index > layout->css_mask) { - if (index > PROG_MAX_RM9200_CSS && layout->have_slck_mck) { - tmp |= AT91_PMC_CSSMCK_MCK; - return 0; - } else { + if (index > PROG_MAX_RM9200_CSS && !layout->have_slck_mck) return -EINVAL; - } + + pckr |= AT91_PMC_CSSMCK_MCK; } - pmc_write(pmc, AT91_PMC_PCKR(prog->id), tmp | index); + regmap_update_bits(prog->regmap, AT91_PMC_PCKR(prog->id), mask, pckr); + return 0; } static u8 clk_programmable_get_parent(struct clk_hw *hw) { - u32 tmp; - u8 ret; struct clk_programmable *prog = to_clk_programmable(hw); - struct at91_pmc *pmc = prog->pmc; const struct clk_programmable_layout *layout = prog->layout; + unsigned int pckr; + u8 ret; + + regmap_read(prog->regmap, AT91_PMC_PCKR(prog->id), &pckr); + + ret = pckr & layout->css_mask; - tmp = pmc_read(pmc, AT91_PMC_PCKR(prog->id)); - ret = tmp & layout->css_mask; - if (layout->have_slck_mck && (tmp & AT91_PMC_CSSMCK_MCK) && !ret) + if (layout->have_slck_mck && (pckr & AT91_PMC_CSSMCK_MCK) && !ret) ret = PROG_MAX_RM9200_CSS + 1; return ret; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:140 @ static int clk_programmable_set_rate(str unsigned long parent_rate) { struct clk_programmable *prog = to_clk_programmable(hw); - struct at91_pmc *pmc = prog->pmc; const struct clk_programmable_layout *layout = prog->layout; unsigned long div = parent_rate / rate; + unsigned int pckr; int shift = 0; - u32 tmp = pmc_read(pmc, AT91_PMC_PCKR(prog->id)) & - ~(PROG_PRES_MASK << layout->pres_shift); + + regmap_read(prog->regmap, AT91_PMC_PCKR(prog->id), &pckr); if (!div) return -EINVAL; shift = fls(div) - 1; - if (div != (1<<shift)) + if (div != (1 << shift)) return -EINVAL; if (shift >= PROG_PRES_MASK) return -EINVAL; - pmc_write(pmc, AT91_PMC_PCKR(prog->id), - tmp | (shift << layout->pres_shift)); + regmap_update_bits(prog->regmap, AT91_PMC_PCKR(prog->id), + PROG_PRES_MASK << layout->pres_shift, + shift << layout->pres_shift); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:174 @ static const struct clk_ops programmable }; static struct clk * __init -at91_clk_register_programmable(struct at91_pmc *pmc, +at91_clk_register_programmable(struct regmap *regmap, const char *name, const char **parent_names, u8 num_parents, u8 id, const struct clk_programmable_layout *layout) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:199 @ at91_clk_register_programmable(struct at prog->id = id; prog->layout = layout; prog->hw.init = &init; - prog->pmc = pmc; + prog->regmap = regmap; clk = clk_register(NULL, &prog->hw); if (IS_ERR(clk)) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:227 @ static const struct clk_programmable_lay }; static void __init -of_at91_clk_prog_setup(struct device_node *np, struct at91_pmc *pmc, +of_at91_clk_prog_setup(struct device_node *np, const struct clk_programmable_layout *layout) { int num; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:237 @ of_at91_clk_prog_setup(struct device_nod const char *parent_names[PROG_SOURCE_MAX]; const char *name; struct device_node *progclknp; + struct regmap *regmap; num_parents = of_clk_get_parent_count(np); if (num_parents <= 0 || num_parents > PROG_SOURCE_MAX) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:249 @ of_at91_clk_prog_setup(struct device_nod if (!num || num > (PROG_ID_MAX + 1)) return; + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) + return; + for_each_child_of_node(np, progclknp) { if (of_property_read_u32(progclknp, "reg", &id)) continue; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:260 @ of_at91_clk_prog_setup(struct device_nod if (of_property_read_string(np, "clock-output-names", &name)) name = progclknp->name; - clk = at91_clk_register_programmable(pmc, name, + clk = at91_clk_register_programmable(regmap, name, parent_names, num_parents, id, layout); if (IS_ERR(clk)) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:271 @ of_at91_clk_prog_setup(struct device_nod } -void __init of_at91rm9200_clk_prog_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91rm9200_clk_prog_setup(struct device_node *np) { - of_at91_clk_prog_setup(np, pmc, &at91rm9200_programmable_layout); + of_at91_clk_prog_setup(np, &at91rm9200_programmable_layout); } +CLK_OF_DECLARE(at91rm9200_clk_prog, "atmel,at91rm9200-clk-programmable", + of_at91rm9200_clk_prog_setup); -void __init of_at91sam9g45_clk_prog_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91sam9g45_clk_prog_setup(struct device_node *np) { - of_at91_clk_prog_setup(np, pmc, &at91sam9g45_programmable_layout); + of_at91_clk_prog_setup(np, &at91sam9g45_programmable_layout); } +CLK_OF_DECLARE(at91sam9g45_clk_prog, "atmel,at91sam9g45-clk-programmable", + of_at91sam9g45_clk_prog_setup); -void __init of_at91sam9x5_clk_prog_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91sam9x5_clk_prog_setup(struct device_node *np) { - of_at91_clk_prog_setup(np, pmc, &at91sam9x5_programmable_layout); + of_at91_clk_prog_setup(np, &at91sam9x5_programmable_layout); } +CLK_OF_DECLARE(at91sam9x5_clk_prog, "atmel,at91sam9x5-clk-programmable", + of_at91sam9x5_clk_prog_setup); Index: linux-4.4.25-rt35/drivers/clk/at91/clk-slow.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/clk-slow.c +++ linux-4.4.25-rt35/drivers/clk/at91/clk-slow.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:16 @ #include <linux/clk.h> #include <linux/clk-provider.h> #include <linux/clkdev.h> -#include <linux/slab.h> #include <linux/clk/at91_pmc.h> #include <linux/delay.h> #include <linux/of.h> -#include <linux/of_address.h> -#include <linux/of_irq.h> -#include <linux/io.h> -#include <linux/interrupt.h> -#include <linux/irq.h> -#include <linux/sched.h> -#include <linux/wait.h> +#include <linux/mfd/syscon.h> +#include <linux/regmap.h> #include "pmc.h" #include "sckc.h" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:56 @ struct clk_slow_rc_osc { struct clk_sam9260_slow { struct clk_hw hw; - struct at91_pmc *pmc; + struct regmap *regmap; }; #define to_clk_sam9260_slow(hw) container_of(hw, struct clk_sam9260_slow, hw) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:390 @ void __init of_at91sam9x5_clk_slow_setup static u8 clk_sam9260_slow_get_parent(struct clk_hw *hw) { struct clk_sam9260_slow *slowck = to_clk_sam9260_slow(hw); + unsigned int status; - return !!(pmc_read(slowck->pmc, AT91_PMC_SR) & AT91_PMC_OSCSEL); + regmap_read(slowck->regmap, AT91_PMC_SR, &status); + + return status & AT91_PMC_OSCSEL ? 1 : 0; } static const struct clk_ops sam9260_slow_ops = { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:402 @ static const struct clk_ops sam9260_slow }; static struct clk * __init -at91_clk_register_sam9260_slow(struct at91_pmc *pmc, +at91_clk_register_sam9260_slow(struct regmap *regmap, const char *name, const char **parent_names, int num_parents) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:411 @ at91_clk_register_sam9260_slow(struct at struct clk *clk = NULL; struct clk_init_data init; - if (!pmc || !name) + if (!name) return ERR_PTR(-EINVAL); if (!parent_names || !num_parents) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:428 @ at91_clk_register_sam9260_slow(struct at init.flags = 0; slowck->hw.init = &init; - slowck->pmc = pmc; + slowck->regmap = regmap; clk = clk_register(NULL, &slowck->hw); if (IS_ERR(clk)) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:439 @ at91_clk_register_sam9260_slow(struct at return clk; } -void __init of_at91sam9260_clk_slow_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91sam9260_clk_slow_setup(struct device_node *np) { struct clk *clk; const char *parent_names[2]; int num_parents; const char *name = np->name; + struct regmap *regmap; num_parents = of_clk_get_parent_count(np); if (num_parents != 2) return; of_clk_parent_fill(np, parent_names, num_parents); + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) + return; of_property_read_string(np, "clock-output-names", &name); - clk = at91_clk_register_sam9260_slow(pmc, name, parent_names, + clk = at91_clk_register_sam9260_slow(regmap, name, parent_names, num_parents); if (IS_ERR(clk)) return; of_clk_add_provider(np, of_clk_src_simple_get, clk); } +CLK_OF_DECLARE(at91sam9260_clk_slow, "atmel,at91sam9260-clk-slow", + of_at91sam9260_clk_slow_setup); /* * FIXME: All slow clk users are not properly claiming it (get + prepare + Index: linux-4.4.25-rt35/drivers/clk/at91/clk-smd.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/clk-smd.c +++ linux-4.4.25-rt35/drivers/clk/at91/clk-smd.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:15 @ #include <linux/clkdev.h> #include <linux/clk/at91_pmc.h> #include <linux/of.h> -#include <linux/of_address.h> -#include <linux/io.h> +#include <linux/mfd/syscon.h> +#include <linux/regmap.h> #include "pmc.h" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:27 @ struct at91sam9x5_clk_smd { struct clk_hw hw; - struct at91_pmc *pmc; + struct regmap *regmap; }; #define to_at91sam9x5_clk_smd(hw) \ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:36 @ struct at91sam9x5_clk_smd { static unsigned long at91sam9x5_clk_smd_recalc_rate(struct clk_hw *hw, unsigned long parent_rate) { - u32 tmp; - u8 smddiv; struct at91sam9x5_clk_smd *smd = to_at91sam9x5_clk_smd(hw); - struct at91_pmc *pmc = smd->pmc; + unsigned int smdr; + u8 smddiv; + + regmap_read(smd->regmap, AT91_PMC_SMD, &smdr); + smddiv = (smdr & AT91_PMC_SMD_DIV) >> SMD_DIV_SHIFT; - tmp = pmc_read(pmc, AT91_PMC_SMD); - smddiv = (tmp & AT91_PMC_SMD_DIV) >> SMD_DIV_SHIFT; return parent_rate / (smddiv + 1); } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:70 @ static long at91sam9x5_clk_smd_round_rat static int at91sam9x5_clk_smd_set_parent(struct clk_hw *hw, u8 index) { - u32 tmp; struct at91sam9x5_clk_smd *smd = to_at91sam9x5_clk_smd(hw); - struct at91_pmc *pmc = smd->pmc; if (index > 1) return -EINVAL; - tmp = pmc_read(pmc, AT91_PMC_SMD) & ~AT91_PMC_SMDS; - if (index) - tmp |= AT91_PMC_SMDS; - pmc_write(pmc, AT91_PMC_SMD, tmp); + + regmap_update_bits(smd->regmap, AT91_PMC_SMD, AT91_PMC_SMDS, + index ? AT91_PMC_SMDS : 0); + return 0; } static u8 at91sam9x5_clk_smd_get_parent(struct clk_hw *hw) { struct at91sam9x5_clk_smd *smd = to_at91sam9x5_clk_smd(hw); - struct at91_pmc *pmc = smd->pmc; + unsigned int smdr; - return pmc_read(pmc, AT91_PMC_SMD) & AT91_PMC_SMDS; + regmap_read(smd->regmap, AT91_PMC_SMD, &smdr); + + return smdr & AT91_PMC_SMDS; } static int at91sam9x5_clk_smd_set_rate(struct clk_hw *hw, unsigned long rate, unsigned long parent_rate) { - u32 tmp; struct at91sam9x5_clk_smd *smd = to_at91sam9x5_clk_smd(hw); - struct at91_pmc *pmc = smd->pmc; unsigned long div = parent_rate / rate; if (parent_rate % rate || div < 1 || div > (SMD_MAX_DIV + 1)) return -EINVAL; - tmp = pmc_read(pmc, AT91_PMC_SMD) & ~AT91_PMC_SMD_DIV; - tmp |= (div - 1) << SMD_DIV_SHIFT; - pmc_write(pmc, AT91_PMC_SMD, tmp); + + regmap_update_bits(smd->regmap, AT91_PMC_SMD, AT91_PMC_SMD_DIV, + (div - 1) << SMD_DIV_SHIFT); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:115 @ static const struct clk_ops at91sam9x5_s }; static struct clk * __init -at91sam9x5_clk_register_smd(struct at91_pmc *pmc, const char *name, +at91sam9x5_clk_register_smd(struct regmap *regmap, const char *name, const char **parent_names, u8 num_parents) { struct at91sam9x5_clk_smd *smd; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:133 @ at91sam9x5_clk_register_smd(struct at91_ init.flags = CLK_SET_RATE_GATE | CLK_SET_PARENT_GATE; smd->hw.init = &init; - smd->pmc = pmc; + smd->regmap = regmap; clk = clk_register(NULL, &smd->hw); if (IS_ERR(clk)) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:142 @ at91sam9x5_clk_register_smd(struct at91_ return clk; } -void __init of_at91sam9x5_clk_smd_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91sam9x5_clk_smd_setup(struct device_node *np) { struct clk *clk; int num_parents; const char *parent_names[SMD_SOURCE_MAX]; const char *name = np->name; + struct regmap *regmap; num_parents = of_clk_get_parent_count(np); if (num_parents <= 0 || num_parents > SMD_SOURCE_MAX) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:158 @ void __init of_at91sam9x5_clk_smd_setup( of_property_read_string(np, "clock-output-names", &name); - clk = at91sam9x5_clk_register_smd(pmc, name, parent_names, + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) + return; + + clk = at91sam9x5_clk_register_smd(regmap, name, parent_names, num_parents); if (IS_ERR(clk)) return; of_clk_add_provider(np, of_clk_src_simple_get, clk); } +CLK_OF_DECLARE(at91sam9x5_clk_smd, "atmel,at91sam9x5-clk-smd", + of_at91sam9x5_clk_smd_setup); Index: linux-4.4.25-rt35/drivers/clk/at91/clk-system.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/clk-system.c +++ linux-4.4.25-rt35/drivers/clk/at91/clk-system.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:15 @ #include <linux/clkdev.h> #include <linux/clk/at91_pmc.h> #include <linux/of.h> -#include <linux/of_address.h> -#include <linux/io.h> -#include <linux/irq.h> -#include <linux/of_irq.h> -#include <linux/interrupt.h> -#include <linux/wait.h> -#include <linux/sched.h> +#include <linux/mfd/syscon.h> +#include <linux/regmap.h> #include "pmc.h" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:27 @ #define to_clk_system(hw) container_of(hw, struct clk_system, hw) struct clk_system { struct clk_hw hw; - struct at91_pmc *pmc; - unsigned int irq; - wait_queue_head_t wait; + struct regmap *regmap; u8 id; }; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:35 @ static inline int is_pck(int id) { return (id >= 8) && (id <= 15); } -static irqreturn_t clk_system_irq_handler(int irq, void *dev_id) + +static inline bool clk_system_ready(struct regmap *regmap, int id) { - struct clk_system *sys = (struct clk_system *)dev_id; + unsigned int status; - wake_up(&sys->wait); - disable_irq_nosync(sys->irq); + regmap_read(regmap, AT91_PMC_SR, &status); - return IRQ_HANDLED; + return status & (1 << id) ? 1 : 0; } static int clk_system_prepare(struct clk_hw *hw) { struct clk_system *sys = to_clk_system(hw); - struct at91_pmc *pmc = sys->pmc; - u32 mask = 1 << sys->id; - pmc_write(pmc, AT91_PMC_SCER, mask); + regmap_write(sys->regmap, AT91_PMC_SCER, 1 << sys->id); if (!is_pck(sys->id)) return 0; - while (!(pmc_read(pmc, AT91_PMC_SR) & mask)) { - if (sys->irq) { - enable_irq(sys->irq); - wait_event(sys->wait, - pmc_read(pmc, AT91_PMC_SR) & mask); - } else - cpu_relax(); - } + while (!clk_system_ready(sys->regmap, sys->id)) + cpu_relax(); + return 0; } static void clk_system_unprepare(struct clk_hw *hw) { struct clk_system *sys = to_clk_system(hw); - struct at91_pmc *pmc = sys->pmc; - pmc_write(pmc, AT91_PMC_SCDR, 1 << sys->id); + regmap_write(sys->regmap, AT91_PMC_SCDR, 1 << sys->id); } static int clk_system_is_prepared(struct clk_hw *hw) { struct clk_system *sys = to_clk_system(hw); - struct at91_pmc *pmc = sys->pmc; + unsigned int status; + + regmap_read(sys->regmap, AT91_PMC_SCSR, &status); - if (!(pmc_read(pmc, AT91_PMC_SCSR) & (1 << sys->id))) + if (!(status & (1 << sys->id))) return 0; if (!is_pck(sys->id)) return 1; - return !!(pmc_read(pmc, AT91_PMC_SR) & (1 << sys->id)); + regmap_read(sys->regmap, AT91_PMC_SR, &status); + + return status & (1 << sys->id) ? 1 : 0; } static const struct clk_ops system_ops = { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:92 @ static const struct clk_ops system_ops = }; static struct clk * __init -at91_clk_register_system(struct at91_pmc *pmc, const char *name, - const char *parent_name, u8 id, int irq) +at91_clk_register_system(struct regmap *regmap, const char *name, + const char *parent_name, u8 id) { struct clk_system *sys; struct clk *clk = NULL; struct clk_init_data init; - int ret; if (!parent_name || id > SYSTEM_MAX_ID) return ERR_PTR(-EINVAL); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:114 @ at91_clk_register_system(struct at91_pmc sys->id = id; sys->hw.init = &init; - sys->pmc = pmc; - sys->irq = irq; - if (irq) { - init_waitqueue_head(&sys->wait); - irq_set_status_flags(sys->irq, IRQ_NOAUTOEN); - ret = request_irq(sys->irq, clk_system_irq_handler, - IRQF_TRIGGER_HIGH, name, sys); - if (ret) { - kfree(sys); - return ERR_PTR(ret); - } - } + sys->regmap = regmap; clk = clk_register(NULL, &sys->hw); - if (IS_ERR(clk)) { - if (irq) - free_irq(sys->irq, sys); + if (IS_ERR(clk)) kfree(sys); - } return clk; } -static void __init -of_at91_clk_sys_setup(struct device_node *np, struct at91_pmc *pmc) +static void __init of_at91rm9200_clk_sys_setup(struct device_node *np) { int num; - int irq = 0; u32 id; struct clk *clk; const char *name; struct device_node *sysclknp; const char *parent_name; + struct regmap *regmap; num = of_get_child_count(np); if (num > (SYSTEM_MAX_ID + 1)) return; + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) + return; + for_each_child_of_node(np, sysclknp) { if (of_property_read_u32(sysclknp, "reg", &id)) continue; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:148 @ of_at91_clk_sys_setup(struct device_node if (of_property_read_string(np, "clock-output-names", &name)) name = sysclknp->name; - if (is_pck(id)) - irq = irq_of_parse_and_map(sysclknp, 0); - parent_name = of_clk_get_parent_name(sysclknp, 0); - clk = at91_clk_register_system(pmc, name, parent_name, id, irq); + clk = at91_clk_register_system(regmap, name, parent_name, id); if (IS_ERR(clk)) continue; of_clk_add_provider(sysclknp, of_clk_src_simple_get, clk); } } - -void __init of_at91rm9200_clk_sys_setup(struct device_node *np, - struct at91_pmc *pmc) -{ - of_at91_clk_sys_setup(np, pmc); -} +CLK_OF_DECLARE(at91rm9200_clk_sys, "atmel,at91rm9200-clk-system", + of_at91rm9200_clk_sys_setup); Index: linux-4.4.25-rt35/drivers/clk/at91/clk-usb.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/clk-usb.c +++ linux-4.4.25-rt35/drivers/clk/at91/clk-usb.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:15 @ #include <linux/clkdev.h> #include <linux/clk/at91_pmc.h> #include <linux/of.h> -#include <linux/of_address.h> -#include <linux/io.h> +#include <linux/mfd/syscon.h> +#include <linux/regmap.h> #include "pmc.h" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:30 @ struct at91sam9x5_clk_usb { struct clk_hw hw; - struct at91_pmc *pmc; + struct regmap *regmap; }; #define to_at91sam9x5_clk_usb(hw) \ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:38 @ struct at91sam9x5_clk_usb { struct at91rm9200_clk_usb { struct clk_hw hw; - struct at91_pmc *pmc; + struct regmap *regmap; u32 divisors[4]; }; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:48 @ struct at91rm9200_clk_usb { static unsigned long at91sam9x5_clk_usb_recalc_rate(struct clk_hw *hw, unsigned long parent_rate) { - u32 tmp; - u8 usbdiv; struct at91sam9x5_clk_usb *usb = to_at91sam9x5_clk_usb(hw); - struct at91_pmc *pmc = usb->pmc; + unsigned int usbr; + u8 usbdiv; - tmp = pmc_read(pmc, AT91_PMC_USB); - usbdiv = (tmp & AT91_PMC_OHCIUSBDIV) >> SAM9X5_USB_DIV_SHIFT; + regmap_read(usb->regmap, AT91_PMC_USB, &usbr); + usbdiv = (usbr & AT91_PMC_OHCIUSBDIV) >> SAM9X5_USB_DIV_SHIFT; return DIV_ROUND_CLOSEST(parent_rate, (usbdiv + 1)); } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:111 @ static int at91sam9x5_clk_usb_determine_ static int at91sam9x5_clk_usb_set_parent(struct clk_hw *hw, u8 index) { - u32 tmp; struct at91sam9x5_clk_usb *usb = to_at91sam9x5_clk_usb(hw); - struct at91_pmc *pmc = usb->pmc; if (index > 1) return -EINVAL; - tmp = pmc_read(pmc, AT91_PMC_USB) & ~AT91_PMC_USBS; - if (index) - tmp |= AT91_PMC_USBS; - pmc_write(pmc, AT91_PMC_USB, tmp); + + regmap_update_bits(usb->regmap, AT91_PMC_USB, AT91_PMC_USBS, + index ? AT91_PMC_USBS : 0); + return 0; } static u8 at91sam9x5_clk_usb_get_parent(struct clk_hw *hw) { struct at91sam9x5_clk_usb *usb = to_at91sam9x5_clk_usb(hw); - struct at91_pmc *pmc = usb->pmc; + unsigned int usbr; - return pmc_read(pmc, AT91_PMC_USB) & AT91_PMC_USBS; + regmap_read(usb->regmap, AT91_PMC_USB, &usbr); + + return usbr & AT91_PMC_USBS; } static int at91sam9x5_clk_usb_set_rate(struct clk_hw *hw, unsigned long rate, unsigned long parent_rate) { - u32 tmp; struct at91sam9x5_clk_usb *usb = to_at91sam9x5_clk_usb(hw); - struct at91_pmc *pmc = usb->pmc; unsigned long div; if (!rate) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:145 @ static int at91sam9x5_clk_usb_set_rate(s if (div > SAM9X5_USB_MAX_DIV + 1 || !div) return -EINVAL; - tmp = pmc_read(pmc, AT91_PMC_USB) & ~AT91_PMC_OHCIUSBDIV; - tmp |= (div - 1) << SAM9X5_USB_DIV_SHIFT; - pmc_write(pmc, AT91_PMC_USB, tmp); + regmap_update_bits(usb->regmap, AT91_PMC_USB, AT91_PMC_OHCIUSBDIV, + (div - 1) << SAM9X5_USB_DIV_SHIFT); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:162 @ static const struct clk_ops at91sam9x5_u static int at91sam9n12_clk_usb_enable(struct clk_hw *hw) { struct at91sam9x5_clk_usb *usb = to_at91sam9x5_clk_usb(hw); - struct at91_pmc *pmc = usb->pmc; - pmc_write(pmc, AT91_PMC_USB, - pmc_read(pmc, AT91_PMC_USB) | AT91_PMC_USBS); + regmap_update_bits(usb->regmap, AT91_PMC_USB, AT91_PMC_USBS, + AT91_PMC_USBS); + return 0; } static void at91sam9n12_clk_usb_disable(struct clk_hw *hw) { struct at91sam9x5_clk_usb *usb = to_at91sam9x5_clk_usb(hw); - struct at91_pmc *pmc = usb->pmc; - pmc_write(pmc, AT91_PMC_USB, - pmc_read(pmc, AT91_PMC_USB) & ~AT91_PMC_USBS); + regmap_update_bits(usb->regmap, AT91_PMC_USB, AT91_PMC_USBS, 0); } static int at91sam9n12_clk_usb_is_enabled(struct clk_hw *hw) { struct at91sam9x5_clk_usb *usb = to_at91sam9x5_clk_usb(hw); - struct at91_pmc *pmc = usb->pmc; + unsigned int usbr; - return !!(pmc_read(pmc, AT91_PMC_USB) & AT91_PMC_USBS); + regmap_read(usb->regmap, AT91_PMC_USB, &usbr); + + return usbr & AT91_PMC_USBS; } static const struct clk_ops at91sam9n12_usb_ops = { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:196 @ static const struct clk_ops at91sam9n12_ }; static struct clk * __init -at91sam9x5_clk_register_usb(struct at91_pmc *pmc, const char *name, +at91sam9x5_clk_register_usb(struct regmap *regmap, const char *name, const char **parent_names, u8 num_parents) { struct at91sam9x5_clk_usb *usb; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:215 @ at91sam9x5_clk_register_usb(struct at91_ CLK_SET_RATE_PARENT; usb->hw.init = &init; - usb->pmc = pmc; + usb->regmap = regmap; clk = clk_register(NULL, &usb->hw); if (IS_ERR(clk)) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:225 @ at91sam9x5_clk_register_usb(struct at91_ } static struct clk * __init -at91sam9n12_clk_register_usb(struct at91_pmc *pmc, const char *name, +at91sam9n12_clk_register_usb(struct regmap *regmap, const char *name, const char *parent_name) { struct at91sam9x5_clk_usb *usb; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:243 @ at91sam9n12_clk_register_usb(struct at91 init.flags = CLK_SET_RATE_GATE | CLK_SET_RATE_PARENT; usb->hw.init = &init; - usb->pmc = pmc; + usb->regmap = regmap; clk = clk_register(NULL, &usb->hw); if (IS_ERR(clk)) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:256 @ static unsigned long at91rm9200_clk_usb_ unsigned long parent_rate) { struct at91rm9200_clk_usb *usb = to_at91rm9200_clk_usb(hw); - struct at91_pmc *pmc = usb->pmc; - u32 tmp; + unsigned int pllbr; u8 usbdiv; - tmp = pmc_read(pmc, AT91_CKGR_PLLBR); - usbdiv = (tmp & AT91_PMC_USBDIV) >> RM9200_USB_DIV_SHIFT; + regmap_read(usb->regmap, AT91_CKGR_PLLBR, &pllbr); + + usbdiv = (pllbr & AT91_PMC_USBDIV) >> RM9200_USB_DIV_SHIFT; if (usb->divisors[usbdiv]) return parent_rate / usb->divisors[usbdiv]; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:309 @ static long at91rm9200_clk_usb_round_rat static int at91rm9200_clk_usb_set_rate(struct clk_hw *hw, unsigned long rate, unsigned long parent_rate) { - u32 tmp; int i; struct at91rm9200_clk_usb *usb = to_at91rm9200_clk_usb(hw); - struct at91_pmc *pmc = usb->pmc; unsigned long div; if (!rate) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:320 @ static int at91rm9200_clk_usb_set_rate(s for (i = 0; i < RM9200_USB_DIV_TAB_SIZE; i++) { if (usb->divisors[i] == div) { - tmp = pmc_read(pmc, AT91_CKGR_PLLBR) & - ~AT91_PMC_USBDIV; - tmp |= i << RM9200_USB_DIV_SHIFT; - pmc_write(pmc, AT91_CKGR_PLLBR, tmp); + regmap_update_bits(usb->regmap, AT91_CKGR_PLLBR, + AT91_PMC_USBDIV, + i << RM9200_USB_DIV_SHIFT); + return 0; } } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:338 @ static const struct clk_ops at91rm9200_u }; static struct clk * __init -at91rm9200_clk_register_usb(struct at91_pmc *pmc, const char *name, +at91rm9200_clk_register_usb(struct regmap *regmap, const char *name, const char *parent_name, const u32 *divisors) { struct at91rm9200_clk_usb *usb; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:356 @ at91rm9200_clk_register_usb(struct at91_ init.flags = CLK_SET_RATE_PARENT; usb->hw.init = &init; - usb->pmc = pmc; + usb->regmap = regmap; memcpy(usb->divisors, divisors, sizeof(usb->divisors)); clk = clk_register(NULL, &usb->hw); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:366 @ at91rm9200_clk_register_usb(struct at91_ return clk; } -void __init of_at91sam9x5_clk_usb_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91sam9x5_clk_usb_setup(struct device_node *np) { struct clk *clk; int num_parents; const char *parent_names[USB_SOURCE_MAX]; const char *name = np->name; + struct regmap *regmap; num_parents = of_clk_get_parent_count(np); if (num_parents <= 0 || num_parents > USB_SOURCE_MAX) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:382 @ void __init of_at91sam9x5_clk_usb_setup( of_property_read_string(np, "clock-output-names", &name); - clk = at91sam9x5_clk_register_usb(pmc, name, parent_names, num_parents); + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) + return; + + clk = at91sam9x5_clk_register_usb(regmap, name, parent_names, + num_parents); if (IS_ERR(clk)) return; of_clk_add_provider(np, of_clk_src_simple_get, clk); } +CLK_OF_DECLARE(at91sam9x5_clk_usb, "atmel,at91sam9x5-clk-usb", + of_at91sam9x5_clk_usb_setup); -void __init of_at91sam9n12_clk_usb_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91sam9n12_clk_usb_setup(struct device_node *np) { struct clk *clk; const char *parent_name; const char *name = np->name; + struct regmap *regmap; parent_name = of_clk_get_parent_name(np, 0); if (!parent_name) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:409 @ void __init of_at91sam9n12_clk_usb_setup of_property_read_string(np, "clock-output-names", &name); - clk = at91sam9n12_clk_register_usb(pmc, name, parent_name); + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) + return; + + clk = at91sam9n12_clk_register_usb(regmap, name, parent_name); if (IS_ERR(clk)) return; of_clk_add_provider(np, of_clk_src_simple_get, clk); } +CLK_OF_DECLARE(at91sam9n12_clk_usb, "atmel,at91sam9n12-clk-usb", + of_at91sam9n12_clk_usb_setup); -void __init of_at91rm9200_clk_usb_setup(struct device_node *np, - struct at91_pmc *pmc) +static void __init of_at91rm9200_clk_usb_setup(struct device_node *np) { struct clk *clk; const char *parent_name; const char *name = np->name; u32 divisors[4] = {0, 0, 0, 0}; + struct regmap *regmap; parent_name = of_clk_get_parent_name(np, 0); if (!parent_name) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:440 @ void __init of_at91rm9200_clk_usb_setup( of_property_read_string(np, "clock-output-names", &name); - clk = at91rm9200_clk_register_usb(pmc, name, parent_name, divisors); + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) + return; + + clk = at91rm9200_clk_register_usb(regmap, name, parent_name, divisors); if (IS_ERR(clk)) return; of_clk_add_provider(np, of_clk_src_simple_get, clk); } +CLK_OF_DECLARE(at91rm9200_clk_usb, "atmel,at91rm9200-clk-usb", + of_at91rm9200_clk_usb_setup); Index: linux-4.4.25-rt35/drivers/clk/at91/clk-utmi.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/clk-utmi.c +++ linux-4.4.25-rt35/drivers/clk/at91/clk-utmi.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:14 @ #include <linux/clk-provider.h> #include <linux/clkdev.h> #include <linux/clk/at91_pmc.h> -#include <linux/interrupt.h> -#include <linux/irq.h> #include <linux/of.h> -#include <linux/of_address.h> -#include <linux/of_irq.h> -#include <linux/io.h> -#include <linux/sched.h> -#include <linux/wait.h> +#include <linux/mfd/syscon.h> +#include <linux/regmap.h> #include "pmc.h" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:24 @ struct clk_utmi { struct clk_hw hw; - struct at91_pmc *pmc; - unsigned int irq; - wait_queue_head_t wait; + struct regmap *regmap; }; #define to_clk_utmi(hw) container_of(hw, struct clk_utmi, hw) -static irqreturn_t clk_utmi_irq_handler(int irq, void *dev_id) +static inline bool clk_utmi_ready(struct regmap *regmap) { - struct clk_utmi *utmi = (struct clk_utmi *)dev_id; + unsigned int status; - wake_up(&utmi->wait); - disable_irq_nosync(utmi->irq); + regmap_read(regmap, AT91_PMC_SR, &status); - return IRQ_HANDLED; + return status & AT91_PMC_LOCKU; } static int clk_utmi_prepare(struct clk_hw *hw) { struct clk_utmi *utmi = to_clk_utmi(hw); - struct at91_pmc *pmc = utmi->pmc; - u32 tmp = pmc_read(pmc, AT91_CKGR_UCKR) | AT91_PMC_UPLLEN | - AT91_PMC_UPLLCOUNT | AT91_PMC_BIASEN; - - pmc_write(pmc, AT91_CKGR_UCKR, tmp); - - while (!(pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_LOCKU)) { - enable_irq(utmi->irq); - wait_event(utmi->wait, - pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_LOCKU); - } + unsigned int uckr = AT91_PMC_UPLLEN | AT91_PMC_UPLLCOUNT | + AT91_PMC_BIASEN; + + regmap_update_bits(utmi->regmap, AT91_CKGR_UCKR, uckr, uckr); + + while (!clk_utmi_ready(utmi->regmap)) + cpu_relax(); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:55 @ static int clk_utmi_prepare(struct clk_h static int clk_utmi_is_prepared(struct clk_hw *hw) { struct clk_utmi *utmi = to_clk_utmi(hw); - struct at91_pmc *pmc = utmi->pmc; - return !!(pmc_read(pmc, AT91_PMC_SR) & AT91_PMC_LOCKU); + return clk_utmi_ready(utmi->regmap); } static void clk_utmi_unprepare(struct clk_hw *hw) { struct clk_utmi *utmi = to_clk_utmi(hw); - struct at91_pmc *pmc = utmi->pmc; - u32 tmp = pmc_read(pmc, AT91_CKGR_UCKR) & ~AT91_PMC_UPLLEN; - pmc_write(pmc, AT91_CKGR_UCKR, tmp); + regmap_update_bits(utmi->regmap, AT91_CKGR_UCKR, AT91_PMC_UPLLEN, 0); } static unsigned long clk_utmi_recalc_rate(struct clk_hw *hw, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:81 @ static const struct clk_ops utmi_ops = { }; static struct clk * __init -at91_clk_register_utmi(struct at91_pmc *pmc, unsigned int irq, +at91_clk_register_utmi(struct regmap *regmap, const char *name, const char *parent_name) { - int ret; struct clk_utmi *utmi; struct clk *clk = NULL; struct clk_init_data init; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:99 @ at91_clk_register_utmi(struct at91_pmc * init.flags = CLK_SET_RATE_GATE; utmi->hw.init = &init; - utmi->pmc = pmc; - utmi->irq = irq; - init_waitqueue_head(&utmi->wait); - irq_set_status_flags(utmi->irq, IRQ_NOAUTOEN); - ret = request_irq(utmi->irq, clk_utmi_irq_handler, - IRQF_TRIGGER_HIGH, "clk-utmi", utmi); - if (ret) { - kfree(utmi); - return ERR_PTR(ret); - } + utmi->regmap = regmap; clk = clk_register(NULL, &utmi->hw); - if (IS_ERR(clk)) { - free_irq(utmi->irq, utmi); + if (IS_ERR(clk)) kfree(utmi); - } return clk; } -static void __init -of_at91_clk_utmi_setup(struct device_node *np, struct at91_pmc *pmc) +static void __init of_at91sam9x5_clk_utmi_setup(struct device_node *np) { - unsigned int irq; struct clk *clk; const char *parent_name; const char *name = np->name; + struct regmap *regmap; parent_name = of_clk_get_parent_name(np, 0); of_property_read_string(np, "clock-output-names", &name); - irq = irq_of_parse_and_map(np, 0); - if (!irq) + regmap = syscon_node_to_regmap(of_get_parent(np)); + if (IS_ERR(regmap)) return; - clk = at91_clk_register_utmi(pmc, irq, name, parent_name); + clk = at91_clk_register_utmi(regmap, name, parent_name); if (IS_ERR(clk)) return; of_clk_add_provider(np, of_clk_src_simple_get, clk); return; } - -void __init of_at91sam9x5_clk_utmi_setup(struct device_node *np, - struct at91_pmc *pmc) -{ - of_at91_clk_utmi_setup(np, pmc); -} +CLK_OF_DECLARE(at91sam9x5_clk_utmi, "atmel,at91sam9x5-clk-utmi", + of_at91sam9x5_clk_utmi_setup); Index: linux-4.4.25-rt35/drivers/clk/at91/pmc.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/pmc.c +++ linux-4.4.25-rt35/drivers/clk/at91/pmc.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:15 @ #include <linux/clkdev.h> #include <linux/clk/at91_pmc.h> #include <linux/of.h> -#include <linux/of_address.h> -#include <linux/io.h> -#include <linux/interrupt.h> -#include <linux/irq.h> -#include <linux/irqchip/chained_irq.h> -#include <linux/irqdomain.h> -#include <linux/of_irq.h> +#include <linux/mfd/syscon.h> +#include <linux/regmap.h> #include <asm/proc-fns.h> #include "pmc.h" -void __iomem *at91_pmc_base; -EXPORT_SYMBOL_GPL(at91_pmc_base); - -void at91rm9200_idle(void) -{ - /* - * Disable the processor clock. The processor will be automatically - * re-enabled by an interrupt or by a reset. - */ - at91_pmc_write(AT91_PMC_SCDR, AT91_PMC_PCK); -} - -void at91sam9_idle(void) -{ - at91_pmc_write(AT91_PMC_SCDR, AT91_PMC_PCK); - cpu_do_idle(); -} - int of_at91_get_clk_range(struct device_node *np, const char *propname, struct clk_range *range) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:44 @ int of_at91_get_clk_range(struct device_ return 0; } EXPORT_SYMBOL_GPL(of_at91_get_clk_range); - -static void pmc_irq_mask(struct irq_data *d) -{ - struct at91_pmc *pmc = irq_data_get_irq_chip_data(d); - - pmc_write(pmc, AT91_PMC_IDR, 1 << d->hwirq); -} - -static void pmc_irq_unmask(struct irq_data *d) -{ - struct at91_pmc *pmc = irq_data_get_irq_chip_data(d); - - pmc_write(pmc, AT91_PMC_IER, 1 << d->hwirq); -} - -static int pmc_irq_set_type(struct irq_data *d, unsigned type) -{ - if (type != IRQ_TYPE_LEVEL_HIGH) { - pr_warn("PMC: type not supported (support only IRQ_TYPE_LEVEL_HIGH type)\n"); - return -EINVAL; - } - - return 0; -} - -static void pmc_irq_suspend(struct irq_data *d) -{ - struct at91_pmc *pmc = irq_data_get_irq_chip_data(d); - - pmc->imr = pmc_read(pmc, AT91_PMC_IMR); - pmc_write(pmc, AT91_PMC_IDR, pmc->imr); -} - -static void pmc_irq_resume(struct irq_data *d) -{ - struct at91_pmc *pmc = irq_data_get_irq_chip_data(d); - - pmc_write(pmc, AT91_PMC_IER, pmc->imr); -} - -static struct irq_chip pmc_irq = { - .name = "PMC", - .irq_disable = pmc_irq_mask, - .irq_mask = pmc_irq_mask, - .irq_unmask = pmc_irq_unmask, - .irq_set_type = pmc_irq_set_type, - .irq_suspend = pmc_irq_suspend, - .irq_resume = pmc_irq_resume, -}; - -static struct lock_class_key pmc_lock_class; - -static int pmc_irq_map(struct irq_domain *h, unsigned int virq, - irq_hw_number_t hw) -{ - struct at91_pmc *pmc = h->host_data; - - irq_set_lockdep_class(virq, &pmc_lock_class); - - irq_set_chip_and_handler(virq, &pmc_irq, - handle_level_irq); - irq_set_chip_data(virq, pmc); - - return 0; -} - -static int pmc_irq_domain_xlate(struct irq_domain *d, - struct device_node *ctrlr, - const u32 *intspec, unsigned int intsize, - irq_hw_number_t *out_hwirq, - unsigned int *out_type) -{ - struct at91_pmc *pmc = d->host_data; - const struct at91_pmc_caps *caps = pmc->caps; - - if (WARN_ON(intsize < 1)) - return -EINVAL; - - *out_hwirq = intspec[0]; - - if (!(caps->available_irqs & (1 << *out_hwirq))) - return -EINVAL; - - *out_type = IRQ_TYPE_LEVEL_HIGH; - - return 0; -} - -static const struct irq_domain_ops pmc_irq_ops = { - .map = pmc_irq_map, - .xlate = pmc_irq_domain_xlate, -}; - -static irqreturn_t pmc_irq_handler(int irq, void *data) -{ - struct at91_pmc *pmc = (struct at91_pmc *)data; - unsigned long sr; - int n; - - sr = pmc_read(pmc, AT91_PMC_SR) & pmc_read(pmc, AT91_PMC_IMR); - if (!sr) - return IRQ_NONE; - - for_each_set_bit(n, &sr, BITS_PER_LONG) - generic_handle_irq(irq_find_mapping(pmc->irqdomain, n)); - - return IRQ_HANDLED; -} - -static const struct at91_pmc_caps at91rm9200_caps = { - .available_irqs = AT91_PMC_MOSCS | AT91_PMC_LOCKA | AT91_PMC_LOCKB | - AT91_PMC_MCKRDY | AT91_PMC_PCK0RDY | - AT91_PMC_PCK1RDY | AT91_PMC_PCK2RDY | - AT91_PMC_PCK3RDY, -}; - -static const struct at91_pmc_caps at91sam9260_caps = { - .available_irqs = AT91_PMC_MOSCS | AT91_PMC_LOCKA | AT91_PMC_LOCKB | - AT91_PMC_MCKRDY | AT91_PMC_PCK0RDY | - AT91_PMC_PCK1RDY, -}; - -static const struct at91_pmc_caps at91sam9g45_caps = { - .available_irqs = AT91_PMC_MOSCS | AT91_PMC_LOCKA | AT91_PMC_MCKRDY | - AT91_PMC_LOCKU | AT91_PMC_PCK0RDY | - AT91_PMC_PCK1RDY, -}; - -static const struct at91_pmc_caps at91sam9n12_caps = { - .available_irqs = AT91_PMC_MOSCS | AT91_PMC_LOCKA | AT91_PMC_LOCKB | - AT91_PMC_MCKRDY | AT91_PMC_PCK0RDY | - AT91_PMC_PCK1RDY | AT91_PMC_MOSCSELS | - AT91_PMC_MOSCRCS | AT91_PMC_CFDEV, -}; - -static const struct at91_pmc_caps at91sam9x5_caps = { - .available_irqs = AT91_PMC_MOSCS | AT91_PMC_LOCKA | AT91_PMC_MCKRDY | - AT91_PMC_LOCKU | AT91_PMC_PCK0RDY | - AT91_PMC_PCK1RDY | AT91_PMC_MOSCSELS | - AT91_PMC_MOSCRCS | AT91_PMC_CFDEV, -}; - -static const struct at91_pmc_caps sama5d2_caps = { - .available_irqs = AT91_PMC_MOSCS | AT91_PMC_LOCKA | AT91_PMC_MCKRDY | - AT91_PMC_LOCKU | AT91_PMC_PCK0RDY | - AT91_PMC_PCK1RDY | AT91_PMC_PCK2RDY | - AT91_PMC_MOSCSELS | AT91_PMC_MOSCRCS | - AT91_PMC_CFDEV | AT91_PMC_GCKRDY, -}; - -static const struct at91_pmc_caps sama5d3_caps = { - .available_irqs = AT91_PMC_MOSCS | AT91_PMC_LOCKA | AT91_PMC_MCKRDY | - AT91_PMC_LOCKU | AT91_PMC_PCK0RDY | - AT91_PMC_PCK1RDY | AT91_PMC_PCK2RDY | - AT91_PMC_MOSCSELS | AT91_PMC_MOSCRCS | - AT91_PMC_CFDEV, -}; - -static struct at91_pmc *__init at91_pmc_init(struct device_node *np, - void __iomem *regbase, int virq, - const struct at91_pmc_caps *caps) -{ - struct at91_pmc *pmc; - - if (!regbase || !virq || !caps) - return NULL; - - at91_pmc_base = regbase; - - pmc = kzalloc(sizeof(*pmc), GFP_KERNEL); - if (!pmc) - return NULL; - - spin_lock_init(&pmc->lock); - pmc->regbase = regbase; - pmc->virq = virq; - pmc->caps = caps; - - pmc->irqdomain = irq_domain_add_linear(np, 32, &pmc_irq_ops, pmc); - - if (!pmc->irqdomain) - goto out_free_pmc; - - pmc_write(pmc, AT91_PMC_IDR, 0xffffffff); - if (request_irq(pmc->virq, pmc_irq_handler, - IRQF_SHARED | IRQF_COND_SUSPEND, "pmc", pmc)) - goto out_remove_irqdomain; - - return pmc; - -out_remove_irqdomain: - irq_domain_remove(pmc->irqdomain); -out_free_pmc: - kfree(pmc); - - return NULL; -} - -static const struct of_device_id pmc_clk_ids[] __initconst = { - /* Slow oscillator */ - { - .compatible = "atmel,at91sam9260-clk-slow", - .data = of_at91sam9260_clk_slow_setup, - }, - /* Main clock */ - { - .compatible = "atmel,at91rm9200-clk-main-osc", - .data = of_at91rm9200_clk_main_osc_setup, - }, - { - .compatible = "atmel,at91sam9x5-clk-main-rc-osc", - .data = of_at91sam9x5_clk_main_rc_osc_setup, - }, - { - .compatible = "atmel,at91rm9200-clk-main", - .data = of_at91rm9200_clk_main_setup, - }, - { - .compatible = "atmel,at91sam9x5-clk-main", - .data = of_at91sam9x5_clk_main_setup, - }, - /* PLL clocks */ - { - .compatible = "atmel,at91rm9200-clk-pll", - .data = of_at91rm9200_clk_pll_setup, - }, - { - .compatible = "atmel,at91sam9g45-clk-pll", - .data = of_at91sam9g45_clk_pll_setup, - }, - { - .compatible = "atmel,at91sam9g20-clk-pllb", - .data = of_at91sam9g20_clk_pllb_setup, - }, - { - .compatible = "atmel,sama5d3-clk-pll", - .data = of_sama5d3_clk_pll_setup, - }, - { - .compatible = "atmel,at91sam9x5-clk-plldiv", - .data = of_at91sam9x5_clk_plldiv_setup, - }, - /* Master clock */ - { - .compatible = "atmel,at91rm9200-clk-master", - .data = of_at91rm9200_clk_master_setup, - }, - { - .compatible = "atmel,at91sam9x5-clk-master", - .data = of_at91sam9x5_clk_master_setup, - }, - /* System clocks */ - { - .compatible = "atmel,at91rm9200-clk-system", - .data = of_at91rm9200_clk_sys_setup, - }, - /* Peripheral clocks */ - { - .compatible = "atmel,at91rm9200-clk-peripheral", - .data = of_at91rm9200_clk_periph_setup, - }, - { - .compatible = "atmel,at91sam9x5-clk-peripheral", - .data = of_at91sam9x5_clk_periph_setup, - }, - /* Programmable clocks */ - { - .compatible = "atmel,at91rm9200-clk-programmable", - .data = of_at91rm9200_clk_prog_setup, - }, - { - .compatible = "atmel,at91sam9g45-clk-programmable", - .data = of_at91sam9g45_clk_prog_setup, - }, - { - .compatible = "atmel,at91sam9x5-clk-programmable", - .data = of_at91sam9x5_clk_prog_setup, - }, - /* UTMI clock */ -#if defined(CONFIG_HAVE_AT91_UTMI) - { - .compatible = "atmel,at91sam9x5-clk-utmi", - .data = of_at91sam9x5_clk_utmi_setup, - }, -#endif - /* USB clock */ -#if defined(CONFIG_HAVE_AT91_USB_CLK) - { - .compatible = "atmel,at91rm9200-clk-usb", - .data = of_at91rm9200_clk_usb_setup, - }, - { - .compatible = "atmel,at91sam9x5-clk-usb", - .data = of_at91sam9x5_clk_usb_setup, - }, - { - .compatible = "atmel,at91sam9n12-clk-usb", - .data = of_at91sam9n12_clk_usb_setup, - }, -#endif - /* SMD clock */ -#if defined(CONFIG_HAVE_AT91_SMD) - { - .compatible = "atmel,at91sam9x5-clk-smd", - .data = of_at91sam9x5_clk_smd_setup, - }, -#endif -#if defined(CONFIG_HAVE_AT91_H32MX) - { - .compatible = "atmel,sama5d4-clk-h32mx", - .data = of_sama5d4_clk_h32mx_setup, - }, -#endif -#if defined(CONFIG_HAVE_AT91_GENERATED_CLK) - { - .compatible = "atmel,sama5d2-clk-generated", - .data = of_sama5d2_clk_generated_setup, - }, -#endif - { /*sentinel*/ } -}; - -static void __init of_at91_pmc_setup(struct device_node *np, - const struct at91_pmc_caps *caps) -{ - struct at91_pmc *pmc; - struct device_node *childnp; - void (*clk_setup)(struct device_node *, struct at91_pmc *); - const struct of_device_id *clk_id; - void __iomem *regbase = of_iomap(np, 0); - int virq; - - if (!regbase) - return; - - virq = irq_of_parse_and_map(np, 0); - if (!virq) - return; - - pmc = at91_pmc_init(np, regbase, virq, caps); - if (!pmc) - return; - for_each_child_of_node(np, childnp) { - clk_id = of_match_node(pmc_clk_ids, childnp); - if (!clk_id) - continue; - clk_setup = clk_id->data; - clk_setup(childnp, pmc); - } -} - -static void __init of_at91rm9200_pmc_setup(struct device_node *np) -{ - of_at91_pmc_setup(np, &at91rm9200_caps); -} -CLK_OF_DECLARE(at91rm9200_clk_pmc, "atmel,at91rm9200-pmc", - of_at91rm9200_pmc_setup); - -static void __init of_at91sam9260_pmc_setup(struct device_node *np) -{ - of_at91_pmc_setup(np, &at91sam9260_caps); -} -CLK_OF_DECLARE(at91sam9260_clk_pmc, "atmel,at91sam9260-pmc", - of_at91sam9260_pmc_setup); - -static void __init of_at91sam9g45_pmc_setup(struct device_node *np) -{ - of_at91_pmc_setup(np, &at91sam9g45_caps); -} -CLK_OF_DECLARE(at91sam9g45_clk_pmc, "atmel,at91sam9g45-pmc", - of_at91sam9g45_pmc_setup); - -static void __init of_at91sam9n12_pmc_setup(struct device_node *np) -{ - of_at91_pmc_setup(np, &at91sam9n12_caps); -} -CLK_OF_DECLARE(at91sam9n12_clk_pmc, "atmel,at91sam9n12-pmc", - of_at91sam9n12_pmc_setup); - -static void __init of_at91sam9x5_pmc_setup(struct device_node *np) -{ - of_at91_pmc_setup(np, &at91sam9x5_caps); -} -CLK_OF_DECLARE(at91sam9x5_clk_pmc, "atmel,at91sam9x5-pmc", - of_at91sam9x5_pmc_setup); - -static void __init of_sama5d2_pmc_setup(struct device_node *np) -{ - of_at91_pmc_setup(np, &sama5d2_caps); -} -CLK_OF_DECLARE(sama5d2_clk_pmc, "atmel,sama5d2-pmc", - of_sama5d2_pmc_setup); - -static void __init of_sama5d3_pmc_setup(struct device_node *np) -{ - of_at91_pmc_setup(np, &sama5d3_caps); -} -CLK_OF_DECLARE(sama5d3_clk_pmc, "atmel,sama5d3-pmc", - of_sama5d3_pmc_setup); Index: linux-4.4.25-rt35/drivers/clk/at91/pmc.h =================================================================== --- linux-4.4.25-rt35.orig/drivers/clk/at91/pmc.h +++ linux-4.4.25-rt35/drivers/clk/at91/pmc.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:17 @ #include <linux/io.h> #include <linux/irqdomain.h> +#include <linux/regmap.h> #include <linux/spinlock.h> +extern spinlock_t pmc_pcr_lock; + struct clk_range { unsigned long min; unsigned long max; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:29 @ struct clk_range { #define CLK_RANGE(MIN, MAX) {.min = MIN, .max = MAX,} -struct at91_pmc_caps { - u32 available_irqs; -}; - -struct at91_pmc { - void __iomem *regbase; - int virq; - spinlock_t lock; - const struct at91_pmc_caps *caps; - struct irq_domain *irqdomain; - u32 imr; -}; - -static inline void pmc_lock(struct at91_pmc *pmc) -{ - spin_lock(&pmc->lock); -} - -static inline void pmc_unlock(struct at91_pmc *pmc) -{ - spin_unlock(&pmc->lock); -} - -static inline u32 pmc_read(struct at91_pmc *pmc, int offset) -{ - return readl(pmc->regbase + offset); -} - -static inline void pmc_write(struct at91_pmc *pmc, int offset, u32 value) -{ - writel(value, pmc->regbase + offset); -} - int of_at91_get_clk_range(struct device_node *np, const char *propname, struct clk_range *range); -void of_at91sam9260_clk_slow_setup(struct device_node *np, - struct at91_pmc *pmc); - -void of_at91rm9200_clk_main_osc_setup(struct device_node *np, - struct at91_pmc *pmc); -void of_at91sam9x5_clk_main_rc_osc_setup(struct device_node *np, - struct at91_pmc *pmc); -void of_at91rm9200_clk_main_setup(struct device_node *np, - struct at91_pmc *pmc); -void of_at91sam9x5_clk_main_setup(struct device_node *np, - struct at91_pmc *pmc); - -void of_at91rm9200_clk_pll_setup(struct device_node *np, - struct at91_pmc *pmc); -void of_at91sam9g45_clk_pll_setup(struct device_node *np, - struct at91_pmc *pmc); -void of_at91sam9g20_clk_pllb_setup(struct device_node *np, - struct at91_pmc *pmc); -void of_sama5d3_clk_pll_setup(struct device_node *np, - struct at91_pmc *pmc); -void of_at91sam9x5_clk_plldiv_setup(struct device_node *np, - struct at91_pmc *pmc); - -void of_at91rm9200_clk_master_setup(struct device_node *np, - struct at91_pmc *pmc); -void of_at91sam9x5_clk_master_setup(struct device_node *np, - struct at91_pmc *pmc); - -void of_at91rm9200_clk_sys_setup(struct device_node *np, - struct at91_pmc *pmc); - -void of_at91rm9200_clk_periph_setup(struct device_node *np, - struct at91_pmc *pmc); -void of_at91sam9x5_clk_periph_setup(struct device_node *np, - struct at91_pmc *pmc); - -void of_at91rm9200_clk_prog_setup(struct device_node *np, - struct at91_pmc *pmc); -void of_at91sam9g45_clk_prog_setup(struct device_node *np, - struct at91_pmc *pmc); -void of_at91sam9x5_clk_prog_setup(struct device_node *np, - struct at91_pmc *pmc); - -void of_at91sam9x5_clk_utmi_setup(struct device_node *np, - struct at91_pmc *pmc); - -void of_at91rm9200_clk_usb_setup(struct device_node *np, - struct at91_pmc *pmc); -void of_at91sam9x5_clk_usb_setup(struct device_node *np, - struct at91_pmc *pmc); -void of_at91sam9n12_clk_usb_setup(struct device_node *np, - struct at91_pmc *pmc); - -void of_at91sam9x5_clk_smd_setup(struct device_node *np, - struct at91_pmc *pmc); - -void of_sama5d4_clk_h32mx_setup(struct device_node *np, - struct at91_pmc *pmc); - -void of_sama5d2_clk_generated_setup(struct device_node *np, - struct at91_pmc *pmc); - #endif /* __PMC_H_ */ Index: linux-4.4.25-rt35/drivers/clocksource/tcb_clksrc.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clocksource/tcb_clksrc.c +++ linux-4.4.25-rt35/drivers/clocksource/tcb_clksrc.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:26 @ * this 32 bit free-running counter. the second channel is not used. * * - The third channel may be used to provide a 16-bit clockevent - * source, used in either periodic or oneshot mode. This runs - * at 32 KiHZ, and can handle delays of up to two seconds. + * source, used in either periodic or oneshot mode. * * A boot clocksource and clockevent source are also currently needed, * unless the relevant platforms (ARM/AT91, AVR32/AT32) are changed so @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:76 @ static struct clocksource clksrc = { struct tc_clkevt_device { struct clock_event_device clkevt; struct clk *clk; + bool clk_enabled; + u32 freq; void __iomem *regs; }; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:86 @ static struct tc_clkevt_device *to_tc_cl return container_of(clkevt, struct tc_clkevt_device, clkevt); } -/* For now, we always use the 32K clock ... this optimizes for NO_HZ, - * because using one of the divided clocks would usually mean the - * tick rate can never be less than several dozen Hz (vs 0.5 Hz). - * - * A divided clock could be good for high resolution timers, since - * 30.5 usec resolution can seem "low". - */ static u32 timer_clock; +static void tc_clk_disable(struct clock_event_device *d) +{ + struct tc_clkevt_device *tcd = to_tc_clkevt(d); + + clk_disable(tcd->clk); + tcd->clk_enabled = false; +} + +static void tc_clk_enable(struct clock_event_device *d) +{ + struct tc_clkevt_device *tcd = to_tc_clkevt(d); + + if (tcd->clk_enabled) + return; + clk_enable(tcd->clk); + tcd->clk_enabled = true; +} + static int tc_shutdown(struct clock_event_device *d) { struct tc_clkevt_device *tcd = to_tc_clkevt(d); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:113 @ static int tc_shutdown(struct clock_even __raw_writel(0xff, regs + ATMEL_TC_REG(2, IDR)); __raw_writel(ATMEL_TC_CLKDIS, regs + ATMEL_TC_REG(2, CCR)); + return 0; +} + +static int tc_shutdown_clk_off(struct clock_event_device *d) +{ + tc_shutdown(d); if (!clockevent_state_detached(d)) - clk_disable(tcd->clk); + tc_clk_disable(d); return 0; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:133 @ static int tc_set_oneshot(struct clock_e if (clockevent_state_oneshot(d) || clockevent_state_periodic(d)) tc_shutdown(d); - clk_enable(tcd->clk); + tc_clk_enable(d); - /* slow clock, count up to RC, then irq and stop */ + /* count up to RC, then irq and stop */ __raw_writel(timer_clock | ATMEL_TC_CPCSTOP | ATMEL_TC_WAVE | ATMEL_TC_WAVESEL_UP_AUTO, regs + ATMEL_TC_REG(2, CMR)); __raw_writel(ATMEL_TC_CPCS, regs + ATMEL_TC_REG(2, IER)); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:155 @ static int tc_set_periodic(struct clock_ /* By not making the gentime core emulate periodic mode on top * of oneshot, we get lower overhead and improved accuracy. */ - clk_enable(tcd->clk); + tc_clk_enable(d); - /* slow clock, count up to RC, then irq and restart */ + /* count up to RC, then irq and restart */ __raw_writel(timer_clock | ATMEL_TC_WAVE | ATMEL_TC_WAVESEL_UP_AUTO, regs + ATMEL_TC_REG(2, CMR)); - __raw_writel((32768 + HZ / 2) / HZ, tcaddr + ATMEL_TC_REG(2, RC)); + __raw_writel((tcd->freq + HZ / 2) / HZ, tcaddr + ATMEL_TC_REG(2, RC)); /* Enable clock and interrupts on RC compare */ __raw_writel(ATMEL_TC_CPCS, regs + ATMEL_TC_REG(2, IER)); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:187 @ static struct tc_clkevt_device clkevt = .features = CLOCK_EVT_FEAT_PERIODIC | CLOCK_EVT_FEAT_ONESHOT, /* Should be lower than at91rm9200's system timer */ +#ifdef CONFIG_ATMEL_TCB_CLKSRC_USE_SLOW_CLOCK .rating = 125, +#else + .rating = 200, +#endif .set_next_event = tc_next_event, - .set_state_shutdown = tc_shutdown, + .set_state_shutdown = tc_shutdown_clk_off, .set_state_periodic = tc_set_periodic, .set_state_oneshot = tc_set_oneshot, }, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:213 @ static irqreturn_t ch2_irq(int irq, void return IRQ_NONE; } -static int __init setup_clkevents(struct atmel_tc *tc, int clk32k_divisor_idx) +static int __init setup_clkevents(struct atmel_tc *tc, int divisor_idx) { + unsigned divisor = atmel_tc_divisors[divisor_idx]; int ret; struct clk *t2_clk = tc->clk[2]; int irq = tc->irq[2]; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:236 @ static int __init setup_clkevents(struct clkevt.regs = tc->regs; clkevt.clk = t2_clk; - timer_clock = clk32k_divisor_idx; + timer_clock = divisor_idx; + if (!divisor) + clkevt.freq = 32768; + else + clkevt.freq = clk_get_rate(t2_clk) / divisor; clkevt.clkevt.cpumask = cpumask_of(0); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:251 @ static int __init setup_clkevents(struct return ret; } - clockevents_config_and_register(&clkevt.clkevt, 32768, 1, 0xffff); + clockevents_config_and_register(&clkevt.clkevt, clkevt.freq, 1, 0xffff); return ret; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:388 @ static int __init tcb_clksrc_init(void) goto err_disable_t1; /* channel 2: periodic and oneshot timer support */ +#ifdef CONFIG_ATMEL_TCB_CLKSRC_USE_SLOW_CLOCK ret = setup_clkevents(tc, clk32k_divisor_idx); +#else + ret = setup_clkevents(tc, best_divisor_idx); +#endif if (ret) goto err_unregister_clksrc; Index: linux-4.4.25-rt35/drivers/clocksource/timer-atmel-pit.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clocksource/timer-atmel-pit.c +++ linux-4.4.25-rt35/drivers/clocksource/timer-atmel-pit.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:49 @ struct pit_data { u32 cycle; u32 cnt; unsigned int irq; + bool irq_requested; struct clk *mck; }; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:100 @ static int pit_clkevt_shutdown(struct cl /* disable irq, leaving the clocksource active */ pit_write(data->base, AT91_PIT_MR, (data->cycle - 1) | AT91_PIT_PITEN); + if (data->irq_requested) { + free_irq(data->irq, data); + data->irq_requested = false; + } return 0; } +static irqreturn_t at91sam926x_pit_interrupt(int irq, void *dev_id); /* * Clockevent device: interrupts every 1/HZ (== pit_cycles * MCK/16) */ static int pit_clkevt_set_periodic(struct clock_event_device *dev) { struct pit_data *data = clkevt_to_pit_data(dev); + int ret; + + ret = request_irq(data->irq, at91sam926x_pit_interrupt, + IRQF_SHARED | IRQF_TIMER | IRQF_IRQPOLL, + "at91_tick", data); + if (ret) + panic(pr_fmt("Unable to setup IRQ\n")); + + data->irq_requested = true; /* update clocksource counter */ data->cnt += data->cycle * PIT_PICNT(pit_read(data->base, AT91_PIT_PIVR)); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:199 @ static void __init at91sam926x_pit_commo { unsigned long pit_rate; unsigned bits; - int ret; /* * Use our actual MCK to figure out how many MCK/16 ticks per @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:223 @ static void __init at91sam926x_pit_commo data->clksrc.flags = CLOCK_SOURCE_IS_CONTINUOUS; clocksource_register_hz(&data->clksrc, pit_rate); - /* Set up irq handler */ - ret = request_irq(data->irq, at91sam926x_pit_interrupt, - IRQF_SHARED | IRQF_TIMER | IRQF_IRQPOLL, - "at91_tick", data); - if (ret) - panic(pr_fmt("Unable to setup IRQ\n")); - /* Set up and register clockevents */ data->clkevt.name = "pit"; data->clkevt.features = CLOCK_EVT_FEAT_PERIODIC; Index: linux-4.4.25-rt35/drivers/clocksource/timer-atmel-st.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/clocksource/timer-atmel-st.c +++ linux-4.4.25-rt35/drivers/clocksource/timer-atmel-st.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:118 @ static void clkdev32k_disable_and_flush_ last_crtr = read_CRTR(); } +static int atmel_st_irq; + static int clkevt32k_shutdown(struct clock_event_device *evt) { clkdev32k_disable_and_flush_irq(); irqmask = 0; regmap_write(regmap_st, AT91_ST_IER, irqmask); + free_irq(atmel_st_irq, regmap_st); return 0; } static int clkevt32k_set_oneshot(struct clock_event_device *dev) { + int ret; + clkdev32k_disable_and_flush_irq(); + ret = request_irq(atmel_st_irq, at91rm9200_timer_interrupt, + IRQF_SHARED | IRQF_TIMER | IRQF_IRQPOLL, + "at91_tick", regmap_st); + if (ret) + panic(pr_fmt("Unable to setup IRQ\n")); + /* * ALM for oneshot irqs, set by next_event() * before 32 seconds have passed. @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:153 @ static int clkevt32k_set_oneshot(struct static int clkevt32k_set_periodic(struct clock_event_device *dev) { + int ret; + clkdev32k_disable_and_flush_irq(); + ret = request_irq(atmel_st_irq, at91rm9200_timer_interrupt, + IRQF_SHARED | IRQF_TIMER | IRQF_IRQPOLL, + "at91_tick", regmap_st); + if (ret) + panic(pr_fmt("Unable to setup IRQ\n")); + /* PIT for periodic irqs; fixed rate of 1/HZ */ irqmask = AT91_ST_PITS; regmap_write(regmap_st, AT91_ST_PIMR, timer_latch); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:220 @ static void __init atmel_st_timer_init(s { struct clk *sclk; unsigned int sclk_rate, val; - int irq, ret; + int ret; regmap_st = syscon_node_to_regmap(node); if (IS_ERR(regmap_st)) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:232 @ static void __init atmel_st_timer_init(s regmap_read(regmap_st, AT91_ST_SR, &val); /* Get the interrupts property */ - irq = irq_of_parse_and_map(node, 0); - if (!irq) + atmel_st_irq = irq_of_parse_and_map(node, 0); + if (!atmel_st_irq) panic(pr_fmt("Unable to get IRQ from DT\n")); - /* Make IRQs happen for the system timer */ - ret = request_irq(irq, at91rm9200_timer_interrupt, - IRQF_SHARED | IRQF_TIMER | IRQF_IRQPOLL, - "at91_tick", regmap_st); - if (ret) - panic(pr_fmt("Unable to setup IRQ\n")); - sclk = of_clk_get(node, 0); if (IS_ERR(sclk)) panic(pr_fmt("Unable to get slow clock\n")); Index: linux-4.4.25-rt35/drivers/cpufreq/Kconfig.x86 =================================================================== --- linux-4.4.25-rt35.orig/drivers/cpufreq/Kconfig.x86 +++ linux-4.4.25-rt35/drivers/cpufreq/Kconfig.x86 @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:126 @ config X86_POWERNOW_K7_ACPI config X86_POWERNOW_K8 tristate "AMD Opteron/Athlon64 PowerNow!" - depends on ACPI && ACPI_PROCESSOR && X86_ACPI_CPUFREQ + depends on ACPI && ACPI_PROCESSOR && X86_ACPI_CPUFREQ && !PREEMPT_RT_BASE help This adds the CPUFreq driver for K8/early Opteron/Athlon64 processors. Support for K10 and newer processors is now in acpi-cpufreq. Index: linux-4.4.25-rt35/drivers/cpuidle/coupled.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/cpuidle/coupled.c +++ linux-4.4.25-rt35/drivers/cpuidle/coupled.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:122 @ struct cpuidle_coupled { #define CPUIDLE_COUPLED_NOT_IDLE (-1) -static DEFINE_MUTEX(cpuidle_coupled_lock); static DEFINE_PER_CPU(struct call_single_data, cpuidle_coupled_poke_cb); /* Index: linux-4.4.25-rt35/drivers/gpu/drm/i915/i915_gem_execbuffer.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/gpu/drm/i915/i915_gem_execbuffer.c +++ linux-4.4.25-rt35/drivers/gpu/drm/i915/i915_gem_execbuffer.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1267 @ i915_gem_ringbuffer_submission(struct i9 if (ret) return ret; +#ifndef CONFIG_PREEMPT_RT_BASE trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags); +#endif i915_gem_execbuffer_move_to_active(vmas, params->request); i915_gem_execbuffer_retire_commands(params); Index: linux-4.4.25-rt35/drivers/gpu/drm/i915/i915_gem_shrinker.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/gpu/drm/i915/i915_gem_shrinker.c +++ linux-4.4.25-rt35/drivers/gpu/drm/i915/i915_gem_shrinker.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:42 @ static bool mutex_is_locked_by(struct mu if (!mutex_is_locked(mutex)) return false; -#if defined(CONFIG_DEBUG_MUTEXES) || defined(CONFIG_MUTEX_SPIN_ON_OWNER) +#if (defined(CONFIG_DEBUG_MUTEXES) || defined(CONFIG_MUTEX_SPIN_ON_OWNER)) && !defined(CONFIG_PREEMPT_RT_BASE) return mutex->owner == task; #else /* Since UP may be pre-empted, we cannot assume that we own the lock */ Index: linux-4.4.25-rt35/drivers/gpu/drm/i915/i915_irq.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/gpu/drm/i915/i915_irq.c +++ linux-4.4.25-rt35/drivers/gpu/drm/i915/i915_irq.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:815 @ static int i915_get_crtc_scanoutpos(stru spin_lock_irqsave(&dev_priv->uncore.lock, irqflags); /* preempt_disable_rt() should go right here in PREEMPT_RT patchset. */ + preempt_disable_rt(); /* Get optional system timestamp before query. */ if (stime) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:867 @ static int i915_get_crtc_scanoutpos(stru *etime = ktime_get(); /* preempt_enable_rt() should go right here in PREEMPT_RT patchset. */ + preempt_enable_rt(); spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags); Index: linux-4.4.25-rt35/drivers/gpu/drm/i915/intel_display.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/gpu/drm/i915/intel_display.c +++ linux-4.4.25-rt35/drivers/gpu/drm/i915/intel_display.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:11403 @ void intel_check_page_flip(struct drm_de struct intel_crtc *intel_crtc = to_intel_crtc(crtc); struct intel_unpin_work *work; - WARN_ON(!in_interrupt()); + WARN_ON_NONRT(!in_interrupt()); if (crtc == NULL) return; Index: linux-4.4.25-rt35/drivers/gpu/drm/i915/intel_sprite.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/gpu/drm/i915/intel_sprite.c +++ linux-4.4.25-rt35/drivers/gpu/drm/i915/intel_sprite.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:41 @ #include "intel_drv.h" #include <drm/i915_drm.h> #include "i915_drv.h" +#include <linux/locallock.h> static bool format_is_yuv(uint32_t format) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:68 @ static int usecs_to_scanlines(const stru 1000 * adjusted_mode->crtc_htotal); } +static DEFINE_LOCAL_IRQ_LOCK(pipe_update_lock); + /** * intel_pipe_update_start() - start update of a set of display registers * @crtc: the crtc of which the registers are going to be updated @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:102 @ void intel_pipe_update_start(struct inte min = vblank_start - usecs_to_scanlines(adjusted_mode, 100); max = vblank_start - 1; - local_irq_disable(); + local_lock_irq(pipe_update_lock); if (min <= 0 || max <= 0) return; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:132 @ void intel_pipe_update_start(struct inte break; } - local_irq_enable(); + local_unlock_irq(pipe_update_lock); timeout = schedule_timeout(timeout); - local_irq_disable(); + local_lock_irq(pipe_update_lock); } finish_wait(wq, &wait); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:170 @ void intel_pipe_update_end(struct intel_ trace_i915_pipe_update_end(crtc, end_vbl_count, scanline_end); - local_irq_enable(); + local_unlock_irq(pipe_update_lock); if (crtc->debug.start_vbl_count && crtc->debug.start_vbl_count != end_vbl_count) { Index: linux-4.4.25-rt35/drivers/gpu/drm/radeon/radeon_display.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/gpu/drm/radeon/radeon_display.c +++ linux-4.4.25-rt35/drivers/gpu/drm/radeon/radeon_display.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1865 @ int radeon_get_crtc_scanoutpos(struct dr struct radeon_device *rdev = dev->dev_private; /* preempt_disable_rt() should go right here in PREEMPT_RT patchset. */ + preempt_disable_rt(); /* Get optional system timestamp before query. */ if (stime) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1958 @ int radeon_get_crtc_scanoutpos(struct dr *etime = ktime_get(); /* preempt_enable_rt() should go right here in PREEMPT_RT patchset. */ + preempt_enable_rt(); /* Decode into vertical and horizontal scanout position. */ *vpos = position & 0x1fff; Index: linux-4.4.25-rt35/drivers/hv/vmbus_drv.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/hv/vmbus_drv.c +++ linux-4.4.25-rt35/drivers/hv/vmbus_drv.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:823 @ static void vmbus_isr(void) tasklet_schedule(&msg_dpc); } - add_interrupt_randomness(HYPERVISOR_CALLBACK_VECTOR, 0); + add_interrupt_randomness(HYPERVISOR_CALLBACK_VECTOR, 0, 0); } Index: linux-4.4.25-rt35/drivers/i2c/busses/i2c-omap.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/i2c/busses/i2c-omap.c +++ linux-4.4.25-rt35/drivers/i2c/busses/i2c-omap.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:998 @ omap_i2c_isr(int irq, void *dev_id) u16 mask; u16 stat; - spin_lock(&omap->lock); - mask = omap_i2c_read_reg(omap, OMAP_I2C_IE_REG); stat = omap_i2c_read_reg(omap, OMAP_I2C_STAT_REG); + mask = omap_i2c_read_reg(omap, OMAP_I2C_IE_REG); if (stat & mask) ret = IRQ_WAKE_THREAD; - spin_unlock(&omap->lock); - return ret; } Index: linux-4.4.25-rt35/drivers/ide/alim15x3.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/ide/alim15x3.c +++ linux-4.4.25-rt35/drivers/ide/alim15x3.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:237 @ static int init_chipset_ali15x3(struct p isa_dev = pci_get_device(PCI_VENDOR_ID_AL, PCI_DEVICE_ID_AL_M1533, NULL); - local_irq_save(flags); + local_irq_save_nort(flags); if (m5229_revision < 0xC2) { /* @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:328 @ out: } pci_dev_put(north); pci_dev_put(isa_dev); - local_irq_restore(flags); + local_irq_restore_nort(flags); return 0; } Index: linux-4.4.25-rt35/drivers/ide/hpt366.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/ide/hpt366.c +++ linux-4.4.25-rt35/drivers/ide/hpt366.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1244 @ static int init_dma_hpt366(ide_hwif_t *h dma_old = inb(base + 2); - local_irq_save(flags); + local_irq_save_nort(flags); dma_new = dma_old; pci_read_config_byte(dev, hwif->channel ? 0x4b : 0x43, &masterdma); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1255 @ static int init_dma_hpt366(ide_hwif_t *h if (dma_new != dma_old) outb(dma_new, base + 2); - local_irq_restore(flags); + local_irq_restore_nort(flags); printk(KERN_INFO " %s: BM-DMA at 0x%04lx-0x%04lx\n", hwif->name, base, base + 7); Index: linux-4.4.25-rt35/drivers/ide/ide-io-std.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/ide/ide-io-std.c +++ linux-4.4.25-rt35/drivers/ide/ide-io-std.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:178 @ void ide_input_data(ide_drive_t *drive, unsigned long uninitialized_var(flags); if ((io_32bit & 2) && !mmio) { - local_irq_save(flags); + local_irq_save_nort(flags); ata_vlb_sync(io_ports->nsect_addr); } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:189 @ void ide_input_data(ide_drive_t *drive, insl(data_addr, buf, words); if ((io_32bit & 2) && !mmio) - local_irq_restore(flags); + local_irq_restore_nort(flags); if (((len + 1) & 3) < 2) return; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:222 @ void ide_output_data(ide_drive_t *drive, unsigned long uninitialized_var(flags); if ((io_32bit & 2) && !mmio) { - local_irq_save(flags); + local_irq_save_nort(flags); ata_vlb_sync(io_ports->nsect_addr); } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:233 @ void ide_output_data(ide_drive_t *drive, outsl(data_addr, buf, words); if ((io_32bit & 2) && !mmio) - local_irq_restore(flags); + local_irq_restore_nort(flags); if (((len + 1) & 3) < 2) return; Index: linux-4.4.25-rt35/drivers/ide/ide-io.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/ide/ide-io.c +++ linux-4.4.25-rt35/drivers/ide/ide-io.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:662 @ void ide_timer_expiry (unsigned long dat /* disable_irq_nosync ?? */ disable_irq(hwif->irq); /* local CPU only, as if we were handling an interrupt */ - local_irq_disable(); + local_irq_disable_nort(); if (hwif->polling) { startstop = handler(drive); } else if (drive_is_ready(drive)) { Index: linux-4.4.25-rt35/drivers/ide/ide-iops.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/ide/ide-iops.c +++ linux-4.4.25-rt35/drivers/ide/ide-iops.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:132 @ int __ide_wait_stat(ide_drive_t *drive, if ((stat & ATA_BUSY) == 0) break; - local_irq_restore(flags); + local_irq_restore_nort(flags); *rstat = stat; return -EBUSY; } } - local_irq_restore(flags); + local_irq_restore_nort(flags); } /* * Allow status to settle, then read it again. Index: linux-4.4.25-rt35/drivers/ide/ide-probe.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/ide/ide-probe.c +++ linux-4.4.25-rt35/drivers/ide/ide-probe.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:199 @ static void do_identify(ide_drive_t *dri int bswap = 1; /* local CPU only; some systems need this */ - local_irq_save(flags); + local_irq_save_nort(flags); /* read 512 bytes of id info */ hwif->tp_ops->input_data(drive, NULL, id, SECTOR_SIZE); - local_irq_restore(flags); + local_irq_restore_nort(flags); drive->dev_flags |= IDE_DFLAG_ID_READ; #ifdef DEBUG Index: linux-4.4.25-rt35/drivers/ide/ide-taskfile.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/ide/ide-taskfile.c +++ linux-4.4.25-rt35/drivers/ide/ide-taskfile.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:253 @ void ide_pio_bytes(ide_drive_t *drive, s page_is_high = PageHighMem(page); if (page_is_high) - local_irq_save(flags); + local_irq_save_nort(flags); buf = kmap_atomic(page) + offset; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:274 @ void ide_pio_bytes(ide_drive_t *drive, s kunmap_atomic(buf); if (page_is_high) - local_irq_restore(flags); + local_irq_restore_nort(flags); len -= nr_bytes; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:417 @ static ide_startstop_t pre_task_out_intr } if ((drive->dev_flags & IDE_DFLAG_UNMASK) == 0) - local_irq_disable(); + local_irq_disable_nort(); ide_set_handler(drive, &task_pio_intr, WAIT_WORSTCASE); Index: linux-4.4.25-rt35/drivers/infiniband/ulp/ipoib/ipoib_multicast.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/infiniband/ulp/ipoib/ipoib_multicast.c +++ linux-4.4.25-rt35/drivers/infiniband/ulp/ipoib/ipoib_multicast.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:860 @ void ipoib_mcast_restart_task(struct wor ipoib_dbg_mcast(priv, "restarting multicast task\n"); - local_irq_save(flags); + local_irq_save_nort(flags); netif_addr_lock(dev); spin_lock(&priv->lock); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:942 @ void ipoib_mcast_restart_task(struct wor spin_unlock(&priv->lock); netif_addr_unlock(dev); - local_irq_restore(flags); + local_irq_restore_nort(flags); /* * make sure the in-flight joins have finished before we attempt Index: linux-4.4.25-rt35/drivers/input/gameport/gameport.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/input/gameport/gameport.c +++ linux-4.4.25-rt35/drivers/input/gameport/gameport.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:94 @ static int gameport_measure_speed(struct tx = ~0; for (i = 0; i < 50; i++) { - local_irq_save(flags); + local_irq_save_nort(flags); t1 = ktime_get_ns(); for (t = 0; t < 50; t++) gameport_read(gameport); t2 = ktime_get_ns(); t3 = ktime_get_ns(); - local_irq_restore(flags); + local_irq_restore_nort(flags); udelay(i * 10); t = (t2 - t1) - (t3 - t2); if (t < tx) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:127 @ static int old_gameport_measure_speed(st tx = 1 << 30; for(i = 0; i < 50; i++) { - local_irq_save(flags); + local_irq_save_nort(flags); GET_TIME(t1); for (t = 0; t < 50; t++) gameport_read(gameport); GET_TIME(t2); GET_TIME(t3); - local_irq_restore(flags); + local_irq_restore_nort(flags); udelay(i * 10); if ((t = DELTA(t2,t1) - DELTA(t3,t2)) < tx) tx = t; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:151 @ static int old_gameport_measure_speed(st tx = 1 << 30; for(i = 0; i < 50; i++) { - local_irq_save(flags); + local_irq_save_nort(flags); t1 = rdtsc(); for (t = 0; t < 50; t++) gameport_read(gameport); t2 = rdtsc(); - local_irq_restore(flags); + local_irq_restore_nort(flags); udelay(i * 10); if (t2 - t1 < tx) tx = t2 - t1; } Index: linux-4.4.25-rt35/drivers/iommu/amd_iommu.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/iommu/amd_iommu.c +++ linux-4.4.25-rt35/drivers/iommu/amd_iommu.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2025 @ static int __attach_device(struct iommu_ int ret; /* - * Must be called with IRQs disabled. Warn here to detect early - * when its not. + * Must be called with IRQs disabled on a non RT kernel. Warn here to + * detect early when its not. */ - WARN_ON(!irqs_disabled()); + WARN_ON_NONRT(!irqs_disabled()); /* lock domain */ spin_lock(&domain->lock); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2191 @ static void __detach_device(struct iommu struct protection_domain *domain; /* - * Must be called with IRQs disabled. Warn here to detect early - * when its not. + * Must be called with IRQs disabled on a non RT kernel. Warn here to + * detect early when its not. */ - WARN_ON(!irqs_disabled()); + WARN_ON_NONRT(!irqs_disabled()); if (WARN_ON(!dev_data->domain)) return; Index: linux-4.4.25-rt35/drivers/leds/trigger/Kconfig =================================================================== --- linux-4.4.25-rt35.orig/drivers/leds/trigger/Kconfig +++ linux-4.4.25-rt35/drivers/leds/trigger/Kconfig @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:64 @ config LEDS_TRIGGER_BACKLIGHT config LEDS_TRIGGER_CPU bool "LED CPU Trigger" - depends on LEDS_TRIGGERS + depends on LEDS_TRIGGERS && !PREEMPT_RT_BASE help This allows LEDs to be controlled by active CPUs. This shows the active CPUs across an array of LEDs so you can see which Index: linux-4.4.25-rt35/drivers/md/bcache/Kconfig =================================================================== --- linux-4.4.25-rt35.orig/drivers/md/bcache/Kconfig +++ linux-4.4.25-rt35/drivers/md/bcache/Kconfig @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ config BCACHE tristate "Block device as cache" + depends on !PREEMPT_RT_FULL ---help--- Allows a block device to be used as cache for other devices; uses a btree for indexing and the layout is optimized for SSDs. Index: linux-4.4.25-rt35/drivers/md/dm.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/md/dm.c +++ linux-4.4.25-rt35/drivers/md/dm.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2130 @ static void dm_request_fn(struct request /* Establish tio->ti before queuing work (map_tio_request) */ tio->ti = ti; queue_kthread_work(&md->kworker, &tio->work); - BUG_ON(!irqs_disabled()); + BUG_ON_NONRT(!irqs_disabled()); } goto out; Index: linux-4.4.25-rt35/drivers/md/raid5.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/md/raid5.c +++ linux-4.4.25-rt35/drivers/md/raid5.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1923 @ static void raid_run_ops(struct stripe_h struct raid5_percpu *percpu; unsigned long cpu; - cpu = get_cpu(); + cpu = get_cpu_light(); percpu = per_cpu_ptr(conf->percpu, cpu); + spin_lock(&percpu->lock); if (test_bit(STRIPE_OP_BIOFILL, &ops_request)) { ops_run_biofill(sh); overlap_clear++; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1981 @ static void raid_run_ops(struct stripe_h if (test_and_clear_bit(R5_Overlap, &dev->flags)) wake_up(&sh->raid_conf->wait_for_overlap); } - put_cpu(); + spin_unlock(&percpu->lock); + put_cpu_light(); } static struct stripe_head *alloc_stripe(struct kmem_cache *sc, gfp_t gfp) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:6419 @ static int raid5_alloc_percpu(struct r5c __func__, cpu); break; } + spin_lock_init(&per_cpu_ptr(conf->percpu, cpu)->lock); } put_online_cpus(); Index: linux-4.4.25-rt35/drivers/md/raid5.h =================================================================== --- linux-4.4.25-rt35.orig/drivers/md/raid5.h +++ linux-4.4.25-rt35/drivers/md/raid5.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:507 @ struct r5conf { int recovery_disabled; /* per cpu variables */ struct raid5_percpu { + spinlock_t lock; /* Protection for -RT */ struct page *spare_page; /* Used when checking P/Q in raid6 */ struct flex_array *scribble; /* space for constructing buffer * lists and performing address Index: linux-4.4.25-rt35/drivers/media/platform/vsp1/vsp1_video.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/media/platform/vsp1/vsp1_video.c +++ linux-4.4.25-rt35/drivers/media/platform/vsp1/vsp1_video.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:523 @ static bool vsp1_pipeline_stopped(struct bool stopped; spin_lock_irqsave(&pipe->irqlock, flags); - stopped = pipe->state == VSP1_PIPELINE_STOPPED, + stopped = pipe->state == VSP1_PIPELINE_STOPPED; spin_unlock_irqrestore(&pipe->irqlock, flags); return stopped; Index: linux-4.4.25-rt35/drivers/misc/Kconfig =================================================================== --- linux-4.4.25-rt35.orig/drivers/misc/Kconfig +++ linux-4.4.25-rt35/drivers/misc/Kconfig @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:57 @ config AD525X_DPOT_SPI config ATMEL_TCLIB bool "Atmel AT32/AT91 Timer/Counter Library" depends on (AVR32 || ARCH_AT91) + default y if PREEMPT_RT_FULL help Select this if you want a library to allocate the Timer/Counter blocks found on many Atmel processors. This facilitates using @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:73 @ config ATMEL_TCB_CLKSRC are combined to make a single 32-bit timer. When GENERIC_CLOCKEVENTS is defined, the third timer channel - may be used as a clock event device supporting oneshot mode - (delays of up to two seconds) based on the 32 KiHz clock. + may be used as a clock event device supporting oneshot mode. config ATMEL_TCB_CLKSRC_BLOCK int @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:87 @ config ATMEL_TCB_CLKSRC_BLOCK TC can be used for other purposes, such as PWM generation and interval timing. +config ATMEL_TCB_CLKSRC_USE_SLOW_CLOCK + bool "TC Block use 32 KiHz clock" + depends on ATMEL_TCB_CLKSRC + default y if !PREEMPT_RT_FULL + help + Select this to use 32 KiHz base clock rate as TC block clock + source for clock events. + + config DUMMY_IRQ tristate "Dummy IRQ handler" default n @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:125 @ config IBM_ASM for information on the specific driver level and support statement for your IBM server. +config HWLAT_DETECTOR + tristate "Testing module to detect hardware-induced latencies" + depends on DEBUG_FS + depends on RING_BUFFER + default m + ---help--- + A simple hardware latency detector. Use this module to detect + large latencies introduced by the behavior of the underlying + system firmware external to Linux. We do this using periodic + use of stop_machine to grab all available CPUs and measure + for unexplainable gaps in the CPU timestamp counter(s). By + default, the module is not enabled until the "enable" file + within the "hwlat_detector" debugfs directory is toggled. + + This module is often used to detect SMI (System Management + Interrupts) on x86 systems, though is not x86 specific. To + this end, we default to using a sample window of 1 second, + during which we will sample for 0.5 seconds. If an SMI or + similar event occurs during that time, it is recorded + into an 8K samples global ring buffer until retreived. + + WARNING: This software should never be enabled (it can be built + but should not be turned on after it is loaded) in a production + environment where high latencies are a concern since the + sampling mechanism actually introduces latencies for + regular tasks while the CPU(s) are being held. + + If unsure, say N + config PHANTOM tristate "Sensable PHANToM (PCI)" depends on PCI Index: linux-4.4.25-rt35/drivers/misc/Makefile =================================================================== --- linux-4.4.25-rt35.orig/drivers/misc/Makefile +++ linux-4.4.25-rt35/drivers/misc/Makefile @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:42 @ obj-$(CONFIG_C2PORT) += c2port/ obj-$(CONFIG_HMC6352) += hmc6352.o obj-y += eeprom/ obj-y += cb710/ +obj-$(CONFIG_HWLAT_DETECTOR) += hwlat_detector.o obj-$(CONFIG_SPEAR13XX_PCIE_GADGET) += spear13xx_pcie_gadget.o obj-$(CONFIG_VMWARE_BALLOON) += vmw_balloon.o obj-$(CONFIG_ARM_CHARLCD) += arm-charlcd.o Index: linux-4.4.25-rt35/drivers/misc/hwlat_detector.c =================================================================== --- /dev/null +++ linux-4.4.25-rt35/drivers/misc/hwlat_detector.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +/* + * hwlat_detector.c - A simple Hardware Latency detector. + * + * Use this module to detect large system latencies induced by the behavior of + * certain underlying system hardware or firmware, independent of Linux itself. + * The code was developed originally to detect the presence of SMIs on Intel + * and AMD systems, although there is no dependency upon x86 herein. + * + * The classical example usage of this module is in detecting the presence of + * SMIs or System Management Interrupts on Intel and AMD systems. An SMI is a + * somewhat special form of hardware interrupt spawned from earlier CPU debug + * modes in which the (BIOS/EFI/etc.) firmware arranges for the South Bridge + * LPC (or other device) to generate a special interrupt under certain + * circumstances, for example, upon expiration of a special SMI timer device, + * due to certain external thermal readings, on certain I/O address accesses, + * and other situations. An SMI hits a special CPU pin, triggers a special + * SMI mode (complete with special memory map), and the OS is unaware. + * + * Although certain hardware-inducing latencies are necessary (for example, + * a modern system often requires an SMI handler for correct thermal control + * and remote management) they can wreak havoc upon any OS-level performance + * guarantees toward low-latency, especially when the OS is not even made + * aware of the presence of these interrupts. For this reason, we need a + * somewhat brute force mechanism to detect these interrupts. In this case, + * we do it by hogging all of the CPU(s) for configurable timer intervals, + * sampling the built-in CPU timer, looking for discontiguous readings. + * + * WARNING: This implementation necessarily introduces latencies. Therefore, + * you should NEVER use this module in a production environment + * requiring any kind of low-latency performance guarantee(s). + * + * Copyright (C) 2008-2009 Jon Masters, Red Hat, Inc. <jcm@redhat.com> + * + * Includes useful feedback from Clark Williams <clark@redhat.com> + * + * This file is licensed under the terms of the GNU General Public + * License version 2. This program is licensed "as is" without any + * warranty of any kind, whether express or implied. + */ + +#include <linux/module.h> +#include <linux/init.h> +#include <linux/ring_buffer.h> +#include <linux/time.h> +#include <linux/hrtimer.h> +#include <linux/kthread.h> +#include <linux/debugfs.h> +#include <linux/seq_file.h> +#include <linux/uaccess.h> +#include <linux/version.h> +#include <linux/delay.h> +#include <linux/slab.h> +#include <linux/trace_clock.h> + +#define BUF_SIZE_DEFAULT 262144UL /* 8K*(sizeof(entry)) */ +#define BUF_FLAGS (RB_FL_OVERWRITE) /* no block on full */ +#define U64STR_SIZE 22 /* 20 digits max */ + +#define VERSION "1.0.0" +#define BANNER "hwlat_detector: " +#define DRVNAME "hwlat_detector" +#define DEFAULT_SAMPLE_WINDOW 1000000 /* 1s */ +#define DEFAULT_SAMPLE_WIDTH 500000 /* 0.5s */ +#define DEFAULT_LAT_THRESHOLD 10 /* 10us */ + +/* Module metadata */ + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Jon Masters <jcm@redhat.com>"); +MODULE_DESCRIPTION("A simple hardware latency detector"); +MODULE_VERSION(VERSION); + +/* Module parameters */ + +static int debug; +static int enabled; +static int threshold; + +module_param(debug, int, 0); /* enable debug */ +module_param(enabled, int, 0); /* enable detector */ +module_param(threshold, int, 0); /* latency threshold */ + +/* Buffering and sampling */ + +static struct ring_buffer *ring_buffer; /* sample buffer */ +static DEFINE_MUTEX(ring_buffer_mutex); /* lock changes */ +static unsigned long buf_size = BUF_SIZE_DEFAULT; +static struct task_struct *kthread; /* sampling thread */ + +/* DebugFS filesystem entries */ + +static struct dentry *debug_dir; /* debugfs directory */ +static struct dentry *debug_max; /* maximum TSC delta */ +static struct dentry *debug_count; /* total detect count */ +static struct dentry *debug_sample_width; /* sample width us */ +static struct dentry *debug_sample_window; /* sample window us */ +static struct dentry *debug_sample; /* raw samples us */ +static struct dentry *debug_threshold; /* threshold us */ +static struct dentry *debug_enable; /* enable/disable */ + +/* Individual samples and global state */ + +struct sample; /* latency sample */ +struct data; /* Global state */ + +/* Sampling functions */ +static int __buffer_add_sample(struct sample *sample); +static struct sample *buffer_get_sample(struct sample *sample); + +/* Threading and state */ +static int kthread_fn(void *unused); +static int start_kthread(void); +static int stop_kthread(void); +static void __reset_stats(void); +static int init_stats(void); + +/* Debugfs interface */ +static ssize_t simple_data_read(struct file *filp, char __user *ubuf, + size_t cnt, loff_t *ppos, const u64 *entry); +static ssize_t simple_data_write(struct file *filp, const char __user *ubuf, + size_t cnt, loff_t *ppos, u64 *entry); +static int debug_sample_fopen(struct inode *inode, struct file *filp); +static ssize_t debug_sample_fread(struct file *filp, char __user *ubuf, + size_t cnt, loff_t *ppos); +static int debug_sample_release(struct inode *inode, struct file *filp); +static int debug_enable_fopen(struct inode *inode, struct file *filp); +static ssize_t debug_enable_fread(struct file *filp, char __user *ubuf, + size_t cnt, loff_t *ppos); +static ssize_t debug_enable_fwrite(struct file *file, + const char __user *user_buffer, + size_t user_size, loff_t *offset); + +/* Initialization functions */ +static int init_debugfs(void); +static void free_debugfs(void); +static int detector_init(void); +static void detector_exit(void); + +/* Individual latency samples are stored here when detected and packed into + * the ring_buffer circular buffer, where they are overwritten when + * more than buf_size/sizeof(sample) samples are received. */ +struct sample { + u64 seqnum; /* unique sequence */ + u64 duration; /* ktime delta */ + u64 outer_duration; /* ktime delta (outer loop) */ + struct timespec timestamp; /* wall time */ + unsigned long lost; +}; + +/* keep the global state somewhere. */ +static struct data { + + struct mutex lock; /* protect changes */ + + u64 count; /* total since reset */ + u64 max_sample; /* max hardware latency */ + u64 threshold; /* sample threshold level */ + + u64 sample_window; /* total sampling window (on+off) */ + u64 sample_width; /* active sampling portion of window */ + + atomic_t sample_open; /* whether the sample file is open */ + + wait_queue_head_t wq; /* waitqeue for new sample values */ + +} data; + +/** + * __buffer_add_sample - add a new latency sample recording to the ring buffer + * @sample: The new latency sample value + * + * This receives a new latency sample and records it in a global ring buffer. + * No additional locking is used in this case. + */ +static int __buffer_add_sample(struct sample *sample) +{ + return ring_buffer_write(ring_buffer, + sizeof(struct sample), sample); +} + +/** + * buffer_get_sample - remove a hardware latency sample from the ring buffer + * @sample: Pre-allocated storage for the sample + * + * This retrieves a hardware latency sample from the global circular buffer + */ +static struct sample *buffer_get_sample(struct sample *sample) +{ + struct ring_buffer_event *e = NULL; + struct sample *s = NULL; + unsigned int cpu = 0; + + if (!sample) + return NULL; + + mutex_lock(&ring_buffer_mutex); + for_each_online_cpu(cpu) { + e = ring_buffer_consume(ring_buffer, cpu, NULL, &sample->lost); + if (e) + break; + } + + if (e) { + s = ring_buffer_event_data(e); + memcpy(sample, s, sizeof(struct sample)); + } else + sample = NULL; + mutex_unlock(&ring_buffer_mutex); + + return sample; +} + +#ifndef CONFIG_TRACING +#define time_type ktime_t +#define time_get() ktime_get() +#define time_to_us(x) ktime_to_us(x) +#define time_sub(a, b) ktime_sub(a, b) +#define init_time(a, b) (a).tv64 = b +#define time_u64(a) ((a).tv64) +#else +#define time_type u64 +#define time_get() trace_clock_local() +#define time_to_us(x) div_u64(x, 1000) +#define time_sub(a, b) ((a) - (b)) +#define init_time(a, b) (a = b) +#define time_u64(a) a +#endif +/** + * get_sample - sample the CPU TSC and look for likely hardware latencies + * + * Used to repeatedly capture the CPU TSC (or similar), looking for potential + * hardware-induced latency. Called with interrupts disabled and with + * data.lock held. + */ +static int get_sample(void) +{ + time_type start, t1, t2, last_t2; + s64 diff, total = 0; + u64 sample = 0; + u64 outer_sample = 0; + int ret = -1; + + init_time(last_t2, 0); + start = time_get(); /* start timestamp */ + + do { + + t1 = time_get(); /* we'll look for a discontinuity */ + t2 = time_get(); + + if (time_u64(last_t2)) { + /* Check the delta from outer loop (t2 to next t1) */ + diff = time_to_us(time_sub(t1, last_t2)); + /* This shouldn't happen */ + if (diff < 0) { + pr_err(BANNER "time running backwards\n"); + goto out; + } + if (diff > outer_sample) + outer_sample = diff; + } + last_t2 = t2; + + total = time_to_us(time_sub(t2, start)); /* sample width */ + + /* This checks the inner loop (t1 to t2) */ + diff = time_to_us(time_sub(t2, t1)); /* current diff */ + + /* This shouldn't happen */ + if (diff < 0) { + pr_err(BANNER "time running backwards\n"); + goto out; + } + + if (diff > sample) + sample = diff; /* only want highest value */ + + } while (total <= data.sample_width); + + ret = 0; + + /* If we exceed the threshold value, we have found a hardware latency */ + if (sample > data.threshold || outer_sample > data.threshold) { + struct sample s; + + ret = 1; + + data.count++; + s.seqnum = data.count; + s.duration = sample; + s.outer_duration = outer_sample; + s.timestamp = CURRENT_TIME; + __buffer_add_sample(&s); + + /* Keep a running maximum ever recorded hardware latency */ + if (sample > data.max_sample) + data.max_sample = sample; + } + +out: + return ret; +} + +/* + * kthread_fn - The CPU time sampling/hardware latency detection kernel thread + * @unused: A required part of the kthread API. + * + * Used to periodically sample the CPU TSC via a call to get_sample. We + * disable interrupts, which does (intentionally) introduce latency since we + * need to ensure nothing else might be running (and thus pre-empting). + * Obviously this should never be used in production environments. + * + * Currently this runs on which ever CPU it was scheduled on, but most + * real-worald hardware latency situations occur across several CPUs, + * but we might later generalize this if we find there are any actualy + * systems with alternate SMI delivery or other hardware latencies. + */ +static int kthread_fn(void *unused) +{ + int ret; + u64 interval; + + while (!kthread_should_stop()) { + + mutex_lock(&data.lock); + + local_irq_disable(); + ret = get_sample(); + local_irq_enable(); + + if (ret > 0) + wake_up(&data.wq); /* wake up reader(s) */ + + interval = data.sample_window - data.sample_width; + do_div(interval, USEC_PER_MSEC); /* modifies interval value */ + + mutex_unlock(&data.lock); + + if (msleep_interruptible(interval)) + break; + } + + return 0; +} + +/** + * start_kthread - Kick off the hardware latency sampling/detector kthread + * + * This starts a kernel thread that will sit and sample the CPU timestamp + * counter (TSC or similar) and look for potential hardware latencies. + */ +static int start_kthread(void) +{ + kthread = kthread_run(kthread_fn, NULL, + DRVNAME); + if (IS_ERR(kthread)) { + pr_err(BANNER "could not start sampling thread\n"); + enabled = 0; + return -ENOMEM; + } + + return 0; +} + +/** + * stop_kthread - Inform the hardware latency samping/detector kthread to stop + * + * This kicks the running hardware latency sampling/detector kernel thread and + * tells it to stop sampling now. Use this on unload and at system shutdown. + */ +static int stop_kthread(void) +{ + int ret; + + ret = kthread_stop(kthread); + + return ret; +} + +/** + * __reset_stats - Reset statistics for the hardware latency detector + * + * We use data to store various statistics and global state. We call this + * function in order to reset those when "enable" is toggled on or off, and + * also at initialization. Should be called with data.lock held. + */ +static void __reset_stats(void) +{ + data.count = 0; + data.max_sample = 0; + ring_buffer_reset(ring_buffer); /* flush out old sample entries */ +} + +/** + * init_stats - Setup global state statistics for the hardware latency detector + * + * We use data to store various statistics and global state. We also use + * a global ring buffer (ring_buffer) to keep raw samples of detected hardware + * induced system latencies. This function initializes these structures and + * allocates the global ring buffer also. + */ +static int init_stats(void) +{ + int ret = -ENOMEM; + + mutex_init(&data.lock); + init_waitqueue_head(&data.wq); + atomic_set(&data.sample_open, 0); + + ring_buffer = ring_buffer_alloc(buf_size, BUF_FLAGS); + + if (WARN(!ring_buffer, KERN_ERR BANNER + "failed to allocate ring buffer!\n")) + goto out; + + __reset_stats(); + data.threshold = threshold ?: DEFAULT_LAT_THRESHOLD; /* threshold us */ + data.sample_window = DEFAULT_SAMPLE_WINDOW; /* window us */ + data.sample_width = DEFAULT_SAMPLE_WIDTH; /* width us */ + + ret = 0; + +out: + return ret; + +} + +/* + * simple_data_read - Wrapper read function for global state debugfs entries + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The userspace provided buffer to read value into + * @cnt: The maximum number of bytes to read + * @ppos: The current "file" position + * @entry: The entry to read from + * + * This function provides a generic read implementation for the global state + * "data" structure debugfs filesystem entries. It would be nice to use + * simple_attr_read directly, but we need to make sure that the data.lock + * is held during the actual read. + */ +static ssize_t simple_data_read(struct file *filp, char __user *ubuf, + size_t cnt, loff_t *ppos, const u64 *entry) +{ + char buf[U64STR_SIZE]; + u64 val = 0; + int len = 0; + + memset(buf, 0, sizeof(buf)); + + if (!entry) + return -EFAULT; + + mutex_lock(&data.lock); + val = *entry; + mutex_unlock(&data.lock); + + len = snprintf(buf, sizeof(buf), "%llu\n", (unsigned long long)val); + + return simple_read_from_buffer(ubuf, cnt, ppos, buf, len); + +} + +/* + * simple_data_write - Wrapper write function for global state debugfs entries + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The userspace provided buffer to write value from + * @cnt: The maximum number of bytes to write + * @ppos: The current "file" position + * @entry: The entry to write to + * + * This function provides a generic write implementation for the global state + * "data" structure debugfs filesystem entries. It would be nice to use + * simple_attr_write directly, but we need to make sure that the data.lock + * is held during the actual write. + */ +static ssize_t simple_data_write(struct file *filp, const char __user *ubuf, + size_t cnt, loff_t *ppos, u64 *entry) +{ + char buf[U64STR_SIZE]; + int csize = min(cnt, sizeof(buf)); + u64 val = 0; + int err = 0; + + memset(buf, '\0', sizeof(buf)); + if (copy_from_user(buf, ubuf, csize)) + return -EFAULT; + + buf[U64STR_SIZE-1] = '\0'; /* just in case */ + err = kstrtoull(buf, 10, &val); + if (err) + return -EINVAL; + + mutex_lock(&data.lock); + *entry = val; + mutex_unlock(&data.lock); + + return csize; +} + +/** + * debug_count_fopen - Open function for "count" debugfs entry + * @inode: The in-kernel inode representation of the debugfs "file" + * @filp: The active open file structure for the debugfs "file" + * + * This function provides an open implementation for the "count" debugfs + * interface to the hardware latency detector. + */ +static int debug_count_fopen(struct inode *inode, struct file *filp) +{ + return 0; +} + +/** + * debug_count_fread - Read function for "count" debugfs entry + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The userspace provided buffer to read value into + * @cnt: The maximum number of bytes to read + * @ppos: The current "file" position + * + * This function provides a read implementation for the "count" debugfs + * interface to the hardware latency detector. Can be used to read the + * number of latency readings exceeding the configured threshold since + * the detector was last reset (e.g. by writing a zero into "count"). + */ +static ssize_t debug_count_fread(struct file *filp, char __user *ubuf, + size_t cnt, loff_t *ppos) +{ + return simple_data_read(filp, ubuf, cnt, ppos, &data.count); +} + +/** + * debug_count_fwrite - Write function for "count" debugfs entry + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The user buffer that contains the value to write + * @cnt: The maximum number of bytes to write to "file" + * @ppos: The current position in the debugfs "file" + * + * This function provides a write implementation for the "count" debugfs + * interface to the hardware latency detector. Can be used to write a + * desired value, especially to zero the total count. + */ +static ssize_t debug_count_fwrite(struct file *filp, + const char __user *ubuf, + size_t cnt, + loff_t *ppos) +{ + return simple_data_write(filp, ubuf, cnt, ppos, &data.count); +} + +/** + * debug_enable_fopen - Dummy open function for "enable" debugfs interface + * @inode: The in-kernel inode representation of the debugfs "file" + * @filp: The active open file structure for the debugfs "file" + * + * This function provides an open implementation for the "enable" debugfs + * interface to the hardware latency detector. + */ +static int debug_enable_fopen(struct inode *inode, struct file *filp) +{ + return 0; +} + +/** + * debug_enable_fread - Read function for "enable" debugfs interface + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The userspace provided buffer to read value into + * @cnt: The maximum number of bytes to read + * @ppos: The current "file" position + * + * This function provides a read implementation for the "enable" debugfs + * interface to the hardware latency detector. Can be used to determine + * whether the detector is currently enabled ("0\n" or "1\n" returned). + */ +static ssize_t debug_enable_fread(struct file *filp, char __user *ubuf, + size_t cnt, loff_t *ppos) +{ + char buf[4]; + + if ((cnt < sizeof(buf)) || (*ppos)) + return 0; + + buf[0] = enabled ? '1' : '0'; + buf[1] = '\n'; + buf[2] = '\0'; + if (copy_to_user(ubuf, buf, strlen(buf))) + return -EFAULT; + return *ppos = strlen(buf); +} + +/** + * debug_enable_fwrite - Write function for "enable" debugfs interface + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The user buffer that contains the value to write + * @cnt: The maximum number of bytes to write to "file" + * @ppos: The current position in the debugfs "file" + * + * This function provides a write implementation for the "enable" debugfs + * interface to the hardware latency detector. Can be used to enable or + * disable the detector, which will have the side-effect of possibly + * also resetting the global stats and kicking off the measuring + * kthread (on an enable) or the converse (upon a disable). + */ +static ssize_t debug_enable_fwrite(struct file *filp, + const char __user *ubuf, + size_t cnt, + loff_t *ppos) +{ + char buf[4]; + int csize = min(cnt, sizeof(buf)); + long val = 0; + int err = 0; + + memset(buf, '\0', sizeof(buf)); + if (copy_from_user(buf, ubuf, csize)) + return -EFAULT; + + buf[sizeof(buf)-1] = '\0'; /* just in case */ + err = kstrtoul(buf, 10, &val); + if (err) + return -EINVAL; + + if (val) { + if (enabled) + goto unlock; + enabled = 1; + __reset_stats(); + if (start_kthread()) + return -EFAULT; + } else { + if (!enabled) + goto unlock; + enabled = 0; + err = stop_kthread(); + if (err) { + pr_err(BANNER "cannot stop kthread\n"); + return -EFAULT; + } + wake_up(&data.wq); /* reader(s) should return */ + } +unlock: + return csize; +} + +/** + * debug_max_fopen - Open function for "max" debugfs entry + * @inode: The in-kernel inode representation of the debugfs "file" + * @filp: The active open file structure for the debugfs "file" + * + * This function provides an open implementation for the "max" debugfs + * interface to the hardware latency detector. + */ +static int debug_max_fopen(struct inode *inode, struct file *filp) +{ + return 0; +} + +/** + * debug_max_fread - Read function for "max" debugfs entry + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The userspace provided buffer to read value into + * @cnt: The maximum number of bytes to read + * @ppos: The current "file" position + * + * This function provides a read implementation for the "max" debugfs + * interface to the hardware latency detector. Can be used to determine + * the maximum latency value observed since it was last reset. + */ +static ssize_t debug_max_fread(struct file *filp, char __user *ubuf, + size_t cnt, loff_t *ppos) +{ + return simple_data_read(filp, ubuf, cnt, ppos, &data.max_sample); +} + +/** + * debug_max_fwrite - Write function for "max" debugfs entry + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The user buffer that contains the value to write + * @cnt: The maximum number of bytes to write to "file" + * @ppos: The current position in the debugfs "file" + * + * This function provides a write implementation for the "max" debugfs + * interface to the hardware latency detector. Can be used to reset the + * maximum or set it to some other desired value - if, then, subsequent + * measurements exceed this value, the maximum will be updated. + */ +static ssize_t debug_max_fwrite(struct file *filp, + const char __user *ubuf, + size_t cnt, + loff_t *ppos) +{ + return simple_data_write(filp, ubuf, cnt, ppos, &data.max_sample); +} + + +/** + * debug_sample_fopen - An open function for "sample" debugfs interface + * @inode: The in-kernel inode representation of this debugfs "file" + * @filp: The active open file structure for the debugfs "file" + * + * This function handles opening the "sample" file within the hardware + * latency detector debugfs directory interface. This file is used to read + * raw samples from the global ring_buffer and allows the user to see a + * running latency history. Can be opened blocking or non-blocking, + * affecting whether it behaves as a buffer read pipe, or does not. + * Implements simple locking to prevent multiple simultaneous use. + */ +static int debug_sample_fopen(struct inode *inode, struct file *filp) +{ + if (!atomic_add_unless(&data.sample_open, 1, 1)) + return -EBUSY; + else + return 0; +} + +/** + * debug_sample_fread - A read function for "sample" debugfs interface + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The user buffer that will contain the samples read + * @cnt: The maximum bytes to read from the debugfs "file" + * @ppos: The current position in the debugfs "file" + * + * This function handles reading from the "sample" file within the hardware + * latency detector debugfs directory interface. This file is used to read + * raw samples from the global ring_buffer and allows the user to see a + * running latency history. By default this will block pending a new + * value written into the sample buffer, unless there are already a + * number of value(s) waiting in the buffer, or the sample file was + * previously opened in a non-blocking mode of operation. + */ +static ssize_t debug_sample_fread(struct file *filp, char __user *ubuf, + size_t cnt, loff_t *ppos) +{ + int len = 0; + char buf[64]; + struct sample *sample = NULL; + + if (!enabled) + return 0; + + sample = kzalloc(sizeof(struct sample), GFP_KERNEL); + if (!sample) + return -ENOMEM; + + while (!buffer_get_sample(sample)) { + + DEFINE_WAIT(wait); + + if (filp->f_flags & O_NONBLOCK) { + len = -EAGAIN; + goto out; + } + + prepare_to_wait(&data.wq, &wait, TASK_INTERRUPTIBLE); + schedule(); + finish_wait(&data.wq, &wait); + + if (signal_pending(current)) { + len = -EINTR; + goto out; + } + + if (!enabled) { /* enable was toggled */ + len = 0; + goto out; + } + } + + len = snprintf(buf, sizeof(buf), "%010lu.%010lu\t%llu\t%llu\n", + sample->timestamp.tv_sec, + sample->timestamp.tv_nsec, + sample->duration, + sample->outer_duration); + + + /* handling partial reads is more trouble than it's worth */ + if (len > cnt) + goto out; + + if (copy_to_user(ubuf, buf, len)) + len = -EFAULT; + +out: + kfree(sample); + return len; +} + +/** + * debug_sample_release - Release function for "sample" debugfs interface + * @inode: The in-kernel inode represenation of the debugfs "file" + * @filp: The active open file structure for the debugfs "file" + * + * This function completes the close of the debugfs interface "sample" file. + * Frees the sample_open "lock" so that other users may open the interface. + */ +static int debug_sample_release(struct inode *inode, struct file *filp) +{ + atomic_dec(&data.sample_open); + + return 0; +} + +/** + * debug_threshold_fopen - Open function for "threshold" debugfs entry + * @inode: The in-kernel inode representation of the debugfs "file" + * @filp: The active open file structure for the debugfs "file" + * + * This function provides an open implementation for the "threshold" debugfs + * interface to the hardware latency detector. + */ +static int debug_threshold_fopen(struct inode *inode, struct file *filp) +{ + return 0; +} + +/** + * debug_threshold_fread - Read function for "threshold" debugfs entry + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The userspace provided buffer to read value into + * @cnt: The maximum number of bytes to read + * @ppos: The current "file" position + * + * This function provides a read implementation for the "threshold" debugfs + * interface to the hardware latency detector. It can be used to determine + * the current threshold level at which a latency will be recorded in the + * global ring buffer, typically on the order of 10us. + */ +static ssize_t debug_threshold_fread(struct file *filp, char __user *ubuf, + size_t cnt, loff_t *ppos) +{ + return simple_data_read(filp, ubuf, cnt, ppos, &data.threshold); +} + +/** + * debug_threshold_fwrite - Write function for "threshold" debugfs entry + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The user buffer that contains the value to write + * @cnt: The maximum number of bytes to write to "file" + * @ppos: The current position in the debugfs "file" + * + * This function provides a write implementation for the "threshold" debugfs + * interface to the hardware latency detector. It can be used to configure + * the threshold level at which any subsequently detected latencies will + * be recorded into the global ring buffer. + */ +static ssize_t debug_threshold_fwrite(struct file *filp, + const char __user *ubuf, + size_t cnt, + loff_t *ppos) +{ + int ret; + + ret = simple_data_write(filp, ubuf, cnt, ppos, &data.threshold); + + if (enabled) + wake_up_process(kthread); + + return ret; +} + +/** + * debug_width_fopen - Open function for "width" debugfs entry + * @inode: The in-kernel inode representation of the debugfs "file" + * @filp: The active open file structure for the debugfs "file" + * + * This function provides an open implementation for the "width" debugfs + * interface to the hardware latency detector. + */ +static int debug_width_fopen(struct inode *inode, struct file *filp) +{ + return 0; +} + +/** + * debug_width_fread - Read function for "width" debugfs entry + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The userspace provided buffer to read value into + * @cnt: The maximum number of bytes to read + * @ppos: The current "file" position + * + * This function provides a read implementation for the "width" debugfs + * interface to the hardware latency detector. It can be used to determine + * for how many us of the total window us we will actively sample for any + * hardware-induced latecy periods. Obviously, it is not possible to + * sample constantly and have the system respond to a sample reader, or, + * worse, without having the system appear to have gone out to lunch. + */ +static ssize_t debug_width_fread(struct file *filp, char __user *ubuf, + size_t cnt, loff_t *ppos) +{ + return simple_data_read(filp, ubuf, cnt, ppos, &data.sample_width); +} + +/** + * debug_width_fwrite - Write function for "width" debugfs entry + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The user buffer that contains the value to write + * @cnt: The maximum number of bytes to write to "file" + * @ppos: The current position in the debugfs "file" + * + * This function provides a write implementation for the "width" debugfs + * interface to the hardware latency detector. It can be used to configure + * for how many us of the total window us we will actively sample for any + * hardware-induced latency periods. Obviously, it is not possible to + * sample constantly and have the system respond to a sample reader, or, + * worse, without having the system appear to have gone out to lunch. It + * is enforced that width is less that the total window size. + */ +static ssize_t debug_width_fwrite(struct file *filp, + const char __user *ubuf, + size_t cnt, + loff_t *ppos) +{ + char buf[U64STR_SIZE]; + int csize = min(cnt, sizeof(buf)); + u64 val = 0; + int err = 0; + + memset(buf, '\0', sizeof(buf)); + if (copy_from_user(buf, ubuf, csize)) + return -EFAULT; + + buf[U64STR_SIZE-1] = '\0'; /* just in case */ + err = kstrtoull(buf, 10, &val); + if (err) + return -EINVAL; + + mutex_lock(&data.lock); + if (val < data.sample_window) + data.sample_width = val; + else { + mutex_unlock(&data.lock); + return -EINVAL; + } + mutex_unlock(&data.lock); + + if (enabled) + wake_up_process(kthread); + + return csize; +} + +/** + * debug_window_fopen - Open function for "window" debugfs entry + * @inode: The in-kernel inode representation of the debugfs "file" + * @filp: The active open file structure for the debugfs "file" + * + * This function provides an open implementation for the "window" debugfs + * interface to the hardware latency detector. The window is the total time + * in us that will be considered one sample period. Conceptually, windows + * occur back-to-back and contain a sample width period during which + * actual sampling occurs. + */ +static int debug_window_fopen(struct inode *inode, struct file *filp) +{ + return 0; +} + +/** + * debug_window_fread - Read function for "window" debugfs entry + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The userspace provided buffer to read value into + * @cnt: The maximum number of bytes to read + * @ppos: The current "file" position + * + * This function provides a read implementation for the "window" debugfs + * interface to the hardware latency detector. The window is the total time + * in us that will be considered one sample period. Conceptually, windows + * occur back-to-back and contain a sample width period during which + * actual sampling occurs. Can be used to read the total window size. + */ +static ssize_t debug_window_fread(struct file *filp, char __user *ubuf, + size_t cnt, loff_t *ppos) +{ + return simple_data_read(filp, ubuf, cnt, ppos, &data.sample_window); +} + +/** + * debug_window_fwrite - Write function for "window" debugfs entry + * @filp: The active open file structure for the debugfs "file" + * @ubuf: The user buffer that contains the value to write + * @cnt: The maximum number of bytes to write to "file" + * @ppos: The current position in the debugfs "file" + * + * This function provides a write implementation for the "window" debufds + * interface to the hardware latency detetector. The window is the total time + * in us that will be considered one sample period. Conceptually, windows + * occur back-to-back and contain a sample width period during which + * actual sampling occurs. Can be used to write a new total window size. It + * is enfoced that any value written must be greater than the sample width + * size, or an error results. + */ +static ssize_t debug_window_fwrite(struct file *filp, + const char __user *ubuf, + size_t cnt, + loff_t *ppos) +{ + char buf[U64STR_SIZE]; + int csize = min(cnt, sizeof(buf)); + u64 val = 0; + int err = 0; + + memset(buf, '\0', sizeof(buf)); + if (copy_from_user(buf, ubuf, csize)) + return -EFAULT; + + buf[U64STR_SIZE-1] = '\0'; /* just in case */ + err = kstrtoull(buf, 10, &val); + if (err) + return -EINVAL; + + mutex_lock(&data.lock); + if (data.sample_width < val) + data.sample_window = val; + else { + mutex_unlock(&data.lock); + return -EINVAL; + } + mutex_unlock(&data.lock); + + return csize; +} + +/* + * Function pointers for the "count" debugfs file operations + */ +static const struct file_operations count_fops = { + .open = debug_count_fopen, + .read = debug_count_fread, + .write = debug_count_fwrite, + .owner = THIS_MODULE, +}; + +/* + * Function pointers for the "enable" debugfs file operations + */ +static const struct file_operations enable_fops = { + .open = debug_enable_fopen, + .read = debug_enable_fread, + .write = debug_enable_fwrite, + .owner = THIS_MODULE, +}; + +/* + * Function pointers for the "max" debugfs file operations + */ +static const struct file_operations max_fops = { + .open = debug_max_fopen, + .read = debug_max_fread, + .write = debug_max_fwrite, + .owner = THIS_MODULE, +}; + +/* + * Function pointers for the "sample" debugfs file operations + */ +static const struct file_operations sample_fops = { + .open = debug_sample_fopen, + .read = debug_sample_fread, + .release = debug_sample_release, + .owner = THIS_MODULE, +}; + +/* + * Function pointers for the "threshold" debugfs file operations + */ +static const struct file_operations threshold_fops = { + .open = debug_threshold_fopen, + .read = debug_threshold_fread, + .write = debug_threshold_fwrite, + .owner = THIS_MODULE, +}; + +/* + * Function pointers for the "width" debugfs file operations + */ +static const struct file_operations width_fops = { + .open = debug_width_fopen, + .read = debug_width_fread, + .write = debug_width_fwrite, + .owner = THIS_MODULE, +}; + +/* + * Function pointers for the "window" debugfs file operations + */ +static const struct file_operations window_fops = { + .open = debug_window_fopen, + .read = debug_window_fread, + .write = debug_window_fwrite, + .owner = THIS_MODULE, +}; + +/** + * init_debugfs - A function to initialize the debugfs interface files + * + * This function creates entries in debugfs for "hwlat_detector", including + * files to read values from the detector, current samples, and the + * maximum sample that has been captured since the hardware latency + * dectector was started. + */ +static int init_debugfs(void) +{ + int ret = -ENOMEM; + + debug_dir = debugfs_create_dir(DRVNAME, NULL); + if (!debug_dir) + goto err_debug_dir; + + debug_sample = debugfs_create_file("sample", 0444, + debug_dir, NULL, + &sample_fops); + if (!debug_sample) + goto err_sample; + + debug_count = debugfs_create_file("count", 0444, + debug_dir, NULL, + &count_fops); + if (!debug_count) + goto err_count; + + debug_max = debugfs_create_file("max", 0444, + debug_dir, NULL, + &max_fops); + if (!debug_max) + goto err_max; + + debug_sample_window = debugfs_create_file("window", 0644, + debug_dir, NULL, + &window_fops); + if (!debug_sample_window) + goto err_window; + + debug_sample_width = debugfs_create_file("width", 0644, + debug_dir, NULL, + &width_fops); + if (!debug_sample_width) + goto err_width; + + debug_threshold = debugfs_create_file("threshold", 0644, + debug_dir, NULL, + &threshold_fops); + if (!debug_threshold) + goto err_threshold; + + debug_enable = debugfs_create_file("enable", 0644, + debug_dir, &enabled, + &enable_fops); + if (!debug_enable) + goto err_enable; + + else { + ret = 0; + goto out; + } + +err_enable: + debugfs_remove(debug_threshold); +err_threshold: + debugfs_remove(debug_sample_width); +err_width: + debugfs_remove(debug_sample_window); +err_window: + debugfs_remove(debug_max); +err_max: + debugfs_remove(debug_count); +err_count: + debugfs_remove(debug_sample); +err_sample: + debugfs_remove(debug_dir); +err_debug_dir: +out: + return ret; +} + +/** + * free_debugfs - A function to cleanup the debugfs file interface + */ +static void free_debugfs(void) +{ + /* could also use a debugfs_remove_recursive */ + debugfs_remove(debug_enable); + debugfs_remove(debug_threshold); + debugfs_remove(debug_sample_width); + debugfs_remove(debug_sample_window); + debugfs_remove(debug_max); + debugfs_remove(debug_count); + debugfs_remove(debug_sample); + debugfs_remove(debug_dir); +} + +/** + * detector_init - Standard module initialization code + */ +static int detector_init(void) +{ + int ret = -ENOMEM; + + pr_info(BANNER "version %s\n", VERSION); + + ret = init_stats(); + if (ret) + goto out; + + ret = init_debugfs(); + if (ret) + goto err_stats; + + if (enabled) + ret = start_kthread(); + + goto out; + +err_stats: + ring_buffer_free(ring_buffer); +out: + return ret; + +} + +/** + * detector_exit - Standard module cleanup code + */ +static void detector_exit(void) +{ + int err; + + if (enabled) { + enabled = 0; + err = stop_kthread(); + if (err) + pr_err(BANNER "cannot stop kthread\n"); + } + + free_debugfs(); + ring_buffer_free(ring_buffer); /* free up the ring buffer */ + +} + +module_init(detector_init); +module_exit(detector_exit); Index: linux-4.4.25-rt35/drivers/mmc/host/mmci.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/mmc/host/mmci.c +++ linux-4.4.25-rt35/drivers/mmc/host/mmci.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1158 @ static irqreturn_t mmci_pio_irq(int irq, struct sg_mapping_iter *sg_miter = &host->sg_miter; struct variant_data *variant = host->variant; void __iomem *base = host->base; - unsigned long flags; u32 status; status = readl(base + MMCISTATUS); dev_dbg(mmc_dev(host->mmc), "irq1 (pio) %08x\n", status); - local_irq_save(flags); - do { unsigned int remain, len; char *buffer; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1203 @ static irqreturn_t mmci_pio_irq(int irq, sg_miter_stop(sg_miter); - local_irq_restore(flags); - /* * If we have less than the fifo 'half-full' threshold to transfer, * trigger a PIO interrupt as soon as any data is available. Index: linux-4.4.25-rt35/drivers/net/ethernet/3com/3c59x.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/net/ethernet/3com/3c59x.c +++ linux-4.4.25-rt35/drivers/net/ethernet/3com/3c59x.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:845 @ static void poll_vortex(struct net_devic { struct vortex_private *vp = netdev_priv(dev); unsigned long flags; - local_irq_save(flags); + local_irq_save_nort(flags); (vp->full_bus_master_rx ? boomerang_interrupt:vortex_interrupt)(dev->irq,dev); - local_irq_restore(flags); + local_irq_restore_nort(flags); } #endif @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1919 @ static void vortex_tx_timeout(struct net * Block interrupts because vortex_interrupt does a bare spin_lock() */ unsigned long flags; - local_irq_save(flags); + local_irq_save_nort(flags); if (vp->full_bus_master_tx) boomerang_interrupt(dev->irq, dev); else vortex_interrupt(dev->irq, dev); - local_irq_restore(flags); + local_irq_restore_nort(flags); } } Index: linux-4.4.25-rt35/drivers/net/ethernet/atheros/atl1c/atl1c_main.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/net/ethernet/atheros/atl1c/atl1c_main.c +++ linux-4.4.25-rt35/drivers/net/ethernet/atheros/atl1c/atl1c_main.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2224 @ static netdev_tx_t atl1c_xmit_frame(stru } tpd_req = atl1c_cal_tpd_req(skb); - if (!spin_trylock_irqsave(&adapter->tx_lock, flags)) { - if (netif_msg_pktdata(adapter)) - dev_info(&adapter->pdev->dev, "tx locked\n"); - return NETDEV_TX_LOCKED; - } + spin_lock_irqsave(&adapter->tx_lock, flags); if (atl1c_tpd_avail(adapter, type) < tpd_req) { /* no enough descriptor, just stop queue */ Index: linux-4.4.25-rt35/drivers/net/ethernet/atheros/atl1e/atl1e_main.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/net/ethernet/atheros/atl1e/atl1e_main.c +++ linux-4.4.25-rt35/drivers/net/ethernet/atheros/atl1e/atl1e_main.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1883 @ static netdev_tx_t atl1e_xmit_frame(stru return NETDEV_TX_OK; } tpd_req = atl1e_cal_tdp_req(skb); - if (!spin_trylock_irqsave(&adapter->tx_lock, flags)) - return NETDEV_TX_LOCKED; + spin_lock_irqsave(&adapter->tx_lock, flags); if (atl1e_tpd_avail(adapter) < tpd_req) { /* no enough descriptor, just stop queue */ Index: linux-4.4.25-rt35/drivers/net/ethernet/chelsio/cxgb/sge.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/net/ethernet/chelsio/cxgb/sge.c +++ linux-4.4.25-rt35/drivers/net/ethernet/chelsio/cxgb/sge.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1667 @ static int t1_sge_tx(struct sk_buff *skb struct cmdQ *q = &sge->cmdQ[qid]; unsigned int credits, pidx, genbit, count, use_sched_skb = 0; - if (!spin_trylock(&q->lock)) - return NETDEV_TX_LOCKED; + spin_lock(&q->lock); reclaim_completed_tx(sge, q); Index: linux-4.4.25-rt35/drivers/net/ethernet/neterion/s2io.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/net/ethernet/neterion/s2io.c +++ linux-4.4.25-rt35/drivers/net/ethernet/neterion/s2io.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4087 @ static netdev_tx_t s2io_xmit(struct sk_b [skb->priority & (MAX_TX_FIFOS - 1)]; fifo = &mac_control->fifos[queue]; - if (do_spin_lock) - spin_lock_irqsave(&fifo->tx_lock, flags); - else { - if (unlikely(!spin_trylock_irqsave(&fifo->tx_lock, flags))) - return NETDEV_TX_LOCKED; - } + spin_lock_irqsave(&fifo->tx_lock, flags); if (sp->config.multiq) { if (__netif_subqueue_stopped(dev, fifo->fifo_no)) { Index: linux-4.4.25-rt35/drivers/net/ethernet/oki-semi/pch_gbe/pch_gbe_main.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/net/ethernet/oki-semi/pch_gbe/pch_gbe_main.c +++ linux-4.4.25-rt35/drivers/net/ethernet/oki-semi/pch_gbe/pch_gbe_main.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2140 @ static int pch_gbe_xmit_frame(struct sk_ struct pch_gbe_tx_ring *tx_ring = adapter->tx_ring; unsigned long flags; - if (!spin_trylock_irqsave(&tx_ring->tx_lock, flags)) { - /* Collision - tell upper layer to requeue */ - return NETDEV_TX_LOCKED; - } + spin_lock_irqsave(&tx_ring->tx_lock, flags); + if (unlikely(!PCH_GBE_DESC_UNUSED(tx_ring))) { netif_stop_queue(netdev); spin_unlock_irqrestore(&tx_ring->tx_lock, flags); Index: linux-4.4.25-rt35/drivers/net/ethernet/realtek/8139too.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/net/ethernet/realtek/8139too.c +++ linux-4.4.25-rt35/drivers/net/ethernet/realtek/8139too.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2232 @ static void rtl8139_poll_controller(stru struct rtl8139_private *tp = netdev_priv(dev); const int irq = tp->pci_dev->irq; - disable_irq(irq); + disable_irq_nosync(irq); rtl8139_interrupt(irq, dev); enable_irq(irq); } Index: linux-4.4.25-rt35/drivers/net/ethernet/tehuti/tehuti.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/net/ethernet/tehuti/tehuti.c +++ linux-4.4.25-rt35/drivers/net/ethernet/tehuti/tehuti.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1632 @ static netdev_tx_t bdx_tx_transmit(struc unsigned long flags; ENTER; - local_irq_save(flags); - if (!spin_trylock(&priv->tx_lock)) { - local_irq_restore(flags); - DBG("%s[%s]: TX locked, returning NETDEV_TX_LOCKED\n", - BDX_DRV_NAME, ndev->name); - return NETDEV_TX_LOCKED; - } + + spin_lock_irqsave(&priv->tx_lock, flags); /* build tx descriptor */ BDX_ASSERT(f->m.wptr >= f->m.memsz); /* started with valid wptr */ Index: linux-4.4.25-rt35/drivers/net/rionet.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/net/rionet.c +++ linux-4.4.25-rt35/drivers/net/rionet.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:177 @ static int rionet_start_xmit(struct sk_b unsigned long flags; int add_num = 1; - local_irq_save(flags); - if (!spin_trylock(&rnet->tx_lock)) { - local_irq_restore(flags); - return NETDEV_TX_LOCKED; - } + spin_lock_irqsave(&rnet->tx_lock, flags); if (is_multicast_ether_addr(eth->h_dest)) add_num = nets[rnet->mport->id].nact; Index: linux-4.4.25-rt35/drivers/net/wireless/orinoco/orinoco_usb.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/net/wireless/orinoco/orinoco_usb.c +++ linux-4.4.25-rt35/drivers/net/wireless/orinoco/orinoco_usb.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:700 @ static void ezusb_req_ctx_wait(struct ez while (!ctx->done.done && msecs--) udelay(1000); } else { - wait_event_interruptible(ctx->done.wait, + swait_event_interruptible(ctx->done.wait, ctx->done.done); } break; Index: linux-4.4.25-rt35/drivers/pci/access.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/pci/access.c +++ linux-4.4.25-rt35/drivers/pci/access.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:564 @ void pci_cfg_access_unlock(struct pci_de WARN_ON(!dev->block_cfg_access); dev->block_cfg_access = 0; - wake_up_all(&pci_cfg_wait); + wake_up_all_locked(&pci_cfg_wait); raw_spin_unlock_irqrestore(&pci_lock, flags); } EXPORT_SYMBOL_GPL(pci_cfg_access_unlock); Index: linux-4.4.25-rt35/drivers/scsi/fcoe/fcoe.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/scsi/fcoe/fcoe.c +++ linux-4.4.25-rt35/drivers/scsi/fcoe/fcoe.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1289 @ static void fcoe_percpu_thread_destroy(u struct sk_buff *skb; #ifdef CONFIG_SMP struct fcoe_percpu_s *p0; - unsigned targ_cpu = get_cpu(); + unsigned targ_cpu = get_cpu_light(); #endif /* CONFIG_SMP */ FCOE_DBG("Destroying receive thread for CPU %d\n", cpu); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1345 @ static void fcoe_percpu_thread_destroy(u kfree_skb(skb); spin_unlock_bh(&p->fcoe_rx_list.lock); } - put_cpu(); + put_cpu_light(); #else /* * This a non-SMP scenario where the singular Rx thread is @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1569 @ err2: static int fcoe_alloc_paged_crc_eof(struct sk_buff *skb, int tlen) { struct fcoe_percpu_s *fps; - int rc; + int rc, cpu = get_cpu_light(); - fps = &get_cpu_var(fcoe_percpu); + fps = &per_cpu(fcoe_percpu, cpu); rc = fcoe_get_paged_crc_eof(skb, tlen, fps); - put_cpu_var(fcoe_percpu); + put_cpu_light(); return rc; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1769 @ static inline int fcoe_filter_frames(str return 0; } - stats = per_cpu_ptr(lport->stats, get_cpu()); + stats = per_cpu_ptr(lport->stats, get_cpu_light()); stats->InvalidCRCCount++; if (stats->InvalidCRCCount < 5) printk(KERN_WARNING "fcoe: dropping frame with CRC error\n"); - put_cpu(); + put_cpu_light(); return -EINVAL; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1817 @ static void fcoe_recv_frame(struct sk_bu */ hp = (struct fcoe_hdr *) skb_network_header(skb); - stats = per_cpu_ptr(lport->stats, get_cpu()); + stats = per_cpu_ptr(lport->stats, get_cpu_light()); if (unlikely(FC_FCOE_DECAPS_VER(hp) != FC_FCOE_VER)) { if (stats->ErrorFrames < 5) printk(KERN_WARNING "fcoe: FCoE version " @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1849 @ static void fcoe_recv_frame(struct sk_bu goto drop; if (!fcoe_filter_frames(lport, fp)) { - put_cpu(); + put_cpu_light(); fc_exch_recv(lport, fp); return; } drop: stats->ErrorFrames++; - put_cpu(); + put_cpu_light(); kfree_skb(skb); } Index: linux-4.4.25-rt35/drivers/scsi/fcoe/fcoe_ctlr.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/scsi/fcoe/fcoe_ctlr.c +++ linux-4.4.25-rt35/drivers/scsi/fcoe/fcoe_ctlr.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:834 @ static unsigned long fcoe_ctlr_age_fcfs( INIT_LIST_HEAD(&del_list); - stats = per_cpu_ptr(fip->lp->stats, get_cpu()); + stats = per_cpu_ptr(fip->lp->stats, get_cpu_light()); list_for_each_entry_safe(fcf, next, &fip->fcfs, list) { deadline = fcf->time + fcf->fka_period + fcf->fka_period / 2; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:870 @ static unsigned long fcoe_ctlr_age_fcfs( sel_time = fcf->time; } } - put_cpu(); + put_cpu_light(); list_for_each_entry_safe(fcf, next, &del_list, list) { /* Removes fcf from current list */ Index: linux-4.4.25-rt35/drivers/scsi/libfc/fc_exch.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/scsi/libfc/fc_exch.c +++ linux-4.4.25-rt35/drivers/scsi/libfc/fc_exch.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:817 @ static struct fc_exch *fc_exch_em_alloc( } memset(ep, 0, sizeof(*ep)); - cpu = get_cpu(); + cpu = get_cpu_light(); pool = per_cpu_ptr(mp->pool, cpu); spin_lock_bh(&pool->lock); - put_cpu(); + put_cpu_light(); /* peek cache of free slot */ if (pool->left != FC_XID_UNKNOWN) { Index: linux-4.4.25-rt35/drivers/scsi/libsas/sas_ata.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/scsi/libsas/sas_ata.c +++ linux-4.4.25-rt35/drivers/scsi/libsas/sas_ata.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:193 @ static unsigned int sas_ata_qc_issue(str /* TODO: audit callers to ensure they are ready for qc_issue to * unconditionally re-enable interrupts */ - local_irq_save(flags); + local_irq_save_nort(flags); spin_unlock(ap->lock); /* If the device fell off, no sense in issuing commands */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:258 @ static unsigned int sas_ata_qc_issue(str out: spin_lock(ap->lock); - local_irq_restore(flags); + local_irq_restore_nort(flags); return ret; } Index: linux-4.4.25-rt35/drivers/scsi/qla2xxx/qla_inline.h =================================================================== --- linux-4.4.25-rt35.orig/drivers/scsi/qla2xxx/qla_inline.h +++ linux-4.4.25-rt35/drivers/scsi/qla2xxx/qla_inline.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:62 @ qla2x00_poll(struct rsp_que *rsp) { unsigned long flags; struct qla_hw_data *ha = rsp->hw; - local_irq_save(flags); + local_irq_save_nort(flags); if (IS_P3P_TYPE(ha)) qla82xx_poll(0, rsp); else ha->isp_ops->intr_handler(0, rsp); - local_irq_restore(flags); + local_irq_restore_nort(flags); } static inline uint8_t * Index: linux-4.4.25-rt35/drivers/thermal/x86_pkg_temp_thermal.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/thermal/x86_pkg_temp_thermal.c +++ linux-4.4.25-rt35/drivers/thermal/x86_pkg_temp_thermal.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:32 @ #include <linux/pm.h> #include <linux/thermal.h> #include <linux/debugfs.h> +#include <linux/swork.h> #include <asm/cpu_device_id.h> #include <asm/mce.h> @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:356 @ static void pkg_temp_thermal_threshold_w } } -static int pkg_temp_thermal_platform_thermal_notify(__u64 msr_val) +static void platform_thermal_notify_work(struct swork_event *event) { unsigned long flags; int cpu = smp_processor_id(); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:373 @ static int pkg_temp_thermal_platform_the pkg_work_scheduled[phy_id]) { disable_pkg_thres_interrupt(); spin_unlock_irqrestore(&pkg_work_lock, flags); - return -EINVAL; + return; } pkg_work_scheduled[phy_id] = 1; spin_unlock_irqrestore(&pkg_work_lock, flags); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:382 @ static int pkg_temp_thermal_platform_the schedule_delayed_work_on(cpu, &per_cpu(pkg_temp_thermal_threshold_work, cpu), msecs_to_jiffies(notify_delay_ms)); +} + +#ifdef CONFIG_PREEMPT_RT_FULL +static struct swork_event notify_work; + +static int thermal_notify_work_init(void) +{ + int err; + + err = swork_get(); + if (err) + return err; + + INIT_SWORK(¬ify_work, platform_thermal_notify_work); return 0; } +static void thermal_notify_work_cleanup(void) +{ + swork_put(); +} + +static int pkg_temp_thermal_platform_thermal_notify(__u64 msr_val) +{ + swork_queue(¬ify_work); + return 0; +} + +#else /* !CONFIG_PREEMPT_RT_FULL */ + +static int thermal_notify_work_init(void) { return 0; } + +static void thermal_notify_work_cleanup(void) { } + +static int pkg_temp_thermal_platform_thermal_notify(__u64 msr_val) +{ + platform_thermal_notify_work(NULL); + + return 0; +} +#endif /* CONFIG_PREEMPT_RT_FULL */ + static int find_siblings_cpu(int cpu) { int i; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:627 @ static int __init pkg_temp_thermal_init( if (!x86_match_cpu(pkg_temp_thermal_ids)) return -ENODEV; + if (!thermal_notify_work_init()) + return -ENODEV; + spin_lock_init(&pkg_work_lock); platform_thermal_package_notify = pkg_temp_thermal_platform_thermal_notify; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:654 @ err_ret: kfree(pkg_work_scheduled); platform_thermal_package_notify = NULL; platform_thermal_package_rate_control = NULL; - + thermal_notify_work_cleanup(); return -ENODEV; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:679 @ static void __exit pkg_temp_thermal_exit mutex_unlock(&phy_dev_list_mutex); platform_thermal_package_notify = NULL; platform_thermal_package_rate_control = NULL; + thermal_notify_work_cleanup(); for_each_online_cpu(i) cancel_delayed_work_sync( &per_cpu(pkg_temp_thermal_threshold_work, i)); Index: linux-4.4.25-rt35/drivers/tty/serial/8250/8250_core.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/tty/serial/8250/8250_core.c +++ linux-4.4.25-rt35/drivers/tty/serial/8250/8250_core.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:61 @ static struct uart_driver serial8250_reg static unsigned int skip_txen_test; /* force skip of txen test at init time */ -#define PASS_LIMIT 512 +/* + * On -rt we can have a more delays, and legitimately + * so - so don't drop work spuriously and spam the + * syslog: + */ +#ifdef CONFIG_PREEMPT_RT_FULL +# define PASS_LIMIT 1000000 +#else +# define PASS_LIMIT 512 +#endif #include <asm/serial.h> /* Index: linux-4.4.25-rt35/drivers/tty/serial/8250/8250_port.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/tty/serial/8250/8250_port.c +++ linux-4.4.25-rt35/drivers/tty/serial/8250/8250_port.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:38 @ #include <linux/nmi.h> #include <linux/mutex.h> #include <linux/slab.h> +#include <linux/kdb.h> #include <linux/uaccess.h> #include <linux/pm_runtime.h> @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2847 @ void serial8250_console_write(struct uar serial8250_rpm_get(up); - if (port->sysrq) + if (port->sysrq || oops_in_progress) locked = 0; - else if (oops_in_progress) + else if (in_kdb_printk()) locked = spin_trylock_irqsave(&port->lock, flags); else spin_lock_irqsave(&port->lock, flags); Index: linux-4.4.25-rt35/drivers/tty/serial/amba-pl011.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/tty/serial/amba-pl011.c +++ linux-4.4.25-rt35/drivers/tty/serial/amba-pl011.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2070 @ pl011_console_write(struct console *co, clk_enable(uap->clk); - local_irq_save(flags); + /* + * local_irq_save(flags); + * + * This local_irq_save() is nonsense. If we come in via sysrq + * handling then interrupts are already disabled. Aside of + * that the port.sysrq check is racy on SMP regardless. + */ if (uap->port.sysrq) locked = 0; else if (oops_in_progress) - locked = spin_trylock(&uap->port.lock); + locked = spin_trylock_irqsave(&uap->port.lock, flags); else - spin_lock(&uap->port.lock); + spin_lock_irqsave(&uap->port.lock, flags); /* * First save the CR then disable the interrupts @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2107 @ pl011_console_write(struct console *co, writew(old_cr, uap->port.membase + UART011_CR); if (locked) - spin_unlock(&uap->port.lock); - local_irq_restore(flags); + spin_unlock_irqrestore(&uap->port.lock, flags); clk_disable(uap->clk); } Index: linux-4.4.25-rt35/drivers/tty/serial/omap-serial.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/tty/serial/omap-serial.c +++ linux-4.4.25-rt35/drivers/tty/serial/omap-serial.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1260 @ serial_omap_console_write(struct console pm_runtime_get_sync(up->dev); - local_irq_save(flags); - if (up->port.sysrq) - locked = 0; - else if (oops_in_progress) - locked = spin_trylock(&up->port.lock); + if (up->port.sysrq || oops_in_progress) + locked = spin_trylock_irqsave(&up->port.lock, flags); else - spin_lock(&up->port.lock); + spin_lock_irqsave(&up->port.lock, flags); /* * First save the IER then disable the interrupts @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1292 @ serial_omap_console_write(struct console pm_runtime_mark_last_busy(up->dev); pm_runtime_put_autosuspend(up->dev); if (locked) - spin_unlock(&up->port.lock); - local_irq_restore(flags); + spin_unlock_irqrestore(&up->port.lock, flags); } static int __init Index: linux-4.4.25-rt35/drivers/tty/serial/sc16is7xx.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/tty/serial/sc16is7xx.c +++ linux-4.4.25-rt35/drivers/tty/serial/sc16is7xx.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1233 @ static int sc16is7xx_probe(struct device /* Setup interrupt */ ret = devm_request_irq(dev, irq, sc16is7xx_irq, - IRQF_ONESHOT | flags, dev_name(dev), s); + flags, dev_name(dev), s); if (!ret) return 0; Index: linux-4.4.25-rt35/drivers/usb/core/hcd.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/usb/core/hcd.c +++ linux-4.4.25-rt35/drivers/usb/core/hcd.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1738 @ static void __usb_hcd_giveback_urb(struc * and no one may trigger the above deadlock situation when * running complete() in tasklet. */ - local_irq_save(flags); + local_irq_save_nort(flags); urb->complete(urb); - local_irq_restore(flags); + local_irq_restore_nort(flags); usb_anchor_resume_wakeups(anchor); atomic_dec(&urb->use_count); Index: linux-4.4.25-rt35/drivers/usb/gadget/function/f_fs.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/usb/gadget/function/f_fs.c +++ linux-4.4.25-rt35/drivers/usb/gadget/function/f_fs.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1407 @ static void ffs_data_put(struct ffs_data pr_info("%s(): freeing\n", __func__); ffs_data_clear(ffs); BUG_ON(waitqueue_active(&ffs->ev.waitq) || - waitqueue_active(&ffs->ep0req_completion.wait)); + swait_active(&ffs->ep0req_completion.wait)); kfree(ffs->dev_name); kfree(ffs); } Index: linux-4.4.25-rt35/drivers/usb/gadget/legacy/inode.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/usb/gadget/legacy/inode.c +++ linux-4.4.25-rt35/drivers/usb/gadget/legacy/inode.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:348 @ ep_io (struct ep_data *epdata, void *buf spin_unlock_irq (&epdata->dev->lock); if (likely (value == 0)) { - value = wait_event_interruptible (done.wait, done.done); + value = swait_event_interruptible (done.wait, done.done); if (value != 0) { spin_lock_irq (&epdata->dev->lock); if (likely (epdata->ep != NULL)) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:357 @ ep_io (struct ep_data *epdata, void *buf usb_ep_dequeue (epdata->ep, epdata->req); spin_unlock_irq (&epdata->dev->lock); - wait_event (done.wait, done.done); + swait_event (done.wait, done.done); if (epdata->status == -ECONNRESET) epdata->status = -EINTR; } else { Index: linux-4.4.25-rt35/drivers/usb/gadget/udc/atmel_usba_udc.c =================================================================== --- linux-4.4.25-rt35.orig/drivers/usb/gadget/udc/atmel_usba_udc.c +++ linux-4.4.25-rt35/drivers/usb/gadget/udc/atmel_usba_udc.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:20 @ #include <linux/device.h> #include <linux/dma-mapping.h> #include <linux/list.h> +#include <linux/mfd/syscon.h> #include <linux/platform_device.h> +#include <linux/regmap.h> #include <linux/usb/ch9.h> #include <linux/usb/gadget.h> #include <linux/usb/atmel_usba_udc.h> @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1893 @ static int atmel_usba_stop(struct usb_ga #ifdef CONFIG_OF static void at91sam9rl_toggle_bias(struct usba_udc *udc, int is_on) { - unsigned int uckr = at91_pmc_read(AT91_CKGR_UCKR); - - if (is_on) - at91_pmc_write(AT91_CKGR_UCKR, uckr | AT91_PMC_BIASEN); - else - at91_pmc_write(AT91_CKGR_UCKR, uckr & ~(AT91_PMC_BIASEN)); + regmap_update_bits(udc->pmc, AT91_CKGR_UCKR, AT91_PMC_BIASEN, + is_on ? AT91_PMC_BIASEN : 0); } static void at91sam9g45_pulse_bias(struct usba_udc *udc) { - unsigned int uckr = at91_pmc_read(AT91_CKGR_UCKR); - - at91_pmc_write(AT91_CKGR_UCKR, uckr & ~(AT91_PMC_BIASEN)); - at91_pmc_write(AT91_CKGR_UCKR, uckr | AT91_PMC_BIASEN); + regmap_update_bits(udc->pmc, AT91_CKGR_UCKR, AT91_PMC_BIASEN, 0); + regmap_update_bits(udc->pmc, AT91_CKGR_UCKR, AT91_PMC_BIASEN, + AT91_PMC_BIASEN); } static const struct usba_udc_errata at91sam9rl_errata = { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1938 @ static struct usba_ep * atmel_udc_of_ini return ERR_PTR(-EINVAL); udc->errata = match->data; + udc->pmc = syscon_regmap_lookup_by_compatible("atmel,at91sam9g45-pmc"); + if (udc->errata && IS_ERR(udc->pmc)) + return ERR_CAST(udc->pmc); udc->num_ep = 0; Index: linux-4.4.25-rt35/drivers/usb/gadget/udc/atmel_usba_udc.h =================================================================== --- linux-4.4.25-rt35.orig/drivers/usb/gadget/udc/atmel_usba_udc.h +++ linux-4.4.25-rt35/drivers/usb/gadget/udc/atmel_usba_udc.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:357 @ struct usba_udc { struct dentry *debugfs_root; struct dentry *debugfs_regs; #endif + + struct regmap *pmc; }; static inline struct usba_ep *to_usba_ep(struct usb_ep *ep) Index: linux-4.4.25-rt35/fs/aio.c =================================================================== --- linux-4.4.25-rt35.orig/fs/aio.c +++ linux-4.4.25-rt35/fs/aio.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:43 @ #include <linux/ramfs.h> #include <linux/percpu-refcount.h> #include <linux/mount.h> +#include <linux/swork.h> #include <asm/kmap_types.h> #include <asm/uaccess.h> @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:119 @ struct kioctx { struct page **ring_pages; long nr_pages; - struct work_struct free_work; + struct swork_event free_work; /* * signals when all in-flight requests are done @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:262 @ static int __init aio_setup(void) .mount = aio_mount, .kill_sb = kill_anon_super, }; + BUG_ON(swork_get()); aio_mnt = kern_mount(&aio_fs); if (IS_ERR(aio_mnt)) panic("Failed to create aio fs mount."); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:578 @ static int kiocb_cancel(struct aio_kiocb return cancel(&kiocb->common); } -static void free_ioctx(struct work_struct *work) +static void free_ioctx(struct swork_event *sev) { - struct kioctx *ctx = container_of(work, struct kioctx, free_work); + struct kioctx *ctx = container_of(sev, struct kioctx, free_work); pr_debug("freeing %p\n", ctx); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:599 @ static void free_ioctx_reqs(struct percp if (ctx->rq_wait && atomic_dec_and_test(&ctx->rq_wait->count)) complete(&ctx->rq_wait->comp); - INIT_WORK(&ctx->free_work, free_ioctx); - schedule_work(&ctx->free_work); + INIT_SWORK(&ctx->free_work, free_ioctx); + swork_queue(&ctx->free_work); } /* @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:608 @ static void free_ioctx_reqs(struct percp * and ctx->users has dropped to 0, so we know no more kiocbs can be submitted - * now it's safe to cancel any that need to be. */ -static void free_ioctx_users(struct percpu_ref *ref) +static void free_ioctx_users_work(struct swork_event *sev) { - struct kioctx *ctx = container_of(ref, struct kioctx, users); + struct kioctx *ctx = container_of(sev, struct kioctx, free_work); struct aio_kiocb *req; spin_lock_irq(&ctx->ctx_lock); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:629 @ static void free_ioctx_users(struct perc percpu_ref_put(&ctx->reqs); } +static void free_ioctx_users(struct percpu_ref *ref) +{ + struct kioctx *ctx = container_of(ref, struct kioctx, users); + + INIT_SWORK(&ctx->free_work, free_ioctx_users_work); + swork_queue(&ctx->free_work); +} + static int ioctx_add_table(struct kioctx *ctx, struct mm_struct *mm) { unsigned i, new_nr; Index: linux-4.4.25-rt35/fs/autofs4/autofs_i.h =================================================================== --- linux-4.4.25-rt35.orig/fs/autofs4/autofs_i.h +++ linux-4.4.25-rt35/fs/autofs4/autofs_i.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:37 @ #include <linux/sched.h> #include <linux/mount.h> #include <linux/namei.h> +#include <linux/delay.h> #include <asm/current.h> #include <asm/uaccess.h> Index: linux-4.4.25-rt35/fs/autofs4/expire.c =================================================================== --- linux-4.4.25-rt35.orig/fs/autofs4/expire.c +++ linux-4.4.25-rt35/fs/autofs4/expire.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:153 @ again: parent = p->d_parent; if (!spin_trylock(&parent->d_lock)) { spin_unlock(&p->d_lock); - cpu_relax(); + cpu_chill(); goto relock; } spin_unlock(&p->d_lock); Index: linux-4.4.25-rt35/fs/buffer.c =================================================================== --- linux-4.4.25-rt35.orig/fs/buffer.c +++ linux-4.4.25-rt35/fs/buffer.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:308 @ static void end_buffer_async_read(struct * decide that the page is now completely done. */ first = page_buffers(page); - local_irq_save(flags); - bit_spin_lock(BH_Uptodate_Lock, &first->b_state); + flags = bh_uptodate_lock_irqsave(first); clear_buffer_async_read(bh); unlock_buffer(bh); tmp = bh; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:321 @ static void end_buffer_async_read(struct } tmp = tmp->b_this_page; } while (tmp != bh); - bit_spin_unlock(BH_Uptodate_Lock, &first->b_state); - local_irq_restore(flags); + bh_uptodate_unlock_irqrestore(first, flags); /* * If none of the buffers had errors and they are all @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:333 @ static void end_buffer_async_read(struct return; still_busy: - bit_spin_unlock(BH_Uptodate_Lock, &first->b_state); - local_irq_restore(flags); - return; + bh_uptodate_unlock_irqrestore(first, flags); } /* @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:361 @ void end_buffer_async_write(struct buffe } first = page_buffers(page); - local_irq_save(flags); - bit_spin_lock(BH_Uptodate_Lock, &first->b_state); + flags = bh_uptodate_lock_irqsave(first); clear_buffer_async_write(bh); unlock_buffer(bh); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:373 @ void end_buffer_async_write(struct buffe } tmp = tmp->b_this_page; } - bit_spin_unlock(BH_Uptodate_Lock, &first->b_state); - local_irq_restore(flags); + bh_uptodate_unlock_irqrestore(first, flags); end_page_writeback(page); return; still_busy: - bit_spin_unlock(BH_Uptodate_Lock, &first->b_state); - local_irq_restore(flags); - return; + bh_uptodate_unlock_irqrestore(first, flags); } EXPORT_SYMBOL(end_buffer_async_write); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:3320 @ struct buffer_head *alloc_buffer_head(gf struct buffer_head *ret = kmem_cache_zalloc(bh_cachep, gfp_flags); if (ret) { INIT_LIST_HEAD(&ret->b_assoc_buffers); + buffer_head_init_locks(ret); preempt_disable(); __this_cpu_inc(bh_accounting.nr); recalc_bh_state(); Index: linux-4.4.25-rt35/fs/dcache.c =================================================================== --- linux-4.4.25-rt35.orig/fs/dcache.c +++ linux-4.4.25-rt35/fs/dcache.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:22 @ #include <linux/mm.h> #include <linux/fs.h> #include <linux/fsnotify.h> +#include <linux/delay.h> #include <linux/slab.h> #include <linux/init.h> #include <linux/hash.h> @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:751 @ static inline bool fast_dput(struct dent */ void dput(struct dentry *dentry) { + struct dentry *parent; + if (unlikely(!dentry)) return; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:789 @ repeat: return; kill_it: - dentry = dentry_kill(dentry); - if (dentry) { - cond_resched(); + parent = dentry_kill(dentry); + if (parent) { + int r; + + if (parent == dentry) { + /* the task with the highest priority won't schedule */ + r = cond_resched(); + if (!r) + cpu_chill(); + } else { + dentry = parent; + } goto repeat; } } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2409 @ again: if (dentry->d_lockref.count == 1) { if (!spin_trylock(&inode->i_lock)) { spin_unlock(&dentry->d_lock); - cpu_relax(); + cpu_chill(); goto again; } dentry->d_flags &= ~DCACHE_CANT_MOUNT; Index: linux-4.4.25-rt35/fs/eventpoll.c =================================================================== --- linux-4.4.25-rt35.orig/fs/eventpoll.c +++ linux-4.4.25-rt35/fs/eventpoll.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:508 @ static int ep_poll_wakeup_proc(void *pri */ static void ep_poll_safewake(wait_queue_head_t *wq) { - int this_cpu = get_cpu(); + int this_cpu = get_cpu_light(); ep_call_nested(&poll_safewake_ncalls, EP_MAX_NESTS, ep_poll_wakeup_proc, NULL, wq, (void *) (long) this_cpu); - put_cpu(); + put_cpu_light(); } static void ep_remove_wait_queue(struct eppoll_entry *pwq) Index: linux-4.4.25-rt35/fs/exec.c =================================================================== --- linux-4.4.25-rt35.orig/fs/exec.c +++ linux-4.4.25-rt35/fs/exec.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:868 @ static int exec_mmap(struct mm_struct *m } } task_lock(tsk); + preempt_disable_rt(); active_mm = tsk->active_mm; tsk->mm = mm; tsk->active_mm = mm; activate_mm(active_mm, mm); tsk->mm->vmacache_seqnum = 0; vmacache_flush(tsk); + preempt_enable_rt(); task_unlock(tsk); if (old_mm) { up_read(&old_mm->mmap_sem); Index: linux-4.4.25-rt35/fs/f2fs/f2fs.h =================================================================== --- linux-4.4.25-rt35.orig/fs/f2fs/f2fs.h +++ linux-4.4.25-rt35/fs/f2fs/f2fs.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:27 @ #ifdef CONFIG_F2FS_CHECK_FS #define f2fs_bug_on(sbi, condition) BUG_ON(condition) -#define f2fs_down_write(x, y) down_write_nest_lock(x, y) #else #define f2fs_bug_on(sbi, condition) \ do { \ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:35 @ set_sbi_flag(sbi, SBI_NEED_FSCK); \ } \ } while (0) -#define f2fs_down_write(x, y) down_write(x) #endif /* @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:960 @ static inline void f2fs_unlock_op(struct static inline void f2fs_lock_all(struct f2fs_sb_info *sbi) { - f2fs_down_write(&sbi->cp_rwsem, &sbi->cp_mutex); + down_write(&sbi->cp_rwsem); } static inline void f2fs_unlock_all(struct f2fs_sb_info *sbi) Index: linux-4.4.25-rt35/fs/jbd2/checkpoint.c =================================================================== --- linux-4.4.25-rt35.orig/fs/jbd2/checkpoint.c +++ linux-4.4.25-rt35/fs/jbd2/checkpoint.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:119 @ void __jbd2_log_wait_for_space(journal_t nblocks = jbd2_space_needed(journal); while (jbd2_log_space_left(journal) < nblocks) { write_unlock(&journal->j_state_lock); + if (current->plug) + io_schedule(); mutex_lock(&journal->j_checkpoint_mutex); /* Index: linux-4.4.25-rt35/fs/namespace.c =================================================================== --- linux-4.4.25-rt35.orig/fs/namespace.c +++ linux-4.4.25-rt35/fs/namespace.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:17 @ #include <linux/mnt_namespace.h> #include <linux/user_namespace.h> #include <linux/namei.h> +#include <linux/delay.h> #include <linux/security.h> #include <linux/idr.h> #include <linux/init.h> /* init_rootfs */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:357 @ int __mnt_want_write(struct vfsmount *m) * incremented count after it has set MNT_WRITE_HOLD. */ smp_mb(); - while (ACCESS_ONCE(mnt->mnt.mnt_flags) & MNT_WRITE_HOLD) - cpu_relax(); + while (ACCESS_ONCE(mnt->mnt.mnt_flags) & MNT_WRITE_HOLD) { + preempt_enable(); + cpu_chill(); + preempt_disable(); + } /* * After the slowpath clears MNT_WRITE_HOLD, mnt_is_readonly will * be set to match its requirements. So we must not load that until Index: linux-4.4.25-rt35/fs/ntfs/aops.c =================================================================== --- linux-4.4.25-rt35.orig/fs/ntfs/aops.c +++ linux-4.4.25-rt35/fs/ntfs/aops.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:110 @ static void ntfs_end_buffer_async_read(s "0x%llx.", (unsigned long long)bh->b_blocknr); } first = page_buffers(page); - local_irq_save(flags); - bit_spin_lock(BH_Uptodate_Lock, &first->b_state); + flags = bh_uptodate_lock_irqsave(first); clear_buffer_async_read(bh); unlock_buffer(bh); tmp = bh; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:125 @ static void ntfs_end_buffer_async_read(s } tmp = tmp->b_this_page; } while (tmp != bh); - bit_spin_unlock(BH_Uptodate_Lock, &first->b_state); - local_irq_restore(flags); + bh_uptodate_unlock_irqrestore(first, flags); /* * If none of the buffers had errors then we can set the page uptodate, * but we first have to perform the post read mst fixups, if the @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:146 @ static void ntfs_end_buffer_async_read(s recs = PAGE_CACHE_SIZE / rec_size; /* Should have been verified before we got here... */ BUG_ON(!recs); - local_irq_save(flags); + local_irq_save_nort(flags); kaddr = kmap_atomic(page); for (i = 0; i < recs; i++) post_read_mst_fixup((NTFS_RECORD*)(kaddr + i * rec_size), rec_size); kunmap_atomic(kaddr); - local_irq_restore(flags); + local_irq_restore_nort(flags); flush_dcache_page(page); if (likely(page_uptodate && !PageError(page))) SetPageUptodate(page); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:160 @ static void ntfs_end_buffer_async_read(s unlock_page(page); return; still_busy: - bit_spin_unlock(BH_Uptodate_Lock, &first->b_state); - local_irq_restore(flags); - return; + bh_uptodate_unlock_irqrestore(first, flags); } /** Index: linux-4.4.25-rt35/fs/timerfd.c =================================================================== --- linux-4.4.25-rt35.orig/fs/timerfd.c +++ linux-4.4.25-rt35/fs/timerfd.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:453 @ static int do_timerfd_settime(int ufd, i break; } spin_unlock_irq(&ctx->wqh.lock); - cpu_relax(); + if (isalarm(ctx)) + hrtimer_wait_for_timer(&ctx->t.alarm.timer); + else + hrtimer_wait_for_timer(&ctx->t.tmr); } /* Index: linux-4.4.25-rt35/include/acpi/platform/aclinux.h =================================================================== --- linux-4.4.25-rt35.orig/include/acpi/platform/aclinux.h +++ linux-4.4.25-rt35/include/acpi/platform/aclinux.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:130 @ #define acpi_cache_t struct kmem_cache #define acpi_spinlock spinlock_t * +#define acpi_raw_spinlock raw_spinlock_t * #define acpi_cpu_flags unsigned long /* Use native linux version of acpi_os_allocate_zeroed */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:149 @ #define ACPI_USE_ALTERNATE_PROTOTYPE_acpi_os_get_thread_id #define ACPI_USE_ALTERNATE_PROTOTYPE_acpi_os_create_lock +#define acpi_os_create_raw_lock(__handle) \ +({ \ + raw_spinlock_t *lock = ACPI_ALLOCATE(sizeof(*lock)); \ + \ + if (lock) { \ + *(__handle) = lock; \ + raw_spin_lock_init(*(__handle)); \ + } \ + lock ? AE_OK : AE_NO_MEMORY; \ + }) + +#define acpi_os_delete_raw_lock(__handle) kfree(__handle) + + /* * OSL interfaces used by debugger/disassembler */ Index: linux-4.4.25-rt35/include/asm-generic/bug.h =================================================================== --- linux-4.4.25-rt35.orig/include/asm-generic/bug.h +++ linux-4.4.25-rt35/include/asm-generic/bug.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:209 @ extern void warn_slowpath_null(const cha # define WARN_ON_SMP(x) ({0;}) #endif +#ifdef CONFIG_PREEMPT_RT_BASE +# define BUG_ON_RT(c) BUG_ON(c) +# define BUG_ON_NONRT(c) do { } while (0) +# define WARN_ON_RT(condition) WARN_ON(condition) +# define WARN_ON_NONRT(condition) do { } while (0) +# define WARN_ON_ONCE_NONRT(condition) do { } while (0) +#else +# define BUG_ON_RT(c) do { } while (0) +# define BUG_ON_NONRT(c) BUG_ON(c) +# define WARN_ON_RT(condition) do { } while (0) +# define WARN_ON_NONRT(condition) WARN_ON(condition) +# define WARN_ON_ONCE_NONRT(condition) WARN_ON_ONCE(condition) +#endif + #endif /* __ASSEMBLY__ */ #endif Index: linux-4.4.25-rt35/include/asm-generic/preempt.h =================================================================== --- linux-4.4.25-rt35.orig/include/asm-generic/preempt.h +++ linux-4.4.25-rt35/include/asm-generic/preempt.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:10 @ static __always_inline int preempt_count(void) { - return current_thread_info()->preempt_count; + return READ_ONCE(current_thread_info()->preempt_count); } -static __always_inline int *preempt_count_ptr(void) +static __always_inline volatile int *preempt_count_ptr(void) { return ¤t_thread_info()->preempt_count; } Index: linux-4.4.25-rt35/include/linux/blk-mq.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/blk-mq.h +++ linux-4.4.25-rt35/include/linux/blk-mq.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:215 @ static inline u16 blk_mq_unique_tag_to_t struct blk_mq_hw_ctx *blk_mq_map_queue(struct request_queue *, const int ctx_index); struct blk_mq_hw_ctx *blk_mq_alloc_single_hw_queue(struct blk_mq_tag_set *, unsigned int, int); +void __blk_mq_complete_request_remote_work(struct work_struct *work); int blk_mq_request_started(struct request *rq); void blk_mq_start_request(struct request *rq); Index: linux-4.4.25-rt35/include/linux/blkdev.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/blkdev.h +++ linux-4.4.25-rt35/include/linux/blkdev.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:92 @ struct request { struct list_head queuelist; union { struct call_single_data csd; + struct work_struct work; unsigned long fifo_time; }; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:459 @ struct request_queue { struct throtl_data *td; #endif struct rcu_head rcu_head; - wait_queue_head_t mq_freeze_wq; + struct swait_queue_head mq_freeze_wq; struct percpu_ref q_usage_counter; struct list_head all_q_node; Index: linux-4.4.25-rt35/include/linux/bottom_half.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/bottom_half.h +++ linux-4.4.25-rt35/include/linux/bottom_half.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:6 @ #include <linux/preempt.h> +#ifdef CONFIG_PREEMPT_RT_FULL + +extern void __local_bh_disable(void); +extern void _local_bh_enable(void); +extern void __local_bh_enable(void); + +static inline void local_bh_disable(void) +{ + __local_bh_disable(); +} + +static inline void __local_bh_disable_ip(unsigned long ip, unsigned int cnt) +{ + __local_bh_disable(); +} + +static inline void local_bh_enable(void) +{ + __local_bh_enable(); +} + +static inline void __local_bh_enable_ip(unsigned long ip, unsigned int cnt) +{ + __local_bh_enable(); +} + +static inline void local_bh_enable_ip(unsigned long ip) +{ + __local_bh_enable(); +} + +#else + #ifdef CONFIG_TRACE_IRQFLAGS extern void __local_bh_disable_ip(unsigned long ip, unsigned int cnt); #else @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:66 @ static inline void local_bh_enable(void) { __local_bh_enable_ip(_THIS_IP_, SOFTIRQ_DISABLE_OFFSET); } +#endif #endif /* _LINUX_BH_H */ Index: linux-4.4.25-rt35/include/linux/buffer_head.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/buffer_head.h +++ linux-4.4.25-rt35/include/linux/buffer_head.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:78 @ struct buffer_head { struct address_space *b_assoc_map; /* mapping this buffer is associated with */ atomic_t b_count; /* users using this buffer_head */ +#ifdef CONFIG_PREEMPT_RT_BASE + spinlock_t b_uptodate_lock; +#if IS_ENABLED(CONFIG_JBD2) + spinlock_t b_state_lock; + spinlock_t b_journal_head_lock; +#endif +#endif }; +static inline unsigned long bh_uptodate_lock_irqsave(struct buffer_head *bh) +{ + unsigned long flags; + +#ifndef CONFIG_PREEMPT_RT_BASE + local_irq_save(flags); + bit_spin_lock(BH_Uptodate_Lock, &bh->b_state); +#else + spin_lock_irqsave(&bh->b_uptodate_lock, flags); +#endif + return flags; +} + +static inline void +bh_uptodate_unlock_irqrestore(struct buffer_head *bh, unsigned long flags) +{ +#ifndef CONFIG_PREEMPT_RT_BASE + bit_spin_unlock(BH_Uptodate_Lock, &bh->b_state); + local_irq_restore(flags); +#else + spin_unlock_irqrestore(&bh->b_uptodate_lock, flags); +#endif +} + +static inline void buffer_head_init_locks(struct buffer_head *bh) +{ +#ifdef CONFIG_PREEMPT_RT_BASE + spin_lock_init(&bh->b_uptodate_lock); +#if IS_ENABLED(CONFIG_JBD2) + spin_lock_init(&bh->b_state_lock); + spin_lock_init(&bh->b_journal_head_lock); +#endif +#endif +} + /* * macro tricks to expand the set_buffer_foo(), clear_buffer_foo() * and buffer_foo() functions. Index: linux-4.4.25-rt35/include/linux/cgroup-defs.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/cgroup-defs.h +++ linux-4.4.25-rt35/include/linux/cgroup-defs.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:19 @ #include <linux/percpu-refcount.h> #include <linux/percpu-rwsem.h> #include <linux/workqueue.h> +#include <linux/swork.h> #ifdef CONFIG_CGROUPS @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:146 @ struct cgroup_subsys_state { /* percpu_ref killing and RCU release */ struct rcu_head rcu_head; struct work_struct destroy_work; + struct swork_event destroy_swork; }; /* Index: linux-4.4.25-rt35/include/linux/clk/at91_pmc.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/clk/at91_pmc.h +++ linux-4.4.25-rt35/include/linux/clk/at91_pmc.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:19 @ #ifndef AT91_PMC_H #define AT91_PMC_H -#ifndef __ASSEMBLY__ -extern void __iomem *at91_pmc_base; - -#define at91_pmc_read(field) \ - readl_relaxed(at91_pmc_base + field) - -#define at91_pmc_write(field, value) \ - writel_relaxed(value, at91_pmc_base + field) -#else -.extern at91_pmc_base -#endif - #define AT91_PMC_SCER 0x00 /* System Clock Enable Register */ #define AT91_PMC_SCDR 0x04 /* System Clock Disable Register */ Index: linux-4.4.25-rt35/include/linux/completion.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/completion.h +++ linux-4.4.25-rt35/include/linux/completion.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:10 @ * Atomic wait-for-completion handler data structures. * See kernel/sched/completion.c for details. */ - -#include <linux/wait.h> +#include <linux/swait.h> /* * struct completion - structure used to maintain state for a "completion" @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:26 @ */ struct completion { unsigned int done; - wait_queue_head_t wait; + struct swait_queue_head wait; }; #define COMPLETION_INITIALIZER(work) \ - { 0, __WAIT_QUEUE_HEAD_INITIALIZER((work).wait) } + { 0, __SWAIT_QUEUE_HEAD_INITIALIZER((work).wait) } #define COMPLETION_INITIALIZER_ONSTACK(work) \ ({ init_completion(&work); work; }) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:75 @ struct completion { static inline void init_completion(struct completion *x) { x->done = 0; - init_waitqueue_head(&x->wait); + init_swait_queue_head(&x->wait); } /** Index: linux-4.4.25-rt35/include/linux/cpu.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/cpu.h +++ linux-4.4.25-rt35/include/linux/cpu.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:227 @ extern void get_online_cpus(void); extern void put_online_cpus(void); extern void cpu_hotplug_disable(void); extern void cpu_hotplug_enable(void); +extern void pin_current_cpu(void); +extern void unpin_current_cpu(void); #define hotcpu_notifier(fn, pri) cpu_notifier(fn, pri) #define __hotcpu_notifier(fn, pri) __cpu_notifier(fn, pri) #define register_hotcpu_notifier(nb) register_cpu_notifier(nb) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:246 @ static inline void cpu_hotplug_done(void #define put_online_cpus() do { } while (0) #define cpu_hotplug_disable() do { } while (0) #define cpu_hotplug_enable() do { } while (0) +static inline void pin_current_cpu(void) { } +static inline void unpin_current_cpu(void) { } #define hotcpu_notifier(fn, pri) do { (void)(fn); } while (0) #define __hotcpu_notifier(fn, pri) do { (void)(fn); } while (0) /* These aren't inline functions due to a GCC bug. */ Index: linux-4.4.25-rt35/include/linux/delay.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/delay.h +++ linux-4.4.25-rt35/include/linux/delay.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:55 @ static inline void ssleep(unsigned int s msleep(seconds * 1000); } +#ifdef CONFIG_PREEMPT_RT_FULL +extern void cpu_chill(void); +#else +# define cpu_chill() cpu_relax() +#endif + #endif /* defined(_LINUX_DELAY_H) */ Index: linux-4.4.25-rt35/include/linux/ftrace.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/ftrace.h +++ linux-4.4.25-rt35/include/linux/ftrace.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:697 @ static inline void __ftrace_enabled_rest #define CALLER_ADDR5 ((unsigned long)ftrace_return_address(5)) #define CALLER_ADDR6 ((unsigned long)ftrace_return_address(6)) +static inline unsigned long get_lock_parent_ip(void) +{ + unsigned long addr = CALLER_ADDR0; + + if (!in_lock_functions(addr)) + return addr; + addr = CALLER_ADDR1; + if (!in_lock_functions(addr)) + return addr; + return CALLER_ADDR2; +} + #ifdef CONFIG_IRQSOFF_TRACER extern void time_hardirqs_on(unsigned long a0, unsigned long a1); extern void time_hardirqs_off(unsigned long a0, unsigned long a1); Index: linux-4.4.25-rt35/include/linux/highmem.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/highmem.h +++ linux-4.4.25-rt35/include/linux/highmem.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:10 @ #include <linux/mm.h> #include <linux/uaccess.h> #include <linux/hardirq.h> +#include <linux/sched.h> #include <asm/cacheflush.h> @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:69 @ static inline void kunmap(struct page *p static inline void *kmap_atomic(struct page *page) { - preempt_disable(); + preempt_disable_nort(); pagefault_disable(); return page_address(page); } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:78 @ static inline void *kmap_atomic(struct p static inline void __kunmap_atomic(void *addr) { pagefault_enable(); - preempt_enable(); + preempt_enable_nort(); } #define kmap_atomic_pfn(pfn) kmap_atomic(pfn_to_page(pfn)) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:90 @ static inline void __kunmap_atomic(void #if defined(CONFIG_HIGHMEM) || defined(CONFIG_X86_32) +#ifndef CONFIG_PREEMPT_RT_FULL DECLARE_PER_CPU(int, __kmap_atomic_idx); +#endif static inline int kmap_atomic_idx_push(void) { +#ifndef CONFIG_PREEMPT_RT_FULL int idx = __this_cpu_inc_return(__kmap_atomic_idx) - 1; -#ifdef CONFIG_DEBUG_HIGHMEM +# ifdef CONFIG_DEBUG_HIGHMEM WARN_ON_ONCE(in_irq() && !irqs_disabled()); BUG_ON(idx >= KM_TYPE_NR); -#endif +# endif return idx; +#else + current->kmap_idx++; + BUG_ON(current->kmap_idx > KM_TYPE_NR); + return current->kmap_idx - 1; +#endif } static inline int kmap_atomic_idx(void) { +#ifndef CONFIG_PREEMPT_RT_FULL return __this_cpu_read(__kmap_atomic_idx) - 1; +#else + return current->kmap_idx - 1; +#endif } static inline void kmap_atomic_idx_pop(void) { -#ifdef CONFIG_DEBUG_HIGHMEM +#ifndef CONFIG_PREEMPT_RT_FULL +# ifdef CONFIG_DEBUG_HIGHMEM int idx = __this_cpu_dec_return(__kmap_atomic_idx); BUG_ON(idx < 0); -#else +# else __this_cpu_dec(__kmap_atomic_idx); +# endif +#else + current->kmap_idx--; +# ifdef CONFIG_DEBUG_HIGHMEM + BUG_ON(current->kmap_idx < 0); +# endif #endif } Index: linux-4.4.25-rt35/include/linux/hrtimer.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/hrtimer.h +++ linux-4.4.25-rt35/include/linux/hrtimer.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:90 @ enum hrtimer_restart { * @function: timer expiry callback function * @base: pointer to the timer base (per cpu and per clock) * @state: state information (See bit values above) + * @cb_entry: list entry to defer timers from hardirq context + * @irqsafe: timer can run in hardirq context + * @praecox: timer expiry time if expired at the time of programming * @is_rel: Set if the timer was armed relative * @start_pid: timer statistics field to store the pid of the task which * started the timer @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:109 @ struct hrtimer { enum hrtimer_restart (*function)(struct hrtimer *); struct hrtimer_clock_base *base; u8 state; + struct list_head cb_entry; + int irqsafe; +#ifdef CONFIG_MISSED_TIMER_OFFSETS_HIST + ktime_t praecox; +#endif u8 is_rel; #ifdef CONFIG_TIMER_STATS int start_pid; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:134 @ struct hrtimer_sleeper { struct task_struct *task; }; -#ifdef CONFIG_64BIT # define HRTIMER_CLOCK_BASE_ALIGN 64 -#else -# define HRTIMER_CLOCK_BASE_ALIGN 32 -#endif /** * struct hrtimer_clock_base - the timer base for a specific clock @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:143 @ struct hrtimer_sleeper { * timer to a base on another cpu. * @clockid: clock id for per_cpu support * @active: red black tree root node for the active timers + * @expired: list head for deferred timers. * @get_time: function to retrieve the current time of the clock * @offset: offset of this clock to the monotonic base */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:152 @ struct hrtimer_clock_base { int index; clockid_t clockid; struct timerqueue_head active; + struct list_head expired; ktime_t (*get_time)(void); ktime_t offset; } __attribute__((__aligned__(HRTIMER_CLOCK_BASE_ALIGN))); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:196 @ struct hrtimer_cpu_base { raw_spinlock_t lock; seqcount_t seq; struct hrtimer *running; + struct hrtimer *running_soft; unsigned int cpu; unsigned int active_bases; unsigned int clock_was_set_seq; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:213 @ struct hrtimer_cpu_base { unsigned int nr_hangs; unsigned int max_hang_time; #endif +#ifdef CONFIG_PREEMPT_RT_BASE + wait_queue_head_t wait; +#endif struct hrtimer_clock_base clock_base[HRTIMER_MAX_CLOCK_BASES]; } ____cacheline_aligned; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:425 @ static inline void hrtimer_restart(struc hrtimer_start_expires(timer, HRTIMER_MODE_ABS); } +/* Softirq preemption could deadlock timer removal */ +#ifdef CONFIG_PREEMPT_RT_BASE + extern void hrtimer_wait_for_timer(const struct hrtimer *timer); +#else +# define hrtimer_wait_for_timer(timer) do { cpu_relax(); } while (0) +#endif + /* Query timers: */ extern ktime_t __hrtimer_get_remaining(const struct hrtimer *timer, bool adjust); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:456 @ static inline int hrtimer_is_queued(stru * Helper function to check, whether the timer is running the callback * function */ -static inline int hrtimer_callback_running(struct hrtimer *timer) +static inline int hrtimer_callback_running(const struct hrtimer *timer) { return timer->base->cpu_base->running == timer; } Index: linux-4.4.25-rt35/include/linux/idr.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/idr.h +++ linux-4.4.25-rt35/include/linux/idr.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:98 @ bool idr_is_empty(struct idr *idp); * Each idr_preload() should be matched with an invocation of this * function. See idr_preload() for details. */ +#ifdef CONFIG_PREEMPT_RT_FULL +void idr_preload_end(void); +#else static inline void idr_preload_end(void) { preempt_enable(); } +#endif /** * idr_find - return pointer for given id Index: linux-4.4.25-rt35/include/linux/init_task.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/init_task.h +++ linux-4.4.25-rt35/include/linux/init_task.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:151 @ extern struct task_group root_task_group # define INIT_PERF_EVENTS(tsk) #endif +#ifdef CONFIG_PREEMPT_RT_BASE +# define INIT_TIMER_LIST .posix_timer_list = NULL, +#else +# define INIT_TIMER_LIST +#endif + #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN # define INIT_VTIME(tsk) \ - .vtime_seqlock = __SEQLOCK_UNLOCKED(tsk.vtime_seqlock), \ + .vtime_seqcount = SEQCNT_ZERO(tsk.vtime_seqcount), \ .vtime_snap = 0, \ .vtime_snap_whence = VTIME_SYS, #else @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:248 @ extern struct task_group root_task_group .cpu_timers = INIT_CPU_TIMERS(tsk.cpu_timers), \ .pi_lock = __RAW_SPIN_LOCK_UNLOCKED(tsk.pi_lock), \ .timer_slack_ns = 50000, /* 50 usec default slack */ \ + INIT_TIMER_LIST \ .pids = { \ [PIDTYPE_PID] = INIT_PID_LINK(PIDTYPE_PID), \ [PIDTYPE_PGID] = INIT_PID_LINK(PIDTYPE_PGID), \ Index: linux-4.4.25-rt35/include/linux/interrupt.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/interrupt.h +++ linux-4.4.25-rt35/include/linux/interrupt.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:64 @ * interrupt handler after suspending interrupts. For system * wakeup devices users need to implement wakeup detection in * their interrupt handlers. + * IRQF_NO_SOFTIRQ_CALL - Do not process softirqs in the irq thread context (RT) */ #define IRQF_SHARED 0x00000080 #define IRQF_PROBE_SHARED 0x00000100 @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:78 @ #define IRQF_NO_THREAD 0x00010000 #define IRQF_EARLY_RESUME 0x00020000 #define IRQF_COND_SUSPEND 0x00040000 +#define IRQF_NO_SOFTIRQ_CALL 0x00080000 #define IRQF_TIMER (__IRQF_TIMER | IRQF_NO_SUSPEND | IRQF_NO_THREAD) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:191 @ extern void devm_free_irq(struct device #ifdef CONFIG_LOCKDEP # define local_irq_enable_in_hardirq() do { } while (0) #else -# define local_irq_enable_in_hardirq() local_irq_enable() +# define local_irq_enable_in_hardirq() local_irq_enable_nort() #endif extern void disable_irq_nosync(unsigned int irq); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:211 @ extern void resume_device_irqs(void); * @irq: Interrupt to which notification applies * @kref: Reference count, for internal use * @work: Work item, for internal use + * @list: List item for deferred callbacks * @notify: Function to be called on change. This will be * called in process context. * @release: Function to be called on release. This will be @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:223 @ struct irq_affinity_notify { unsigned int irq; struct kref kref; struct work_struct work; + struct list_head list; void (*notify)(struct irq_affinity_notify *, const cpumask_t *mask); void (*release)(struct kref *ref); }; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:386 @ extern int irq_set_irqchip_state(unsigne bool state); #ifdef CONFIG_IRQ_FORCED_THREADING +# ifndef CONFIG_PREEMPT_RT_BASE extern bool force_irqthreads; +# else +# define force_irqthreads (true) +# endif #else -#define force_irqthreads (0) +#define force_irqthreads (false) #endif #ifndef __ARCH_SET_SOFTIRQ_PENDING @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:449 @ struct softirq_action void (*action)(struct softirq_action *); }; +#ifndef CONFIG_PREEMPT_RT_FULL asmlinkage void do_softirq(void); asmlinkage void __do_softirq(void); - +static inline void thread_do_softirq(void) { do_softirq(); } #ifdef __ARCH_HAS_DO_SOFTIRQ void do_softirq_own_stack(void); #else @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:461 @ static inline void do_softirq_own_stack( __do_softirq(); } #endif +#else +extern void thread_do_softirq(void); +#endif extern void open_softirq(int nr, void (*action)(struct softirq_action *)); extern void softirq_init(void); extern void __raise_softirq_irqoff(unsigned int nr); +#ifdef CONFIG_PREEMPT_RT_FULL +extern void __raise_softirq_irqoff_ksoft(unsigned int nr); +#else +static inline void __raise_softirq_irqoff_ksoft(unsigned int nr) +{ + __raise_softirq_irqoff(nr); +} +#endif extern void raise_softirq_irqoff(unsigned int nr); extern void raise_softirq(unsigned int nr); +extern void softirq_check_pending_idle(void); DECLARE_PER_CPU(struct task_struct *, ksoftirqd); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:501 @ static inline struct task_struct *this_c to be executed on some cpu at least once after this. * If the tasklet is already scheduled, but its execution is still not started, it will be executed only once. - * If this tasklet is already running on another CPU (or schedule is called - from tasklet itself), it is rescheduled for later. + * If this tasklet is already running on another CPU, it is rescheduled + for later. + * Schedule must not be called from the tasklet itself (a lockup occurs) * Tasklet is strictly serialized wrt itself, but not wrt another tasklets. If client needs some intertask synchronization, he makes it with spinlocks. @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:528 @ struct tasklet_struct name = { NULL, 0, enum { TASKLET_STATE_SCHED, /* Tasklet is scheduled for execution */ - TASKLET_STATE_RUN /* Tasklet is running (SMP only) */ + TASKLET_STATE_RUN, /* Tasklet is running (SMP only) */ + TASKLET_STATE_PENDING /* Tasklet is pending */ }; -#ifdef CONFIG_SMP +#define TASKLET_STATEF_SCHED (1 << TASKLET_STATE_SCHED) +#define TASKLET_STATEF_RUN (1 << TASKLET_STATE_RUN) +#define TASKLET_STATEF_PENDING (1 << TASKLET_STATE_PENDING) + +#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT_RT_FULL) static inline int tasklet_trylock(struct tasklet_struct *t) { return !test_and_set_bit(TASKLET_STATE_RUN, &(t)->state); } +static inline int tasklet_tryunlock(struct tasklet_struct *t) +{ + return cmpxchg(&t->state, TASKLET_STATEF_RUN, 0) == TASKLET_STATEF_RUN; +} + static inline void tasklet_unlock(struct tasklet_struct *t) { smp_mb__before_atomic(); clear_bit(TASKLET_STATE_RUN, &(t)->state); } -static inline void tasklet_unlock_wait(struct tasklet_struct *t) -{ - while (test_bit(TASKLET_STATE_RUN, &(t)->state)) { barrier(); } -} +extern void tasklet_unlock_wait(struct tasklet_struct *t); + #else #define tasklet_trylock(t) 1 +#define tasklet_tryunlock(t) 1 #define tasklet_unlock_wait(t) do { } while (0) #define tasklet_unlock(t) do { } while (0) #endif @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:606 @ static inline void tasklet_disable(struc smp_mb(); } -static inline void tasklet_enable(struct tasklet_struct *t) -{ - smp_mb__before_atomic(); - atomic_dec(&t->count); -} - +extern void tasklet_enable(struct tasklet_struct *t); extern void tasklet_kill(struct tasklet_struct *t); extern void tasklet_kill_immediate(struct tasklet_struct *t, unsigned int cpu); extern void tasklet_init(struct tasklet_struct *t, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:637 @ void tasklet_hrtimer_cancel(struct taskl tasklet_kill(&ttimer->tasklet); } +#ifdef CONFIG_PREEMPT_RT_FULL +extern void softirq_early_init(void); +#else +static inline void softirq_early_init(void) { } +#endif + /* * Autoprobing for irqs: * Index: linux-4.4.25-rt35/include/linux/irq.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/irq.h +++ linux-4.4.25-rt35/include/linux/irq.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:75 @ enum irqchip_irq_state; * IRQ_IS_POLLED - Always polled by another interrupt. Exclude * it from the spurious interrupt detection * mechanism and from core side polling. + * IRQ_NO_SOFTIRQ_CALL - No softirq processing in the irq thread context (RT) * IRQ_DISABLE_UNLAZY - Disable lazy irq disable */ enum { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:103 @ enum { IRQ_PER_CPU_DEVID = (1 << 17), IRQ_IS_POLLED = (1 << 18), IRQ_DISABLE_UNLAZY = (1 << 19), + IRQ_NO_SOFTIRQ_CALL = (1 << 20), }; #define IRQF_MODIFY_MASK \ (IRQ_TYPE_SENSE_MASK | IRQ_NOPROBE | IRQ_NOREQUEST | \ IRQ_NOAUTOEN | IRQ_MOVE_PCNTXT | IRQ_LEVEL | IRQ_NO_BALANCING | \ IRQ_PER_CPU | IRQ_NESTED_THREAD | IRQ_NOTHREAD | IRQ_PER_CPU_DEVID | \ - IRQ_IS_POLLED | IRQ_DISABLE_UNLAZY) + IRQ_IS_POLLED | IRQ_DISABLE_UNLAZY | IRQ_NO_SOFTIRQ_CALL) #define IRQ_NO_BALANCING_MASK (IRQ_PER_CPU | IRQ_NO_BALANCING) Index: linux-4.4.25-rt35/include/linux/irq_work.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/irq_work.h +++ linux-4.4.25-rt35/include/linux/irq_work.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:19 @ #define IRQ_WORK_BUSY 2UL #define IRQ_WORK_FLAGS 3UL #define IRQ_WORK_LAZY 4UL /* Doesn't want IPI, wait for tick */ +#define IRQ_WORK_HARD_IRQ 8UL /* Run hard IRQ context, even on RT */ struct irq_work { unsigned long flags; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:55 @ static inline bool irq_work_needs_cpu(vo static inline void irq_work_run(void) { } #endif +#if defined(CONFIG_IRQ_WORK) && defined(CONFIG_PREEMPT_RT_FULL) +void irq_work_tick_soft(void); +#else +static inline void irq_work_tick_soft(void) { } +#endif + #endif /* _LINUX_IRQ_WORK_H */ Index: linux-4.4.25-rt35/include/linux/irqdesc.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/irqdesc.h +++ linux-4.4.25-rt35/include/linux/irqdesc.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:64 @ struct irq_desc { unsigned int irqs_unhandled; atomic_t threads_handled; int threads_handled_last; + u64 random_ip; raw_spinlock_t lock; struct cpumask *percpu_enabled; #ifdef CONFIG_SMP Index: linux-4.4.25-rt35/include/linux/irqflags.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/irqflags.h +++ linux-4.4.25-rt35/include/linux/irqflags.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:28 @ # define trace_softirqs_enabled(p) ((p)->softirqs_enabled) # define trace_hardirq_enter() do { current->hardirq_context++; } while (0) # define trace_hardirq_exit() do { current->hardirq_context--; } while (0) -# define lockdep_softirq_enter() do { current->softirq_context++; } while (0) -# define lockdep_softirq_exit() do { current->softirq_context--; } while (0) # define INIT_TRACE_IRQFLAGS .softirqs_enabled = 1, #else # define trace_hardirqs_on() do { } while (0) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:40 @ # define trace_softirqs_enabled(p) 0 # define trace_hardirq_enter() do { } while (0) # define trace_hardirq_exit() do { } while (0) +# define INIT_TRACE_IRQFLAGS +#endif + +#if defined(CONFIG_TRACE_IRQFLAGS) && !defined(CONFIG_PREEMPT_RT_FULL) +# define lockdep_softirq_enter() do { current->softirq_context++; } while (0) +# define lockdep_softirq_exit() do { current->softirq_context--; } while (0) +#else # define lockdep_softirq_enter() do { } while (0) # define lockdep_softirq_exit() do { } while (0) -# define INIT_TRACE_IRQFLAGS #endif #if defined(CONFIG_IRQSOFF_TRACER) || \ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:155 @ #define irqs_disabled_flags(flags) raw_irqs_disabled_flags(flags) +/* + * local_irq* variants depending on RT/!RT + */ +#ifdef CONFIG_PREEMPT_RT_FULL +# define local_irq_disable_nort() do { } while (0) +# define local_irq_enable_nort() do { } while (0) +# define local_irq_save_nort(flags) local_save_flags(flags) +# define local_irq_restore_nort(flags) (void)(flags) +# define local_irq_disable_rt() local_irq_disable() +# define local_irq_enable_rt() local_irq_enable() +#else +# define local_irq_disable_nort() local_irq_disable() +# define local_irq_enable_nort() local_irq_enable() +# define local_irq_save_nort(flags) local_irq_save(flags) +# define local_irq_restore_nort(flags) local_irq_restore(flags) +# define local_irq_disable_rt() do { } while (0) +# define local_irq_enable_rt() do { } while (0) +#endif + #endif Index: linux-4.4.25-rt35/include/linux/jbd2.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/jbd2.h +++ linux-4.4.25-rt35/include/linux/jbd2.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:355 @ static inline struct journal_head *bh2jh static inline void jbd_lock_bh_state(struct buffer_head *bh) { +#ifndef CONFIG_PREEMPT_RT_BASE bit_spin_lock(BH_State, &bh->b_state); +#else + spin_lock(&bh->b_state_lock); +#endif } static inline int jbd_trylock_bh_state(struct buffer_head *bh) { +#ifndef CONFIG_PREEMPT_RT_BASE return bit_spin_trylock(BH_State, &bh->b_state); +#else + return spin_trylock(&bh->b_state_lock); +#endif } static inline int jbd_is_locked_bh_state(struct buffer_head *bh) { +#ifndef CONFIG_PREEMPT_RT_BASE return bit_spin_is_locked(BH_State, &bh->b_state); +#else + return spin_is_locked(&bh->b_state_lock); +#endif } static inline void jbd_unlock_bh_state(struct buffer_head *bh) { +#ifndef CONFIG_PREEMPT_RT_BASE bit_spin_unlock(BH_State, &bh->b_state); +#else + spin_unlock(&bh->b_state_lock); +#endif } static inline void jbd_lock_bh_journal_head(struct buffer_head *bh) { +#ifndef CONFIG_PREEMPT_RT_BASE bit_spin_lock(BH_JournalHead, &bh->b_state); +#else + spin_lock(&bh->b_journal_head_lock); +#endif } static inline void jbd_unlock_bh_journal_head(struct buffer_head *bh) { +#ifndef CONFIG_PREEMPT_RT_BASE bit_spin_unlock(BH_JournalHead, &bh->b_state); +#else + spin_unlock(&bh->b_journal_head_lock); +#endif } #define J_ASSERT(assert) BUG_ON(!(assert)) Index: linux-4.4.25-rt35/include/linux/kdb.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/kdb.h +++ linux-4.4.25-rt35/include/linux/kdb.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:170 @ extern __printf(2, 0) int vkdb_printf(en extern __printf(1, 2) int kdb_printf(const char *, ...); typedef __printf(1, 2) int (*kdb_printf_t)(const char *, ...); +#define in_kdb_printk() (kdb_trap_printk) extern void kdb_init(int level); /* Access to kdb specific polling devices */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:205 @ extern int kdb_register_flags(char *, kd extern int kdb_unregister(char *); #else /* ! CONFIG_KGDB_KDB */ static inline __printf(1, 2) int kdb_printf(const char *fmt, ...) { return 0; } +#define in_kdb_printk() (0) static inline void kdb_init(int level) {} static inline int kdb_register(char *cmd, kdb_func_t func, char *usage, char *help, short minlen) { return 0; } Index: linux-4.4.25-rt35/include/linux/kernel.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/kernel.h +++ linux-4.4.25-rt35/include/linux/kernel.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:191 @ extern int _cond_resched(void); */ # define might_sleep() \ do { __might_sleep(__FILE__, __LINE__, 0); might_resched(); } while (0) + +# define might_sleep_no_state_check() \ + do { ___might_sleep(__FILE__, __LINE__, 0); might_resched(); } while (0) # define sched_annotate_sleep() (current->task_state_change = 0) #else static inline void ___might_sleep(const char *file, int line, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:201 @ extern int _cond_resched(void); static inline void __might_sleep(const char *file, int line, int preempt_offset) { } # define might_sleep() do { might_resched(); } while (0) +# define might_sleep_no_state_check() do { might_resched(); } while (0) # define sched_annotate_sleep() do { } while (0) #endif @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:262 @ extern long (*panic_blink)(int state); __printf(1, 2) void panic(const char *fmt, ...) __noreturn __cold; +void nmi_panic(struct pt_regs *regs, const char *msg); extern void oops_enter(void); extern void oops_exit(void); void print_oops_end_marker(void); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:456 @ extern int sysctl_panic_on_stackoverflow extern bool crash_kexec_post_notifiers; /* + * panic_cpu is used for synchronizing panic() and crash_kexec() execution. It + * holds a CPU number which is executing panic() currently. A value of + * PANIC_CPU_INVALID means no CPU has entered panic() or crash_kexec(). + */ +extern atomic_t panic_cpu; +#define PANIC_CPU_INVALID -1 + +/* * Only to be used by arch init code. If the user over-wrote the default * CONFIG_PANIC_TIMEOUT, honor it. */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:491 @ extern enum system_states { SYSTEM_HALT, SYSTEM_POWER_OFF, SYSTEM_RESTART, + SYSTEM_SUSPEND, } system_state; #define TAINT_PROPRIETARY_MODULE 0 Index: linux-4.4.25-rt35/include/linux/kvm_host.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/kvm_host.h +++ linux-4.4.25-rt35/include/linux/kvm_host.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:28 @ #include <linux/irqflags.h> #include <linux/context_tracking.h> #include <linux/irqbypass.h> +#include <linux/swait.h> #include <asm/signal.h> #include <linux/kvm.h> @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:247 @ struct kvm_vcpu { int fpu_active; int guest_fpu_loaded, guest_xcr0_loaded; unsigned char fpu_counter; - wait_queue_head_t wq; + struct swait_queue_head wq; struct pid *pid; int sigset_active; sigset_t sigset; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:798 @ static inline bool kvm_arch_has_assigned } #endif -static inline wait_queue_head_t *kvm_arch_vcpu_wq(struct kvm_vcpu *vcpu) +static inline struct swait_queue_head *kvm_arch_vcpu_wq(struct kvm_vcpu *vcpu) { #ifdef __KVM_HAVE_ARCH_WQP return vcpu->arch.wqp; Index: linux-4.4.25-rt35/include/linux/lglock.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/lglock.h +++ linux-4.4.25-rt35/include/linux/lglock.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:37 @ #endif struct lglock { +#ifdef CONFIG_PREEMPT_RT_FULL + struct rt_mutex __percpu *lock; +#else arch_spinlock_t __percpu *lock; +#endif #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lock_class_key lock_key; struct lockdep_map lock_dep_map; #endif }; +#ifdef CONFIG_PREEMPT_RT_FULL +# define DEFINE_LGLOCK(name) \ + static DEFINE_PER_CPU(struct rt_mutex, name ## _lock) \ + = __RT_MUTEX_INITIALIZER( name ## _lock); \ + struct lglock name = { .lock = &name ## _lock } + +# define DEFINE_STATIC_LGLOCK(name) \ + static DEFINE_PER_CPU(struct rt_mutex, name ## _lock) \ + = __RT_MUTEX_INITIALIZER( name ## _lock); \ + static struct lglock name = { .lock = &name ## _lock } + +#else + #define DEFINE_LGLOCK(name) \ static DEFINE_PER_CPU(arch_spinlock_t, name ## _lock) \ = __ARCH_SPIN_LOCK_UNLOCKED; \ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:70 @ struct lglock { static DEFINE_PER_CPU(arch_spinlock_t, name ## _lock) \ = __ARCH_SPIN_LOCK_UNLOCKED; \ static struct lglock name = { .lock = &name ## _lock } +#endif void lg_lock_init(struct lglock *lg, char *name); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:85 @ void lg_double_unlock(struct lglock *lg, void lg_global_lock(struct lglock *lg); void lg_global_unlock(struct lglock *lg); +#ifndef CONFIG_PREEMPT_RT_FULL +#define lg_global_trylock_relax(name) lg_global_lock(name) +#else +void lg_global_trylock_relax(struct lglock *lg); +#endif + #else /* When !CONFIG_SMP, map lglock to spinlock */ #define lglock spinlock Index: linux-4.4.25-rt35/include/linux/list_bl.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/list_bl.h +++ linux-4.4.25-rt35/include/linux/list_bl.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:5 @ #define _LINUX_LIST_BL_H #include <linux/list.h> +#include <linux/spinlock.h> #include <linux/bit_spinlock.h> /* @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:36 @ struct hlist_bl_head { struct hlist_bl_node *first; +#ifdef CONFIG_PREEMPT_RT_BASE + raw_spinlock_t lock; +#endif }; struct hlist_bl_node { struct hlist_bl_node *next, **pprev; }; -#define INIT_HLIST_BL_HEAD(ptr) \ - ((ptr)->first = NULL) + +#ifdef CONFIG_PREEMPT_RT_BASE +#define INIT_HLIST_BL_HEAD(h) \ +do { \ + (h)->first = NULL; \ + raw_spin_lock_init(&(h)->lock); \ +} while (0) +#else +#define INIT_HLIST_BL_HEAD(h) (h)->first = NULL +#endif static inline void INIT_HLIST_BL_NODE(struct hlist_bl_node *h) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:133 @ static inline void hlist_bl_del_init(str static inline void hlist_bl_lock(struct hlist_bl_head *b) { +#ifndef CONFIG_PREEMPT_RT_BASE bit_spin_lock(0, (unsigned long *)b); +#else + raw_spin_lock(&b->lock); +#if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK) + __set_bit(0, (unsigned long *)b); +#endif +#endif } static inline void hlist_bl_unlock(struct hlist_bl_head *b) { +#ifndef CONFIG_PREEMPT_RT_BASE __bit_spin_unlock(0, (unsigned long *)b); +#else +#if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK) + __clear_bit(0, (unsigned long *)b); +#endif + raw_spin_unlock(&b->lock); +#endif } static inline bool hlist_bl_is_locked(struct hlist_bl_head *b) Index: linux-4.4.25-rt35/include/linux/locallock.h =================================================================== --- /dev/null +++ linux-4.4.25-rt35/include/linux/locallock.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +#ifndef _LINUX_LOCALLOCK_H +#define _LINUX_LOCALLOCK_H + +#include <linux/percpu.h> +#include <linux/spinlock.h> + +#ifdef CONFIG_PREEMPT_RT_BASE + +#ifdef CONFIG_DEBUG_SPINLOCK +# define LL_WARN(cond) WARN_ON(cond) +#else +# define LL_WARN(cond) do { } while (0) +#endif + +/* + * per cpu lock based substitute for local_irq_*() + */ +struct local_irq_lock { + spinlock_t lock; + struct task_struct *owner; + int nestcnt; + unsigned long flags; +}; + +#define DEFINE_LOCAL_IRQ_LOCK(lvar) \ + DEFINE_PER_CPU(struct local_irq_lock, lvar) = { \ + .lock = __SPIN_LOCK_UNLOCKED((lvar).lock) } + +#define DECLARE_LOCAL_IRQ_LOCK(lvar) \ + DECLARE_PER_CPU(struct local_irq_lock, lvar) + +#define local_irq_lock_init(lvar) \ + do { \ + int __cpu; \ + for_each_possible_cpu(__cpu) \ + spin_lock_init(&per_cpu(lvar, __cpu).lock); \ + } while (0) + +/* + * spin_lock|trylock|unlock_local flavour that does not migrate disable + * used for __local_lock|trylock|unlock where get_local_var/put_local_var + * already takes care of the migrate_disable/enable + * for CONFIG_PREEMPT_BASE map to the normal spin_* calls. + */ +#ifdef CONFIG_PREEMPT_RT_FULL +# define spin_lock_local(lock) rt_spin_lock__no_mg(lock) +# define spin_trylock_local(lock) rt_spin_trylock__no_mg(lock) +# define spin_unlock_local(lock) rt_spin_unlock__no_mg(lock) +#else +# define spin_lock_local(lock) spin_lock(lock) +# define spin_trylock_local(lock) spin_trylock(lock) +# define spin_unlock_local(lock) spin_unlock(lock) +#endif + +static inline void __local_lock(struct local_irq_lock *lv) +{ + if (lv->owner != current) { + spin_lock_local(&lv->lock); + LL_WARN(lv->owner); + LL_WARN(lv->nestcnt); + lv->owner = current; + } + lv->nestcnt++; +} + +#define local_lock(lvar) \ + do { __local_lock(&get_local_var(lvar)); } while (0) + +#define local_lock_on(lvar, cpu) \ + do { __local_lock(&per_cpu(lvar, cpu)); } while (0) + +static inline int __local_trylock(struct local_irq_lock *lv) +{ + if (lv->owner != current && spin_trylock_local(&lv->lock)) { + LL_WARN(lv->owner); + LL_WARN(lv->nestcnt); + lv->owner = current; + lv->nestcnt = 1; + return 1; + } + return 0; +} + +#define local_trylock(lvar) \ + ({ \ + int __locked; \ + __locked = __local_trylock(&get_local_var(lvar)); \ + if (!__locked) \ + put_local_var(lvar); \ + __locked; \ + }) + +static inline void __local_unlock(struct local_irq_lock *lv) +{ + LL_WARN(lv->nestcnt == 0); + LL_WARN(lv->owner != current); + if (--lv->nestcnt) + return; + + lv->owner = NULL; + spin_unlock_local(&lv->lock); +} + +#define local_unlock(lvar) \ + do { \ + __local_unlock(this_cpu_ptr(&lvar)); \ + put_local_var(lvar); \ + } while (0) + +#define local_unlock_on(lvar, cpu) \ + do { __local_unlock(&per_cpu(lvar, cpu)); } while (0) + +static inline void __local_lock_irq(struct local_irq_lock *lv) +{ + spin_lock_irqsave(&lv->lock, lv->flags); + LL_WARN(lv->owner); + LL_WARN(lv->nestcnt); + lv->owner = current; + lv->nestcnt = 1; +} + +#define local_lock_irq(lvar) \ + do { __local_lock_irq(&get_local_var(lvar)); } while (0) + +#define local_lock_irq_on(lvar, cpu) \ + do { __local_lock_irq(&per_cpu(lvar, cpu)); } while (0) + +static inline void __local_unlock_irq(struct local_irq_lock *lv) +{ + LL_WARN(!lv->nestcnt); + LL_WARN(lv->owner != current); + lv->owner = NULL; + lv->nestcnt = 0; + spin_unlock_irq(&lv->lock); +} + +#define local_unlock_irq(lvar) \ + do { \ + __local_unlock_irq(this_cpu_ptr(&lvar)); \ + put_local_var(lvar); \ + } while (0) + +#define local_unlock_irq_on(lvar, cpu) \ + do { \ + __local_unlock_irq(&per_cpu(lvar, cpu)); \ + } while (0) + +static inline int __local_lock_irqsave(struct local_irq_lock *lv) +{ + if (lv->owner != current) { + __local_lock_irq(lv); + return 0; + } else { + lv->nestcnt++; + return 1; + } +} + +#define local_lock_irqsave(lvar, _flags) \ + do { \ + if (__local_lock_irqsave(&get_local_var(lvar))) \ + put_local_var(lvar); \ + _flags = __this_cpu_read(lvar.flags); \ + } while (0) + +#define local_lock_irqsave_on(lvar, _flags, cpu) \ + do { \ + __local_lock_irqsave(&per_cpu(lvar, cpu)); \ + _flags = per_cpu(lvar, cpu).flags; \ + } while (0) + +static inline int __local_unlock_irqrestore(struct local_irq_lock *lv, + unsigned long flags) +{ + LL_WARN(!lv->nestcnt); + LL_WARN(lv->owner != current); + if (--lv->nestcnt) + return 0; + + lv->owner = NULL; + spin_unlock_irqrestore(&lv->lock, lv->flags); + return 1; +} + +#define local_unlock_irqrestore(lvar, flags) \ + do { \ + if (__local_unlock_irqrestore(this_cpu_ptr(&lvar), flags)) \ + put_local_var(lvar); \ + } while (0) + +#define local_unlock_irqrestore_on(lvar, flags, cpu) \ + do { \ + __local_unlock_irqrestore(&per_cpu(lvar, cpu), flags); \ + } while (0) + +#define local_spin_trylock_irq(lvar, lock) \ + ({ \ + int __locked; \ + local_lock_irq(lvar); \ + __locked = spin_trylock(lock); \ + if (!__locked) \ + local_unlock_irq(lvar); \ + __locked; \ + }) + +#define local_spin_lock_irq(lvar, lock) \ + do { \ + local_lock_irq(lvar); \ + spin_lock(lock); \ + } while (0) + +#define local_spin_unlock_irq(lvar, lock) \ + do { \ + spin_unlock(lock); \ + local_unlock_irq(lvar); \ + } while (0) + +#define local_spin_lock_irqsave(lvar, lock, flags) \ + do { \ + local_lock_irqsave(lvar, flags); \ + spin_lock(lock); \ + } while (0) + +#define local_spin_unlock_irqrestore(lvar, lock, flags) \ + do { \ + spin_unlock(lock); \ + local_unlock_irqrestore(lvar, flags); \ + } while (0) + +#define get_locked_var(lvar, var) \ + (*({ \ + local_lock(lvar); \ + this_cpu_ptr(&var); \ + })) + +#define put_locked_var(lvar, var) local_unlock(lvar); + +#define local_lock_cpu(lvar) \ + ({ \ + local_lock(lvar); \ + smp_processor_id(); \ + }) + +#define local_unlock_cpu(lvar) local_unlock(lvar) + +#else /* PREEMPT_RT_BASE */ + +#define DEFINE_LOCAL_IRQ_LOCK(lvar) __typeof__(const int) lvar +#define DECLARE_LOCAL_IRQ_LOCK(lvar) extern __typeof__(const int) lvar + +static inline void local_irq_lock_init(int lvar) { } + +#define local_lock(lvar) preempt_disable() +#define local_unlock(lvar) preempt_enable() +#define local_lock_irq(lvar) local_irq_disable() +#define local_unlock_irq(lvar) local_irq_enable() +#define local_lock_irqsave(lvar, flags) local_irq_save(flags) +#define local_unlock_irqrestore(lvar, flags) local_irq_restore(flags) + +#define local_spin_trylock_irq(lvar, lock) spin_trylock_irq(lock) +#define local_spin_lock_irq(lvar, lock) spin_lock_irq(lock) +#define local_spin_unlock_irq(lvar, lock) spin_unlock_irq(lock) +#define local_spin_lock_irqsave(lvar, lock, flags) \ + spin_lock_irqsave(lock, flags) +#define local_spin_unlock_irqrestore(lvar, lock, flags) \ + spin_unlock_irqrestore(lock, flags) + +#define get_locked_var(lvar, var) get_cpu_var(var) +#define put_locked_var(lvar, var) put_cpu_var(var) + +#define local_lock_cpu(lvar) get_cpu() +#define local_unlock_cpu(lvar) put_cpu() + +#endif + +#endif Index: linux-4.4.25-rt35/include/linux/mm_types.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/mm_types.h +++ linux-4.4.25-rt35/include/linux/mm_types.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:14 @ #include <linux/completion.h> #include <linux/cpumask.h> #include <linux/uprobes.h> +#include <linux/rcupdate.h> #include <linux/page-flags-layout.h> #include <asm/page.h> #include <asm/mmu.h> @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:508 @ struct mm_struct { bool tlb_flush_pending; #endif struct uprobes_state uprobes_state; +#ifdef CONFIG_PREEMPT_RT_BASE + struct rcu_head delayed_drop; +#endif #ifdef CONFIG_X86_INTEL_MPX /* address of the bounds directory */ void __user *bd_addr; Index: linux-4.4.25-rt35/include/linux/mutex.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/mutex.h +++ linux-4.4.25-rt35/include/linux/mutex.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:22 @ #include <asm/processor.h> #include <linux/osq_lock.h> +#ifdef CONFIG_DEBUG_LOCK_ALLOC +# define __DEP_MAP_MUTEX_INITIALIZER(lockname) \ + , .dep_map = { .name = #lockname } +#else +# define __DEP_MAP_MUTEX_INITIALIZER(lockname) +#endif + +#ifdef CONFIG_PREEMPT_RT_FULL +# include <linux/mutex_rt.h> +#else + /* * Simple, straightforward mutexes with strict semantics: * @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:113 @ do { \ static inline void mutex_destroy(struct mutex *lock) {} #endif -#ifdef CONFIG_DEBUG_LOCK_ALLOC -# define __DEP_MAP_MUTEX_INITIALIZER(lockname) \ - , .dep_map = { .name = #lockname } -#else -# define __DEP_MAP_MUTEX_INITIALIZER(lockname) -#endif - #define __MUTEX_INITIALIZER(lockname) \ { .count = ATOMIC_INIT(1) \ , .wait_lock = __SPIN_LOCK_UNLOCKED(lockname.wait_lock) \ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:180 @ extern int __must_check mutex_lock_killa extern int mutex_trylock(struct mutex *lock); extern void mutex_unlock(struct mutex *lock); +#endif /* !PREEMPT_RT_FULL */ + extern int atomic_dec_and_mutex_lock(atomic_t *cnt, struct mutex *lock); #endif /* __LINUX_MUTEX_H */ Index: linux-4.4.25-rt35/include/linux/mutex_rt.h =================================================================== --- /dev/null +++ linux-4.4.25-rt35/include/linux/mutex_rt.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +#ifndef __LINUX_MUTEX_RT_H +#define __LINUX_MUTEX_RT_H + +#ifndef __LINUX_MUTEX_H +#error "Please include mutex.h" +#endif + +#include <linux/rtmutex.h> + +/* FIXME: Just for __lockfunc */ +#include <linux/spinlock.h> + +struct mutex { + struct rt_mutex lock; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; +#endif +}; + +#define __MUTEX_INITIALIZER(mutexname) \ + { \ + .lock = __RT_MUTEX_INITIALIZER(mutexname.lock) \ + __DEP_MAP_MUTEX_INITIALIZER(mutexname) \ + } + +#define DEFINE_MUTEX(mutexname) \ + struct mutex mutexname = __MUTEX_INITIALIZER(mutexname) + +extern void __mutex_do_init(struct mutex *lock, const char *name, struct lock_class_key *key); +extern void __lockfunc _mutex_lock(struct mutex *lock); +extern int __lockfunc _mutex_lock_interruptible(struct mutex *lock); +extern int __lockfunc _mutex_lock_killable(struct mutex *lock); +extern void __lockfunc _mutex_lock_nested(struct mutex *lock, int subclass); +extern void __lockfunc _mutex_lock_nest_lock(struct mutex *lock, struct lockdep_map *nest_lock); +extern int __lockfunc _mutex_lock_interruptible_nested(struct mutex *lock, int subclass); +extern int __lockfunc _mutex_lock_killable_nested(struct mutex *lock, int subclass); +extern int __lockfunc _mutex_trylock(struct mutex *lock); +extern void __lockfunc _mutex_unlock(struct mutex *lock); + +#define mutex_is_locked(l) rt_mutex_is_locked(&(l)->lock) +#define mutex_lock(l) _mutex_lock(l) +#define mutex_lock_interruptible(l) _mutex_lock_interruptible(l) +#define mutex_lock_killable(l) _mutex_lock_killable(l) +#define mutex_trylock(l) _mutex_trylock(l) +#define mutex_unlock(l) _mutex_unlock(l) +#define mutex_destroy(l) rt_mutex_destroy(&(l)->lock) + +#ifdef CONFIG_DEBUG_LOCK_ALLOC +# define mutex_lock_nested(l, s) _mutex_lock_nested(l, s) +# define mutex_lock_interruptible_nested(l, s) \ + _mutex_lock_interruptible_nested(l, s) +# define mutex_lock_killable_nested(l, s) \ + _mutex_lock_killable_nested(l, s) + +# define mutex_lock_nest_lock(lock, nest_lock) \ +do { \ + typecheck(struct lockdep_map *, &(nest_lock)->dep_map); \ + _mutex_lock_nest_lock(lock, &(nest_lock)->dep_map); \ +} while (0) + +#else +# define mutex_lock_nested(l, s) _mutex_lock(l) +# define mutex_lock_interruptible_nested(l, s) \ + _mutex_lock_interruptible(l) +# define mutex_lock_killable_nested(l, s) \ + _mutex_lock_killable(l) +# define mutex_lock_nest_lock(lock, nest_lock) mutex_lock(lock) +#endif + +# define mutex_init(mutex) \ +do { \ + static struct lock_class_key __key; \ + \ + rt_mutex_init(&(mutex)->lock); \ + __mutex_do_init((mutex), #mutex, &__key); \ +} while (0) + +# define __mutex_init(mutex, name, key) \ +do { \ + rt_mutex_init(&(mutex)->lock); \ + __mutex_do_init((mutex), name, key); \ +} while (0) + +#endif Index: linux-4.4.25-rt35/include/linux/netdevice.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/netdevice.h +++ linux-4.4.25-rt35/include/linux/netdevice.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2289 @ void netdev_freemem(struct net_device *d void synchronize_net(void); int init_dummy_netdev(struct net_device *dev); +#ifdef CONFIG_PREEMPT_RT_FULL +static inline int dev_recursion_level(void) +{ + return current->xmit_recursion; +} + +#else + DECLARE_PER_CPU(int, xmit_recursion); static inline int dev_recursion_level(void) { return this_cpu_read(xmit_recursion); } +#endif struct net_device *dev_get_by_index(struct net *net, int ifindex); struct net_device *__dev_get_by_index(struct net *net, int ifindex); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2613 @ struct softnet_data { unsigned int dropped; struct sk_buff_head input_pkt_queue; struct napi_struct backlog; + struct sk_buff_head tofree_queue; }; Index: linux-4.4.25-rt35/include/linux/netfilter/x_tables.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/netfilter/x_tables.h +++ linux-4.4.25-rt35/include/linux/netfilter/x_tables.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:7 @ #include <linux/netdevice.h> #include <linux/static_key.h> +#include <linux/locallock.h> #include <uapi/linux/netfilter/x_tables.h> /** @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:293 @ void xt_free_table_info(struct xt_table_ */ DECLARE_PER_CPU(seqcount_t, xt_recseq); +DECLARE_LOCAL_IRQ_LOCK(xt_write_lock); + /* xt_tee_enabled - true if x_tables needs to handle reentrancy * * Enabled if current ip(6)tables ruleset has at least one -j TEE rule. @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:315 @ static inline unsigned int xt_write_recs { unsigned int addend; + /* RT protection */ + local_lock(xt_write_lock); + /* * Low order bit of sequence is set if we already * called xt_write_recseq_begin(). @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:348 @ static inline void xt_write_recseq_end(u /* this is kind of a write_seqcount_end(), but addend is 0 or 1 */ smp_wmb(); __this_cpu_add(xt_recseq.sequence, addend); + local_unlock(xt_write_lock); } /* Index: linux-4.4.25-rt35/include/linux/notifier.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/notifier.h +++ linux-4.4.25-rt35/include/linux/notifier.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:9 @ * * Alan Cox <Alan.Cox@linux.org> */ - + #ifndef _LINUX_NOTIFIER_H #define _LINUX_NOTIFIER_H #include <linux/errno.h> @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:45 @ * in srcu_notifier_call_chain(): no cache bounces and no memory barriers. * As compensation, srcu_notifier_chain_unregister() is rather expensive. * SRCU notifier chains should be used when the chain will be called very - * often but notifier_blocks will seldom be removed. Also, SRCU notifier - * chains are slightly more difficult to use because they require special - * runtime initialization. + * often but notifier_blocks will seldom be removed. */ typedef int (*notifier_fn_t)(struct notifier_block *nb, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:89 @ struct srcu_notifier_head { (name)->head = NULL; \ } while (0) -/* srcu_notifier_heads must be initialized and cleaned up dynamically */ +/* srcu_notifier_heads must be cleaned up dynamically */ extern void srcu_init_notifier_head(struct srcu_notifier_head *nh); #define srcu_cleanup_notifier_head(name) \ cleanup_srcu_struct(&(name)->srcu); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:102 @ extern void srcu_init_notifier_head(stru .head = NULL } #define RAW_NOTIFIER_INIT(name) { \ .head = NULL } -/* srcu_notifier_heads cannot be initialized statically */ + +#define SRCU_NOTIFIER_INIT(name, pcpu) \ + { \ + .mutex = __MUTEX_INITIALIZER(name.mutex), \ + .head = NULL, \ + .srcu = __SRCU_STRUCT_INIT(name.srcu, pcpu), \ + } #define ATOMIC_NOTIFIER_HEAD(name) \ struct atomic_notifier_head name = \ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:120 @ extern void srcu_init_notifier_head(stru struct raw_notifier_head name = \ RAW_NOTIFIER_INIT(name) +#define _SRCU_NOTIFIER_HEAD(name, mod) \ + static DEFINE_PER_CPU(struct srcu_struct_array, \ + name##_head_srcu_array); \ + mod struct srcu_notifier_head name = \ + SRCU_NOTIFIER_INIT(name, name##_head_srcu_array) + +#define SRCU_NOTIFIER_HEAD(name) \ + _SRCU_NOTIFIER_HEAD(name, ) + +#define SRCU_NOTIFIER_HEAD_STATIC(name) \ + _SRCU_NOTIFIER_HEAD(name, static) + #ifdef __KERNEL__ extern int atomic_notifier_chain_register(struct atomic_notifier_head *nh, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:201 @ static inline int notifier_to_errno(int /* * Declared notifiers so far. I can imagine quite a few more chains - * over time (eg laptop power reset chains, reboot chain (to clean + * over time (eg laptop power reset chains, reboot chain (to clean * device units up), device [un]mount chain, module load/unload chain, - * low memory chain, screenblank chain (for plug in modular screenblankers) + * low memory chain, screenblank chain (for plug in modular screenblankers) * VC switch chains (for loadable kernel svgalib VC switch helpers) etc... */ - + /* CPU notfiers are defined in include/linux/cpu.h. */ /* netdevice notifiers are defined in include/linux/netdevice.h */ Index: linux-4.4.25-rt35/include/linux/percpu.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/percpu.h +++ linux-4.4.25-rt35/include/linux/percpu.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:27 @ PERCPU_MODULE_RESERVE) #endif +#ifdef CONFIG_PREEMPT_RT_FULL + +#define get_local_var(var) (*({ \ + migrate_disable(); \ + this_cpu_ptr(&var); })) + +#define put_local_var(var) do { \ + (void)&(var); \ + migrate_enable(); \ +} while (0) + +# define get_local_ptr(var) ({ \ + migrate_disable(); \ + this_cpu_ptr(var); }) + +# define put_local_ptr(var) do { \ + (void)(var); \ + migrate_enable(); \ +} while (0) + +#else + +#define get_local_var(var) get_cpu_var(var) +#define put_local_var(var) put_cpu_var(var) +#define get_local_ptr(var) get_cpu_ptr(var) +#define put_local_ptr(var) put_cpu_ptr(var) + +#endif + /* minimum unit size, also is the maximum supported allocation size */ #define PCPU_MIN_UNIT_SIZE PFN_ALIGN(32 << 10) Index: linux-4.4.25-rt35/include/linux/pid.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/pid.h +++ linux-4.4.25-rt35/include/linux/pid.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:5 @ #define _LINUX_PID_H #include <linux/rcupdate.h> +#include <linux/atomic.h> enum pid_type { Index: linux-4.4.25-rt35/include/linux/preempt.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/preempt.h +++ linux-4.4.25-rt35/include/linux/preempt.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:53 @ #define HARDIRQ_OFFSET (1UL << HARDIRQ_SHIFT) #define NMI_OFFSET (1UL << NMI_SHIFT) -#define SOFTIRQ_DISABLE_OFFSET (2 * SOFTIRQ_OFFSET) +#ifndef CONFIG_PREEMPT_RT_FULL +# define SOFTIRQ_DISABLE_OFFSET (2 * SOFTIRQ_OFFSET) +#else +# define SOFTIRQ_DISABLE_OFFSET (0) +#endif /* We use the MSB mostly because its available */ #define PREEMPT_NEED_RESCHED 0x80000000 @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:66 @ #include <asm/preempt.h> #define hardirq_count() (preempt_count() & HARDIRQ_MASK) -#define softirq_count() (preempt_count() & SOFTIRQ_MASK) #define irq_count() (preempt_count() & (HARDIRQ_MASK | SOFTIRQ_MASK \ | NMI_MASK)) +#ifndef CONFIG_PREEMPT_RT_FULL +# define softirq_count() (preempt_count() & SOFTIRQ_MASK) +# define in_serving_softirq() (softirq_count() & SOFTIRQ_OFFSET) +#else +# define softirq_count() (0UL) +extern int in_serving_softirq(void); +#endif /* * Are we doing bottom half or hardware interrupt processing? @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:85 @ #define in_irq() (hardirq_count()) #define in_softirq() (softirq_count()) #define in_interrupt() (irq_count()) -#define in_serving_softirq() (softirq_count() & SOFTIRQ_OFFSET) /* * Are we in NMI context? @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:103 @ /* * The preempt_count offset after spin_lock() */ +#if !defined(CONFIG_PREEMPT_RT_FULL) #define PREEMPT_LOCK_OFFSET PREEMPT_DISABLE_OFFSET +#else +#define PREEMPT_LOCK_OFFSET 0 +#endif /* * The preempt_count offset needed for things like: @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:156 @ extern void preempt_count_sub(int val); #define preempt_count_inc() preempt_count_add(1) #define preempt_count_dec() preempt_count_sub(1) +#ifdef CONFIG_PREEMPT_LAZY +#define add_preempt_lazy_count(val) do { preempt_lazy_count() += (val); } while (0) +#define sub_preempt_lazy_count(val) do { preempt_lazy_count() -= (val); } while (0) +#define inc_preempt_lazy_count() add_preempt_lazy_count(1) +#define dec_preempt_lazy_count() sub_preempt_lazy_count(1) +#define preempt_lazy_count() (current_thread_info()->preempt_lazy_count) +#else +#define add_preempt_lazy_count(val) do { } while (0) +#define sub_preempt_lazy_count(val) do { } while (0) +#define inc_preempt_lazy_count() do { } while (0) +#define dec_preempt_lazy_count() do { } while (0) +#define preempt_lazy_count() (0) +#endif + #ifdef CONFIG_PREEMPT_COUNT #define preempt_disable() \ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:178 @ do { \ barrier(); \ } while (0) +#define preempt_lazy_disable() \ +do { \ + inc_preempt_lazy_count(); \ + barrier(); \ +} while (0) + #define sched_preempt_enable_no_resched() \ do { \ barrier(); \ preempt_count_dec(); \ } while (0) -#define preempt_enable_no_resched() sched_preempt_enable_no_resched() +#ifdef CONFIG_PREEMPT_RT_BASE +# define preempt_enable_no_resched() sched_preempt_enable_no_resched() +# define preempt_check_resched_rt() preempt_check_resched() +#else +# define preempt_enable_no_resched() preempt_enable() +# define preempt_check_resched_rt() barrier(); +#endif #define preemptible() (preempt_count() == 0 && !irqs_disabled()) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:221 @ do { \ __preempt_schedule(); \ } while (0) +#define preempt_lazy_enable() \ +do { \ + dec_preempt_lazy_count(); \ + barrier(); \ + preempt_check_resched(); \ +} while (0) + #else /* !CONFIG_PREEMPT */ #define preempt_enable() \ do { \ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:273 @ do { \ #define preempt_disable_notrace() barrier() #define preempt_enable_no_resched_notrace() barrier() #define preempt_enable_notrace() barrier() +#define preempt_check_resched_rt() barrier() #define preemptible() 0 #endif /* CONFIG_PREEMPT_COUNT */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:294 @ do { \ } while (0) #define preempt_fold_need_resched() \ do { \ - if (tif_need_resched()) \ + if (tif_need_resched_now()) \ set_preempt_need_resched(); \ } while (0) +#ifdef CONFIG_PREEMPT_RT_FULL +# define preempt_disable_rt() preempt_disable() +# define preempt_enable_rt() preempt_enable() +# define preempt_disable_nort() barrier() +# define preempt_enable_nort() barrier() +# ifdef CONFIG_SMP + extern void migrate_disable(void); + extern void migrate_enable(void); +# else /* CONFIG_SMP */ +# define migrate_disable() barrier() +# define migrate_enable() barrier() +# endif /* CONFIG_SMP */ +#else +# define preempt_disable_rt() barrier() +# define preempt_enable_rt() barrier() +# define preempt_disable_nort() preempt_disable() +# define preempt_enable_nort() preempt_enable() +# define migrate_disable() preempt_disable() +# define migrate_enable() preempt_enable() +#endif + #ifdef CONFIG_PREEMPT_NOTIFIERS struct preempt_notifier; Index: linux-4.4.25-rt35/include/linux/printk.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/printk.h +++ linux-4.4.25-rt35/include/linux/printk.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:120 @ int no_printk(const char *fmt, ...) #ifdef CONFIG_EARLY_PRINTK extern asmlinkage __printf(1, 2) void early_printk(const char *fmt, ...); +extern void printk_kill(void); #else static inline __printf(1, 2) __cold void early_printk(const char *s, ...) { } +static inline void printk_kill(void) { } #endif typedef __printf(1, 0) int (*printk_func_t)(const char *fmt, va_list args); Index: linux-4.4.25-rt35/include/linux/radix-tree.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/radix-tree.h +++ linux-4.4.25-rt35/include/linux/radix-tree.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:280 @ radix_tree_gang_lookup(struct radix_tree unsigned int radix_tree_gang_lookup_slot(struct radix_tree_root *root, void ***results, unsigned long *indices, unsigned long first_index, unsigned int max_items); +#ifndef CONFIG_PREEMPT_RT_FULL int radix_tree_preload(gfp_t gfp_mask); int radix_tree_maybe_preload(gfp_t gfp_mask); +#else +static inline int radix_tree_preload(gfp_t gm) { return 0; } +static inline int radix_tree_maybe_preload(gfp_t gfp_mask) { return 0; } +#endif void radix_tree_init(void); void *radix_tree_tag_set(struct radix_tree_root *root, unsigned long index, unsigned int tag); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:311 @ unsigned long radix_tree_locate_item(str static inline void radix_tree_preload_end(void) { - preempt_enable(); + preempt_enable_nort(); } /** Index: linux-4.4.25-rt35/include/linux/random.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/random.h +++ linux-4.4.25-rt35/include/linux/random.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:23 @ struct random_ready_callback { extern void add_device_randomness(const void *, unsigned int); extern void add_input_randomness(unsigned int type, unsigned int code, unsigned int value); -extern void add_interrupt_randomness(int irq, int irq_flags); +extern void add_interrupt_randomness(int irq, int irq_flags, __u64 ip); extern void get_random_bytes(void *buf, int nbytes); extern int add_random_ready_callback(struct random_ready_callback *rdy); Index: linux-4.4.25-rt35/include/linux/rbtree.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/rbtree.h +++ linux-4.4.25-rt35/include/linux/rbtree.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:34 @ #include <linux/kernel.h> #include <linux/stddef.h> -#include <linux/rcupdate.h> struct rb_node { unsigned long __rb_parent_color; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:88 @ static inline void rb_link_node(struct r *rb_link = node; } -static inline void rb_link_node_rcu(struct rb_node *node, struct rb_node *parent, - struct rb_node **rb_link) -{ - node->__rb_parent_color = (unsigned long)parent; - node->rb_left = node->rb_right = NULL; - - rcu_assign_pointer(*rb_link, node); -} +void rb_link_node_rcu(struct rb_node *node, struct rb_node *parent, + struct rb_node **rb_link); #define rb_entry_safe(ptr, type, member) \ ({ typeof(ptr) ____ptr = (ptr); \ Index: linux-4.4.25-rt35/include/linux/rcupdate.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/rcupdate.h +++ linux-4.4.25-rt35/include/linux/rcupdate.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:172 @ void call_rcu(struct rcu_head *head, #endif /* #else #ifdef CONFIG_PREEMPT_RCU */ +#ifdef CONFIG_PREEMPT_RT_FULL +#define call_rcu_bh call_rcu +#else /** * call_rcu_bh() - Queue an RCU for invocation after a quicker grace period. * @head: structure to be used for queueing the RCU updates. @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:198 @ void call_rcu(struct rcu_head *head, */ void call_rcu_bh(struct rcu_head *head, rcu_callback_t func); +#endif /** * call_rcu_sched() - Queue an RCU for invocation after sched grace period. @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:299 @ void synchronize_rcu(void); * types of kernel builds, the rcu_read_lock() nesting depth is unknowable. */ #define rcu_preempt_depth() (current->rcu_read_lock_nesting) +#ifndef CONFIG_PREEMPT_RT_FULL +#define sched_rcu_preempt_depth() rcu_preempt_depth() +#else +static inline int sched_rcu_preempt_depth(void) { return 0; } +#endif #else /* #ifdef CONFIG_PREEMPT_RCU */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:329 @ static inline int rcu_preempt_depth(void return 0; } +#define sched_rcu_preempt_depth() rcu_preempt_depth() + #endif /* #else #ifdef CONFIG_PREEMPT_RCU */ /* Internal to kernel */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:503 @ extern struct lockdep_map rcu_callback_m int debug_lockdep_rcu_enabled(void); int rcu_read_lock_held(void); +#ifdef CONFIG_PREEMPT_RT_FULL +static inline int rcu_read_lock_bh_held(void) +{ + return rcu_read_lock_held(); +} +#else int rcu_read_lock_bh_held(void); +#endif /** * rcu_read_lock_sched_held() - might we be in RCU-sched read-side critical section? @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:958 @ static inline void rcu_read_unlock(void) static inline void rcu_read_lock_bh(void) { local_bh_disable(); +#ifdef CONFIG_PREEMPT_RT_FULL + rcu_read_lock(); +#else __acquire(RCU_BH); rcu_lock_acquire(&rcu_bh_lock_map); RCU_LOCKDEP_WARN(!rcu_is_watching(), "rcu_read_lock_bh() used illegally while idle"); +#endif } /* @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:975 @ static inline void rcu_read_lock_bh(void */ static inline void rcu_read_unlock_bh(void) { +#ifdef CONFIG_PREEMPT_RT_FULL + rcu_read_unlock(); +#else RCU_LOCKDEP_WARN(!rcu_is_watching(), "rcu_read_unlock_bh() used illegally while idle"); rcu_lock_release(&rcu_bh_lock_map); __release(RCU_BH); +#endif local_bh_enable(); } Index: linux-4.4.25-rt35/include/linux/rcutree.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/rcutree.h +++ linux-4.4.25-rt35/include/linux/rcutree.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:47 @ static inline void rcu_virt_note_context rcu_note_context_switch(); } +#ifdef CONFIG_PREEMPT_RT_FULL +# define synchronize_rcu_bh synchronize_rcu +#else void synchronize_rcu_bh(void); +#endif void synchronize_sched_expedited(void); void synchronize_rcu_expedited(void); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:79 @ static inline void synchronize_rcu_bh_ex } void rcu_barrier(void); +#ifdef CONFIG_PREEMPT_RT_FULL +# define rcu_barrier_bh rcu_barrier +#else void rcu_barrier_bh(void); +#endif void rcu_barrier_sched(void); unsigned long get_state_synchronize_rcu(void); void cond_synchronize_rcu(unsigned long oldstate); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:96 @ unsigned long rcu_batches_started(void); unsigned long rcu_batches_started_bh(void); unsigned long rcu_batches_started_sched(void); unsigned long rcu_batches_completed(void); -unsigned long rcu_batches_completed_bh(void); unsigned long rcu_batches_completed_sched(void); void show_rcu_gp_kthreads(void); void rcu_force_quiescent_state(void); -void rcu_bh_force_quiescent_state(void); void rcu_sched_force_quiescent_state(void); void rcu_idle_enter(void); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:114 @ extern int rcu_scheduler_active __read_m bool rcu_is_watching(void); +#ifndef CONFIG_PREEMPT_RT_FULL +void rcu_bh_force_quiescent_state(void); +unsigned long rcu_batches_completed_bh(void); +#else +# define rcu_bh_force_quiescent_state rcu_force_quiescent_state +# define rcu_batches_completed_bh rcu_batches_completed +#endif + void rcu_all_qs(void); #endif /* __LINUX_RCUTREE_H */ Index: linux-4.4.25-rt35/include/linux/rtmutex.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/rtmutex.h +++ linux-4.4.25-rt35/include/linux/rtmutex.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:16 @ #define __LINUX_RT_MUTEX_H #include <linux/linkage.h> +#include <linux/spinlock_types_raw.h> #include <linux/rbtree.h> -#include <linux/spinlock_types.h> extern int max_lock_depth; /* for sysctl */ +#ifdef CONFIG_DEBUG_MUTEXES +#include <linux/debug_locks.h> +#endif + /** * The rt_mutex structure * @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:38 @ struct rt_mutex { struct rb_root waiters; struct rb_node *waiters_leftmost; struct task_struct *owner; -#ifdef CONFIG_DEBUG_RT_MUTEXES int save_state; +#ifdef CONFIG_DEBUG_RT_MUTEXES const char *name, *file; int line; void *magic; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:62 @ struct hrtimer_sleeper; # define rt_mutex_debug_check_no_locks_held(task) do { } while (0) #endif +# define rt_mutex_init(mutex) \ + do { \ + raw_spin_lock_init(&(mutex)->wait_lock); \ + __rt_mutex_init(mutex, #mutex); \ + } while (0) + #ifdef CONFIG_DEBUG_RT_MUTEXES # define __DEBUG_RT_MUTEX_INITIALIZER(mutexname) \ , .name = #mutexname, .file = __FILE__, .line = __LINE__ -# define rt_mutex_init(mutex) __rt_mutex_init(mutex, __func__) extern void rt_mutex_debug_task_free(struct task_struct *tsk); #else # define __DEBUG_RT_MUTEX_INITIALIZER(mutexname) -# define rt_mutex_init(mutex) __rt_mutex_init(mutex, NULL) # define rt_mutex_debug_task_free(t) do { } while (0) #endif -#define __RT_MUTEX_INITIALIZER(mutexname) \ - { .wait_lock = __RAW_SPIN_LOCK_UNLOCKED(mutexname.wait_lock) \ +#define __RT_MUTEX_INITIALIZER_PLAIN(mutexname) \ + .wait_lock = __RAW_SPIN_LOCK_UNLOCKED(mutexname.wait_lock) \ , .waiters = RB_ROOT \ , .owner = NULL \ - __DEBUG_RT_MUTEX_INITIALIZER(mutexname)} + __DEBUG_RT_MUTEX_INITIALIZER(mutexname) + +#define __RT_MUTEX_INITIALIZER(mutexname) \ + { __RT_MUTEX_INITIALIZER_PLAIN(mutexname) } + +#define __RT_MUTEX_INITIALIZER_SAVE_STATE(mutexname) \ + { __RT_MUTEX_INITIALIZER_PLAIN(mutexname) \ + , .save_state = 1 } #define DEFINE_RT_MUTEX(mutexname) \ struct rt_mutex mutexname = __RT_MUTEX_INITIALIZER(mutexname) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:109 @ extern void rt_mutex_destroy(struct rt_m extern void rt_mutex_lock(struct rt_mutex *lock); extern int rt_mutex_lock_interruptible(struct rt_mutex *lock); +extern int rt_mutex_lock_killable(struct rt_mutex *lock); extern int rt_mutex_timed_lock(struct rt_mutex *lock, struct hrtimer_sleeper *timeout); Index: linux-4.4.25-rt35/include/linux/rwlock_rt.h =================================================================== --- /dev/null +++ linux-4.4.25-rt35/include/linux/rwlock_rt.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +#ifndef __LINUX_RWLOCK_RT_H +#define __LINUX_RWLOCK_RT_H + +#ifndef __LINUX_SPINLOCK_H +#error Do not include directly. Use spinlock.h +#endif + +#define rwlock_init(rwl) \ +do { \ + static struct lock_class_key __key; \ + \ + rt_mutex_init(&(rwl)->lock); \ + __rt_rwlock_init(rwl, #rwl, &__key); \ +} while (0) + +extern void __lockfunc rt_write_lock(rwlock_t *rwlock); +extern void __lockfunc rt_read_lock(rwlock_t *rwlock); +extern int __lockfunc rt_write_trylock(rwlock_t *rwlock); +extern int __lockfunc rt_write_trylock_irqsave(rwlock_t *trylock, unsigned long *flags); +extern int __lockfunc rt_read_trylock(rwlock_t *rwlock); +extern void __lockfunc rt_write_unlock(rwlock_t *rwlock); +extern void __lockfunc rt_read_unlock(rwlock_t *rwlock); +extern unsigned long __lockfunc rt_write_lock_irqsave(rwlock_t *rwlock); +extern unsigned long __lockfunc rt_read_lock_irqsave(rwlock_t *rwlock); +extern void __rt_rwlock_init(rwlock_t *rwlock, char *name, struct lock_class_key *key); + +#define read_trylock(lock) __cond_lock(lock, rt_read_trylock(lock)) +#define write_trylock(lock) __cond_lock(lock, rt_write_trylock(lock)) + +#define write_trylock_irqsave(lock, flags) \ + __cond_lock(lock, rt_write_trylock_irqsave(lock, &flags)) + +#define read_lock_irqsave(lock, flags) \ + do { \ + typecheck(unsigned long, flags); \ + flags = rt_read_lock_irqsave(lock); \ + } while (0) + +#define write_lock_irqsave(lock, flags) \ + do { \ + typecheck(unsigned long, flags); \ + flags = rt_write_lock_irqsave(lock); \ + } while (0) + +#define read_lock(lock) rt_read_lock(lock) + +#define read_lock_bh(lock) \ + do { \ + local_bh_disable(); \ + rt_read_lock(lock); \ + } while (0) + +#define read_lock_irq(lock) read_lock(lock) + +#define write_lock(lock) rt_write_lock(lock) + +#define write_lock_bh(lock) \ + do { \ + local_bh_disable(); \ + rt_write_lock(lock); \ + } while (0) + +#define write_lock_irq(lock) write_lock(lock) + +#define read_unlock(lock) rt_read_unlock(lock) + +#define read_unlock_bh(lock) \ + do { \ + rt_read_unlock(lock); \ + local_bh_enable(); \ + } while (0) + +#define read_unlock_irq(lock) read_unlock(lock) + +#define write_unlock(lock) rt_write_unlock(lock) + +#define write_unlock_bh(lock) \ + do { \ + rt_write_unlock(lock); \ + local_bh_enable(); \ + } while (0) + +#define write_unlock_irq(lock) write_unlock(lock) + +#define read_unlock_irqrestore(lock, flags) \ + do { \ + typecheck(unsigned long, flags); \ + (void) flags; \ + rt_read_unlock(lock); \ + } while (0) + +#define write_unlock_irqrestore(lock, flags) \ + do { \ + typecheck(unsigned long, flags); \ + (void) flags; \ + rt_write_unlock(lock); \ + } while (0) + +#endif Index: linux-4.4.25-rt35/include/linux/rwlock_types.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/rwlock_types.h +++ linux-4.4.25-rt35/include/linux/rwlock_types.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ #ifndef __LINUX_RWLOCK_TYPES_H #define __LINUX_RWLOCK_TYPES_H +#if !defined(__LINUX_SPINLOCK_TYPES_H) +# error "Do not include directly, include spinlock_types.h" +#endif + /* * include/linux/rwlock_types.h - generic rwlock type definitions * and initializers @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:50 @ typedef struct { RW_DEP_MAP_INIT(lockname) } #endif -#define DEFINE_RWLOCK(x) rwlock_t x = __RW_LOCK_UNLOCKED(x) +#define DEFINE_RWLOCK(name) \ + rwlock_t name __cacheline_aligned_in_smp = __RW_LOCK_UNLOCKED(name) #endif /* __LINUX_RWLOCK_TYPES_H */ Index: linux-4.4.25-rt35/include/linux/rwlock_types_rt.h =================================================================== --- /dev/null +++ linux-4.4.25-rt35/include/linux/rwlock_types_rt.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +#ifndef __LINUX_RWLOCK_TYPES_RT_H +#define __LINUX_RWLOCK_TYPES_RT_H + +#ifndef __LINUX_SPINLOCK_TYPES_H +#error "Do not include directly. Include spinlock_types.h instead" +#endif + +/* + * rwlocks - rtmutex which allows single reader recursion + */ +typedef struct { + struct rt_mutex lock; + int read_depth; + unsigned int break_lock; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; +#endif +} rwlock_t; + +#ifdef CONFIG_DEBUG_LOCK_ALLOC +# define RW_DEP_MAP_INIT(lockname) .dep_map = { .name = #lockname } +#else +# define RW_DEP_MAP_INIT(lockname) +#endif + +#define __RW_LOCK_UNLOCKED(name) \ + { .lock = __RT_MUTEX_INITIALIZER_SAVE_STATE(name.lock), \ + RW_DEP_MAP_INIT(name) } + +#define DEFINE_RWLOCK(name) \ + rwlock_t name __cacheline_aligned_in_smp = __RW_LOCK_UNLOCKED(name) + +#endif Index: linux-4.4.25-rt35/include/linux/rwsem.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/rwsem.h +++ linux-4.4.25-rt35/include/linux/rwsem.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:21 @ #include <linux/osq_lock.h> #endif +#ifdef CONFIG_PREEMPT_RT_FULL +#include <linux/rwsem_rt.h> +#else /* PREEMPT_RT_FULL */ + struct rw_semaphore; #ifdef CONFIG_RWSEM_GENERIC_SPINLOCK @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:184 @ extern void up_read_non_owner(struct rw_ # define up_read_non_owner(sem) up_read(sem) #endif +#endif /* !PREEMPT_RT_FULL */ + #endif /* _LINUX_RWSEM_H */ Index: linux-4.4.25-rt35/include/linux/rwsem_rt.h =================================================================== --- /dev/null +++ linux-4.4.25-rt35/include/linux/rwsem_rt.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +#ifndef _LINUX_RWSEM_RT_H +#define _LINUX_RWSEM_RT_H + +#ifndef _LINUX_RWSEM_H +#error "Include rwsem.h" +#endif + +/* + * RW-semaphores are a spinlock plus a reader-depth count. + * + * Note that the semantics are different from the usual + * Linux rw-sems, in PREEMPT_RT mode we do not allow + * multiple readers to hold the lock at once, we only allow + * a read-lock owner to read-lock recursively. This is + * better for latency, makes the implementation inherently + * fair and makes it simpler as well. + */ + +#include <linux/rtmutex.h> + +struct rw_semaphore { + struct rt_mutex lock; + int read_depth; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; +#endif +}; + +#define __RWSEM_INITIALIZER(name) \ + { .lock = __RT_MUTEX_INITIALIZER(name.lock), \ + RW_DEP_MAP_INIT(name) } + +#define DECLARE_RWSEM(lockname) \ + struct rw_semaphore lockname = __RWSEM_INITIALIZER(lockname) + +extern void __rt_rwsem_init(struct rw_semaphore *rwsem, const char *name, + struct lock_class_key *key); + +#define __rt_init_rwsem(sem, name, key) \ + do { \ + rt_mutex_init(&(sem)->lock); \ + __rt_rwsem_init((sem), (name), (key));\ + } while (0) + +#define __init_rwsem(sem, name, key) __rt_init_rwsem(sem, name, key) + +# define rt_init_rwsem(sem) \ +do { \ + static struct lock_class_key __key; \ + \ + __rt_init_rwsem((sem), #sem, &__key); \ +} while (0) + +extern void rt_down_write(struct rw_semaphore *rwsem); +extern void rt_down_read_nested(struct rw_semaphore *rwsem, int subclass); +extern void rt_down_write_nested(struct rw_semaphore *rwsem, int subclass); +extern void rt_down_write_nested_lock(struct rw_semaphore *rwsem, + struct lockdep_map *nest); +extern void rt__down_read(struct rw_semaphore *rwsem); +extern void rt_down_read(struct rw_semaphore *rwsem); +extern int rt_down_write_trylock(struct rw_semaphore *rwsem); +extern int rt__down_read_trylock(struct rw_semaphore *rwsem); +extern int rt_down_read_trylock(struct rw_semaphore *rwsem); +extern void __rt_up_read(struct rw_semaphore *rwsem); +extern void rt_up_read(struct rw_semaphore *rwsem); +extern void rt_up_write(struct rw_semaphore *rwsem); +extern void rt_downgrade_write(struct rw_semaphore *rwsem); + +#define init_rwsem(sem) rt_init_rwsem(sem) +#define rwsem_is_locked(s) rt_mutex_is_locked(&(s)->lock) + +static inline int rwsem_is_contended(struct rw_semaphore *sem) +{ + /* rt_mutex_has_waiters() */ + return !RB_EMPTY_ROOT(&sem->lock.waiters); +} + +static inline void __down_read(struct rw_semaphore *sem) +{ + rt__down_read(sem); +} + +static inline void down_read(struct rw_semaphore *sem) +{ + rt_down_read(sem); +} + +static inline int __down_read_trylock(struct rw_semaphore *sem) +{ + return rt__down_read_trylock(sem); +} + +static inline int down_read_trylock(struct rw_semaphore *sem) +{ + return rt_down_read_trylock(sem); +} + +static inline void down_write(struct rw_semaphore *sem) +{ + rt_down_write(sem); +} + +static inline int down_write_trylock(struct rw_semaphore *sem) +{ + return rt_down_write_trylock(sem); +} + +static inline void __up_read(struct rw_semaphore *sem) +{ + __rt_up_read(sem); +} + +static inline void up_read(struct rw_semaphore *sem) +{ + rt_up_read(sem); +} + +static inline void up_write(struct rw_semaphore *sem) +{ + rt_up_write(sem); +} + +static inline void downgrade_write(struct rw_semaphore *sem) +{ + rt_downgrade_write(sem); +} + +static inline void down_read_nested(struct rw_semaphore *sem, int subclass) +{ + return rt_down_read_nested(sem, subclass); +} + +static inline void down_write_nested(struct rw_semaphore *sem, int subclass) +{ + rt_down_write_nested(sem, subclass); +} +#ifdef CONFIG_DEBUG_LOCK_ALLOC +static inline void down_write_nest_lock(struct rw_semaphore *sem, + struct rw_semaphore *nest_lock) +{ + rt_down_write_nested_lock(sem, &nest_lock->dep_map); +} + +#else + +static inline void down_write_nest_lock(struct rw_semaphore *sem, + struct rw_semaphore *nest_lock) +{ + rt_down_write_nested_lock(sem, NULL); +} +#endif +#endif Index: linux-4.4.25-rt35/include/linux/sched.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/sched.h +++ linux-4.4.25-rt35/include/linux/sched.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:29 @ struct sched_param { #include <linux/nodemask.h> #include <linux/mm_types.h> #include <linux/preempt.h> +#include <asm/kmap_types.h> #include <asm/page.h> #include <asm/ptrace.h> @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:186 @ extern void update_cpu_load_nohz(void); static inline void update_cpu_load_nohz(void) { } #endif -extern unsigned long get_parent_ip(unsigned long addr); - extern void dump_cpu_task(int cpu); struct seq_file; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:244 @ extern char ___assert_task_state[1 - 2*! TASK_UNINTERRUPTIBLE | __TASK_STOPPED | \ __TASK_TRACED | EXIT_ZOMBIE | EXIT_DEAD) -#define task_is_traced(task) ((task->state & __TASK_TRACED) != 0) #define task_is_stopped(task) ((task->state & __TASK_STOPPED) != 0) -#define task_is_stopped_or_traced(task) \ - ((task->state & (__TASK_STOPPED | __TASK_TRACED)) != 0) #define task_contributes_to_load(task) \ ((task->state & TASK_UNINTERRUPTIBLE) != 0 && \ (task->flags & PF_FROZEN) == 0 && \ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:310 @ extern char ___assert_task_state[1 - 2*! #endif +#define __set_current_state_no_track(state_value) \ + do { current->state = (state_value); } while (0) +#define set_current_state_no_track(state_value) \ + set_mb(current->state, (state_value)) + /* Task command name length */ #define TASK_COMM_LEN 16 @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:974 @ struct wake_q_head { struct wake_q_head name = { WAKE_Q_TAIL, &name.first } extern void wake_q_add(struct wake_q_head *head, - struct task_struct *task); -extern void wake_up_q(struct wake_q_head *head); + struct task_struct *task); +extern void __wake_up_q(struct wake_q_head *head, bool sleeper); + +static inline void wake_up_q(struct wake_q_head *head) +{ + __wake_up_q(head, false); +} + +static inline void wake_up_q_sleeper(struct wake_q_head *head) +{ + __wake_up_q(head, true); +} /* * sched-domains (multiprocessor balancing) declarations: @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1393 @ struct tlbflush_unmap_batch { struct task_struct { volatile long state; /* -1 unrunnable, 0 runnable, >0 stopped */ + volatile long saved_state; /* saved state for "spinlock sleepers" */ void *stack; atomic_t usage; unsigned int flags; /* per process flags, defined below */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1430 @ struct task_struct { #endif unsigned int policy; +#ifdef CONFIG_PREEMPT_RT_FULL + int migrate_disable; +# ifdef CONFIG_SCHED_DEBUG + int migrate_disable_atomic; +# endif +#endif int nr_cpus_allowed; cpumask_t cpus_allowed; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1543 @ struct task_struct { cputime_t gtime; struct prev_cputime prev_cputime; #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN - seqlock_t vtime_seqlock; + seqcount_t vtime_seqcount; unsigned long long vtime_snap; enum { - VTIME_SLEEPING = 0, + /* Task is sleeping or running in a CPU with VTIME inactive */ + VTIME_INACTIVE = 0, + /* Task runs in userspace in a CPU with VTIME active */ VTIME_USER, + /* Task runs in kernelspace in a CPU with VTIME active */ VTIME_SYS, } vtime_snap_whence; #endif @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1562 @ struct task_struct { struct task_cputime cputime_expires; struct list_head cpu_timers[3]; +#ifdef CONFIG_PREEMPT_RT_BASE + struct task_struct *posix_timer_list; +#endif /* process credentials */ const struct cred __rcu *real_cred; /* objective and real subjective task @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1595 @ struct task_struct { /* signal handlers */ struct signal_struct *signal; struct sighand_struct *sighand; + struct sigqueue *sigqueue_cache; sigset_t blocked, real_blocked; sigset_t saved_sigmask; /* restored if set_restore_sigmask() was used */ struct sigpending pending; +#ifdef CONFIG_PREEMPT_RT_FULL + /* TODO: move me into ->restart_block ? */ + struct siginfo forced_info; +#endif unsigned long sas_ss_sp; size_t sas_ss_size; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1827 @ struct task_struct { unsigned long trace; /* bitmask and counter of trace recursion */ unsigned long trace_recursion; +#ifdef CONFIG_WAKEUP_LATENCY_HIST + u64 preempt_timestamp_hist; +#ifdef CONFIG_MISSED_TIMER_OFFSETS_HIST + long timer_offset; +#endif +#endif #endif /* CONFIG_TRACING */ #ifdef CONFIG_MEMCG struct mem_cgroup *memcg_in_oom; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1849 @ struct task_struct { unsigned int sequential_io; unsigned int sequential_io_avg; #endif +#ifdef CONFIG_PREEMPT_RT_BASE + struct rcu_head put_rcu; + int softirq_nestcnt; + unsigned int softirqs_raised; +#endif +#ifdef CONFIG_PREEMPT_RT_FULL +# if defined CONFIG_HIGHMEM || defined CONFIG_X86_32 + int kmap_idx; + pte_t kmap_pte[KM_TYPE_NR]; +# endif +#endif #ifdef CONFIG_DEBUG_ATOMIC_SLEEP unsigned long task_state_change; #endif +#ifdef CONFIG_PREEMPT_RT_FULL + int xmit_recursion; +#endif int pagefault_disabled; /* CPU-specific state of this task */ struct thread_struct thread; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1883 @ extern int arch_task_struct_size __read_ # define arch_task_struct_size (sizeof(struct task_struct)) #endif -/* Future-safe accessor for struct task_struct's cpus_allowed. */ -#define tsk_cpus_allowed(tsk) (&(tsk)->cpus_allowed) - #define TNF_MIGRATED 0x01 #define TNF_NO_GROUP 0x02 #define TNF_SHARED 0x04 @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2072 @ extern struct pid *cad_pid; extern void free_task(struct task_struct *tsk); #define get_task_struct(tsk) do { atomic_inc(&(tsk)->usage); } while(0) +#ifdef CONFIG_PREEMPT_RT_BASE +extern void __put_task_struct_cb(struct rcu_head *rhp); + +static inline void put_task_struct(struct task_struct *t) +{ + if (atomic_dec_and_test(&t->usage)) + call_rcu(&t->put_rcu, __put_task_struct_cb); +} +#else extern void __put_task_struct(struct task_struct *t); static inline void put_task_struct(struct task_struct *t) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2088 @ static inline void put_task_struct(struc if (atomic_dec_and_test(&t->usage)) __put_task_struct(t); } +#endif #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN extern void task_cputime(struct task_struct *t, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2127 @ extern void thread_group_cputime_adjuste /* * Per process flags */ +#define PF_IN_SOFTIRQ 0x00000001 /* Task is serving softirq */ #define PF_EXITING 0x00000004 /* getting shut down */ #define PF_EXITPIDONE 0x00000008 /* pi exit done on shut down */ #define PF_VCPU 0x00000010 /* I'm a virtual CPU */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2292 @ extern void do_set_cpus_allowed(struct t extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask); +int migrate_me(void); +void tell_sched_cpu_down_begin(int cpu); +void tell_sched_cpu_down_done(int cpu); + #else static inline void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2308 @ static inline int set_cpus_allowed_ptr(s return -EINVAL; return 0; } +static inline int migrate_me(void) { return 0; } +static inline void tell_sched_cpu_down_begin(int cpu) { } +static inline void tell_sched_cpu_down_done(int cpu) { } #endif #ifdef CONFIG_NO_HZ_COMMON @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2520 @ extern void xtime_update(unsigned long t extern int wake_up_state(struct task_struct *tsk, unsigned int state); extern int wake_up_process(struct task_struct *tsk); +extern int wake_up_lock_sleeper(struct task_struct * tsk); extern void wake_up_new_task(struct task_struct *tsk); #ifdef CONFIG_SMP extern void kick_process(struct task_struct *tsk); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2644 @ extern struct mm_struct * mm_alloc(void) /* mmdrop drops the mm and the page tables */ extern void __mmdrop(struct mm_struct *); + static inline void mmdrop(struct mm_struct * mm) { if (unlikely(atomic_dec_and_test(&mm->mm_count))) __mmdrop(mm); } +#ifdef CONFIG_PREEMPT_RT_BASE +extern void __mmdrop_delayed(struct rcu_head *rhp); +static inline void mmdrop_delayed(struct mm_struct *mm) +{ + if (atomic_dec_and_test(&mm->mm_count)) + call_rcu(&mm->delayed_drop, __mmdrop_delayed); +} +#else +# define mmdrop_delayed(mm) mmdrop(mm) +#endif + /* mmput gets rid of the mappings and all user-space */ extern void mmput(struct mm_struct *); /* Grab a reference to a task's mm, if it is not already going away */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:2971 @ static inline int test_tsk_need_resched( return unlikely(test_tsk_thread_flag(tsk,TIF_NEED_RESCHED)); } +#ifdef CONFIG_PREEMPT_LAZY +static inline void set_tsk_need_resched_lazy(struct task_struct *tsk) +{ + set_tsk_thread_flag(tsk,TIF_NEED_RESCHED_LAZY); +} + +static inline void clear_tsk_need_resched_lazy(struct task_struct *tsk) +{ + clear_tsk_thread_flag(tsk,TIF_NEED_RESCHED_LAZY); +} + +static inline int test_tsk_need_resched_lazy(struct task_struct *tsk) +{ + return unlikely(test_tsk_thread_flag(tsk,TIF_NEED_RESCHED_LAZY)); +} + +static inline int need_resched_lazy(void) +{ + return test_thread_flag(TIF_NEED_RESCHED_LAZY); +} + +static inline int need_resched_now(void) +{ + return test_thread_flag(TIF_NEED_RESCHED); +} + +#else +static inline void clear_tsk_need_resched_lazy(struct task_struct *tsk) { } +static inline int need_resched_lazy(void) { return 0; } + +static inline int need_resched_now(void) +{ + return test_thread_flag(TIF_NEED_RESCHED); +} + +#endif + static inline int restart_syscall(void) { set_tsk_thread_flag(current, TIF_SIGPENDING); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:3039 @ static inline int signal_pending_state(l return (state & TASK_INTERRUPTIBLE) || __fatal_signal_pending(p); } +static inline bool __task_is_stopped_or_traced(struct task_struct *task) +{ + if (task->state & (__TASK_STOPPED | __TASK_TRACED)) + return true; +#ifdef CONFIG_PREEMPT_RT_FULL + if (task->saved_state & (__TASK_STOPPED | __TASK_TRACED)) + return true; +#endif + return false; +} + +static inline bool task_is_stopped_or_traced(struct task_struct *task) +{ + bool traced_stopped; + +#ifdef CONFIG_PREEMPT_RT_FULL + unsigned long flags; + + raw_spin_lock_irqsave(&task->pi_lock, flags); + traced_stopped = __task_is_stopped_or_traced(task); + raw_spin_unlock_irqrestore(&task->pi_lock, flags); +#else + traced_stopped = __task_is_stopped_or_traced(task); +#endif + return traced_stopped; +} + +static inline bool task_is_traced(struct task_struct *task) +{ + bool traced = false; + + if (task->state & __TASK_TRACED) + return true; +#ifdef CONFIG_PREEMPT_RT_FULL + /* in case the task is sleeping on tasklist_lock */ + raw_spin_lock_irq(&task->pi_lock); + if (task->state & __TASK_TRACED) + traced = true; + else if (task->saved_state & __TASK_TRACED) + traced = true; + raw_spin_unlock_irq(&task->pi_lock); +#endif + return traced; +} + /* * cond_resched() and cond_resched_lock(): latency reduction via * explicit rescheduling in places that are safe. The return @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:3105 @ extern int __cond_resched_lock(spinlock_ __cond_resched_lock(lock); \ }) +#ifndef CONFIG_PREEMPT_RT_FULL extern int __cond_resched_softirq(void); #define cond_resched_softirq() ({ \ ___might_sleep(__FILE__, __LINE__, SOFTIRQ_DISABLE_OFFSET); \ __cond_resched_softirq(); \ }) +#else +# define cond_resched_softirq() cond_resched() +#endif static inline void cond_resched_rcu(void) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:3276 @ static inline void set_task_cpu(struct t #endif /* CONFIG_SMP */ +static inline int __migrate_disabled(struct task_struct *p) +{ +#ifdef CONFIG_PREEMPT_RT_FULL + return p->migrate_disable; +#else + return 0; +#endif +} + +/* Future-safe accessor for struct task_struct's cpus_allowed. */ +static inline const struct cpumask *tsk_cpus_allowed(struct task_struct *p) +{ + if (__migrate_disabled(p)) + return cpumask_of(task_cpu(p)); + + return &p->cpus_allowed; +} + +static inline int tsk_nr_cpus_allowed(struct task_struct *p) +{ + if (__migrate_disabled(p)) + return 1; + return p->nr_cpus_allowed; +} + extern long sched_setaffinity(pid_t pid, const struct cpumask *new_mask); extern long sched_getaffinity(pid_t pid, struct cpumask *mask); Index: linux-4.4.25-rt35/include/linux/seqlock.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/seqlock.h +++ linux-4.4.25-rt35/include/linux/seqlock.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:223 @ static inline int read_seqcount_retry(co return __read_seqcount_retry(s, start); } - - -static inline void raw_write_seqcount_begin(seqcount_t *s) +static inline void __raw_write_seqcount_begin(seqcount_t *s) { s->sequence++; smp_wmb(); } -static inline void raw_write_seqcount_end(seqcount_t *s) +static inline void raw_write_seqcount_begin(seqcount_t *s) +{ + preempt_disable_rt(); + __raw_write_seqcount_begin(s); +} + +static inline void __raw_write_seqcount_end(seqcount_t *s) { smp_wmb(); s->sequence++; } +static inline void raw_write_seqcount_end(seqcount_t *s) +{ + __raw_write_seqcount_end(s); + preempt_enable_rt(); +} + /** * raw_write_seqcount_barrier - do a seq write barrier * @s: pointer to seqcount_t @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:438 @ typedef struct { /* * Read side functions for starting and finalizing a read side section. */ +#ifndef CONFIG_PREEMPT_RT_FULL static inline unsigned read_seqbegin(const seqlock_t *sl) { return read_seqcount_begin(&sl->seqcount); } +#else +/* + * Starvation safe read side for RT + */ +static inline unsigned read_seqbegin(seqlock_t *sl) +{ + unsigned ret; + +repeat: + ret = ACCESS_ONCE(sl->seqcount.sequence); + if (unlikely(ret & 1)) { + /* + * Take the lock and let the writer proceed (i.e. evtl + * boost it), otherwise we could loop here forever. + */ + spin_unlock_wait(&sl->lock); + goto repeat; + } + return ret; +} +#endif static inline unsigned read_seqretry(const seqlock_t *sl, unsigned start) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:478 @ static inline unsigned read_seqretry(con static inline void write_seqlock(seqlock_t *sl) { spin_lock(&sl->lock); - write_seqcount_begin(&sl->seqcount); + __raw_write_seqcount_begin(&sl->seqcount); } static inline void write_sequnlock(seqlock_t *sl) { - write_seqcount_end(&sl->seqcount); + __raw_write_seqcount_end(&sl->seqcount); spin_unlock(&sl->lock); } static inline void write_seqlock_bh(seqlock_t *sl) { spin_lock_bh(&sl->lock); - write_seqcount_begin(&sl->seqcount); + __raw_write_seqcount_begin(&sl->seqcount); } static inline void write_sequnlock_bh(seqlock_t *sl) { - write_seqcount_end(&sl->seqcount); + __raw_write_seqcount_end(&sl->seqcount); spin_unlock_bh(&sl->lock); } static inline void write_seqlock_irq(seqlock_t *sl) { spin_lock_irq(&sl->lock); - write_seqcount_begin(&sl->seqcount); + __raw_write_seqcount_begin(&sl->seqcount); } static inline void write_sequnlock_irq(seqlock_t *sl) { - write_seqcount_end(&sl->seqcount); + __raw_write_seqcount_end(&sl->seqcount); spin_unlock_irq(&sl->lock); } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:516 @ static inline unsigned long __write_seql unsigned long flags; spin_lock_irqsave(&sl->lock, flags); - write_seqcount_begin(&sl->seqcount); + __raw_write_seqcount_begin(&sl->seqcount); return flags; } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:526 @ static inline unsigned long __write_seql static inline void write_sequnlock_irqrestore(seqlock_t *sl, unsigned long flags) { - write_seqcount_end(&sl->seqcount); + __raw_write_seqcount_end(&sl->seqcount); spin_unlock_irqrestore(&sl->lock, flags); } Index: linux-4.4.25-rt35/include/linux/signal.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/signal.h +++ linux-4.4.25-rt35/include/linux/signal.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:236 @ static inline void init_sigpending(struc } extern void flush_sigqueue(struct sigpending *queue); +extern void flush_task_sigqueue(struct task_struct *tsk); /* Test if 'sig' is valid signal. Use this instead of testing _NSIG directly */ static inline int valid_signal(unsigned long sig) Index: linux-4.4.25-rt35/include/linux/skbuff.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/skbuff.h +++ linux-4.4.25-rt35/include/linux/skbuff.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:206 @ struct sk_buff_head { __u32 qlen; spinlock_t lock; + raw_spinlock_t raw_lock; }; struct sk_buff; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1469 @ static inline void skb_queue_head_init(s __skb_queue_head_init(list); } +static inline void skb_queue_head_init_raw(struct sk_buff_head *list) +{ + raw_spin_lock_init(&list->raw_lock); + __skb_queue_head_init(list); +} + static inline void skb_queue_head_init_class(struct sk_buff_head *list, struct lock_class_key *class) { Index: linux-4.4.25-rt35/include/linux/smp.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/smp.h +++ linux-4.4.25-rt35/include/linux/smp.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:188 @ static inline void smp_init(void) { } #define get_cpu() ({ preempt_disable(); smp_processor_id(); }) #define put_cpu() preempt_enable() +#define get_cpu_light() ({ migrate_disable(); smp_processor_id(); }) +#define put_cpu_light() migrate_enable() + /* * Callback to arch code if there's nosmp or maxcpus=0 on the * boot command line: Index: linux-4.4.25-rt35/include/linux/spinlock.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/spinlock.h +++ linux-4.4.25-rt35/include/linux/spinlock.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:274 @ static inline void do_raw_spin_unlock(ra #define raw_spin_can_lock(lock) (!raw_spin_is_locked(lock)) /* Include rwlock functions */ -#include <linux/rwlock.h> +#ifdef CONFIG_PREEMPT_RT_FULL +# include <linux/rwlock_rt.h> +#else +# include <linux/rwlock.h> +#endif /* * Pull the _spin_*()/_read_*()/_write_*() functions/declarations: @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:289 @ static inline void do_raw_spin_unlock(ra # include <linux/spinlock_api_up.h> #endif +#ifdef CONFIG_PREEMPT_RT_FULL +# include <linux/spinlock_rt.h> +#else /* PREEMPT_RT_FULL */ + /* * Map the spin_lock functions to the raw variants for PREEMPT_RT=n */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:358 @ static __always_inline void spin_unlock( raw_spin_unlock(&lock->rlock); } +static __always_inline int spin_unlock_no_deboost(spinlock_t *lock) +{ + raw_spin_unlock(&lock->rlock); + return 0; +} + static __always_inline void spin_unlock_bh(spinlock_t *lock) { raw_spin_unlock_bh(&lock->rlock); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:433 @ extern int _atomic_dec_and_lock(atomic_t #define atomic_dec_and_lock(atomic, lock) \ __cond_lock(lock, _atomic_dec_and_lock(atomic, lock)) +#endif /* !PREEMPT_RT_FULL */ + #endif /* __LINUX_SPINLOCK_H */ Index: linux-4.4.25-rt35/include/linux/spinlock_api_smp.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/spinlock_api_smp.h +++ linux-4.4.25-rt35/include/linux/spinlock_api_smp.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:192 @ static inline int __raw_spin_trylock_bh( return 0; } -#include <linux/rwlock_api_smp.h> +#ifndef CONFIG_PREEMPT_RT_FULL +# include <linux/rwlock_api_smp.h> +#endif #endif /* __LINUX_SPINLOCK_API_SMP_H */ Index: linux-4.4.25-rt35/include/linux/spinlock_rt.h =================================================================== --- /dev/null +++ linux-4.4.25-rt35/include/linux/spinlock_rt.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +#ifndef __LINUX_SPINLOCK_RT_H +#define __LINUX_SPINLOCK_RT_H + +#ifndef __LINUX_SPINLOCK_H +#error Do not include directly. Use spinlock.h +#endif + +#include <linux/bug.h> + +extern void +__rt_spin_lock_init(spinlock_t *lock, char *name, struct lock_class_key *key); + +#define spin_lock_init(slock) \ +do { \ + static struct lock_class_key __key; \ + \ + rt_mutex_init(&(slock)->lock); \ + __rt_spin_lock_init(slock, #slock, &__key); \ +} while (0) + +void __lockfunc rt_spin_lock__no_mg(spinlock_t *lock); +void __lockfunc rt_spin_unlock__no_mg(spinlock_t *lock); +int __lockfunc rt_spin_trylock__no_mg(spinlock_t *lock); + +extern void __lockfunc rt_spin_lock(spinlock_t *lock); +extern unsigned long __lockfunc rt_spin_lock_trace_flags(spinlock_t *lock); +extern void __lockfunc rt_spin_lock_nested(spinlock_t *lock, int subclass); +extern void __lockfunc rt_spin_unlock(spinlock_t *lock); +extern int __lockfunc rt_spin_unlock_no_deboost(spinlock_t *lock); +extern void __lockfunc rt_spin_unlock_wait(spinlock_t *lock); +extern int __lockfunc rt_spin_trylock_irqsave(spinlock_t *lock, unsigned long *flags); +extern int __lockfunc rt_spin_trylock_bh(spinlock_t *lock); +extern int __lockfunc rt_spin_trylock(spinlock_t *lock); +extern int atomic_dec_and_spin_lock(atomic_t *atomic, spinlock_t *lock); + +/* + * lockdep-less calls, for derived types like rwlock: + * (for trylock they can use rt_mutex_trylock() directly. + */ +extern void __lockfunc __rt_spin_lock__no_mg(struct rt_mutex *lock); +extern void __lockfunc __rt_spin_lock(struct rt_mutex *lock); +extern void __lockfunc __rt_spin_unlock(struct rt_mutex *lock); +extern int __lockfunc __rt_spin_trylock(struct rt_mutex *lock); + +#define spin_lock(lock) rt_spin_lock(lock) + +#define spin_lock_bh(lock) \ + do { \ + local_bh_disable(); \ + rt_spin_lock(lock); \ + } while (0) + +#define spin_lock_irq(lock) spin_lock(lock) + +#define spin_do_trylock(lock) __cond_lock(lock, rt_spin_trylock(lock)) + +#define spin_trylock(lock) \ +({ \ + int __locked; \ + __locked = spin_do_trylock(lock); \ + __locked; \ +}) + +#ifdef CONFIG_LOCKDEP +# define spin_lock_nested(lock, subclass) \ + do { \ + rt_spin_lock_nested(lock, subclass); \ + } while (0) + +#define spin_lock_bh_nested(lock, subclass) \ + do { \ + local_bh_disable(); \ + rt_spin_lock_nested(lock, subclass); \ + } while (0) + +# define spin_lock_irqsave_nested(lock, flags, subclass) \ + do { \ + typecheck(unsigned long, flags); \ + flags = 0; \ + rt_spin_lock_nested(lock, subclass); \ + } while (0) +#else +# define spin_lock_nested(lock, subclass) spin_lock(lock) +# define spin_lock_bh_nested(lock, subclass) spin_lock_bh(lock) + +# define spin_lock_irqsave_nested(lock, flags, subclass) \ + do { \ + typecheck(unsigned long, flags); \ + flags = 0; \ + spin_lock(lock); \ + } while (0) +#endif + +#define spin_lock_irqsave(lock, flags) \ + do { \ + typecheck(unsigned long, flags); \ + flags = 0; \ + spin_lock(lock); \ + } while (0) + +static inline unsigned long spin_lock_trace_flags(spinlock_t *lock) +{ + unsigned long flags = 0; +#ifdef CONFIG_TRACE_IRQFLAGS + flags = rt_spin_lock_trace_flags(lock); +#else + spin_lock(lock); /* lock_local */ +#endif + return flags; +} + +/* FIXME: we need rt_spin_lock_nest_lock */ +#define spin_lock_nest_lock(lock, nest_lock) spin_lock_nested(lock, 0) + +#define spin_unlock(lock) rt_spin_unlock(lock) +#define spin_unlock_no_deboost(lock) rt_spin_unlock_no_deboost(lock) + +#define spin_unlock_bh(lock) \ + do { \ + rt_spin_unlock(lock); \ + local_bh_enable(); \ + } while (0) + +#define spin_unlock_irq(lock) spin_unlock(lock) + +#define spin_unlock_irqrestore(lock, flags) \ + do { \ + typecheck(unsigned long, flags); \ + (void) flags; \ + spin_unlock(lock); \ + } while (0) + +#define spin_trylock_bh(lock) __cond_lock(lock, rt_spin_trylock_bh(lock)) +#define spin_trylock_irq(lock) spin_trylock(lock) + +#define spin_trylock_irqsave(lock, flags) \ + rt_spin_trylock_irqsave(lock, &(flags)) + +#define spin_unlock_wait(lock) rt_spin_unlock_wait(lock) + +#ifdef CONFIG_GENERIC_LOCKBREAK +# define spin_is_contended(lock) ((lock)->break_lock) +#else +# define spin_is_contended(lock) (((void)(lock), 0)) +#endif + +static inline int spin_can_lock(spinlock_t *lock) +{ + return !rt_mutex_is_locked(&lock->lock); +} + +static inline int spin_is_locked(spinlock_t *lock) +{ + return rt_mutex_is_locked(&lock->lock); +} + +static inline void assert_spin_locked(spinlock_t *lock) +{ + BUG_ON(!spin_is_locked(lock)); +} + +#define atomic_dec_and_lock(atomic, lock) \ + atomic_dec_and_spin_lock(atomic, lock) + +#endif Index: linux-4.4.25-rt35/include/linux/spinlock_types.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/spinlock_types.h +++ linux-4.4.25-rt35/include/linux/spinlock_types.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:12 @ * Released under the General Public License (GPL). */ -#if defined(CONFIG_SMP) -# include <asm/spinlock_types.h> -#else -# include <linux/spinlock_types_up.h> -#endif - -#include <linux/lockdep.h> - -typedef struct raw_spinlock { - arch_spinlock_t raw_lock; -#ifdef CONFIG_GENERIC_LOCKBREAK - unsigned int break_lock; -#endif -#ifdef CONFIG_DEBUG_SPINLOCK - unsigned int magic, owner_cpu; - void *owner; -#endif -#ifdef CONFIG_DEBUG_LOCK_ALLOC - struct lockdep_map dep_map; -#endif -} raw_spinlock_t; - -#define SPINLOCK_MAGIC 0xdead4ead - -#define SPINLOCK_OWNER_INIT ((void *)-1L) - -#ifdef CONFIG_DEBUG_LOCK_ALLOC -# define SPIN_DEP_MAP_INIT(lockname) .dep_map = { .name = #lockname } -#else -# define SPIN_DEP_MAP_INIT(lockname) -#endif +#include <linux/spinlock_types_raw.h> -#ifdef CONFIG_DEBUG_SPINLOCK -# define SPIN_DEBUG_INIT(lockname) \ - .magic = SPINLOCK_MAGIC, \ - .owner_cpu = -1, \ - .owner = SPINLOCK_OWNER_INIT, +#ifndef CONFIG_PREEMPT_RT_FULL +# include <linux/spinlock_types_nort.h> +# include <linux/rwlock_types.h> #else -# define SPIN_DEBUG_INIT(lockname) +# include <linux/rtmutex.h> +# include <linux/spinlock_types_rt.h> +# include <linux/rwlock_types_rt.h> #endif -#define __RAW_SPIN_LOCK_INITIALIZER(lockname) \ - { \ - .raw_lock = __ARCH_SPIN_LOCK_UNLOCKED, \ - SPIN_DEBUG_INIT(lockname) \ - SPIN_DEP_MAP_INIT(lockname) } - -#define __RAW_SPIN_LOCK_UNLOCKED(lockname) \ - (raw_spinlock_t) __RAW_SPIN_LOCK_INITIALIZER(lockname) - -#define DEFINE_RAW_SPINLOCK(x) raw_spinlock_t x = __RAW_SPIN_LOCK_UNLOCKED(x) - -typedef struct spinlock { - union { - struct raw_spinlock rlock; - -#ifdef CONFIG_DEBUG_LOCK_ALLOC -# define LOCK_PADSIZE (offsetof(struct raw_spinlock, dep_map)) - struct { - u8 __padding[LOCK_PADSIZE]; - struct lockdep_map dep_map; - }; -#endif - }; -} spinlock_t; - -#define __SPIN_LOCK_INITIALIZER(lockname) \ - { { .rlock = __RAW_SPIN_LOCK_INITIALIZER(lockname) } } - -#define __SPIN_LOCK_UNLOCKED(lockname) \ - (spinlock_t ) __SPIN_LOCK_INITIALIZER(lockname) - -#define DEFINE_SPINLOCK(x) spinlock_t x = __SPIN_LOCK_UNLOCKED(x) - -#include <linux/rwlock_types.h> - #endif /* __LINUX_SPINLOCK_TYPES_H */ Index: linux-4.4.25-rt35/include/linux/spinlock_types_nort.h =================================================================== --- /dev/null +++ linux-4.4.25-rt35/include/linux/spinlock_types_nort.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +#ifndef __LINUX_SPINLOCK_TYPES_NORT_H +#define __LINUX_SPINLOCK_TYPES_NORT_H + +#ifndef __LINUX_SPINLOCK_TYPES_H +#error "Do not include directly. Include spinlock_types.h instead" +#endif + +/* + * The non RT version maps spinlocks to raw_spinlocks + */ +typedef struct spinlock { + union { + struct raw_spinlock rlock; + +#ifdef CONFIG_DEBUG_LOCK_ALLOC +# define LOCK_PADSIZE (offsetof(struct raw_spinlock, dep_map)) + struct { + u8 __padding[LOCK_PADSIZE]; + struct lockdep_map dep_map; + }; +#endif + }; +} spinlock_t; + +#define __SPIN_LOCK_INITIALIZER(lockname) \ + { { .rlock = __RAW_SPIN_LOCK_INITIALIZER(lockname) } } + +#define __SPIN_LOCK_UNLOCKED(lockname) \ + (spinlock_t ) __SPIN_LOCK_INITIALIZER(lockname) + +#define DEFINE_SPINLOCK(x) spinlock_t x = __SPIN_LOCK_UNLOCKED(x) + +#endif Index: linux-4.4.25-rt35/include/linux/spinlock_types_raw.h =================================================================== --- /dev/null +++ linux-4.4.25-rt35/include/linux/spinlock_types_raw.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +#ifndef __LINUX_SPINLOCK_TYPES_RAW_H +#define __LINUX_SPINLOCK_TYPES_RAW_H + +#if defined(CONFIG_SMP) +# include <asm/spinlock_types.h> +#else +# include <linux/spinlock_types_up.h> +#endif + +#include <linux/lockdep.h> + +typedef struct raw_spinlock { + arch_spinlock_t raw_lock; +#ifdef CONFIG_GENERIC_LOCKBREAK + unsigned int break_lock; +#endif +#ifdef CONFIG_DEBUG_SPINLOCK + unsigned int magic, owner_cpu; + void *owner; +#endif +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; +#endif +} raw_spinlock_t; + +#define SPINLOCK_MAGIC 0xdead4ead + +#define SPINLOCK_OWNER_INIT ((void *)-1L) + +#ifdef CONFIG_DEBUG_LOCK_ALLOC +# define SPIN_DEP_MAP_INIT(lockname) .dep_map = { .name = #lockname } +#else +# define SPIN_DEP_MAP_INIT(lockname) +#endif + +#ifdef CONFIG_DEBUG_SPINLOCK +# define SPIN_DEBUG_INIT(lockname) \ + .magic = SPINLOCK_MAGIC, \ + .owner_cpu = -1, \ + .owner = SPINLOCK_OWNER_INIT, +#else +# define SPIN_DEBUG_INIT(lockname) +#endif + +#define __RAW_SPIN_LOCK_INITIALIZER(lockname) \ + { \ + .raw_lock = __ARCH_SPIN_LOCK_UNLOCKED, \ + SPIN_DEBUG_INIT(lockname) \ + SPIN_DEP_MAP_INIT(lockname) } + +#define __RAW_SPIN_LOCK_UNLOCKED(lockname) \ + (raw_spinlock_t) __RAW_SPIN_LOCK_INITIALIZER(lockname) + +#define DEFINE_RAW_SPINLOCK(x) raw_spinlock_t x = __RAW_SPIN_LOCK_UNLOCKED(x) + +#endif Index: linux-4.4.25-rt35/include/linux/spinlock_types_rt.h =================================================================== --- /dev/null +++ linux-4.4.25-rt35/include/linux/spinlock_types_rt.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +#ifndef __LINUX_SPINLOCK_TYPES_RT_H +#define __LINUX_SPINLOCK_TYPES_RT_H + +#ifndef __LINUX_SPINLOCK_TYPES_H +#error "Do not include directly. Include spinlock_types.h instead" +#endif + +#include <linux/cache.h> + +/* + * PREEMPT_RT: spinlocks - an RT mutex plus lock-break field: + */ +typedef struct spinlock { + struct rt_mutex lock; + unsigned int break_lock; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; +#endif +} spinlock_t; + +#ifdef CONFIG_DEBUG_RT_MUTEXES +# define __RT_SPIN_INITIALIZER(name) \ + { \ + .wait_lock = __RAW_SPIN_LOCK_UNLOCKED(name.wait_lock), \ + .save_state = 1, \ + .file = __FILE__, \ + .line = __LINE__ , \ + } +#else +# define __RT_SPIN_INITIALIZER(name) \ + { \ + .wait_lock = __RAW_SPIN_LOCK_UNLOCKED(name.wait_lock), \ + .save_state = 1, \ + } +#endif + +/* +.wait_list = PLIST_HEAD_INIT_RAW((name).lock.wait_list, (name).lock.wait_lock) +*/ + +#define __SPIN_LOCK_UNLOCKED(name) \ + { .lock = __RT_SPIN_INITIALIZER(name.lock), \ + SPIN_DEP_MAP_INIT(name) } + +#define __DEFINE_SPINLOCK(name) \ + spinlock_t name = __SPIN_LOCK_UNLOCKED(name) + +#define DEFINE_SPINLOCK(name) \ + spinlock_t name __cacheline_aligned_in_smp = __SPIN_LOCK_UNLOCKED(name) + +#endif Index: linux-4.4.25-rt35/include/linux/srcu.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/srcu.h +++ linux-4.4.25-rt35/include/linux/srcu.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:87 @ int init_srcu_struct(struct srcu_struct void process_srcu(struct work_struct *work); -#define __SRCU_STRUCT_INIT(name) \ +#define __SRCU_STRUCT_INIT(name, pcpu_name) \ { \ .completed = -300, \ - .per_cpu_ref = &name##_srcu_array, \ + .per_cpu_ref = &pcpu_name, \ .queue_lock = __SPIN_LOCK_UNLOCKED(name.queue_lock), \ .running = false, \ .batch_queue = RCU_BATCH_INIT(name.batch_queue), \ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:107 @ void process_srcu(struct work_struct *wo */ #define __DEFINE_SRCU(name, is_static) \ static DEFINE_PER_CPU(struct srcu_struct_array, name##_srcu_array);\ - is_static struct srcu_struct name = __SRCU_STRUCT_INIT(name) + is_static struct srcu_struct name = __SRCU_STRUCT_INIT(name, name##_srcu_array) #define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */) #define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static) Index: linux-4.4.25-rt35/include/linux/suspend.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/suspend.h +++ linux-4.4.25-rt35/include/linux/suspend.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:197 @ struct platform_freeze_ops { void (*end)(void); }; +#if defined(CONFIG_SUSPEND) || defined(CONFIG_HIBERNATION) +extern bool pm_in_action; +#else +# define pm_in_action false +#endif + #ifdef CONFIG_SUSPEND /** * suspend_set_ops - set platform dependent suspend operations Index: linux-4.4.25-rt35/include/linux/swait.h =================================================================== --- /dev/null +++ linux-4.4.25-rt35/include/linux/swait.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +#ifndef _LINUX_SWAIT_H +#define _LINUX_SWAIT_H + +#include <linux/list.h> +#include <linux/stddef.h> +#include <linux/spinlock.h> +#include <asm/current.h> + +/* + * Simple wait queues + * + * While these are very similar to the other/complex wait queues (wait.h) the + * most important difference is that the simple waitqueue allows for + * deterministic behaviour -- IOW it has strictly bounded IRQ and lock hold + * times. + * + * In order to make this so, we had to drop a fair number of features of the + * other waitqueue code; notably: + * + * - mixing INTERRUPTIBLE and UNINTERRUPTIBLE sleeps on the same waitqueue; + * all wakeups are TASK_NORMAL in order to avoid O(n) lookups for the right + * sleeper state. + * + * - the exclusive mode; because this requires preserving the list order + * and this is hard. + * + * - custom wake functions; because you cannot give any guarantees about + * random code. + * + * As a side effect of this; the data structures are slimmer. + * + * One would recommend using this wait queue where possible. + */ + +struct task_struct; + +struct swait_queue_head { + raw_spinlock_t lock; + struct list_head task_list; +}; + +struct swait_queue { + struct task_struct *task; + struct list_head task_list; +}; + +#define __SWAITQUEUE_INITIALIZER(name) { \ + .task = current, \ + .task_list = LIST_HEAD_INIT((name).task_list), \ +} + +#define DECLARE_SWAITQUEUE(name) \ + struct swait_queue name = __SWAITQUEUE_INITIALIZER(name) + +#define __SWAIT_QUEUE_HEAD_INITIALIZER(name) { \ + .lock = __RAW_SPIN_LOCK_UNLOCKED(name.lock), \ + .task_list = LIST_HEAD_INIT((name).task_list), \ +} + +#define DECLARE_SWAIT_QUEUE_HEAD(name) \ + struct swait_queue_head name = __SWAIT_QUEUE_HEAD_INITIALIZER(name) + +extern void __init_swait_queue_head(struct swait_queue_head *q, const char *name, + struct lock_class_key *key); + +#define init_swait_queue_head(q) \ + do { \ + static struct lock_class_key __key; \ + __init_swait_queue_head((q), #q, &__key); \ + } while (0) + +#ifdef CONFIG_LOCKDEP +# define __SWAIT_QUEUE_HEAD_INIT_ONSTACK(name) \ + ({ init_swait_queue_head(&name); name; }) +# define DECLARE_SWAIT_QUEUE_HEAD_ONSTACK(name) \ + struct swait_queue_head name = __SWAIT_QUEUE_HEAD_INIT_ONSTACK(name) +#else +# define DECLARE_SWAIT_QUEUE_HEAD_ONSTACK(name) \ + DECLARE_SWAIT_QUEUE_HEAD(name) +#endif + +static inline int swait_active(struct swait_queue_head *q) +{ + return !list_empty(&q->task_list); +} + +extern void swake_up(struct swait_queue_head *q); +extern void swake_up_all(struct swait_queue_head *q); +extern void swake_up_locked(struct swait_queue_head *q); +extern void swake_up_all_locked(struct swait_queue_head *q); + +extern void __prepare_to_swait(struct swait_queue_head *q, struct swait_queue *wait); +extern void prepare_to_swait(struct swait_queue_head *q, struct swait_queue *wait, int state); +extern long prepare_to_swait_event(struct swait_queue_head *q, struct swait_queue *wait, int state); + +extern void __finish_swait(struct swait_queue_head *q, struct swait_queue *wait); +extern void finish_swait(struct swait_queue_head *q, struct swait_queue *wait); + +/* as per ___wait_event() but for swait, therefore "exclusive == 0" */ +#define ___swait_event(wq, condition, state, ret, cmd) \ +({ \ + struct swait_queue __wait; \ + long __ret = ret; \ + \ + INIT_LIST_HEAD(&__wait.task_list); \ + for (;;) { \ + long __int = prepare_to_swait_event(&wq, &__wait, state);\ + \ + if (condition) \ + break; \ + \ + if (___wait_is_interruptible(state) && __int) { \ + __ret = __int; \ + break; \ + } \ + \ + cmd; \ + } \ + finish_swait(&wq, &__wait); \ + __ret; \ +}) + +#define __swait_event(wq, condition) \ + (void)___swait_event(wq, condition, TASK_UNINTERRUPTIBLE, 0, \ + schedule()) + +#define swait_event(wq, condition) \ +do { \ + if (condition) \ + break; \ + __swait_event(wq, condition); \ +} while (0) + +#define __swait_event_timeout(wq, condition, timeout) \ + ___swait_event(wq, ___wait_cond_timeout(condition), \ + TASK_UNINTERRUPTIBLE, timeout, \ + __ret = schedule_timeout(__ret)) + +#define swait_event_timeout(wq, condition, timeout) \ +({ \ + long __ret = timeout; \ + if (!___wait_cond_timeout(condition)) \ + __ret = __swait_event_timeout(wq, condition, timeout); \ + __ret; \ +}) + +#define __swait_event_interruptible(wq, condition) \ + ___swait_event(wq, condition, TASK_INTERRUPTIBLE, 0, \ + schedule()) + +#define swait_event_interruptible(wq, condition) \ +({ \ + int __ret = 0; \ + if (!(condition)) \ + __ret = __swait_event_interruptible(wq, condition); \ + __ret; \ +}) + +#define __swait_event_interruptible_timeout(wq, condition, timeout) \ + ___swait_event(wq, ___wait_cond_timeout(condition), \ + TASK_INTERRUPTIBLE, timeout, \ + __ret = schedule_timeout(__ret)) + +#define swait_event_interruptible_timeout(wq, condition, timeout) \ +({ \ + long __ret = timeout; \ + if (!___wait_cond_timeout(condition)) \ + __ret = __swait_event_interruptible_timeout(wq, \ + condition, timeout); \ + __ret; \ +}) + +#endif /* _LINUX_SWAIT_H */ Index: linux-4.4.25-rt35/include/linux/swap.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/swap.h +++ linux-4.4.25-rt35/include/linux/swap.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:14 @ #include <linux/fs.h> #include <linux/atomic.h> #include <linux/page-flags.h> +#include <linux/locallock.h> #include <asm/page.h> struct notifier_block; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:256 @ struct swap_info_struct { void *workingset_eviction(struct address_space *mapping, struct page *page); bool workingset_refault(void *shadow); void workingset_activation(struct page *page); -extern struct list_lru workingset_shadow_nodes; +extern struct list_lru __workingset_shadow_nodes; +DECLARE_LOCAL_IRQ_LOCK(workingset_shadow_lock); static inline unsigned int workingset_node_pages(struct radix_tree_node *node) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:303 @ extern unsigned long nr_free_pagecache_p /* linux/mm/swap.c */ +DECLARE_LOCAL_IRQ_LOCK(swapvec_lock); extern void lru_cache_add(struct page *); extern void lru_cache_add_anon(struct page *page); extern void lru_cache_add_file(struct page *page); Index: linux-4.4.25-rt35/include/linux/swork.h =================================================================== --- /dev/null +++ linux-4.4.25-rt35/include/linux/swork.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +#ifndef _LINUX_SWORK_H +#define _LINUX_SWORK_H + +#include <linux/list.h> + +struct swork_event { + struct list_head item; + unsigned long flags; + void (*func)(struct swork_event *); +}; + +static inline void INIT_SWORK(struct swork_event *event, + void (*func)(struct swork_event *)) +{ + event->flags = 0; + event->func = func; +} + +bool swork_queue(struct swork_event *sev); + +int swork_get(void); +void swork_put(void); + +#endif /* _LINUX_SWORK_H */ Index: linux-4.4.25-rt35/include/linux/thread_info.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/thread_info.h +++ linux-4.4.25-rt35/include/linux/thread_info.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:105 @ static inline int test_ti_thread_flag(st #define test_thread_flag(flag) \ test_ti_thread_flag(current_thread_info(), flag) -#define tif_need_resched() test_thread_flag(TIF_NEED_RESCHED) +#ifdef CONFIG_PREEMPT_LAZY +#define tif_need_resched() (test_thread_flag(TIF_NEED_RESCHED) || \ + test_thread_flag(TIF_NEED_RESCHED_LAZY)) +#define tif_need_resched_now() (test_thread_flag(TIF_NEED_RESCHED)) +#define tif_need_resched_lazy() test_thread_flag(TIF_NEED_RESCHED_LAZY)) + +#else +#define tif_need_resched() test_thread_flag(TIF_NEED_RESCHED) +#define tif_need_resched_now() test_thread_flag(TIF_NEED_RESCHED) +#define tif_need_resched_lazy() 0 +#endif #if defined TIF_RESTORE_SIGMASK && !defined HAVE_SET_RESTORE_SIGMASK /* Index: linux-4.4.25-rt35/include/linux/timer.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/timer.h +++ linux-4.4.25-rt35/include/linux/timer.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:228 @ extern void add_timer(struct timer_list extern int try_to_del_timer_sync(struct timer_list *timer); -#ifdef CONFIG_SMP +#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT_RT_FULL) extern int del_timer_sync(struct timer_list *timer); #else # define del_timer_sync(t) del_timer(t) Index: linux-4.4.25-rt35/include/linux/trace_events.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/trace_events.h +++ linux-4.4.25-rt35/include/linux/trace_events.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:69 @ struct trace_entry { unsigned char flags; unsigned char preempt_count; int pid; + unsigned short migrate_disable; + unsigned short padding; + unsigned char preempt_lazy_count; }; #define TRACE_EVENT_TYPE_MAX \ Index: linux-4.4.25-rt35/include/linux/uaccess.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/uaccess.h +++ linux-4.4.25-rt35/include/linux/uaccess.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:27 @ static __always_inline void pagefault_di */ static inline void pagefault_disable(void) { + migrate_disable(); pagefault_disabled_inc(); /* * make sure to have issued the store before a pagefault @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:44 @ static inline void pagefault_enable(void */ barrier(); pagefault_disabled_dec(); + migrate_enable(); } /* Index: linux-4.4.25-rt35/include/linux/uprobes.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/uprobes.h +++ linux-4.4.25-rt35/include/linux/uprobes.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:30 @ #include <linux/errno.h> #include <linux/rbtree.h> #include <linux/types.h> +#include <linux/wait.h> struct vm_area_struct; struct mm_struct; Index: linux-4.4.25-rt35/include/linux/vmstat.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/vmstat.h +++ linux-4.4.25-rt35/include/linux/vmstat.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:36 @ DECLARE_PER_CPU(struct vm_event_state, v */ static inline void __count_vm_event(enum vm_event_item item) { + preempt_disable_rt(); raw_cpu_inc(vm_event_states.event[item]); + preempt_enable_rt(); } static inline void count_vm_event(enum vm_event_item item) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:48 @ static inline void count_vm_event(enum v static inline void __count_vm_events(enum vm_event_item item, long delta) { + preempt_disable_rt(); raw_cpu_add(vm_event_states.event[item], delta); + preempt_enable_rt(); } static inline void count_vm_events(enum vm_event_item item, long delta) Index: linux-4.4.25-rt35/include/linux/wait.h =================================================================== --- linux-4.4.25-rt35.orig/include/linux/wait.h +++ linux-4.4.25-rt35/include/linux/wait.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:11 @ #include <linux/spinlock.h> #include <asm/current.h> #include <uapi/linux/wait.h> +#include <linux/atomic.h> typedef struct __wait_queue wait_queue_t; typedef int (*wait_queue_func_t)(wait_queue_t *wait, unsigned mode, int flags, void *key); Index: linux-4.4.25-rt35/include/net/dst.h =================================================================== --- linux-4.4.25-rt35.orig/include/net/dst.h +++ linux-4.4.25-rt35/include/net/dst.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:440 @ static inline void dst_confirm(struct ds static inline int dst_neigh_output(struct dst_entry *dst, struct neighbour *n, struct sk_buff *skb) { - const struct hh_cache *hh; + struct hh_cache *hh; if (dst->pending_confirm) { unsigned long now = jiffies; Index: linux-4.4.25-rt35/include/net/neighbour.h =================================================================== --- linux-4.4.25-rt35.orig/include/net/neighbour.h +++ linux-4.4.25-rt35/include/net/neighbour.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:449 @ static inline int neigh_hh_bridge(struct } #endif -static inline int neigh_hh_output(const struct hh_cache *hh, struct sk_buff *skb) +static inline int neigh_hh_output(struct hh_cache *hh, struct sk_buff *skb) { unsigned int seq; int hh_len; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:504 @ struct neighbour_cb { #define NEIGH_CB(skb) ((struct neighbour_cb *)(skb)->cb) -static inline void neigh_ha_snapshot(char *dst, const struct neighbour *n, +static inline void neigh_ha_snapshot(char *dst, struct neighbour *n, const struct net_device *dev) { unsigned int seq; Index: linux-4.4.25-rt35/include/net/netns/ipv4.h =================================================================== --- linux-4.4.25-rt35.orig/include/net/netns/ipv4.h +++ linux-4.4.25-rt35/include/net/netns/ipv4.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:73 @ struct netns_ipv4 { int sysctl_icmp_echo_ignore_all; int sysctl_icmp_echo_ignore_broadcasts; + int sysctl_icmp_echo_sysrq; int sysctl_icmp_ignore_bogus_error_responses; int sysctl_icmp_ratelimit; int sysctl_icmp_ratemask; Index: linux-4.4.25-rt35/include/trace/events/hist.h =================================================================== --- /dev/null +++ linux-4.4.25-rt35/include/trace/events/hist.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM hist + +#if !defined(_TRACE_HIST_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_HIST_H + +#include "latency_hist.h" +#include <linux/tracepoint.h> + +#if !defined(CONFIG_PREEMPT_OFF_HIST) && !defined(CONFIG_INTERRUPT_OFF_HIST) +#define trace_preemptirqsoff_hist(a, b) +#define trace_preemptirqsoff_hist_rcuidle(a, b) +#else +TRACE_EVENT(preemptirqsoff_hist, + + TP_PROTO(int reason, int starthist), + + TP_ARGS(reason, starthist), + + TP_STRUCT__entry( + __field(int, reason) + __field(int, starthist) + ), + + TP_fast_assign( + __entry->reason = reason; + __entry->starthist = starthist; + ), + + TP_printk("reason=%s starthist=%s", getaction(__entry->reason), + __entry->starthist ? "start" : "stop") +); +#endif + +#ifndef CONFIG_MISSED_TIMER_OFFSETS_HIST +#define trace_hrtimer_interrupt(a, b, c, d) +#else +TRACE_EVENT(hrtimer_interrupt, + + TP_PROTO(int cpu, long long offset, struct task_struct *curr, + struct task_struct *task), + + TP_ARGS(cpu, offset, curr, task), + + TP_STRUCT__entry( + __field(int, cpu) + __field(long long, offset) + __array(char, ccomm, TASK_COMM_LEN) + __field(int, cprio) + __array(char, tcomm, TASK_COMM_LEN) + __field(int, tprio) + ), + + TP_fast_assign( + __entry->cpu = cpu; + __entry->offset = offset; + memcpy(__entry->ccomm, curr->comm, TASK_COMM_LEN); + __entry->cprio = curr->prio; + memcpy(__entry->tcomm, task != NULL ? task->comm : "<none>", + task != NULL ? TASK_COMM_LEN : 7); + __entry->tprio = task != NULL ? task->prio : -1; + ), + + TP_printk("cpu=%d offset=%lld curr=%s[%d] thread=%s[%d]", + __entry->cpu, __entry->offset, __entry->ccomm, + __entry->cprio, __entry->tcomm, __entry->tprio) +); +#endif + +#endif /* _TRACE_HIST_H */ + +/* This part must be outside protection */ +#include <trace/define_trace.h> Index: linux-4.4.25-rt35/include/trace/events/latency_hist.h =================================================================== --- /dev/null +++ linux-4.4.25-rt35/include/trace/events/latency_hist.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:4 @ +#ifndef _LATENCY_HIST_H +#define _LATENCY_HIST_H + +enum hist_action { + IRQS_ON, + PREEMPT_ON, + TRACE_STOP, + IRQS_OFF, + PREEMPT_OFF, + TRACE_START, +}; + +static char *actions[] = { + "IRQS_ON", + "PREEMPT_ON", + "TRACE_STOP", + "IRQS_OFF", + "PREEMPT_OFF", + "TRACE_START", +}; + +static inline char *getaction(int action) +{ + if (action >= 0 && action <= sizeof(actions)/sizeof(actions[0])) + return actions[action]; + return "unknown"; +} + +#endif /* _LATENCY_HIST_H */ Index: linux-4.4.25-rt35/include/trace/events/writeback.h =================================================================== --- linux-4.4.25-rt35.orig/include/trace/events/writeback.h +++ linux-4.4.25-rt35/include/trace/events/writeback.h @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:137 @ DEFINE_EVENT(writeback_dirty_inode_templ #ifdef CREATE_TRACE_POINTS #ifdef CONFIG_CGROUP_WRITEBACK -static inline size_t __trace_wb_cgroup_size(struct bdi_writeback *wb) +static inline unsigned int __trace_wb_assign_cgroup(struct bdi_writeback *wb) { - return kernfs_path_len(wb->memcg_css->cgroup->kn) + 1; + return wb->memcg_css->cgroup->kn->ino; } -static inline void __trace_wb_assign_cgroup(char *buf, struct bdi_writeback *wb) -{ - struct cgroup *cgrp = wb->memcg_css->cgroup; - char *path; - - path = cgroup_path(cgrp, buf, kernfs_path_len(cgrp->kn) + 1); - WARN_ON_ONCE(path != buf); -} - -static inline size_t __trace_wbc_cgroup_size(struct writeback_control *wbc) -{ - if (wbc->wb) - return __trace_wb_cgroup_size(wbc->wb); - else - return 2; -} - -static inline void __trace_wbc_assign_cgroup(char *buf, - struct writeback_control *wbc) +static inline unsigned int __trace_wbc_assign_cgroup(struct writeback_control *wbc) { if (wbc->wb) - __trace_wb_assign_cgroup(buf, wbc->wb); + return __trace_wb_assign_cgroup(wbc->wb); else - strcpy(buf, "/"); + return -1U; } - #else /* CONFIG_CGROUP_WRITEBACK */ -static inline size_t __trace_wb_cgroup_size(struct bdi_writeback *wb) -{ - return 2; -} - -static inline void __trace_wb_assign_cgroup(char *buf, struct bdi_writeback *wb) -{ - strcpy(buf, "/"); -} - -static inline size_t __trace_wbc_cgroup_size(struct writeback_control *wbc) +static inline unsigned int __trace_wb_assign_cgroup(struct bdi_writeback *wb) { - return 2; + return -1U; } -static inline void __trace_wbc_assign_cgroup(char *buf, - struct writeback_control *wbc) +static inline unsigned int __trace_wbc_assign_cgroup(struct writeback_control *wbc) { - strcpy(buf, "/"); + return -1U; } #endif /* CONFIG_CGROUP_WRITEBACK */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:174 @ DECLARE_EVENT_CLASS(writeback_write_inod __array(char, name, 32) __field(unsigned long, ino) __field(int, sync_mode) - __dynamic_array(char, cgroup, __trace_wbc_cgroup_size(wbc)) + __field(unsigned int, cgroup_ino) ), TP_fast_assign( @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:182 @ DECLARE_EVENT_CLASS(writeback_write_inod dev_name(inode_to_bdi(inode)->dev), 32); __entry->ino = inode->i_ino; __entry->sync_mode = wbc->sync_mode; - __trace_wbc_assign_cgroup(__get_str(cgroup), wbc); + __entry->cgroup_ino = __trace_wbc_assign_cgroup(wbc); ), - TP_printk("bdi %s: ino=%lu sync_mode=%d cgroup=%s", + TP_printk("bdi %s: ino=%lu sync_mode=%d cgroup_ino=%u", __entry->name, __entry->ino, __entry->sync_mode, - __get_str(cgroup) + __entry->cgroup_ino ) ); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:219 @ DECLARE_EVENT_CLASS(writeback_work_class __field(int, range_cyclic) __field(int, for_background) __field(int, reason) - __dynamic_array(char, cgroup, __trace_wb_cgroup_size(wb)) + __field(unsigned int, cgroup_ino) ), TP_fast_assign( strncpy(__entry->name, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:231 @ DECLARE_EVENT_CLASS(writeback_work_class __entry->range_cyclic = work->range_cyclic; __entry->for_background = work->for_background; __entry->reason = work->reason; - __trace_wb_assign_cgroup(__get_str(cgroup), wb); + __entry->cgroup_ino = __trace_wb_assign_cgroup(wb); ), TP_printk("bdi %s: sb_dev %d:%d nr_pages=%ld sync_mode=%d " - "kupdate=%d range_cyclic=%d background=%d reason=%s cgroup=%s", + "kupdate=%d range_cyclic=%d background=%d reason=%s cgroup_ino=%u", __entry->name, MAJOR(__entry->sb_dev), MINOR(__entry->sb_dev), __entry->nr_pages, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:243 @ DECLARE_EVENT_CLASS(writeback_work_class __entry->range_cyclic, __entry->for_background, __print_symbolic(__entry->reason, WB_WORK_REASON), - __get_str(cgroup) + __entry->cgroup_ino ) ); #define DEFINE_WRITEBACK_WORK_EVENT(name) \ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:273 @ DECLARE_EVENT_CLASS(writeback_class, TP_ARGS(wb), TP_STRUCT__entry( __array(char, name, 32) - __dynamic_array(char, cgroup, __trace_wb_cgroup_size(wb)) + __field(unsigned int, cgroup_ino) ), TP_fast_assign( strncpy(__entry->name, dev_name(wb->bdi->dev), 32); - __trace_wb_assign_cgroup(__get_str(cgroup), wb); + __entry->cgroup_ino = __trace_wb_assign_cgroup(wb); ), - TP_printk("bdi %s: cgroup=%s", + TP_printk("bdi %s: cgroup_ino=%u", __entry->name, - __get_str(cgroup) + __entry->cgroup_ino ) ); #define DEFINE_WRITEBACK_EVENT(name) \ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:320 @ DECLARE_EVENT_CLASS(wbc_class, __field(int, range_cyclic) __field(long, range_start) __field(long, range_end) - __dynamic_array(char, cgroup, __trace_wbc_cgroup_size(wbc)) + __field(unsigned int, cgroup_ino) ), TP_fast_assign( @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:334 @ DECLARE_EVENT_CLASS(wbc_class, __entry->range_cyclic = wbc->range_cyclic; __entry->range_start = (long)wbc->range_start; __entry->range_end = (long)wbc->range_end; - __trace_wbc_assign_cgroup(__get_str(cgroup), wbc); + __entry->cgroup_ino = __trace_wbc_assign_cgroup(wbc); ), TP_printk("bdi %s: towrt=%ld skip=%ld mode=%d kupd=%d " "bgrd=%d reclm=%d cyclic=%d " - "start=0x%lx end=0x%lx cgroup=%s", + "start=0x%lx end=0x%lx cgroup_ino=%u", __entry->name, __entry->nr_to_write, __entry->pages_skipped, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:350 @ DECLARE_EVENT_CLASS(wbc_class, __entry->range_cyclic, __entry->range_start, __entry->range_end, - __get_str(cgroup) + __entry->cgroup_ino ) ) @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:371 @ TRACE_EVENT(writeback_queue_io, __field(long, age) __field(int, moved) __field(int, reason) - __dynamic_array(char, cgroup, __trace_wb_cgroup_size(wb)) + __field(unsigned int, cgroup_ino) ), TP_fast_assign( unsigned long *older_than_this = work->older_than_this; @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:381 @ TRACE_EVENT(writeback_queue_io, (jiffies - *older_than_this) * 1000 / HZ : -1; __entry->moved = moved; __entry->reason = work->reason; - __trace_wb_assign_cgroup(__get_str(cgroup), wb); + __entry->cgroup_ino = __trace_wb_assign_cgroup(wb); ), - TP_printk("bdi %s: older=%lu age=%ld enqueue=%d reason=%s cgroup=%s", + TP_printk("bdi %s: older=%lu age=%ld enqueue=%d reason=%s cgroup_ino=%u", __entry->name, __entry->older, /* older_than_this in jiffies */ __entry->age, /* older_than_this in relative milliseconds */ __entry->moved, __print_symbolic(__entry->reason, WB_WORK_REASON), - __get_str(cgroup) + __entry->cgroup_ino ) ); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:457 @ TRACE_EVENT(bdi_dirty_ratelimit, __field(unsigned long, dirty_ratelimit) __field(unsigned long, task_ratelimit) __field(unsigned long, balanced_dirty_ratelimit) - __dynamic_array(char, cgroup, __trace_wb_cgroup_size(wb)) + __field(unsigned int, cgroup_ino) ), TP_fast_assign( @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:469 @ TRACE_EVENT(bdi_dirty_ratelimit, __entry->task_ratelimit = KBps(task_ratelimit); __entry->balanced_dirty_ratelimit = KBps(wb->balanced_dirty_ratelimit); - __trace_wb_assign_cgroup(__get_str(cgroup), wb); + __entry->cgroup_ino = __trace_wb_assign_cgroup(wb); ), TP_printk("bdi %s: " "write_bw=%lu awrite_bw=%lu dirty_rate=%lu " "dirty_ratelimit=%lu task_ratelimit=%lu " - "balanced_dirty_ratelimit=%lu cgroup=%s", + "balanced_dirty_ratelimit=%lu cgroup_ino=%u", __entry->bdi, __entry->write_bw, /* write bandwidth */ __entry->avg_write_bw, /* avg write bandwidth */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:483 @ TRACE_EVENT(bdi_dirty_ratelimit, __entry->dirty_ratelimit, /* base ratelimit */ __entry->task_ratelimit, /* ratelimit with position control */ __entry->balanced_dirty_ratelimit, /* the balanced ratelimit */ - __get_str(cgroup) + __entry->cgroup_ino ) ); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:521 @ TRACE_EVENT(balance_dirty_pages, __field( long, pause) __field(unsigned long, period) __field( long, think) - __dynamic_array(char, cgroup, __trace_wb_cgroup_size(wb)) + __field(unsigned int, cgroup_ino) ), TP_fast_assign( @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:544 @ TRACE_EVENT(balance_dirty_pages, __entry->period = period * 1000 / HZ; __entry->pause = pause * 1000 / HZ; __entry->paused = (jiffies - start_time) * 1000 / HZ; - __trace_wb_assign_cgroup(__get_str(cgroup), wb); + __entry->cgroup_ino = __trace_wb_assign_cgroup(wb); ), @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:553 @ TRACE_EVENT(balance_dirty_pages, "bdi_setpoint=%lu bdi_dirty=%lu " "dirty_ratelimit=%lu task_ratelimit=%lu " "dirtied=%u dirtied_pause=%u " - "paused=%lu pause=%ld period=%lu think=%ld cgroup=%s", + "paused=%lu pause=%ld period=%lu think=%ld cgroup_ino=%u", __entry->bdi, __entry->limit, __entry->setpoint, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:568 @ TRACE_EVENT(balance_dirty_pages, __entry->pause, /* ms */ __entry->period, /* ms */ __entry->think, /* ms */ - __get_str(cgroup) + __entry->cgroup_ino ) ); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:582 @ TRACE_EVENT(writeback_sb_inodes_requeue, __field(unsigned long, ino) __field(unsigned long, state) __field(unsigned long, dirtied_when) - __dynamic_array(char, cgroup, - __trace_wb_cgroup_size(inode_to_wb(inode))) + __field(unsigned int, cgroup_ino) ), TP_fast_assign( @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:591 @ TRACE_EVENT(writeback_sb_inodes_requeue, __entry->ino = inode->i_ino; __entry->state = inode->i_state; __entry->dirtied_when = inode->dirtied_when; - __trace_wb_assign_cgroup(__get_str(cgroup), inode_to_wb(inode)); + __entry->cgroup_ino = __trace_wb_assign_cgroup(inode_to_wb(inode)); ), - TP_printk("bdi %s: ino=%lu state=%s dirtied_when=%lu age=%lu cgroup=%s", + TP_printk("bdi %s: ino=%lu state=%s dirtied_when=%lu age=%lu cgroup_ino=%u", __entry->name, __entry->ino, show_inode_state(__entry->state), __entry->dirtied_when, (jiffies - __entry->dirtied_when) / HZ, - __get_str(cgroup) + __entry->cgroup_ino ) ); @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:656 @ DECLARE_EVENT_CLASS(writeback_single_ino __field(unsigned long, writeback_index) __field(long, nr_to_write) __field(unsigned long, wrote) - __dynamic_array(char, cgroup, __trace_wbc_cgroup_size(wbc)) + __field(unsigned int, cgroup_ino) ), TP_fast_assign( @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:668 @ DECLARE_EVENT_CLASS(writeback_single_ino __entry->writeback_index = inode->i_mapping->writeback_index; __entry->nr_to_write = nr_to_write; __entry->wrote = nr_to_write - wbc->nr_to_write; - __trace_wbc_assign_cgroup(__get_str(cgroup), wbc); + __entry->cgroup_ino = __trace_wbc_assign_cgroup(wbc); ), TP_printk("bdi %s: ino=%lu state=%s dirtied_when=%lu age=%lu " - "index=%lu to_write=%ld wrote=%lu cgroup=%s", + "index=%lu to_write=%ld wrote=%lu cgroup_ino=%u", __entry->name, __entry->ino, show_inode_state(__entry->state), @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:681 @ DECLARE_EVENT_CLASS(writeback_single_ino __entry->writeback_index, __entry->nr_to_write, __entry->wrote, - __get_str(cgroup) + __entry->cgroup_ino ) ); Index: linux-4.4.25-rt35/init/Kconfig =================================================================== --- linux-4.4.25-rt35.orig/init/Kconfig +++ linux-4.4.25-rt35/init/Kconfig @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:501 @ config TINY_RCU config RCU_EXPERT bool "Make expert-level adjustments to RCU configuration" - default n + default y if PREEMPT_RT_FULL help This option needs to be enabled if you wish to make expert-level adjustments to RCU configuration. By default, @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:617 @ config RCU_FANOUT_LEAF config RCU_FAST_NO_HZ bool "Accelerate last non-dyntick-idle CPU's grace periods" - depends on NO_HZ_COMMON && SMP && RCU_EXPERT + depends on NO_HZ_COMMON && SMP && RCU_EXPERT && !PREEMPT_RT_FULL default n help This option permits CPUs to enter dynticks-idle state even if @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:644 @ config TREE_RCU_TRACE config RCU_BOOST bool "Enable RCU priority boosting" depends on RT_MUTEXES && PREEMPT_RCU && RCU_EXPERT - default n + default y if PREEMPT_RT_FULL help This option boosts the priority of preempted RCU readers that block the current preemptible RCU grace period for too long. @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1109 @ config CFS_BANDWIDTH config RT_GROUP_SCHED bool "Group scheduling for SCHED_RR/FIFO" depends on CGROUP_SCHED + depends on !PREEMPT_RT_FULL default n help This feature lets you explicitly allocate real CPU bandwidth @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1723 @ choice config SLAB bool "SLAB" + depends on !PREEMPT_RT_FULL help The regular slab allocator that is established and known to work well in all environments. It organizes cache hot objects in @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1742 @ config SLUB config SLOB depends on EXPERT bool "SLOB (Simple Allocator)" + depends on !PREEMPT_RT_FULL help SLOB replaces the stock allocator with a drastically simpler allocator. SLOB is generally more space efficient but @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:1752 @ endchoice config SLUB_CPU_PARTIAL default y - depends on SLUB && SMP + depends on SLUB && SMP && !PREEMPT_RT_FULL bool "SLUB per cpu partial cache" help Per cpu partial caches accellerate objects allocation and freeing Index: linux-4.4.25-rt35/init/Makefile =================================================================== --- linux-4.4.25-rt35.orig/init/Makefile +++ linux-4.4.25-rt35/init/Makefile @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:36 @ silent_chk_compile.h = : include/generated/compile.h: FORCE @$($(quiet)chk_compile.h) $(Q)$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ - "$(UTS_MACHINE)" "$(CONFIG_SMP)" "$(CONFIG_PREEMPT)" "$(CC) $(KBUILD_CFLAGS)" + "$(UTS_MACHINE)" "$(CONFIG_SMP)" "$(CONFIG_PREEMPT)" "$(CONFIG_PREEMPT_RT_FULL)" "$(CC) $(KBUILD_CFLAGS)" Index: linux-4.4.25-rt35/init/main.c =================================================================== --- linux-4.4.25-rt35.orig/init/main.c +++ linux-4.4.25-rt35/init/main.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:533 @ asmlinkage __visible void __init start_k setup_command_line(command_line); setup_nr_cpu_ids(); setup_per_cpu_areas(); + softirq_early_init(); smp_prepare_boot_cpu(); /* arch-specific boot-cpu hooks */ build_all_zonelists(NULL, NULL); Index: linux-4.4.25-rt35/ipc/msg.c =================================================================== --- linux-4.4.25-rt35.orig/ipc/msg.c +++ linux-4.4.25-rt35/ipc/msg.c @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:186 @ static void ss_wakeup(struct list_head * } } -static void expunge_all(struct msg_queue *msq, int res) +static void expunge_all(struct msg_queue *msq, int res, + struct wake_q_head *wake_q) { struct msg_receiver *msr, *t; list_for_each_entry_safe(msr, t, &msq->q_receivers, r_list) { - msr->r_msg = NULL; /* initialize expunge ordering */ - wake_up_process(msr->r_tsk); - /* - * Ensure that the wakeup is visible before setting r_msg as - * the receiving end depends on it: either spinning on a nil, - * or dealing with -EAGAIN cases. See lockless receive part 1 - * and 2 in do_msgrcv(). - */ - smp_wmb(); /* barrier (B) */ + + wake_q_add(wake_q, msr->r_tsk); msr->r_msg = ERR_PTR(res); } } @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:210 @ static void freeque(struct ipc_namespace { struct msg_msg *msg, *t; struct msg_queue *msq = container_of(ipcp, struct msg_queue, q_perm); + WAKE_Q(wake_q); - expunge_all(msq, -EIDRM); + expunge_all(msq, -EIDRM, &wake_q); ss_wakeup(&msq->q_senders, 1); msg_rmid(ns, msq); ipc_unlock_object(&msq->q_perm); + wake_up_q(&wake_q); rcu_read_unlock(); list_for_each_entry_safe(msg, t, &msq->q_messages, m_list) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:341 @ static int msgctl_down(struct ipc_namesp struct kern_ipc_perm *ipcp; struct msqid64_ds uninitialized_var(msqid64); struct msg_queue *msq; + WAKE_Q(wake_q); int err; if (cmd == IPC_SET) { @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:389 @ static int msgctl_down(struct ipc_namesp /* sleeping receivers might be excluded by * stricter permissions. */ - expunge_all(msq, -EAGAIN); + expunge_all(msq, -EAGAIN, &wake_q); /* sleeping senders might be able to send * due to a larger queue size. */ @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:402 @ static int msgctl_down(struct ipc_namesp out_unlock0: ipc_unlock_object(&msq->q_perm); + wake_up_q(&wake_q); out_unlock1: rcu_read_unlock(); out_up: @ linux-4.4.25-rt35/Documentation/hwlat_detector.txt:567 @ static int testmsg(struct msg_msg