Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux

asm-generic/tlb, arch: Invert CONFIG_HAVE_RCU_TABLE_INVALIDATE

Make issuing a TLB invalidate for page-table pages the normal case.

The reason is twofold:

- too many invalidates is safer than too few,
- most architectures use the linux page-tables natively
and would thus require this.

Make it an opt-out, instead of an opt-in.

No change in behavior intended.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Ingo Molnar <mingo@kernel.org>

authored by

Peter Zijlstra and committed by
Ingo Molnar
96bc9567 8b6dd0c4

+9 -8
+1 -1
arch/Kconfig
··· 383 383 config HAVE_RCU_TABLE_FREE 384 384 bool 385 385 386 - config HAVE_RCU_TABLE_INVALIDATE 386 + config HAVE_RCU_TABLE_NO_INVALIDATE 387 387 bool 388 388 389 389 config HAVE_MMU_GATHER_PAGE_SIZE
-1
arch/arm64/Kconfig
··· 149 149 select HAVE_PERF_USER_STACK_DUMP 150 150 select HAVE_REGS_AND_STACK_ACCESS_API 151 151 select HAVE_RCU_TABLE_FREE 152 - select HAVE_RCU_TABLE_INVALIDATE 153 152 select HAVE_RSEQ 154 153 select HAVE_STACKPROTECTOR 155 154 select HAVE_SYSCALL_TRACEPOINTS
+1
arch/powerpc/Kconfig
··· 218 218 select HAVE_PERF_REGS 219 219 select HAVE_PERF_USER_STACK_DUMP 220 220 select HAVE_RCU_TABLE_FREE if SMP 221 + select HAVE_RCU_TABLE_NO_INVALIDATE if HAVE_RCU_TABLE_FREE 221 222 select HAVE_MMU_GATHER_PAGE_SIZE 222 223 select HAVE_REGS_AND_STACK_ACCESS_API 223 224 select HAVE_RELIABLE_STACKTRACE if PPC_BOOK3S_64 && CPU_LITTLE_ENDIAN
+1
arch/sparc/Kconfig
··· 63 63 select HAVE_KRETPROBES 64 64 select HAVE_KPROBES 65 65 select HAVE_RCU_TABLE_FREE if SMP 66 + select HAVE_RCU_TABLE_NO_INVALIDATE if HAVE_RCU_TABLE_FREE 66 67 select HAVE_MEMBLOCK_NODE_MAP 67 68 select HAVE_ARCH_TRANSPARENT_HUGEPAGE 68 69 select HAVE_DYNAMIC_FTRACE
-1
arch/x86/Kconfig
··· 183 183 select HAVE_PERF_REGS 184 184 select HAVE_PERF_USER_STACK_DUMP 185 185 select HAVE_RCU_TABLE_FREE if PARAVIRT 186 - select HAVE_RCU_TABLE_INVALIDATE if HAVE_RCU_TABLE_FREE 187 186 select HAVE_REGS_AND_STACK_ACCESS_API 188 187 select HAVE_RELIABLE_STACKTRACE if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION 189 188 select HAVE_FUNCTION_ARG_ACCESS_API
+5 -4
include/asm-generic/tlb.h
··· 135 135 * When used, an architecture is expected to provide __tlb_remove_table() 136 136 * which does the actual freeing of these pages. 137 137 * 138 - * HAVE_RCU_TABLE_INVALIDATE 138 + * HAVE_RCU_TABLE_NO_INVALIDATE 139 139 * 140 - * This makes HAVE_RCU_TABLE_FREE call tlb_flush_mmu_tlbonly() before freeing 141 - * the page-table pages. Required if you use HAVE_RCU_TABLE_FREE and your 142 - * architecture uses the Linux page-tables natively. 140 + * This makes HAVE_RCU_TABLE_FREE avoid calling tlb_flush_mmu_tlbonly() before 141 + * freeing the page-table pages. This can be avoided if you use 142 + * HAVE_RCU_TABLE_FREE and your architecture does _NOT_ use the Linux 143 + * page-tables natively. 143 144 * 144 145 * MMU_GATHER_NO_RANGE 145 146 *
+1 -1
mm/mmu_gather.c
··· 157 157 */ 158 158 static inline void tlb_table_invalidate(struct mmu_gather *tlb) 159 159 { 160 - #ifdef CONFIG_HAVE_RCU_TABLE_INVALIDATE 160 + #ifndef CONFIG_HAVE_RCU_TABLE_NO_INVALIDATE 161 161 /* 162 162 * Invalidate page-table caches used by hardware walkers. Then we still 163 163 * need to RCU-sched wait while freeing the pages because software