Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux

jump label, locking/static_keys: Update docs

Signed-off-by: Jason Baron <jbaron@akamai.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: benh@kernel.crashing.org
Cc: bp@alien8.de
Cc: davem@davemloft.net
Cc: ddaney@caviumnetworks.com
Cc: heiko.carstens@de.ibm.com
Cc: linux-kernel@vger.kernel.org
Cc: liuj97@gmail.com
Cc: luto@amacapital.net
Cc: michael@ellerman.id.au
Cc: rabin@rab.in
Cc: ralf@linux-mips.org
Cc: rostedt@goodmis.org
Cc: vbabka@suse.cz
Cc: will.deacon@arm.com
Link: http://lkml.kernel.org/r/6b50f2f6423a2244f37f4b1d2d6c211b9dcdf4f8.1438227999.git.jbaron@akamai.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>

authored by

Jason Baron and committed by
Ingo Molnar
412758cb 2bf9e0ab

+99 -69
+53 -48
Documentation/static-keys.txt
··· 1 1 Static Keys 2 2 ----------- 3 3 4 - By: Jason Baron <jbaron@redhat.com> 4 + DEPRECATED API: 5 + 6 + The use of 'struct static_key' directly, is now DEPRECATED. In addition 7 + static_key_{true,false}() is also DEPRECATED. IE DO NOT use the following: 8 + 9 + struct static_key false = STATIC_KEY_INIT_FALSE; 10 + struct static_key true = STATIC_KEY_INIT_TRUE; 11 + static_key_true() 12 + static_key_false() 13 + 14 + The updated API replacements are: 15 + 16 + DEFINE_STATIC_KEY_TRUE(key); 17 + DEFINE_STATIC_KEY_FALSE(key); 18 + static_key_likely() 19 + statick_key_unlikely() 5 20 6 21 0) Abstract 7 22 ··· 24 9 performance-sensitive fast-path kernel code, via a GCC feature and a code 25 10 patching technique. A quick example: 26 11 27 - struct static_key key = STATIC_KEY_INIT_FALSE; 12 + DEFINE_STATIC_KEY_FALSE(key); 28 13 29 14 ... 30 15 31 - if (static_key_false(&key)) 16 + if (static_branch_unlikely(&key)) 32 17 do unlikely code 33 18 else 34 19 do likely code 35 20 36 21 ... 37 - static_key_slow_inc(); 22 + static_branch_enable(&key); 38 23 ... 39 - static_key_slow_inc(); 24 + static_branch_disable(&key); 40 25 ... 41 26 42 - The static_key_false() branch will be generated into the code with as little 27 + The static_branch_unlikely() branch will be generated into the code with as little 43 28 impact to the likely code path as possible. 44 29 45 30 ··· 71 56 72 57 For example, if we have a simple branch that is disabled by default: 73 58 74 - if (static_key_false(&key)) 59 + if (static_branch_unlikely(&key)) 75 60 printk("I am the true branch\n"); 76 61 77 62 Thus, by default the 'printk' will not be emitted. And the code generated will ··· 90 75 91 76 In order to make use of this optimization you must first define a key: 92 77 93 - struct static_key key; 94 - 95 - Which is initialized as: 96 - 97 - struct static_key key = STATIC_KEY_INIT_TRUE; 78 + DEFINE_STATIC_KEY_TRUE(key); 98 79 99 80 or: 100 81 101 - struct static_key key = STATIC_KEY_INIT_FALSE; 82 + DEFINE_STATIC_KEY_FALSE(key); 102 83 103 - If the key is not initialized, it is default false. The 'struct static_key', 104 - must be a 'global'. That is, it can't be allocated on the stack or dynamically 84 + 85 + The key must be global, that is, it can't be allocated on the stack or dynamically 105 86 allocated at run-time. 106 87 107 88 The key is then used in code as: 108 89 109 - if (static_key_false(&key)) 90 + if (static_branch_unlikely(&key)) 110 91 do unlikely code 111 92 else 112 93 do likely code 113 94 114 95 Or: 115 96 116 - if (static_key_true(&key)) 97 + if (static_branch_likely(&key)) 117 98 do likely code 118 99 else 119 100 do unlikely code 120 101 121 - A key that is initialized via 'STATIC_KEY_INIT_FALSE', must be used in a 122 - 'static_key_false()' construct. Likewise, a key initialized via 123 - 'STATIC_KEY_INIT_TRUE' must be used in a 'static_key_true()' construct. A 124 - single key can be used in many branches, but all the branches must match the 125 - way that the key has been initialized. 102 + Keys defined via DEFINE_STATIC_KEY_TRUE(), or DEFINE_STATIC_KEY_FALSE, may 103 + be used in either static_branch_likely() or static_branch_unlikely() 104 + statemnts. 126 105 127 - The branch(es) can then be switched via: 106 + Branch(es) can be set true via: 128 107 129 - static_key_slow_inc(&key); 108 + static_branch_enable(&key); 109 + 110 + or false via: 111 + 112 + static_branch_disable(&key); 113 + 114 + The branch(es) can then be switched via reference counts: 115 + 116 + static_branch_inc(&key); 130 117 ... 131 - static_key_slow_dec(&key); 118 + static_branch_dec(&key); 132 119 133 - Thus, 'static_key_slow_inc()' means 'make the branch true', and 134 - 'static_key_slow_dec()' means 'make the branch false' with appropriate 120 + Thus, 'static_branch_inc()' means 'make the branch true', and 121 + 'static_branch_dec()' means 'make the branch false' with appropriate 135 122 reference counting. For example, if the key is initialized true, a 136 - static_key_slow_dec(), will switch the branch to false. And a subsequent 137 - static_key_slow_inc(), will change the branch back to true. Likewise, if the 138 - key is initialized false, a 'static_key_slow_inc()', will change the branch to 139 - true. And then a 'static_key_slow_dec()', will again make the branch false. 140 - 141 - An example usage in the kernel is the implementation of tracepoints: 142 - 143 - static inline void trace_##name(proto) \ 144 - { \ 145 - if (static_key_false(&__tracepoint_##name.key)) \ 146 - __DO_TRACE(&__tracepoint_##name, \ 147 - TP_PROTO(data_proto), \ 148 - TP_ARGS(data_args), \ 149 - TP_CONDITION(cond)); \ 150 - } 151 - 152 - Tracepoints are disabled by default, and can be placed in performance critical 153 - pieces of the kernel. Thus, by using a static key, the tracepoints can have 154 - absolutely minimal impact when not in use. 123 + static_branch_dec(), will switch the branch to false. And a subsequent 124 + static_branch_inc(), will change the branch back to true. Likewise, if the 125 + key is initialized false, a 'static_branch_inc()', will change the branch to 126 + true. And then a 'static_branch_dec()', will again make the branch false. 155 127 156 128 157 129 4) Architecture level code patching interface, 'jump labels' ··· 152 150 153 151 * #define JUMP_LABEL_NOP_SIZE, see: arch/x86/include/asm/jump_label.h 154 152 155 - * __always_inline bool arch_static_branch(struct static_key *key), see: 153 + * __always_inline bool arch_static_branch(struct static_key *key, bool branch), see: 156 154 arch/x86/include/asm/jump_label.h 155 + 156 + * __always_inline bool arch_static_branch_jump(struct static_key *key, bool branch), 157 + see: arch/x86/include/asm/jump_label.h 157 158 158 159 * void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type), 159 160 see: arch/x86/kernel/jump_label.c ··· 178 173 { 179 174 int pid; 180 175 181 - + if (static_key_false(&key)) 176 + + if (static_branch_unlikely(&key)) 182 177 + printk("I am the true branch\n"); 183 178 184 179 rcu_read_lock();
+46 -21
include/linux/jump_label.h
··· 7 7 * Copyright (C) 2009-2012 Jason Baron <jbaron@redhat.com> 8 8 * Copyright (C) 2011-2012 Peter Zijlstra <pzijlstr@redhat.com> 9 9 * 10 + * DEPRECATED API: 11 + * 12 + * The use of 'struct static_key' directly, is now DEPRECATED. In addition 13 + * static_key_{true,false}() is also DEPRECATED. IE DO NOT use the following: 14 + * 15 + * struct static_key false = STATIC_KEY_INIT_FALSE; 16 + * struct static_key true = STATIC_KEY_INIT_TRUE; 17 + * static_key_true() 18 + * static_key_false() 19 + * 20 + * The updated API replacements are: 21 + * 22 + * DEFINE_STATIC_KEY_TRUE(key); 23 + * DEFINE_STATIC_KEY_FALSE(key); 24 + * static_key_likely() 25 + * statick_key_unlikely() 26 + * 10 27 * Jump labels provide an interface to generate dynamic branches using 11 - * self-modifying code. Assuming toolchain and architecture support, the result 12 - * of a "if (static_key_false(&key))" statement is an unconditional branch (which 13 - * defaults to false - and the true block is placed out of line). 28 + * self-modifying code. Assuming toolchain and architecture support, if we 29 + * define a "key" that is initially false via "DEFINE_STATIC_KEY_FALSE(key)", 30 + * an "if (static_branch_unlikely(&key))" statement is an unconditional branch 31 + * (which defaults to false - and the true block is placed out of line). 32 + * Similarly, we can define an initially true key via 33 + * "DEFINE_STATIC_KEY_TRUE(key)", and use it in the same 34 + * "if (static_branch_unlikely(&key))", in which case we will generate an 35 + * unconditional branch to the out-of-line true branch. Keys that are 36 + * initially true or false can be using in both static_branch_unlikely() 37 + * and static_branch_likely() statements. 14 38 * 15 - * However at runtime we can change the branch target using 16 - * static_key_slow_{inc,dec}(). These function as a 'reference' count on the key 17 - * object, and for as long as there are references all branches referring to 18 - * that particular key will point to the (out of line) true block. 39 + * At runtime we can change the branch target by setting the key 40 + * to true via a call to static_branch_enable(), or false using 41 + * static_branch_disable(). If the direction of the branch is switched by 42 + * these calls then we run-time modify the branch target via a 43 + * no-op -> jump or jump -> no-op conversion. For example, for an 44 + * initially false key that is used in an "if (static_branch_unlikely(&key))" 45 + * statement, setting the key to true requires us to patch in a jump 46 + * to the out-of-line of true branch. 19 47 * 20 - * Since this relies on modifying code, the static_key_slow_{inc,dec}() functions 48 + * In addtion to static_branch_{enable,disable}, we can also reference count 49 + * the key or branch direction via static_branch_{inc,dec}. Thus, 50 + * static_branch_inc() can be thought of as a 'make more true' and 51 + * static_branch_dec() as a 'make more false'. The inc()/dec() 52 + * interface is meant to be used exclusively from the inc()/dec() for a given 53 + * key. 54 + * 55 + * Since this relies on modifying code, the branch modifying functions 21 56 * must be considered absolute slow paths (machine wide synchronization etc.). 22 57 * OTOH, since the affected branches are unconditional, their runtime overhead 23 58 * will be absolutely minimal, esp. in the default (off) case where the total ··· 64 29 * cause significant performance degradation. Struct static_key_deferred and 65 30 * static_key_slow_dec_deferred() provide for this. 66 31 * 67 - * Lacking toolchain and or architecture support, jump labels fall back to a simple 68 - * conditional branch. 32 + * Lacking toolchain and or architecture support, static keys fall back to a 33 + * simple conditional branch. 69 34 * 70 - * struct static_key my_key = STATIC_KEY_INIT_TRUE; 71 - * 72 - * if (static_key_true(&my_key)) { 73 - * } 74 - * 75 - * will result in the true case being in-line and starts the key with a single 76 - * reference. Mixing static_key_true() and static_key_false() on the same key is not 77 - * allowed. 78 - * 79 - * Not initializing the key (static data is initialized to 0s anyway) is the 80 - * same as using STATIC_KEY_INIT_FALSE. 35 + * Additional babbling in: Documentation/static-keys.txt 81 36 */ 82 37 83 38 #if defined(CC_HAVE_ASM_GOTO) && defined(CONFIG_JUMP_LABEL)