Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux

documentation: Update circular buffer for load-acquire/store-release

This commit replaces full barriers by targeted use of load-acquire and
store-release.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
[ paulmck: Restore comments as suggested by David Howells. ]

+19 -18
+19 -18
Documentation/circular-buffers.txt
··· 160 160 spin_lock(&producer_lock); 161 161 162 162 unsigned long head = buffer->head; 163 + /* The spin_unlock() and next spin_lock() provide needed ordering. */ 163 164 unsigned long tail = ACCESS_ONCE(buffer->tail); 164 165 165 166 if (CIRC_SPACE(head, tail, buffer->size) >= 1) { ··· 169 168 170 169 produce_item(item); 171 170 172 - smp_wmb(); /* commit the item before incrementing the head */ 173 - 174 - ACCESS_ONCE(buffer->head) = (head + 1) & (buffer->size - 1); 171 + smp_store_release(buffer->head, 172 + (head + 1) & (buffer->size - 1)); 175 173 176 174 /* wake_up() will make sure that the head is committed before 177 175 * waking anyone up */ ··· 200 200 201 201 spin_lock(&consumer_lock); 202 202 203 - unsigned long head = ACCESS_ONCE(buffer->head); 203 + /* Read index before reading contents at that index. */ 204 + unsigned long head = smp_load_acquire(buffer->head); 204 205 unsigned long tail = buffer->tail; 205 206 206 207 if (CIRC_CNT(head, tail, buffer->size) >= 1) { 207 - /* read index before reading contents at that index */ 208 - smp_rmb(); 209 208 210 209 /* extract one item from the buffer */ 211 210 struct item *item = buffer[tail]; 212 211 213 212 consume_item(item); 214 213 215 - smp_mb(); /* finish reading descriptor before incrementing tail */ 216 - 217 - ACCESS_ONCE(buffer->tail) = (tail + 1) & (buffer->size - 1); 214 + /* Finish reading descriptor before incrementing tail. */ 215 + smp_store_release(buffer->tail, 216 + (tail + 1) & (buffer->size - 1)); 218 217 } 219 218 220 219 spin_unlock(&consumer_lock); ··· 222 223 the new item, and then it shall make sure the CPU has finished reading the item 223 224 before it writes the new tail pointer, which will erase the item. 224 225 225 - 226 - Note the use of ACCESS_ONCE() in both algorithms to read the opposition index. 227 - This prevents the compiler from discarding and reloading its cached value - 228 - which some compilers will do across smp_read_barrier_depends(). This isn't 229 - strictly needed if you can be sure that the opposition index will _only_ be 230 - used the once. Similarly, ACCESS_ONCE() is used in both algorithms to 231 - write the thread's index. This documents the fact that we are writing 232 - to something that can be read concurrently and also prevents the compiler 233 - from tearing the store. 226 + Note the use of ACCESS_ONCE() and smp_load_acquire() to read the 227 + opposition index. This prevents the compiler from discarding and 228 + reloading its cached value - which some compilers will do across 229 + smp_read_barrier_depends(). This isn't strictly needed if you can 230 + be sure that the opposition index will _only_ be used the once. 231 + The smp_load_acquire() additionally forces the CPU to order against 232 + subsequent memory references. Similarly, smp_store_release() is used 233 + in both algorithms to write the thread's index. This documents the 234 + fact that we are writing to something that can be read concurrently, 235 + prevents the compiler from tearing the store, and enforces ordering 236 + against previous accesses. 234 237 235 238 236 239 ===============