1 .\" Copyright (c) 2000-2001 John H. Baldwin <jhb@FreeBSD.org>
3 .\" Redistribution and use in source and binary forms, with or without
4 .\" modification, are permitted provided that the following conditions
6 .\" 1. Redistributions of source code must retain the above copyright
7 .\" notice, this list of conditions and the following disclaimer.
8 .\" 2. Redistributions in binary form must reproduce the above copyright
9 .\" notice, this list of conditions and the following disclaimer in the
10 .\" documentation and/or other materials provided with the distribution.
12 .\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR
13 .\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
14 .\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
15 .\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT,
16 .\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
17 .\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
18 .\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
19 .\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
20 .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
21 .\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
34 .Nm atomic_interrupt_fence ,
36 .Nm atomic_readandclear ,
40 .Nm atomic_thread_fence
46 .Fn atomic_add_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
48 .Fn atomic_clear_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
50 .Fo atomic_cmpset_[acq_|rel_]<type>
51 .Fa "volatile <type> *dst"
56 .Fo atomic_fcmpset_[acq_|rel_]<type>
57 .Fa "volatile <type> *dst"
62 .Fn atomic_fetchadd_<type> "volatile <type> *p" "<type> v"
64 .Fn atomic_interrupt_fence "void"
66 .Fn atomic_load_[acq_]<type> "volatile <type> *p"
68 .Fn atomic_readandclear_<type> "volatile <type> *p"
70 .Fn atomic_set_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
72 .Fn atomic_subtract_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
74 .Fn atomic_store_[rel_]<type> "volatile <type> *p" "<type> v"
76 .Fn atomic_swap_<type> "volatile <type> *p" "<type> v"
78 .Fn atomic_testandclear_<type> "volatile <type> *p" "u_int v"
80 .Fn atomic_testandset_<type> "volatile <type> *p" "u_int v"
82 .Fn atomic_thread_fence_[acq|acq_rel|rel|seq_cst] "void"
84 Atomic operations are commonly used to implement reference counts and as
85 building blocks for synchronization primitives, such as mutexes.
87 All of these operations are performed
89 across multiple threads and in the presence of interrupts, meaning that they
90 are performed in an indivisible manner from the perspective of concurrently
91 running threads and interrupt handlers.
93 On all architectures supported by
95 ordinary loads and stores of integers in cache-coherent memory are
96 inherently atomic if the integer is naturally aligned and its size does not
97 exceed the processor's word size.
98 However, such loads and stores may be elided from the program by
99 the compiler, whereas atomic operations are always performed.
101 When atomic operations are performed on cache-coherent memory, all
102 operations on the same location are totally ordered.
104 When an atomic load is performed on a location in cache-coherent memory,
105 it reads the entire value that was defined by the last atomic store to
106 each byte of the location.
107 An atomic load will never return a value out of thin air.
108 When an atomic store is performed on a location, no other thread or
109 interrupt handler will observe a
111 or partial modification of the location.
113 Except as noted below, the semantics of these operations are almost
114 identical to the semantics of similarly named C11 atomic operations.
116 Most atomic operations act upon a specific
118 That type is indicated in the function name.
119 In contrast to C11 atomic operations,
121 atomic operations are performed on ordinary integer types.
122 The available types are:
124 .Bl -tag -offset indent -width short -compact
128 unsigned long integer
130 unsigned integer the size of a pointer
132 unsigned 32-bit integer
134 unsigned 64-bit integer
137 For example, the function to atomically add two integers is called
140 Certain architectures also provide operations for types smaller than
143 .Bl -tag -offset indent -width short -compact
147 unsigned short integer
149 unsigned 8-bit integer
151 unsigned 16-bit integer
154 These types must not be used in machine-independent code.
155 .Ss Acquire and Release Operations
156 By default, a thread's accesses to different memory locations might not be
159 that is, the order in which the accesses appear in the source code.
160 To optimize the program's execution, both the compiler and processor might
161 reorder the thread's accesses.
162 However, both ensure that their reordering of the accesses is not visible to
164 Otherwise, the traditional memory model that is expected by single-threaded
165 programs would be violated.
166 Nonetheless, other threads in a multithreaded program, such as the
168 kernel, might observe the reordering.
169 Moreover, in some cases, such as the implementation of synchronization between
170 threads, arbitrary reordering might result in the incorrect execution of the
172 To constrain the reordering that both the compiler and processor might perform
173 on a thread's accesses, a programmer can use atomic operations with
179 Atomic operations on memory have up to three variants.
182 variant, performs the operation without imposing any ordering constraints on
183 accesses to other memory locations.
184 This variant is the default.
185 The second variant has acquire semantics, and the third variant has release
188 When an atomic operation has acquire semantics, the operation must have
189 completed before any subsequent load or store (by program order) is
191 Conversely, acquire semantics do not require that prior loads or stores have
192 completed before the atomic operation is performed.
193 An atomic operation can only have acquire semantics if it performs a load
195 To denote acquire semantics, the suffix
197 is inserted into the function name immediately prior to the
198 .Dq Li _ Ns Aq Fa type
200 For example, to subtract two integers ensuring that the subtraction is
201 completed before any subsequent loads and stores are performed, use
202 .Fn atomic_subtract_acq_int .
204 When an atomic operation has release semantics, all prior loads or stores
205 (by program order) must have completed before the operation is performed.
206 Conversely, release semantics do not require that the atomic operation must
207 have completed before any subsequent load or store is performed.
208 An atomic operation can only have release semantics if it performs a store
210 To denote release semantics, the suffix
212 is inserted into the function name immediately prior to the
213 .Dq Li _ Ns Aq Fa type
215 For example, to add two long integers ensuring that all prior loads and
216 stores are completed before the addition is performed, use
217 .Fn atomic_add_rel_long .
219 When a release operation by one thread
220 .Em synchronizes with
221 an acquire operation by another thread, usually meaning that the acquire
222 operation reads the value written by the release operation, then the effects
223 of all prior stores by the releasing thread must become visible to
224 subsequent loads by the acquiring thread.
225 Moreover, the effects of all stores (by other threads) that were visible to
226 the releasing thread must also become visible to the acquiring thread.
227 These rules only apply to the synchronizing threads.
228 Other threads might observe these stores in a different order.
230 In effect, atomic operations with acquire and release semantics establish
231 one-way barriers to reordering that enable the implementations of
232 synchronization primitives to express their ordering requirements without
233 also imposing unnecessary ordering.
234 For example, for a critical section guarded by a mutex, an acquire operation
235 when the mutex is locked and a release operation when the mutex is unlocked
236 will prevent any loads or stores from moving outside of the critical
238 However, they will not prevent the compiler or processor from moving loads
239 or stores into the critical section, which does not violate the semantics of
241 .Ss Thread Fence Operations
242 Alternatively, a programmer can use atomic thread fence operations to
243 constrain the reordering of accesses.
244 In contrast to other atomic operations, fences do not, themselves, access
247 When a fence has acquire semantics, all prior loads (by program order) must
248 have completed before any subsequent load or store is performed.
249 Thus, an acquire fence is a two-way barrier for load operations.
250 To denote acquire semantics, the suffix
252 is appended to the function name, for example,
253 .Fn atomic_thread_fence_acq .
255 When a fence has release semantics, all prior loads or stores (by program
256 order) must have completed before any subsequent store operation is
258 Thus, a release fence is a two-way barrier for store operations.
259 To denote release semantics, the suffix
261 is appended to the function name, for example,
262 .Fn atomic_thread_fence_rel .
265 .Fn atomic_thread_fence_acq_rel
266 implements both acquire and release semantics, it is not a full barrier.
267 For example, a store prior to the fence (in program order) may be completed
268 after a load subsequent to the fence.
270 .Fn atomic_thread_fence_seq_cst
271 implements a full barrier.
272 Neither loads nor stores may cross this barrier in either direction.
274 In C11, a release fence by one thread synchronizes with an acquire fence by
275 another thread when an atomic load that is prior to the acquire fence (by
276 program order) reads the value written by an atomic store that is subsequent
277 to the release fence.
280 because of the atomicity of ordinary, naturally
281 aligned loads and stores, fences can also be synchronized by ordinary loads
283 This simplifies the implementation and use of some synchronization
287 Since neither a compiler nor a processor can foresee which (atomic) load
288 will read the value written by an (atomic) store, the ordering constraints
289 imposed by fences must be more restrictive than acquire loads and release
291 Essentially, this is why fences are two-way barriers.
293 Although fences impose more restrictive ordering than acquire loads and
294 release stores, by separating access from ordering, they can sometimes
295 facilitate more efficient implementations of synchronization primitives.
296 For example, they can be used to avoid executing a memory barrier until a
297 memory access shows that some condition is satisfied.
298 .Ss Interrupt Fence Operations
300 .Fn atomic_interrupt_fence()
301 function establishes ordering between its call location and any interrupt
302 handler executing on the same CPU.
303 It is modeled after the similar C11 function
304 .Fn atomic_signal_fence() ,
305 and adapted for the kernel environment.
306 .Ss Multiple Processors
307 In multiprocessor systems, the atomicity of the atomic operations on memory
308 depends on support for cache coherence in the underlying architecture.
309 In general, cache coherence on the default memory type,
310 .Dv VM_MEMATTR_DEFAULT ,
311 is guaranteed by all architectures that are supported by
313 For example, cache coherence is guaranteed on write-back memory by the
318 However, on some architectures, cache coherence might not be enabled on all
320 To determine if cache coherence is enabled for a non-default memory type,
321 consult the architecture's documentation.
323 This section describes the semantics of each operation using a C like notation.
325 .It Fn atomic_add p v
326 .Bd -literal -compact
329 .It Fn atomic_clear p v
330 .Bd -literal -compact
333 .It Fn atomic_cmpset dst old new
334 .Bd -literal -compact
343 Some architectures do not implement the
345 functions for the types
352 .It Fn atomic_fcmpset dst *old new
355 On architectures implementing
357 operation in hardware, the functionality can be described as
358 .Bd -literal -offset indent -compact
367 On architectures which provide
368 .Em Load Linked/Store Conditional
369 primitive, the write to
371 might also fail for several reasons, most important of which
372 is a parallel write to
374 cache line by other CPU.
377 function also returns
382 Some architectures do not implement the
384 functions for the types
391 .It Fn atomic_fetchadd p v
392 .Bd -literal -compact
401 functions are only implemented for the types
406 and do not have any variants with memory barriers at this time.
409 .Bd -literal -compact
412 .It Fn atomic_readandclear p
413 .Bd -literal -compact
421 .Fn atomic_readandclear
422 functions are not implemented for the types
429 and do not have any variants with memory barriers at this time.
431 .It Fn atomic_set p v
432 .Bd -literal -compact
435 .It Fn atomic_subtract p v
436 .Bd -literal -compact
439 .It Fn atomic_store p v
440 .Bd -literal -compact
443 .It Fn atomic_swap p v
444 .Bd -literal -compact
453 functions are not implemented for the types
460 and do not have any variants with memory barriers at this time.
462 .It Fn atomic_testandclear p v
463 .Bd -literal -compact
464 bit = 1 << (v % (sizeof(*p) * NBBY));
465 tmp = (*p & bit) != 0;
471 .It Fn atomic_testandset p v
472 .Bd -literal -compact
473 bit = 1 << (v % (sizeof(*p) * NBBY));
474 tmp = (*p & bit) != 0;
481 .Fn atomic_testandset
483 .Fn atomic_testandclear
484 functions are only implemented for the types
489 and do not have any variants with memory barriers at this time.
493 is currently not implemented for some of the atomic operations on the
502 function returns the result of the compare operation.
507 if the operation succeeded.
514 .Fn atomic_fetchadd ,
516 .Fn atomic_readandclear ,
519 functions return the value at the specified address.
521 .Fn atomic_testandset
523 .Fn atomic_testandclear
524 function returns the result of the test operation.
526 This example uses the
527 .Fn atomic_cmpset_acq_ptr
530 functions to obtain a sleep mutex and handle recursion.
539 /* Try to obtain mtx_lock once. */
540 #define _obtain_lock(mp, tid) \\
541 atomic_cmpset_acq_ptr(&(mp)->mtx_lock, MTX_UNOWNED, (tid))
543 /* Get a sleep lock, deal with recursion inline. */
544 #define _get_sleep_lock(mp, tid, opts, file, line) do { \\
545 uintptr_t _tid = (uintptr_t)(tid); \\
547 if (!_obtain_lock(mp, tid)) { \\
548 if (((mp)->mtx_lock & MTX_FLAGMASK) != _tid) \\
549 _mtx_lock_sleep((mp), _tid, (opts), (file), (line));\\
551 atomic_set_ptr(&(mp)->mtx_lock, MTX_RECURSE); \\
552 (mp)->mtx_recurse++; \\
564 operations were introduced in
566 Initially, these operations were defined on the types
575 .Fn atomic_load_acq ,
576 .Fn atomic_readandclear ,
579 operations were added in
581 Simultaneously, the acquire and release variants were introduced, and
582 support was added for operation on the types
592 operation was added in
598 .Fn atomic_testandset
599 operations were added in
603 .Fn atomic_testandclear
605 .Fn atomic_thread_fence
606 operations were added in
609 The relaxed variants of
617 .Fn atomic_interrupt_fence
618 operation was added in