1 .\" Copyright (c) 2000-2001 John H. Baldwin <jhb@FreeBSD.org>
2 .\" All rights reserved.
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice, this list of conditions and the following disclaimer.
9 .\" 2. Redistributions in binary form must reproduce the above copyright
10 .\" notice, this list of conditions and the following disclaimer in the
11 .\" documentation and/or other materials provided with the distribution.
13 .\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR
14 .\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
15 .\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
16 .\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT,
17 .\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
18 .\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
19 .\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
20 .\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
21 .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
22 .\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
36 .Nm atomic_readandclear ,
40 .Nm atomic_thread_fence
46 .Fn atomic_add_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
48 .Fn atomic_clear_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
50 .Fo atomic_cmpset_[acq_|rel_]<type>
51 .Fa "volatile <type> *dst"
56 .Fo atomic_fcmpset_[acq_|rel_]<type>
57 .Fa "volatile <type> *dst"
62 .Fn atomic_fetchadd_<type> "volatile <type> *p" "<type> v"
64 .Fn atomic_load_[acq_]<type> "volatile <type> *p"
66 .Fn atomic_readandclear_<type> "volatile <type> *p"
68 .Fn atomic_set_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
70 .Fn atomic_subtract_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
72 .Fn atomic_store_[rel_]<type> "volatile <type> *p" "<type> v"
74 .Fn atomic_swap_<type> "volatile <type> *p" "<type> v"
76 .Fn atomic_testandclear_<type> "volatile <type> *p" "u_int v"
78 .Fn atomic_testandset_<type> "volatile <type> *p" "u_int v"
80 .Fn atomic_thread_fence_[acq|acq_rel|rel|seq_cst] "void"
82 Atomic operations are commonly used to implement reference counts and as
83 building blocks for synchronization primitives, such as mutexes.
85 All of these operations are performed
87 across multiple threads and in the presence of interrupts, meaning that they
88 are performed in an indivisible manner from the perspective of concurrently
89 running threads and interrupt handlers.
91 On all architectures supported by
93 ordinary loads and stores of integers in cache-coherent memory are
94 inherently atomic if the integer is naturally aligned and its size does not
95 exceed the processor's word size.
96 However, such loads and stores may be elided from the program by
97 the compiler, whereas atomic operations are always performed.
99 When atomic operations are performed on cache-coherent memory, all
100 operations on the same location are totally ordered.
102 When an atomic load is performed on a location in cache-coherent memory,
103 it reads the entire value that was defined by the last atomic store to
104 each byte of the location.
105 An atomic load will never return a value out of thin air.
106 When an atomic store is performed on a location, no other thread or
107 interrupt handler will observe a
109 or partial modification of the location.
111 Except as noted below, the semantics of these operations are almost
112 identical to the semantics of similarly named C11 atomic operations.
114 Most atomic operations act upon a specific
116 That type is indicated in the function name.
117 In contrast to C11 atomic operations,
119 atomic operations are performed on ordinary integer types.
120 The available types are:
122 .Bl -tag -offset indent -width short -compact
126 unsigned long integer
128 unsigned integer the size of a pointer
130 unsigned 32-bit integer
132 unsigned 64-bit integer
135 For example, the function to atomically add two integers is called
138 Certain architectures also provide operations for types smaller than
141 .Bl -tag -offset indent -width short -compact
145 unsigned short integer
147 unsigned 8-bit integer
149 unsigned 16-bit integer
152 These types must not be used in machine-independent code.
153 .Ss Acquire and Release Operations
154 By default, a thread's accesses to different memory locations might not be
157 that is, the order in which the accesses appear in the source code.
158 To optimize the program's execution, both the compiler and processor might
159 reorder the thread's accesses.
160 However, both ensure that their reordering of the accesses is not visible to
162 Otherwise, the traditional memory model that is expected by single-threaded
163 programs would be violated.
164 Nonetheless, other threads in a multithreaded program, such as the
166 kernel, might observe the reordering.
167 Moreover, in some cases, such as the implementation of synchronization between
168 threads, arbitrary reordering might result in the incorrect execution of the
170 To constrain the reordering that both the compiler and processor might perform
171 on a thread's accesses, a programmer can use atomic operations with
177 Atomic operations on memory have up to three variants.
178 The first variant performs the operation without imposing any ordering
179 constraints on memory accesses to other locations.
180 The second variant has acquire semantics, and the third variant has release
183 When an atomic operation has acquire semantics, the operation must have
184 completed before any subsequent load or store (by program order) is
186 Conversely, acquire semantics do not require that prior loads or stores have
187 completed before the atomic operation is performed.
188 An atomic operation can only have acquire semantics if it performs a load
190 To denote acquire semantics, the suffix
192 is inserted into the function name immediately prior to the
193 .Dq Li _ Ns Aq Fa type
195 For example, to subtract two integers ensuring that the subtraction is
196 completed before any subsequent loads and stores are performed, use
197 .Fn atomic_subtract_acq_int .
199 When an atomic operation has release semantics, all prior loads or stores
200 (by program order) must have completed before the operation is performed.
201 Conversely, release semantics do not require that the atomic operation must
202 have completed before any subsequent load or store is performed.
203 An atomic operation can only have release semantics if it performs a store
205 To denote release semantics, the suffix
207 is inserted into the function name immediately prior to the
208 .Dq Li _ Ns Aq Fa type
210 For example, to add two long integers ensuring that all prior loads and
211 stores are completed before the addition is performed, use
212 .Fn atomic_add_rel_long .
214 When a release operation by one thread
215 .Em synchronizes with
216 an acquire operation by another thread, usually meaning that the acquire
217 operation reads the value written by the release operation, then the effects
218 of all prior stores by the releasing thread must become visible to
219 subsequent loads by the acquiring thread.
220 Moreover, the effects of all stores (by other threads) that were visible to
221 the releasing thread must also become visible to the acquiring thread.
222 These rules only apply to the synchronizing threads.
223 Other threads might observe these stores in a different order.
225 In effect, atomic operations with acquire and release semantics establish
226 one-way barriers to reordering that enable the implementations of
227 synchronization primitives to express their ordering requirements without
228 also imposing unnecessary ordering.
229 For example, for a critical section guarded by a mutex, an acquire operation
230 when the mutex is locked and a release operation when the mutex is unlocked
231 will prevent any loads or stores from moving outside of the critical
233 However, they will not prevent the compiler or processor from moving loads
234 or stores into the critical section, which does not violate the semantics of
236 .Ss Thread Fence Operations
237 Alternatively, a programmer can use atomic thread fence operations to
238 constrain the reordering of accesses.
239 In contrast to other atomic operations, fences do not, themselves, access
242 When a fence has acquire semantics, all prior loads (by program order) must
243 have completed before any subsequent load or store is performed.
244 Thus, an acquire fence is a two-way barrier for load operations.
245 To denote acquire semantics, the suffix
247 is appended to the function name, for example,
248 .Fn atomic_thread_fence_acq .
250 When a fence has release semantics, all prior loads or stores (by program
251 order) must have completed before any subsequent store operation is
253 Thus, a release fence is a two-way barrier for store operations.
254 To denote release semantics, the suffix
256 is appended to the function name, for example,
257 .Fn atomic_thread_fence_rel .
260 .Fn atomic_thread_fence_acq_rel
261 implements both acquire and release semantics, it is not a full barrier.
262 For example, a store prior to the fence (in program order) may be completed
263 after a load subsequent to the fence.
265 .Fn atomic_thread_fence_seq_cst
266 implements a full barrier.
267 Neither loads nor stores may cross this barrier in either direction.
269 In C11, a release fence by one thread synchronizes with an acquire fence by
270 another thread when an atomic load that is prior to the acquire fence (by
271 program order) reads the value written by an atomic store that is subsequent
272 to the release fence.
273 In constrast, in FreeBSD, because of the atomicity of ordinary, naturally
274 aligned loads and stores, fences can also be synchronized by ordinary loads
276 This simplifies the implementation and use of some synchronization
280 Since neither a compiler nor a processor can foresee which (atomic) load
281 will read the value written by an (atomic) store, the ordering constraints
282 imposed by fences must be more restrictive than acquire loads and release
284 Essentially, this is why fences are two-way barriers.
286 Although fences impose more restrictive ordering than acquire loads and
287 release stores, by separating access from ordering, they can sometimes
288 facilitate more efficient implementations of synchronization primitives.
289 For example, they can be used to avoid executing a memory barrier until a
290 memory access shows that some condition is satisfied.
291 .Ss Multiple Processors
292 In multiprocessor systems, the atomicity of the atomic operations on memory
293 depends on support for cache coherence in the underlying architecture.
294 In general, cache coherence on the default memory type,
295 .Dv VM_MEMATTR_DEFAULT ,
296 is guaranteed by all architectures that are supported by
298 For example, cache coherence is guaranteed on write-back memory by the
303 However, on some architectures, cache coherence might not be enabled on all
305 To determine if cache coherence is enabled for a non-default memory type,
306 consult the architecture's documentation.
308 This section describes the semantics of each operation using a C like notation.
310 .It Fn atomic_add p v
311 .Bd -literal -compact
314 .It Fn atomic_clear p v
315 .Bd -literal -compact
318 .It Fn atomic_cmpset dst old new
319 .Bd -literal -compact
328 Some architectures do not implement the
330 functions for the types
337 .It Fn atomic_fcmpset dst *old new
340 On architectures implementing
342 operation in hardware, the functionality can be described as
343 .Bd -literal -offset indent -compact
352 On architectures which provide
353 .Em Load Linked/Store Conditional
354 primitive, the write to
356 might also fail for several reasons, most important of which
357 is a parallel write to
359 cache line by other CPU.
362 function also returns
367 Some architectures do not implement the
369 functions for the types
376 .It Fn atomic_fetchadd p v
377 .Bd -literal -compact
386 functions are only implemented for the types
391 and do not have any variants with memory barriers at this time.
394 .Bd -literal -compact
397 .It Fn atomic_readandclear p
398 .Bd -literal -compact
406 .Fn atomic_readandclear
407 functions are not implemented for the types
414 and do not have any variants with memory barriers at this time.
416 .It Fn atomic_set p v
417 .Bd -literal -compact
420 .It Fn atomic_subtract p v
421 .Bd -literal -compact
424 .It Fn atomic_store p v
425 .Bd -literal -compact
428 .It Fn atomic_swap p v
429 .Bd -literal -compact
438 functions are not implemented for the types
445 and do not have any variants with memory barriers at this time.
447 .It Fn atomic_testandclear p v
448 .Bd -literal -compact
449 bit = 1 << (v % (sizeof(*p) * NBBY));
450 tmp = (*p & bit) != 0;
456 .It Fn atomic_testandset p v
457 .Bd -literal -compact
458 bit = 1 << (v % (sizeof(*p) * NBBY));
459 tmp = (*p & bit) != 0;
466 .Fn atomic_testandset
468 .Fn atomic_testandclear
469 functions are only implemented for the types
474 and do not have any variants with memory barriers at this time.
478 is currently not implemented for any of the atomic operations on the
487 function returns the result of the compare operation.
492 if the operation succeeded.
499 .Fn atomic_fetchadd ,
501 .Fn atomic_readandclear ,
504 functions return the value at the specified address.
506 .Fn atomic_testandset
508 .Fn atomic_testandclear
509 function returns the result of the test operation.
511 This example uses the
512 .Fn atomic_cmpset_acq_ptr
515 functions to obtain a sleep mutex and handle recursion.
524 /* Try to obtain mtx_lock once. */
525 #define _obtain_lock(mp, tid) \\
526 atomic_cmpset_acq_ptr(&(mp)->mtx_lock, MTX_UNOWNED, (tid))
528 /* Get a sleep lock, deal with recursion inline. */
529 #define _get_sleep_lock(mp, tid, opts, file, line) do { \\
530 uintptr_t _tid = (uintptr_t)(tid); \\
532 if (!_obtain_lock(mp, tid)) { \\
533 if (((mp)->mtx_lock & MTX_FLAGMASK) != _tid) \\
534 _mtx_lock_sleep((mp), _tid, (opts), (file), (line));\\
536 atomic_set_ptr(&(mp)->mtx_lock, MTX_RECURSE); \\
537 (mp)->mtx_recurse++; \\
549 operations were first introduced in
551 This first set only supported the types
560 .Fn atomic_readandclear ,
563 operations were added in
572 and all of the acquire and release variants
578 operations were added in
583 .Fn atomic_testandset
584 operations were added in
586 .Fn atomic_testandclear
587 operation was added in