1 .\" Copyright (c) 2000-2001 John H. Baldwin <jhb@FreeBSD.org>
3 .\" Redistribution and use in source and binary forms, with or without
4 .\" modification, are permitted provided that the following conditions
6 .\" 1. Redistributions of source code must retain the above copyright
7 .\" notice, this list of conditions and the following disclaimer.
8 .\" 2. Redistributions in binary form must reproduce the above copyright
9 .\" notice, this list of conditions and the following disclaimer in the
10 .\" documentation and/or other materials provided with the distribution.
12 .\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR
13 .\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
14 .\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
15 .\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT,
16 .\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
17 .\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
18 .\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
19 .\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
20 .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
21 .\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
35 .Nm atomic_readandclear ,
39 .Nm atomic_thread_fence
45 .Fn atomic_add_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
47 .Fn atomic_clear_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
49 .Fo atomic_cmpset_[acq_|rel_]<type>
50 .Fa "volatile <type> *dst"
55 .Fo atomic_fcmpset_[acq_|rel_]<type>
56 .Fa "volatile <type> *dst"
61 .Fn atomic_fetchadd_<type> "volatile <type> *p" "<type> v"
63 .Fn atomic_load_[acq_]<type> "volatile <type> *p"
65 .Fn atomic_readandclear_<type> "volatile <type> *p"
67 .Fn atomic_set_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
69 .Fn atomic_subtract_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
71 .Fn atomic_store_[rel_]<type> "volatile <type> *p" "<type> v"
73 .Fn atomic_swap_<type> "volatile <type> *p" "<type> v"
75 .Fn atomic_testandclear_<type> "volatile <type> *p" "u_int v"
77 .Fn atomic_testandset_<type> "volatile <type> *p" "u_int v"
79 .Fn atomic_thread_fence_[acq|acq_rel|rel|seq_cst] "void"
81 Atomic operations are commonly used to implement reference counts and as
82 building blocks for synchronization primitives, such as mutexes.
84 All of these operations are performed
86 across multiple threads and in the presence of interrupts, meaning that they
87 are performed in an indivisible manner from the perspective of concurrently
88 running threads and interrupt handlers.
90 On all architectures supported by
92 ordinary loads and stores of integers in cache-coherent memory are
93 inherently atomic if the integer is naturally aligned and its size does not
94 exceed the processor's word size.
95 However, such loads and stores may be elided from the program by
96 the compiler, whereas atomic operations are always performed.
98 When atomic operations are performed on cache-coherent memory, all
99 operations on the same location are totally ordered.
101 When an atomic load is performed on a location in cache-coherent memory,
102 it reads the entire value that was defined by the last atomic store to
103 each byte of the location.
104 An atomic load will never return a value out of thin air.
105 When an atomic store is performed on a location, no other thread or
106 interrupt handler will observe a
108 or partial modification of the location.
110 Except as noted below, the semantics of these operations are almost
111 identical to the semantics of similarly named C11 atomic operations.
113 Most atomic operations act upon a specific
115 That type is indicated in the function name.
116 In contrast to C11 atomic operations,
118 atomic operations are performed on ordinary integer types.
119 The available types are:
121 .Bl -tag -offset indent -width short -compact
125 unsigned long integer
127 unsigned integer the size of a pointer
129 unsigned 32-bit integer
131 unsigned 64-bit integer
134 For example, the function to atomically add two integers is called
137 Certain architectures also provide operations for types smaller than
140 .Bl -tag -offset indent -width short -compact
144 unsigned short integer
146 unsigned 8-bit integer
148 unsigned 16-bit integer
151 These types must not be used in machine-independent code.
152 .Ss Acquire and Release Operations
153 By default, a thread's accesses to different memory locations might not be
156 that is, the order in which the accesses appear in the source code.
157 To optimize the program's execution, both the compiler and processor might
158 reorder the thread's accesses.
159 However, both ensure that their reordering of the accesses is not visible to
161 Otherwise, the traditional memory model that is expected by single-threaded
162 programs would be violated.
163 Nonetheless, other threads in a multithreaded program, such as the
165 kernel, might observe the reordering.
166 Moreover, in some cases, such as the implementation of synchronization between
167 threads, arbitrary reordering might result in the incorrect execution of the
169 To constrain the reordering that both the compiler and processor might perform
170 on a thread's accesses, a programmer can use atomic operations with
176 Atomic operations on memory have up to three variants.
179 variant, performs the operation without imposing any ordering constraints on
180 accesses to other memory locations.
181 This variant is the default.
182 The second variant has acquire semantics, and the third variant has release
185 When an atomic operation has acquire semantics, the operation must have
186 completed before any subsequent load or store (by program order) is
188 Conversely, acquire semantics do not require that prior loads or stores have
189 completed before the atomic operation is performed.
190 An atomic operation can only have acquire semantics if it performs a load
192 To denote acquire semantics, the suffix
194 is inserted into the function name immediately prior to the
195 .Dq Li _ Ns Aq Fa type
197 For example, to subtract two integers ensuring that the subtraction is
198 completed before any subsequent loads and stores are performed, use
199 .Fn atomic_subtract_acq_int .
201 When an atomic operation has release semantics, all prior loads or stores
202 (by program order) must have completed before the operation is performed.
203 Conversely, release semantics do not require that the atomic operation must
204 have completed before any subsequent load or store is performed.
205 An atomic operation can only have release semantics if it performs a store
207 To denote release semantics, the suffix
209 is inserted into the function name immediately prior to the
210 .Dq Li _ Ns Aq Fa type
212 For example, to add two long integers ensuring that all prior loads and
213 stores are completed before the addition is performed, use
214 .Fn atomic_add_rel_long .
216 When a release operation by one thread
217 .Em synchronizes with
218 an acquire operation by another thread, usually meaning that the acquire
219 operation reads the value written by the release operation, then the effects
220 of all prior stores by the releasing thread must become visible to
221 subsequent loads by the acquiring thread.
222 Moreover, the effects of all stores (by other threads) that were visible to
223 the releasing thread must also become visible to the acquiring thread.
224 These rules only apply to the synchronizing threads.
225 Other threads might observe these stores in a different order.
227 In effect, atomic operations with acquire and release semantics establish
228 one-way barriers to reordering that enable the implementations of
229 synchronization primitives to express their ordering requirements without
230 also imposing unnecessary ordering.
231 For example, for a critical section guarded by a mutex, an acquire operation
232 when the mutex is locked and a release operation when the mutex is unlocked
233 will prevent any loads or stores from moving outside of the critical
235 However, they will not prevent the compiler or processor from moving loads
236 or stores into the critical section, which does not violate the semantics of
238 .Ss Thread Fence Operations
239 Alternatively, a programmer can use atomic thread fence operations to
240 constrain the reordering of accesses.
241 In contrast to other atomic operations, fences do not, themselves, access
244 When a fence has acquire semantics, all prior loads (by program order) must
245 have completed before any subsequent load or store is performed.
246 Thus, an acquire fence is a two-way barrier for load operations.
247 To denote acquire semantics, the suffix
249 is appended to the function name, for example,
250 .Fn atomic_thread_fence_acq .
252 When a fence has release semantics, all prior loads or stores (by program
253 order) must have completed before any subsequent store operation is
255 Thus, a release fence is a two-way barrier for store operations.
256 To denote release semantics, the suffix
258 is appended to the function name, for example,
259 .Fn atomic_thread_fence_rel .
262 .Fn atomic_thread_fence_acq_rel
263 implements both acquire and release semantics, it is not a full barrier.
264 For example, a store prior to the fence (in program order) may be completed
265 after a load subsequent to the fence.
267 .Fn atomic_thread_fence_seq_cst
268 implements a full barrier.
269 Neither loads nor stores may cross this barrier in either direction.
271 In C11, a release fence by one thread synchronizes with an acquire fence by
272 another thread when an atomic load that is prior to the acquire fence (by
273 program order) reads the value written by an atomic store that is subsequent
274 to the release fence.
277 because of the atomicity of ordinary, naturally
278 aligned loads and stores, fences can also be synchronized by ordinary loads
280 This simplifies the implementation and use of some synchronization
284 Since neither a compiler nor a processor can foresee which (atomic) load
285 will read the value written by an (atomic) store, the ordering constraints
286 imposed by fences must be more restrictive than acquire loads and release
288 Essentially, this is why fences are two-way barriers.
290 Although fences impose more restrictive ordering than acquire loads and
291 release stores, by separating access from ordering, they can sometimes
292 facilitate more efficient implementations of synchronization primitives.
293 For example, they can be used to avoid executing a memory barrier until a
294 memory access shows that some condition is satisfied.
295 .Ss Multiple Processors
296 In multiprocessor systems, the atomicity of the atomic operations on memory
297 depends on support for cache coherence in the underlying architecture.
298 In general, cache coherence on the default memory type,
299 .Dv VM_MEMATTR_DEFAULT ,
300 is guaranteed by all architectures that are supported by
302 For example, cache coherence is guaranteed on write-back memory by the
307 However, on some architectures, cache coherence might not be enabled on all
309 To determine if cache coherence is enabled for a non-default memory type,
310 consult the architecture's documentation.
312 This section describes the semantics of each operation using a C like notation.
314 .It Fn atomic_add p v
315 .Bd -literal -compact
318 .It Fn atomic_clear p v
319 .Bd -literal -compact
322 .It Fn atomic_cmpset dst old new
323 .Bd -literal -compact
332 Some architectures do not implement the
334 functions for the types
341 .It Fn atomic_fcmpset dst *old new
344 On architectures implementing
346 operation in hardware, the functionality can be described as
347 .Bd -literal -offset indent -compact
356 On architectures which provide
357 .Em Load Linked/Store Conditional
358 primitive, the write to
360 might also fail for several reasons, most important of which
361 is a parallel write to
363 cache line by other CPU.
366 function also returns
371 Some architectures do not implement the
373 functions for the types
380 .It Fn atomic_fetchadd p v
381 .Bd -literal -compact
390 functions are only implemented for the types
395 and do not have any variants with memory barriers at this time.
398 .Bd -literal -compact
401 .It Fn atomic_readandclear p
402 .Bd -literal -compact
410 .Fn atomic_readandclear
411 functions are not implemented for the types
418 and do not have any variants with memory barriers at this time.
420 .It Fn atomic_set p v
421 .Bd -literal -compact
424 .It Fn atomic_subtract p v
425 .Bd -literal -compact
428 .It Fn atomic_store p v
429 .Bd -literal -compact
432 .It Fn atomic_swap p v
433 .Bd -literal -compact
442 functions are not implemented for the types
449 and do not have any variants with memory barriers at this time.
451 .It Fn atomic_testandclear p v
452 .Bd -literal -compact
453 bit = 1 << (v % (sizeof(*p) * NBBY));
454 tmp = (*p & bit) != 0;
460 .It Fn atomic_testandset p v
461 .Bd -literal -compact
462 bit = 1 << (v % (sizeof(*p) * NBBY));
463 tmp = (*p & bit) != 0;
470 .Fn atomic_testandset
472 .Fn atomic_testandclear
473 functions are only implemented for the types
478 and do not have any variants with memory barriers at this time.
482 is currently not implemented for some of the atomic operations on the
491 function returns the result of the compare operation.
496 if the operation succeeded.
503 .Fn atomic_fetchadd ,
505 .Fn atomic_readandclear ,
508 functions return the value at the specified address.
510 .Fn atomic_testandset
512 .Fn atomic_testandclear
513 function returns the result of the test operation.
515 This example uses the
516 .Fn atomic_cmpset_acq_ptr
519 functions to obtain a sleep mutex and handle recursion.
528 /* Try to obtain mtx_lock once. */
529 #define _obtain_lock(mp, tid) \\
530 atomic_cmpset_acq_ptr(&(mp)->mtx_lock, MTX_UNOWNED, (tid))
532 /* Get a sleep lock, deal with recursion inline. */
533 #define _get_sleep_lock(mp, tid, opts, file, line) do { \\
534 uintptr_t _tid = (uintptr_t)(tid); \\
536 if (!_obtain_lock(mp, tid)) { \\
537 if (((mp)->mtx_lock & MTX_FLAGMASK) != _tid) \\
538 _mtx_lock_sleep((mp), _tid, (opts), (file), (line));\\
540 atomic_set_ptr(&(mp)->mtx_lock, MTX_RECURSE); \\
541 (mp)->mtx_recurse++; \\
553 operations were introduced in
555 Initially, these operations were defined on the types
564 .Fn atomic_load_acq ,
565 .Fn atomic_readandclear ,
568 operations were added in
570 Simultaneously, the acquire and release variants were introduced, and
571 support was added for operation on the types
581 operation was added in
587 .Fn atomic_testandset
588 operations were added in
592 .Fn atomic_testandclear
594 .Fn atomic_thread_fence
595 operations were added in
598 The relaxed variants of