1 .\" Copyright (c) 2000-2001 John H. Baldwin <jhb@FreeBSD.org>
3 .\" Redistribution and use in source and binary forms, with or without
4 .\" modification, are permitted provided that the following conditions
6 .\" 1. Redistributions of source code must retain the above copyright
7 .\" notice, this list of conditions and the following disclaimer.
8 .\" 2. Redistributions in binary form must reproduce the above copyright
9 .\" notice, this list of conditions and the following disclaimer in the
10 .\" documentation and/or other materials provided with the distribution.
12 .\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR
13 .\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
14 .\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
15 .\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT,
16 .\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
17 .\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
18 .\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
19 .\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
20 .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
21 .\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
35 .Nm atomic_readandclear ,
39 .Nm atomic_thread_fence
45 .Fn atomic_add_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
47 .Fn atomic_clear_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
49 .Fo atomic_cmpset_[acq_|rel_]<type>
50 .Fa "volatile <type> *dst"
55 .Fo atomic_fcmpset_[acq_|rel_]<type>
56 .Fa "volatile <type> *dst"
61 .Fn atomic_fetchadd_<type> "volatile <type> *p" "<type> v"
63 .Fn atomic_load_[acq_]<type> "volatile <type> *p"
65 .Fn atomic_readandclear_<type> "volatile <type> *p"
67 .Fn atomic_set_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
69 .Fn atomic_subtract_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
71 .Fn atomic_store_[rel_]<type> "volatile <type> *p" "<type> v"
73 .Fn atomic_swap_<type> "volatile <type> *p" "<type> v"
75 .Fn atomic_testandclear_<type> "volatile <type> *p" "u_int v"
77 .Fn atomic_testandset_<type> "volatile <type> *p" "u_int v"
79 .Fn atomic_thread_fence_[acq|acq_rel|rel|seq_cst] "void"
81 Atomic operations are commonly used to implement reference counts and as
82 building blocks for synchronization primitives, such as mutexes.
84 All of these operations are performed
86 across multiple threads and in the presence of interrupts, meaning that they
87 are performed in an indivisible manner from the perspective of concurrently
88 running threads and interrupt handlers.
90 On all architectures supported by
92 ordinary loads and stores of integers in cache-coherent memory are
93 inherently atomic if the integer is naturally aligned and its size does not
94 exceed the processor's word size.
95 However, such loads and stores may be elided from the program by
96 the compiler, whereas atomic operations are always performed.
98 When atomic operations are performed on cache-coherent memory, all
99 operations on the same location are totally ordered.
101 When an atomic load is performed on a location in cache-coherent memory,
102 it reads the entire value that was defined by the last atomic store to
103 each byte of the location.
104 An atomic load will never return a value out of thin air.
105 When an atomic store is performed on a location, no other thread or
106 interrupt handler will observe a
108 or partial modification of the location.
110 Except as noted below, the semantics of these operations are almost
111 identical to the semantics of similarly named C11 atomic operations.
113 Most atomic operations act upon a specific
115 That type is indicated in the function name.
116 In contrast to C11 atomic operations,
118 atomic operations are performed on ordinary integer types.
119 The available types are:
121 .Bl -tag -offset indent -width short -compact
125 unsigned long integer
127 unsigned integer the size of a pointer
129 unsigned 32-bit integer
131 unsigned 64-bit integer
134 For example, the function to atomically add two integers is called
137 Certain architectures also provide operations for types smaller than
140 .Bl -tag -offset indent -width short -compact
144 unsigned short integer
146 unsigned 8-bit integer
148 unsigned 16-bit integer
151 These types must not be used in machine-independent code.
152 .Ss Acquire and Release Operations
153 By default, a thread's accesses to different memory locations might not be
156 that is, the order in which the accesses appear in the source code.
157 To optimize the program's execution, both the compiler and processor might
158 reorder the thread's accesses.
159 However, both ensure that their reordering of the accesses is not visible to
161 Otherwise, the traditional memory model that is expected by single-threaded
162 programs would be violated.
163 Nonetheless, other threads in a multithreaded program, such as the
165 kernel, might observe the reordering.
166 Moreover, in some cases, such as the implementation of synchronization between
167 threads, arbitrary reordering might result in the incorrect execution of the
169 To constrain the reordering that both the compiler and processor might perform
170 on a thread's accesses, a programmer can use atomic operations with
176 Atomic operations on memory have up to three variants.
179 variant, performs the operation without imposing any ordering constraints on
180 accesses to other memory locations.
181 This variant is the default.
182 The second variant has acquire semantics, and the third variant has release
185 When an atomic operation has acquire semantics, the operation must have
186 completed before any subsequent load or store (by program order) is
188 Conversely, acquire semantics do not require that prior loads or stores have
189 completed before the atomic operation is performed.
190 An atomic operation can only have acquire semantics if it performs a load
192 To denote acquire semantics, the suffix
194 is inserted into the function name immediately prior to the
195 .Dq Li _ Ns Aq Fa type
197 For example, to subtract two integers ensuring that the subtraction is
198 completed before any subsequent loads and stores are performed, use
199 .Fn atomic_subtract_acq_int .
201 When an atomic operation has release semantics, all prior loads or stores
202 (by program order) must have completed before the operation is performed.
203 Conversely, release semantics do not require that the atomic operation must
204 have completed before any subsequent load or store is performed.
205 An atomic operation can only have release semantics if it performs a store
207 To denote release semantics, the suffix
209 is inserted into the function name immediately prior to the
210 .Dq Li _ Ns Aq Fa type
212 For example, to add two long integers ensuring that all prior loads and
213 stores are completed before the addition is performed, use
214 .Fn atomic_add_rel_long .
216 When a release operation by one thread
217 .Em synchronizes with
218 an acquire operation by another thread, usually meaning that the acquire
219 operation reads the value written by the release operation, then the effects
220 of all prior stores by the releasing thread must become visible to
221 subsequent loads by the acquiring thread.
222 Moreover, the effects of all stores (by other threads) that were visible to
223 the releasing thread must also become visible to the acquiring thread.
224 These rules only apply to the synchronizing threads.
225 Other threads might observe these stores in a different order.
227 In effect, atomic operations with acquire and release semantics establish
228 one-way barriers to reordering that enable the implementations of
229 synchronization primitives to express their ordering requirements without
230 also imposing unnecessary ordering.
231 For example, for a critical section guarded by a mutex, an acquire operation
232 when the mutex is locked and a release operation when the mutex is unlocked
233 will prevent any loads or stores from moving outside of the critical
235 However, they will not prevent the compiler or processor from moving loads
236 or stores into the critical section, which does not violate the semantics of
238 .Ss Thread Fence Operations
239 Alternatively, a programmer can use atomic thread fence operations to
240 constrain the reordering of accesses.
241 In contrast to other atomic operations, fences do not, themselves, access
244 When a fence has acquire semantics, all prior loads (by program order) must
245 have completed before any subsequent load or store is performed.
246 Thus, an acquire fence is a two-way barrier for load operations.
247 To denote acquire semantics, the suffix
249 is appended to the function name, for example,
250 .Fn atomic_thread_fence_acq .
252 When a fence has release semantics, all prior loads or stores (by program
253 order) must have completed before any subsequent store operation is
255 Thus, a release fence is a two-way barrier for store operations.
256 To denote release semantics, the suffix
258 is appended to the function name, for example,
259 .Fn atomic_thread_fence_rel .
262 .Fn atomic_thread_fence_acq_rel
263 implements both acquire and release semantics, it is not a full barrier.
264 For example, a store prior to the fence (in program order) may be completed
265 after a load subsequent to the fence.
267 .Fn atomic_thread_fence_seq_cst
268 implements a full barrier.
269 Neither loads nor stores may cross this barrier in either direction.
271 In C11, a release fence by one thread synchronizes with an acquire fence by
272 another thread when an atomic load that is prior to the acquire fence (by
273 program order) reads the value written by an atomic store that is subsequent
274 to the release fence.
275 In constrast, in FreeBSD, because of the atomicity of ordinary, naturally
276 aligned loads and stores, fences can also be synchronized by ordinary loads
278 This simplifies the implementation and use of some synchronization
282 Since neither a compiler nor a processor can foresee which (atomic) load
283 will read the value written by an (atomic) store, the ordering constraints
284 imposed by fences must be more restrictive than acquire loads and release
286 Essentially, this is why fences are two-way barriers.
288 Although fences impose more restrictive ordering than acquire loads and
289 release stores, by separating access from ordering, they can sometimes
290 facilitate more efficient implementations of synchronization primitives.
291 For example, they can be used to avoid executing a memory barrier until a
292 memory access shows that some condition is satisfied.
293 .Ss Multiple Processors
294 In multiprocessor systems, the atomicity of the atomic operations on memory
295 depends on support for cache coherence in the underlying architecture.
296 In general, cache coherence on the default memory type,
297 .Dv VM_MEMATTR_DEFAULT ,
298 is guaranteed by all architectures that are supported by
300 For example, cache coherence is guaranteed on write-back memory by the
305 However, on some architectures, cache coherence might not be enabled on all
307 To determine if cache coherence is enabled for a non-default memory type,
308 consult the architecture's documentation.
310 This section describes the semantics of each operation using a C like notation.
312 .It Fn atomic_add p v
313 .Bd -literal -compact
316 .It Fn atomic_clear p v
317 .Bd -literal -compact
320 .It Fn atomic_cmpset dst old new
321 .Bd -literal -compact
330 Some architectures do not implement the
332 functions for the types
339 .It Fn atomic_fcmpset dst *old new
342 On architectures implementing
344 operation in hardware, the functionality can be described as
345 .Bd -literal -offset indent -compact
354 On architectures which provide
355 .Em Load Linked/Store Conditional
356 primitive, the write to
358 might also fail for several reasons, most important of which
359 is a parallel write to
361 cache line by other CPU.
364 function also returns
369 Some architectures do not implement the
371 functions for the types
378 .It Fn atomic_fetchadd p v
379 .Bd -literal -compact
388 functions are only implemented for the types
393 and do not have any variants with memory barriers at this time.
396 .Bd -literal -compact
399 .It Fn atomic_readandclear p
400 .Bd -literal -compact
408 .Fn atomic_readandclear
409 functions are not implemented for the types
416 and do not have any variants with memory barriers at this time.
418 .It Fn atomic_set p v
419 .Bd -literal -compact
422 .It Fn atomic_subtract p v
423 .Bd -literal -compact
426 .It Fn atomic_store p v
427 .Bd -literal -compact
430 .It Fn atomic_swap p v
431 .Bd -literal -compact
440 functions are not implemented for the types
447 and do not have any variants with memory barriers at this time.
449 .It Fn atomic_testandclear p v
450 .Bd -literal -compact
451 bit = 1 << (v % (sizeof(*p) * NBBY));
452 tmp = (*p & bit) != 0;
458 .It Fn atomic_testandset p v
459 .Bd -literal -compact
460 bit = 1 << (v % (sizeof(*p) * NBBY));
461 tmp = (*p & bit) != 0;
468 .Fn atomic_testandset
470 .Fn atomic_testandclear
471 functions are only implemented for the types
476 and do not have any variants with memory barriers at this time.
480 is currently not implemented for any of the atomic operations on the
489 function returns the result of the compare operation.
494 if the operation succeeded.
501 .Fn atomic_fetchadd ,
503 .Fn atomic_readandclear ,
506 functions return the value at the specified address.
508 .Fn atomic_testandset
510 .Fn atomic_testandclear
511 function returns the result of the test operation.
513 This example uses the
514 .Fn atomic_cmpset_acq_ptr
517 functions to obtain a sleep mutex and handle recursion.
526 /* Try to obtain mtx_lock once. */
527 #define _obtain_lock(mp, tid) \\
528 atomic_cmpset_acq_ptr(&(mp)->mtx_lock, MTX_UNOWNED, (tid))
530 /* Get a sleep lock, deal with recursion inline. */
531 #define _get_sleep_lock(mp, tid, opts, file, line) do { \\
532 uintptr_t _tid = (uintptr_t)(tid); \\
534 if (!_obtain_lock(mp, tid)) { \\
535 if (((mp)->mtx_lock & MTX_FLAGMASK) != _tid) \\
536 _mtx_lock_sleep((mp), _tid, (opts), (file), (line));\\
538 atomic_set_ptr(&(mp)->mtx_lock, MTX_RECURSE); \\
539 (mp)->mtx_recurse++; \\
551 operations were introduced in
553 Initially, these operations were defined on the types
562 .Fn atomic_load_acq ,
563 .Fn atomic_readandclear ,
566 operations were added in
568 Simultaneously, the acquire and release variants were introduced, and
569 support was added for operation on the types
579 operation was added in
585 .Fn atomic_testandset
586 operations were added in
590 .Fn atomic_testandclear
592 .Fn atomic_thread_fence
593 operations were added in
596 The relaxed variants of