1 .\" Copyright (c) 2000-2001 John H. Baldwin <jhb@FreeBSD.org>
2 .\" All rights reserved.
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice, this list of conditions and the following disclaimer.
9 .\" 2. Redistributions in binary form must reproduce the above copyright
10 .\" notice, this list of conditions and the following disclaimer in the
11 .\" documentation and/or other materials provided with the distribution.
13 .\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR
14 .\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
15 .\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
16 .\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT,
17 .\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
18 .\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
19 .\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
20 .\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
21 .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
22 .\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
36 .Nm atomic_readandclear ,
45 .Fn atomic_add_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
47 .Fn atomic_clear_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
49 .Fo atomic_cmpset_[acq_|rel_]<type>
50 .Fa "volatile <type> *dst"
55 .Fo atomic_fcmpset_[acq_|rel_]<type>
56 .Fa "volatile <type> *dst"
61 .Fn atomic_fetchadd_<type> "volatile <type> *p" "<type> v"
63 .Fn atomic_load_acq_<type> "volatile <type> *p"
65 .Fn atomic_readandclear_<type> "volatile <type> *p"
67 .Fn atomic_set_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
69 .Fn atomic_subtract_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
71 .Fn atomic_store_rel_<type> "volatile <type> *p" "<type> v"
73 .Fn atomic_swap_<type> "volatile <type> *p" "<type> v"
75 .Fn atomic_testandclear_<type> "volatile <type> *p" "u_int v"
77 .Fn atomic_testandset_<type> "volatile <type> *p" "u_int v"
79 Each of the atomic operations is guaranteed to be atomic across multiple
80 threads and in the presence of interrupts.
81 They can be used to implement reference counts or as building blocks for more
82 advanced synchronization primitives such as mutexes.
84 Each atomic operation operates on a specific
86 The type to use is indicated in the function name.
87 The available types that can be used are:
89 .Bl -tag -offset indent -width short -compact
95 unsigned integer the size of a pointer
97 unsigned 32-bit integer
99 unsigned 64-bit integer
102 For example, the function to atomically add two integers is called
105 Certain architectures also provide operations for types smaller than
108 .Bl -tag -offset indent -width short -compact
112 unsigned short integer
114 unsigned 8-bit integer
116 unsigned 16-bit integer
119 These must not be used in MI code because the instructions to implement them
120 efficiently might not be available.
121 .Ss Acquire and Release Operations
122 By default, a thread's accesses to different memory locations might not be
125 that is, the order in which the accesses appear in the source code.
126 To optimize the program's execution, both the compiler and processor might
127 reorder the thread's accesses.
128 However, both ensure that their reordering of the accesses is not visible to
130 Otherwise, the traditional memory model that is expected by single-threaded
131 programs would be violated.
132 Nonetheless, other threads in a multithreaded program, such as the
134 kernel, might observe the reordering.
135 Moreover, in some cases, such as the implementation of synchronization between
136 threads, arbitrary reordering might result in the incorrect execution of the
138 To constrain the reordering that both the compiler and processor might perform
139 on a thread's accesses, the thread should use atomic operations with
145 Most of the atomic operations on memory have three variants.
146 The first variant performs the operation without imposing any ordering
147 constraints on memory accesses to other locations.
148 The second variant has acquire semantics, and the third variant has release
150 In effect, operations with acquire and release semantics establish one-way
151 barriers to reordering.
153 When an atomic operation has acquire semantics, the effects of the operation
154 must have completed before any subsequent load or store (by program order) is
156 Conversely, acquire semantics do not require that prior loads or stores have
157 completed before the atomic operation is performed.
158 To denote acquire semantics, the suffix
160 is inserted into the function name immediately prior to the
161 .Dq Li _ Ns Aq Fa type
163 For example, to subtract two integers ensuring that subsequent loads and
164 stores happen after the subtraction is performed, use
165 .Fn atomic_subtract_acq_int .
167 When an atomic operation has release semantics, the effects of all prior
168 loads or stores (by program order) must have completed before the operation
170 Conversely, release semantics do not require that the effects of the
171 atomic operation must have completed before any subsequent load or store is
173 To denote release semantics, the suffix
175 is inserted into the function name immediately prior to the
176 .Dq Li _ Ns Aq Fa type
178 For example, to add two long integers ensuring that all prior loads and
179 stores happen before the addition, use
180 .Fn atomic_add_rel_long .
182 The one-way barriers provided by acquire and release operations allow the
183 implementations of common synchronization primitives to express their
184 ordering requirements without also imposing unnecessary ordering.
185 For example, for a critical section guarded by a mutex, an acquire operation
186 when the mutex is locked and a release operation when the mutex is unlocked
187 will prevent any loads or stores from moving outside of the critical
189 However, they will not prevent the compiler or processor from moving loads
190 or stores into the critical section, which does not violate the semantics of
192 .Ss Multiple Processors
193 In multiprocessor systems, the atomicity of the atomic operations on memory
194 depends on support for cache coherence in the underlying architecture.
195 In general, cache coherence on the default memory type,
196 .Dv VM_MEMATTR_DEFAULT ,
197 is guaranteed by all architectures that are supported by
199 For example, cache coherence is guaranteed on write-back memory by the
204 However, on some architectures, cache coherence might not be enabled on all
206 To determine if cache coherence is enabled for a non-default memory type,
207 consult the architecture's documentation.
209 This section describes the semantics of each operation using a C like notation.
211 .It Fn atomic_add p v
212 .Bd -literal -compact
215 .It Fn atomic_clear p v
216 .Bd -literal -compact
219 .It Fn atomic_cmpset dst old new
220 .Bd -literal -compact
231 functions are not implemented for the types
238 .It Fn atomic_fcmpset dst *old new
241 On architectures implementing
243 operation in hardware, the functionality can be described as
244 .Bd -literal -offset indent -compact
253 On architectures which provide
254 .Em Load Linked/Store Conditional
255 primitive, the write to
257 might also fail for several reasons, most important of which
258 is a parallel write to
260 cache line by other CPU.
263 function also returns
270 functions are not implemented for the types
277 .It Fn atomic_fetchadd p v
278 .Bd -literal -compact
287 functions are only implemented for the types
292 and do not have any variants with memory barriers at this time.
295 .Bd -literal -compact
302 functions are only provided with acquire memory barriers.
304 .It Fn atomic_readandclear p
305 .Bd -literal -compact
313 .Fn atomic_readandclear
314 functions are not implemented for the types
321 and do not have any variants with memory barriers at this time.
323 .It Fn atomic_set p v
324 .Bd -literal -compact
327 .It Fn atomic_subtract p v
328 .Bd -literal -compact
331 .It Fn atomic_store p v
332 .Bd -literal -compact
339 functions are only provided with release memory barriers.
341 .It Fn atomic_swap p v
342 .Bd -literal -compact
351 functions are not implemented for the types
358 and do not have any variants with memory barriers at this time.
360 .It Fn atomic_testandclear p v
361 .Bd -literal -compact
362 bit = 1 << (v % (sizeof(*p) * NBBY));
363 tmp = (*p & bit) != 0;
369 .It Fn atomic_testandset p v
370 .Bd -literal -compact
371 bit = 1 << (v % (sizeof(*p) * NBBY));
372 tmp = (*p & bit) != 0;
379 .Fn atomic_testandset
381 .Fn atomic_testandclear
382 functions are only implemented for the types
387 and do not have any variants with memory barriers at this time.
391 is currently not implemented for any of the atomic operations on the
400 function returns the result of the compare operation.
405 if the operation succeeded.
412 .Fn atomic_fetchadd ,
414 .Fn atomic_readandclear ,
417 functions return the value at the specified address.
419 .Fn atomic_testandset
421 .Fn atomic_testandclear
422 function returns the result of the test operation.
424 This example uses the
425 .Fn atomic_cmpset_acq_ptr
428 functions to obtain a sleep mutex and handle recursion.
437 /* Try to obtain mtx_lock once. */
438 #define _obtain_lock(mp, tid) \\
439 atomic_cmpset_acq_ptr(&(mp)->mtx_lock, MTX_UNOWNED, (tid))
441 /* Get a sleep lock, deal with recursion inline. */
442 #define _get_sleep_lock(mp, tid, opts, file, line) do { \\
443 uintptr_t _tid = (uintptr_t)(tid); \\
445 if (!_obtain_lock(mp, tid)) { \\
446 if (((mp)->mtx_lock & MTX_FLAGMASK) != _tid) \\
447 _mtx_lock_sleep((mp), _tid, (opts), (file), (line));\\
449 atomic_set_ptr(&(mp)->mtx_lock, MTX_RECURSE); \\
450 (mp)->mtx_recurse++; \\
462 operations were first introduced in
464 This first set only supported the types
473 .Fn atomic_readandclear ,
476 operations were added in
485 and all of the acquire and release variants
491 operations were added in
496 .Fn atomic_testandset
497 operations were added in
499 .Fn atomic_testandclear
500 operation was added in