1 .\" Copyright (c) 2000-2001 John H. Baldwin <jhb@FreeBSD.org>
2 .\" All rights reserved.
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice, this list of conditions and the following disclaimer.
9 .\" 2. Redistributions in binary form must reproduce the above copyright
10 .\" notice, this list of conditions and the following disclaimer in the
11 .\" documentation and/or other materials provided with the distribution.
13 .\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR
14 .\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
15 .\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
16 .\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT,
17 .\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
18 .\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
19 .\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
20 .\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
21 .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
22 .\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
36 .Nm atomic_readandclear ,
40 .Nm atomic_thread_fence
46 .Fn atomic_add_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
48 .Fn atomic_clear_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
50 .Fo atomic_cmpset_[acq_|rel_]<type>
51 .Fa "volatile <type> *dst"
56 .Fo atomic_fcmpset_[acq_|rel_]<type>
57 .Fa "volatile <type> *dst"
62 .Fn atomic_fetchadd_<type> "volatile <type> *p" "<type> v"
64 .Fn atomic_load_[acq_]<type> "volatile <type> *p"
66 .Fn atomic_readandclear_<type> "volatile <type> *p"
68 .Fn atomic_set_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
70 .Fn atomic_subtract_[acq_|rel_]<type> "volatile <type> *p" "<type> v"
72 .Fn atomic_store_[rel_]<type> "volatile <type> *p" "<type> v"
74 .Fn atomic_swap_<type> "volatile <type> *p" "<type> v"
76 .Fn atomic_testandclear_<type> "volatile <type> *p" "u_int v"
78 .Fn atomic_testandset_<type> "volatile <type> *p" "u_int v"
80 .Fn atomic_thread_fence_[acq|acq_rel|rel|seq_cst] "void"
82 Atomic operations are commonly used to implement reference counts and as
83 building blocks for synchronization primitives, such as mutexes.
85 All of these operations are performed
87 across multiple threads and in the presence of interrupts, meaning that they
88 are performed in an indivisible manner from the perspective of concurrently
89 running threads and interrupt handlers.
91 On all architectures supported by
93 ordinary loads and stores of integers in cache-coherent memory are
94 inherently atomic if the integer is naturally aligned and its size does not
95 exceed the processor's word size.
96 However, such loads and stores may be elided from the program by
97 the compiler, whereas atomic operations are always performed.
99 When atomic operations are performed on cache-coherent memory, all
100 operations on the same location are totally ordered.
102 When an atomic load is performed on a location in cache-coherent memory,
103 it reads the entire value that was defined by the last atomic store to
104 each byte of the location.
105 An atomic load will never return a value out of thin air.
106 When an atomic store is performed on a location, no other thread or
107 interrupt handler will observe a
109 or partial modification of the location.
111 Except as noted below, the semantics of these operations are almost
112 identical to the semantics of similarly named C11 atomic operations.
114 Most atomic operations act upon a specific
116 That type is indicated in the function name.
117 In contrast to C11 atomic operations,
119 atomic operations are performed on ordinary integer types.
120 The available types are:
122 .Bl -tag -offset indent -width short -compact
126 unsigned long integer
128 unsigned integer the size of a pointer
130 unsigned 32-bit integer
132 unsigned 64-bit integer
135 For example, the function to atomically add two integers is called
138 Certain architectures also provide operations for types smaller than
141 .Bl -tag -offset indent -width short -compact
145 unsigned short integer
147 unsigned 8-bit integer
149 unsigned 16-bit integer
152 These types must not be used in machine-independent code.
153 .Ss Acquire and Release Operations
154 By default, a thread's accesses to different memory locations might not be
157 that is, the order in which the accesses appear in the source code.
158 To optimize the program's execution, both the compiler and processor might
159 reorder the thread's accesses.
160 However, both ensure that their reordering of the accesses is not visible to
162 Otherwise, the traditional memory model that is expected by single-threaded
163 programs would be violated.
164 Nonetheless, other threads in a multithreaded program, such as the
166 kernel, might observe the reordering.
167 Moreover, in some cases, such as the implementation of synchronization between
168 threads, arbitrary reordering might result in the incorrect execution of the
170 To constrain the reordering that both the compiler and processor might perform
171 on a thread's accesses, a programmer can use atomic operations with
177 Atomic operations on memory have up to three variants.
180 variant, performs the operation without imposing any ordering constraints on
181 accesses to other memory locations.
182 This variant is the default.
183 The second variant has acquire semantics, and the third variant has release
186 When an atomic operation has acquire semantics, the operation must have
187 completed before any subsequent load or store (by program order) is
189 Conversely, acquire semantics do not require that prior loads or stores have
190 completed before the atomic operation is performed.
191 An atomic operation can only have acquire semantics if it performs a load
193 To denote acquire semantics, the suffix
195 is inserted into the function name immediately prior to the
196 .Dq Li _ Ns Aq Fa type
198 For example, to subtract two integers ensuring that the subtraction is
199 completed before any subsequent loads and stores are performed, use
200 .Fn atomic_subtract_acq_int .
202 When an atomic operation has release semantics, all prior loads or stores
203 (by program order) must have completed before the operation is performed.
204 Conversely, release semantics do not require that the atomic operation must
205 have completed before any subsequent load or store is performed.
206 An atomic operation can only have release semantics if it performs a store
208 To denote release semantics, the suffix
210 is inserted into the function name immediately prior to the
211 .Dq Li _ Ns Aq Fa type
213 For example, to add two long integers ensuring that all prior loads and
214 stores are completed before the addition is performed, use
215 .Fn atomic_add_rel_long .
217 When a release operation by one thread
218 .Em synchronizes with
219 an acquire operation by another thread, usually meaning that the acquire
220 operation reads the value written by the release operation, then the effects
221 of all prior stores by the releasing thread must become visible to
222 subsequent loads by the acquiring thread.
223 Moreover, the effects of all stores (by other threads) that were visible to
224 the releasing thread must also become visible to the acquiring thread.
225 These rules only apply to the synchronizing threads.
226 Other threads might observe these stores in a different order.
228 In effect, atomic operations with acquire and release semantics establish
229 one-way barriers to reordering that enable the implementations of
230 synchronization primitives to express their ordering requirements without
231 also imposing unnecessary ordering.
232 For example, for a critical section guarded by a mutex, an acquire operation
233 when the mutex is locked and a release operation when the mutex is unlocked
234 will prevent any loads or stores from moving outside of the critical
236 However, they will not prevent the compiler or processor from moving loads
237 or stores into the critical section, which does not violate the semantics of
239 .Ss Thread Fence Operations
240 Alternatively, a programmer can use atomic thread fence operations to
241 constrain the reordering of accesses.
242 In contrast to other atomic operations, fences do not, themselves, access
245 When a fence has acquire semantics, all prior loads (by program order) must
246 have completed before any subsequent load or store is performed.
247 Thus, an acquire fence is a two-way barrier for load operations.
248 To denote acquire semantics, the suffix
250 is appended to the function name, for example,
251 .Fn atomic_thread_fence_acq .
253 When a fence has release semantics, all prior loads or stores (by program
254 order) must have completed before any subsequent store operation is
256 Thus, a release fence is a two-way barrier for store operations.
257 To denote release semantics, the suffix
259 is appended to the function name, for example,
260 .Fn atomic_thread_fence_rel .
263 .Fn atomic_thread_fence_acq_rel
264 implements both acquire and release semantics, it is not a full barrier.
265 For example, a store prior to the fence (in program order) may be completed
266 after a load subsequent to the fence.
268 .Fn atomic_thread_fence_seq_cst
269 implements a full barrier.
270 Neither loads nor stores may cross this barrier in either direction.
272 In C11, a release fence by one thread synchronizes with an acquire fence by
273 another thread when an atomic load that is prior to the acquire fence (by
274 program order) reads the value written by an atomic store that is subsequent
275 to the release fence.
276 In constrast, in FreeBSD, because of the atomicity of ordinary, naturally
277 aligned loads and stores, fences can also be synchronized by ordinary loads
279 This simplifies the implementation and use of some synchronization
283 Since neither a compiler nor a processor can foresee which (atomic) load
284 will read the value written by an (atomic) store, the ordering constraints
285 imposed by fences must be more restrictive than acquire loads and release
287 Essentially, this is why fences are two-way barriers.
289 Although fences impose more restrictive ordering than acquire loads and
290 release stores, by separating access from ordering, they can sometimes
291 facilitate more efficient implementations of synchronization primitives.
292 For example, they can be used to avoid executing a memory barrier until a
293 memory access shows that some condition is satisfied.
294 .Ss Multiple Processors
295 In multiprocessor systems, the atomicity of the atomic operations on memory
296 depends on support for cache coherence in the underlying architecture.
297 In general, cache coherence on the default memory type,
298 .Dv VM_MEMATTR_DEFAULT ,
299 is guaranteed by all architectures that are supported by
301 For example, cache coherence is guaranteed on write-back memory by the
306 However, on some architectures, cache coherence might not be enabled on all
308 To determine if cache coherence is enabled for a non-default memory type,
309 consult the architecture's documentation.
311 This section describes the semantics of each operation using a C like notation.
313 .It Fn atomic_add p v
314 .Bd -literal -compact
317 .It Fn atomic_clear p v
318 .Bd -literal -compact
321 .It Fn atomic_cmpset dst old new
322 .Bd -literal -compact
331 Some architectures do not implement the
333 functions for the types
340 .It Fn atomic_fcmpset dst *old new
343 On architectures implementing
345 operation in hardware, the functionality can be described as
346 .Bd -literal -offset indent -compact
355 On architectures which provide
356 .Em Load Linked/Store Conditional
357 primitive, the write to
359 might also fail for several reasons, most important of which
360 is a parallel write to
362 cache line by other CPU.
365 function also returns
370 Some architectures do not implement the
372 functions for the types
379 .It Fn atomic_fetchadd p v
380 .Bd -literal -compact
389 functions are only implemented for the types
394 and do not have any variants with memory barriers at this time.
397 .Bd -literal -compact
400 .It Fn atomic_readandclear p
401 .Bd -literal -compact
409 .Fn atomic_readandclear
410 functions are not implemented for the types
417 and do not have any variants with memory barriers at this time.
419 .It Fn atomic_set p v
420 .Bd -literal -compact
423 .It Fn atomic_subtract p v
424 .Bd -literal -compact
427 .It Fn atomic_store p v
428 .Bd -literal -compact
431 .It Fn atomic_swap p v
432 .Bd -literal -compact
441 functions are not implemented for the types
448 and do not have any variants with memory barriers at this time.
450 .It Fn atomic_testandclear p v
451 .Bd -literal -compact
452 bit = 1 << (v % (sizeof(*p) * NBBY));
453 tmp = (*p & bit) != 0;
459 .It Fn atomic_testandset p v
460 .Bd -literal -compact
461 bit = 1 << (v % (sizeof(*p) * NBBY));
462 tmp = (*p & bit) != 0;
469 .Fn atomic_testandset
471 .Fn atomic_testandclear
472 functions are only implemented for the types
477 and do not have any variants with memory barriers at this time.
481 is currently not implemented for any of the atomic operations on the
490 function returns the result of the compare operation.
495 if the operation succeeded.
502 .Fn atomic_fetchadd ,
504 .Fn atomic_readandclear ,
507 functions return the value at the specified address.
509 .Fn atomic_testandset
511 .Fn atomic_testandclear
512 function returns the result of the test operation.
514 This example uses the
515 .Fn atomic_cmpset_acq_ptr
518 functions to obtain a sleep mutex and handle recursion.
527 /* Try to obtain mtx_lock once. */
528 #define _obtain_lock(mp, tid) \\
529 atomic_cmpset_acq_ptr(&(mp)->mtx_lock, MTX_UNOWNED, (tid))
531 /* Get a sleep lock, deal with recursion inline. */
532 #define _get_sleep_lock(mp, tid, opts, file, line) do { \\
533 uintptr_t _tid = (uintptr_t)(tid); \\
535 if (!_obtain_lock(mp, tid)) { \\
536 if (((mp)->mtx_lock & MTX_FLAGMASK) != _tid) \\
537 _mtx_lock_sleep((mp), _tid, (opts), (file), (line));\\
539 atomic_set_ptr(&(mp)->mtx_lock, MTX_RECURSE); \\
540 (mp)->mtx_recurse++; \\
552 operations were introduced in
554 Initially, these operations were defined on the types
563 .Fn atomic_load_acq ,
564 .Fn atomic_readandclear ,
567 operations were added in
569 Simultaneously, the acquire and release variants were introduced, and
570 support was added for operation on the types
580 operation was added in
586 .Fn atomic_testandset
587 operations were added in
591 .Fn atomic_testandclear
593 .Fn atomic_thread_fence
594 operations were added in
597 The relaxed variants of