2 .\" Copyright (c) 2017 Robert N. M. Watson
3 .\" All rights reserved.
5 .\" Redistribution and use in source and binary forms, with or without
6 .\" modification, are permitted provided that the following conditions
8 .\" 1. Redistributions of source code must retain the above copyright
9 .\" notice, this list of conditions and the following disclaimer.
10 .\" 2. Redistributions in binary form must reproduce the above copyright
11 .\" notice, this list of conditions and the following disclaimer in the
12 .\" documentation and/or other materials provided with the distribution.
14 .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
15 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
16 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
17 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
18 .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
19 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
20 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
21 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
22 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
23 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
33 .Nd Kernel Dynamic Per-CPU Memory Allocator
36 .Ss Per-CPU Variable Definition and Declaration
37 .Fn DPCPU_DEFINE "type" "name"
38 .Fn DPCPU_DEFINE_STATIC "type" "name"
39 .Fn DPCPU_DECLARE "type" "name"
40 .Ss Current CPU Accessor Functions
43 .Fn DPCPU_SET "name" "value"
44 .Ss Named CPU Accessor Functions
45 .Fn DPCPU_ID_PTR "cpu" "name"
46 .Fn DPCPU_ID_GET "cpu" "name"
47 .Fn DPCPU_ID_SET "cpu" "name" "value"
50 instantiates one instance of a global variable with each CPU in the system.
51 Dynamically allocated per-CPU variables are defined using
53 which defines a variable of name
57 Arbitrary C types may be used, including structures and arrays.
58 If no initialization is provided, then each per-CPU instance of the variable
59 will be zero-filled (i.e., as though allocated in BSS):
60 .Bd -literal -offset 1234
61 DPCPU_DEFINE(int, foo_int);
64 Values may also be initialized statically with the definition, causing each
65 per-CPU instance to be initialized with the value:
66 .Bd -literal -offset 1234
67 DPCPU_DEFINE(int, foo_int) = 1;
70 Values that can be defined as
73 .Fn DPCPU_DEFINE_STATIC :
74 .Bd -literal -offset 1234
75 DPCPU_DEFINE_STATIC(int, foo_int);
79 produces a declaration of the per-CPU variable suitable for use in header
82 The current CPU's variable instance can be accessed via
84 (which returns a pointer to the per-CPU instance),
86 (which retrieves the value of the per-CPU instance),
89 (which sets the value of the per-CPU instance).
91 Instances of variables associated with specific CPUs can be accessed via the
96 accessor functions, which accept an additional CPU ID argument,
99 In addition to the ordinary synchronization concerns associated with global
100 variables, which may imply the use of
103 or other kernel synchronization primitives, it is further the case that
104 thread migration could dynamically change the instance of a variable being
105 accessed by a thread between operations.
106 This requires additional care when reasoning about and protecting per-CPU
109 For example, it may be desirable to protect access using
110 .Xr critical_section 9
111 to prevent both preemption and migration during use.
112 Alternatively, it may be desirable to cache the CPU ID at the start of a
113 sequence of accesses, using suitable synchronization to make non-atomic
114 sequences safe in the presence of migration.
115 .Bd -literal -offset 1234
116 DPCPU_DEFINE_STATIC(int, foo_int);
117 DPCPU_DEFINE_STATIC(struct mutex, foo_lock);
120 foo_int_increment(void)
124 /* Safe as atomic access. */
125 atomic_add_int(DPCPU_PTR(foo_int), 1);
128 * Protect with a critical section, which prevents preemption
129 * and migration. However, access to instances from remote CPUs
130 * is not safe, as critical sections prevent concurrent access
131 * only from the current CPU.
134 value = DPCPU_GET(foo_int);
136 DPCPU_SET(foo_int, value);
140 * Protect with a per-CPU mutex, tolerating migration, but
141 * potentially accessing the variable from multiple CPUs if
142 * migration occurs after reading curcpu. Remote access to a
143 * per-CPU variable is safe as long as the correct mutex is
147 mtx_lock(DPCPU_ID_PTR(cpu, foo_lock));
148 value = DPCPU_ID_GET(cpu, foo_int);
150 DPCPU_ID_SET(cpu, foo_int);
151 mtx_unlock(DPCPU_ID_PTR(cpu, foo_lock));
156 .Xr critical_enter 9 ,
160 was first introduced by
164 This manual page was written by
165 .An Robert N. M. Watson.