2 .\" Copyright (c) 2010 The FreeBSD Foundation
3 .\" All rights reserved.
5 .\" This documentation was written by CK Software GmbH under sponsorship from
6 .\" the FreeBSD Foundation.
8 .\" Redistribution and use in source and binary forms, with or without
9 .\" modification, are permitted provided that the following conditions
11 .\" 1. Redistributions of source code must retain the above copyright
12 .\" notice, this list of conditions and the following disclaimer.
13 .\" 2. Redistributions in binary form must reproduce the above copyright
14 .\" notice, this list of conditions and the following disclaimer in the
15 .\" documentation and/or other materials provided with the distribution.
17 .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
18 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
19 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
20 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
21 .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
22 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
23 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
24 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
25 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
26 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
36 .Nd "network subsystem virtualization infrastructure"
39 .Cd "options VNET_DEBUG"
43 .\"------------------------------------------------------------
44 .Ss "Constants and Global Variables"
50 .Vt extern struct vnet *vnet0;
51 .\"------------------------------------------------------------
52 .Ss "Variable Declaration"
69 .Fo VNET_DEFINE_STATIC
74 #define V_name VNET(name)
76 .\" ------------------------------------------------------------
77 .Ss "Virtual Instance Selection"
103 .Fo CURVNET_SET_QUIET
109 .Fo VNET_ITERATOR_DECL
116 .\" ------------------------------------------------------------
120 .Fn VNET_LIST_RUNLOCK
121 .Fn VNET_LIST_RLOCK_NOSLEEP
122 .Fn VNET_LIST_RUNLOCK_NOSLEEP
123 .\" ------------------------------------------------------------
124 .Ss "Startup and Teardown Functions"
138 .Fa "enum sysinit_sub_id subsystem"
139 .Fa "enum sysinit_elem_order order"
140 .Fa "sysinit_cfunc_t func"
141 .Fa "const void *arg"
146 .Fa "enum sysinit_sub_id subsystem"
147 .Fa "enum sysinit_elem_order order"
148 .Fa "sysinit_cfunc_t func"
149 .Fa "const void *arg"
151 .\" ------------------------------------------------------------
154 .Fo VNET_GLOBAL_EVENTHANDLER_REGISTER
155 .Fa "const char *name"
161 .Fo VNET_GLOBAL_EVENTHANDLER_REGISTER_TAG
162 .Fa "eventhandler_tag tag"
163 .Fa "const char *name"
168 .\" ------------------------------------------------------------
169 .Ss "Sysctl Handling"
171 .Fa parent nbr name access ptr val descr
174 .Fa parent nbr name access ptr arg handler fmt descr
176 .Fo SYSCTL_VNET_STRING
177 .Fa parent nbr name access arg len descr
179 .Fo SYSCTL_VNET_STRUCT
180 .Fa parent nbr name access ptr type descr
183 .Fa parent nbr name access ptr val descr
188 .\" ------------------------------------------------------------
191 is the name of a technique to virtualize the network stack.
192 The basic idea is to change global resources most notably variables into
193 per network stack resources and have functions, sysctls, eventhandlers,
194 etc. access and handle them in the context of the correct instance.
195 Each (virtual) network stack is attached to a
199 being the unrestricted default network stack of the base system.
201 The global defines for
207 to access internals for debugging reasons.
208 .\" ------------------------------------------------------------
209 .Ss "Variable Declaration"
211 Variables are virtualized by using the
213 macro rather than writing them out as
215 One can still use static initialization, e.g.,
217 .Dl Li VNET_DEFINE(int, foo) = 1;
219 Variables declared with the static keyword can use the
220 .Fn VNET_DEFINE_STATIC
223 .Dl Li VNET_DEFINE_STATIC(SLIST_HEAD(, bar), bars);
225 Static initialization is not possible when the virtualized variable
226 would need to be referenced, e.g., with
227 .Dq TAILQ_HEAD_INITIALIZER() .
230 based initialization function must be used.
232 External variables have to be declared using the
235 In either case the convention is to define another macro,
236 that is then used throughout the implementation to access that variable.
237 The variable name is usually prefixed by
239 to express that it is virtualized.
242 macro will then translate accesses to that variable to the copy of the
243 currently selected instance (see the
244 .Sx "Virtual instance selection"
247 .Dl Li #define V_name VNET(name)
250 Do not confuse this with the convention used by
255 macro returns the offset within the memory region of the virtual network
257 It is usually only used with
260 .\" ------------------------------------------------------------
261 .Ss "Virtual Instance Selection"
263 There are three different places where the current virtual
264 network stack pointer is stored and can be taken from:
265 .Bl -enum -offset indent
269 .Dl "(struct prison *)->pr_vnet"
271 For convenience the following macros are provided:
272 .Bd -literal -compact -offset indent
273 .Fn CRED_TO_VNET "struct ucred *"
274 .Fn TD_TO_VNET "struct thread *"
275 .Fn P_TO_VNET "struct proc *"
280 .Dl "(struct socket *)->so_vnet"
284 .Dl "(struct ifnet *)->if_vnet"
288 In addition the currently active instance is cached in
289 .Dq "curthread->td_vnet"
290 which is usually only accessed through the
295 To set the correct context of the current virtual network instance, use the
298 .Fn CURVNET_SET_QUIET
301 .Fn CURVNET_SET_QUIET
302 version will not record vnet recursions in case the kernel was compiled
304 .Cd "options VNET_DEBUG"
305 and should thus only be used in well known cases, where recursion is
307 Both macros will save the previous state on the stack and it must be restored
313 As the previous state is saved on the stack, you cannot have multiple
315 calls in the same block.
318 As the previous state is saved on the stack, a
320 call has to be in the same block as the
322 call or in a subblock with the same idea of the saved instances as the
326 As each macro is a set of operations and, as previously explained, cannot
327 be put into its own block when defined, one cannot conditionally set
328 the current vnet context.
332 .Bd -literal -offset indent
338 .Bd -literal -offset indent
346 Sometimes one needs to loop over all virtual instances, for example to update
347 virtual from global state, to run a function from a
349 for each instance, etc.
351 .Fn VNET_ITERATOR_DECL
355 The former macro defines the variable that iterates over the loop,
356 and the latter loops over all of the virtual network stack instances.
359 for how to savely traverse the list of all virtual instances.
364 macro provides a safe way to check whether the currently active instance is the
365 unrestricted default network stack of the base system
371 macro provides a way to conditionally add assertions that are only active with
373 compiled in and either
374 .Cd "options VNET_DEBUG"
376 .Cd "options INVARIANTS"
378 It uses the same semantics as
380 .\" ------------------------------------------------------------
383 For public access to the list of virtual network stack instances
386 macro, read locks are provided.
387 Macros are used to abstract from the actual type of the locks.
388 If a caller may sleep while traversing the list, it must use the
391 .Fn VNET_LIST_RUNLOCK
393 Otherwise, the caller can use
394 .Fn VNET_LIST_RLOCK_NOSLEEP
396 .Fn VNET_LIST_RUNLOCK_NOSLEEP .
397 .\" ------------------------------------------------------------
398 .Ss "Startup and Teardown Functions"
400 To start or tear down a virtual network stack instance the internal
405 are provided and called from the jail framework.
406 They run the publicly provided methods to handle network stack
407 startup and teardown.
409 For public control, the system startup interface has been enhanced
410 to not only handle a system boot but to also handle a virtual
411 network stack startup and teardown.
412 To the base system the
416 macros look exactly as if there were no virtual network stack.
419 is not compiled in they are compiled to the standard
422 In addition to that they are run for each virtual network stack
423 when starting or, in reverse order, when shutting down.
424 .\" ------------------------------------------------------------
427 Eventhandlers can be handled in two ways:
429 .Bl -enum -offset indent -compact
433 returned in each virtual instance and properly free the eventhandlers
434 on teardown using those, or
436 use one eventhandler that will iterate over all virtual network
440 For the first case one can just use the normal
442 functions, while for the second case the
443 .Fn VNET_GLOBAL_EVENTHANDLER_REGISTER
445 .Fn VNET_GLOBAL_EVENTHANDLER_REGISTER_TAG
448 .Fn VNET_GLOBAL_EVENTHANDLER_REGISTER_TAG
449 takes an extra first argument that will carry the
452 Eventhandlers registered with either of these will not run
456 will be called from an internal iterator function for each vnet.
457 Both macros can only be used for eventhandlers that do not take
458 additional arguments, as the variadic arguments from an
459 .Xr EVENTHANDLER_INVOKE 9
460 call will be ignored.
461 .\" ------------------------------------------------------------
462 .Ss "Sysctl Handling"
466 can be virtualized by using one of the
470 They take the same arguments as the standard
472 functions, with the only difference, that the
474 argument has to be passed as
478 so that the variable can be selected from the correct memory
479 region of the virtual network stack instance of the caller.
481 For the very rare case a sysctl handler function would want to
485 .Fn VNET_SYSCTL_ARG req arg1
486 is provided that will translate the
488 argument to the correct memory address in the virtual network stack
489 context of the caller.
490 .\" ------------------------------------------------------------
500 Marko Zec, Implementing a Clonable Network Stack in the FreeBSD Kernel,
501 USENIX ATC'03, June 2003, Boston
503 The virtual network stack implementation first appeared in
509 framework was designed and implemented at the University of Zagreb by
511 under sponsorship of the FreeBSD Foundation and NLnet Foundation,
512 and later extended and refined by
514 (also under FreeBSD Foundation sponsorship), and
517 This manual page was written by
518 .An Bjoern A. Zeeb, CK Software GmbH,
519 under sponsorship from the FreeBSD Foundation.