2 .\" Copyright (c) 2002 Poul-Henning Kamp
3 .\" Copyright (c) 2002 Networks Associates Technology, Inc.
4 .\" All rights reserved.
6 .\" This software was developed for the FreeBSD Project by Poul-Henning Kamp
7 .\" and NAI Labs, the Security Research Division of Network Associates, Inc.
8 .\" under DARPA/SPAWAR contract N66001-01-C-8035 ("CBOSS"), as part of the
9 .\" DARPA CHATS research program.
11 .\" Redistribution and use in source and binary forms, with or without
12 .\" modification, are permitted provided that the following conditions
14 .\" 1. Redistributions of source code must retain the above copyright
15 .\" notice, this list of conditions and the following disclaimer.
16 .\" 2. Redistributions in binary form must reproduce the above copyright
17 .\" notice, this list of conditions and the following disclaimer in the
18 .\" documentation and/or other materials provided with the distribution.
19 .\" 3. The names of the authors may not be used to endorse or promote
20 .\" products derived from this software without specific prior written
23 .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
24 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
25 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
26 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
27 .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
28 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
29 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
30 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
31 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
32 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
42 .Nd "modular disk I/O request transformation framework"
47 .Cd options GEOM_CACHE
48 .Cd options GEOM_CONCAT
52 .Cd options GEOM_JOURNAL
53 .Cd options GEOM_LABEL
54 .Cd options GEOM_LINUX_LVM
57 .Cd options GEOM_MIRROR
58 .Cd options GEOM_MULTIPATH
60 .Cd options GEOM_PART_APM
61 .Cd options GEOM_PART_BSD
62 .Cd options GEOM_PART_BSD64
63 .Cd options GEOM_PART_EBR
64 .Cd options GEOM_PART_EBR_COMPAT
65 .Cd options GEOM_PART_GPT
66 .Cd options GEOM_PART_LDM
67 .Cd options GEOM_PART_MBR
68 .Cd options GEOM_PART_PC98
69 .Cd options GEOM_PART_VTOC8
72 .Cd options GEOM_RAID3
73 .Cd options GEOM_SHSEC
74 .Cd options GEOM_STRIPE
75 .Cd options GEOM_SUNLABEL
76 .Cd options GEOM_UNCOMPRESS
78 .Cd options GEOM_VIRSTOR
84 framework provides an infrastructure in which
86 can perform transformations on disk I/O requests on their path from
87 the upper kernel to the device drivers and back.
91 context range from the simple geometric
92 displacement performed in typical disk partitioning modules over RAID
93 algorithms and device multipath resolution to full blown cryptographic
94 protection of the stored data.
96 Compared to traditional
97 .Dq "volume management" ,
100 and in some cases all previous implementations in the following ways:
105 It is trivially simple to write a new class
106 of transformation and it will not be given stepchild treatment.
108 someone for some reason wanted to mount IBM MVS diskpacks, a class
109 recognizing and configuring their VTOC information would be a trivial
113 is topologically agnostic.
114 Most volume management implementations
115 have very strict notions of how classes can fit together, very often
116 one fixed hierarchy is provided, for instance, subdisk - plex -
120 Being extensible means that new transformations are treated no differently
121 than existing transformations.
123 Fixed hierarchies are bad because they make it impossible to express
124 the intent efficiently.
125 In the fixed hierarchy above, it is not possible to mirror two
126 physical disks and then partition the mirror into subdisks, instead
127 one is forced to make subdisks on the physical volumes and to mirror
128 these two and two, resulting in a much more complex configuration.
130 on the other hand does not care in which order things are done,
131 the only restriction is that cycles in the graph will not be allowed.
132 .Sh "TERMINOLOGY AND TOPOLOGY"
134 is quite object oriented and consequently the terminology
135 borrows a lot of context and semantics from the OO vocabulary:
139 represented by the data structure
142 particular kind of transformation.
143 Typical examples are MBR disk
144 partition, BSD disklabel, and RAID5 classes.
146 An instance of a class is called a
148 and represented by the data structure
153 will be one geom of class MBR for each disk.
157 represented by the data structure
159 is the front gate at which a geom offers service.
162 a disk-like thing which appears in
166 All providers have three main properties:
174 is the backdoor through which a geom connects to another
175 geom provider and through which I/O requests are sent.
177 The topological relationship between these entities are as follows:
180 A class has zero or more geom instances.
182 A geom has exactly one class it is derived from.
184 A geom has zero or more consumers.
186 A geom has zero or more providers.
188 A consumer can be attached to zero or one providers.
190 A provider can have zero or more consumers attached.
193 All geoms have a rank-number assigned, which is used to detect and
194 prevent loops in the acyclic directed graph.
199 A geom with no attached consumers has rank=1.
201 A geom with attached consumers has a rank one higher than the
202 highest rank of the geoms of the providers its consumers are
205 .Sh "SPECIAL TOPOLOGICAL MANEUVERS"
206 In addition to the straightforward attach, which attaches a consumer
207 to a provider, and detach, which breaks the bond, a number of special
208 topological maneuvers exists to facilitate configuration and to
209 improve the overall flexibility.
212 is a process that happens whenever a new class or new provider
213 is created, and it provides the class a chance to automatically configure an
214 instance on providers which it recognizes as its own.
215 A typical example is the MBR disk-partition class which will look for
216 the MBR table in the first sector and, if found and validated, will
217 instantiate a geom to multiplex according to the contents of the MBR.
219 A new class will be offered to all existing providers in turn and a new
220 provider will be offered to all classes in turn.
222 Exactly what a class does to recognize if it should accept the offered
223 provider is not defined by
225 but the sensible set of options are:
228 Examine specific data structures on the disk.
230 Examine properties like
236 Examine the rank number of the provider's geom.
238 Examine the method name of the provider's geom.
241 is the process by which a provider is removed while
242 it potentially is still being used.
244 When a geom orphans a provider, all future I/O requests will
246 on the provider with an error code set by the geom.
248 consumers attached to the provider will receive notification about
249 the orphanization when the event loop gets around to it, and they
250 can take appropriate action at that time.
252 A geom which came into being as a result of a normal taste operation
253 should self-destruct unless it has a way to keep functioning whilst
254 lacking the orphaned provider.
255 Geoms like disk slicers should therefore self-destruct whereas
256 RAID5 or mirror geoms will be able to continue as long as they do
259 When a provider is orphaned, this does not necessarily result in any
260 immediate change in the topology: any attached consumers are still
261 attached, any opened paths are still open, any outstanding I/O
262 requests are still outstanding.
264 The typical scenario is:
266 .Bl -bullet -offset indent -compact
268 A device driver detects a disk has departed and orphans the provider for it.
270 The geoms on top of the disk receive the orphanization event and
271 orphan all their providers in turn.
272 Providers which are not attached to will typically self-destruct
274 This process continues in a quasi-recursive fashion until all
275 relevant pieces of the tree have heard the bad news.
277 Eventually the buck stops when it reaches geom_dev at the top
282 to stop any more requests from
284 It will sleep until any and all outstanding I/O requests have
286 It will explicitly close (i.e.: zero the access counts), a change
287 which will propagate all the way down through the mesh.
288 It will then detach and destroy its geom.
290 The geom whose provider is now detached will destroy the provider,
291 detach and destroy its consumer and destroy its geom.
293 This process percolates all the way down through the mesh, until
294 the cleanup is complete.
297 While this approach seems byzantine, it does provide the maximum
298 flexibility and robustness in handling disappearing devices.
300 The one absolutely crucial detail to be aware of is that if the
301 device driver does not return all I/O requests, the tree will
304 is a special case of orphanization used to protect
305 against stale metadata.
306 It is probably easiest to understand spoiling by going through
311 on top of which an MBR geom provides
321 and that both the MBR and BSD geoms have
322 autoconfigured based on data structures on the disk media.
323 Now imagine the case where
325 is opened for writing and those
326 data structures are modified or overwritten: now the geoms would
327 be operating on stale metadata unless some notification system
328 can inform them otherwise.
330 To avoid this situation, when the open of
333 all attached consumers are told about this and geoms like
334 MBR and BSD will self-destruct as a result.
337 is closed, it will be offered for tasting again
338 and, if the data structures for MBR and BSD are still there, new
339 geoms will instantiate themselves anew.
341 Now for the fine print:
343 If any of the paths through the MBR or BSD module were open, they
344 would have opened downwards with an exclusive bit thus rendering it
347 for writing in that case.
349 the requested exclusive bit would render it impossible to open a
350 path through the MBR geom while
354 From this it also follows that changing the size of open geoms can
355 only be done with their cooperation.
357 Finally: the spoiling only happens when the write count goes from
358 zero to non-zero and the retasting happens only when the write count goes
359 from non-zero to zero.
361 is the process where the administrator issues instructions
362 for a particular class to instantiate itself.
364 ways to express intent in this case - a particular provider may be
365 specified with a level of override forcing, for instance, a BSD
366 disklabel module to attach to a provider which was not found palatable
367 during the TASTE operation.
369 Finally, I/O is the reason we even do this: it concerns itself with
370 sending I/O requests through the graph.
371 .It Em "I/O REQUESTS" ,
374 originate at a consumer,
375 are scheduled on its attached provider and, when processed, are returned
377 It is important to realize that the
379 which enters through the provider of a particular geom does not
381 come out on the other side
383 Even simple transformations like MBR and BSD will clone the
385 modify the clone, and schedule the clone on their
387 Note that cloning the
389 does not involve cloning the
390 actual data area specified in the I/O request.
392 In total, four different I/O requests exist in
394 read, write, delete, and
397 Read and write are self explanatory.
399 Delete indicates that a certain range of data is no longer used
400 and that it can be erased or freed as the underlying technology
402 Technologies like flash adaptation layers can arrange to erase
403 the relevant blocks before they will become reassigned and
404 cryptographic devices may want to fill random bits into the
405 range to reduce the amount of data available for attack.
407 It is important to recognize that a delete indication is not a
408 request and consequently there is no guarantee that the data actually
409 will be erased or made unavailable unless guaranteed by specific
413 semantics are required, a
414 geom should be pushed which converts delete indications into (a
415 sequence of) write requests.
418 supports inspection and manipulation
419 of out-of-band attributes on a particular provider or path.
420 Attributes are named by
422 strings and they will be discussed in
423 a separate section below.
426 (Stay tuned while the author rests his brain and fingers: more to come.)
428 Several flags are provided for tracing
430 operations and unlocking
431 protection mechanisms via the
432 .Va kern.geom.debugflags
434 All of these flags are off by default, and great care should be taken in
436 .Bl -tag -width indent
437 .It 0x01 Pq Dv G_T_TOPOLOGY
438 Provide tracing of topology change events.
439 .It 0x02 Pq Dv G_T_BIO
440 Provide tracing of buffer I/O requests.
441 .It 0x04 Pq Dv G_T_ACCESS
442 Provide tracing of access check controls.
444 .It 0x10 (allow foot shooting)
445 Allow writing to Rank 1 providers.
446 This would, for example, allow the super-user to overwrite the MBR on the root
447 disk or write random sectors elsewhere to a mounted disk.
448 The implications are obvious.
449 .It 0x40 Pq Dv G_F_DISKIOCTL
450 This is unused at this time.
451 .It 0x80 Pq Dv G_F_CTLDUMP
452 Dump contents of gctl requests.
457 .Xr DECLARE_GEOM_CLASS 9 ,
466 .Xr g_provider_by_name 9
468 This software was developed for the
471 .An Poul-Henning Kamp
472 and NAI Labs, the Security Research Division of Network Associates, Inc.\&
473 under DARPA/SPAWAR contract N66001-01-C-8035
476 DARPA CHATS research program.
478 The first precursor for
480 was a gruesome hack to Minix 1.2 and was
482 An earlier attempt to implement a less general scheme
487 .An "Poul-Henning Kamp" Aq phk@FreeBSD.org