1 .\" Copyright (c) 2007 Matthew Jacob
2 .\" All rights reserved.
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice, this list of conditions and the following disclaimer.
9 .\" 2. Redistributions in binary form must reproduce the above copyright
10 .\" notice, this list of conditions and the following disclaimer in the
11 .\" documentation and/or other materials provided with the distribution.
13 .\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
14 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
15 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
16 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
17 .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
18 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
19 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
20 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
21 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
22 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
32 .Nd "disk multipath control utility"
100 utility is used for device multipath configuration.
102 The multipath device can be configured using two different methods:
108 method, no metadata are stored on the devices, so the multipath
109 device has to be configured by hand every time it is needed.
110 Additional device paths also won't be detected automatically.
113 method uses on-disk metadata to detect device and all it's paths.
114 Metadata use the last sector of the underlying disk device and
115 include device name and UUID.
116 The UUID guarantees uniqueness in a shared storage environment
117 but is in general too cumbersome to use.
118 The name is what is exported via the device interface.
120 The first argument to
122 indicates an action to be performed:
123 .Bl -tag -width ".Cm destroy"
125 Create multipath device with
127 method without writing any on-disk metadata.
128 It is up to administrator, how to properly identify device paths.
129 Kernel will only check that all given providers have same media and
133 option enables Active/Active mode,
135 option enables Active/Read mode, otherwise Active/Passive mode is used
138 Create multipath device with
141 Label the first given provider with on-disk metadata using the specified
143 The rest of given providers will be retasted to detect these metadata.
144 It reliably protects against specifying unrelated providers.
145 Providers with no matching metadata detected will not be added to the device.
148 option enables Active/Active mode,
150 option enables Active/Read mode, otherwise Active/Passive mode is used
153 Configure the given multipath device.
156 option enables Active/Active mode,
158 option enables Active/Passive mode,
160 option enables Active/Read mode.
162 Add the given provider as a path to the given multipath device.
163 Should normally be used only for devices created with
165 method, unless you know what you are doing (you are sure that it is another
166 device path, but tasting its metadata in regular
168 way is not possible).
170 Remove the given provider as a path from the given multipath device.
171 If the last path removed, the multipath device will be destroyed.
173 Mark specified provider as a path of the specified multipath device as failed.
174 If there are other paths present, new requests will be forwarded there.
176 Mark specified provider as a path of the specified multipath device as
177 operational, allowing it to handle requests.
179 Change the active provider/path to the next available provider in Active/Passive mode.
181 Change the active provider/path to the specified provider in Active/Passive mode.
183 Get the currently active provider(s)/path(s).
185 Destroy the given multipath device clearing metadata.
187 Stop the given multipath device without clearing metadata.
189 Clear metadata on the given provider.
206 variable can be used to control the behavior of the
209 .Bl -tag -width indent
210 .It Va kern.geom.multipath.debug : No 0
214 This can be set to 0 (default) or 1 to disable or enable various
216 .It Va kern.geom.multipath.exclusive : No 1
217 Open underlying providers exclusively, preventing individual paths access.
220 Exit status is 0 on success, and 1 if the command fails.
221 .Sh MULTIPATH ARCHITECTURE
222 This is a multiple path architecture with no device knowledge or
223 presumptions other than size matching built in.
224 Therefore the user must exercise some care
225 in selecting providers that do indeed represent multiple paths to the
226 same underlying disk device.
227 The reason for this is that there are several
228 criteria across multiple underlying transport types that can
230 identity, but in all respects such identity can rarely be considered
233 For example, if you use the World Word Port Name of a Fibre Channel
234 disk object you might believe that two disks that have the same WWPN
235 on different paths (or even disjoint fabrics) might be considered
237 Nearly always this would be a safe assumption, until
238 you realize that a WWPN, like an Ethernet MAC address, is a soft
239 programmable entity, and that a misconfigured Director Class switch
240 could lead you to believe incorrectly that you have found multiple
241 paths to the same device.
242 This is an extreme and theoretical case, but
243 it is possible enough to indicate that the policy for deciding which
244 of multiple pathnames refer to the same device should be left to the
245 system operator who will use tools and knowledge of their own storage
246 subsystem to make the correct configuration selection.
248 There are Active/Passive, Active/Read and Active/Active operation modes
250 In Active/Passive mode only one path has I/O moving on it
251 at any point in time.
252 This I/O continues until an I/O is returned with
253 a generic I/O error or a "Nonexistent Device" error.
254 When this occurs, that path is marked FAIL, the next path
255 in a list is selected as active and the failed I/O reissued.
256 In Active/Active mode all paths not marked FAIL may handle I/O at the same time.
257 Requests are distributed between paths to equalize load.
258 For capable devices it allows to utilize the bandwidth of all paths.
259 In Active/Read mode all paths not marked FAIL may handle reads at the same time,
260 but unlike in Active/Active mode only one path handles write requests at any
262 It allows to closer follow the original write request order if the layer above
263 needs it for data consistency (not waiting for requisite write completion
264 before sending dependent write).
266 When new devices are added to the system the
268 GEOM class is given an opportunity to taste these new devices.
272 on-disk metadata label, the device is either used to create a new
274 GEOM, or added to the list of paths for an existing
278 It is this mechanism that works reasonably with
282 based Fibre Channel disk devices.
283 For these devices, when a device disappears
284 (due to e.g., a cable pull or power failure to a switch), the device is
285 proactively marked as gone and I/O to it failed.
288 failure event just described.
290 When Fibre Channel events inform either
294 host bus adapters that new devices may have arrived (e.g., the arrival
295 of an RSCN event from the Fabric Domain Controller), they can cause
296 a rescan to occur and cause the attachment and configuration of any
297 (now) new devices to occur, causing the taste event described above.
299 This means that this multipath architecture is not a one-shot path
300 failover, but can be considered to be steady state as long as failed
301 paths are repaired (automatically or otherwise).
303 Automatic rescanning is not a requirement.
304 Nor is Fibre Channel.
306 same failover mechanisms work equally well for traditional "Parallel"
307 SCSI but may require manual intervention with
309 to cause the reattachment of repaired device links.
311 The following example shows how to use
313 to find possible multiple path devices and to create a
316 .Bd -literal -offset indent
317 mysys# camcontrol devlist
318 <ECNCTX @WESTVILLE > at scbus0 target 0 lun 0 (da0,pass0)
319 <ECNCTX @WESTVILLE > at scbus0 target 0 lun 1 (da1,pass1)
320 <ECNCTX @WESTVILLE > at scbus1 target 0 lun 0 (da2,pass2)
321 <ECNCTX @WESTVILLE > at scbus1 target 0 lun 1 (da3,pass3)
322 mysys# camcontrol inquiry da0 -S
323 ECNTX0LUN000000SER10ac0d01
324 mysys# camcontrol inquiry da2 -S
325 ECNTX0LUN000000SER10ac0d01
328 Now that you have used the Serial Number to compare two disk paths
329 it is not entirely unreasonable to conclude that these are multiple
330 paths to the same device.
331 However, only the user who is familiar
332 with their storage is qualified to make this judgement.
336 command to label and create a
340 .Bd -literal -offset indent
341 gmultipath label -v FRED /dev/da0 /dev/da2
342 disklabel -Brw /dev/multipath/FRED auto
343 newfs /dev/multipath/FREDa
344 mount /dev/multipath/FREDa /mnt....
347 The resultant console output looks something like:
348 .Bd -literal -offset indent
349 GEOM_MULTIPATH: da0 added to FRED
350 GEOM_MULTIPATH: da0 is now active path in FRED
351 GEOM_MULTIPATH: da2 added to FRED
364 .An Matthew Jacob Aq mjacob@FreeBSD.org
365 .An Alexander Motin Aq mav@FreeBSD.org