1 .\" $OpenBSD: carp.4,v 1.16 2004/12/07 23:41:35 jmc Exp $
3 .\" Copyright (c) 2003, Ryan McBride. All rights reserved.
4 .\" Copyright (c) 2011, Gleb Smirnoff <glebius@FreeBSD.org>
6 .\" Redistribution and use in source and binary forms, with or without
7 .\" modification, are permitted provided that the following conditions
9 .\" 1. Redistributions of source code must retain the above copyright
10 .\" notice, this list of conditions and the following disclaimer.
11 .\" 2. Redistributions in binary form must reproduce the above copyright
12 .\" notice, this list of conditions and the following disclaimer in the
13 .\" documentation and/or other materials provided with the distribution.
15 .\" THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND
16 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
17 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
18 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE
19 .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
20 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
21 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
22 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
23 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
24 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
34 .Nd Common Address Redundancy Protocol
38 The CARP allows multiple hosts on the same local network to share a set of
39 IPv4 and/or IPv6 addresses.
40 Its primary purpose is to ensure that these
41 addresses are always available.
45 the administrator needs to configure at a minimum a common virtual host ID
46 (vhid), and attach at least one IP address to this vhid on each machine which
47 is to take part in the virtual group.
48 Additional parameters can also be set on a per-vhid basis:
52 which are used to control how frequently the host sends advertisements when it
53 is the master for a virtual host, and
55 which is used to authenticate
61 .Dq "advertisement base" .
62 It is measured in seconds and specifies the base of the advertisement interval.
66 .Dq "advertisement skew" .
67 It is measured in 1/256 of seconds.
68 It is added to the base advertisement interval to make one host advertise
69 a bit slower that the other does.
74 are put inside CARP advertisements.
75 These values can be configured using
81 CARP defaults to using multicast messages, but can be configured to unicast
82 announcements to peers using the
86 parameters. Default addresses can be restored using
90 Note that TTL verification is disabled if the peer address is not a multicast
92 These values can be configured using
98 CARP virtual hosts can be configured on multicast-capable interfaces: Ethernet,
99 layer 2 VLAN, FDDI and Token Ring.
100 An arbitrary number of virtual host IDs can be configured on an interface.
101 An arbitrary number of IPv4 or IPv6 addresses can be attached to a particular
103 It is important that all hosts participating in a vhid have the same list
104 of prefixes configured on the vhid, since all the prefixes are included in the
105 cryptographic checksum supplied in each advertisement.
106 Multiple vhids running on one interface participate in master/backup
107 elections independently.
109 Additionally, there are a number of global parameters which can be set using
111 .Bl -tag -width ".Va net.inet.carp.ifdown_demotion_factor"
112 .It Va net.inet.carp.allow
116 When disabled, virtual hosts remain in initial state, neither sending nor
117 receiving announcements or traffic.
119 .It Va net.inet.carp.preempt
120 Allow virtual hosts to preempt each other.
121 When enabled, a vhid in a backup state would preempt a master that
122 is announcing itself with a lower advskew.
124 .It Va net.inet.carp.dscp
125 DSCP value in carp packet.
126 Valid Values are 0 to 63.
127 A value of 4 is equivalent to the old standard of TOS LOW_DELAY.
128 TOS values were deprecated and replaced by DSCP in 1998.
129 The default value is 56 (CS7/Network Control).
130 .It Va net.inet.carp.log
131 Determines what events relating to
134 A value of 0 disables any logging.
135 A value of 1 enables logging state changes of
138 Values above 1 enable logging of bad
141 The default value is 1.
142 .It Va net.inet.carp.demotion
143 This value shows the current level of CARP demotion.
144 The value is added to the actual advskew sent in announcements for
146 During normal system operation the demotion factor is zero.
147 However, problematic conditions raise its level: when
149 experiences problem with sending announcements, when an interface
150 running a vhid goes down, or while the
152 interface is not synchronized.
153 The demotion factor can be adjusted writing to the sysctl oid.
154 The signed value supplied to the
156 command is added to current demotion factor.
157 This allows to control
159 behaviour depending on some external conditions, for example on the status
160 of some daemon utility.
161 .It Va net.inet.carp.ifdown_demotion_factor
162 This value is added to
163 .Va net.inet.carp.demotion
164 when an interface running a vhid goes down.
165 The default value is 240 (the maximum advskew value).
166 .It Va net.inet.carp.senderr_demotion_factor
167 This value is added to
168 .Va net.inet.carp.demotion
171 experiences errors sending its announcements.
172 The default value is 240 (the maximum advskew value).
174 .\".Sh ARP level load balancing
177 .\"interface has limited abilities for load balancing incoming connections
178 .\"between hosts in an Ethernet network.
179 .\"For load-balancing operation, one needs several CARP interfaces that
180 .\"are configured to the same IP address, but to a different vhids.
181 .\"Once an ARP request is received, the CARP protocol will use a hashing
182 .\"function against the source IP address in the ARP request to determine
183 .\"which vhid the request will be assigned to.
184 .\"If the corresponding CARP interface is the current
185 .\"master interface, a reply will
186 .\"be sent to the ARP request;
187 .\"otherwise it will be ignored.
190 .\"section for a practical example of load balancing.
192 .\"The ARP load balancing implemented in
194 .\"has some limitations.
195 .\"First, ARP balancing only works on the local network segment.
196 .\"It cannot balance traffic that crosses a router, because the
197 .\"router itself will always be balanced to the same virtual host.
198 .\"Second, ARP load balancing can lead to asymmetric routing
199 .\"of incoming and outgoing traffic, and thus combining it with
201 .\"is dangerous, because this creates a race condition between
202 .\"balanced routers and a host they are serving.
203 .\"Imagine an incoming packet creating state on the first router, being
204 .\"forwarded to its destination, and the destination replying faster
205 .\"than the state information is packed and synced with the second router.
206 .\"If the reply would be load balanced to second router, it will be
207 .\"dropped since the second router has not yet received information about
208 .\"the connection state.
209 .Sh STATE CHANGE NOTIFICATIONS
210 Sometimes it is useful to get notified about
212 status change events.
213 This can be accomplished by using
216 Master/slave events are signalled under system
218 The subsystem specifies the vhid and name of the interface where
219 the master/slave event occurred.
220 The type of the message displays the new state of the vhid.
225 section for more information.
227 For firewalls and routers with multiple interfaces, it is desirable to
228 failover all of the addresses running
230 together, when one of the physical interfaces goes down.
231 This is achieved by the use of the preempt option.
232 Enable it on both hosts A and B:
234 .Dl sysctl net.inet.carp.preempt=1
236 Assume that host A is the preferred master and we are running the
237 192.168.1.0/24 prefix on em0 and 192.168.2.0/24 on em1.
238 This is the setup for host A (advskew is above 0 so it could be overwritten
239 in the emergency situation from the other host):
240 .Bd -literal -offset indent
241 ifconfig em0 vhid 1 advskew 100 pass mekmitasdigoat 192.168.1.1/24
242 ifconfig em1 vhid 2 advskew 100 pass mekmitasdigoat 192.168.2.1/24
245 The setup for host B is identical, but it has a higher
247 .Bd -literal -offset indent
248 ifconfig em0 vhid 1 advskew 200 pass mekmitasdigoat 192.168.1.1/24
249 ifconfig em1 vhid 2 advskew 200 pass mekmitasdigoat 192.168.2.1/24
252 When one of the physical interfaces of host A fails,
254 is demoted to a configured value on all its
257 Due to the preempt option, host B would start announcing itself, and thus
258 preempt host A on both interfaces instead of just the failed one.
260 .\"In order to set up an ARP balanced virtual host, it is necessary to configure
261 .\"one virtual host for each physical host which would respond to ARP requests
262 .\"and thus handle the traffic.
263 .\"In the following example, two virtual hosts are configured on two hosts to
264 .\"provide balancing and failover for the IP address 192.168.1.10.
268 .\"interfaces on host A are configured.
271 .\"of 100 on the second virtual host means that its advertisements will be sent
272 .\"out slightly less frequently.
273 .\".Bd -literal -offset indent
274 .\"ifconfig carp0 create
275 .\"ifconfig carp0 vhid 1 advskew 100 pass mekmitasdigoat 192.168.1.10/24
276 .\"ifconfig carp1 create
277 .\"ifconfig carp1 vhid 2 advskew 200 pass mekmitasdigoat 192.168.1.10/24
280 .\"The configuration for host B is identical, except the
282 .\"is on virtual host 1 rather than virtual host 2.
283 .\".Bd -literal -offset indent
284 .\"ifconfig carp0 create
285 .\"ifconfig carp0 vhid 1 advskew 200 pass mekmitasdigoat 192.168.1.10/24
286 .\"ifconfig carp1 create
287 .\"ifconfig carp1 vhid 2 advskew 100 pass mekmitasdigoat 192.168.1.10/24
290 .\"Finally, the ARP balancing feature must be enabled on both hosts:
292 .\".Dl sysctl net.inet.carp.arpbalance=1
294 .\"When the hosts receive an ARP request for 192.168.1.10, the source IP address
295 .\"of the request is used to compute which virtual host should answer the request.
296 .\"The host which is master of the selected virtual host will reply to the
297 .\"request, the other(s) will ignore it.
299 .\"This way, locally connected systems will receive different ARP replies and
300 .\"subsequent IP traffic will be balanced among the hosts.
301 .\"If one of the hosts fails, the other will take over the virtual MAC address,
302 .\"and begin answering ARP requests on its behalf.
306 status change events can be set up by using the following devd.conf rule:
307 .Bd -literal -offset indent
309 match "system" "CARP";
310 match "subsystem" "[0-9]+@[0-9a-z\.]+";
311 match "type" "(MASTER|BACKUP)";
312 action "/root/carpcontrol.sh $subsystem $type";
320 output, one needs to specify the
324 will interpret them as VRRP packets:
325 .Bd -literal -offset indent
326 tcpdump -npi vlan0 -T carp
339 device first appeared in
343 device was imported into
348 was significantly rewritten, and is no longer a pseudo-interface.