CARP Firewall Failover

CARP on FreeBSD 12 with Pf Firewall Failover for IPv4 and Ipv6 NAT networks

On many networks the edge firewall is the single point of failure for Internet access. When the firewall fails, access in or out of the network is lost. If this is a local system then it is time to drop everything and walk to the server room. If the server is at a remote data center then it is time for a road trip hoping we do not forget to bring anything we might need in the rush to leave. Either way you are going to be flooded with user emails asking, "is the Internet down?". A better configuration is to have two or more firewalls that will fail over to each other within seconds if one of them goes down. We could then take our time getting the secondary firewall back up.

Thanks to the OpenBSD group, we have an option. With Common Address Redundancy Protocol (CARP) and Pfsync, when the primary firewall fails all firewall state information will be inherited by the backup firewall. None of the current TCP stateful connections will drop and users will not notice an interruption in service. In an operational environment this is an absolute necessity.

CARP is the Common Address Redundancy Protocol. It's a free, secure alternative to the "Virtual Router Redundancy Protocol" and the "Hot Standby Router Protocol." CARP was created and is maintained by the OpenBSD project and available on FreeBSD. Properly setup, carp can be a secure insurance policy in case of software malfunction or hardware failure.

These are our notes for building a production quality firewall using FreeBSD 12, pfsync and CARP. Why use FreeBSD instead of OpenBSD? FreeBSD 12 includes the multi processor friendly Pf, meaning the firewall can use multiple cpu cores to create states significantly faster and more efficiently then OpenBSD's current Pf. We also prefer FreeBSD's network code, support for 10gig network cards and the ZFS file system. Note, with FreeBSD 12 the syntax of CARP has changed from OpenBSD and even FreeBSD 9. CARP does not use separate virtual carp interfaces anymore. FreeBSD 12 CARP uses simple interface aliases bound to the native network interfaces. We will go through full examples of both IPv4 and IPv6 CARP aliases.

The ideology behind "CARP firewall fail over" and setup

We are going to setup two(2) firewalls called "fw0" and "fw1". The basic OS installation procedure is explained on our FreeBSD ZFS Root Install guide. Firewall fw0 will be the "MASTER" and it is used exclusively as long as the machine is up, running properly and we have not "demoted" it. Firewall fw1 is the "BACKUP" firewall and used only if the MASTER fw0 firewall goes down. Both firewalls exchange and update Pf firewall state information from each other over the pfsync interface (em0) and are ready to activate if the other firewall becomes unresponsive.

This is a diagram of our current pair of FreeBSD 12 firewalls with ten(10) gigabit network interfaces on the internal network (LAN) and the outside (Internet) using Myricom Myri10GE 10 Gigabit Ethernet optical adapters (mxge0). The pfsync interface is an on-board one(1) gigabit copper network interface, in this case an Intel (em0) chipset. The exact hardware is listed on our FreeBSD Network Tuning and Optimization page. The firewall is easily able to sustain 9.8 Gbit/sec of bandwidth NAT'd through Pf (1.22 gigabytes/sec).

                                       WEB/Internet
                                             |
                       +--------------------/ \------------------+
                       |   carp alias (mxge0) = 88.88.88.100     |
                       |                                         |
          88.88.88.10 mxge0                                    mxge0 88.88.88.20
                       |                                         |
                   +-------+                    10.10.10.20  +-------+
    MASTER         |  fw0  |- em0 ------  pfsync   ---- em0 -|  fw1  |         BACKUP
                   +-------+  10.10.10.10                    +-------+
                       |                                         |
         192.168.2.10 mxge1                                    mxge1 192.168.2.20
                       |                                         |
                       |   carp alias (mxge1) = 192.168.2.100    |
                       +--------------------\ /------------------+
                                             |
                                       Internal LAN 
                                      (192.168.2/24)

Both firewalls have three(3) physical interfaces mxge0, mxge1 and em0 as well as aliased CARP addresses bound to mxge0 and mxge1. Lets look at the MASTER firewall as an example (fw0):

mxge0 : is the external WEB/Internet interface. The public internet ip on the first firewall is (88.88.88.10) and is bound to the real interface. The only purpose for this ip is to send and receive Carpv2 broadcast packets to the second firewall (fw1) on the 88.88.88/24 subnet. This ip will not be accepting any connections for any services.
mxge1 : is the internal LAN interface and is assigned the ip address, 192.168.2.10 . This is also an ip on a real interface and will be used to broadcast Carpv2 packets to the other firewalls on our internal LAN. As an internal interface you may want to allow ssh access in order to administer the box. This way you can work on the firewall and not worry about which machine is the CARP Master. The pf.conf found later on this page will show an example setup allowing ssh into the machine's LAN interface.
em0 : is the pfsync interface. The interface's sole purpose is the exchange Pf firewall state table information between the MASTER (fw0) firewall to the BACKUP (fw1) firewall(s). When the MASTER firewall fails for any reason the BACKUP firewall will have a real time copy of all the current connection states. Note, the link between the two interfaces should be completely secure like a crossover cable, VLAN, IPSEC tunnel or a dedicated switch. If you only have two firewalls and they a physically close to each other then a crossover cable is a good idea. We highly suggest segregating the CARP network so no one can interfere or snoop its traffic as pfsync packets are NOT secure. The bandwidth on this interface is extremely low, around 500 kilobytes per second on a busy firewall. A 100 megabit or a one(1) gigabit interface is more then sufficient.
carp alias : are the ip's which will be answered by the MASTER CARP firewall. If the first firewall fw0 is up and the master then CARP will bind the ip 88.88.88.100 to the external interface and 192.168.2.100 to the internal interface. The CARP aliases are the ips which services are bound to. This way, when a firewall fails, the carp aliased ips will migrate to the BACKUP firewall and traffic will continue to flow. FreeBSD 10 does NOT use pseudo CARP interfaces anymore. CARP ip addresses are simply aliases on a real interface which makes setup a lot easier.

Configuring both FreeBSD 10 firewalls

For our production machines we use the ZFS boot disk install which can be found on our FreeBSD ZFS Root Install guide. For added protection we also use triple copies per hard drive and a pair of drives as boot disk mirrors (raid 1). ZFS is completely optional, but highly recommended. The systems are using the default kernel so you can continue to use the "freebsd-update fetch && freebsd-update install" tools to keep your machine patched and up to date without having to rebuild the kernel.

The following three(3) configurations files need to be modified on both firewalls:

/boot/loader.conf

Both firewalls use the same /boot/loader.conf file. The loader.conf will load the CARP kernel module and the Myri10GE kernel module on boot. If you do not have a Myricom 10gig card then you will not need the mxge driver; it is really just a reminder for us.

#### FreeBSD 10 /boot/loader.conf
##

# load the PF CARP module
carp_load="YES"

# load the Myricom Myri10GE 10 Gigabit Ethernet adapter driver on boot.
# if_mxge_load="YES"

/etc/sysctl.conf

Both firewalls use the same /etc/sysctl.conf configuration file. We are allowing both machines to forward IPv4 and IPv6 packets through both interfaces and enabling CARP preemption. Preempt allows all CARP interfaces to fail over all at once if any one CARP aliased interface fails or goes down. The following settings are the basics needed to get CARP and ip forwarding to work, check out our FreeBSD Tuning and Optimization page for additional FreeBSD 12 network performance tweaks.

#### FreeBSD 12 /etc/sysctl.conf
##

# Ip Forwarding to allow packets to traverse between interfaces and is used for
# firewalls, bridges and routers. When fast IP forwarding is also enabled, IP packets
# are forwarded directly to the appropriate network interface with direct
# processing to completion, which greatly improves the throughput. All packets
# for local IP addresses, non-unicast, or with IP options are handled by the
# normal IP input processing path. All features of the normal (slow) IP
# forwarding path are supported by fast forwarding including firewall (through
# pfil(9) hooks) checking, except ipsec tunnel brokering. The IP fast
# forwarding path does not generate ICMP redirect or source quench messages
# though. Compared to normal IP forwarding, fastforwarding can give a speedup
# of 40 to 60% in packet forwarding performance which is great for interactive
# connections like online games or VOIP where low latency is critical.
net.inet.ip.forwarding=1      # (default 0)
net.inet.ip.fastforwarding=1  # (default 0)
net.inet6.ip6.forwarding=1    # (default 0)

# CARP - failover all carp interfaces as a group. The preempt option means when
# one of the carp enabled physical interfaces goes down, advskew is changed to
# 240 on all carp interfaces. CARP demoted to 240 stops this machine from
# accepting traffic and allows another CARP enabled machine to accept the
# traffic.
net.inet.carp.preempt=1  # (default 0)

/etc/rc.conf

The MASTER and BACKUP firewalls will have their own /etc/rc.conf . The differences are the unique native ip addresses assigned to each interface. Note the "advskew" on the MASTER is zero(0) and on the BACKUP firewall the advskew is 100. Look at the section following the configuration files for details about the CARP aliases. We have included multiple examples of both IPv4 and IPv6 CARP aliases to show the configuration patterns. The syntax ordering is important, make sure not to randomize the directives on each line. The "static_routes" line is included in case you have multiple private subnets on your LAN. You can setup a static route to your internal router.

MASTER (fw0 primary) firewall configuration

#### FreeBSD 12 /etc/rc.conf
##

# zfs_enable="YES"

## PF firewall
pf_enable="YES"                 # Enable PF (load module if required)
pf_rules="/etc/pf.conf"         # rules definition file for pf
pf_flags=""                     # additional flags for pfctl start up
pflog_enable="YES"              # start pflogd(8)
pflog_logfile="/var/log/pflog"  # where pflogd should store the logfile
pflog_flags=""                  # additional flags for pflogd start up

#####################################################################

hostname="fw0"

## ipv4 - native addresses
 ifconfig_mxge0="inet 88.88.88.10/24"
 defaultrouter="88.88.88.1"
 ifconfig_mxge1="inet 192.168.2.10/24"
#static_routes="lan_net"
#route_lan_net="-net 192.168.0.0/16 192.168.2.1"

## ipv6 - native addresses
#ifconfig_mxge0_ipv6="inet6 2001:2d1:1118:100::10/64"
#ipv6_defaultrouter="2001:2d1:1118:100::1"
#ifconfig_mxge1_ipv6="inet6 fddd:0:0:2::10/64"
#ipv6_static_routes="lan_net"
#ipv6_route_lan_net="fddd:: -prefixlen 48 fddd:0:0:2::1"

## CARP aliases - EXTERNAL
 ifconfig_mxge0_alias0="vhid 10 advskew 0 pass 12af7ae5ecb1b497bd4eb7r2ca7ddd2c alias 88.88.88.100/32"
#ifconfig_mxge0_alias1="vhid 11 advskew 0 pass 249bad2afe7d851d9bbede600rcc4804 alias 88.88.88.101/32"
#ifconfig_mxge0_alias2="vhid 12 advskew 0 pass a8c7687rd5abb65689d50a063242897d alias 88.88.88.102/32"
#ifconfig_mxge0_alias3="inet6 2001:2d1:1118:100::100/64 vhid 13 advskew 0 pass 96ee916bb09b9af491d8693fb51f3842"
#ifconfig_mxge0_alias4="inet6 2001:2d1:1118:100::101/64 vhid 14 advskew 0 pass 7e7f2047a63b0330c424773a3a3eb50f"
#ifconfig_mxge0_alias5="inet6 2001:2d1:1118:100::102/64 vhid 15 advskew 0 pass a264fe0aa7210c555f7f6c5a3d8ce0f8"

## CARP aliases - INTERNAL
 ifconfig_mxge1_alias0="vhid 16 advskew 0 pass 5bf2d98cc3ad8c8e74ecae43c8df297f alias 192.168.2.100/32"
#ifconfig_mxge1_alias1="inet6 fddd:0:0:2::100/64 vhid 17 advskew 0 pass 624731c1d3015b202b9a16e6afac73f9"

## pfsync 
 ifconfig_em0="10.10.10.10/24"
 pfsync_enable="YES"
 pfsync_syncdev="em0"

#####################################################################

### EOF ###

BACKUP (fw1 secondary) firewall configuration

#### FreeBSD 12 /etc/rc.conf
##

# zfs_enable="YES"

## PF firewall
pf_enable="YES"                 # Enable PF (load module if required)
pf_rules="/etc/pf.conf"         # rules definition file for pf
pf_flags=""                     # additional flags for pfctl start up
pflog_enable="YES"              # start pflogd(8)
pflog_logfile="/var/log/pflog"  # where pflogd should store the logfile
pflog_flags=""                  # additional flags for pflogd start up

#####################################################################

hostname="fw1"

## ipv4 - native addresses
 ifconfig_mxge0="inet 88.88.88.20/24"
 defaultrouter="88.88.88.1"
 ifconfig_mxge1="inet 192.168.2.20/24"
#static_routes="lan_net"
#route_lan_net="-net 192.168.0.0/16 192.168.2.1"

## ipv6 - native addresses
#ifconfig_mxge0_ipv6="inet6 2001:2d1:1118:100::20/64"
#ipv6_defaultrouter="2001:2d1:1118:100::1"
#ifconfig_mxge1_ipv6="inet6 fddd:0:0:2::20/64"
#ipv6_static_routes="lan_net"
#ipv6_route_lan_net="fddd:: -prefixlen 48 fddd:0:0:2::1"

## CARP aliases - EXTERNAL
 ifconfig_mxge0_alias0="vhid 10 advskew 100 pass 12af7ae5ecb1b497bd4eb7r2ca7ddd2c alias 88.88.88.100/32"
#ifconfig_mxge0_alias1="vhid 11 advskew 100 pass 249bad2afe7d851d9bbede600rcc4804 alias 88.88.88.101/32"
#ifconfig_mxge0_alias2="vhid 12 advskew 100 pass a8c7687rd5abb65689d50a063242897d alias 88.88.88.102/32"
#ifconfig_mxge0_alias3="inet6 2001:2d1:1118:100::100/64 vhid 13 advskew 100 pass 96ee916bb09b9af491d8693fb51f3842"
#ifconfig_mxge0_alias4="inet6 2001:2d1:1118:100::101/64 vhid 14 advskew 100 pass 7e7f2047a63b0330c424773a3a3eb50f"
#ifconfig_mxge0_alias5="inet6 2001:2d1:1118:100::102/64 vhid 15 advskew 100 pass a264fe0aa7210c555f7f6c5a3d8ce0f8"

## CARP aliases - INTERNAL
 ifconfig_mxge1_alias0="vhid 16 advskew 100 pass 5bf2d98cc3ad8c8e74ecae43c8df297f alias 192.168.2.100/32"
#ifconfig_mxge1_alias1="inet6 fddd:0:0:2::100/64 vhid 17 advskew 100 pass 624731c1d3015b202b9a16e6afac73f9"

## pfsync 
 ifconfig_em0="10.10.10.20/24"
 pfsync_enable="YES"
 pfsync_syncdev="em0"

#####################################################################

### EOF ###

Explanation of the FreeBSD Carp aliases

vhid 10 is the Virtual Host ID. This is a unique number which is used to identify the redundancy group to other nodes on the network. Acceptable values are from 1 to 255. The "vhid" must match the same carp interface on both firewalls. Notice each carp alias on fw0 matches the corresponding vhid of the carp alias on fw1.

advskew 0 is an optional parameter specifying how much to skew the advbase when sending CARP advertisements. The default value of advbase is one(1) which equates to sending a CARP advertisement 1/256 of a second. If the advskew is zero(0) then the advbase plus advskew still equals one; CARP advertisements will be sent out one per second. By manipulating advskew, the master CARP host can be chosen or forced. The lower the advskew value, the MORE preferred the host will be when choosing a master. The default is 0. Acceptable values are from 0 to 254. Notice the advskew on the master is 0 and the advskew on the backup is 100. If the BACKUP firewall (fw1) ever becomes the master the CARP advertisements will be sent out once every 1.391 seconds (advbase of 1 plus skew of 100/256 seconds).

pass abcdef... is the authentication password used when talking to other CARP enabled hosts in the same redundancy group. The "pass" string for each CARP alias must be the same value on the master and the slave firewalls. The "pass" strings do not have to be all unique like in our example above, but for security we generated a unique string for each and every CARP alias. CARP uses a cryptographically strong SHA-1 HMAC to protect each advertisement. We suggest a password of 30 characters which is the maximum length allowed.

To assist in making random passwords for carp aliases we suggest the following line. It queries /dev/random and outputs a clean 30 character string which you can cut and paste in place for "pass".

cat /dev/random | hexdump -n 30| cut -d \  -f 2-| head -n 1 | tr -d " "

CARP: general information and notes

Protocols

IP Protocol: CARP uses IP protocol number 112 (0x70).

Multicast Advertisements: CARP advertisements are multicast to the 224.0.0.18 for IPv4 or FF02::12 for IPv6 multicast groups. The watch CARP advertisements use "tcpdump -npi mxge0 -T carp". You will see one CARP advertisement packet per vhid from the MASTER firewall. The BACKUP firewalls will not advertise, only the machine which is the current MASTER.

TTL/Hop Limit: CARP packets are always sent with a TTL/HLIM of 255. This is so CARP packets which have crossed a subnet boundary, for example being passed on by a router, can be recognized and dropped.

Timers

The host that advertises the most frequently will become the master for the CARP group if adskew is the same. The timer values configured on each host are sent as part of the CARP advertisements so that all other hosts can make a determinate decision as to which host will become the master.

Advertisement Interval: This is the base interval at which CARP hellos will be sent. The default is 1 second and is configured with the advbase keyword equal to zero(0).

Advertisement Skew: This value is used to skew the advertisement interval in order to make the host more or less preferred in becoming master. The valid range is 0 to 254, with lower values making the host more preferred to be master. The default is 0 and is configured with the advskew keyword.

The advertisement window is calculated by taking the skew, dividing it by 256 and adding it to the adbase interval. The CARP host will send a hello every "window" seconds.

Failover Timer: If a backup CARP host doesn't see an advertisement from the master for 3 consecutive advertisement windows then it assumes the master is down. It will take over the CARP group and start advertising itself as the master. The number of advertisement windows (3) to delay before assuming the master is down is hard-coded into CARP and is not tunable.

In the event that two or more hosts have the same timer values configured, the following behavior results:

If preempt is disabled: whichever host starts advertising first (i.e., is configured first) will become the master.
If preempt is enabled: whichever host starts advertising last (i.e., is configured last) will become the master.

Want to graph your Pf traffic statistics to see patterns on your network? Check out our guide to setting up RRDtool to graph PF logs. With just a bit of time you can have full color graphs representing all of your traffic going though Pf.

Option 1 of 2: The minimum necessary to setup Pf with Carp

We need to make sure pf knows how and where to send the state information between firewalls. The following lines in /etc/pf.conf are the absolute minimum necessary to get carp working. This will setup em0 as the CarpIf variable sending states to each firewall every second. The two pass lines will allow pf to accept Carpv2 broadcast packets from any Carp host on the network. Note: Only the master firewall at the time sends out broadcast packets. The backup firewalls just listen until they become the master.

Add the following lines or a similar setup to your existing pf.conf on both firewalls. Remember that the pf.conf on the master should be copied to all other firewalls. If the master goes down then the new master must act the same. If you need further assistance with Pf then check our Pf Firewall tutorial" :

## moneyslow.com FreeBSD 10 Carp Example
##    https://moneyslow.com/html/webconf/pf_carp.html
#
################ Macros ###################################
### Interfaces ###
 ExtIf="mxge0"
 IntIf="mxge1"
 PFSync="em0"

################ Filtering #################################
# CARP firewall failover
 pass quick on $PFSync proto pfsync keep state (no-sync)
 pass quick on $ExtIf  proto carp   keep state (no-sync)
 pass quick on $IntIf  proto carp   keep state (no-sync)

Option 2 of 2: A full working pf.conf to setup Pf with Carp

The following is a fully working pf.conf to use as an example. There is a lot in the file so take your time when reading through it. To show an example of the format we setup:

ssh port 22 access to the internal interface of each firewall through mxge1 from LAN ips. Notice we have sshd listening on port 22 and an rdr rule allowing ssh access.
Internet access from LAN clients to ports 80. LAN clients will also be NAT'ed to the carp aliased ip address.
The table BLOCKTEMP was made to house the clients who abuse our webserver by connecting more than 300 times in 120 seconds. Not CARP specific, but it is good practice to set upper limits on your services.

For more information about FreeBSD's Pf firewall and HFSC quality of service options check out our PF Config (pf.conf)

## moneyslow.com FreeBSD 12 Carp Example
##    https://moneyslow.com/html/webconf/pf_carp.html
#
################ FreeBSD 12 pf.conf ##########################
# Required order: options, normalization, queueing, translation, filtering.
# Note: translation rules are first match while filter rules are last match.
################ Macros ###################################

### Interfaces ###
 ExtIf="mxge0"
 IntIf="mxge1"
 PFSync="em0"

### Carp Virtual Interfaces ###
 CARPvhid10 = "88.88.88.100"
#CARPvhid13 ="2001:2d1:1118:100::100/64"
 CARPvhid16 ="192.168.2.100"

### Hosts ###
 IntNet  = "192.168.2/24"
#IntNet6 = "fddd:0:0:2::/64"
 IntWeb  = "192.168.2.30"

### States & Queues ###
 TcpState="flags S/SAFR keep state"
 UdpState="keep state"

### Stateful Tracking Options ###
 SshSTO ="(max 5,   source-track rule, max-src-states 5, max-src-nodes 5,   max-src-conn-rate 5/60)"
 WebSTO ="(max 800, source-track rule, max-src-conn 200, max-src-nodes 500, max-src-conn-rate 300/120, overload <BLOCKTEMP> flush global)"

################ Tables ####################################
 table <BLOCKTEMP> persist

################ Options ##################################
# Misc Options
 set debug urgent
 set require-order yes
 set block-policy drop
 set loginterface $ExtIf
 set state-policy if-bound
 set fingerprints "/etc/pf.os"
 set ruleset-optimization none

################ Normalization #############################
 scrub out log on $ExtIf all min-ttl 15 fragment reassemble set-tos 0x1c
 scrub in  log on $ExtIf all min-ttl 15 fragment reassemble

################ Translation ###############################
# Network Address Translation (NAT)
 no rdr on lo0 from any to any

# NAT ipv4 internal traffic
 nat on $ExtIf from $IntNet to any -> $CARPvhid10 

# NAT ipv6 internal traffic
#nat on $ExtIf from $IntNet6 to any -> $CARPvhid13

# Redirect Internet traffic to our internal web server
 rdr on $ExtIf proto tcp from any to $CARPvhid10 port http -> $IntWeb

# Allow ssh access from the internal lan only
 rdr on $IntIf proto tcp from $IntNet to $IntIf port ssh -> lo0

################ Filtering #################################

# Block blacklisted
 block in quick on $ExtIf from <BLOCKTEMP> to any

# Default BLOCK all in/out on all interfaces and log
 block        in     on $ExtIf
 block return in log on $IntIf

# CARP firewall failover
 pass quick on $PFSync proto pfsync keep state (no-sync)
 pass quick on $ExtIf  proto carp   keep state (no-sync)
 pass quick on $IntIf  proto carp   keep state (no-sync)

# IntIf Inbound ipv4
 pass in log on $IntIf inet proto tcp  from $IntNet to lo0 port ssh $TcpState $SshSTO
 pass in log on $IntIf inet proto icmp from $OBPG   to any icmp-type 8 code 0 $UdpState

# $IntIf Inbound ipv6 
#pass in log on $IntIf inet6 proto ipv6-icmp all icmp6-type {neighbradv, neighbrsol, echoreq} keep state

# IntIf Outbound
 pass out log on $IntIf       proto tcp   $TcpState
 pass out log on $IntIf       proto udp   $UdpState
 pass out log on $IntIf inet  proto icmp  $UdpState
 pass out log on $IntIf inet6 proto icmp6 $UdpState

# ExtIf Inbound ipv4
 pass in log on $ExtIf inet proto tcp  from any to $IntWeb port http $TcpState $WebSTO

# $ExtIf Inbound ipv6
#pass in log on $ExtIf inet6 proto ipv6-icmp all icmp6-type {neighbradv, neighbrsol, echoreq} keep state

# $ExtIf Outbound
 pass out log on $ExtIf       proto tcp   $TcpState
 pass out log on $ExtIf       proto udp   $UdpState
 pass out log on $ExtIf inet  proto icmp  $UdpState
 pass out log on $ExtIf inet6 proto icmp6 $UdpState

################ END #######################################

For more information about the Pf firewall and HFSC quality of service options check out our PF Config (pf.conf) and PF quality of service HFSC "how to's".

Testing the firewalls

If you have not already, you are welcome to reboot the firewalls now. It does not matter which one reboots first or what order they come up in. According to the CARP settings fw0 will always be the master when both firewalls are up and running. Once the firewalls are up make sure all the interfaces are active and pf loads

Look at the interfaces with "ifconfig". You should see three physical interfaces (like mxge0, mxge1 and em0) and the carp aliases. Make sure you also see pflog0 and pfsync0 in the list. If you do not see them then pf was not properly started with the lines in /etc/rc.conf . This is an example from our test box of what you should see on the MASTER firewall. Notice the carp aliases say "carp: MASTER" as this is an ifconfig of fw0.

root@fw0: ifconfig
mxge0: flags=8943 metric 0 mtu 9000
        options=6c03bb
        ether 00:60:dd:45:35:e9
        inet 88.88.88.10 netmask 0xffffff00 broadcast 88.88.88.255 
        inet6 fe80::260:ddff:fe45:35e0%mxge0 prefixlen 64 scopeid 0x1 
        inet6 2001:2d1:1118:100::10 prefixlen 64 
        inet 88.88.88.100 netmask 0xffffffff broadcast 88.88.88.100 vhid 10 
        nd6 options=21
        media: Ethernet 10Gbase-SR 
        status: active
        carp: MASTER vhid 10 advbase 1 advskew 0
mxge1: flags=8943 metric 0 mtu 9000
        options=6c03bb
        ether 00:60:dd:45:35:d3
        inet 192.168.2.10 netmask 0xffffff00 broadcast 192.168.2.255 
        inet6 fe80::260:ddff:fe45:35e1%mxge1 prefixlen 64 scopeid 0x2 
        inet 192.168.2.100 netmask 0xffffffff broadcast 192.168.2.100 vhid 16 
        nd6 options=29
        media: Ethernet 10Gbase-SR 
        status: active
        carp: MASTER vhid 16 advbase 1 advskew 0
em0: flags=8843 metric 0 mtu 1500
        options=4219b
        ether 00:25:90:22:14:d1
        inet 10.10.10.10 netmask 0xffffff00 broadcast 10.10.10.255 
        inet6 fe80::225:90ff:fe22:14c8%em0 prefixlen 64 scopeid 0x3 
        nd6 options=29
        media: Ethernet autoselect (1000baseT )
        status: active
lo0: flags=8049 metric 0 mtu 16384
        options=600003
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 
        inet 127.0.0.1 netmask 0xff000000 
        nd6 options=21
pfsync0: flags=41 metric 0 mtu 1500
        pfsync: syncdev: em0 syncpeer: 224.0.0.240 maxupd: 128 defer: off
pflog0: flags=141 metric 0 mtu 33160

Watching the 'state' of things...

systat -ifstat 1 will allow you to watch the bandwidth on each of the interfaces including pfsync.

Pftop is an excellent program to make sure CARP pfsyncing is working. After installing pftop you can run it on both firewalls and watch the states in real time. You will see that when the state is created on the MASTER firewall, CARP will sync the state to the BACKUP firewall within a second through the pfsync interface (em0). This is how both firewalls are kept in sync and how the BACKUP firewall can take over the duties of the MASTER.

Simulating a firewall failure

To simulate a firewall going down we have a few choices. Lets first look at the current state of the firewalls. To check the status of the firewall use "ifconfig". Look to see if the carp aliases are in one of these three states:

Interface States

INIT : All CARP interfaces start in this state. Also, when a CARP interface is admin down, i.e. "ifconfig mxge0 down", it is put into this state. When a CARP interface is admin up, it immediately transitions to INIT and then to BACKUP.

BACKUP : The host is listening for advertisements from the master. If no advertisements are seen after 3 advertisement windows (adbase=0 is one second per window for example) then assume the master is down, transition to MASTER state and start sending advertisements. If an advertisement is seen with a worse (higher) advertisement window than ours, and if preempt is enabled, transition to MASTER and start sending advertisements.

MASTER : The host is forwarding traffic directed to the CARP alias IP address(es). The host is also sending advertisements once per advertisement window (adbase=0 equals one second) to announce its presence to other CARP hosts within the CARP group. The host still listens for advertisements from other CARP hosts. If an advertisement is seen with a better (lower or equal to ours) advertisement window, transition to BACKUP and allow the other host to become MASTER.

IMPORTANT NOTE: changing any values associated with a CARP interface (timers, password, etc) will automatically result in the interface being put into the INIT state.

Under normal circumstances, there can be multiple hosts within a CARP group in the BACKUP state, but only one host should ever be in MASTER state.

To simulate the MASTER firewall (fw0) going down you can:

Halt or poweroff the box. This leaves only the BACKUP machine and it will change to MASTER. Before the MASTER goes down all CARP alias are demoted.
"ifconfig mxge0 down" to take a physical interface down. This will force CARP to see one of the three physical interfaces are down, thus carp1 is unable to bind to it and the BACKUP will change to MASTER.
Demote the CARP counter with by 240 points "sysctl net.inet.carp.demotion=240" which will make the machine always have a higher advskew then the BACKUP. To promote this machine, "sysctl net.inet.carp.demotion=-240" which will remove 240 points and the machine, if it was the MASTER with an initial advskew=0, will become MASTER again in a few seconds.