Redundant ethernet bridge with FreeBSD and tinc

FreeBSD comes with the if_bridge(4) driver which interconnects network cards in order to form an Ethernet switch (or historically a multiport bridge, hence the name). if_bridge even supports the Rapid Spanning Tree Protocol (RSTP) which avoids network loops. Thus, you could connect two (or more) ports of a FreeBSD box running if_bridge to another FreeBSD box running if_bridge, too, and won't experience a network storm. That way both boxes (or software switches, if you like) had a redundant connection to each other and could tolerate the loss of one interconnection. But what if one connection should use the Internet? Here comes tinc into play. Tinc is an SSL VPN software which can also transport layer 2 ethernet frames when configured on a tap device. That tap device will then be connected to the bridge. Here's a complete picture:

                    FreeBSD box 1                                                       FreeBSD box 2
           /------------------------------\                                    /------------------------------\
           |                              |                                    |                              |
   lan 1   |                              |     direct ethernet connection     |                              |   lan 2
-----------|----(em2)           (em0)-----|------------------------------------|-----(em0)           (em2)----|-----------
           |      \---(bridge0)---/       |                                    |       \---(bridge0)---/      |
           |             |                |                                    |              |               |
           |           (tap0)             |                                    |            (tap0)            |
           |             |                |                                    |              |               |
           |        (tinc daemon)         |                                    |         (tinc daemon)        |
           |             |                |            (oooooooooo)            |              |               |
           |             \------(em1)-----|-----------(--Internet--)-----------|-----(em1)----/               |
           |                (public ip 1) |            (oooooooooo)            | (public ip 2)                |
           |                              |                                    |                              |
           \------------------------------/                                    \------------------------------/

Instead of em(4) based ethernet cards you can use any supported nic, of course. em1 on box 1 can even be some kind of ppp uplink with a dynamic ip address. Since tap devices present themselves as 10 MBits/sec interfaces, their RSTP costs are 2000000. 100 MBit nics or 1GE cards have lower costs of 200000 and 20000, respectively. Thus, RSTP will prefer the direct connection, and only if that one is down, it'll use the Internet connection (unless that one is down, too). Here are the steps which are needed to build such a scenario on FreeBSD 10.3 as depicted above:

Install tinc on box 1 and 2:
```
# pkg install tinc
```
We will name our tinc connection lebridge and have to create a directory to place its configuration in on box 1 and 2:
```
# mkdir -p /usr/local/etc/tinc/lebridge/hosts
```
On box 1 and 2 create a keypair:
```
# tincd -n lebridge -K
```
On box 2 (tinc server) create the configration file /usr/local/etc/tinc/lebridge/tinc.conf:
```
Name = box2
Device = /dev/tap0
MaxTimeout = 120
Mode = hub
ProcessPriority = high
```
On box 1 (tinc client) create a slightly different configration file /usr/local/etc/tinc/lebridge/tinc.conf:
```
Name = box1
Device = /dev/tap0
MaxTimeout = 120
Mode = hub
ProcessPriority = high
ConnectTo = box2
```
On box 2 (tinc server) create its host configuration file /usr/local/etc/tinc/lebridge/hosts/box2:
```
Address = public ip 2
Compression = 10
```

Still on box 2, append its public key to its host configuration file:

# cat /usr/local/etc/tinc/lebridge/rsa_key.pub >> /usr/local/etc/tinc/lebridge/hosts/box2

On box 1 (tinc client) also create its host configuration file /usr/local/etc/tinc/lebridge/hosts/box1, but omit the address as it'll be assigned a dynamic ip address:
```
Compression = 10
```

Still on box 1, append its public key to its host configuration file:

# cat /usr/local/etc/tinc/lebridge/rsa_key.pub >> /usr/local/etc/tinc/lebridge/hosts/box1

Now, copy /usr/local/etc/tinc/lebridge/hosts/box1 from box 1 to box 2, and /usr/local/etc/tinc/lebridge/hosts/box2 from box2 to box1. If you're editing both files in a text editor within an SSH session, you might copy&paste their contents. Or simply use scp on box1:
```
# scp box2:/usr/local/etc/tinc/lebridge/hosts/box2 /usr/local/etc/tinc/lebridge/hosts/
# scp /usr/local/etc/tinc/lebridge/hosts/box1 box2:/usr/local/etc/tinc/lebridge/hosts/
```
On both box 1 and 2, create a shell script /usr/local/etc/tinc/lebridge/tinc-up. It gets called whenever tinc starts and adds the tap interface to the bridge:
```
#!/bin/sh

ifconfig "$INTERFACE" up
ifconfig bridge0 addm "$INTERFACE" stp "$INTERFACE"
```
Create another shell script /usr/local/etc/tinc/lebridge/tinc-down which gets called right before tinc exits. It will remove the tap interface from the system:
```
#!/bin/sh

ifconfig "$INTERFACE" destroy
```

Make both scripts executable:

# chmod 0755 /usr/local/etc/tinc/lebridge/tinc-up /usr/local/etc/tinc/lebridge/tinc-down

On box 2 (tinc server), add these lines to your /etc/rc.conf. Replace the ip address and netmask for em1 with their actual values as well as the default gateway:

cloned_interfaces="bridge0"
ifconfig_bridge0="maxaddr 16384 timeout 30 addm em0 stp em0 addm em2 edge em2 up"
ifconfig_em0="up"
ifconfig_em2="up"
ifconfig_em1="inet $PUBLIC_IP_2 netmask x.x.x.x"
defaultrouter="$GATEWAY_OF_PUBLIC_IP_2"
tincd_enable="YES"
tincd_cfg="lebridge"
tincd_flags="-d 2 -L"

On box 1 (tinc client), add almost the same to /etc/rc.conf:

cloned_interfaces="bridge0"
ifconfig_bridge0="maxaddr 16384 timeout 30 addm em0 stp em0 addm em2 edge em2 up"
ifconfig_em0="up"
ifconfig_em2="up"
ifconfig_em1="inet $PUBLIC_IP_1 netmask x.x.x.x"
defaultrouter="$GATEWAY_OF_PUBLIC_IP_1"
tincd_enable="YES"
tincd_cfg="lebridge"
tincd_flags="-d 2 -L"

Add this to /etc/sysctl.conf if the kernel should log RSTP topology changes:
```
net.link.bridge.log_stp=1
```

Optionally, you can assign internal ip addresses to both bridges, e.g. /etc/rc.conf on box 2:

ifconfig_bridge0="inet 10.23.42.2 netmask 255.255.255.0 maxaddr 16384 timeout 30 addm em0 stp em0 addm em2 edge em2 up"

And /etc/rc.conf on box 1:

ifconfig_bridge0="inet 10.23.42.1 netmask 255.255.255.0 maxaddr 16384 timeout 30 addm em0 stp em0 addm em2 edge em2 up"

Reboot both boxes. Afterwards, you will have a redundant layer 2 connection between the two. In case of problems, check /var/log/messages for any tinc related log entries, and inspect the bridges' mac tables:
```
# ifconfig bridge0 addr
```

Please note that you should apply appropriate firewall filters on interface em1 on both boxes.