Bringing Up a Bonded Interface in Linux Without a Reboot

Originally by apeschel on Jan. 28, 2015, 7:18 p.m.

There was an issue with some servers I was working on recently where two interfaces were meant to be set up with bonding. Unfortunately, due to a misunderstanding, the servers were brought up and put into production without the bonding being set up. When the problem was noticed later, the initial assumption was that fixing the bonding would require a reboot, which meant we would need to schedule downtime for the servers. We had a week scheduled to cycle through the servers, but I was determined to find a way to cut down the time by figuring out how to fix the bonding without a reboot.

After digging around, and asking around on FreeNode, I was not able to find anyone who had come up with a solution to this problem. So I did some experimentation, and came up with a process that only resulted in a couple seconds of downtime for each server while the IP switched from eth0 to bond0.

The initial state of the servers looked like the following.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
    inet 1.1.1.2/24 brd 1.1.1.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether aa:bb:cc:dd:ee:f1 brd ff:ff:ff:ff:ff:ff

Note that the IP address is associated with the eth0 interface, and neither interface is associated with the bond.

The first step is to get the corrected configs for the interfaces in place. These servers were running Ubuntu, which uses /etc/network/interfaces for managing interfaces. After reading the associated kernel documentation, I came up with the following config. Read through the kernel docs and update the config for your distribution according to what you need.

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual
  bond-master bond0
  bond-primary eth0 eth1

auto eth1
iface eth1 inet manual
  bond-master bond0
  bond-primary eth0 eth1

auto bond0
iface bond0 inet static
  bond-slaves none
  bond-mode 802.3ad
  bond-miimon 100
  address 1.1.1.2
  netmask 255.255.255.0
  gateway 1.1.1.2
  bond-xmit_hash_policy layer3+4

The final step is to enact the changes. There is risk involved in doing this via a network connection. I was able to do this via SSH by taking down the interfaces and bringing them back up all in one command. Having a local console open on the machine is a better method, in case something goes wrong.

sudo ifdown eth0 && sudo ifdown eth1 && sudo ifup eth0 && sudo ifup eth1

At this point, the IP address will be associated with both the eth0 and bond0 interfaces, and needs to be cleared from eth0.

sudo ip addr del 1.1.1.2/24 dev eth0

Now everything should be set up correctly.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
    inet 1.1.1.2/24 brd 1.1.1.255 scope global bond0