NIC Bonding

NIC bonding (or NIC teaming) is used to aggregate the link to increase the throughput and provide the redundancy. On our Dell servers running Redhat Linux and Centos, we do NIC bonding on the first two embedded NICs: em1 and em2 (read the new Network Interface naming convention).

To enable NIC bonding, first you will need to load the kernel module by creating a file /etc/modprobe.d/bonding.conf

alias bond0 bonding

If you need second bonding interface for your 3rd and 4th NICs, you add one more line alias bond1 bonding

NOTE: On my Redhat RHEL 6.2 box, I got warnings: Loading kernel module for a network device with CAP_SYS_MODULE (deprecated).  Use CAP_NET_ADMIN and alias netdev-bond0 instead While you can safely ignore this warning, I felt it annoying and changed the /etc/modprobe.d/bond.conf to read alias netdev-bond0 bonding and the warning went away.

You then create the bond0 interface configuration file /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE="bond0"
BOOTPROTO="static"
IPADDR="10.141.11.128"
NETMASK="255.255.255.0"
GATEWAY="10.141.11.1"
NETWORK="10.141.11.0"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE=BOND
BONDING_OPTS="mode=6 miimon=100"

The last line is for the bonding options, which, we will cover details later. The IPADDR, NETMASK, GATEWAY and NETWORK should be copied from your existing /etc/sysconfig/network-scripts/ifcfg-em1. You should comment out those two lines in ifcfg-em1 since they are now defined on the bonding interface. In ifcfg-em1 and ifcfg-em2, make sure ONBOOT=yes so that both interfaces can be UP after system boots to make bond0 up. Please also add those lines in both ifcfg-em1 and ifcfg-em2:

MASTER=bond0
SLAVE=yes

so that system knows em1 and em2 are now part of the bonding interface bond0, as slaves.

Now you can reboot or just restart the network /etc/init.d/network restart If all is working fine, the bonding interface should come up and cat /proc/net/bonding/bond0 will show your bonding mode and em1/em2 as two slave interfaces.

Details about the bonding interface parameters/options can be found in Redhat Deployment Guide. In particular, I'd like to mentioned the "mode".

mode=<value>

Allows you to specify the bonding policy. The <value> can be one of:

• balance-rr or 0 — Sets a round-robin policy for fault tolerance and load balancing. Transmissions are received and sent out sequentially on each bonded slave interface beginning with the first one available.
• active-backup or 1 — Sets an active-backup policy for fault tolerance. Transmissions are received and sent out via the first available bonded slave interface. Another bonded slave interface is only used if the active bonded slave interface fails.
• balance-xor or 2 — Sets an XOR (exclusive-or) policy for fault tolerance and load balancing. Using this method, the interface matches up the incoming request's MAC address with the MAC address for one of the slave NICs. Once this link is established, transmissions are sent out sequentially beginning with the first available interface.
• broadcast or 3 — Sets a broadcast policy for fault tolerance. All transmissions are sent on all slave interfaces.
• 802.3ad or 4 — Sets an IEEE 802.3ad dynamic link aggregation policy. Creates aggregation groups that share the same speed and duplex settings. Transmits and receives on all slaves in the active aggregator. Requires a switch that is 802.3ad compliant.
• balance-tlb or 5 — Sets a Transmit Load Balancing (TLB) policy for fault tolerance and load balancing. The outgoing traffic is distributed according to the current load on each slave interface. Incoming traffic is received by the current slave. If the receiving slave fails, another slave takes over the MAC address of the failed slave.
• balance-alb or 6 — Sets an Active Load Balancing (ALB) policy for fault tolerance and load balancing. Includes transmit and receive load balancing for IPV4 traffic. Receive load balancing is achieved through ARP negotiation.

We picked mode 6 because it provides both fault tolerance and load balancing (both transmission and receiving), and it requires no special configuration on the switch port. The later could turn into a big maintenance headache with special port configs everywhere, besides, not all switches support 802.3ad.

One last thing I want to mention is our Cisco switches have spanning tree protocol on, to make sure the network ports that all our servers are connected come up faster, you want to make sure the spanning tree is disabled on those server ports (aka "edge" ports) with command spanning-tree portfast