Linux Admin Reference – Network Bonding – Redhat Enterprise Linux ( RHEL5 / RHEL6)

 

Bonding is a feature that can be enabled bu the Linux kernel, to allow system administrators to combine two or more network interface to form a single, logical “bonded” interface, for redundancy or increased throughput purpose.

We can configure Linux bonding in 7 different modes. The behavior of the bonded interfaces depends upon the mode.

Below are the available bonding modes

  • Balance-rr (mode 0)
  • Active-backup (mode 1)
  • Balance-xor (mode 2)
  • Broadcast (mode 3)
  • 802.3ad (mode 4)
  • Balance-tlb (mode 5)
  • Balance-alb (mode 6)

 

Configuring a bonding Channel between eth0 and eth1 using the default mode i.e. Mode-0 (balance-rr)

1. Network manager should not be running on the system, as NIM doesn’t support bonding. So please stop and disable the networkmanager service.

 

# service NetworkManager stop 
# chkconfig NetworkManager off
 

2. Bonding module should be configured and loaded. To enable bonding module, create a bonding.conf file as below.

Note : you can add multiple entries to create multiple bonding channels like bond0, bond1, bond2…etc

 

# cat > /etc/modprobe.d/bonding.conf

alias bond0 bonding

options bond0 mode=1 miimon=100 

3. Create a configuration file for the bonding channel i.e. bond0, as below

# cat /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
IPADDR=192.168.50.111
NETMASK=255.255.255.0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
BONDING_OPTS=”mode=0 miimon=100″

4. Create configuration files for all the interfaces  i.e. eth0 and eth1, which are participating in the bonding channel

# cat > /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=none
HWADDR=aa:bb:cc:dd:ee:ff
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

# cat > /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
BOOTPROTO=none
HWADDR=yy:xx:zz:aa:bb:cc
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

 

5. Activating Bonding Channel 

a. With out rebooting

# modprobe bonding

# service network restart

b. with reboot

# reboot

 

6. Verify that Bonding channel is active, by using the commands

 

# cat /proc/net/bonding/bond0 
Ethernet Channel Bonding Driver: v3.0.2 (Nov 25, 2012) 

Bonding Mode: adaptive load balancing 
Primary Slave: None 
Currently Active Slave: eth0 
MII Status: up 
MII Polling Interval (ms): 100 
Up Delay (ms): 0 
Down Delay (ms): 0 

Slave Interface: eth0
MII Status: up 
Link Failure Count: 0 
Permanent HW addr: aa:bb:cc:dd:ee:ff

Configurable Bonding Parameters

max_bonds : specifies the number of bonding devices that can be configured on single bonding driver.

xmit_hash_policy : Slecect the transmit has policy to use for slave selection in balance-xor and 802.3d modevice. Possible values , layer2 (default) , larer2+3 or layer3+4

arp_interval : Specifies the ARP link monitoring frequency in milli seconds

arp_ip_target : Specifies the IP addresses to use as ARP monitoring peers when arp_interval is > 0. Multiple IP addresses must be seperated by comma. At least one IP addres must be given for ARP monitoring to work. Max. no. of target IPs are 16.

 

arp_validate : Specifies Whether or not ARP probes and replies should be validated in the active-backup mode. This causes the ARP monitor to examine the incoming ARP requests and replies, and only consider a slave to be up if it is receiving the appropriate ARP traffic. Possible values  

  • none(0) – default value
  • active(1) – validate only the active slave
  • backup(2) – validate only the backup slave
  • all(3) – validate all slaves

miimon : Specifies the MII link monitoring frequency in milliseconds. A value of 0 disables the MII link monitroing. A value of 100 is recommended.

updelay : specifies the time, in milliseconds, to wait before disabling a slave after a link recovery has been detected.

downdelay : specifies the time, in milliseconds, to wait before disabling a slava after a link failure has been detected.

use_carrier : specifies whether or not miimon shoudl use MII/ETHTOO “ioctls for value of 0 ” or ” netif_carrier_ok() for value 1 ( default value)” functions to determine the link status.


F AQ to Troubleshoot Network Bonding

>>>> 1. /proc/net/bond0/bonding display the network interface card’s link is up when the cable is unplugged within a bonding environment. why ?

Bonding driver uses “use_carrier”  with default value of 1 ( i.e. to use netif_carrier_ok() function), to identify the link status. Just incase if the bonding driver doesn’t support netif_carrier_ok() function, it cannot determine the link status properly. As a workaround we can set use_carrier to value 0 ( i.e. to use  MII/ETHTOOL ioctl method) to determine the link status, as mentioned below

# cat /etc/modprobe.conf
alias bond0 bonding
options bonding mode=1 miimon=100 use_carrier=0
 
if you are using mode-1 , we can also set the arp_interval and arp_ip_target to determine the link status as given below
 
# cat /etc/modprobe.conf
alias bond0 bonding
options bonding mode=1 miimon=100 arp_interval=200 arp_ip_target=192.168.0.1,192.168.0.2
 
 

>>>> 2. Although the Bonding configured in balance-rr / mode-0 ( round-robin policy), outgoing network traffic doesn’t appear to go through all interfaces (RHEL5)

Possible cause could be that Bonded interfaces is predominantly communicating with a low number of IP address (for example, only NAS NFS traffic is used on the interface). To make sure that traffic was getting distributed to all available slave interfaces change the xmit_hash_policy value as given below:
 
Option 1:  modify the /etc/modprobe.conf to add the xmit_hash_policy value
 
#vi /etc/modprobe.conf
:::::: output truncated::::
options bond0 mode=0 miimon=100 xmit_hash_policy=1
:::::: output truncated::::
 
Option 2:  Modify /etc/sysconfig/network-scripts/ifcfg-bondX and add/modify the following entry
#vi /etc/sysconfig/network-scripts/ifcfg-bondX
BONDING_OPTS=”mode=0 miimon=100 xmit_hash_policy=1″
 
Then restart the bonding interface using 
 
# ifdown bondX
# ifup bondX
 

For more reference to use there values you can check, in your linux box :  /usr/share/doc/kernel-doc-2.6.18/Documentation/networking/bonding.txt 

 

>>>> 3.  How to Gracefully Break the Bonded Network Interface?

In Sample Configuration is we have bond0 configure for two interface name eth0 and eht1, and bond running configuration looks as 

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.4.0 Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 53:44:00:43:91:04

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 52:33:00:6d:0e:70

 
The process to break the bonded interface is :
 
a.  Online remove one slave network interface which will be assigned with a new IP address from the bonding device. In this example, we are going to remove the eth1 from bond0. Execute the following command as root:
# ifenslave -d bond0 eth1
Or
# echo -eth1 > /sys/class/net/bond0/bonding/slaves
After issue the command above , You will see the eth1 device has been remove from bond0# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.4.0 

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr:  53:44:00:43:91:04

b. Create a new network configuration for eth1 via graphic tool or text tool and then using following command to start eth1 again. Make sure the physical link has been changed correctly to match the new network configuration of eth1.
 
# ifup eth1
During the above 2 steps, the network of bond0 will work fine without any break up.
 
c. Create the same network configuration as bond0 for eth0 
 
 — just copy the content of /etc/sysconfig/network-scripts/ifcfg-bondX to  /etc/sysconfig/network-scripts/ifcfg-eth0, except the interface name and HW MAC address.
 
d. Remove the eth0 from bond0, destory the bond0 device and start eth0 immediately.
 
Please note this disruptive operation, The network of bond0/eth0 will break shortly during this step.# ifenslave -d bond0 eth0; ifdown bond0; ifup eth0

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.4.0 

Bonding Mode: load balancing (round-robin)
MII Status: down
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

now ifconfig will show the configuration for only eth0 and eth1, no bonding interface appears.

e. Remove all bonding setting of bond0 in /etc/modprobe.conf and remove the ifcfg-bond0 in /etc/sysconfig/network-script directory. Then issue following command to remove the bond0 device from memory completely.
 
# echo -bond0 > /sys/class/net/bonding_masters
 

>>>> 4.  The interface eth0 configured with the multiple virtual interfaces like eth0:1 , eht0:2  and eth0:3.  When we disable the virtual interface eth0:0 , the other two interfaces also getting disabled.

 

In RHEL5, the lowest numbered virtual interface was treated as “primary” interface, and remaining all as secondary interfaces as “secondary” . If we disable primary interface , it will by default disable all the secondary interface.

And to avoid this default behavior, we should set a kernel parameter  “net.ipv4.conf.all.promote_secondaries” to  value “1”, so that the secondary interfaces will be promoted as primary, in case of primary down.

There are two ways to set this parameter:

Method-1

# Vi /etc/sysctl.conf      

net.ipv4.conf.all.promote_secondaries = 1

#  sysctl -p

Method-2

echo “1” > /proc/sys/net/ipv4/conf/eth1/promote_secondaries

 

>>>> 5. Difference between arp monitoring and MII monitoring

 

ARP monitoring periodically sends an arp packet to the designated hosts. When the reply does not come back, then the bonding device considers the network on that specific device is down.
 
MII monitoring periodically monitors the information provided by MII (Media Independent Interface). When the interface of the driver detects the network failure, the bonding device considers the network on that specific device is down.
 
MII is a driver-dependent function. It cannot detect network failure if it occurred beyond the nearest switch, because the monitoring covers the link from the device to the nearest connected switch. Whenever emitting excessive network packets is accepted, it is safe to use ARP monitoring.

 

 

 

Ramdev

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

19 Responses

  1. Kennedy says:

    Lovely! Very easy to follow instructions and simple explanation. Good job – Keep it up.!

  2. Karthick says:

    mode-0 is Balance-rr or Active-backup? The start of the article mentioned mode-0 as Balance-rr and then throughout the article mode-0 referred as Active-backup configuration. Please clarify.

    • Ramdev Ramdev says:

      Karthik – sorry for the confusion. the mode-0 is balance-rr.
      For both mode-0 and mode-1 , interfaces will be configured as slaves.
      The only difference is, in mode-0 all the interfaces in use in round robin mode to provide load balancing and Fault tolerance. Where a in mode-1 there is one active slave at a time, and only provide the fault tolerance.

  3. vaibhav kanchan says:

    HI Ramdev,

    I have done the below steps.

    [root@cent62 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
    DEVICE=bond0
    IPADDR=192.168.1.124
    NETMASK=255.255.255.0
    NETWORK=192.168.1.0
    USERCTL=no
    BOOTPROTO=none
    ONBOOT=yes
    [root@cent62 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
    DEVICE=eth0
    BOOTPROTO=none
    ONBOOT=yes
    MASTER=bond0
    SLAVE=yes
    USERCTL=no
    [root@cent62 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
    DEVICE=eth1
    BOOTPROTO=none
    ONBOOT=yes
    MASTER=bond0
    SLAVE=yes
    USERCTL=no
    [root@cent62 ~]# cat /etc/modprobe.d/bonding.conf
    alias bond0 bonding
    options bond0 mode=1 miimon=100
    [root@cent62 ~]# cat /proc/net/bonding/bond0
    Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

    Bonding Mode: fault-tolerance (active-backup)
    Primary Slave: None
    Currently Active Slave: eth0
    MII Status: up
    MII Polling Interval (ms): 100
    Up Delay (ms): 0
    Down Delay (ms): 0

    Slave Interface: eth0
    MII Status: up
    Speed: 1000 Mbps
    Duplex: full
    Link Failure Count: 0
    Permanent HW addr: 08:00:27:88:80:0e
    Slave queue ID: 0

    Slave Interface: eth1
    MII Status: up
    Speed: 1000 Mbps
    Duplex: full
    Link Failure Count: 1
    Permanent HW addr: 08:00:27:c7:e1:e1
    Slave queue ID: 0

    but ifconfig -a is showing the below status.

    bond0 Link encap:Ethernet HWaddr 08:00:27:88:80:0E
    inet addr:192.168.1.124 Bcast:192.168.1.255 Mask:255.255.255.0
    inet6 addr: fe80::a00:27ff:fe88:800e/64 Scope:Link
    UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
    RX packets:515 errors:0 dropped:0 overruns:0 frame:0
    TX packets:390 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:45200 (44.1 KiB) TX bytes:54529 (53.2 KiB)

    eth0 Link encap:Ethernet HWaddr 08:00:27:88:80:0E
    inet addr:192.168.1.124 Bcast:192.168.1.255 Mask:255.255.255.0
    UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
    RX packets:515 errors:0 dropped:0 overruns:0 frame:0
    TX packets:379 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:45200 (44.1 KiB) TX bytes:52055 (50.8 KiB)

    eth1 Link encap:Ethernet HWaddr 08:00:27:C7:E1:E1
    inet addr:192.168.1.124 Bcast:192.168.1.255 Mask:255.255.255.0
    UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:0 (0.0 b) TX bytes:2474 (2.4 KiB)

    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:16436 Metric:1
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

  4. vaibhav kanchan says:

    what is the issue in my case?

  5. Ramdev Ramdev says:

    @vaibhav, I didn’t get your question completely, can you explain what is the issue you are talking?

  6. cuturrr says:

    options bond0 mode=1 miimon=100 should be options bonding mode=1 miimon=100 and i don’t knwo why you specified paramaters in both places 1:BONDING_OPTS && /etc/modprobe.d/bonding.conf

  7. rony says:

    Hi,
     I have Redhat 5.9 running on the hp blade server. I already have bond0 configure with mode=0 with eth0 and eth1 interface and its running fine. Now , my team wants to change mode=0 to mode =1 for this bonding bond0. What are the steps to change the bonding mode=0 to mode=1, while the bond0 is running

     Waiting for your reply as this is really urgent. Thanks

    thanks,
    Rony

  8. Javeed says:

    Nice explanation!!! keep it up

  9. Mahesh Vijapure says:

    [root@cent62 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
    DEVICE=eth0

    Please modify the device name in your example.. I know it must be typo. FYI.

    Thanks,
    Mahesh

  10. Mahesh Vijapure says:

    Cool.. Welcome.. You doing great Job Sir.. :)

  11. anon says:

    # cat > /etc/modprobe.d/bonding.conf

    alia bond0 bonding

    typo – ‘alia’ should be changed to ‘alias

  12. Manisha says:

    Hello Ramdev/team,
    I don’t see few updates in the sample box in the description. It shows blank. I found it in other topics as well.

    Cheers,
    Manisha

What is in your mind, about this post ? Leave a Reply

Close
  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us