IPMP Configurations – In Solaris 8 & 9

Summary of typical IPMP Configurations:

1.  Production and test interfaces in the same IP subnet

1.1  With defaultrouter
1.2  Without defaultrouter
1.3  With dedicated hosts acting as test targets with “host-routes”
1.4  Configuration example for 1.1, 1.2 and 1.3

2.  Production and test interfaces in different IP subnets but the same physical network

2.1  With defaultrouter in production subnet and test subnet
2.2  With defaultrouter in production subnet but without defaultrouter in test subnet
2.3  With dedicated hosts acting as test targets with “host-routes”
2.4  Configuration example for 2.1, 2.2 and 2.3
2.5  Configuration example for 2.2 if you use IPMP on the test subnet

A.  Start script for adding static “host routes” permanently  /etc/init.d/ipmp.targets

Introduction:

The operation of IP Multipathing (in.mpathd) depends on the routing configuration. Therefore in.mpathd always refers to the routing-table  (IRE-cache) to distinguish which test partner(s) are going to be used. Test partners are required for deciding if the interface is working properly.

in.mpathd by default chooses the defaultrouter as single test-target for probing. If no defaultrouter exists for the test-interface ip address, arbitrary hosts on the link are detected by sending out “all hosts” multicast packets (224.0.0.1) on the wire to detect its test-partners.

An “all routers” multicasts (224.0.0.2) will never be sent! The first five  hosts that are responding to the echo packets are chosen as targets for probing. In such a non-defaultrouter environment, the in.mpathd  always tries to find five probe targets via an “all hosts” multicast.

The in.mpathd detects failures and repairs by sending out ‘icmp-echo’  probes (like pinging) from all interfaces that are part of the IPMP group.

If there are no replies to five consecutive probes, the interface is considered to have failed and performs a failover of the network access to another interface in the IPMP group. The probing rate depends on the  failure detection time which is defined in /etc/default/mpathd. By default, failure detection time is 10 seconds. Thus the five probes will be sent within the failure detection time.

1.  Production and test interfaces in the same IP subnet

1.1 With defaultrouter

IPMP only use the defaultrouter as probe target. Each test interface of the IPMP group send ICMP requests only to the defaultrouter. To get the configuration, IPMP looks to the routing table and is independent from /etc/defaultrouter file. NO “all hosts” multicast (224.0.0.1) will be sent.

Advantages:
– Easiest configuration for IPMP.

Disadvantages:

– When the defaultrouter is down IPMP failover does not work anymore.  The in.mpathd does NOT send out multicasts to get other probe  targets, therefore all interfaces in the IPMP group get the state  “failed”. You can ignore this bug/feature when you have a defaultrouter which is 100% online! Please look to RFE 4431511 and 4489960 for further information.

– If you have a lot of IPMP groups, the defaultrouter has to reply to a lot of ICMP requests. Take care of defaultrouter. Do not overload  the defaultrouter.
– The defaultrouter has to reliably answer ICMP echo requests. (e.g.  firewalls sometimes do not)

1.2 Without defaultrouter

IPMP dynamically determines five arbitrary hosts on the link via “all hosts” multicast address (224.0.0.1). At the least, you need one probe target that IPMP will work. Beware that one probe target is not reliable enough. If there are less than five targets available the in.mpathd sent out the “all hosts” multicasts to get a complete list of five probe targets.

Advantages:
– easiest configuration for IPMP in a subnet without a defaultrouter.
– is very reliable due to the five targets.

Disadvantages:
– a subnet without an defaultrouter is very rare.

1.3 With dedicated hosts acting as test targets with “host-routes”

Solaris 8 and 9 – Some “host routes” will be defined with a startscript in /etc/rc2.d/S70staticroutes.  (The script is attached to this document.)

When IPMP refer to the routing table it will choose the first five defined “host routes” as probe targets. This is due to the fact that normally the “host routes” are before the defaultrouter in the routing table. If you have less than five “host routes” also the defaultrouter (when available) will be used as probe target as well.

Example:
a) Configuration with host1, host2 … hostN (with N=5 or N>5), defaultrouter :

==> The first five hosts (host1 … host5) will be defined as target.

b) Configuration with less than 5 hosts : for instance, host1, host2, defaultrouter :

==> The three systems (host1, host2, defaultrouter) will be defined as target.

Also in this case the in.mpathd tries to get five probe targets all the time from the routing table. Remember in this configuration the in.mpathd does NOT send “all hosts” multicasts!
Advantages:
– The defaultrouter is not important for the IPMP configuration because if the defaultrouter is not available you have still some “host routes” for probing.
– IPMP is always high available due to independency to the defaultrouter

Disadvantages:
– More administrative work to do.
– Due to static configuration you should check that some of the probe targets   are always available.

1.4 Configuration examples for 1.1, 1.2 and 1.3

/etc/hosts
———-
127.0.0.1                localhost
172.20.20.10        host10       loghost
172.20.20.210     host10-test-qfe0
172.20.20.220     host10-test-qfe4

/etc/hostname.qfe0
——————
host10 netmask + broadcast + group ipmp0 up
addif host10-test-qfe0 deprecated -failover netmask + broadcast + up

/etc/hostname.qfe4
——————
host10-test-qfe4 deprecated -failover netmask + broadcast + group ipmp0 up

ifconfig output:
—————-
qfe0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e8:88:dc
qfe0:1:
flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 172.20.20.210 netmask ffffff00 broadcast 172.20.20.255
qfe4:
flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 4
inet 172.20.20.220 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e8:89:34

Note: The example describe the setup of an active-active IPMP configuration  which is best practices. But you can also configure an active-standby   configuration. You simple have to add the “standby” flag to the
/etc/hostname.qfe4 file.

2.  Production and test interfaces in different IP subnets

If you have not enough additional ip-addresses on hand for setting up IPMP, you can configure the ipmp-test-interfaces in a different ip-network than your production network (e.g. 192.168., 10. ..). But you must make sure that there are enough test-partners (also in the new test-network) who are responding to the ipmp-test-interfaces. You may also configure a defaultrouter in the new test-network in case you have an existing 100.1% reliable test-partner which should act as a single test-partner. In such a configuration in.mpathd will only use its test-subnet IP addresses as source address for outgoing probe packets.
Note: The in.mpathd only looks to the test subnet. Therefore if you have no IP addresses available in the test subnet the IPMP group will fail although if the production subnet is available.

2.1 With defaultrouter in production subnet and test subnet

IPMP only use the defaultrouter as probe target. Each test interface of the IPMP group send ICMP requests only to the defaultrouter. To get the configuration IPMP looks to the routing table and is independent from
/etc/defaultrouter file. NO “all hosts” multicast (224.0.0.1) will be sent.

Advantages:
– Test interfaces don’t need IP addresses of the production subnet

Disadvantages:
– The interface of the defaultrouter has to reside in both the production AND test subnet.
– Exceptional configuration of defaultrouter.
– All which are mentioned in section 1.1

2.2 With defaultrouter in production subnet net but without defaultrouter in test subnet

IPMP dynamically determines five arbitrary hosts in the test subnet via “all hosts” multicast  address (224.0.0.1). At least you need one probe target that IPMP will work. Beware that one probe target is not reliable enough. If there are less than five targets available the in.mpathd sent out the “all hosts” multicasts to get a complete list of five probe targets.

Advantages:
– easiest configuration for IPMP if you have too less IP addresses  available in the production subnet.
– is very reliable due to the five targets.

Disadvantages:
– the probe targets must be available before you can setup the IPMP host.
– more administrative work because you have to setup the probe targets as an additional interface in the test subnet.

Recommendation: Setup an additional logical network interface on target host. (e.g. add a new interface to /etc/hostname.qfe0 with the ‘addif’ option) If your test partners are on systems which run also IPMP you must add an logical network interface NOT flagged deprecated and NOT flagged nofailover. An address associated with this interface will then be used as source address by responding properly to “all hosts” multicast packets which are used for the automatic IPMP test partner detection.

Beware that Solaris 8 does NOT require the additional logical network  interface in the test subnet on the target host. So, in case of an upgrade from Solaris 8 to Solaris[TM] 9 or higher you have to change your configuration.

2.3 with dedicated hosts acting as test targets with “host-routes”

Solaris 8 and 9 – Some “host routes” will be defined in the test subnet with a startscript in /etc/rc2.d/S70ipmp.targets (The script is attached to this document.). When IPMP refer to the routing table it will choose the first five defined “host routes” as probe targets in the test subnet. This is due to the fact that normally the “host routes” are before the defaultrouter in the routing table. If you have less than five “host routes” also the defaultrouter (when available in the test subnet) will be used as probe target as well.

Example:
– please look to the example of section 1.3

Solaris 10 – Use the “route -p” option to add a persistent route to the kernel. No additional script will be necessary.

Example:
In this example, the active interface on the system is 10.8.118.104 The test host targets are 5 systems that are external to this physical system. The IP address used for the default router is the same IP address that was in the /etc/defaultrouter file.

# rm /etc/defaultrouter
# route -p add host 10.8.118.1 10.8.118.1
# route -p add host 10.8.118.3 10.8.118.3
# route -p add host 10.8.118.40 10.8.118.40
# route -p add host 10.8.118.41 10.8.118.41
# route -p add host 10.8.118.193 10.8.118.193
# route -p add default 10.8.118.248

Advantages:
– test interfaces don’t need IP addresses of the production subnet
– all which are mentioned in section 1.3

Disadvantages:
– all which are mentioned in section 1.3
– all which are mentioned in section 2.2

2.4 Configuration examples for 2.1, 2.2 and 2.3

/etc/hosts
———-
127.0.0.1        localhost
172.20.20.10     host10       loghost
192.168.1.210    host10-test-qfe0
192.168.1.220    host10-test-qfe4

/etc/hostname.qfe0
——————
host10 netmask + broadcast + group ipmp0 up
addif host10-test-qfe0 deprecated -failover netmask + broadcast + up

/etc/hostname.qfe4
——————
host10-test-qfe4 deprecated -failover netmask + broadcast + group ipmp0 up

ifconfig output:
—————-
qfe0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e8:88:dc
qfe0:1:
flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>  mtu 1500 index 3
inet 192.168.1.210 netmask ffffff00 broadcast 192.168.1.255
qfe4:
flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
inet 192.168.1.220 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp0
ether 8:0:20:e8:89:34

Note: The example describe the setup of an active-active IPMP configuration which is best practices. But you can also configure an active-standby  configuration. You simple have to add the “standby” flag to the /etc/hostname.qfe4 file.

2.5  Configuration example for 2.2 if you use IPMP on the test subnet

/etc/hosts
———-
127.0.0.1                        localhost

# IPMP active IP address:
172.20.20.10               host10       loghost

# IPMP test interface
192.168.1.210              host10-test-ce0
# additional active interface in the ‘test subnet’
# to be able to respond to probe packets from other
# IPMP setups (See 2.2). Will also failover.
192.168.1.211              host10-prod-ce0
# IPMP test interface
192.168.1.220              host10-test-ce1

/etc/hostname.ce0
—————–
172.20.20.10 netmask + broadcast + group ipmp0 up
addif 192.168.1.211 netmask + broadcast + up
addif 192.168.1.210 netmask + broadcast + deprecated -failover up

/etc/hostname.ce1
—————–
192.168.1.220 netmask + broadcast + group ipmp0 deprecated -failover up

ifconfig output:
—————-
ce0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e2:da:d5
ce0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.1.211 netmask ffffff00 broadcast 192.168.1.255
ce0:2: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
inet 192.168.1.210 netmask ffffff00 broadcast 192.168.1.255
ce1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 192.168.1.220 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp0
ether 8:0:20:e2:da:d6

===============================================================================
———– Begin of start script /etc/init.d/ipmp.targets ————–

#!/sbin/sh
# /etc/rc2.d/S70ipmp.targets /etc/init.d/ipmp.targets
# Copyright (c) 2005 by Sun Microsystems, Inc.
# All rights reserved.
#
#ident  “@(#)ipmp.targets      1.0.0
#
# Edit the following IPMP test  TARGETS to suit your needs.
# To install:
# 1) cp ipmp.targets /etc/init.d
# 2) perform edits on the script as required (e.g: add TARGETS)
# 3) chmod 744 /etc/init.d/ipmp.targets
# 4) chown root:sys /etc/init.d/ipmp.targets
# 5) ln /etc/init.d/ipmp.targets /etc/rc2.d/S70ipmp.targets
#
TARGETS=”192.168.85.117 192.168.85.127 192.168.85.137″

case “$1” in
‘start’)
/usr/bin/echo “Adding static routes for use as IPMP targets”
for target in $TARGETS; do
/usr/sbin/route add -host $target $target
done
;;
‘stop’)
/usr/bin/echo “Removing static routes for use as IPMP targets”
for target in $TARGETS; do
/usr/sbin/route delete -host $target $target
done
;;
esac
———– End of start script /etc/init.d/ipmp.targets ————–

 

Ramdev

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

What is in your mind, about this post ? Leave a Reply

Close
  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us