In many of the entries this blog contains the topic lied on MPxIO configuration for fiberchannel storage links:http://southbrain.com/south/2008/02/solaris-10-with-mpxio-scsi-vhc.html (for Solaris 10) or
http://southbrain.com/south/2009/09/sun-mpxio-storage-scsi-vhci-so.html (for OpenSolaris)
Now let's have a look into Solaris IP networking redundancy.
As opposed to the Compaq/HP "teaming" solution (offered for Windows Operating Systems and Linux) the Sun solution operates on the IP layer and not on the ethernet layer.
It needs a reachable gateway (or adjacent host) and three distinct IP addresses as minimum per multipathing group.
Look at the graphic above on the right (click on it to detach a larger version).
On the lower right, you see the (Open)Solaris host. In this example two network interfaces (e1000g0 and bge0) will form a so-called "multipathing group". Each multipathing group will be represented by a virtual interface named "ipmp". In this case, e1000g0 and bge0 form the multipath group interface ipmp0.
The Intel Gigabit Interface e1000g0 is configured to IPv4 192.168.71.89, the Broadcom Gigabit Ethernet Interface is configured to 192.168.71.90.
To define a multipathing group for these two interfaces first choose a name for this group. As an example we'll use "testgroup" here.
Each Solaris interface definition (controllable by ifconfig) has the following additional keywords to define multipathing properties:
In our case, the three IP addresses must be defined as follows:
192.168.71.89 -failover
192.168.71.90 -failover
192.168.71.88 failover
The "group" property defines the multipathing group name. All interfaces sharing the same group name are in one multipathing group. The addresses marked as "failover" may slide from one interface to another.
It's time to configure the interfaces!
First, let's do it manually.
To define the two interfaces, you would type:
Note the "-failover", the IP adresses are normally reachable - the multipathing will not (yet!) take control of it.
Now define a subinterface on one of these two interfaces, defining the failover address (multipath group address):
And, things begin to run! In that moment, your Solaris operating system begins to spawn a "in.mpathd" daemon and a new interface has appeared: ipmp0:
And - the "real" interfaces do not show up any more when invoking "ifconfig -a". They've disappeared - in fact they're controlled by in.mpathd:
This daemon just started in the moment when you defined an ip address as a subinterface on an interface belonging to a multipath group (group <groupname> option in ifconfig) and assigning the "failover" option.
It is VERY important that you have defined all relevant addresses in your multipathing group with "-failover" before defining a "failover" (subinterface) address.
Now let's make the startup/boot config:
That's what the "hostname.xxxx"-files are for (in the /etc directory).
For our example you would create three files:
/etc/hostname.e1000g0 :
/etc/hostname.bge0 :
/etc/hostname.e1000g0:1 :
That's all.
The IP addresses of the "real" interfaces are still occupied and used! Sun MPxIO uses them to reach adjacent hosts or the gateway.
TO MAKE THIS WORK YOUR DEFAULT GATEWAY ADDRESS OR THE ADDRESSES WHICH ARE GATHERED BY THE in.rdiscd OR in.routed DAEMONS MUST RESPOND TO ICMP ECHO REQUESTS ("ping"). IF THE GATEWAY/ROUTER IS A FIREWALL YOU MUST TELL IT TO REPLY TO ICMP ECHO REQUEST PACKETS BY A VALID ECHO REPLY. THIS IS NOT A SECURITY PROBLEM HERE, IT IS NEEDED FOR THIS FEATURE TO FUNCTION!
It needs a reachable gateway (or adjacent host) and three distinct IP addresses as minimum per multipathing group.
Look at the graphic above on the right (click on it to detach a larger version).
On the lower right, you see the (Open)Solaris host. In this example two network interfaces (e1000g0 and bge0) will form a so-called "multipathing group". Each multipathing group will be represented by a virtual interface named "ipmp". In this case, e1000g0 and bge0 form the multipath group interface ipmp0.
The Intel Gigabit Interface e1000g0 is configured to IPv4 192.168.71.89, the Broadcom Gigabit Ethernet Interface is configured to 192.168.71.90.
To define a multipathing group for these two interfaces first choose a name for this group. As an example we'll use "testgroup" here.
Each Solaris interface definition (controllable by ifconfig) has the following additional keywords to define multipathing properties:
- group <groupname>
- failover
In our case, the three IP addresses must be defined as follows:
192.168.71.89 -failover
192.168.71.90 -failover
192.168.71.88 failover
The "group" property defines the multipathing group name. All interfaces sharing the same group name are in one multipathing group. The addresses marked as "failover" may slide from one interface to another.
It's time to configure the interfaces!
First, let's do it manually.
To define the two interfaces, you would type:
ifconfig e1000g0 plumb
ifconfig bge0 plumb
ifconfig e1000g0 192.168.71.89 netmask 255.255.255.0 -failover group testgroup up
ifconfig bge0 192.168.71.90 netmask 255.255.255.0 -failover group testgroup upNote the "-failover", the IP adresses are normally reachable - the multipathing will not (yet!) take control of it.
Now define a subinterface on one of these two interfaces, defining the failover address (multipath group address):
ifconfig e1000g0:1 plumb
ifconfig e1000g0:1 192.168.71.88 netmask 255.255.255.0 failoverAnd, things begin to run! In that moment, your Solaris operating system begins to spawn a "in.mpathd" daemon and a new interface has appeared: ipmp0:
ipmp0: flags=8001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP> mtu 1500 index 5
inet 192.168.71.88 netmask ffffff00 broadcast 192.168.71.255
groupname testgroupAnd - the "real" interfaces do not show up any more when invoking "ifconfig -a". They've disappeared - in fact they're controlled by in.mpathd:
root 42 1 0 Nov 06 ? 64:58 /lib/inet/in.mpathd
This daemon just started in the moment when you defined an ip address as a subinterface on an interface belonging to a multipath group (group <groupname> option in ifconfig) and assigning the "failover" option.
It is VERY important that you have defined all relevant addresses in your multipathing group with "-failover" before defining a "failover" (subinterface) address.
Now let's make the startup/boot config:
That's what the "hostname.xxxx"-files are for (in the /etc directory).
For our example you would create three files:
/etc/hostname.e1000g0 :
192.168.71.89 netmask 255.255.255.0 -failover group testgroup/etc/hostname.bge0 :
192.168.71.90 netmask 255.255.255.0 -failover group testgroup/etc/hostname.e1000g0:1 :
192.168.71.88 netmask 255.255.255.0 failover
That's all.
The IP addresses of the "real" interfaces are still occupied and used! Sun MPxIO uses them to reach adjacent hosts or the gateway.
TO MAKE THIS WORK YOUR DEFAULT GATEWAY ADDRESS OR THE ADDRESSES WHICH ARE GATHERED BY THE in.rdiscd OR in.routed DAEMONS MUST RESPOND TO ICMP ECHO REQUESTS ("ping"). IF THE GATEWAY/ROUTER IS A FIREWALL YOU MUST TELL IT TO REPLY TO ICMP ECHO REQUEST PACKETS BY A VALID ECHO REPLY. THIS IS NOT A SECURITY PROBLEM HERE, IT IS NEEDED FOR THIS FEATURE TO FUNCTION!
Leave a comment