Configuring Transitive IPMP on Solaris 11

http://www.tokiwinter.com/configuring-transitive-ipmp-on-solaris-11/

We all know the pain of configuring probe-based IPMP under Solaris, with a slew of test addresses being required, and a long line of ifconfig configuration in our /etc/hostname.<interface> files.

With Solaris 11, there is a new type of probe-based IPMP called transitive probing. This new type of probing does not require test addresses, as per the documentation: "Transitive probes are sent by the alternate interfaces in the group to probe the active interface. An alternate interface is an underlying interface that does not actively receive any inbound IP packets".

In this article, I will configure failover (active/passive) IPMP on clusternode1 (the first node of a Solaris Cluster I'm building). Interface net0 has an address of 10.1.1.80 (configured at install time), and I'll be adding this into an IPMP group ipmp0 along with a standby interface, net1. Make sure you are performing these steps via a console connection, as the original address associated with net0 will need to be removed before attempting to add it to an IPMP group.

The first step, ensure that there is an entry in /etc/hosts for the IP address you're configuring IPMP for:

# grep '^10\.1\.1\.80' /etc/hosts

10.1.1.80    clusternode1

Next, ensure that automatic network configuration is disabled. In my case it was as I'd configured networking manually during the installation of Solaris 11:

# netadm list -p ncp –x

TYPE        PROFILE        STATE          AUXILIARY STATE

ncp         Automatic      disabled       disabled by administrator

ncp         DefaultFixed   online         active

Verify that the appropriate physical interfaces are available. In the following output, I'll be bonding e1000g0 (net0) and e1000g1 (net1) into a failover IPMP group.

# dladm show-phys

LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE

net1              Ethernet             unknown    0      unknown   e1000g1

net2              Ethernet             unknown    0      unknown   e1000g2

net3              Ethernet             unknown    0      unknown   e1000g3

net0              Ethernet             up         1000   full      e1000g0

List the current addresses – from the output of ipadm show-addr I can see that I'll need to delete net0/v4 and net0/v6, otherwise I'll be unable to add net0 to the IPMP group.

# ipadm delete-addr net0/v4

# ipadm delete-addr net0/v6

As the net0 IP interface is already created, I only need to create the net1 interface:

# ipadm create-ip net1

I can then create the IPMP group, which I'll call ipmp0:

# ipadm add-ipmp -i net0 -i net1 ipmp0

Next, enable transitive probing, which is disabled by default:

# svccfg -s svc:/network/ipmp setprop config/transitive-probing=true

# svccfg -s svc:/network/ipmp listprop config/transitive-probing

config/transitive-probing boolean     true

# svcadm refresh svc:/network/ipmp:default

And configure the appropriate interface (in my case net1) to be a standby interface (as I'm using failover):

# ipadm set-ifprop -p standby=on -m ip net1

Now I can create my IPv4 address on the IPMP group:

# ipadm create-addr -T static -a clusternode1/24 ipmp0/v4

# ipadm show-addr ipmp0

ADDROBJ           TYPE     STATE        ADDR

ipmp0/v4          static   ok           10.1.1.80/24

Finally, fix the default route. I removed the existing route and added a new default route using the new and correct interface – ipmp0:

# route -p delete default 10.1.1.1

# route -p add default 10.1.1.1 -ifp ipmp0

# netstat -rn -f inet

   

Routing Table: IPv4

  Destination           Gateway           Flags  Ref     Use     Interface

-------------------- -------------------- ----- ----- ---------- ---------

default              10.1.1.1             UG        1          0 ipmp0

10.1.1.0             10.1.1.80            U         8        388 ipmp0

127.0.0.1            127.0.0.1            UH        2        554 lo0

You can use ipmpstat to verify the configuration and health of the IPMP group:

# ipmpstat -g

GROUP       GROUPNAME   STATE     FDT       INTERFACES

ipmp0       ipmp0       ok        10.00s    net0 (net1)

# ipmpstat -a

ADDRESS                   STATE  GROUP       INBOUND     OUTBOUND

::                         down   ipmp0       --          --

clusternode1              up     ipmp0       net0        net0

# ipmpstat -t

INTERFACE   MODE       TESTADDR            TARGETS

net1        transitive <net1>              <net0>

net0        routes     clusternode1        10.1.1.1

Let's perform a failover test. I'll disable net0 and ensure that the clusternode1 address fails over:

# ipadm disable-if -t net0

# ipmpstat -t

INTERFACE   MODE       TESTADDR            TARGETS

net1        routes     clusternode1        10.1.1.1

It works! (and my SSH connection is still active…) – net1 is now active with the correct IP address. Let's fail it back:

# ipadm enable-if -t net0

# ipmpstat -t

INTERFACE   MODE       TESTADDR            TARGETS

net0        routes     clusternode1        10.1.1.1

net1        transitive <net1>              <net0>

The address has failed back to net0, and again my SSH connection is still active. I can now continue with clusternode2, and the rest of the cluster install.