Fiberchannel Solaris Part 2: Configuring SAN and TCP/IP

After having looked at some fundamental fiberchannel principles, let's configure our Solaris host to use fiberchannel devices. The machines are connected to a fiber channel fabric as well as two storage devices - all in the same fiberchannel zone.

1. SAN (Storage)


To rescan all fiberchannel devices to be able to use them and to see the result, issue a "cfgadm -al" command:

# cfgadm -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c3                             scsi-bus     connected    configured   unknown
c3::dsk/c3t0d0                 disk         connected    configured   unknown
c4                             fc-fabric    connected    configured   unknown
c4::2100001b321fb79d           unknown      connected    unconfigured unknown
c4::210000d0231c50ab           disk         connected    configured   unknown
c4::210000e08b0d82c6           unknown      connected    unconfigured unknown
c4::220000d0232c50be           disk         connected    configured   unknown
c5                             fc-fabric    connected    configured   unknown
c5::2100001b321f9eb3           unknown      connected    unconfigured unknown
c5::210000d0231c50be           disk         connected    configured   unknown
c5::220000d0232c50ab           disk         connected    configured   unknown

[... other lines omitted...]

In this example we have two storage device with the following fiberchannel addresses:

Device 1:
WWNN: 200000d0230c50ab
WWPN: 210000d0231c50ab,220000d0232c50ab

Device 2:
WWNN: 200000d0230c50be
WWPN: 210000d0231c50be,220000d0232c50be

"c4" in the cfgadm output represents Fiberchannel Interface Card 1, c5 Card 2. Each storage device is seen by c4 and c5 - but with different WWPNs - this seems to be a rendundant fiberchannel setup.

For disks to show up, just type:

# devfsadm -c disk


This command creates new /devices and /dev-entries if needed. The storage devices are now available to the Solaris operating system and they should appear when invoking the "format" command:

# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c3t0d0 <DEFAULT cyl 8872 alt 2 hd 255 sec 63>
          /pci@7b,0/pci1022,7458@11/pci1000,3060@2/sd@0,0
       1. c4t210000D0231C50ABd0 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
          /pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0/disk@w210000d0231c50ab,0
       2. c4t210000D0231C50ABd1 <ADVUNI-OXYGENRAID 416F4-347B-698.38GB>
          /pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0/disk@w210000d0231c50ab,1
       3. c4t210000D0231C50ABd2 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
          /pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0/disk@w210000d0231c50ab,2
       4. c4t220000D0232C50BEd0 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
          /pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0/disk@w220000d0232c50be,0
       5. c4t220000D0232C50BEd1 <ADVUNI-OXYGENRAID 416F4-347B-698.38GB>
          /pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0/disk@w220000d0232c50be,1
       6. c4t220000D0232C50BEd2 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
          /pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0/disk@w220000d0232c50be,2
       7. c5t210000D0231C50BEd0 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
          /pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0/disk@w210000d0231c50be,0
       8. c5t210000D0231C50BEd1 <ADVUNI-OXYGENRAID 416F4-347B-698.38GB>
          /pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0/disk@w210000d0231c50be,1
       9. c5t210000D0231C50BEd2 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
          /pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0/disk@w210000d0231c50be,2
      10. c5t220000D0232C50ABd0 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
          /pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0/disk@w220000d0232c50ab,0
11. c5t220000D0232C50ABd1 <ADVUNI-OXYGENRAID 416F4-347B-698.38GB>
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0/disk@w220000d0232c50ab,1
12. c5t220000D0232C50ABd2 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0/disk@w220000d0232c50ab,2

First, look at the device names:

c5t22000D0232C50ABd0:

Controller 5: c5
Target 22000D0232C50AB (WWPN): t22000D0232C50AB
LUN 0: d0

The storage device is divided in multiple LUNs. SCSI allows to define different Logical Units (LUNs) available at the same target. RAID devices very often use this to diffentiate between created volumes on the storage device.
The target specification is just the WWPN of the device.

Another thing is apparent: The devices are appearing twice. Example: It is quite obvious that

       5. c4t220000D0232C50BEd1 <ADVUNI-OXYGENRAID 416F4-347B-698.38GB>
          /pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0/disk@w220000d0232c50be,1
represents the same disk volume as

       8. c5t210000D0231C50BEd1 <ADVUNI-OXYGENRAID 416F4-347B-698.38GB>
          /pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0/disk@w210000d0231c50be,1

They belong to the same WWNN, the same device. And the LUN is the same as well: d1. It's the same "disk".

So we need a software layer to correct this - otherwise this leads to chaos. Imagine you format disk 5, get data on it and you think disk 8 is available and you format this disk as well and mount it - the system will crash or immediately stops from using disk 5. The Operating System does not now that disk 8 is the same as 5 but at the point you begin to write on disk 8 it will notice that disk 5 has changed data without its knowledge - stop.

Now the scsi_vhci driver gets on his track.
First, we need to allow "fp" (the fiberchannel driver) to be able to use multipathing. Look at the configuration file

/kernel/drv/fp.conf :

Just make sure that MPxIO (Multipathing) is enabled. Normally you see that line in fp.conf:

mpxio-disable="yes";

If you see such a line, replace "yes" by "no":

mpxio-disable="no";

Next step: Be sure that scsi_vhci supports your device. Look at

http://southbrain.com/south/2008/02/solaris-10-with-mpxio-scsi-vhc.html (for Solaris 10) or

http://southbrain.com/south/2009/09/sun-mpxio-storage-scsi-vhci-so.html (for OpenSolaris)

In my case (using Solaris 10 and a non-supported non-Sun array), I had to write this in my

/kernel/drv/scsi_vhci.conf :


device-type-scsi-options-list =
"ADVUNI  OXYGENRAID", "symmetric-option";

symmetric-option = 0x1000000;


In case you want to use an asymmetric link (only one link is active and the others are standby) with such a custom device (e.g. the device does not accept symmetric usage), just write this:


device-type-scsi-options-list =
"ADVUNI  OXYGENRAID", "asymmetric-option";

asymmetric-option = 0x2000000;

Many devices are supported out of the box with scsi_vhci, so you won't need to configure it. Setting    mpxio-disable="no";   in fp.conf as mentioned above is sufficient in this case.

Do a reconfiguration reboot:

# reboot -- -r

After boot, "format" will give you another output:

# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c3t0d0 <DEFAULT cyl 8872 alt 2 hd 255 sec 63>
          /pci@7b,0/pci1022,7458@11/pci1000,3060@2/sd@0,0
       1. c6t600D0230006B66680C50AB0F36ADF100d0 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
          /scsi_vhci/disk@g600d0230006b66680c50ab0f36adf100
       2. c6t600D0230006B66680C50AB4F92F61000d0 <ADVUNI-OXYGENRAID 416F4-347B-698.38GB>
          /scsi_vhci/disk@g600d0230006b66680c50ab4f92f61000
       3. c6t600D0230006B66680C50AB5675F91300d0 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
          /scsi_vhci/disk@g600d0230006b66680c50ab5675f91300
       4. c6t600D0230006C1C4C0C50BE4DFE511B00d0 <ADVUNI-OXYGENRAID 416F4-347B-698.38GB>
          /scsi_vhci/disk@g600d0230006c1c4c0c50be4dfe511b00
       5. c6t600D0230006C1C4C0C50BE16FF1FE200d0 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
          /scsi_vhci/disk@g600d0230006c1c4c0c50be16ff1fe200
       6. c6t600D0230006C1C4C0C50BE57396E9F00d0 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
          /scsi_vhci/disk@g600d0230006c1c4c0c50be57396e9f00


The links are aggregated now. And - the controller number has changed!

MPxIO has replaced c4 and c5 by a virtual host controller interface (v h c i ...) c6! In fact c4 and c5 are hidden by the scsi_vhci driver.

Now it is safe to use these "disks" - despite of their long device name! You will love a terminal with copy & paste support to enter commands involving these disk devices.

To create a zpool on disk 6 above just type:

# zpool create mypool c6t600D0230006C1C4C0C50BE57396E9F00d0
Like you would do with a physical hardware scsi disk.

2. IP (Network)

To set up networking, just plumb the fcipX interfaces (X = fiberchannel port, X=0,1,2,3,....):

# ifconfig fcip0 plumb

Result:

# ifconfig fcip0
fcip0: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 65280 index 3
        inet 0.0.0.0 netmask 0
        ether 0:1b:32:1f:bd:9c

You're ready to assign an IP address:

# ifconfig fcip0 192.168.47.11 netmask 0xffffff00 broadcast 192.168.47.255 up
# ifconfig fcip0
fcip0: flags=1001843<UP,BROADCAST,RUNNING,MULTICAST,MULTI_BCAST,IPv4> mtu 65280 index 3
        inet 192.168.47.11 netmask ffffff00 broadcast 192.168.47.255
        ether 0:1b:32:1f:bd:9c

Alias interfaces are also possible:

# ifconfig fcip0:1 plumb
# ifconfig fcip0:1 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255 up
# ifconfig fcip0:1
fcip0:1: flags=1001843<UP,BROADCAST,RUNNING,MULTICAST,MULTI_BCAST,IPv4> mtu 65280 index 3
        inet 10.1.1.1 netmask ffffff00 broadcast 10.1.1.255
The high MTU value (65280) makes Fiberchannel IP very efficient to transfer high data rates (up to  8 Gbit/sec fiberchannel).

Be careful to use proper zoning when using fiberchannel storage across firewalled networks. Firewalls are easily circumvented with Fiberchannel IP when all storage is in one zone...

NETWORK MULTIPATHING IS NOT DONE VIA SCSI_VHCI! EVEN IF YOU USE MPXIO FOR SAN (AS ABOVE) YOU'LL HAVE FCIP0 AND FCIP1. USE NETWORK MULTIPATHING IF YOU PLAN TO LINK TWO FIBERCHANNEL IP LINKS TOGETHER.

Part 3 deals with diagnostic commands.

0 TrackBacks

Listed below are links to blogs that reference this entry: Fiberchannel Solaris Part 2: Configuring SAN and TCP/IP.

TrackBack URL for this entry: http://southbrain.com/mt/mt-tb.cgi/113

3 Comments

excellent. thank you.

Excellent topic. It helped me a lot to understand the SAN concepts

Hi Pascal,
Many thanks for sharing the configuration in detail.

I am trying for asymetric option for Netapp Luns.
When I am opting for symetric option Netapp devices are being detec ted under MPxIO and I am getting the stms device names.

device-type-scsi-options-list =
"NETAPP LUN", "symmetric-option";
symmetric-option = 0x1000000;

However when I am trying to configure the Netapp Luns for asymetric, they are not getting detected by MPxIO.
device-type-scsi-options-list =
"NETAPP LUN", "asymmetric-option";
asymmetric-option = 0x2000000;

Could you please help on this.

Does the vhci driver support netapp luns for asymmetric options?

December 2015

Sun Mon Tue Wed Thu Fri Sat
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    

About

This blog is owned by:

Pascal Gienger
J├Ągerstrasse 77
8406 Winterthur
Switzerland


Google+: Profile
YouTube Channel: pascalgienger