Fiberchannel Solaris Part 3: SAN Diagnostics

1. Kernel modules loaded?

Check whether the needed kernel modules/drivers are loaded:

# modinfo|grep fc 
 93 ffffffffef17d000  19ac0  87   1  fcp (SunFC FCP v20090210-1.134)
 94 ffffffffef196000   a2a0   -   1  fctl (SunFC Transport v20090210-1.59)
157 ffffffffef3e8000   9620  83   1  fcip (SunFC FCIP v20090210-1.50)
160 ffffffffef494850   5610  92   1  fcsm (Sun FC SAN Management v20070116)

166 ffffffffef4b5bd8   1fc0 151   1  ippctl (IP Policy Configuration Driver )

The needed modules are printed in bold. The last line just matches because it also contains "fc".

2. Which FC interfaces do I have?

Use "fcinfo":

# fcinfo hba-port
HBA Port WWN: 2100001b321fbd9c
        OS Device Name: /dev/cfg/c4
        Manufacturer: QLogic Corp.
        Model: 375-3355-01
        Firmware Version: 4.04.01
        FCode/BIOS Version:  BIOS: 1.24; fcode: 1.24; EFI: 1.8;
        Serial Number: 0402R00-0725369970
        Driver Name: qlc
        Driver Version: 20081115-2.29
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2000001b321fbd9c
HBA Port WWN: 2100001b321fe69b
        OS Device Name: /dev/cfg/c5
        Manufacturer: QLogic Corp.
        Model: 375-3355-01
        Firmware Version: 4.04.01
        FCode/BIOS Version:  BIOS: 1.24; fcode: 1.24; EFI: 1.8;
        Serial Number: 0402R00-0725369921
        Driver Name: qlc
        Driver Version: 20081115-2.29
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2000001b321fe69b


Result: Two ports available, one port per fiberchannel card. The WWPNs are printed in bold.
There's also the port type (N_port in this case) and the current data bit rate.

3. Which WWPNs (storage devices) are seen from my fiberchannel interface ports?

Use "fcinfo remote-port -p" with the interface port WWPN found in step 2:

# fcinfo remote-port -p 2100001b321fbd9c
Remote Port WWN: 210000e08b0d82c6
        Active FC4 Types: SCSI,IP
        SCSI Target: no
        Node WWN: 200000e08b0d82c6
Remote Port WWN: 2100001b321fb79d
        Active FC4 Types: SCSI,IP
        SCSI Target: no
        Node WWN: 2000001b321fb79d
Remote Port WWN: 210000d0231c50ab
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 200000d0231c50ab
Remote Port WWN: 220000d0232c50be
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 200000d0232c50be


"SCSI" types are storage disks (storage volumes).
"SCSI,IP" types are devices capable of talking IP and SCSI (storage). In this case this is a Solaris host with IP configured.

4. Are all disks configured as solaris disks (usable by Solaris)?

Use "cfgadm -al" (this will configure also newly added fc disks):

# cfgadm -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c3                             scsi-bus     connected    configured   unknown
c3::dsk/c3t0d0                 disk         connected    configured   unknown
c4                             fc-fabric    connected    configured   unknown
c4::2100001b321fb79d           unknown      connected    unconfigured unknown
c4::210000d0231c50ab           disk         connected    configured   unknown
c4::210000e08b0d82c6           unknown      connected    unconfigured unknown
c4::220000d0232c50be           disk         connected    configured   unknown
c5                             fc-fabric    connected    configured   unknown
c5::2100001b321f9eb3           unknown      connected    unconfigured unknown
c5::210000d0231c50be           disk         connected    configured   unknown
c5::220000d0232c50ab           disk         connected    configured   unknown

usb0/1                         unknown      empty        unconfigured ok
usb0/2                         unknown      empty        unconfigured ok
usb0/3                         usb-device   connected    configured   ok
usb0/4                         usb-storage  connected    configured   ok
usb0/5                         usb-floppy   connected    configured   ok
usb0/6                         unknown      empty        unconfigured ok
usb0/7                         unknown      empty        unconfigured ok
usb1/1                         unknown      empty        unconfigured ok
usb1/2                         unknown      empty        unconfigured ok
usb1/3                         unknown      empty        unconfigured ok
usb1/4                         unknown      empty        unconfigured ok
usb1/5                         unknown      empty        unconfigured ok
usb1/6                         usb-hub      connected    configured   ok
usb1/6.1                       unknown      empty        unconfigured ok
usb1/6.2                       unknown      empty        unconfigured ok
usb1/6.3                       unknown      empty        unconfigured ok
usb1/6.4                       unknown      empty        unconfigured ok
usb1/7                         unknown      empty        unconfigured ok

The relevant lines are printed in bold.

5. Is multipathing working?

Use "luxadm display" with the target WWPN of the storage disk device to list the paths:

# luxadm display 220000d0232c50be
DEVICE PROPERTIES for disk: /dev/rdsk/c6t600D0230006C1C4C0C50BE16FF1FE200d0s2
  Vendor:               ADVUNI 
  Product ID:           OXYGENRAID 416F4
  Revision:             347B
  Serial Num:           0C50BE16FF1FE200
  Unformatted capacity:  0.000 MBytes
  Write Cache:          Enabled
  Read Cache:           Enabled
    Minimum prefetch:   0x0
    Maximum prefetch:   0xffff
  Device Type:          Disk device
  Path(s):

  /dev/rdsk/c6t600D0230006C1C4C0C50BE16FF1FE200d0s2
  /devices/scsi_vhci/disk@g600d0230006c1c4c0c50be16ff1fe200:c,raw
   Controller           /dev/cfg/c4
    Device Address              220000d0232c50be,2
    Host controller port WWN    2100001b321fbd9c
    Class                       primary
    State                       ONLINE
   Controller           /dev/cfg/c5
    Device Address              210000d0231c50be,2
    Host controller port WWN    2100001b321fe69b
    Class                       primary
    State                       ONLINE

DEVICE PROPERTIES for disk: /dev/rdsk/c6t600D0230006C1C4C0C50BE4DFE511B00d0s2
  Vendor:               ADVUNI 
  Product ID:           OXYGENRAID 416F4
  Revision:             347B
  Serial Num:           0C50BE4DFE511B00
  Unformatted capacity: 715144.000 MBytes
  Write Cache:          Enabled
  Read Cache:           Enabled
    Minimum prefetch:   0x0
    Maximum prefetch:   0xffff
  Device Type:          Disk device
  Path(s):

  /dev/rdsk/c6t600D0230006C1C4C0C50BE4DFE511B00d0s2
  /devices/scsi_vhci/disk@g600d0230006c1c4c0c50be4dfe511b00:c,raw
   Controller           /dev/cfg/c4
    Device Address              220000d0232c50be,1
    Host controller port WWN    2100001b321fbd9c
    Class                       primary
    State                       ONLINE
   Controller           /dev/cfg/c5
    Device Address              210000d0231c50be,1
    Host controller port WWN    2100001b321fe69b
    Class                       primary
    State                       ONLINE

DEVICE PROPERTIES for disk: /dev/rdsk/c6t600D0230006C1C4C0C50BE57396E9F00d0s2
  Vendor:               ADVUNI 
  Product ID:           OXYGENRAID 416F4
  Revision:             347B
  Serial Num:           0C50BE57396E9F00
  Unformatted capacity:  0.000 MBytes
  Write Cache:          Enabled
  Read Cache:           Enabled
    Minimum prefetch:   0x0
    Maximum prefetch:   0xffff
  Device Type:          Disk device
  Path(s):

  /dev/rdsk/c6t600D0230006C1C4C0C50BE57396E9F00d0s2
  /devices/scsi_vhci/disk@g600d0230006c1c4c0c50be57396e9f00:c,raw
   Controller           /dev/cfg/c4
    Device Address              220000d0232c50be,0
    Host controller port WWN    2100001b321fbd9c
    Class                       primary
    State                       ONLINE
   Controller           /dev/cfg/c5
    Device Address              210000d0231c50be,0
    Host controller port WWN    2100001b321fe69b
    Class                       primary
    State                       ONLINE


There are three LUNs here: 0,1 and 2; in Solaris "FC Device Address" syntax they are defined as wwpn,lun.

6. A simple MPxIO overview: vhci_stat

To have a short output of MPxIO SAN status, you may use vhci_stat.
This is a little utility written by myself, available for download here: http://southbrain.com/south/software/vhci-stat.html

# vhci_stat -iz
vhci_stat
1.3.1; Pascal Gienger <pascal@southbrain.com>; http://southbrain.com/south/
Host: atlanta

PATH                                              WWPN,LUN            STATE
------------------------------------------------- ------------------- -----
/scsi_vhci/disk@g600d0230006b66680c50ab5675f91300
  (Vendor = ADVUNI   Model = OXYGENRAID 416F4 Rev. = 347B VolID = 5675F913)
   ZFS: pool name = mail, ZFS Version: 10
   ZFS: size = 3.41T, member type = mirror
    /pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0   220000d0232c50ab,2  ONLINE
    /pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0   210000d0231c50ab,2  ONLINE
/scsi_vhci/disk@g600d0230006b66680c50ab4f92f61000
  (Vendor = ADVUNI   Model = OXYGENRAID 416F4 Rev. = 347B VolID = 4F92F610)
   ZFS: pool name = cyrus, ZFS Version: 10
   ZFS: size = 698.37G, member type = mirror
    /pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0   220000d0232c50ab,1  ONLINE
    /pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0   210000d0231c50ab,1  ONLINE
/scsi_vhci/disk@g600d0230006b66680c50ab0f36adf100
  (Vendor = ADVUNI   Model = OXYGENRAID 416F4 Rev. = 347B VolID = 0F36ADF1)
   ZFS: pool name = mail, ZFS Version: 10
   ZFS: size = 3.41T, member type = mirror
    /pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0   220000d0232c50ab,0  ONLINE
    /pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0   210000d0231c50ab,0  ONLINE
/scsi_vhci/disk@g600d0230006c1c4c0c50be16ff1fe200
  (Vendor = ADVUNI   Model = OXYGENRAID 416F4 Rev. = 347B VolID = 16FF1FE2)
   ZFS: pool name = mail, ZFS Version: 10
   ZFS: size = 3.41T, member type = mirror
    /pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0   210000d0231c50be,2  ONLINE
    /pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0   220000d0232c50be,2  ONLINE
/scsi_vhci/disk@g600d0230006c1c4c0c50be4dfe511b00
  (Vendor = ADVUNI   Model = OXYGENRAID 416F4 Rev. = 347B VolID = 4DFE511B)
   ZFS: pool name = cyrus, ZFS Version: 10
   ZFS: size = 698.37G, member type = mirror
    /pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0   210000d0231c50be,1  ONLINE
    /pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0   220000d0232c50be,1  ONLINE
/scsi_vhci/disk@g600d0230006c1c4c0c50be57396e9f00
  (Vendor = ADVUNI   Model = OXYGENRAID 416F4 Rev. = 347B VolID = 57396E9F)
   ZFS: pool name = mail, ZFS Version: 10
   ZFS: size = 3.41T, member type = mirror
    /pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0   210000d0231c50be,0  ONLINE
    /pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0   220000d0232c50be,0  ONLINE


In this example you see also ZFS filesystem information and SCSI inquiry data.

0 TrackBacks

Listed below are links to blogs that reference this entry: Fiberchannel Solaris Part 3: SAN Diagnostics.

TrackBack URL for this entry: http://southbrain.com/mt/mt-tb.cgi/117

6 Comments

Hi Pascal,

This guide of yours is excellent!! Very simple and to the point. I was greatly benefited by this tutorial. Some of my doubts were cleared. Keep writing such things. God bless you!!!!

Thanks,

Ajay

I have a nagging performance problem on one of my two-server clusters. Both clusters are loaded with the same release of Solaris (10/08), and the same version of Sun Cluster (3.2, don't know the update, but they're the same). All four servers have redundant connections to two Brocade Silkworm 4100 switches. Each pair of servers is zoned to see a 6140 on the back-end. The 6140s are also redundantly connected through the switches. The problem I'm seeing is that on one server cluster I can copy a 150MB file into a SAN-based filesystem in less than 2 secs, and the other takes 20-30 secs. In a comparison of the two clusters, here is what I see:

The FC HBAs are running the same firmware, but on the 6140s the slow side has firmware 6.60.xx, and the other has 7.xx.xx.

In the output from "fcinfo remote-port -p wwpn", I see the WWN of the other server's HBA, and the value for "Active FC4 Type" is different on each server. On node 1 of each cluster, the value is empty, and on node 2 on the slow cluster it is "SCSI,IP", and node 2 on the other cluster it is just "SCSI".

On the slow cluster, the array controllers are seen as devices in Solaris, with separate devices for each path. On the other cluster, the controllers do not show up as devices.

The slow cluster was the one originally setup, and I just thought had been optimistic in my performance expectations. Then I setup the second, and saw the better performance, so I reloaded the first exactly the same as the second with Solaris and Sun Cluster, expecting an improvement which did not appear.

I am now leaning towards firmware updates, but am wondering if what you think, especially about the controller devices. The only other thing that I can think of that would cause a performance problem would be the construction of the volumes and filesystems (UFS on both), but that seems much less likely, especially for this much performance difference (x10). In any case, both clusters use RAID5 hardware volumes concatenated by SVM.

Any help would be greatly appreciated. Let me know if you would like to see any command output. Thanks..
- Shane R.

I know this is an old post but I can't help myself.

There is a big difference between the 6.60 fw and the 7.xx fw on the 6140 arrays. The config you have is not optimal. Uprgrading is where you're going to get the biggest improvement.

I'd start your investigation on this side of the config. For example:
- Remember that the 6140 array controllers are active/passive.
- The cache flush rate can be tuned.
- Check that the volumes have been created from the same storage profile.
- Check that the volumes are of a similar disk configuration. No point mirroring a R5 on FCAL to a R5 on SATA disks etc...
- Check the port settings in the brocade switches.

Cheers

Hi Pascal, Thank you for this very helpful page of yours.

I am not very much familiar with fiberchannels so this page help me a lot.

I am checking right now a server connected to a fiberchannel. I am not sure if this behavior is normal. When I df a filesytem as an ordinary user this is what i get.

$ df -k /phl/root/mnl/data04
df: (/phl/root/mnl/data04) not a block device, directory or mounted resource

...But when i logged in as ROOT. I can view the filesystem.

# df -k /phl/root/mnl/data04
Filesystem size used avail capacity Mounted on
/dev/dsk/c4t60060160F3F72200C7DCE12B093EDE11d0s0
25G 17G 6.9G 72% /phl/root/mnl/data04

# cd /phl/root/mnl/data04
# pwd
/phl/root/mnl/data04

Is this normal?


"
$ df -k /phl/root/mnl/data04
df: (/phl/root/mnl/data04) not a block device, directory or mounted resource
"

You don't have any rights to enter this directory. Look whether you have r and x set for that directory (and all directories in the path: /phl, /phl/root, /phl/root/mnl).

Hi Pascal,
Thanks for the great article. On one of my setup (Solaris 11) I am observing that the fp module does not get automatically loaded on a reboot. Any pointers on what could be the issue?

December 2015

Sun Mon Tue Wed Thu Fri Sat
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    

About

This blog is owned by:

Pascal Gienger
J├Ągerstrasse 77
8406 Winterthur
Switzerland


Google+: Profile
YouTube Channel: pascalgienger