Fiberchannel Solaris Part 3: SAN Diagnostics
1. Kernel modules loaded?
Check whether the needed kernel modules/drivers are loaded:
The needed modules are printed in bold. The last line just matches because it also contains "fc".
2. Which FC interfaces do I have?
Use "fcinfo":
Result: Two ports available, one port per fiberchannel card. The WWPNs are printed in bold.
There's also the port type (N_port in this case) and the current data bit rate.
3. Which WWPNs (storage devices) are seen from my fiberchannel interface ports?
Use "fcinfo remote-port -p" with the interface port WWPN found in step 2:
"SCSI" types are storage disks (storage volumes).
"SCSI,IP" types are devices capable of talking IP and SCSI (storage). In this case this is a Solaris host with IP configured.
4. Are all disks configured as solaris disks (usable by Solaris)?
Use "cfgadm -al" (this will configure also newly added fc disks):
The relevant lines are printed in bold.
5. Is multipathing working?
Use "luxadm display" with the target WWPN of the storage disk device to list the paths:
There are three LUNs here: 0,1 and 2; in Solaris "FC Device Address" syntax they are defined as wwpn,lun.
6. A simple MPxIO overview: vhci_stat
To have a short output of MPxIO SAN status, you may use vhci_stat.
This is a little utility written by myself, available for download here: http://southbrain.com/south/software/vhci-stat.html
In this example you see also ZFS filesystem information and SCSI inquiry data.
Check whether the needed kernel modules/drivers are loaded:
# modinfo|grep fc
93 ffffffffef17d000 19ac0 87 1 fcp (SunFC FCP v20090210-1.134)
94 ffffffffef196000 a2a0 - 1 fctl (SunFC Transport v20090210-1.59)
157 ffffffffef3e8000 9620 83 1 fcip (SunFC FCIP v20090210-1.50)
160 ffffffffef494850 5610 92 1 fcsm (Sun FC SAN Management v20070116)
166 ffffffffef4b5bd8 1fc0 151 1 ippctl (IP Policy Configuration Driver )
The needed modules are printed in bold. The last line just matches because it also contains "fc".
2. Which FC interfaces do I have?
Use "fcinfo":
# fcinfo hba-port
HBA Port WWN: 2100001b321fbd9c
OS Device Name: /dev/cfg/c4
Manufacturer: QLogic Corp.
Model: 375-3355-01
Firmware Version: 4.04.01
FCode/BIOS Version: BIOS: 1.24; fcode: 1.24; EFI: 1.8;
Serial Number: 0402R00-0725369970
Driver Name: qlc
Driver Version: 20081115-2.29
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 2000001b321fbd9c
HBA Port WWN: 2100001b321fe69b
OS Device Name: /dev/cfg/c5
Manufacturer: QLogic Corp.
Model: 375-3355-01
Firmware Version: 4.04.01
FCode/BIOS Version: BIOS: 1.24; fcode: 1.24; EFI: 1.8;
Serial Number: 0402R00-0725369921
Driver Name: qlc
Driver Version: 20081115-2.29
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 2000001b321fe69b
Result: Two ports available, one port per fiberchannel card. The WWPNs are printed in bold.
There's also the port type (N_port in this case) and the current data bit rate.
3. Which WWPNs (storage devices) are seen from my fiberchannel interface ports?
Use "fcinfo remote-port -p" with the interface port WWPN found in step 2:
# fcinfo remote-port -p 2100001b321fbd9c
Remote Port WWN: 210000e08b0d82c6
Active FC4 Types: SCSI,IP
SCSI Target: no
Node WWN: 200000e08b0d82c6
Remote Port WWN: 2100001b321fb79d
Active FC4 Types: SCSI,IP
SCSI Target: no
Node WWN: 2000001b321fb79d
Remote Port WWN: 210000d0231c50ab
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 200000d0231c50ab
Remote Port WWN: 220000d0232c50be
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 200000d0232c50be
"SCSI" types are storage disks (storage volumes).
"SCSI,IP" types are devices capable of talking IP and SCSI (storage). In this case this is a Solaris host with IP configured.
4. Are all disks configured as solaris disks (usable by Solaris)?
Use "cfgadm -al" (this will configure also newly added fc disks):
# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c3 scsi-bus connected configured unknown
c3::dsk/c3t0d0 disk connected configured unknown
c4 fc-fabric connected configured unknown
c4::2100001b321fb79d unknown connected unconfigured unknown
c4::210000d0231c50ab disk connected configured unknown
c4::210000e08b0d82c6 unknown connected unconfigured unknown
c4::220000d0232c50be disk connected configured unknown
c5 fc-fabric connected configured unknown
c5::2100001b321f9eb3 unknown connected unconfigured unknown
c5::210000d0231c50be disk connected configured unknown
c5::220000d0232c50ab disk connected configured unknown
usb0/1 unknown empty unconfigured ok
usb0/2 unknown empty unconfigured ok
usb0/3 usb-device connected configured ok
usb0/4 usb-storage connected configured ok
usb0/5 usb-floppy connected configured ok
usb0/6 unknown empty unconfigured ok
usb0/7 unknown empty unconfigured ok
usb1/1 unknown empty unconfigured ok
usb1/2 unknown empty unconfigured ok
usb1/3 unknown empty unconfigured ok
usb1/4 unknown empty unconfigured ok
usb1/5 unknown empty unconfigured ok
usb1/6 usb-hub connected configured ok
usb1/6.1 unknown empty unconfigured ok
usb1/6.2 unknown empty unconfigured ok
usb1/6.3 unknown empty unconfigured ok
usb1/6.4 unknown empty unconfigured ok
usb1/7 unknown empty unconfigured ok
The relevant lines are printed in bold.
5. Is multipathing working?
Use "luxadm display" with the target WWPN of the storage disk device to list the paths:
# luxadm display 220000d0232c50be
DEVICE PROPERTIES for disk: /dev/rdsk/c6t600D0230006C1C4C0C50BE16FF1FE200d0s2
Vendor: ADVUNI
Product ID: OXYGENRAID 416F4
Revision: 347B
Serial Num: 0C50BE16FF1FE200
Unformatted capacity: 0.000 MBytes
Write Cache: Enabled
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0xffff
Device Type: Disk device
Path(s):
/dev/rdsk/c6t600D0230006C1C4C0C50BE16FF1FE200d0s2
/devices/scsi_vhci/disk@g600d0230006c1c4c0c50be16ff1fe200:c,raw
Controller /dev/cfg/c4
Device Address 220000d0232c50be,2
Host controller port WWN 2100001b321fbd9c
Class primary
State ONLINE
Controller /dev/cfg/c5
Device Address 210000d0231c50be,2
Host controller port WWN 2100001b321fe69b
Class primary
State ONLINE
DEVICE PROPERTIES for disk: /dev/rdsk/c6t600D0230006C1C4C0C50BE4DFE511B00d0s2
Vendor: ADVUNI
Product ID: OXYGENRAID 416F4
Revision: 347B
Serial Num: 0C50BE4DFE511B00
Unformatted capacity: 715144.000 MBytes
Write Cache: Enabled
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0xffff
Device Type: Disk device
Path(s):
/dev/rdsk/c6t600D0230006C1C4C0C50BE4DFE511B00d0s2
/devices/scsi_vhci/disk@g600d0230006c1c4c0c50be4dfe511b00:c,raw
Controller /dev/cfg/c4
Device Address 220000d0232c50be,1
Host controller port WWN 2100001b321fbd9c
Class primary
State ONLINE
Controller /dev/cfg/c5
Device Address 210000d0231c50be,1
Host controller port WWN 2100001b321fe69b
Class primary
State ONLINE
DEVICE PROPERTIES for disk: /dev/rdsk/c6t600D0230006C1C4C0C50BE57396E9F00d0s2
Vendor: ADVUNI
Product ID: OXYGENRAID 416F4
Revision: 347B
Serial Num: 0C50BE57396E9F00
Unformatted capacity: 0.000 MBytes
Write Cache: Enabled
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0xffff
Device Type: Disk device
Path(s):
/dev/rdsk/c6t600D0230006C1C4C0C50BE57396E9F00d0s2
/devices/scsi_vhci/disk@g600d0230006c1c4c0c50be57396e9f00:c,raw
Controller /dev/cfg/c4
Device Address 220000d0232c50be,0
Host controller port WWN 2100001b321fbd9c
Class primary
State ONLINE
Controller /dev/cfg/c5
Device Address 210000d0231c50be,0
Host controller port WWN 2100001b321fe69b
Class primary
State ONLINE
There are three LUNs here: 0,1 and 2; in Solaris "FC Device Address" syntax they are defined as wwpn,lun.
6. A simple MPxIO overview: vhci_stat
To have a short output of MPxIO SAN status, you may use vhci_stat.
This is a little utility written by myself, available for download here: http://southbrain.com/south/software/vhci-stat.html
# vhci_stat -iz
vhci_stat
1.3.1; Pascal Gienger <pascal@southbrain.com>; http://southbrain.com/south/
Host: atlanta
PATH WWPN,LUN STATE
------------------------------------------------- ------------------- -----
/scsi_vhci/disk@g600d0230006b66680c50ab5675f91300
(Vendor = ADVUNI Model = OXYGENRAID 416F4 Rev. = 347B VolID = 5675F913)
ZFS: pool name = mail, ZFS Version: 10
ZFS: size = 3.41T, member type = mirror
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0 220000d0232c50ab,2 ONLINE
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0 210000d0231c50ab,2 ONLINE
/scsi_vhci/disk@g600d0230006b66680c50ab4f92f61000
(Vendor = ADVUNI Model = OXYGENRAID 416F4 Rev. = 347B VolID = 4F92F610)
ZFS: pool name = cyrus, ZFS Version: 10
ZFS: size = 698.37G, member type = mirror
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0 220000d0232c50ab,1 ONLINE
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0 210000d0231c50ab,1 ONLINE
/scsi_vhci/disk@g600d0230006b66680c50ab0f36adf100
(Vendor = ADVUNI Model = OXYGENRAID 416F4 Rev. = 347B VolID = 0F36ADF1)
ZFS: pool name = mail, ZFS Version: 10
ZFS: size = 3.41T, member type = mirror
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0 220000d0232c50ab,0 ONLINE
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0 210000d0231c50ab,0 ONLINE
/scsi_vhci/disk@g600d0230006c1c4c0c50be16ff1fe200
(Vendor = ADVUNI Model = OXYGENRAID 416F4 Rev. = 347B VolID = 16FF1FE2)
ZFS: pool name = mail, ZFS Version: 10
ZFS: size = 3.41T, member type = mirror
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0 210000d0231c50be,2 ONLINE
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0 220000d0232c50be,2 ONLINE
/scsi_vhci/disk@g600d0230006c1c4c0c50be4dfe511b00
(Vendor = ADVUNI Model = OXYGENRAID 416F4 Rev. = 347B VolID = 4DFE511B)
ZFS: pool name = cyrus, ZFS Version: 10
ZFS: size = 698.37G, member type = mirror
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0 210000d0231c50be,1 ONLINE
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0 220000d0232c50be,1 ONLINE
/scsi_vhci/disk@g600d0230006c1c4c0c50be57396e9f00
(Vendor = ADVUNI Model = OXYGENRAID 416F4 Rev. = 347B VolID = 57396E9F)
ZFS: pool name = mail, ZFS Version: 10
ZFS: size = 3.41T, member type = mirror
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0 210000d0231c50be,0 ONLINE
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0 220000d0232c50be,0 ONLINE
In this example you see also ZFS filesystem information and SCSI inquiry data.
0 TrackBacks
Listed below are links to blogs that reference this entry: Fiberchannel Solaris Part 3: SAN Diagnostics.
TrackBack URL for this entry: http://southbrain.com/mt/mt-tb.cgi/117
Hi Pascal,
This guide of yours is excellent!! Very simple and to the point. I was greatly benefited by this tutorial. Some of my doubts were cleared. Keep writing such things. God bless you!!!!
Thanks,
Ajay
I have a nagging performance problem on one of my two-server clusters. Both clusters are loaded with the same release of Solaris (10/08), and the same version of Sun Cluster (3.2, don't know the update, but they're the same). All four servers have redundant connections to two Brocade Silkworm 4100 switches. Each pair of servers is zoned to see a 6140 on the back-end. The 6140s are also redundantly connected through the switches. The problem I'm seeing is that on one server cluster I can copy a 150MB file into a SAN-based filesystem in less than 2 secs, and the other takes 20-30 secs. In a comparison of the two clusters, here is what I see:
The FC HBAs are running the same firmware, but on the 6140s the slow side has firmware 6.60.xx, and the other has 7.xx.xx.
In the output from "fcinfo remote-port -p wwpn", I see the WWN of the other server's HBA, and the value for "Active FC4 Type" is different on each server. On node 1 of each cluster, the value is empty, and on node 2 on the slow cluster it is "SCSI,IP", and node 2 on the other cluster it is just "SCSI".
On the slow cluster, the array controllers are seen as devices in Solaris, with separate devices for each path. On the other cluster, the controllers do not show up as devices.
The slow cluster was the one originally setup, and I just thought had been optimistic in my performance expectations. Then I setup the second, and saw the better performance, so I reloaded the first exactly the same as the second with Solaris and Sun Cluster, expecting an improvement which did not appear.
I am now leaning towards firmware updates, but am wondering if what you think, especially about the controller devices. The only other thing that I can think of that would cause a performance problem would be the construction of the volumes and filesystems (UFS on both), but that seems much less likely, especially for this much performance difference (x10). In any case, both clusters use RAID5 hardware volumes concatenated by SVM.
Any help would be greatly appreciated. Let me know if you would like to see any command output. Thanks..
- Shane R.