With two RAID devices in different locations onsite you may want to use a ZFS mirror to access them - as well as Sun MPxIO to have redundant links to them.
The relevant configuration is /kernel/drv/scsi_vhci.conf as scsi_vhci is the MPxIO driver for redundant scsi/fc links.
Yes Solaris 10 is quite good but this configuration is like telling your grandma to disguise herself as a man to be able to use trousers:
Using Sun Arrays all works without any problem. The latter do arise when you want to use other devices (non-Sun), which is not really an uncommon practice...
You have to tell scsi_vhci to ACCEPT your device based on the vendor- und product-id which it tells you at SCSI inquiry stage.
In my case, I had to add this to scsi_vhci.conf:
You see the grandma's trousers?
These lines tell the driver that whenever it encounters a "ADVUNI OXYGENRAID" device it should use the value "0x1000000" as an option - which stands for "behaves like a sun fc array and is able to handle redundant links" (I think Sun believes that only Sun devices can do that...).
You notice the TWO SPACES between Vendor and Product-ID? They are part of the game. The scsi_vhci driver expects the vendor and product id as 8 character-words each. You have to fill up with spaces. ADVUNI has only 6 characters, so you have to use two spaces. If your SCSI vendor id has 8 chars, just don't use any space! Otherwise the driver just ignores your devices. Without a warning or error, of course.
To find out SCSI Vendor and Product-ID,just use format(1), select one of the instances of your device (each link to your device shows up as a different entry):
Only use the first part of "Product:", until the first space!
After doing a reboot -- -r
it should work:
My two different(!) hardware devices are using a new virtual controller (c6) to access the multiple links (each c6 device has 4 redundant links to it, without scsi_vhci you would see all these links as different disk instances).
[Update Apr 19th, 2008:] You may check the multiple paths handled by scsi_vhci by using vhci_stat.
Now I found performance was abysmal. I smiled too early. It just would have been too easy until now.
The solution is a technique called "logical block" round-robin.
First round-robin should speed up things as all fiberchannel links are used and after a failing link recovery is faster as you don't have to transfer all outstanding operations to the next functioning link. But it did not work well.
I found that each of the controllers of these raid devices do have their own RAM cache which won't be used efficiently when every subsequent 512-byte-block request is done by the next controller... And yes, scsi_vhci has a solution for this, if you have much luck. This option came in with a patch or by installing at least Solaris 10 8/07. Otherwise it just won't recognize the option:
Complicated? This line tells scsi_vhci to override some default options for device type "ADVUNI OXYGENRAID" (also with 2 spaces, see above!).
What's logical-block round-robin? You define a block size to summarize requests in this block size range to be served by ONE link. region-size=20 means 2^20 bytes = 1 MB. With this "trick" the onboard controller caches do their thing of prefetching and caching right.
The relevant configuration is /kernel/drv/scsi_vhci.conf as scsi_vhci is the MPxIO driver for redundant scsi/fc links.
Yes Solaris 10 is quite good but this configuration is like telling your grandma to disguise herself as a man to be able to use trousers:
Using Sun Arrays all works without any problem. The latter do arise when you want to use other devices (non-Sun), which is not really an uncommon practice...
You have to tell scsi_vhci to ACCEPT your device based on the vendor- und product-id which it tells you at SCSI inquiry stage.
In my case, I had to add this to scsi_vhci.conf:
device-type-scsi-options-list =
"ADVUNI OXYGENRAID", "symmetric-option";
symmetric-option = 0x1000000;
You see the grandma's trousers?
These lines tell the driver that whenever it encounters a "ADVUNI OXYGENRAID" device it should use the value "0x1000000" as an option - which stands for "behaves like a sun fc array and is able to handle redundant links" (I think Sun believes that only Sun devices can do that...).
You notice the TWO SPACES between Vendor and Product-ID? They are part of the game. The scsi_vhci driver expects the vendor and product id as 8 character-words each. You have to fill up with spaces. ADVUNI has only 6 characters, so you have to use two spaces. If your SCSI vendor id has 8 chars, just don't use any space! Otherwise the driver just ignores your devices. Without a warning or error, of course.
To find out SCSI Vendor and Product-ID,just use format(1), select one of the instances of your device (each link to your device shows up as a different entry):
format> inquiry
Vendor: ADVUNI
Product: OXYGENRAID 416F4
Revision: 347B
Only use the first part of "Product:", until the first space!
After doing a reboot -- -r
it should work:
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c3t0d0 <DEFAULT cyl 8872 alt 2 hd 255 sec 63>
/pci@7b,0/pci1022,7458@11/pci1000,3060@2/sd@0,0
1. c6t600D0230006B66680C50AB7821F0E900d0 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
/scsi_vhci/disk@g600d0230006b66680c50ab7821f0e900
2. c6t600D0230006C1C4C0C50BE5BC9D49100d0 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
/scsi_vhci/disk@g600d0230006c1c4c0c50be5bc9d49100
My two different(!) hardware devices are using a new virtual controller (c6) to access the multiple links (each c6 device has 4 redundant links to it, without scsi_vhci you would see all these links as different disk instances).
[Update Apr 19th, 2008:] You may check the multiple paths handled by scsi_vhci by using vhci_stat.
Now I found performance was abysmal. I smiled too early. It just would have been too easy until now.
The solution is a technique called "logical block" round-robin.
First round-robin should speed up things as all fiberchannel links are used and after a failing link recovery is faster as you don't have to transfer all outstanding operations to the next functioning link. But it did not work well.
I found that each of the controllers of these raid devices do have their own RAM cache which won't be used efficiently when every subsequent 512-byte-block request is done by the next controller... And yes, scsi_vhci has a solution for this, if you have much luck. This option came in with a patch or by installing at least Solaris 10 8/07. Otherwise it just won't recognize the option:
device-type-mpxio-options-list =
"device-type=ADVUNI OXYGENRAID", "load-balance-options=logical-block-options";
logical-block-options="load-balance=logical-block", "region-size=20";
Complicated? This line tells scsi_vhci to override some default options for device type "ADVUNI OXYGENRAID" (also with 2 spaces, see above!).
What's logical-block round-robin? You define a block size to summarize requests in this block size range to be served by ONE link. region-size=20 means 2^20 bytes = 1 MB. With this "trick" the onboard controller caches do their thing of prefetching and caching right.
This was very informative comment, because even I use a 3rd party target on which ALUA is disabled, LUN Affinity is disabled and its developed for SCSI 3 command set. Did configure the host which is a SPARC T2000 as per your suggestion but still nothing seems to be working. Also checked few other SUN webpages which said the same stuff. Still I am not seeing devices listed under multipath(i.e 'mpathadm list lu'), it just lists the local devices connected to the SPARC, but I do see the initiator-ports getting listed. I did enter the product-id, vendor-id in the 'scsi_vhci.conf' as per the guidelines, no go. I also went thru the 'fp.conf' and made sure that the MPxIO is enabled. Devices are getting listed in Solaris and not under mpxio. Please share your experience.
Without your scsi_vhci.conf and the output of a SCSI inquiry (format) it is difficult to give a hint...
Did you ever have to run "stmsboot -e"?
Another thing we had to do was add
mpxio-disable="no";
to /kernel/drv/fp.conf
Yes, you're right, mpxio-disable="no" is needed in fp.conf.
And, no, I did not need stmsboot -e, adding the right device name for my 3rd party device in scsi_vhci.conf was sufficient.
This week, I got two Sun StorageTek 6340 arrays and here no change to scsi_vhci.conf was needed.
We are currently struggeling getting a SE6130 properly connected to a T2000 with Solaris10 and for the volume to be properly recognized after reboot. Since this is a asymmetric storage device, MPxIO has to be instructed to have a primary and a secondary path. Unfortunately this seems to be nowhere explained. Not even in the Sun StorEdge documentation.
Any idea if there are other options for this file, other than the symmetric example(s) at Sun of keyword "symmetric-option = 0x1000000;"?
I have not found a list anywhere and am very intrigued to see you using the parameter "device-type-mpxio-options-list =", as I am only aware of the parameter "device-type-scsi-options-list =" in my configurations. Let me know if there is a list out there?
I have a StorageTek 6140 here and scsi_vhci.conf "knows" the ONLINE/STANDBY configuration of this array automatically:
/scsi_vhci/disk@g600a0b800048a692000006404905840a
(Vendor = SUN Model = CSM200_R Rev. = 0710 VolID = )
ZFS: pool name = fc, ZFS Version: 4
ZFS: size = 1.36T, member type = disk
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0 201700a0b848a692,1 STANDBY
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0 201600a0b848a692,1 ONLINE
"device-type-mpxio-options-list =" is used to give special scsi_vhci-Options per Device Identifier. It is used to set the logical-block size for symmetric multipathing in my example.
Do not use "symmetric-option" with your Sun StorageTek Array. It will not work, the vhci-Driver will not use MPxIO for your array in that case.
Feel free to contact me by mail, including your scsi_vhci.conf and the result of an SCSI inquiry of your SE6130 device. Mine (6140) has "SUN" as Vendor and "CRM200_R" as Model Identifier.