Solaris Admin Reference – Solaris Storage Multipathing – MPxIO ( aka STMS, SCSI_VHCI)

Solaris Multipathing - MPxIO - STMS

 What is Oracle Solaris Storage Multipathing MPxIO?

  • MPxIO Provides multipath access to storage luns by grouping together the physical paths and presenting a single logical path.
  • Supports both symmetric and asymmetric storage 
  • Controls failover / failback of LUNs as appropriate due to path failures, either manually or automatically.
  • Provides load balancing policies.
  • Provides support automatically for known storage arrays and ALUA enabled arrays, or can be manually configured to support other arrays.
  • Provides support for multipathed Sun Oracle Fibre Channel, SAS and iSCSi LUNs accessed over Sun Oracle HBAs using Sun Oracle drivers. MPxIO does not work with third party HBAs unless they are using the Sun Oracle driver stack.
  • Bundled with Solaris 10 and Solaris 11

 

Note 1 : Sometimes MPxIO may also be referenced as STMS (Sun StorEdge Traffic Manager Software)  or SCSI_VHCI due to the commands used and logical device paths created.

 

Note  2:   Asymmetric Logical Unit Access (ALUA)  allows you to see any given LUN via both storage processors as active but only one of these storage processors “owns” the LUN and because of that there will be optimized and un-optimized paths. The optimized paths are the ones with a direct path to the storage processor that owns the LUN. The un-optimized paths have a connection with the storage processor that does not own the LUN but have an indirect path to the storage processor that does own it via an interconnect bus.

 

 

How does MPxIO work ?

 

During boot or when new LUNs are presented to the Solaris host, the configuration pages of the LUN are automatically read to determine the LUNs capabilities and Identity.

When MPxIO is enabled using the “stmsboot -e” (see man page for stmsboot) the discovered LUNs are grouped by identity and physical path , then a single logical LUN / logical path is created and presented to Solaris for access via applications. MPxIO will also (based on the capabilities of the LUN) assign failover and load balancing policies to the physical paths associated with the new logical path.

Any I/O then issued to the logical LUN/path will then be automatically routed by MPxIO to one of the physical paths to the device.

If a physical path fails, MPxIO will reroute the I/O to a surviving path. This may involve the failing over of LUNs where the paths are attached to asymmetric storage.

MPxIO will monitor failed paths and restore the path when it is rediscovered.

Note: MPxIO manages paths based on path state and not on the state of commands that are transported on the path. So as long as the transport of a command succeeds, even if the command failed for some non path related issue,  MPxIO will _not_ consider the command failure as an indication of path failure. For example a command that times out is considered a command failure and not a path failure.

Pre-requisites to configure MPxIO for the storage

1) Storage Configuration should  be via either of the following:

a) Fibre Channel storage using a Sun Oracle HBA attached via the Sun Oracle “qlc” or “emlxs” HBA drivers

b) SAS storage using a Sun Oracle HBA attached via the Sun Oracle “mpt” scsi HBA driver

c) iSCSI storage attached via the Sun Oracle iSCSI driver stack

(MPxIO does not support 3rd party HBA drivers)

2) MPxIO must know the storage type, that it is multipath capable and its operational characteristics. This can be defined to MPxIO in one of the following ways.

a) the storage is already known to MPxIO. These will be all Sun Oracle MPXIO capable storage and some selected 3rd party MPxIO capable storage.  To check which storage is already known to MPxIO run

#mpathadm show mpath-support libmpscsi_vhci.so

b) the array is ALUA (asymmetric logical unit access) capable, ALUA is an industry standard method for arrays to advertize their multipathing capabilities.

c) the storage capabilities are defined in the /kernel/drv/scsi_vhci.conf file using the “device-type-scsi-options-list” method. See the /kernel/drv/scsi_vhci.conf file on your version of Solaris for details.

3)  If necessary the storage must be enabled to work with MPxIO.  You will need to check with your storage vendor if any specific settings are required for the storage to function with MPxIO. Note :  these setting vary for vendor to vendor.

Note1. For third party Storage, your main point of contact for configuration details including compatibility with and settings for MPxIO is the Storage Vendor.

Note2. For third party Storage, all qualification is done by the 3rd party and Oracle is not aware of all settings or operational requirements  to make the storage work with MPxIO

How does MPxIO know how to group paths for a LUN ?

 

All ‘scsi target devices’ or ‘arrays emulating scsi target devices’ make available to the scsi initiator pages of data that can be accessed by the initiator using scsi commands. The pages provide information about the target, such as it’s vendor, model, serial number, word wide number, geometry, capacity, caching, ALUA capabilities, etc.

MPxIO, when started or when a new LUN is presented on a path, reads these pages to identify the LUN. From the data read from these pages, MPxIO is able to extract a unique identifier for the LUN.  MPxIO will then compare that LUN’s unique identifier with the unique identifiers of LUNs presented on all  paths. If MPxIO detects more than one LUN presentation with the same unique identifier, it knows that these presentations are in fact multiple paths to the same LUN. It then groups these presentations together.

In particular MPxIO accesses the EVPD  (Extended Vital Product Data) pages to unique identify the LUN, including pages:

00h – Supported vital product data
        80h – Unit serial number
        83h – Device Identification

Note: not all storage devices by default present the required information for MPxIO to unique identify the LUNs. This can occur mostly on devices that simulate scsi hardware LUNs in firmware such as storage raid arrays. For this reason, it is important to contact the array supplier , to make sure if any settings are required and enabled on the array to make it compatible with MPxIO.  In cases where these settings are not enabled, MPxIO may not be able to group the paths for a specific LUN.

How does MPxIO provide Load Balancing ?

 

In addition to providing failover support, MPxIO also provides the following load balancing options to send and receive I/O.

● Round Robin
Selects all paths in a round robin way.
 
● Logical Block
I/Os within the range specified by range_size would go on the same path.Improves performance on some storage arrays which have per path caching of data.
Logical Block segments the LUN into chunks and dedicates the I/O for those chunks to a particular HBA path. Any I/O to a given region will always go down the
same path.
 
● None
No load balancing – only one path is used.

These parameters are set in the /kernel/drv/scsi_vhci.conf file.

#
# Copyright 2004 Sun Microsystems, Inc. All rights reserved.
# Use is subject to license terms.
#
#pragma ident “@(#)scsi_vhci.conf 1.9 04/08/26 SMI”
#
name=”scsi_vhci”;
#
# Load balancing global configuration: setting load-balance=”none” will cause
# all I/O to a given device (which supports multipath I/O) to occur via one
# path. Setting load-balance=”round-robin” will cause each path to the device
# to be used in turn.
#
load-balance=”round-robin”;
#
# Automatic failback configuration
# possible values are auto-failback=”enable” or auto-failback=”disable”
auto-failback=”enable”;
#
# For enabling MPxIO support for 3rd party symmetric device need an
# entry similar to following in this file. Just replace the “SUN SENA”
# part with the Vendor ID/Product ID for the device, exactly as reported by
# Inquiry cmd.
#
# device-type-scsi-options-list =
# “SUN SENA”, “symmetric-option”;
#
# symmetric-option = 0x1000000;
device-type-mpxio-options-list =
“device-type=HITACHI OPEN-V -SUN”,”load-balance-options=logical-block-options”,
“device-type=HITACHI DF600F”, “load-balance-options=logical-block-options”;
logical-block-options=”load-balance=logical-block”, “region-size=16”;

As an example, you can confirm that the logical-block load balancing algorithm is in use by inspecting the output from dmesg for the following:

genunix: [ID 834635 kern.info] /scsi_vhci/ssd@g60060e80102ae5800511a1e0000000c8 (ssd23) multipath status: optimal, path /pci@0/pci@0/pci@8/SUNW,qlc@0/fp@0,0 (fp2) to target address: w50060e80102ae580,0 is online Load balancing: logical-block, region-size: 16

 

How does MPxIO failover works?

 

There are three types of failover.

1. Automatic Failover: Happens when the last Online path fails.

After 90 seconds, the Offline timeout occurs and initiates the automatic failovers, by switching to available Standby paths.

2. Manual Failover: User initiated via luxadm failover or mpathadm failover

3. External Failover Detection: In a multihost environment, need to switch paths when failover initiated by some other host is detected.As long as MPxIO does not see a Fatal path error, it will retry the I/O on the same path.

There is an offline timeout (90 seconds) built into the driver stack.Therefore, if one of the redundant paths is Offline for less than  90 seconds, no failover is initiated by MPxIO. Approximately 20 seconds after the expiration of the 90-second timeout, MPxIO will mark the non-responding path as failed and divert all pending and future I/Os to functional paths.

fp_offline_ticker: When a link goes offline (i.e. when the HBA port loses a valid FC light input), the fp driver starts a timer with the value of fp_offline_ticker. When the timer expires, the driver fp, notifies the fcp driver to offline the devices. The default value for this is 90 seconds.

fcp_offline_delay: The fcp driver delays the offlining by the value set in this timer. If the devices come back within this interval, this timer safeguards the premature offline event. The default value is 20 second.

An External Failover is one which is not initiated by the host directly, but rather by a different host, or by the array itself based on some condition outside the physical connections to this specific host. The host should report this as an externally initiated failover and change it’s online path(s) accordingly:

Nov 14 12:15:34 kvsfs2 scsi: [ID 243001 kern.info] /scsi_vhci (scsi_vhci0):
Nov 14 12:15:34 kvsfs2  /scsi_vhci/ssd@g600a0b80004881da0000063b48d8ee41 (ssd252): path /pci@12,600000/SUNW,qlc@0/fp@0,0 (fp5) target address 201400a0b84881da,0 is now ONLINE because of an externally initiated failover

Nov 14 12:15:34 kvsfs2 scsi: [ID 243001 kern.info] /scsi_vhci (scsi_vhci0):
Nov 14 12:15:34 kvsfs2  /scsi_vhci/ssd@g600a0b80004881da0000063b48d8ee41 (ssd252): path /pci@2,600000/SUNW,qlc@0,1/fp@0,0 (fp13) target address 204500a0b84881da,0 is now STANDBY because of an externally initiated failover

After these messages, I/O to the LUN(s) will continue on the new online path(s).

MPxIO Configuration Files 

For Solaris 10 :

For Solaris 10 and earlier releases of the OS there are four major modifiable configuration files used by MPxIO.

/kernel/drv/fp.conf :      The “Fibre Channel Port driver configuration” file

Purpose :  used to configure MPxIO to work with FC (Fibre Channel) devices, can be used to define MPXIO globally for all FC devices, or on a per port basis.

/kernel/drv/mpt.conf :       The “mpt scsi/SAS HBA controller driver configuration” file

Purpose :   used to configure MPxIO to work with SAS (Serial Attached SCSI) devices, can be used to define MPXIO globally for all SAS devices, or on a per controller basis.

/kernel/drv/scsi_vhci.conf :         The MPXIO operation configuration file

Purpose :  used to define MPXIO load balancing and failback modes of operation , and to configure third party devices not already known to MPXIO or not using ALUA (asymmetric logical unit access)

/kernel/drv/iscsi_conf :        The iSCSI MPxIO configuration file.

Purpose :  Used to configure MPxIO to work with iSCSI devices

For Solaris 11: 

The files in /kernel/drv/ are the default files as delivered with the OS and should not be edited and are not used by mpxio for configuration. For Solaris 11 copies of the default files are stored in/etc/driver/drv. It is the /etc/driver/drv/ directory that configuration changes can be made, are used and maintained by MPXIO and stmsboot. Otherwise the files perform the same tasks, although the file syntax may have changed from Solaris 10 but will be documented in the file.

/etc/driver/drv/fp.conf :      The “Fibre Channel Port driver configuration” file

Purpose : Used to configure MPxIO to work with FC (Fibre Channel) devices, can be used to define MPXIO globally for all FC devices, or on a per port basis.

/etc/driver/drv/mpt.conf :       The “mpt scsi/SAS HBA controller driver configuration” file

Purpose :  Used to configure MPxIO to work with SAS (Serial Attached SCSI) devices, can be used to define MPXIO globally for all SAS devices, or on a per controller basis.

/etc/driver/drv/scsi_vhci.conf :         The MPXIO operation configuration file

Purpose :  Used to define MPXIO load balancing and failback modes of operation , and to configure third party devices not already known to MPXIO or not using ALUA (asymmetric logical unit access)

/etc/driver/drv/iscsi_conf :        The iSCSI MPxIO configuration file.

Purpose :  Used to configure MPxIO to work with iSCSI devices

 

Note: these configuration files contain parameters used for other driver configuration in addition to MPxIO settings and as such should only be modified by qualified personnel. Normally to enable or disable MPxIO these files should not be manually modified but instead the “stmsboot” utility should be used.

 

MPxIO Hidden State Files

The stmsboot command saves a copy of the original  /kernel/drv/fp.conf,  /kernel/drv/mpt.conf, /kernel/drv/iscsi.conf and /etc/vfstab files before modifying them. 

  • /etc/mpxio/fp.conf (For FC multipathing)
  • /etc/mpxio/mpt.conf (For SAS multipathing)
  • /etc/mpxio/mpt_sas.conf (For SAS-2 multipathing)
  • /etc/mpxio/iscsi.conf (For iSCSI multipathing)
  • /etc/mpxio/vfstab

The unmodified file contents of the files which are modified (a “backup” copy) is stored in /etc/mpxio in a filename formed by concatenating the original filename, a timestamp and an indication of whether this was enable or disable.

In addition, the /etc/mpxio/bootenv.rc file is a copy the original /boot/solaris/bootenv.rc file that the stmsboot command saved prior to updating the bootenv.rc file.

A /etc/mpxio/bootenv.rc file will not exist, if the stmsboot command has not modified your bootenv.rc file.

These files can be used to recover, if the system fails to boot during stmsboot or the system crashes during stmsboot.

 

 

Basic  commands used to control MPxIO 

To enable / disable MPxIO use:
 
stmsboot [-D fp|iscsi|mpt] [-e|-d|-u]
 
for example to enable mpxio for all devices:

#stmsboot -e
 
example to enable mpxio for all fibre channel devices:

#stmsboot -D fp -e
 
example to disable mpxio for all devices: #stmsboot -d
 
 
To confirm if MPxIO is enabled for a LUN simply check format if the device has a logical path beginning with “/scsi_vhci” it is under MPXIO control for example:
 
4. c6t60060E800561CF00000061CF00000002d0 <HITACHI-OPEN-V-SUN-6007 cyl 1363 alt 2 hd 15 sec 512>
/scsi_vhci/ssd@g60060e800561cf00000061cf00000002
 
 
To check path status use either of the utilities “mpathadm” or “luxadm”
 
for example using mpathadm:

 
# mpathadm show lu /dev/rdsk/c6t60060E800561CF00000061CF00000002d0s2
Logical Unit: /dev/rdsk/c6t60060E800561CF00000061CF00000002d0s2
mpath-support: libmpscsi_vhci.so
Vendor: HITACHI
Product: OPEN-V -SUN
Revision: 6007
Name Type: unknown type
Name: 60060e800561cf00000061cf00000002
Asymmetric: no
Current Load Balance: round-robin
Logical Unit Group ID: NA
Auto Failback: on
Auto Probing: NA
 
Paths:
Initiator Port Name: 10000000c97ecef2
Target Port Name: 50060e800561cf00
Override Path: NA
Path State: OK
Disabled: no
 
Initiator Port Name: 10000000c9c08634
Target Port Name: 50060e800561cf10
Override Path: NA
Path State: OK
Disabled: no
 
Target Ports:
Name: 50060e800561cf00
Relative ID: 0
 
Name: 50060e800561cf10
Relative ID: 0
 
# mpathadm show initiator-port 10000000c97ecef2
Initiator Port: 10000000c97ecef2
Transport Type: Fibre Channel
OS Device File: /devices/pci@1e,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0
# mpathadm show initiator-port 10000000c9c08634
Initiator Port: 10000000c9c08634
Transport Type: Fibre Channel
OS Device File: /devices/pci@1f,700000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0
 

 
Using luxadm:

 
# luxadm probe
 
Found Fibre Channel device(s):
……
Logical Path:/dev/rdsk/c6t60060E800561CF00000061CF00000002d0s2
Node WWN:50060e800561cf00 Device Type:Disk device
……
 
# luxadm display 50060e800561cf00
 
DEVICE PROPERTIES for disk: 50060e800561cf00
Vendor: HITACHI
Product ID: OPEN-V -SUN
Revision: 6007
Serial Num: 50 061CF0502
Unformatted capacity: 103065.000 MBytes
Write Cache: Enabled
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0x0
Device Type: Disk device
Path(s):
 
/dev/rdsk/c6t60060E800561CF00000061CF00000502d0s2
/devices/scsi_vhci/ssd@g60060e800561cf00000061cf00000502:c,raw
Controller /devices/pci@1e,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0
Device Address 50060e800561cf00,6
Host controller port WWN 10000000c97ecef2
Class primary
State ONLINE
Controller /devices/pci@1f,700000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0
Device Address 50060e800561cf10,6
Host controller port WWN 10000000c9c08634
Class primary
State ONLINE

 

Configuration of MPxIO

 

Configuration 1 :  Enabling MPxIO/STMS for EMC Luns

 

for Solaris 10

Set to the file /kernel/drv/fp.conf the following parameter:

 
mpxio-disable=”no”;

Set the following parameters to the file /kernel/drv/scsi_vhci.conf:
 

load-balance=”round-robin”;
auto-failback=”enable”;
device-type-scsi-options-list=”EMC SYMMETRIX”, “symmetric-option”;
symmetric-option=0x1000000;

 

After modifying the scsi_vhci.conf file, perform a  “boot -r”  for the changes to take effect.

 

for Solaris 11 :

  • Driver’s configuration files is now located /etc/driver/drv/
  • Set to the file /etc/driver/drv/fp.conf the following parameter:

 
It will set the mpxio-disable to “no” in the file :

mpxio-disable=”no”;

no need for more changes, support for EMC SYMMETRIX arrays is hardcoded in the multipathing support library :

 
# mpathadm show mpath-support libmpscsi_vhci.so
 
(…)
 
 
Vendor: EMC
Product: SYMMETRIX
Revision:
Supported Load Balance Types:
round-robin
(…)

Notes:

  • After device-type-scsi-options-list=, there are five spaces between EMC and SYMMETRIX.  
  • The scsi_vhci.conf must not contain tabs or other whitespaces than space itself to separate vendor ID from product ID.
  • Solaris 10 : Ensure that the qlc driver configuration file /kernel/drv/qlc.conf does not contain the global setting: mpxio-disable=”yes”
  • Any Solaris version : MPxIO functionality requires enabling the Symmetrix director C-bitflag
  • Entries that use a “=” sign will work with or without spaces around the “=” sign.

 

Configuration 2:  Enable MPxIO /STMS for Hitachi HDS storage ( HITACHI-DF600F )

For Solaris 10u6 or later

The support is native starting Solaris 10u6 release, therefore no modifications to scsi_vhci.conf are required.
 
Enable multipathing using following command:
 
# stmsboot -e -D fp

 

For Solaris 10u5 only

Support for these array is coming in the following minimum patch revision :

for Sparc platform : SunPatch 138308-02 (obsoleted by 137137-09)
for x86 platform : SunPatch 138309-02 (obsoleted by 137138-09)
No modification to scsi_vhci.conf are required with above patch.

Enable multipathing using following command:

# stmsboot -e

 

For  Solaris 8, Solaris 9 and Solaris 10u4 or earlier

One need to modify scsi_vhci configuration file /kernel/drv/scsi_vhci.conf.
These settings are provided by Hitachi in the HDS AMS host configuration guide:

load-balance=”round-robin”;
autofail-back=”enable”;
device-type-scsi-options-list =
“HITACHI DF600F”, “symmetric-option”;
symmetric-option = 0x1000000;
Enable multipathing using following command:

# stmsboot -e

Configuration 3 : MPxIO configuraiton for EMC Clarion LUNS

Patch requirement to configure MPxIO for EMC Clarion:

For Solaris8 : 111412-18 or higher
For Solaris9 : 113039-10 or higher
For Solaris10 : 120054-01 or higher ( That patch hasnt been released for download, first patch including the fix and available for download is 119715-05 )
For Solarisx86 : 120055-01 or higher ( That patch hasnt been released for download, first patch including the fix and available for download is 119716-05 )
Once the patch is installed, Clariion devices will be automatically discovered under MPXIO after the following settings have been done :

Configuration for Solaris 8 or Solaris 9

Set the following parameters to the file/kernel/drv/scsi_vhci.conf:

mpxio-disable=”no”;
load-balance=”round-robin”;
auto-failback=”enable”;

 

Configuration for Solaris 10

Set to the file /kernel/drv/fp.conf parameter:

mpxio-disable=”no”;

Set the following parameters to the file /kernel/drv/scsi_vhci.conf:

load-balance=”round-robin”;
auto-failback=”enable”;

Ensure that the qlc driver configuration file /kernel/drv/qlc.conf does not contain the global setting:

mpxio-disable=”yes”

After reboot, CX (DGC) luns will be seen correctly :

2. c4t6006016068730D00FE7C46CC1DC6DA11d0 <DGC-DISK-0207 cyl 32766 …>
/scsi_vhci/ssd@g6006016068730d00fe7c46cc1dc6da11
3. c4t6006016068730D00FF7C46CC1DC6DA11d0 <DGC-DISK-0207 cyl 20478 …>
/scsi_vhci/ssd@g6006016068730d00ff7c46cc1dc6da11
4. c4t6006016068730D00007D46CC1DC6DA11d0 <DGC-DISK-0207 cyl 20478 …>
/scsi_vhci/ssd@g6006016068730d00007d46cc1dc6da11
5. c4t6006016068730D00017D46CC1DC6DA11d0 <DGC-DISK-0207 cyl 40940 …>
/scsi_vhci/ssd@g6006016068730d00017d46cc1dc6da11

 

Troubleshooting MPxIO Issues

 

Scenario 1:   System Fails to Boot During stmsboot

During the reboot following stmsboot enable (-e), disable (-d), or update (-u) operations if the system fails to boot but lets you enter single user mode,  perform the following to recover.

a) Type the root password and enter single user mode.

b) Restart the mpxio-upgrade service.

# svcadm restart svc:/platform/sun4u/mpxio-upgrade:default

If it doesn’t work follow the instructions in the next section to recover your original  configuration.

 

 

Scenario 2 :   System Crashes During stmsboot

During the reboot following stmsboot enable (-e), disable (-d), or update (-u) operations, if your system hangs, panics, or fails to boot follow the instructions below to recover your original configuration.

Step a) Boot the system from another disk, CD, DVD, or over the network.

Step b) Run the fsck <your-root-device> command where <your-root-device> is your  original root device.

# fsck <your-root-device>

Step c) Mount your original root device.

# mount <your-root-device> /mnt

Step d) Restore your original fp.conf file by performing one of the following steps:

If you ran stmsboot -e or -d:

# cp /mnt/etc/mpxio/fp.conf /mnt/kernel/drv/fp.conf

The /mnt/etc/mpxio/fp.conf file is a copy of your original fp.conf file that stmsboot command saved prior to updating your fp.conf file.  If you ran the stmsboot -u command, and you modified the fp.conf file prior to running the stmsboot command, undo the changes you made to this file by editing /mnt/kernel/drv/fp.conf file.

Step e) Undo any other changes you made to the multipathing configuration prior to running the stmsboot command.

For example, if you modified the scsi_vhci.conf file, undo the changes you made to this file by editing the /mnt/kernel/drv/scsi_vhci.conf file.

If you modified the device settings of your storage arrays, restore their original settings.

Step f) Restore your original /etc/vfstab file:

# cp /mnt/etc/mpxio/vfstab /mnt/etc/vfstab

The /mnt/etc/mpxio/vfstab file is a copy your original /etc/vfstab file that the stmsboot command saved prior to updating your vfstab file. A /mnt/etc/mpxio/vfstab file will not exist if the stmsboot command has not modified your vfstab file.

Step g) Disable the mpxio-upgrade service:

# /usr/sbin/svccfg -f /mnt/etc/mpxio/svccfg_recover

Step h)Unmount the device.

# umount /mnt

Step i) Reboot.

 

 

 Scenario 3: Verifying MPxIO Health Status during troubleshooting

Step 1 . Validate that MPxIO is enabled :

Verify if the mpxio-disable parameter is set to “no ”

for Solaris 7,8,9

# egrep “^mpxio-disable=” /kernel/drv/*.conf
/kernel/drv/scsi_vhci.conf:mpxio-disable=”no”; <– MPxIO is enabled

for Solaris 10

# egrep “^mpxio-disable=” /kernel/drv/*.conf
/kernel/drv/fp.conf:mpxio-disable=”no”; <– MPxIO is enabled for FC hba
/kernel/drv/iscsi.conf:mpxio-disable=”no”;
/kernel/drv/mpt.conf:mpxio-disable=”yes”; <– MPxIO is explicitely disabled for mpt driver
/kernel/drv/mpt_sas.conf:mpxio-disable=”no”;

for Solaris 11
#egrep “^mpxio-disable=” /etc/driver/drv/*.conf
/etc/driver/drv/fp.conf:mpxio-disable=”no”; <– MPxIO is enabled for FC hba
/etc/driver/drv/mpt.conf:mpxio-disable=”yes”; <– MPxIO is explicitely disabled for mpt driver
/etc/driver/drv/mpt_sas.conf:mpxio-disable=”no”;
For Solaris 10 and 11, MPxIO is enabled by default on X86 platforms. Verify that the scsi_vhci module is loaded

for Solaris 7,8,9,10,11 – X86

# modinfo -c |grep scsi_vhci
21 1 scsi_vhci LOADED/INSTALLED

scsi_vhci is the virtual host controller interconnect driver used to manage path of logical uni, see man page of scsi_vhci(7D)
If you find that the scsi_vhci module is not loaded, then most likely MPxIO is not enabled. Refer to document 1006596.1

Step 2. Validate the logical unit path health

For the Oracle StorageTek 25xx(M2)/6xxx array series

For Solaris 7,8,9,10,11

# luxadm probe
No Network Array enclosures found in /dev/es
 
Found Fibre Channel device(s):
Node WWN:200600a0b82a1612 Device Type:Disk device
Logical Path:/dev/rdsk/c4t600A0B80002A16120000C11D486C9A52d0s2
 
# luxadm display /dev/rdsk/c4t600A0B80002A16120000C11D486C9A52d0s2
DEVICE PROPERTIES for disk: /dev/rdsk/c4t600A0B80002A16120000C11D486C9A52d0s2
Vendor: SUN
Product ID: CSM200_R
Revision: 0760
Serial Num: SG70837220
Unformatted capacity: 10240.000 MBytes
Write Cache: Enabled
Read Cache: Enabled
Minimum prefetch: 0xffff
Maximum prefetch: 0x0
Device Type: Disk device
Path(s):
 
/dev/rdsk/c4t600A0B80002A16120000C11D486C9A52d0s2
/devices/scsi_vhci/ssd@g600a0b80002a16120000c11d486c9a52:c,raw
Controller /devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2/fp@0,0
Device Address 203600a0b82a1612,3
Host controller port WWN 2100001b32074eda
Class secondary <–
State STANDBY <–
Controller /devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2,1/fp@0,0
Device Address 202700a0b82a1612,3
Host controller port WWN 2101001b32274eda
Class primary <–
State ONLINE <–

For Asymmetric array the state is dependent on array behavior. Active state are ONLINE/primary and STANDBY/secondary (MPxIO use only on the optimized path which are of State/Class ONLINE/primary for i/o).

For Symmetric array the active state can only be ONLINE/primary (MPxIO will use all online paths and load ballance across the paths as per the load balancing policy, please see: An introduction to MPxIO Load Balancing (Doc ID 1510767.1) for details of policys available).

For Solaris 10, 11 only

# mpathadm list lu
/dev/rdsk/c4t600A0B80002A170200003B04486CAD6Dd0s2
Total Path Count: 2
Operational Path Count: 2 <–
 
# mpathadm show lu /dev/rdsk/c4t600A0B80002A16120000C11D486C9A52d0s2
Logical Unit: /dev/rdsk/c4t600A0B80002A16120000C11D486C9A52d0s2
mpath-support: libmpscsi_vhci.so
Vendor: SUN
Product: CSM200_R
Revision: 0760
Name Type: unknown type
Name: 600a0b80002a16120000c11d486c9a52
Asymmetric: yes
Current Load Balance: round-robin
Logical Unit Group ID: NA
Auto Failback: on
Auto Probing: NA
 
Paths:
Initiator Port Name: 2100001b32074eda
Target Port Name: 203600a0b82a1612
Override Path: NA
Path State: OK <–
Disabled: no <–
 
Initiator Port Name: 2101001b32274eda
Target Port Name: 202700a0b82a1612
Override Path: NA
Path State: OK <–
Disabled: no <–
 
Target Port Groups:
ID: 3
Explicit Failover: yes
Access State: standby <–
Target Ports:
Name: 203600a0b82a1612
Relative ID: 0
 
ID: 1
Explicit Failover: yes
Access State: active <–
Target Ports:
Name: 202700a0b82a1612
Relative ID: 0

For Asymmetric array the Access State is dependent on array behavior. Active state are Ok/active, Ok/standby, Ok/optimized, and Ok/non optimized (the last two one are seen with Target Port Group Support compliant array).

MPxIO use only on the optimized path which are of State/Class Ok/active or Ok/optimized for i/o.   For Symmetric array there’s not Access State, only indication of Path State .

  • For Solaris 7,8,9 luxadm is the only available command to check path status.
  •  For Solaris 10,11 mpathadm is the other available command to check path status but can also be used to administer path state (force failover, disable/enable path, etc …). 

 

 

 

 

Ramdev

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

5 Responses

  1. Joerg says:

    Very good explanation, contains all aspects of multipathing in Solaris – great job, thank you!

  2. Jason says:

    Great Job :) high 5

  3. Neal Ginsberg says:

    Let’s say the FC path is still up for the primary path but one of the RAID LUNs either disappears or a new RAID LUN is added, does Solaris automatically detect this? It would seem to me that some kind of polling mechanism would be required. While there looks to be something like this “Failback Polling Rate”, I cannot find info on it. If this is not it, then it looks like a manual re-scan of the paths would turn up the missing/added LUN, correct?

  4. Mohan says:

    how do you verify MpxIO working properly ?

  1. September 17, 2015

    […] Read – Understanding Solaris Storage Multipathing MPxIO ( aka STMS, SCSI_VHCI) […]

What is in your mind, about this post ? Leave a Reply

Close
  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us