SAN (Storage area Networking) for System Administrators

 In conventional IT systems, storage devices are connected to servers by means of SCSI cables. The idea behind storage networks is that these SCSI cables are replaced by a network, which is installed in addition to the existing LAN.  Server and storage devices can exchange data over this new network using the SCSI protocol

SAN Components

 

1. Server Hardware – The actual machines which are configured to use the data from the storage devices. The major vendors for Server hardware are ORACLE ( formerly SUN Microsystems) , IBM, HP, Fujitsu ..etc.

2. Storage Hardware – The actual Disk arrays from smaller size to enterprise size. The major Vendors for Storage Hardware are EMC, Oracle (formerly knows as SUN Microsystems), IBM, HP , DELL …etc.

3. HBA – Host Bus adapters acts as initiator in SCSI storage subsystem and main function is converting SCSI commands into Fiber Channel format and establish connection from server side to  the SAN. You can compare HBA cards with Ethernet adapters in our regular networking. Major vendors are SUN, Emulex, Qlogic, JNI ..etc.

4. SAN Switches – These are similar to Ethernet switches , but do switching for fiber networks. Major vendors are  Brocade, Cisco ….etc.

 

 

SAN Topology

In this exercise, we will be looking at various phases of SAN configuration and practically looking  at the configuration of  two Solaris machines with Emulex HBAs  connected to a EMC Clarion Storage.

From the diagram  you can notice  two Solaris servers named “gurkul1 and gurkul2”, two SAN switches named “SN1 and SN2” and EMC device with two Storage arrays “array1 and array2”.

The SAN Switch and the Storage array together comprises a fabric in the SAN.

 

 

 

 

PORTS

 

In regular networking we do have ports in Ethernet adapters to interconnect the networking devices using network cables, in the similar way in Storage Area Networking we do have different types of ports which are used to interconnect Servers , Switches and Storage. Below are the common type of ports that we commonly discuss while talking about SAN.

  • ‘N’ port: Node ports used for connecting peripheral storage devices to switch fabric or for point to point configurations
  • ‘F’ port: Fabric ports reside on switches and allow connection of storage peripherals (‘N’ port devices)
  • ‘E’ port: Expansion ports are essentially trunk ports used to connect two Fibre Channel switches
  • ‘G’ port: A generic port capable of operating as either an ‘E’ or ‘F’ port. Its also capable of acting in an ‘L’ port capacity— known as a ‘G’ port
  • ‘L’ port: Loop ports are used in arbitrated loop configurations to build storage peripheral networks without FC switches. These ports often also have ‘N’ port capabilities and are called ‘NL’ ports

 

Important point to remember here is End to end connection managed by N ports,  and  Switching and addressing handled by fabric Ports.

 

 

SAN Device Identification

 

 

World-wide name (WWN):  Similar to an Ethernet address in the traditional networking world, this is a unique name assigned by the manufacturer to the HBA. This name is then used to grant and restrict access to other components of the SAN. There are two basic forms of WWN.

1. World Wide Node Name (WWNN)—Assigned to Fibre Channel node device by vendor

2. World Wide Port Name (WWPN) —Assigned to Fibre Channel Host BusAdapter port by vendor

Normally that the HBA vendor will provide a tool that allows the system administrator to query the wwn of the HBA (e.g. Emulex supplies the lputil application). The world-wide name of the symmetrix fibre adapters (FA) can be obtained from EMC, and inaddition to the WWPN for EMC we should also note the SCSI target ID assigned for specific FA in order to make configuration in Solaris Side.

 

For host bus adapters, the world wide name is typically displayed in the /var/adm/messages file after the fibre card and software driver have been installed. An example of for an Emulex PCI fibre HBA is as follows. In this case, the relevant value the WWPN, the World-Wide Port Name.

May 18 09:26:28 gurkul1 lpfc: [ID 242157 kern.info] NOTICE:

lpfc0:031:Link Up Event received Data: 1 1 0 0
May 18 09:26:31 gurkul1 lpfc: [ID 129691 kern.notice] NOTICE:
lpfc0: Firmware Rev 3.20 (D2D3.20X3)
May 18 09:26:31 gurkul1 lpfc: [ID 664688 kern.notice] NOTICE:
lpfc0: WWPN:11:11:11:11:11:11:11:01
WWNN:20:22:22:22:22:22:22:22 DID 0x210913

May 18 09:26:31 gurkul1 lpfc: [ID 494464 kern.info] NOTICE:
lpfc1:031:Link Up Event received Data: 1 1 0 0
May 18 09:26:34 gurkul1 lpfc: [ID 129691 kern.notice] NOTICE:
lpfc1: Firmware Rev 3.20 (D2D3.20X3)
May 18 09:26:34 gurkul1 lpfc: [ID 664688 kern.notice] NOTICE:
lpfc1: WWPN:11:11:1111:11:11:11:02
WWNN:20:22:22:22:22:22:22:02 DID 0x210913

For the purpose of our configuration example I am adding the list of WWPNs numbers that we are going to use here ( we should collect this information before configuring the Server/Storage)

 

Host name port SCSI target World Wide Name
Gurkul1 Hba1   1111111111111101
Gurkul1 Hba2   1111111111111102
Gurkul2 Hba1   2121212121212101
Gurkul2 Hba2   2121212121212102
EMC Fa1a target 11 555555555555551a
EMC Fa1b target 12 555555555555551b
EMC FA2a target 21 555555555555552a
EMC Fa2b target 22 555555555555552b

 

Gurkulindia.com  

 

SAN Connections

 

The below diagram illustrates several principals common to most san topologies:

  • There are two separate switched (“SN1” and “SN2”). Each switch therefore comprises an independent “fabric”.
  • The array provides multiple I/O paths to every disk device. In the diagram beside, for example, the disk device “Disk01” is accessible via both FA1A and FA1B. Depending on the array, the two paths may be simultaneously active (active/active) or only one path may be valid at a time (active/passive).
  • Each host has a connection into each fabric. Host-based software load-balances the traffic between the two links if the array supports active/active disk access. Furthermore, the host should adjust to a single link failure by rerouting all traffic along the surviving path.

 

 

Storage PATH for Gurkul1 Server

 

Take a moment to examine the diagram below and consider the host “gurkul1”.  Assuming that it requires highly-available access to disk Disk02 in Array2 , note that there are two separate paths to that device.

Path1 : HBA1–> SN1 (P1) –> SN1 (P5) –> FA1B –> Array2 –> Disk02

Path2 : HBA2–> SN2 (P1) –> SN2 (P5) –> FA2B –> Array2 –> Disk02

Storage PATH for Gurkul2 Server

 

Similar way to Gurkul1 now we can examine the diagram beside to identify the storage paths available for  “gurkul2”.  Assuming that it requires highly-available access to disk Disk01 in Array1 . And again here we have total separate paths to that device.

Path1 : HBA1 –> SN1 (P3) –> SN1 (P4) –> FA2A –> Array1 –> Disk01

Path2 : HBA2 –> SN2 (P3) –> SN2 (P4) –> FA1A –> Array1 –> Disk01

 

Target configuration that we want to configure

  • For gurkul1: need access to the LUN02 (Disk02 ) to LUN05 (Disk05) from the storage array2, through FA1B and FA2B
  • For gurkul2: need access to the LUN01 (Disk01) to LUN04( Disk04) from the storage array1, through FA1A and FA2A
  • And both servers require access to disk00 – lun00 on each adapter they are connected

 

Solaris 8/9  host side configuration

 

Typically, there are two configuration files that need to be updated once the vendor’s hba software has been installed. The hba driver’s configuration file typically resides in the /kernel/drv directory, and must be updated to support persistent binding and any other configuration requirements specified by the array vendor. Secondly, the Solaris “sd” driver configuration file sd.conf must be updated to tell the operating system to scan for more than the default list of scsi disk devices. The examples below describe the process for configuring Emulex cards in to support an EMC Symmetrix array.

1. Configuring /kernel/drv/lpfc.conf

A. To configure Gurkul1 Server with below storage paths

Path1  : HBA1 (lpfc0) –> SN1 (P1) –> SN1 (P5) –> FA1B  ( target 12 )–> Array2 –> Disk02
Path2 : HBA2 (lpfc1) –> SN2 (P1) –> SN2 (P5) –> FA2B ( target 22)  –> Array2 –> Disk02

We Have to update below entries in /kernel/drv/lpfc.conf

fcp-bind-WWPN=”555555555555551B:lpfc0t12″,
“555555555555552B:lpfc1t22”;

B.To configure Gurkul2 Server with below storage paths

Path1 : HBA1(lpfc0) –> SN1 (P3) –> SN1 (P4) –> FA2A ( target 11) –> Array1 –> Disk01
Path2 : HBA2 (lpfc1) –> SN2 (P3) –> SN2 (P4) –> FA1A (target 21) –> Array1 –> Disk01

We have to update below entries in /kernel/drv/lpfc.conf

fcp-bind-WWPN=”555555555555551A:lpfc0t11″,
“555555555555552A:lpfc1t21”;

2. Configuring /kernel/drv/sd.conf

By default, the Solaris server will scan for a limited number of scsi devices. The administrator has to update the /kernel/drv/sd.conf file to tell the sd driver to scan for a broader range of scsi devices. In both cases, the target number associated with the WWPN of the fiber array adapter is arbitrary. In our case, we’ve assigned scsi targets 11, 12, 21, and 22 to the four array adapters. The following list describes the additions to the /kernel/drv/sd.conf file for each of the three hosts:

A. Gurkul1 Server:

# Entries added for host gurkul1 to “see” lun ( 0,2,3,4,5 )  on FA1B and FA2B with target 12 and 22
# FA1B = 555555555555551B
name=”sd” target=12 lun=0 hba=”lpfc0″ wwn=”555555555555551B”;
name=”sd” target=12 lun=2 hba=”lpfc0″ wwn=”555555555555551B”;
name=”sd” target=12 lun=3 hba=”lpfc0″ wwn=”555555555555551B”;
name=”sd” target=12 lun=4 hba=”lpfc0″ wwn=”555555555555551B”;
name=”sd” target=12 lun=5 hba=”lpfc0″ wwn=”555555555555551B”;

# FA2Bb = 555555555555552B
name=”sd” target=22 lun=0 hba=”lpfc1″ wwn=”555555555555552B”;
name=”sd” target=22 lun=2 hba=”lpfc1″ wwn=”555555555555552B”;
name=”sd” target=22 lun=3 hba=”lpfc1″ wwn=”555555555555552B”;
name=”sd” target=22 lun=4 hba=”lpfc1″ wwn=”555555555555552B”;
name=”sd” target=22 lun=5 hba=”lpfc1″ wwn=”555555555555552B”;

B. Gurkul2 Server:

# Entries added for host gurkul1 to “see” lun(0,1,2,3,4)  on FA1A and FA2A with target 11 and 21
# FA1B = 555555555555551A
name=”sd” target=11 lun=0 hba=”lpfc0″ wwn=”555555555555551A”;
name=”sd” target=11 lun=1 hba=”lpfc0″ wwn=”555555555555551A”;
name=”sd” target=11 lun=2 hba=”lpfc0″ wwn=”555555555555551A”;
name=”sd” target=11 lun=3 hba=”lpfc0″ wwn=”555555555555551A”;
name=”sd” target=11 lun=4 hba=”lpfc0″ wwn=”555555555555551A”;

# FA2Bb = 555555555555552A
name=”sd” target=21 lun=0 hba=”lpfc1″ wwn=”555555555555552A”;
name=”sd” target=21 lun=1 hba=”lpfc1″ wwn=”555555555555552A”;
name=”sd” target=21 lun=2 hba=”lpfc1″ wwn=”555555555555552A”;
name=”sd” target=21 lun=3 hba=”lpfc1″ wwn=”555555555555552A”;
name=”sd” target=21 lun=4 hba=”lpfc1″ wwn=”555555555555552A”;

 

3. Update the /etc/system file as per EMC’s requirements for the Symmetrix.  

 

set sd:sd_max_throttle=20
set scsi_options = 0x7F8

* Powerpath? Enter
* No sd:sd_io_time=0x78
* Yes sd:sd_io_time=0x3C
set sd:sd_io_time=0x3C

4. Perform the Final Reconfiguration Reboot

Perform a reconfiguration reboot (e.g. “reboot — -r”) on all three servers.
And after the reboot see the outout for

#format

You should see the desired disks. Put a Sun label on them via the “format” command and the configuration is complete.

 

 

Ramdev

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

39 Responses

  1. Anil says:

    Good Article Ram, helps sys admins to understand the SAN env.

  2. admin admin says:

    @Anil, Thank you … Welcome to Gurkulindia

  3. nik says:

    good article, thanks!

  4. gkout says:

    Why would you need to explicitly define the LUNs you need to access in the solaris respective conf files, while you can do that on the fabric level by san switch zoning and EMC switch LUN presentation to specific hosts.

  5. Gurkulindia Gurkulindia says:

    @gkout – in solaris 10 you wont require any local configuration, it can dynamically discover, build the device information and can make the LUNs visible to OS as local disks. But prior versions required this manual configuration to map the LUN devices into Solaris understandable Disk devices.

  6. Vinu says:

    hi guru ..Nice article

    . we have emc cx20 and we need to upgrade storage space . how we can add new luns in solaris 10 that alredy configured .

  7. sonu says:

    Good one

  8. Kombuter says:

    Hi , 

    please correct me if i am not mistaken by adding the devices this will create a single path to the disk or each disk will have two target path in the /dev directory also format command ? 

    please clearify . 

  9. Ramdev Ramdev says:

    @ Kombuter – if you have two hba adapters to your server, which is connecting to storage array with two separate fiber cables . For each disk assigned to your server, you will see two device names. That will be visible in both /dev and format output. ….

    .for example if you assign two disks using two paths .. you will see them from one path as — c3t1d0 & c3t2d0 , from the other path it will be like c4t1d0 & c4t2d0. …

    by having multipath installed in your machine, you can configure ( like labeling & formatting operations) both the instances related to single disk using multipath device names e.g.

    emcpower1a for c3t1d0 & c4t1d0
    emcpower2a for c3t2d0 & c44t2d0

  10. Mushtaq says:

    Hi, 

    as per my understanding of this article the target are the for N PORTs on the Storage device where we have WWPN. 

    as far as making persistent binding in the lpfc.conf file and configuring sd.conf file , i have a question .

    what does the target means since above diagrams doesnt define that ?  

    this article states ” the target number associated with the WWPN of the fiber array adapter is arbitrary” 

    to reformat my question as an SA where can i find this information that the LUN is located on so and so target ? 

    i think you got it i am confuse what and how to retrieve this target :) 

    thanks for the help in advance.

  11. Mushtaq says:

    to further confirm my above statement  please advise how we as SA will get these targets

    name=”sd” parent=”lpfc” target=40 lun=29;
    name=”sd” parent=”lpfc” target=41 lun=29;
    name=”sd” parent=”lpfc” target=42 lun=29;
    name=”sd” parent=”lpfc” target=43 lun=29;
    name=”sd” parent=”lpfc” target=44 lun=29;
    name=”sd” parent=”lpfc” target=45 lun=29;
    name=”sd” parent=”lpfc” target=46 lun=29;
    name=”sd” parent=”lpfc” target=47 lun=29;

    thanks again.

  12. Ramdev Ramdev says:

    Hi Musthaq, for Solaris 10 and RHEL , we don’t need to do any of the information about targets, since it is all dynamic. what we just need is lun numbers to cross the whether the detected luns matching with the allocated luns.

    and for solaris 8 , normally storage team will tell you what is the target ID for the luns allocated for us.

  13. Mushtaq says:

    Thanks for the clarification ramdev , i was looking for 8/9 where we have to edit the sd.conf (disk) , st.conf (tape)

    one more confirmation i would require from you if possible.

    One lun can have multiple target and also one lun can have one target (both possible) ?

    lets suppose if LUN ID 40 have one target given by the SAN admin is target 23 , so what will be the syntax to write in sd.conf

    if one LUN ID 50 have 8 target given by the SAN admin target 10-18 how would that be . 

    finally are these both scenarios applicable in (talking about solaris 8/9) 

  14. Ramdev Ramdev says:

    @Musthaq – Please find the answers below.

    your question : One lun can have multiple target and also one lun can have one target (both possible) ?

    Answer : Technically yes,one example is when you have one lun allocated from an array which connected using the dual path e.g. EMC clarion disk. And you have second lun from different array which has only one path e.g. a scsi based storage array. And you want to configure them both in your server

    And the simple way to understand this is : No.of targets you have for one lun = no.of.paths you have for the lun

    in first case – lun from EMC has dual path, connected using emulex hba. E.g. SAN team told to use you LUN ID 50 and the two targets from two paths are10 and 18
    sd.conf entries will be :
    ==================================
    name=”sd” class=”lpfc” target=10 lun=50;
    name=”sd” class=”lpfc” target=18 lun=50;
    ===============================

    for the second lun which is coming from scsi storage array with lun id 40 and target 23, you will make entries to the sd.conf as below
    ========================================
    name=”sd” class=”scsi” target=23 lun=40
    ========================================

    Hope this clarifies.

  15. Yogesh Raheja says:

    @Ram, very well explained. @Musthaq, One lun can have multiple target and also one lun can have one target (both possible) ?–> The ans is Yes but you wont see one lun one target configuration in bigger environments. Clients usually prefer one Lun many target configuration. Bzu one Bug has been indentfied in sol 8 or 9 I dont remember the version where in after performing storage migrations targets got mismatched in one to one configuration at that time EMC recommended one to many configuration in sd.conf to aviod any of the undetermined outage for SAN. So the issues are expected in one to one target configuration.

  16. Mushtaq says:

    Thanks a lot @Ram and also @Yogesh , i will try on that . thanks for the help.
     

  17. shekar says:

    Ram /Yogesh

    How can i track and check the link between HBA and SWITCH because i’m getting packet transmission error occured in the hba driver .I was told by oralce that the latest patch to be applied .is there any way i can confirm to check the connectivity between hba and san switch ( p1,p2,p3) , I think beyond this point Solaris admin can’t access .Pl advise .

  18. Mushtaq says:

    shekar , we can find this information from the lputil in case of Emulex card , depend on the card and the solaris version , just a tip , for sure Ramdev/Yogesh can answer better :)

  19. Mushtaq says:

    Hi Ramdev/Yogesh/

    Can you please explain this file and how to get this information , i am confuse about target and lun numbers , thanks.

    name=”sd” parent=”lpfc” target=0 lun=0;
    name=”sd” parent=”lpfc” target=1 lun=0;
    name=”sd” parent=”lpfc” target=2 lun=0;
    name=”sd” parent=”lpfc” target=3 lun=0;
    name=”sd” parent=”lpfc” target=4 lun=0;
    name=”sd” parent=”lpfc” target=5 lun=0;
    name=”sd” parent=”lpfc” target=6 lun=0;
    name=”sd” parent=”lpfc” target=7 lun=0;
    name=”sd” parent=”lpfc” target=8 lun=0;
    name=”sd” parent=”lpfc” target=9 lun=0;
    name=”sd” parent=”lpfc” target=10 lun=0;
    name=”sd” parent=”lpfc” target=11 lun=0;
    name=”sd” parent=”lpfc” target=12 lun=0;
    name=”sd” parent=”lpfc” target=13 lun=0;
    name=”sd” parent=”lpfc” target=14 lun=0;
    name=”sd” parent=”lpfc” target=15 lun=0;
    name=”sd” parent=”lpfc” target=16 lun=0;
    name=”sd” parent=”lpfc” target=17 lun=0;
    name=”sd” parent=”lpfc” target=17 lun=1;
    name=”sd” parent=”lpfc” target=17 lun=2;
    name=”sd” parent=”lpfc” target=17 lun=3;

  20. Mushtaq says:

    sorry forgot to mention it is sd.conf file. Please advise , thanks in advance

  21. Ajay Sahu says:

    I want to know how to set account auto-logout after ideal period of an user in solaris.please revert friends as this is a audit issue in my envirnoment…

  22. Mushtaq says:

    From the sshd_config manpage

    ClientAliveInterval
    Sets a timeout interval in seconds after which if no data has
    been received from the client, sshd will send a message through
    the encrypted channel to request a response from the client.
    The default is 0, indicating that these messages will not be
    sent to the client. This option applies to protocol version 2
    only.

    In your sshd config, add the line

    ClientAliveInterval 1200 #in seconds
    Protocol 2 #Doesn’t work on protocol 1

    Then restart sshd and you are off and running. 

  23. prajwla says:

    Thanks Ram….

  24. Mushtaq says:

    Hi Ramdev , 

    one new Question pop’s in my mind may be it can describe you what i am confuse at 

    What is a target in sd.conf file ? are we referring to Port Number on Fabric Ex : FA1B FA2A or  P1, P2 , P3 on SAN switch or Is it something else ? 

    anybody :? 

    thanks in advance.

  25. shekar says:

    Dynamic Reconfiguration of SAN disks from Solaris 10, without Solaris reconfiguration reboot:

    # cfgadm -al ( verification of FC conroller connections )
    # devfsadm -c disk -v (rescanning disks )
    # echo|format ( verifiy disks from OS)

    The above cmd is for Solaris .

    The same way how can i check in LinUX RHEL 4.x server ,if the new LUN is allocated to the host and how to ensure how many paths are avalaiable.

    what is equalent cmd in linux for the below .

    fcinof hba-port
    prtconf -vp | grep -i wwn
    luxadm -e port

  26. Mushtaq says:

    Hi, 

    is it possible to have one PWWN from HBA connected to multiple Remote ports output is as below , can you elaborate how this is configured 

    mccah1sap029:/root> fcinfo remote-port -sl -p 10000000c6666712 |grep ‘Remote Port WWN’
    Remote Port WWN: 50001fe9000ec098
    Remote Port WWN: 50001fe9000ec09a
    Remote Port WWN: 50001fe9000ec09c
    Remote Port WWN: 50001fe9000ec09e

    • Ramdev Ramdev says:

      @Musthaq. Yes it is possible.

      HBA@server–> Multiple Fibler conenction —> each conenction goes to one remote port of the SAN switch –> all switch ports target to single storage device

  27. Mushtaq says:

    ok , that said , N Port (host) –> F port -> N1 Port (fabric  FA1A) 
                                                                     N2 Port (fabric  FA2A) 

                            N Port (host) –> F port -> N1 Port (fabric  FA1B) 
                                                                      N2 Port (fabric  FA2B) 

    am i correct ?

  28. shiva says:

    EMC Fa1a target 11 555555555555551a
    EMC Fa1b target 12 555555555555551b
    EMC FA2a target 21 555555555555552a
    EMC Fa2b target 22 555555555555552b

    from where these numbers came

  29. Ganesh says:

    That’s really an amazing info you shared. It will be very useful if you explain for the linux SAN configuration step by step setup. Please let us know the link if its already there.

  30. MM says:

    It is a very good article, let me more clear of the SAN technology, thx~~~

  31. sushil prasad says:

    can you please tell me installation process/procedure of SAN in linux or you can recommended me to some website/video link .

  32. vijayababu says:

    How to re-setting the ilo passwd for T5120. we are for getting the ilo passwd. After connecting the Serial-MGMT port, what are all the steps to fallow.

  33. hashinmy says:

    can you doit for windows

  1. September 15, 2015

    […] SAN (Storage area Networking) for System Administrators […]

  2. September 17, 2015

    […] Read – SAN Guide for Beginners […]

What is in your mind, about this post ? Leave a Reply

Close
  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us