Beginner’s Lesson – VERITAS Volume Manager for Solaris

 

Veritas Volume Manager is a storage management application by Symantec ,  which allows you to manage physical disks as logical devices called volumes.

VxVM uses two types of objects to perform the storage management

1. Physical objects – are direct mappings to physical disks

2 . Virtual objects – are volumes, plexes, subdisks and diskgroups.

a. Disk groups are composed of Volumes
b. Volumes are composed of Plexes and Subdisks
c. Plexes are composed of SubDisks
d. Subdisks are actual disk space segments of VxVM disk  ( directly mapped from the physical disks)

 

1. Physical Disks

Physical disk is a basic storage where ultimate data will be stored. In Solaris physical disk names  uses the  convention like “c#t#d#”  where c# refers to controller/adapter connection, t# refers to the SCSI target Id , and d# refers to disk device Id.  Below figure illustrates how the disk name changes depending on the connection.

Physical disks could be coming from different sources within the servers e.g. Internal disks to the server , Disks from the Disk Array  and Disks from the SAN.

 

 

Check the disks recognized by Solaris

 

#echo|format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c0t0d0
/sbus@1f,0/SUNW,fas@e,8800000/sd@0,0
1. c0t1d0
/sbus@1f,0/SUNW,fas@e,8800000/sd@1,0

2. Solaris Native Disk Partitioning

In solaris, physical disks will partitioned into slices numbered as S0,S1,S3,S4,S5,S6,S7 and the slice number S2 normally called as overlap slice and points to the entire disk.  In Solaris we use the format utility used to partition the physical disks into slices.

Once we added new disks to the Server, first we should recognize the disks from the solaris level before proceeding for any other storage management utility.

 

Steps to add new disk to Solaris:

 

If the disks that are recently added to the server not visible, you can use below procedure

Option 1: Reconfiguration Reboot ( for the server hardware models that doesn’t support hot swapping/dynamic addition of disks )

# touch /reconfigure; init 6

or

#reboot — -r ( only if no applications running on the machine)

Option 2: Recognize  the disks added to external SCSI, without reboot

# devfsadm

# echo | format

Option 3: Recognize disks that are added to internal scsi, hot swappable, disk connections.

Just run the command “cfgadm -al” and check for any newly added devices in “unconfigured” state, and configure them.

# cfgadm -al
Ap_Id                         Type            Receptacle   Occupant     Condition
c0                                  scsi-bus     connected    configured   unknown
c0::dsk/c0t0d0      disk              connected    configured   unknown
c0::dsk/c0t0d0      disk              connected    configured   unknown
c0::rmt/0                  tape             connected    configured   unknown
c1                                  scsi-bus      connected    configured   unknown
c1::dsk/c1t0d0       unavailable  connected    unconfigured unknown
c1::dsk/c1t1d0       unavailable  connected    unconfigured unknown < == disk not configured

 

# cfgadm -c configure c1::dsk/c1t0d0

# cfgadm -c configure c1::dsk/c1t0d0

# cfgadm -al
Ap_Id                         Type            Receptacle   Occupant     Condition
c0                                  scsi-bus     connected    configured   unknown
c0::dsk/c0t0d0      disk              connected    configured   unknown
c0::rmt/0                  tape             connected    configured   unknown
c1                                  scsi-bus      connected    configured   unknown
c1::dsk/c1t0d0       disk              connected    configured unknown
c1::dsk/c1t1d0       disk              connected    configured unknown

# devfsadm

#echo|format

Additional Procedure : Click here to check the procedure to add Internal FC-AL disks

3. Initialize Physical Disks under VxVM control

A formatted physical disk is considered uninitialized until it is initialized for use by VxVM. When a disk is initialized, partitions for the public and private regions are created, VM disk header information is written to the private region and actual data is written to Public region.  During the notmal initialization process any data or partitions that may have existed on the disk are removed.

Note: Encapsulation is another method of placing a disk under VxVM control in which existing data on the disk is preserved

An initialized disk is placed into the VxVM free disk pool. The VxVM free disk pool contains disks that have been initialized but that have not yet been assigned to a disk group. These disks are under Volume Manager control but cannot be used by Volume Manager until they are added to a disk group

Device Naming Schemes

In VxVM, device names can be represented in two ways:

  • Using the traditional operating system-dependent format c#t#d#
  • Using an operating system-independent format that is based on enclosure names

c#t#d# Naming Scheme

Traditionally, device names in VxVM have been represented in the way that the operating system represents them. For example, Solaris and HP-UX both use the format c#t#d# in device naming, which is derived from the controller, target, and disk number. In VxVM version 3.1.1 and earlier, all disks are named using the c#t#d# format. VxVM parses disk names in this format to retrieve connectivity information for disks.

Enclosure-Based Naming Scheme

With VxVM version 3.2 and later, VxVM provides a new device naming scheme, called enclosure-based naming. With enclosure-based naming, the name of a disk is based on the logical name of the enclosure, or disk array, in which the disk resides.

Steps to Recognize new disks under VxVM control

1. Run the below command to see the available disks under VxVM control

# vxdisk list

in the output you will see below status

  • error indicates that the disk has neither been initialized nor encapsulated by VxVM. The disk is uninitialized.
  • online indicates that the drive has been initialized or encapsulated.
  • online invalid indicated that disk is visible to VxVM but not controlled by VxVM

If disks are visible with “format” command but not visible with  “vxdisk list” command, run below command to scan the new disks for VxVM

# vxdctl enable

Now you should see new disks with the status of “Online Invalid

2. Initialize each disk with “vxdisksetup” command

#/etc/vx/bin/vxdisksetup -i

after running this command “vxdisk list” should see the status as “online” for all the newly initialized disks

4. Virtual Objects (DiskGroups / Volumes / Plexs )  in VxVM

Disk Groups

A disk group is a collection of  VxVM disks ( going forward we will call them as VM Disks ) that share a common configuration.  Disk groups allow you to group disks into logical group of Subdisks called plexes which in turn forms the volumes.

Volumes

A volume is a virtual disk device that appears to applications, databases, and file systems like a physical disk device, but does not have the physical limitations of a physical disk device. A volume consists of one or more plexes, each holding a copy of the selected data in the volume.

Plexes:

VxVM uses subdisks to create virtual objects called plexes. A plex consists of one or more subdisks located on one or more physical disks.

From the below diagram you can observer below points

  • The Diskgroup named “Diskgroup_o1” is created using 4 different VM disks named as “vxdisk_0x, vxdisk_oy, vxdisk_oz and vxdisk_oa
  • The diskfgroup  “Diskgroup_01″ was configured with 4 different volumes i.e. “1. Concat_vol 2. striped_vol 3. mirror_vol and 4. raid5_vol
  • Concat_vol is a concatenation volume with single plex i.e. con_plex01 inside, and con_plex01 was build up using 4 subdisks of different size
  • Striped_vol is a striped volume with single plex i.e. stripe_plex01 inside, and stripe_plex01 was build up using 4 subdisks of same size
  • mirror_vol is a mirrored volume with two plexes named “mplex01 and mplex02”  inside, each plex is copy of other. Both the plexes formed with different subdisks  of  either same size or different size.
  • raid5_vol is a raid5 ( striped with parity) volume which build up using 3 different plexes formed with the subdisks of 3 different VM disks.

 

 

Summary – Transformation of Physical disks into Veritas Volumes

Below diagram shows you the complete transformation of a physical disk into a veritas volume. And below is the summary of complete process

1. Recognize disks under solaris using devfsadm, cfgadm or reconfiguration reboot , and verify using format command
2. Recognize the disks under VxVM using “vxdctl enable
3. Initialize the disks under VxVM using vxdisksetup
4. Add the disks to Veritas Disk Group using vxdg commands
5. Create Volumes under Disk Group using vxmake or vxassist commands
6. Create filesystem on top of volumes using mkfs or newfs, and you can create either VXFS filesystem or UFS filesystem

 

 

Solaris Volume Manager Vs Veritas Volume Manager

Below diagram explains how Solaris volume manager differs from the Veritas volume manager during the process of creating a new file system on top of physical storage.

 

 

 


Continuation Post : VxVM Beginners Lesson2 – Commands and Examples

Ramdev

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

52 Responses

  1. Rex di Bona says:

    Very nice, but… why would you want to use VxVM on Solaris anyway?
    SVM is cheaper (free), and (from my tests), in every case faster than the equivalent VxVM layout. With soft partitons SVM no longer requires fixed partitions.
    Or, with ZFS you have no need for the brain spasms that are required for using VxVM and a whole lot more besides. In my opinion, given Soalris 10 is now over 6 years old VxVM for Solaris is a strictly legacy product only.

  2. Sanjay Kumar Verma says:

    Good One!!

    Please keep posted on the same.

  3. ramdev says:

    Rex, your point is very valid. my post is basically for those sa who are still supporting decade old enterprise networks. for example the financial firms cannot have the luxury of moving all their applications out of this legacy configuration ( solaris + vxvm) , and the support team who work there still need the expertise on those configurations. :)

  4. ramdev says:

    @Sanjay- you will see more related posts , for sure.

  5. Carlito says:

    Nice lesson. Do you know where I can download Veritas volume manager for Solaris. Thanks.

  6. Raul says:

    Thank you!!!!!!!!

  7. Mohammad says:

    Hi Ram Kumar,

    very good lesson for veritas beginers..
    Mohammad

  8. Naresh says:

    Hi ,
    can you please give a complete notes or pdf any link for veritas VM .I like this post, same like this do u have any book or videos please send me the link 

  9. Yogesh Raheja says:

    @Rex, their are still some advantages of VXVM over SVM. Specially in VCS where VXVM is highly recommended and required to support VCS. Though its complete business policy by Symantec :). On the other hand ZFS is much powerful but still it will take some time to grab the market demand. Till that time VXVM will remain dominent aas far as Solaris is concerned.

  10. Naresh says:

    Hi Ram

    can u post anything regarding VERITAS NET BACKUP

    Regards,
    Naresh

  11. Viru says:

    HI Ram,

    Please post more about VXVM ,,, This lesson is very good for new starter..

  12. Yogesh Raheja says:

    Hi Viru, You can get lots of posts on VXVM under Veritas Tab. Also we will keep posting more on VXVM. :).

  13. Ramdev Ramdev says:

    Hello Viru, Actually Yogesh referring to below page … just have a look at beginners lessons

    http://gurkulindia.com/main/category/system-admin/veritas/veritas-volume-manager/

  14. Rajeswari says:

    Can you please explain the relation between offset value and disk alignment number.

    what does it mean • Subdisks should be cylinder aligned.

    • Ramdev Ramdev says:

      Hi Rajeswari,

      what does it mean • Subdisks should be cylinder aligned.
      >> Diskalign/nodiskalign is a layout feature used while performing Vxassist operations. When we say Subdisks are cylinder aligned ( default option), subdisks will be created beginning on cylinder boundaries and they will be extended to match the end of a cylinder.

      Can you please explain the relation between offset value and disk alignment number.
      >> I will explain you if you have specific example.

  15. pramath says:

    Please explain Most and Common Troubleshooting Processes and easy methods, Disaster recovery at various instances in VXVM (Real Time Scenarios) if available.

  16. Yogesh Raheja says:

    @Pramath, kindly refer below link and let us know if you want solution of any problem which encountered:
    http://gurkulindia.com/main/category/veritas/veritas-volume-manager/

  17. Mushtaq says:

    Hi , 

    i want to confirm which disk is used by oracle , which disk falls under vxvm and which disk is utilised by the OS only. 

    vxvm list shows different names 
    solaris shows different. 

    any advise to confirm . 

    my environment have , 

    solaris 10 > vxvm > vcs > oracle 

    thanks in advance.

  18. Mushtaq says:

    further explanation : 

    one one system : 
    root@aurhselux115# vxdisk list
    DEVICE       TYPE            DISK         GROUP        STATUS
    Disk_0       auto:cdsdisk    zonedisk     zonedg       online
    Disk_2       auto:cdsdisk    –            –            online udid_mismatch
    Disk_6       auto            –            –            error
    Disk_7       auto            –            –            error
    Disk_8       auto            –            –            error
    Disk_9       auto:none       –            –            online invalid
    Disk_10      auto:cdsdisk    –            –            online
    Disk_11      auto:cdsdisk    –            –            online
    Disk_12      auto:cdsdisk    –            –            online

    On second system 
    root@aurhselux116# vxdisk list
    DEVICE       TYPE            DISK         GROUP        STATUS
    Disk_0       auto:cdsdisk    –            –            online
    Disk_1       auto            –            –            error
    Disk_2       auto:cdsdisk    –            –            online udid_mismatch
    Disk_4       auto            –            –            error
    Disk_5       auto:none       –            –            online invalid
    Disk_6       auto:none       –            –            online invalid
    Disk_7       auto:none       –            –            online invalid
    Disk_8       auto:none       –            –            online invalid

    both are under vxvm / vcs  , want to identify what is there on which disk , anyway from the header etc we can identify ? 

    thanks again.

  19. Mushtaq says:

    how to compare between , vxdisk list and format command to confirm ? which disk we are talking about ? any idea thanks

  20. Yogesh Raheja says:

    @Mushtaq, below is the explanation to your query:

    There are mainly three types of layout of disk in vxvm:

    Sliced Disk layout:
    private and public region slices are on separate partitions (usually slices 3 & 4). This type of disk is not suitable for moving between different O/S’s but are suitable for boot partitions. OS disk under veritas are of Sliced layout

    CDS (Cross-platform Data Sharing):
    private and public regions are one slice (usually slice 7). This type is suitable for moving between different O/S’s but not suitable for boot partitions. So this cannot be used for OS disk.

    Simple:
    Private and public are the same partition but continuous (usually slice 3).

    Firstly dont be confused in format or vxdisk outputs. Format will show you total no. of disks recongnised by your OS irrespective of veritas/emc/local disks. If veritas is installed then vxdisk list is the command which is ths major command to identify the disks are under Veritas or Not.

    Now considering your example, check the TYPE & STATUS FIELD:

    Disk_0, Type: auto:cdsdisk, FIELD: online —> Means the disk is under veritas and is online.

    Disk_9 Type: auto:none FIELD: online invalid –> Means the disk is under OS but not under Veritas (online invalid)

    Disk_2 Type: auto:cdsdisk FIELD: online udid_mismatch –> Means the disk is under veritas, usually you will see udid_mismatch in EMC disks.

    • Ramdev Ramdev says:

      @yogesh – those are useful tips.

      @musthaq – I guess you are asking this question either to configure, unconfigure or to perform the maintenance on the disk.

      as long as the question is related to isolating the veritas disks with OS it is pretty straight forward. what every appears as “online invalid” that is not taken care by veritas. And again there is an exception, there are cases that we add EMC disks and leave them for later configurations. And if you are not the one who configured the disk, then you should be little careful to differentiate an new and unconfigured vxdisk from an OS disk.

      The few checks that i recommend in addition to ” format or vxdisks ” are :

      >> df output to check what filesystems are using what disks
      >> metastat -p output to check what disks are occupied by SDS
      >> check the iostat -xdn to see whether the disk making any io read/write, and this is useful if you see any disks are in use by oracle / sybase as raw devices.
      >> finally prttvtoc on the disk to check if the layout is familiar to me.

  21. Praveen says:

    Hi Ramdev,

    i m regularly watching your articles. It is very nice and insteresing to read. I m working as linux sysadmin. i want to learn vertias in our home, but reading such article i can gain knowlodge withour any practicals. i m seriously searching for articles which gives clear installation of vertias in linux machine. But i didnt find any doc related to installation. can you please provides me the such a article or any link for installation of vertias in linux 32 bit machine.

    Regards,
    Praveen.

  22. AIX ADMIN says:

    Hi Rex di Bona,

    Storage is costly and depending on the need u can keep prod (critical) data / devlopment (less critical) data on different storage such as hitachi/emc and netapp respectively.they are all integrated by veritas in such a way that different os/ hw platform and storages can coexists and the same command can be used in all the flavours and get rids of compatibility issues of vendor o.s’s using their own technologies and systems as all of them can use it.

  23. Bruce Shaw says:

    This is OK for simple disks. How about the situation where you’re attaching to a SAN using Veritas multipathing instead of Solaris’. I need to add more disk. I’ve currently got two LUNs presented but 50+ disks in my list for “format”. When I get the extra LUN, I’m going to get even more. How do I sort out which belongs to what?

  24. Ramdev Ramdev says:

    Hi Bruce, In case of  large number disks we just have to do some prework to map the each filesystem to corresponding veritas disk and then to the actual native disk mentioned in the format command. This is a mandatory pre-check task during the storage migrations. Probably i should add some notes on this part, Thanks for raising the question.

  25. Laxxi says:

    Hi, The above article is really very comprehendable..but as i went through the doc, i got few doubts like

    1) role of Disk group. 2)are plexeses made of subdisks of disks under same diskgroup. 3) a pictorial representation would be more helpful.

    thanks!!!

  26. Ramdev Ramdev says:

    DiskGroups are used to group the all related physical disks with similar purpose  ( e.g. one disk group for sybase database volumes, one fro oracle database volume, once for application data …etc),

    Whenever we think about any veritas logical components i.e. subdisk, plex, volume , they all should come from same disk group.  because these components cannot span across multiple diskgroups.

  27. Laxxi says:

    thnaks Ram!!

  28. Nav says:

    I have a sparc box with root disks encapsulated. It has free space.

    Is it ok to create a new volume on already encapsulated disk. It is running vertias 4.x. I need to create more swap space and wondering if it is ok to create new volume on the encapsulated rootdg. Any precautions to take while doing this? OR is it just same like how you create a new volume on vxvm/vxfs.

    The other option i have is to create a huge swapfile on the SAN lun [vxfs].

    Any input is appreciated?

    • Ramdev Ramdev says:

      Hi Nav, We cannot shrink or grow the volumes which are created before the encapsulation. But, for the creation of new volumes it should be fine as long as you have sufficient space.

  29. Santosh says:

    I want to install Veritas in solaris at my pc. Please let me know from which site i will get veritas software

  30. Hema says:

    ver useful… Thanx..

  31. Rajkamal says:

    Actually, it would have been more helpful if you could explain with command on how to create the layout you provided in the example figure:

    Like how to create / name those disks with vxdisk_0x, vxdisk_0y, vxdisk_0z,vxdisk_0a and then how to initialize with command (vxdisksetup), add them to dg etc. It will be good.

  32. Ramdev Ramdev says:

    Hi Raj, I believe you are talking about the topic about this continuation post….. 

    http://gurkulindia.com/main/2011/08/vxvm-beginners-lesson2-commands-and-examples/

  33. shiva says:

    hey it may be silly question but i’m not understanding what is difference b/w snap object and snap volume , dco object and dco volume

  34. shiva says:

    what is diff b/w snap object and snap volume ,dco object and dco volume

  35. LAKSHMI says:

    When we a add a new device to Veritas,, which commands like vxconfigd, vxdctl init or vxdctl enable are needed to execute to identify the new device.
    Please let me know , I’m bit confused at this point

  36. Ramdev Ramdev says:

    Lakshmi, –

    Few basic things to understand here –

    1 . The vxconfigd communicates configuration changes to the kernel and modifies configuration information stored on disk. Startup scripts usually invoke vxconfigd at system boot time. The vxconfigd daemon must be running for VxVM to operate properly.
    2. The vxdctl utility manages the state of the volume configuration daemon, vxconfigd.
    3. Each system will have a volboot file that contains contains a host ID that VxVM uses to establish ownership of physical disks

    ================================================
    About your question

    >>> vxdctl init – Reinitializes the volboot file with a new host ID , and that required only once during initial setup of the vxvm.

    >>> vxdctl enable – import all disk groups, and rebuild the volume device node directories. And This operation causes vxconfigd to scan for any disks that were newly added since vxconfigd was last started.

    And above two command assumes that you vxconfigd actively running in your server, and if it is not running they will alert you to start vxconfigd before you actually use the vxdctl commands. You can also check the vxconfigd status with below command

    # vxdctl mode
    If mode: enabled, then vxconfigd running
    if mode: not-running , then This message indicates that vxconfigd is not running:

    then you can start vxconfigd with

    # vxconfigd

  37. Amit Sharma says:

    Very Clear & easy Doc for beginners. Really appreciated to the way of presentation of SVM & VXVM.
    very nice mapping of physical disks and VX disks.

    do you have any next step of the Doc ? for further studies.

    Cheers \\ Amit Sharma

  38. Suman Das says:

    if we create a volume with four disk like your eample how many disk failure are tollerable?

    • Ramdev Ramdev says:

      Hi Suman, it depends on what kind of volume you are going to create? if you chose Raid-1 or Raid-5, it can only tolerate max 1. disk

  39. Sarang Sangram says:

    unable to pictures and diagram. please help

  1. August 10, 2011

    […] have already covered the first Learning Objective from the post  ”Veritas Volume Manager for System Adminstrators ” […]

  2. September 15, 2015

    […] Beginner’s Lesson – VERITAS Volume Manager for Solaris […]

  3. September 18, 2015

    […] Read – VxVM Beginner’s Lesson […]

What is in your mind, about this post ? Leave a Reply

Close
  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us