RHEL LVM : How to recover a deleted Logical volume in RHEL 5/6

[vc_row][vc_column width=”2/3″][vc_text_separator title=”Other Learning Articles that you may like to read” color=”black”][vc_tta_tour style=”flat” spacing=”2″ gap=”2″ controls_size=”sm” active_section=”1″ css=”.vc_custom_1469259876636{background-color: #ffffff !important;}” no_fill_content_area=”true”][vc_tta_section title=”Latest Articles” tab_id=”1468871183598-9466d85c-9ce4″][vc_masonry_grid post_type=”post” max_items=”50″ style=”load-more” items_per_page=”7″ element_width=”12″ gap=”2″ item=”17530″ grid_id=”vc_gid:1469282723468-c7b025065176ca8c9d9de4399966c6ec-7″ btn_add_icon=”true” btn_custom_onclick=”true” css=”.vc_custom_1469258760195{background-color: #eaeaea !important;}”][/vc_tta_section][vc_tta_section title=”Information Centers” tab_id=”1468869118809-6c32c26d-7a73″][vc_masonry_grid post_type=”post” max_items=”30″ style=”load-more” items_per_page=”7″ element_width=”12″ gap=”2″ item=”17530″ btn_title=”Show me More Posts” btn_shape=”round” grid_id=”vc_gid:1469282723519-5bda201289ffc3488d36d95b9a511fdc-10″ taxonomies=”674″ btn_add_icon=”true” btn_custom_onclick=”true” css=”.vc_custom_1469282769356{background-color: #dbdbdb !important;}”][/vc_tta_section][vc_tta_section title=”Linux Admin” tab_id=”1468863117966-71d46596-1349″][vc_masonry_grid post_type=”post” max_items=”100″ style=”load-more” items_per_page=”7″ element_width=”12″ gap=”2″ item=”17530″ initial_loading_animation=”none” grid_id=”vc_gid:1469282723571-63cba23831dc7526adf19df94fd0828e-3″ taxonomies=”35, 649, 427″ btn_add_icon=”true” btn_custom_onclick=”true” css=”.vc_custom_1469258736301{background-color: #ededed !important;}”][/vc_tta_section][vc_tta_section title=”Solaris Admin” tab_id=”1468859378329-df794d2f-31ff”][vc_masonry_grid post_type=”post” max_items=”50″ style=”load-more” items_per_page=”7″ element_width=”12″ gap=”5″ item=”17530″ initial_loading_animation=”none” grid_id=”vc_gid:1469282723619-7293ebb36536eb1860a075d903dcb71f-5″ taxonomies=”125, 65, 84″ btn_add_icon=”true” btn_custom_onclick=”true” css=”.vc_custom_1469258528977{background-color: #eaeaea !important;}”][/vc_tta_section][vc_tta_section title=”VxVM Admin” tab_id=”1468863449237-8094f6e6-e47a”][vc_masonry_grid post_type=”post” max_items=”40″ style=”load-more” items_per_page=”7″ element_width=”12″ gap=”5″ item=”17530″ initial_loading_animation=”none” grid_id=”vc_gid:1469282723666-38716db4886d75c4e259e23ceed8d18e-4″ taxonomies=”20″ btn_add_icon=”true” btn_custom_onclick=”true” css=”.vc_custom_1469258547772{background-color: #eaeaea !important;}”][/vc_tta_section][vc_tta_section title=”VCS Admin” tab_id=”1468859393073-b591c633-e80c”][vc_masonry_grid post_type=”post” max_items=”40″ style=”load-more” items_per_page=”7″ element_width=”12″ gap=”2″ item=”17530″ grid_id=”vc_gid:1469282723714-67f3aac26e429b44918631262b219431-0″ taxonomies=”587, 636, 95, 93, 94, 96, 98, 583″ btn_add_icon=”true” btn_custom_onclick=”true” css=”.vc_custom_1469258815100{background-color: #e5e5e5 !important;}”][/vc_tta_section][vc_tta_section title=”Career Guidance” tab_id=”1468859409803-7e8f1d27-b880″][vc_masonry_grid post_type=”post” max_items=”50″ style=”load-more” items_per_page=”7″ element_width=”12″ gap=”2″ item=”17530″ grid_id=”vc_gid:1469282723757-bf496e80da38119f50feee30d2d9bae7-1″ taxonomies=”27, 156″ btn_add_icon=”true” btn_custom_onclick=”true” css=”.vc_custom_1469258835863{background-color: #e5e5e5 !important;}”][/vc_tta_section][vc_tta_section title=”Scripting” tab_id=”1468863616077-d75c044f-0d6a”][vc_masonry_grid post_type=”post” max_items=”50″ style=”load-more” items_per_page=”7″ element_width=”12″ gap=”5″ item=”17530″ grid_id=”vc_gid:1469282723805-e82cf3f477d33494c907fdfc921fb261-2″ taxonomies=”117″ btn_add_icon=”true” btn_custom_onclick=”true”][/vc_tta_section][vc_tta_section title=”Storage Admin” tab_id=”1468864446328-8f46c04c-e442″][vc_masonry_grid post_type=”post” max_items=”50″ style=”load-more” items_per_page=”7″ element_width=”12″ gap=”2″ item=”17530″ initial_loading_animation=”none” grid_id=”vc_gid:1469282723828-fc94cceb74954650b42d157258f9c6e0-7″ taxonomies=”87, 63, 86, 124, 109″ btn_add_icon=”true” btn_custom_onclick=”true” css=”.vc_custom_1469259411409{background-color: #e0e0e0 !important;}”][/vc_tta_section][vc_tta_section title=”Quick Reference” tab_id=”1468864576012-b7f2e683-10d4″][vc_masonry_grid post_type=”post” max_items=”30″ style=”load-more” items_per_page=”6″ element_width=”12″ gap=”2″ item=”17530″ grid_id=”vc_gid:1469282723883-7e90c912885cc85cdc19d133fe4849f2-2″ taxonomies=”116, 625, 624, 626″ btn_add_icon=”true” btn_custom_onclick=”true” css=”.vc_custom_1469258877449{background-color: #dddddd !important;}”][/vc_tta_section][/vc_tta_tour][/vc_column][vc_column width=”1/3″][vc_text_separator title=”Free Courses We Offer” color=”black”][vc_btn title=”Linux Administration Interview Preparation `{`Free Email Crash Course`}`” style=”outline-custom” outline_custom_color=”#666666″ outline_custom_hover_background=”#666666″ outline_custom_hover_text=”#ffffff” shape=”square” align=”center” link=”url:http%3A%2F%2Funixadminschool.com%2Fblog%2F2016%2F03%2Ffree-email-crash-course-for-linux-admin-interview-preparation%2F||target:%20_blank|” button_block=”true”][vc_btn title=”Roles and Responsibilities of System Admin `{`Free Course`}` ” style=”outline-custom” outline_custom_color=”#666666″ outline_custom_hover_background=”#666666″ outline_custom_hover_text=”#ffffff” shape=”square” align=”center” link=”url:http%3A%2F%2Funixadminschool.com%2Fmember%2Fcourse%2Fknow-about-solaris-system-administration-job%2F||target:%20_blank|” button_block=”true”][vc_btn title=”Getting Started with RHEL7 Administration `{`Free Course`}` ” style=”outline-custom” outline_custom_color=”#666666″ outline_custom_hover_background=”#666666″ outline_custom_hover_text=”#ffffff” shape=”square” align=”center” link=”url:http%3A%2F%2Funixadminschool.com%2Fmember%2Fcourse%2Ffree-course-getting-started-with-red-hat-enterprise-linux-7%2F||target:%20_blank|” button_block=”true”][vc_text_separator title=”Paid Training Courses we Offer” color=”black”][vc_tta_tabs style=”flat” color=”black” gap=”2″ active_section=”1″ no_fill_content_area=”true” css=”.vc_custom_1469259083495{background-color: #f2f2f2 !important;border-radius: 1px !important;}”][vc_tta_section title=”Self Paced Courses” tab_id=”1469255118344-7aa625f2-e823″][vc_btn title=”Solaris11 Self Paced Video Learning” style=”classic” color=”primary” align=”center” link=”url:http%3A%2F%2Funixadminschool.com%2Fmember%2Fcourse%2Fsolaris11-administration-for-experience-solaris10-administrators%2F||target:%20_blank|” button_block=”true”][vc_btn title=”RHEL7 Self Paced Video Learning” style=”classic” color=”primary” align=”center” link=”url:http%3A%2F%2Funixadminschool.com%2Fmember%2Fproduct%2Fbecome-an-expert-in-red-hat-enterprise-linux-7%2F||target:%20_blank|” button_block=”true”][vc_btn title=”VxVM + VCS Self Paced Video Learning” style=”classic” color=”primary” align=”center” link=”url:http%3A%2F%2Funixadminschool.com%2Fmember%2Fproduct%2Fstorage-foundation-and-vcs-administration-for-unix-and-linux-engineers%2F||target:%20_blank|” button_block=”true”][vc_btn title=”Solaris10 Self Paced Video Learning” style=”classic” color=”primary” align=”center” link=”url:http%3A%2F%2Funixadminschool.com%2Fmember%2Fproduct%2Fsolaris-associate-membership%2F||target:%20_blank|” button_block=”true”][vc_btn title=”Puppet Automation Self Paced Video Learning” style=”classic” color=”primary” align=”center” link=”url:http%3A%2F%2Funixadminschool.com%2Fmember%2Fcourse%2Fautomation-with-puppet-self-paced-video-learning%2F||target:%20_blank|” button_block=”true”][/vc_tta_section][vc_tta_section title=”Live Webex” tab_id=”1469255118345-d6338902-a6d4″][vc_btn title=”Solaris 11 Administration Live WebEx” style=”classic” color=”warning” align=”center” link=”url:http%3A%2F%2Funixadminschool.com%2Fmember%2Fcourse%2Fsolaris11-administration-for-experience-solaris10-administrators%2F||target:%20_blank|” button_block=”true”][vc_btn title=”Puppet Automation – Live WebEx” style=”classic” color=”warning” align=”center” link=”url:http%3A%2F%2Funixadminschool.com%2Fmember%2Fcourse%2Fautomation-with-puppet-self-paced-video-learning%2F||target:%20_blank|” button_block=”true”][vc_btn title=”Veritas ( VCS+VxVM) Live Webex” style=”classic” color=”warning” align=”center” link=”url:http%3A%2F%2Funixadminschool.com%2Fmember%2Fproduct%2Fveritas-training-vxvm-vcs-live-webex-training%2F||target:%20_blank|” button_block=”true”][/vc_tta_section][/vc_tta_tabs][vc_separator][/vc_column][/vc_row]

In this post, you can find the procedure to recover a deleted logical volume, from the linux server  with below configuration.

Server Name :  GURKULRHEL

Version          :  Redhat Enterprise Linux 6

LVM Configuraiton:    

            VG NAME : datavg

             LV NAME:  datalv

 

Test the restore before actually recovering it

GURKULRHEL6# vgcfgrestore datavg –test -f /etc/lvm/archive/datavg_00002-1304122072.vg
Test mode: Metadata will NOT be updated and volumes will not be (de)activated.

Restore the volume group datavg with vgcfgrestore

GURKULRHEL6# vgcfgrestore datavg -f /etc/lvm/archive/datavg_00002-1304122072.vg

Verify the Restored volume group with lvscan

GURKULRHEL6# lvscan
ACTIVE ‘/dev/vg_rhel6test1/lv_root’ [5.47 GiB] inherit
ACTIVE ‘/dev/vg_rhel6test1/lv_swap’ [1016.00 MiB] inherit
inactive ‘/dev/datavg/datalv’ [900.00 MiB] inherit

Activate the restored LV with lvchange

GURKULRHEL6# lvchange -a y /dev/datavg/datalv

 When LV Actication Fails

GURKULRHEL6# lvchange -a y /dev/datavg/datalv
device-mapper: create ioctl on datavg-datalv failed: Device or resource busy 

 First Understand the issue

To Resolve this error, let’s try different way

Remove the volume using lvremove command

GURKULRHEL6# lvremove /dev/datavg/datalv
Logical volume “datalv” successfully removed

Verify if the DM devices still exists for the deleted Volume using “dmsetup -c”

   From the below command we can still see the device is existing after removal. The affected LV highlighted in Red

GURKULRHEL6# dmsetup info -c
Name Maj Min Stat Open Targ Event UUID
datavg-datalv         253 2 L–w 0 1 0 LVM-FWged1lMqw00KsadDMKsLkDt8waYhhLeGT2cO 
vg_rhel6test1-lv_swap 253 1 L–w 1 1 0 LVM-bozPsOPjzbkjUiRVOU33vp3aZxqNKJRvTdHvz
vg_rhel6test1-lv_root 253 0 L–w 1 1 0 LVM-3bozPsOPjzbkjUiRVOU33vp3aZxqNKJRvYp0l

Verify if the deleted  LV visible with lvscan

From the Below command related LV is not visible with the lvscan as shown below

GURKULRHEL6# lvscan
ACTIVE ‘/dev/vg_rhel6test1/lv_root’ [5.47 GiB] inherit
ACTIVE ‘/dev/vg_rhel6test1/lv_swap’ [1016.00 MiB] inherit 

  Work Around to Resolve the issue

Remove the  dm device  using dmsetup

GURKULRHEL6# dmsetup remove -f /dev/dm-2

Restore the Logical Volume from the datavg from the backup again

GURKULRHEL6# vgcfgrestore datavg -f /etc/lvm/archive/datavg_00002-1304122072.vg

Verify the restored Logical Volume with lvscan command

GURKULRHEL6# lvscan
ACTIVE ‘/dev/vg_rhel6test1/lv_root’ [5.47 GiB] inherit
ACTIVE ‘/dev/vg_rhel6test1/lv_swap’ [1016.00 MiB] inherit
inactive ‘/dev/datavg/datalv’ [900.00 MiB] inherit

Activate the LV volume with lvchange

GURKULRHEL6# lvchange -a y /dev/datavg/datalv

Verify the logical volume status

GURKULRHEL6# lvscan
ACTIVE ‘/dev/vg_rhel6test1/lv_root’ [5.47 GiB] inherit
ACTIVE ‘/dev/vg_rhel6test1/lv_swap’ [1016.00 MiB] inherit
ACTIVE ‘/dev/datavg/datalv’ [900.00 MiB] inherit 

Mount the activated Logical volume

GURKULRHEL6# mount /dev/datavg/datalv /test

    If mount command fails

GURKULRHEL6# mount /dev/datavg/datalv /test
mount: you must specify the filesystem type

Run e2fsck with -n first to verify the file system Status

GURKULRHEL6# e2fsck -n /dev/datavg/datalv
e2fsck 1.41.12 (17-May-2010)
e2fsck: Superblock invalid, trying backup blocks…
/dev/datavg/datalv was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group GURKULRHEL6#0 (29088, counted=29073).
Fix? no

Free blocks count wrong for group GURKULRHEL6#1 (32710, counted=28993).
Fix? no

Free blocks count wrong (222392, counted=218660).
Fix? no

Inode bitmap differences: -(7201–7211) -(14401–14402)
Fix? no

Free inodes count wrong for group GURKULRHEL6#0 (7189, counted=7076).
Fix? no

Directories count wrong for group GURKULRHEL6#0 (2, counted=17).
Fix? no

Free inodes count wrong for group GURKULRHEL6#1 (7200, counted=7189).
Fix? no

Free inodes count wrong for group GURKULRHEL6#2 (7200, counted=7198).
Fix? no

Free inodes count wrong (57589, counted=57463).
Fix? no

/dev/datavg/datalv: ***** FILE SYSTEM WAS MODIFIED *****

/dev/datavg/datalv: ********** WARNING: Filesystem still has errors **********

/dev/datavg/datalv: 11/57600 files (9.1% non-contiguous), 8008/230400 blocks
Error writing block 1 (Attempt to write block from filesystem resulted in short write). Ignore error? no

Error writing block 1 (Attempt to write block from filesystem resulted in short write). Ignore error? no

Error writing block 1 (Attempt to write block from filesystem resulted in short write). Ignore error? no

        Find Backup Super block to fix the inconsistent file system

GURKULRHEL6# mkfs.ext4 -n /dev/datavg/datalv
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
57600 inodes, 230400 blocks
11520 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=239075328
8 block groups
32768 blocks per group, 32768 fragments per group
7200 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376

Rerun the  e2fsck using backup superblock

GURKULRHEL6# e2fsck -yb 32768 /dev/datavg/datalv
e2fsck 1.41.12 (17-May-2010)
/dev/datavg/datalv was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group GURKULRHEL6#0 (29088, counted=29073).
Fix? yes

Free blocks count wrong for group GURKULRHEL6#1 (32710, counted=28993).
Fix? yes

Free blocks count wrong (222392, counted=218660).
Fix? yes

Inode bitmap differences: -(7201–7211) -(14401–14402)
Fix? yes

Free inodes count wrong for group GURKULRHEL6#0 (7189, counted=7076).
Fix? yes

Directories count wrong for group GURKULRHEL6#0 (2, counted=17).
Fix? yes

Free inodes count wrong (57589, counted=57476).
Fix? yes

/dev/datavg/datalv: ***** FILE SYSTEM WAS MODIFIED *****
/dev/datavg/datalv: 124/57600 files (0.8% non-contiguous), 11740/230400 blocks

Verify the filesystem consistency once again

GURKULRHEL6# e2fsck -n /dev/datavg/datalv
e2fsck 1.41.12 (17-May-2010)
/dev/datavg/datalv: clean, 124/57600 files, 11740/230400 blocks
[root@dhcp-1-17 /]GURKULRHEL6# e2fsck -fy /dev/datavg/datalv
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/datavg/datalv: 124/57600 files (0.8% non-contiguous), 11740/230400 blocks

Finally , try mounting the filesystem again

GURKULRHEL6# mount /dev/datavg/datalv /test

 

Final Verification of data from the mounted volume

GURKULRHEL6# ls /test
log lost+found

GURKULRHEL6# df -h /test
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/datavg-datalv
886M 32M 810M 4% /test

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

6 Responses

  1. September 4, 2015

    […] RHEL LVM : How to recover a deleted Logical volume in RHEL 5/6 […]

  2. September 17, 2015

    […] Read – How to recover a deleted Logical volume in RHEL 5/6 […]

  3. September 17, 2015

    […] Read –  How to recover a deleted Logical volume in RHEL 5/6 […]

  4. September 17, 2015

    […] Read – How to recover a deleted Logical volume in RHEL 5/6 […]

  5. September 17, 2015

    […] Read – How to recover a deleted Logical volume in RHEL 5/6 […]

Leave a Reply

Your email address will not be published. Required fields are marked *