RHEL LVM : How to recover a deleted Logical volume in RHEL 5/6

In this post, you can find the procedure to recover a deleted logical volume, from the linux server  with below configuration.

Server Name :  GURKULRHEL

Version          :  Redhat Enterprise Linux 6

LVM Configuraiton:    

            VG NAME : datavg

             LV NAME:  datalv

 

Test the restore before actually recovering it

GURKULRHEL6# vgcfgrestore datavg –test -f /etc/lvm/archive/datavg_00002-1304122072.vg
Test mode: Metadata will NOT be updated and volumes will not be (de)activated.

Restore the volume group datavg with vgcfgrestore

GURKULRHEL6# vgcfgrestore datavg -f /etc/lvm/archive/datavg_00002-1304122072.vg

Verify the Restored volume group with lvscan

GURKULRHEL6# lvscan
ACTIVE ‘/dev/vg_rhel6test1/lv_root’ [5.47 GiB] inherit
ACTIVE ‘/dev/vg_rhel6test1/lv_swap’ [1016.00 MiB] inherit
inactive ‘/dev/datavg/datalv’ [900.00 MiB] inherit

Activate the restored LV with lvchange

GURKULRHEL6# lvchange -a y /dev/datavg/datalv

 When LV Actication Fails

GURKULRHEL6# lvchange -a y /dev/datavg/datalv
device-mapper: create ioctl on datavg-datalv failed: Device or resource busy 

 First Understand the issue

To Resolve this error, let’s try different way

Remove the volume using lvremove command

GURKULRHEL6# lvremove /dev/datavg/datalv
Logical volume “datalv” successfully removed

Verify if the DM devices still exists for the deleted Volume using “dmsetup -c”

   From the below command we can still see the device is existing after removal. The affected LV highlighted in Red

GURKULRHEL6# dmsetup info -c
Name Maj Min Stat Open Targ Event UUID
datavg-datalv         253 2 L–w 0 1 0 LVM-FWged1lMqw00KsadDMKsLkDt8waYhhLeGT2cO 
vg_rhel6test1-lv_swap 253 1 L–w 1 1 0 LVM-bozPsOPjzbkjUiRVOU33vp3aZxqNKJRvTdHvz
vg_rhel6test1-lv_root 253 0 L–w 1 1 0 LVM-3bozPsOPjzbkjUiRVOU33vp3aZxqNKJRvYp0l

Verify if the deleted  LV visible with lvscan

From the Below command related LV is not visible with the lvscan as shown below

GURKULRHEL6# lvscan
ACTIVE ‘/dev/vg_rhel6test1/lv_root’ [5.47 GiB] inherit
ACTIVE ‘/dev/vg_rhel6test1/lv_swap’ [1016.00 MiB] inherit 

  Work Around to Resolve the issue

Remove the  dm device  using dmsetup

GURKULRHEL6# dmsetup remove -f /dev/dm-2

Restore the Logical Volume from the datavg from the backup again

GURKULRHEL6# vgcfgrestore datavg -f /etc/lvm/archive/datavg_00002-1304122072.vg

Verify the restored Logical Volume with lvscan command

GURKULRHEL6# lvscan
ACTIVE ‘/dev/vg_rhel6test1/lv_root’ [5.47 GiB] inherit
ACTIVE ‘/dev/vg_rhel6test1/lv_swap’ [1016.00 MiB] inherit
inactive ‘/dev/datavg/datalv’ [900.00 MiB] inherit

Activate the LV volume with lvchange

GURKULRHEL6# lvchange -a y /dev/datavg/datalv

Verify the logical volume status

GURKULRHEL6# lvscan
ACTIVE ‘/dev/vg_rhel6test1/lv_root’ [5.47 GiB] inherit
ACTIVE ‘/dev/vg_rhel6test1/lv_swap’ [1016.00 MiB] inherit
ACTIVE ‘/dev/datavg/datalv’ [900.00 MiB] inherit 

Mount the activated Logical volume

GURKULRHEL6# mount /dev/datavg/datalv /test

    If mount command fails

GURKULRHEL6# mount /dev/datavg/datalv /test
mount: you must specify the filesystem type

Run e2fsck with -n first to verify the file system Status

GURKULRHEL6# e2fsck -n /dev/datavg/datalv
e2fsck 1.41.12 (17-May-2010)
e2fsck: Superblock invalid, trying backup blocks…
/dev/datavg/datalv was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group GURKULRHEL6#0 (29088, counted=29073).
Fix? no

Free blocks count wrong for group GURKULRHEL6#1 (32710, counted=28993).
Fix? no

Free blocks count wrong (222392, counted=218660).
Fix? no

Inode bitmap differences: -(7201–7211) -(14401–14402)
Fix? no

Free inodes count wrong for group GURKULRHEL6#0 (7189, counted=7076).
Fix? no

Directories count wrong for group GURKULRHEL6#0 (2, counted=17).
Fix? no

Free inodes count wrong for group GURKULRHEL6#1 (7200, counted=7189).
Fix? no

Free inodes count wrong for group GURKULRHEL6#2 (7200, counted=7198).
Fix? no

Free inodes count wrong (57589, counted=57463).
Fix? no

/dev/datavg/datalv: ***** FILE SYSTEM WAS MODIFIED *****

/dev/datavg/datalv: ********** WARNING: Filesystem still has errors **********

/dev/datavg/datalv: 11/57600 files (9.1% non-contiguous), 8008/230400 blocks
Error writing block 1 (Attempt to write block from filesystem resulted in short write). Ignore error? no

Error writing block 1 (Attempt to write block from filesystem resulted in short write). Ignore error? no

Error writing block 1 (Attempt to write block from filesystem resulted in short write). Ignore error? no

        Find Backup Super block to fix the inconsistent file system

GURKULRHEL6# mkfs.ext4 -n /dev/datavg/datalv
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
57600 inodes, 230400 blocks
11520 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=239075328
8 block groups
32768 blocks per group, 32768 fragments per group
7200 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376

Rerun the  e2fsck using backup superblock

GURKULRHEL6# e2fsck -yb 32768 /dev/datavg/datalv
e2fsck 1.41.12 (17-May-2010)
/dev/datavg/datalv was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group GURKULRHEL6#0 (29088, counted=29073).
Fix? yes

Free blocks count wrong for group GURKULRHEL6#1 (32710, counted=28993).
Fix? yes

Free blocks count wrong (222392, counted=218660).
Fix? yes

Inode bitmap differences: -(7201–7211) -(14401–14402)
Fix? yes

Free inodes count wrong for group GURKULRHEL6#0 (7189, counted=7076).
Fix? yes

Directories count wrong for group GURKULRHEL6#0 (2, counted=17).
Fix? yes

Free inodes count wrong (57589, counted=57476).
Fix? yes

/dev/datavg/datalv: ***** FILE SYSTEM WAS MODIFIED *****
/dev/datavg/datalv: 124/57600 files (0.8% non-contiguous), 11740/230400 blocks

Verify the filesystem consistency once again

GURKULRHEL6# e2fsck -n /dev/datavg/datalv
e2fsck 1.41.12 (17-May-2010)
/dev/datavg/datalv: clean, 124/57600 files, 11740/230400 blocks
[root@dhcp-1-17 /]GURKULRHEL6# e2fsck -fy /dev/datavg/datalv
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/datavg/datalv: 124/57600 files (0.8% non-contiguous), 11740/230400 blocks

Finally , try mounting the filesystem again

GURKULRHEL6# mount /dev/datavg/datalv /test

 

Final Verification of data from the mounted volume

GURKULRHEL6# ls /test
log lost+found

GURKULRHEL6# df -h /test
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/datavg-datalv
886M 32M 810M 4% /test

Ramdev

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

6 Responses

  1. September 4, 2015

    […] RHEL LVM : How to recover a deleted Logical volume in RHEL 5/6 […]

  2. September 17, 2015

    […] Read – How to recover a deleted Logical volume in RHEL 5/6 […]

  3. September 17, 2015

    […] Read –  How to recover a deleted Logical volume in RHEL 5/6 […]

  4. September 17, 2015

    […] Read – How to recover a deleted Logical volume in RHEL 5/6 […]

  5. September 17, 2015

    […] Read – How to recover a deleted Logical volume in RHEL 5/6 […]

What is in your mind, about this post ? Leave a Reply

Close
  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us