Linux LVM : Cleaning UP stale storage Devices from LVM, after Storage Reclaim

During the last weekend, I have noticed one of the interesting task performed by one of my collegue as part of the storage migration, i have captured the logs from his task. And the actual scenario is :

In any Storage migration ( with Host Side Data Copy ) below are the common steps that SA will follow :

  1. — Configure the New Storage Devices on the servers
  2. — Create new volumes from new Storage
  3. — Copy the Data from old volumes to new volumes ( either by volume level mirroring or Filesystem level Copy)
  4. — Unmount old filesystems and mount new filesystems – for application/ database access
  5. — Once Applicatin/DB works fine, remove old volumes
  6. — Remove old Storage from the Volume groups
  7. — Ask Storage team to reclaim the storage
  8. — Reconfiguration Reboot to remove any stale entries from the server device list

In the scenario that I am explaining below all the steps went fine except the step 6  and the server got rebooted directly after the storage reclaim. since the LVM cannot see it’s disks related to the Volume groups it started trowing “couldn’t find device errors”.

 As given in red below

[root@gurkuldinia-linux-server ~]# pvs -o +uuid

Couldn’t find device with uuid EyputQ-41zk-PPdr-CFAb-WzHc-oAPM-HzIvgq.

Couldn’t find device with uuid 9Csh1n-xuWq-ZGw3-1Rtz-WMnT-3Mfx-HuZ2wJ.

Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.

Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.

Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.

Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.

Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.

Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.

Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.

Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.

Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.

Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.

Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.

Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.

PV                   VG Fmt Attr PSize PFree PV UUID

/dev/cciss/c0d0p2    rootvg lvm2 a- 67.72G 33.56G try1O0-kPHY-3QTC-SEET-zC1f-065Q-OHuiq7

/dev/mapper/mpath10  ora_dump lvm2 a- 30.09G 104.00M lagcCD-09e5-p6wA-u70y-2I69-Obbt-5m4eHa

/dev/mapper/mpath11  ora_home lvm2 a- 20.09G 100.00M aCrz8Z-hP6v-YyVU-1PBx-l8R5-Vfmd-mR2Oe5

/dev/mapper/mpath12  ora_home lvm2 a- 8.09G 100.00M zuOHAp-ukLe-F2qA-NiXp-SeEC-a5sp-mreep4

/dev/mapper/mpath13  ora_home lvm2 a- 2.09G 108.00M eLmz3v-Akgj-epkH-dvzW-cKAY-G7f3-eQ7daU

unknown device ora_dump lvm2 a- 30.00G 30.00G EyputQ-41zk-PPdr-CFAb-WzHc-oAPM-HzIvgq

unknown device ora_dump lvm2 a- 30.00G 30.00G EyputQ-41zk-PPdr-CFAb-WzHc-oAPM-HzIvgq

unknown device ora_home lvm2 a- 2.00G 2.00G yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT

unknown device ora_home lvm2 a- 2.00G 2.00G yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT

unknown device ora_home lvm2 a- 2.00G 2.00G yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT

unknown device ora_home lvm2 a- 2.00G 2.00G yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT

 

 Ideally to remove any disks from the existing LVM volume group we do use vgreduce command as below

              # vgreduce volumegroup <devicepath>

Since the devices are completely missing here we couldn’t see actual devicepaths for missing devices. And then the Fix was performed as below using  the below command

              # vgreduce –removemissing volume-group

   Removing missing devices from ora_dump volume Group:

[root@gurkuldinia-linux-server ~]# vgreduce –removemissing ora_dump

Couldn’t find device with uuid EyputQ-41zk-PPdr-CFAb-WzHc-oAPM-HzIvgq.

Couldn’t find device with uuid 9Csh1n-xuWq-ZGw3-1Rtz-WMnT-3Mfx-HuZ2wJ.

Couldn’t find device with uuid EyputQ-41zk-PPdr-CFAb-WzHc-oAPM-HzIvgq.

Couldn’t find device with uuid 9Csh1n-xuWq-ZGw3-1Rtz-WMnT-3Mfx-HuZ2wJ.

Wrote out consistent volume group ora_dump

 

Above 5 Devices disappeared from the error messages from LVS command as give below:

 

[root@gurkuldinia-linux-server ~]# lvs -a -o +devices

Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.

Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.

Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.

Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.

Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.

Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.

Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.

Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.

Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.

Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.

Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.

Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.

LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices

dump01 ora_dump -wi-ao 29.99G /dev/dm-12(0)

admin01 ora_home -wi-ao 7.98G /dev/mapper/mpath12(513)

admin01 ora_home -wi-ao 7.98G /dev/mapper/mpath13(0)

app01 ora_home -wi-ao 20.00G /dev/mapper/mpath11(512)

app01 ora_home -wi-ao 20.00G /dev/mapper/mpath12(0)

datalocal01 ora_home -wi-ao 2.00G /dev/mapper/mpath11(0)

kdumpvol rootvg -wi-ao 15.66G /dev/cciss/c0d0p2(592)

rootvol rootvg -wi-ao 4.00G /dev/cciss/c0d0p2(0)

datalocalvol rootvg -wi-ao 2.00G /dev/cciss/c0d0p2(128)

swapvol rootvg -wi-ao 8.00G /dev/cciss/c0d0p2(336)

tmpvol rootvg -wi-ao 512.00M /dev/cciss/c0d0p2(192)

varvol rootvg -wi-ao 4.00G /dev/cciss/c0d0p2(208)

Removing Missing deviced from Volume group ora_home:

[root@gurkuldinia-linux-server bin]# vgreduce –removemissing ora_home

Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.

Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.

Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.

Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.

Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.

Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.

Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.

Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.

Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.

Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.

Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.

Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.

Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.

Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.

Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.

Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.

Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.

Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.

Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.

Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.

Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.

Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.

Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.

Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.

Wrote out consistent volume group ora_home

 

All Missing devices disappeared from the LVM :

[root@gurkuldinia-linux-server bin]# lvs -a -o +devices

LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices

dump01 ora_dump -wi-ao 29.99G /dev/dm-12(0)

admin01 ora_home -wi-ao 7.98G /dev/dm-6(513)

admin01 ora_home -wi-ao 7.98G /dev/dm-7(0)

app01 ora_home -wi-ao 20.00G /dev/dm-13(512)

app01 ora_home -wi-ao 20.00G /dev/dm-6(0)

datalocal01 ora_home -wi-ao 2.00G /dev/dm-13(0)

kdumpvol rootvg -wi-ao 15.66G /dev/cciss/c0d0p2(592)

rootvol rootvg -wi-ao 4.00G /dev/cciss/c0d0p2(0)

datalocalvol rootvg -wi-ao 2.00G /dev/cciss/c0d0p2(128)

swapvol rootvg -wi-ao 8.00G /dev/cciss/c0d0p2(336)

tmpvol rootvg -wi-ao 512.00M /dev/cciss/c0d0p2(192)

varvol rootvg -wi-ao 4.00G /dev/cciss/c0d0p2(208)

 

[root@gurkuldinia-linux-server bin]#

 

 

Ramdev

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

11 Responses

  1. Yogesh Raheja says:

    @Ram, very nice post, I have faced the same issue and the fix was same as posted by you. I faced it for SuSe Linux,

  2. Karthik says:

    Thanks a lot for the article.

    Some LVM modules related question was lingering in my mind and want to post it here.

    Having a question related to LVM and increasing the size of storage volumes.
    To simplify how to make LVM modules dynamically get the increased storage size so that we can increase the storage volumes on the fly.

    The scenario is:

    Underlying Storage devices are EMC devices and using Powerpath software.

    Storage Team extended a volume say /u01 by 50 G (ie old vol – 50 G and now new vol is 100 G).

    In this case as usual rescanning the LUNs was done on the server and then tried pvresize /dev/emcpowera to get the new PV size and its fine showing as 100 G.

    vgdisplay is fine and showing as 100 G but the problem is lvextend -L +50G /dev/vg1/u01 is giving error like this:

    Extending logical volume u01 to 100.00 GB
    device-mapper: reload ioctl failed: Invalid argument
    Failed to suspend u01

    Note: Also unmounted /u01 and tried lvextend and result is same and no luck.

    Not sure why lvm modules are not able to get the new size.

    And it required a reboot to fix but is there any way to do it online and want to avoid reboot?

    Thanks
    Karthik

  3. Karthik says:

    Missed this earlier:

    using devicemapper-lvm module and not sure whether we can tweak anything using dmsetup.

    Thanks
    Karthik

  4. Ramdev Ramdev says:

    @Karthik – after unmounting the filesystem try ” vgchange -a n volume-group” to completely restrict Vg access to the kernel. And then try the lvextend.

    If it is still not resolved, then please register into unixbook and post your problem along with pvdisplay and lvdisplay output. and also let us know the lvm version there. we will try to answer it there, so that it will be useful for other people for easy future reference.

  5. Karthik says:

    Thanks for your help on this Ramdev !!!

    I will check this and post it when this type of extension request comes to my team.

    Will register myself for Unixbook.

  6. Ramdev Ramdev says:

    @Karthik, Thanks for registering to Unix Book. Please post your questions and learnings to unix book. We really appreciate User contribution.

  7. Kiran M.S says:

    Hi Anna,

    Can we have a document for changing normal boot to SAN boot in SOL and Linux ?

  8. Ramdev Ramdev says:

    Hey kiran, you mean for which hardware? san boot wont support for every hardware. What is your requirement

  9. Santanu Roy says:

    It is useful one.

  1. September 17, 2015

    […] Read – Cleaning UP stale storage Devices from LVM, after Storage Reclaim […]

  2. July 22, 2016

    […] Read – Cleaning UP stale storage Devices from LVM, after Storage Reclaim […]

What is in your mind, about this post ? Leave a Reply

Close
  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us