Linux LVM : Cleaning UP stale storage Devices from LVM, after Storage Reclaim
Other Learning Articles that you may like to read
Free Courses We Offer
Paid Training Courses we Offer
During the last weekend, I have noticed one of the interesting task performed by one of my collegue as part of the storage migration, i have captured the logs from his task. And the actual scenario is :
In any Storage migration ( with Host Side Data Copy ) below are the common steps that SA will follow :
- — Configure the New Storage Devices on the servers
- — Create new volumes from new Storage
- —Â Copy the Data from old volumes to new volumes ( either by volume level mirroring or Filesystem level Copy)
- — Unmount old filesystems and mount new filesystems – for application/ database access
- — Once Applicatin/DB works fine, remove old volumes
- — Remove old Storage from the Volume groups
- — Ask Storage team to reclaim the storage
- — Reconfiguration Reboot to remove any stale entries from the server device list
In the scenario that I am explaining below all the steps went fine except the step 6 and the server got rebooted directly after the storage reclaim. since the LVM cannot see it’s disks related to the Volume groups it started trowing “couldn’t find device errors”.
 As given in red below
[root@gurkuldinia-linux-server ~]# pvs -o +uuid
Couldn’t find device with uuid EyputQ-41zk-PPdr-CFAb-WzHc-oAPM-HzIvgq.
Couldn’t find device with uuid 9Csh1n-xuWq-ZGw3-1Rtz-WMnT-3Mfx-HuZ2wJ.
Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.
Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.
Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.
Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.
Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.
Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.
Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.
Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.
Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.
Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.
Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.
Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.
PVÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â VG Fmt Attr PSize PFree PV UUID
/dev/cciss/c0d0p2Â Â Â rootvg lvm2 a- 67.72G 33.56G try1O0-kPHY-3QTC-SEET-zC1f-065Q-OHuiq7
/dev/mapper/mpath10Â Â ora_dump lvm2 a- 30.09G 104.00M lagcCD-09e5-p6wA-u70y-2I69-Obbt-5m4eHa
/dev/mapper/mpath11Â Â ora_home lvm2 a- 20.09G 100.00M aCrz8Z-hP6v-YyVU-1PBx-l8R5-Vfmd-mR2Oe5
/dev/mapper/mpath12Â ora_home lvm2 a- 8.09G 100.00M zuOHAp-ukLe-F2qA-NiXp-SeEC-a5sp-mreep4
/dev/mapper/mpath13Â ora_home lvm2 a- 2.09G 108.00M eLmz3v-Akgj-epkH-dvzW-cKAY-G7f3-eQ7daU
unknown device ora_dump lvm2 a- 30.00G 30.00G EyputQ-41zk-PPdr-CFAb-WzHc-oAPM-HzIvgq
unknown device ora_dump lvm2 a- 30.00G 30.00G EyputQ-41zk-PPdr-CFAb-WzHc-oAPM-HzIvgq
unknown device ora_home lvm2 a- 2.00G 2.00G yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT
unknown device ora_home lvm2 a- 2.00G 2.00G yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT
unknown device ora_home lvm2 a- 2.00G 2.00G yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT
unknown device ora_home lvm2 a- 2.00G 2.00G yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT
 Ideally to remove any disks from the existing LVM volume group we do use vgreduce command as below
              # vgreduce volumegroup <devicepath>
Since the devices are completely missing here we couldn’t see actual devicepaths for missing devices. And then the Fix was performed as below using the below command
             # vgreduce –removemissing volume-group
   Removing missing devices from ora_dump volume Group:
[root@gurkuldinia-linux-server ~]# vgreduce –removemissing ora_dump
Couldn’t find device with uuid EyputQ-41zk-PPdr-CFAb-WzHc-oAPM-HzIvgq.
Couldn’t find device with uuid 9Csh1n-xuWq-ZGw3-1Rtz-WMnT-3Mfx-HuZ2wJ.
Couldn’t find device with uuid EyputQ-41zk-PPdr-CFAb-WzHc-oAPM-HzIvgq.
Couldn’t find device with uuid 9Csh1n-xuWq-ZGw3-1Rtz-WMnT-3Mfx-HuZ2wJ.
Wrote out consistent volume group ora_dump
Above 5 Devices disappeared from the error messages from LVS command as give below:
[root@gurkuldinia-linux-server ~]# lvs -a -o +devices
Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.
Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.
Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.
Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.
Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.
Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.
Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.
Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.
Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.
Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.
Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.
Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.
LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices
dump01 ora_dump -wi-ao 29.99G /dev/dm-12(0)
admin01 ora_home -wi-ao 7.98G /dev/mapper/mpath12(513)
admin01 ora_home -wi-ao 7.98G /dev/mapper/mpath13(0)
app01 ora_home -wi-ao 20.00G /dev/mapper/mpath11(512)
app01 ora_home -wi-ao 20.00G /dev/mapper/mpath12(0)
datalocal01 ora_home -wi-ao 2.00G /dev/mapper/mpath11(0)
kdumpvol rootvg -wi-ao 15.66G /dev/cciss/c0d0p2(592)
rootvol rootvg -wi-ao 4.00G /dev/cciss/c0d0p2(0)
datalocalvol rootvg -wi-ao 2.00G /dev/cciss/c0d0p2(128)
swapvol rootvg -wi-ao 8.00G /dev/cciss/c0d0p2(336)
tmpvol rootvg -wi-ao 512.00M /dev/cciss/c0d0p2(192)
varvol rootvg -wi-ao 4.00G /dev/cciss/c0d0p2(208)
Removing Missing deviced from Volume group ora_home:
[root@gurkuldinia-linux-server bin]# vgreduce –removemissing ora_home
Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.
Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.
Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.
Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.
Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.
Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.
Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.
Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.
Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.
Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.
Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.
Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.
Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.
Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.
Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.
Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.
Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.
Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.
Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.
Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.
Couldn’t find device with uuid yQzjp4-Fnpi-ZXzt-JGSM-Z1J6-E20h-CdhgWT.
Couldn’t find device with uuid dw95OO-yMyO-RLfd-AzT7-tSyA-oshl-jfQbR7.
Couldn’t find device with uuid TxyzL8-xKjh-SNPG-zXUS-pdHu-yZ90-JX9Unz.
Couldn’t find device with uuid 5VgQ7U-QeHv-PRQu-n6Kx-VGlT-1elL-AxyKOC.
Wrote out consistent volume group ora_home
All Missing devices disappeared from the LVM :
[root@gurkuldinia-linux-server bin]# lvs -a -o +devices
LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices
dump01 ora_dump -wi-ao 29.99G /dev/dm-12(0)
admin01 ora_home -wi-ao 7.98G /dev/dm-6(513)
admin01 ora_home -wi-ao 7.98G /dev/dm-7(0)
app01 ora_home -wi-ao 20.00G /dev/dm-13(512)
app01 ora_home -wi-ao 20.00G /dev/dm-6(0)
datalocal01 ora_home -wi-ao 2.00G /dev/dm-13(0)
kdumpvol rootvg -wi-ao 15.66G /dev/cciss/c0d0p2(592)
rootvol rootvg -wi-ao 4.00G /dev/cciss/c0d0p2(0)
datalocalvol rootvg -wi-ao 2.00G /dev/cciss/c0d0p2(128)
swapvol rootvg -wi-ao 8.00G /dev/cciss/c0d0p2(336)
tmpvol rootvg -wi-ao 512.00M /dev/cciss/c0d0p2(192)
varvol rootvg -wi-ao 4.00G /dev/cciss/c0d0p2(208)
[root@gurkuldinia-linux-server bin]#
@Ram, very nice post, I have faced the same issue and the fix was same as posted by you. I faced it for SuSe Linux,
Thanks a lot for the article.
Some LVM modules related question was lingering in my mind and want to post it here.
Having a question related to LVM and increasing the size of storage volumes.
To simplify how to make LVM modules dynamically get the increased storage size so that we can increase the storage volumes on the fly.
The scenario is:
Underlying Storage devices are EMC devices and using Powerpath software.
Storage Team extended a volume say /u01 by 50 G (ie old vol – 50 G and now new vol is 100 G).
In this case as usual rescanning the LUNs was done on the server and then tried pvresize /dev/emcpowera to get the new PV size and its fine showing as 100 G.
vgdisplay is fine and showing as 100 G but the problem is lvextend -L +50G /dev/vg1/u01 is giving error like this:
Extending logical volume u01 to 100.00 GB
device-mapper: reload ioctl failed: Invalid argument
Failed to suspend u01
Note: Also unmounted /u01 and tried lvextend and result is same and no luck.
Not sure why lvm modules are not able to get the new size.
And it required a reboot to fix but is there any way to do it online and want to avoid reboot?
Thanks
Karthik
Missed this earlier:
using devicemapper-lvm module and not sure whether we can tweak anything using dmsetup.
Thanks
Karthik
@Karthik – after unmounting the filesystem try ” vgchange -a n volume-group” to completely restrict Vg access to the kernel. And then try the lvextend.
If it is still not resolved, then please register into unixbook and post your problem along with pvdisplay and lvdisplay output. and also let us know the lvm version there. we will try to answer it there, so that it will be useful for other people for easy future reference.
Thanks for your help on this Ramdev !!!
I will check this and post it when this type of extension request comes to my team.
Will register myself for Unixbook.
@Karthik, Thanks for registering to Unix Book. Please post your questions and learnings to unix book. We really appreciate User contribution.
Hi Anna,
Can we have a document for changing normal boot to SAN boot in SOL and Linux ?
Hey kiran, you mean for which hardware? san boot wont support for every hardware. What is your requirement
It is useful one.