Recovering after a root-partition resize gone wrong

Recently I tried to resize a LVM logical volume with its volume group and the physical disk belonging to the vg.Because the logical volume contained all partitions except for /boot I knew it would be hard.

I also knew that I had no backup and no snapshot of the virtual machine. :)

Here’s the initial situation:

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 499711 497664 243M 83 Linux
/dev/sda2 501758 204797951 204296194 97,4G 5 Extended
/dev/sda5 501760 204797951 204296192 97,4G 8e Linux LVM

Note that sda5 is a logical partition on top of the extended partition sda2.

Here’s the full volume group:

root@master:/home/sebastian# vgdisplay
  --- Volume group ---
  VG Name master-vg
  System ID
  Format lvm2
  Metadata Areas 1
  Metadata Sequence No 4
  VG Access read/write
  VG Status resizable
  MAX LV 0
  Cur LV 2
  Open LV 2
  Max PV 0
  Cur PV 1
  Act PV 1
  VG Size 97,41 GiB
  PE Size 4,00 MiB
  Total PE 24938
  Alloc PE / Size 24938 / 97,41 GiB
  Free PE / Size 0 / 0
  VG UUID yka0ZI-k7Q3-aeYD-vxU7-xajX-OVOy-qyogee

And here is the underlying physical volume:

root@master:/home/sebastian# pvdisplay
  --- Physical volume ---
  PV Name /dev/sda5
  VG Name master-vg
  PV Size 97,42 GiB / not usable 0
  Allocatable yes (but full)
  PE Size 4,00 MiB
  Total PE 24938
  Free PE 0
  Allocated PE 24938
  PV UUID TDHGY0-t1IO-qKuf-BQNf-fsWo-abut-PHZ8KJ

To resize the disk I used sfdisk because I read that fdisk alone couldn’t do the job.

Actually I followed this tutorial on Stack Exchange: https://unix.stackexchange.com/questions/379549/resize-extended-partition-lvm/379681#379681

I’ll paste the answer below for completeness:

To extend your sda5, you need to extend its container too, sda2. Using command-line tools, the simplest way to do this is to use sfdisk:

sfdisk /dev/sda

This will print the current partition table, which should match what you saw in fdisk:

Disk image: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe59ec859

Old situation:

Device Boot Start End Sectors Size Id Type
image1 * 2048 499711 497664 243M 83 Linux
image2 501758 104855551 104353794 49.8G 5 Extended
image5 501760 104855551 104353792 49.8G 83 Linux

Type 'help' to get more information.

At the >>> prompt, start re-defining all your partitions:

2048,497664,83,*

(this is the first partition: start sector, length in sectors, type, and a * to make it bootable). sfdisk will respond with

Created a new DOS disklabel with disk identifier 0x03408377.
Created a new partition 1 of type 'Linux' and of size 243 MiB.
image1 : 2048 499711 (243M) Linux

and prompts for image2 (sda2). Enter

501758,,5

which tells sfdisk to create an extended partition starting at sector 501758 and occupying all the available space after that; sfdisk will output

Created a new partition 2 of type 'Extended' and of size 99.8 GiB.
image2 : 501758 209715199 (99.8G) Extended

For image3, enter simply

501760

sfdisk will then output

Created a new partition 5 of type 'Linux' and of size 99.8 GiB.
image5 : 501760 209715199 (99.8G) Linux

and prompt for image6, which we don’t need, so enter

quit

which will cause sfdisk to print the new partition table and ask if you want to write it to disk:

New situation:

Device Boot Start End Sectors Size Id Type
image1 * 2048 499711 497664 243M 83 Linux
image2 501758 209715199 209213442 99.8G 5 Extended
image5 501760 209715199 209213440 99.8G 83 Linux

Do you want to write this to disk? [Y]es/[N]o:

If you’re convinced the starting sectors all match, and sda1 is still OK, press Y to write the partition table and return to your shell.

Once that’s done, run

pvresize /dev/sda5

to resize your LVM PV; you should then be able to use the newly-allocated disk space (in new LVs, or to extend existing LVs).

After doing the above steps I rebooted and of course the server did not come back up.I connected with the vmware kvm tools to the server and saw an error message (among others) that told me volume group not found. It seemed I destroyed the partition layour or disk.

To fix this issue I mounted a live-CD into the server and booted into it. There I had access to the disk and could try repairs.

The LVM tools (pvscan, pvck) did not find any information regarding lvms on the disks:

> pvck /dev/sda5
Could not find LVM label on /dev/sda5

Also there was no backup in /etc/lvm/backup that I could use. But the internet told me that the configuration of the lvm is stored in the first bytes/sectors of the partition and that it is possible to restore this configuration with dd:

dd if=/dev/sdc3 bs=512 count=255 skip=1

And this did work! I spat out the configuration that contained the volume group, physical disk and logical volumes on this disk.

I then used pvcreate to recreate the physical disk and make it known to the lvm:

> pvcreate --uuid "U3qzRV-jakK-0JFC-zVmq-F1aa-nxrg-9JYsXy" --restore /etc/lvm/backup/testvg /dev/sda5

I don’t really know if this command was necessary as I did use vgcfgbackup to restore the whole configuration.

vgcfgrestore master-vg

vgcfgrestore restores the metadata of a VG from a text back up file produced by vgcfgbackup. This writes VG metadata onto the devices specifed in back up file.

After that I could see the logical volume again and everything looked just like before I tried to resize the partition. I did one final reboot and the system came back online!My disaster recovery worked! Now all I need to do is resize the disk..

Some useful links I found during my repair attempts:

• https://unix.stackexchange.com/questions/420577/how-do-i-recover-a-lost-lvm-volume-group-not-found-by-vgscan-and-lvmdiskscan
• https://unix.stackexchange.com/questions/489182/lvm-pv-missing-after-reboot
• https://unix.stackexchange.com/questions/520688/broken-lvm-physical-volume-after-resizing-partition
• https://unix.stackexchange.com/questions/329582/drives-not-showing-when-sudo-pvdisplay
• https://serverfault.com/questions/275679/rescue-disk-is-unable-to-see-the-lvm-physical-volumes
• https://www.howtoforge.com/recover\_data\_from\_raid\_lvm\_partitions