Weasel disk partition plan

From Finninday
Jump to: navigation, search

Currently I have two partitions, root and big. They are both managed by lvm and look like this:

root@weasel:/boot# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/Ubuntu-root
                      224G  107G  106G  51% /
/dev/mapper/vg00-big  230G  187G   32G  86% /big

It would be much more reasonable to have a small root partition that is easily backed up and separate partitions for /home, /store, and /big.

/store is a shared network-accessible space. /big is used for automatic backups of root, /home, and /store.

Partition Size
root 4G
store 99G
home 3.4G
var 1.2G
big 187G
Total 300G

root is 107G (root + home + var + store), so if I would split them apart, root could be as small as 4G.

It would be handy if the root partition could be kept small enough to be backed up on a DVD.

So the new scheme would have 4 physical volumes:

Device Size Use
/dev/hda 250G root, store, home, var
/dev/hdb 250G backup root, store, home, var
/dev/hdc 500G big
/dev/hdd 500G big, network backup

The main idea is to partition disk failures, so I don't want to bunch all the disks together into a single volume group. When one disk goes bad, I don't want to lose access to part of another disk just because it is in the same volume group. I'm assuming that a volume group will fail when any physical volume in that group fails.

The logical volumes should be divided into primary and backup volumes and the sizes of the two volumes should match.

There should be a primary root, store, home, var, big. And there should be an equivalent backup root, store, home, var, big.

There should also be sufficient unformatted space in the primary volume group for a snapshot of the root partition so that an image can be taken and stored offline. But if root is small, that shouldn't be a problem.

So I'll need to move big (sdb1 vg00) to a 500G drive.

The big plan

Physical volumes sda (250G) sdb (250G) DEAD sdc (500G) sdd (500G)
Volume groups Ubuntu vg00 rootback netstore netback
Logical volumes root,var,home,store,swap big rootback netstore,store netback
Percent free 57% 57% 62% 62%
Model number ST3250823AS ST3250823AS WD500AACS ST3500630AS
Serial number 4ND06C1Q 4ND05VJS WD-WCASU1903539 5QG2B7HE

Currently lvscan looks like this:

 ACTIVE            '/dev/vg00/big' [232.88 GB] inherit
 ACTIVE            '/dev/Ubuntu/root' [226.77 GB] inherit
 ACTIVE            '/dev/Ubuntu/swap_1' [5.88 GB] inherit

bring sdc and sdd online

create volume group netstore on sdc

pvcreate /dev/sdc
vgcreate netstore /dev/sdc

{create logical volume netstore}

mkfs -t ext3 /dev/netstore/netstore

create volume group netback on sdd

pvcreate /dev/sdd
vgcreate netback /dev/sdd

create logical volume netback on sdd

Determine the number of physical extents on the volume group.

root@weasel:~# vgdisplay netback | grep "Total PE"
 Total PE              119234

Use that number to create a logical volume that consumes the disk.

root@weasel:~# lvcreate -l 119234 netback -n netback
 Logical volume "netback" created
root@weasel:~# lvscan
 ACTIVE            '/dev/netback/netback' [465.76 GB] inherit
 ACTIVE            '/dev/netstore/netstore' [465.76 GB] inherit
 ACTIVE            '/dev/vg00/big' [232.88 GB] inherit
 ACTIVE            '/dev/Ubuntu/root' [226.77 GB] inherit
 ACTIVE            '/dev/Ubuntu/swap_1' [5.88 GB] inherit
root@weasel:~# mkfs -t ext3 /dev/netback/netback

move big from sdb to sdc

mount /dev/netstore/netstore /big2
rsync -av /big/share /big2
rsync -av /big/weasel /big2
vol_id -u /dev/netstore/netstore
cd /dev/disk/by-uuid
ln -s ../../mapper/netstore-netstore 91fdcaaa-1ff7-4644-9976-10bda5f4919e

edit /etc/fstab to associate /big with the UUID for netstore

umount /big2 
umount /big
mount /big

move /store from root to volume group netstore

Root is on sda5, volume group Ubuntu.

root@weasel:/# pvscan
 PV /dev/sdd    VG netback    lvm2 [465.76 GB / 0    free]
 PV /dev/sdc    VG netstore   lvm2 [465.76 GB / 0    free]
 PV /dev/sdb1   VG vg00       lvm2 [232.88 GB / 0    free]
 PV /dev/sda5   VG Ubuntu     lvm2 [232.64 GB / 0    free]
 Total: 4 [1.36 TB] / in use: 4 [1.36 TB] / in no VG: 0 [0   ]

Ubuntu does not have any free space to create a new logical volume, much less 3 new logical volumes. The first command that pops to mind is lvreduce, but according to the man page, I have to shrink the filesystem first:

DESCRIPTION
      lvreduce  allows  you  to reduce the size of a logical volume.  Be careful when reducing a
      logical volume’s size, because data in the reduced part is lost!!!
      You should therefore ensure that any filesystem on the volume is  resized  before  running
      lvreduce so that the extents that are to be removed are not in use.

According to "man -k ext3", this looks like what I need:

resize2fs (8)        - ext2/ext3 file system resizer

So I'll need to unmount root, resize the filesystem, and use lvreduce to create some space for new logical volumes for /home, /store, /var.

Root is a 224G device with 107G in use and 106G available. /store is about 100GB. It doesn't really belong on root. If I moved it to /big where it belongs, then root would be a total of about 10G. Then moving /home and /var should be much easier.

reduce netstore

So, to make room for /store on the netstore volume group, I unmounted /big and tried to reduce the filesystem size to 250G from 463G:

root@weasel:~# umount /big
root@weasel:~# resize2fs -p /dev/mapper/netstore-netstore 250G
resize2fs 1.40.8 (13-Mar-2008)
Please run 'e2fsck -f /dev/mapper/netstore-netstore' first.

OK, I can do that.

root@weasel:~# e2fsck -f /dev/mapper/netstore-netstore
e2fsck 1.40.8 (13-Mar-2008)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/netstore-netstore: 829/30531584 files (27.6% non-contiguous), 20540088/122095616 blocks
root@weasel:~# resize2fs -p /dev/mapper/netstore-netstore 250G
resize2fs 1.40.8 (13-Mar-2008)
Resizing the filesystem on /dev/mapper/netstore-netstore to 65536000 (4k) blocks.
Begin pass 2 (max = 718383)
Relocating blocks             XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Begin pass 3 (max = 3727)
Scanning inode table          XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Begin pass 4 (max = 37)
Updating inode references     XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
The filesystem on /dev/mapper/netstore-netstore is now 65536000 blocks long.
root@weasel:~# lvreduce -L 251G netstore/netstore
File descriptor 4 left open
File descriptor 5 left open
File descriptor 7 left open
File descriptor 8 left open
  WARNING: Reducing active logical volume to 251.00 GB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce netstore? [y/n]: y
  Reducing logical volume netstore to 251.00 GB
  Logical volume netstore successfully resized
root@weasel:~# mount /big
root@weasel:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/Ubuntu-root
                      224G  107G  106G  51% /
varrun                1.5G  448K  1.5G   1% /var/run
varlock               1.5G     0  1.5G   0% /var/lock
udev                  1.5G   80K  1.5G   1% /dev
devshm                1.5G     0  1.5G   0% /dev/shm
/dev/mapper/vg00-big  230G  125G   94G  58% /rootback
tmpfs                 1.5G   43M  1.5G   3% /lib/modules/2.6.24-19-generic/volatile
/dev/mapper/netback-netback
                      463G   75G  365G  18% /bigback
/dev/mapper/netstore-netstore
                      249G   75G  164G  32% /big

create logical volume store on volume group netstore

Determine number of extents available and use that to create store:

root@weasel:~# vgdisplay netstore
File descriptor 4 left open
File descriptor 5 left open
File descriptor 7 left open
File descriptor 8 left open
  --- Volume group ---
  VG Name               netstore
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               465.76 GB
  PE Size               4.00 MB
  Total PE              119234
  Alloc PE / Size       64256 / 251.00 GB
  Free  PE / Size       54978 / 214.76 GB
  VG UUID               H7XaHj-lksL-9X0j-J5y8-ggUk-zqZc-zls3VV
   
root@weasel:~# lvcreate -l 54978 netstore -n store
File descriptor 4 left open
File descriptor 5 left open
File descriptor 7 left open
File descriptor 8 left open
  Logical volume "store" created

root@weasel:~# mkfs -t ext3 /dev/netstore/store

root@weasel:~# mkdir /store2
root@weasel:~# mount /dev/mapper/netstore-store /store2
root@weasel:/# rsync -av /store/* /store2

move /var and /home from root

This will have to be done in single user mode, so I want to make sure I have the procedure down before I start.

root is now 7.7G

of that, /home is 3.3G
and /var is 1.2G
  • resize root down to about 50G
  • lvreduce Ubuntu down to 51G
  • create logical volumes on Ubuntu for var and home
  • mkfs for var and home
  • mount new logical volumes as /home2 and /var2
  • edit fstab to mount the new partitions with UUID
  • rsync data to new logical volumes
  • switch /home with /home2
  • switch /var with /var2
  • verify that there are still no differences with an rsync --dry-run
  • delete the /var2 and /home2 directories

use LVM to take a snapshot of root and back it up

When done, lvscan should look like this:

/dev/Ubuntu/root
/dev/Ubuntu/swap_1
/dev/Ubuntu/var
/dev/Ubuntu/home
/dev/netstore/store
/dev/rootback/rootback
/dev/netstore/netstore
/dev/netback/netback

new plan

With the arrival of a drobo disk array, I no longer have the need for redundant storage provided by netstore and netback. And I would like to use that space in the rootback partition which is at 94% capacity now.

I have moved everything from netstore and netback to drobo. I need to determine the lvm command to release those logical volumes so that their resources may be applied to store.

The procedure should go like this:

  • pvdisplay to ensure the physical volumes are not in use
  • vgreduce to remove a physical volume from a volume group
  • vgextend to add a physical volume to a volume group (do I have to initialize it?)
  • lvextend to tell the logical volume to use the new space

I think I'll try this procedure through the graphical LVM manager. I haven't tried that yet and it looks like it will be quite slick.

rootback

needs more space

volume group: vg00
physical view: /dev/sdb
logical view: big

bigback

has unused space

volume group: netback
physical view: sdd
logical view: netback

procedure

  • removed logical volume netback from volume group netback
  • removed physical volume /dev/sdd from volume group netback
  • /dev/sdd now shows as an unallocated volume
  • add /dev/sdd to volume group vg00
  • now vg00 contains
    • 2 physical volumes: sdd and sdb1
    • 1 logical volume: big
    • lots of unused space
  • edit of logical volume (slide the extent slider) allows increase of extents (as long as it can be unmounted)

And it all just worked. The LVM graphical admin tool was definitely the easier way to do this.

status

/rootback now has 143G available and 204G used for a comfortable 59% use percentage. Even better, the logical volume that backs it, has 328G available to be deployed whenever necessary. This logical volume now is spread across two physical hard disks so it will fail sooner, but this is just a backup volume anyway.

I also have the netstore-netstore logical volume that is no longer in use and can be configured into one of the other volumes as necessary. netstore represents 250G of available space.

drive failure on hdb

I found that my backups were yielding some errors about the filesystem being read-only. I found the problem was with /rootback. I tried to umount rootback and mount it normally, but it wouldn't umount, saying it was busy.

I rebooted and got a lot of errors on hdb and hdb1. Now LVM doesn't see hdb and it appears to palimpsest to be an unrecognized and unformatted partition. It also makes funny noises like a broken fan. The drive is ATA ST3250823AS and is supposed to be a member of a backup set (rootback).

remove physical device sdb from volume group rootback

I tried a vgscan to see what I have and got a ton of errors when it was trying to read from sdb

vgreduce rootback /dev/sdb

That failed because it couldn't read from sdb. It gave lots of I/O errors and then said 'rootback' doesn't exist.

I'd like to use sdd (netback) as a replacement since netback is currently offline.

 I uncommented the entry for netback in /etc/fstab and then tried to mount /netback, but it said the UID didn't exist.

how do I get a list of the mappings from device to UID? blkid

root@weasel:~# blkid
/dev/sda1: LABEL="/boot" UUID="94bf1476-39c7-45f4-8db4-1ffd85229e6a" SEC_TYPE="ext2" TYPE="ext3" 
/dev/sda5: UUID="hzY3In-XHIh-RGwS-1ilj-nL6P-Ajts-KmySkw" TYPE="LVM2_member" 
/dev/sdc: UUID="LohUWj-NZog-qAgB-6DyF-KhbV-pLQI-37MPeb" TYPE="LVM2_member" 
/dev/mapper/netstore-netstore: UUID="91fdcaaa-1ff7-4644-9976-10bda5f4919e" SEC_TYPE="ext2" TYPE="ext3" 
/dev/sdd: UUID="O6VzPq-IxUY-dzGf-NfSn-x3kD-uN3z-eQdvO4" TYPE="LVM2_member" 
/dev/mapper/netstore-store: UUID="f529dcf4-95e8-4a03-b142-f61500aaa882" SEC_TYPE="ext2" TYPE="ext3" 
/dev/mapper/Ubuntu-root: LABEL="/" UUID="29dc39eb-1250-4c52-8292-780078836162" SEC_TYPE="ext2" TYPE="ext3" 
/dev/mapper/Ubuntu-swap_1: UUID="abc3862a-f271-4898-a442-4544a1ff7014" TYPE="swap" 
/dev/sde: LABEL="drobo" UUID="7fcc72fe-0884-4d66-b4f3-962901875650" TYPE="ext3" 

OK, that's cool, but incomplete. Here is what fstab is expecting:

# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# /dev/mapper/Ubuntu-root
UUID=29dc39eb-1250-4c52-8292-780078836162 /               ext3    relatime,errors=remount-ro 0       1
# /dev/mapper/vg00-big
UUID=a76cb706-3e52-4c66-9d27-fba502ea0a58 /rootback            ext3    relatime        0       2
# /dev/netstore/netstore
UUID=91fdcaaa-1ff7-4644-9976-10bda5f4919e /big            ext3    relatime        0       2
# /dev/netback/netback
UUID=c70f5595-7dde-460c-af9f-622b683db38a /bigback            ext3    relatime        0       2
# /dev/netstore/store
UUID=f529dcf4-95e8-4a03-b142-f61500aaa882 /store            ext3    relatime        0       2
# /dev/mapper/Ubuntu-swap_1
UUID=abc3862a-f271-4898-a442-4544a1ff7014 none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto,exec,utf8 0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto,exec,utf8 0       0
#/dev/sdc        /media/sdc  auto    rw,user,noauto,exec,utf8 0       0
#/dev/sdc1       /media/usb  ext3    relatime 0       0
#/dev/sde1		/media/iaudio	auto	rw,user,noauto,exec,utf8 0	0
# /dev/sde = drobo
UUID=7fcc72fe-0884-4d66-b4f3-962901875650		/drobo	ext3	defaults0 0
  • I physically disconnected and removed the disk sdb and now lvscan and vgscan show something more sane:
root@weasel:~# lvscan
  Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'.
  Couldn't find all physical volumes for volume group vg00.
  Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'.
  Couldn't find all physical volumes for volume group vg00.
  Volume group "vg00" not found
  ACTIVE            '/dev/netstore/netstore' [251.00 GB] inherit
  ACTIVE            '/dev/netstore/store' [214.76 GB] inherit
  ACTIVE            '/dev/Ubuntu/root' [226.77 GB] inherit
  ACTIVE            '/dev/Ubuntu/swap_1' [5.88 GB] inherit
root@weasel:~# vgscan
  Reading all physical volumes.  This may take a while...
  Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'.
  Couldn't find all physical volumes for volume group vg00.
  Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'.
  Couldn't find all physical volumes for volume group vg00.
  Volume group "vg00" not found
  Found volume group "netstore" using metadata type lvm2
  Found volume group "Ubuntu" using metadata type lvm2

blkid doesn't report any uuid "mhOUHf..." so I don't know exactly what it is complaining about, but I know that volume group vg00 should contain

Perhaps a better tool than blkid is pvscan:

root@weasel:~# pvscan
  Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'.
  Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'.
  PV unknown device   VG vg00       lvm2 [232.88 GB / 0    free]
  PV /dev/sdd         VG vg00       lvm2 [465.76 GB / 328.42 GB free]
  PV /dev/sdc         VG netstore   lvm2 [465.76 GB / 0    free]
  PV /dev/sda5        VG Ubuntu     lvm2 [232.64 GB / 0    free]
  Total: 4 [1.36 TB] / in use: 4 [1.36 TB] / in no VG: 0 [0   ]

I thought I replaced the volume group vg00 with rootback...

If I assume that vg00 is the current name for rootback, and the uuid that it can't find is associated with sdb, what does that mean?

That means that sdd and sdb make up rootback/vg00 and that I need to remove sdb somehow to allow sdd to take over. How do I remove a physical volume that doesn't exist anymore? My first shot in the dark is to try to remove sdb from vg00:

root@weasel:~# vgreduce vg00 /dev/sdb
  Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'.
  Couldn't find all physical volumes for volume group vg00.
  Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'.
  Couldn't find all physical volumes for volume group vg00.
  Volume group "vg00" doesn't exist

the graphical LVM manager

The graphical LVM manager only saw two volume groups (Ubuntu and netstore). It saw sdd as uninitialized, so I told it to initialize sdd and put it into a new volume group called rootback. That was easy. Now the command line looks like this:

root@weasel:~# pvscan
  PV /dev/sdd1   VG rootback   lvm2 [465.76 GB / 465.76 GB free]
  PV /dev/sdc    VG netstore   lvm2 [465.76 GB / 0    free]
  PV /dev/sda5   VG Ubuntu     lvm2 [232.64 GB / 0    free]
  Total: 3 [1.14 TB] / in use: 3 [1.14 TB] / in no VG: 0 [0   ]
root@weasel:~# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "rootback" using metadata type lvm2
  Found volume group "netstore" using metadata type lvm2
  Found volume group "Ubuntu" using metadata type lvm2
root@weasel:~# lvscan
  ACTIVE            '/dev/netstore/netstore' [251.00 GB] inherit
  ACTIVE            '/dev/netstore/store' [214.76 GB] inherit
  ACTIVE            '/dev/Ubuntu/root' [226.77 GB] inherit
  ACTIVE            '/dev/Ubuntu/swap_1' [5.88 GB] inherit

I guess I still have to create a logical volume to make it useable. I can do that. I used the graphical LVM tool to create a new LV on VG rootback called rootback. I really need a better naming scheme. I slid the size slider all the way to the max and picked ext3 as the filesystem told it to mount and mount when rebooted and mount it on /rootback.

It took awhile.

While I'm in here, I should remove the unused netstore logical volume and give its space to store. netstore and netback were obsoleted by the addition of drobo. Actually, I just noticed that I'm using netstore as a backup of data from my work desktop, so I can't just repurpose it.

Now lvscan looks like this:

root@weasel:/big# lvscan
  ACTIVE            '/dev/rootback/rootback' [465.76 GB] inherit
  ACTIVE            '/dev/netstore/netstore' [251.00 GB] inherit
  ACTIVE            '/dev/netstore/store' [214.76 GB] inherit
  ACTIVE            '/dev/Ubuntu/root' [226.77 GB] inherit
  ACTIVE            '/dev/Ubuntu/swap_1' [5.88 GB] inherit

Backups of home now look much healthier.