Weasel disk partition plan
Currently I have two partitions, root and big. They are both managed by lvm and look like this:
root@weasel:/boot# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/Ubuntu-root
224G 107G 106G 51% /
/dev/mapper/vg00-big 230G 187G 32G 86% /big
It would be much more reasonable to have a small root partition that is easily backed up and separate partitions for /home, /store, and /big.
/store is a shared network-accessible space. /big is used for automatic backups of root, /home, and /store.
| Partition | Size |
|---|---|
| root | 4G |
| store | 99G |
| home | 3.4G |
| var | 1.2G |
| big | 187G |
| Total | 300G |
root is 107G (root + home + var + store), so if I would split them apart, root could be as small as 4G.
It would be handy if the root partition could be kept small enough to be backed up on a DVD.
So the new scheme would have 4 physical volumes:
| Device | Size | Use |
|---|---|---|
| /dev/hda | 250G | root, store, home, var |
| /dev/hdb | 250G | backup root, store, home, var |
| /dev/hdc | 500G | big |
| /dev/hdd | 500G | big, network backup |
The main idea is to partition disk failures, so I don't want to bunch all the disks together into a single volume group. When one disk goes bad, I don't want to lose access to part of another disk just because it is in the same volume group. I'm assuming that a volume group will fail when any physical volume in that group fails.
The logical volumes should be divided into primary and backup volumes and the sizes of the two volumes should match.
There should be a primary root, store, home, var, big. And there should be an equivalent backup root, store, home, var, big.
There should also be sufficient unformatted space in the primary volume group for a snapshot of the root partition so that an image can be taken and stored offline. But if root is small, that shouldn't be a problem.
So I'll need to move big (sdb1 vg00) to a 500G drive.
The big plan[edit]
| Physical volumes | sda (250G) | sdb (250G) DEAD | sdc (500G) | sdd (500G) |
|---|---|---|---|---|
| Volume groups | Ubuntu | netstore | netback | |
| Logical volumes | root,var,home, |
netstore,store | netback | |
| Percent free | 57% | 57% | 62% | 62% |
| Model number | ST3250823AS | ST3250823AS | WD500AACS | ST3500630AS |
| Serial number | 4ND06C1Q | 4ND05VJS | WD-WCASU1903539 | 5QG2B7HE |
Currently lvscan looks like this:
ACTIVE '/dev/vg00/big' [232.88 GB] inherit ACTIVE '/dev/Ubuntu/root' [226.77 GB] inherit ACTIVE '/dev/Ubuntu/swap_1' [5.88 GB] inherit
bring sdc and sdd online[edit]
create volume group netstore on sdc[edit]
pvcreate /dev/sdc vgcreate netstore /dev/sdc
{create logical volume netstore}
mkfs -t ext3 /dev/netstore/netstore
create volume group netback on sdd[edit]
pvcreate /dev/sdd vgcreate netback /dev/sdd
create logical volume netback on sdd[edit]
Determine the number of physical extents on the volume group.
root@weasel:~# vgdisplay netback | grep "Total PE" Total PE 119234
Use that number to create a logical volume that consumes the disk.
root@weasel:~# lvcreate -l 119234 netback -n netback Logical volume "netback" created
root@weasel:~# lvscan ACTIVE '/dev/netback/netback' [465.76 GB] inherit ACTIVE '/dev/netstore/netstore' [465.76 GB] inherit ACTIVE '/dev/vg00/big' [232.88 GB] inherit ACTIVE '/dev/Ubuntu/root' [226.77 GB] inherit ACTIVE '/dev/Ubuntu/swap_1' [5.88 GB] inherit
root@weasel:~# mkfs -t ext3 /dev/netback/netback
move big from sdb to sdc[edit]
mount /dev/netstore/netstore /big2 rsync -av /big/share /big2 rsync -av /big/weasel /big2 vol_id -u /dev/netstore/netstore cd /dev/disk/by-uuid ln -s ../../mapper/netstore-netstore 91fdcaaa-1ff7-4644-9976-10bda5f4919e
edit /etc/fstab to associate /big with the UUID for netstore
umount /big2 umount /big mount /big
move /store from root to volume group netstore[edit]
Root is on sda5, volume group Ubuntu.
root@weasel:/# pvscan PV /dev/sdd VG netback lvm2 [465.76 GB / 0 free] PV /dev/sdc VG netstore lvm2 [465.76 GB / 0 free] PV /dev/sdb1 VG vg00 lvm2 [232.88 GB / 0 free] PV /dev/sda5 VG Ubuntu lvm2 [232.64 GB / 0 free] Total: 4 [1.36 TB] / in use: 4 [1.36 TB] / in no VG: 0 [0 ]
Ubuntu does not have any free space to create a new logical volume, much less 3 new logical volumes. The first command that pops to mind is lvreduce, but according to the man page, I have to shrink the filesystem first:
DESCRIPTION
lvreduce allows you to reduce the size of a logical volume. Be careful when reducing a
logical volume’s size, because data in the reduced part is lost!!!
You should therefore ensure that any filesystem on the volume is resized before running
lvreduce so that the extents that are to be removed are not in use.
According to "man -k ext3", this looks like what I need:
resize2fs (8) - ext2/ext3 file system resizer
So I'll need to unmount root, resize the filesystem, and use lvreduce to create some space for new logical volumes for /home, /store, /var.
Root is a 224G device with 107G in use and 106G available. /store is about 100GB. It doesn't really belong on root. If I moved it to /big where it belongs, then root would be a total of about 10G. Then moving /home and /var should be much easier.
reduce netstore[edit]
So, to make room for /store on the netstore volume group, I unmounted /big and tried to reduce the filesystem size to 250G from 463G:
root@weasel:~# umount /big root@weasel:~# resize2fs -p /dev/mapper/netstore-netstore 250G resize2fs 1.40.8 (13-Mar-2008) Please run 'e2fsck -f /dev/mapper/netstore-netstore' first.
OK, I can do that.
root@weasel:~# e2fsck -f /dev/mapper/netstore-netstore
e2fsck 1.40.8 (13-Mar-2008)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/netstore-netstore: 829/30531584 files (27.6% non-contiguous), 20540088/122095616 blocks
root@weasel:~# resize2fs -p /dev/mapper/netstore-netstore 250G
resize2fs 1.40.8 (13-Mar-2008)
Resizing the filesystem on /dev/mapper/netstore-netstore to 65536000 (4k) blocks.
Begin pass 2 (max = 718383)
Relocating blocks XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Begin pass 3 (max = 3727)
Scanning inode table XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Begin pass 4 (max = 37)
Updating inode references XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
The filesystem on /dev/mapper/netstore-netstore is now 65536000 blocks long.
root@weasel:~# lvreduce -L 251G netstore/netstore
File descriptor 4 left open
File descriptor 5 left open
File descriptor 7 left open
File descriptor 8 left open
WARNING: Reducing active logical volume to 251.00 GB
THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce netstore? [y/n]: y
Reducing logical volume netstore to 251.00 GB
Logical volume netstore successfully resized
root@weasel:~# mount /big
root@weasel:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/Ubuntu-root
224G 107G 106G 51% /
varrun 1.5G 448K 1.5G 1% /var/run
varlock 1.5G 0 1.5G 0% /var/lock
udev 1.5G 80K 1.5G 1% /dev
devshm 1.5G 0 1.5G 0% /dev/shm
/dev/mapper/vg00-big 230G 125G 94G 58% /rootback
tmpfs 1.5G 43M 1.5G 3% /lib/modules/2.6.24-19-generic/volatile
/dev/mapper/netback-netback
463G 75G 365G 18% /bigback
/dev/mapper/netstore-netstore
249G 75G 164G 32% /big
create logical volume store on volume group netstore[edit]
Determine number of extents available and use that to create store:
root@weasel:~# vgdisplay netstore File descriptor 4 left open File descriptor 5 left open File descriptor 7 left open File descriptor 8 left open --- Volume group --- VG Name netstore System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 465.76 GB PE Size 4.00 MB Total PE 119234 Alloc PE / Size 64256 / 251.00 GB Free PE / Size 54978 / 214.76 GB VG UUID H7XaHj-lksL-9X0j-J5y8-ggUk-zqZc-zls3VV root@weasel:~# lvcreate -l 54978 netstore -n store File descriptor 4 left open File descriptor 5 left open File descriptor 7 left open File descriptor 8 left open Logical volume "store" created root@weasel:~# mkfs -t ext3 /dev/netstore/store root@weasel:~# mkdir /store2 root@weasel:~# mount /dev/mapper/netstore-store /store2 root@weasel:/# rsync -av /store/* /store2
move /var and /home from root[edit]
This will have to be done in single user mode, so I want to make sure I have the procedure down before I start.
root is now 7.7G
- of that, /home is 3.3G
- and /var is 1.2G
- resize root down to about 50G
- lvreduce Ubuntu down to 51G
- create logical volumes on Ubuntu for var and home
- mkfs for var and home
- mount new logical volumes as /home2 and /var2
- edit fstab to mount the new partitions with UUID
- rsync data to new logical volumes
- switch /home with /home2
- switch /var with /var2
- verify that there are still no differences with an rsync --dry-run
- delete the /var2 and /home2 directories
use LVM to take a snapshot of root and back it up[edit]
When done, lvscan should look like this:
/dev/Ubuntu/root /dev/Ubuntu/swap_1 /dev/Ubuntu/var /dev/Ubuntu/home /dev/netstore/store /dev/rootback/rootback /dev/netstore/netstore /dev/netback/netback
new plan[edit]
With the arrival of a drobo disk array, I no longer have the need for redundant storage provided by netstore and netback. And I would like to use that space in the rootback partition which is at 94% capacity now.
I have moved everything from netstore and netback to drobo. I need to determine the lvm command to release those logical volumes so that their resources may be applied to store.
The procedure should go like this:
- pvdisplay to ensure the physical volumes are not in use
- vgreduce to remove a physical volume from a volume group
- vgextend to add a physical volume to a volume group (do I have to initialize it?)
- lvextend to tell the logical volume to use the new space
I think I'll try this procedure through the graphical LVM manager. I haven't tried that yet and it looks like it will be quite slick.
rootback[edit]
needs more space
- volume group: vg00
- physical view: /dev/sdb
- logical view: big
bigback[edit]
has unused space
- volume group: netback
- physical view: sdd
- logical view: netback
procedure[edit]
- removed logical volume netback from volume group netback
- removed physical volume /dev/sdd from volume group netback
- /dev/sdd now shows as an unallocated volume
- add /dev/sdd to volume group vg00
- now vg00 contains
- 2 physical volumes: sdd and sdb1
- 1 logical volume: big
- lots of unused space
- edit of logical volume (slide the extent slider) allows increase of extents (as long as it can be unmounted)
And it all just worked. The LVM graphical admin tool was definitely the easier way to do this.
status[edit]
/rootback now has 143G available and 204G used for a comfortable 59% use percentage. Even better, the logical volume that backs it, has 328G available to be deployed whenever necessary. This logical volume now is spread across two physical hard disks so it will fail sooner, but this is just a backup volume anyway.
I also have the netstore-netstore logical volume that is no longer in use and can be configured into one of the other volumes as necessary. netstore represents 250G of available space.
drive failure on hdb[edit]
I found that my backups were yielding some errors about the filesystem being read-only. I found the problem was with /rootback. I tried to umount rootback and mount it normally, but it wouldn't umount, saying it was busy.
I rebooted and got a lot of errors on hdb and hdb1. Now LVM doesn't see hdb and it appears to palimpsest to be an unrecognized and unformatted partition. It also makes funny noises like a broken fan. The drive is ATA ST3250823AS and is supposed to be a member of a backup set (rootback).
remove physical device sdb from volume group rootback[edit]
I tried a vgscan to see what I have and got a ton of errors when it was trying to read from sdb
vgreduce rootback /dev/sdb
That failed because it couldn't read from sdb. It gave lots of I/O errors and then said 'rootback' doesn't exist.
I'd like to use sdd (netback) as a replacement since netback is currently offline.[edit]
I uncommented the entry for netback in /etc/fstab and then tried to mount /netback, but it said the UID didn't exist.
how do I get a list of the mappings from device to UID? blkid[edit]
root@weasel:~# blkid /dev/sda1: LABEL="/boot" UUID="94bf1476-39c7-45f4-8db4-1ffd85229e6a" SEC_TYPE="ext2" TYPE="ext3" /dev/sda5: UUID="hzY3In-XHIh-RGwS-1ilj-nL6P-Ajts-KmySkw" TYPE="LVM2_member" /dev/sdc: UUID="LohUWj-NZog-qAgB-6DyF-KhbV-pLQI-37MPeb" TYPE="LVM2_member" /dev/mapper/netstore-netstore: UUID="91fdcaaa-1ff7-4644-9976-10bda5f4919e" SEC_TYPE="ext2" TYPE="ext3" /dev/sdd: UUID="O6VzPq-IxUY-dzGf-NfSn-x3kD-uN3z-eQdvO4" TYPE="LVM2_member" /dev/mapper/netstore-store: UUID="f529dcf4-95e8-4a03-b142-f61500aaa882" SEC_TYPE="ext2" TYPE="ext3" /dev/mapper/Ubuntu-root: LABEL="/" UUID="29dc39eb-1250-4c52-8292-780078836162" SEC_TYPE="ext2" TYPE="ext3" /dev/mapper/Ubuntu-swap_1: UUID="abc3862a-f271-4898-a442-4544a1ff7014" TYPE="swap" /dev/sde: LABEL="drobo" UUID="7fcc72fe-0884-4d66-b4f3-962901875650" TYPE="ext3"
OK, that's cool, but incomplete. Here is what fstab is expecting:
# /etc/fstab: static file system information. # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults 0 0 # /dev/mapper/Ubuntu-root UUID=29dc39eb-1250-4c52-8292-780078836162 / ext3 relatime,errors=remount-ro 0 1 # /dev/mapper/vg00-big UUID=a76cb706-3e52-4c66-9d27-fba502ea0a58 /rootback ext3 relatime 0 2 # /dev/netstore/netstore UUID=91fdcaaa-1ff7-4644-9976-10bda5f4919e /big ext3 relatime 0 2 # /dev/netback/netback UUID=c70f5595-7dde-460c-af9f-622b683db38a /bigback ext3 relatime 0 2 # /dev/netstore/store UUID=f529dcf4-95e8-4a03-b142-f61500aaa882 /store ext3 relatime 0 2 # /dev/mapper/Ubuntu-swap_1 UUID=abc3862a-f271-4898-a442-4544a1ff7014 none swap sw 0 0 /dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec,utf8 0 0 /dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0 #/dev/sdc /media/sdc auto rw,user,noauto,exec,utf8 0 0 #/dev/sdc1 /media/usb ext3 relatime 0 0 #/dev/sde1 /media/iaudio auto rw,user,noauto,exec,utf8 0 0 # /dev/sde = drobo UUID=7fcc72fe-0884-4d66-b4f3-962901875650 /drobo ext3 defaults0 0
- I physically disconnected and removed the disk sdb and now lvscan and vgscan show something more sane:
root@weasel:~# lvscan Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'. Couldn't find all physical volumes for volume group vg00. Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'. Couldn't find all physical volumes for volume group vg00. Volume group "vg00" not found ACTIVE '/dev/netstore/netstore' [251.00 GB] inherit ACTIVE '/dev/netstore/store' [214.76 GB] inherit ACTIVE '/dev/Ubuntu/root' [226.77 GB] inherit ACTIVE '/dev/Ubuntu/swap_1' [5.88 GB] inherit
root@weasel:~# vgscan Reading all physical volumes. This may take a while... Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'. Couldn't find all physical volumes for volume group vg00. Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'. Couldn't find all physical volumes for volume group vg00. Volume group "vg00" not found Found volume group "netstore" using metadata type lvm2 Found volume group "Ubuntu" using metadata type lvm2
blkid doesn't report any uuid "mhOUHf..." so I don't know exactly what it is complaining about, but I know that volume group vg00 should contain
Perhaps a better tool than blkid is pvscan:
root@weasel:~# pvscan Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'. Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'. PV unknown device VG vg00 lvm2 [232.88 GB / 0 free] PV /dev/sdd VG vg00 lvm2 [465.76 GB / 328.42 GB free] PV /dev/sdc VG netstore lvm2 [465.76 GB / 0 free] PV /dev/sda5 VG Ubuntu lvm2 [232.64 GB / 0 free] Total: 4 [1.36 TB] / in use: 4 [1.36 TB] / in no VG: 0 [0 ]
I thought I replaced the volume group vg00 with rootback...
If I assume that vg00 is the current name for rootback, and the uuid that it can't find is associated with sdb, what does that mean?[edit]
That means that sdd and sdb make up rootback/vg00 and that I need to remove sdb somehow to allow sdd to take over. How do I remove a physical volume that doesn't exist anymore? My first shot in the dark is to try to remove sdb from vg00:
root@weasel:~# vgreduce vg00 /dev/sdb Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'. Couldn't find all physical volumes for volume group vg00. Couldn't find device with uuid 'mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu'. Couldn't find all physical volumes for volume group vg00. Volume group "vg00" doesn't exist
the graphical LVM manager[edit]
The graphical LVM manager only saw two volume groups (Ubuntu and netstore). It saw sdd as uninitialized, so I told it to initialize sdd and put it into a new volume group called rootback. That was easy. Now the command line looks like this:
root@weasel:~# pvscan PV /dev/sdd1 VG rootback lvm2 [465.76 GB / 465.76 GB free] PV /dev/sdc VG netstore lvm2 [465.76 GB / 0 free] PV /dev/sda5 VG Ubuntu lvm2 [232.64 GB / 0 free] Total: 3 [1.14 TB] / in use: 3 [1.14 TB] / in no VG: 0 [0 ] root@weasel:~# vgscan Reading all physical volumes. This may take a while... Found volume group "rootback" using metadata type lvm2 Found volume group "netstore" using metadata type lvm2 Found volume group "Ubuntu" using metadata type lvm2 root@weasel:~# lvscan ACTIVE '/dev/netstore/netstore' [251.00 GB] inherit ACTIVE '/dev/netstore/store' [214.76 GB] inherit ACTIVE '/dev/Ubuntu/root' [226.77 GB] inherit ACTIVE '/dev/Ubuntu/swap_1' [5.88 GB] inherit
I guess I still have to create a logical volume to make it useable. I can do that. I used the graphical LVM tool to create a new LV on VG rootback called rootback. I really need a better naming scheme. I slid the size slider all the way to the max and picked ext3 as the filesystem told it to mount and mount when rebooted and mount it on /rootback.
It took awhile.
While I'm in here, I should remove the unused netstore logical volume and give its space to store. netstore and netback were obsoleted by the addition of drobo. Actually, I just noticed that I'm using netstore as a backup of data from my work desktop, so I can't just repurpose it.
Now lvscan looks like this:
root@weasel:/big# lvscan ACTIVE '/dev/rootback/rootback' [465.76 GB] inherit ACTIVE '/dev/netstore/netstore' [251.00 GB] inherit ACTIVE '/dev/netstore/store' [214.76 GB] inherit ACTIVE '/dev/Ubuntu/root' [226.77 GB] inherit ACTIVE '/dev/Ubuntu/swap_1' [5.88 GB] inherit
Backups of home now look much healthier.