Activating second disk on ubuntu
I have two disks installed on my ubuntu dapper machine. Each is 250GB. But only one of them appears to be mounted and usable. This has been the case since my initial install of Ubuntu on this machine. I'm not sure how I confused the installer so badly. But I clearly did not understand the subtleties of LVM.
Collected data
root@weasel:/var/log# df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/Ubuntu-root
224G 87G 125G 42% /
varrun 1007M 208K 1007M 1% /var/run
varlock 1007M 4.0K 1007M 1% /var/lock
udev 1007M 108K 1007M 1% /dev
devshm 1007M 0 1007M 0% /dev/shm
lrm 1007M 22M 986M 3% /lib/modules/2.6.15-26-amd64-k8/volatile
/dev/sda1 228M 102M 115M 48% /boot
Logical volume manager (LVM) has something to do with this.
root@weasel:/var/log# fdisk -l Disk /dev/sda: 250.0 GB, 250059350016 bytes 255 heads, 63 sectors/track, 30401 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 31 248976 83 Linux /dev/sda2 32 30401 243947025 5 Extended /dev/sda5 32 30401 243946993+ 8e Linux LVM Disk /dev/sdb: 250.0 GB, 250059350016 bytes 255 heads, 63 sectors/track, 30401 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 * 1 30401 244196001 8e Linux LVM
The end of the LVM Howto has some common tasks and how to do them. I noticed the pvdisplay command:
root@weasel:/etc/lvm# pvdisplay /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error Couldn't find device with uuid 'Y67lyo-mEdN-F0un-yPLl-HCTa-hWLC-mCentd'. --- Physical volume --- PV Name unknown device VG Name VolGroup00 PV Size 232.78 GB / not usable 0 Allocatable yes (but full) PE Size (KByte) 32768 Total PE 7449 Free PE 0 Allocated PE 7449 PV UUID Y67lyo-mEdN-F0un-yPLl-HCTa-hWLC-mCentd --- Physical volume --- PV Name /dev/sdb1 VG Name VolGroup00 PV Size 232.88 GB / not usable 0 Allocatable yes PE Size (KByte) 32768 Total PE 7452 Free PE 2 Allocated PE 7450 PV UUID mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu --- Physical volume --- PV Name /dev/sda5 VG Name Ubuntu PV Size 232.64 GB / not usable 0 Allocatable yes (but full) PE Size (KByte) 4096 Total PE 59557 Free PE 0 Allocated PE 59557 PV UUID hzY3In-XHIh-RGwS-1ilj-nL6P-Ajts-KmySkw
There's a volume group display command as well:
root@weasel:/etc/lvm# vgdisplay /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error Couldn't find device with uuid 'Y67lyo-mEdN-F0un-yPLl-HCTa-hWLC-mCentd'. Couldn't find all physical volumes for volume group VolGroup00. /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error Couldn't find device with uuid 'Y67lyo-mEdN-F0un-yPLl-HCTa-hWLC-mCentd'. Couldn't find all physical volumes for volume group VolGroup00. Volume group "VolGroup00" doesn't exist --- Volume group --- VG Name Ubuntu System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 2 Max PV 0 Cur PV 1 Act PV 1 VG Size 232.64 GB PE Size 4.00 MB Total PE 59557 Alloc PE / Size 59557 / 232.64 GB Free PE / Size 0 / 0 VG UUID 85vnO6-FtRR-SZnS-s0UQ-aFNf-OY5X-ZiwMm0
Physical volume scan (pvscan) shows this:
root@weasel:/etc/lvm# pvscan /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error Couldn't find device with uuid 'Y67lyo-mEdN-F0un-yPLl-HCTa-hWLC-mCentd'. PV unknown device VG VolGroup00 lvm2 [232.78 GB / 0 free] PV /dev/sdb1 VG VolGroup00 lvm2 [232.88 GB / 64.00 MB free] PV /dev/sda5 VG Ubuntu lvm2 [232.64 GB / 0 free] Total: 3 [698.30 GB] / in use: 3 [698.30 GB] / in no VG: 0 [0 ]
Analysis
It looks to me like I have two volume groups named VolGroup00 and Ubuntu. The Ubuntu volume group is working, but the VolGroup00 is not. Device /dev/sda5 is associated with Ubuntu and /dev/sdb1 is associated with VolGroup00.
The cunning plan
I think the steps toward a working configuration are these:
- remove logical volumes in VolGroup00?
lvremove /dev/???
- remove volume group VolGroup00
vgchange -a n VolGroup00 vgremove VolGroup00
The above commands failed with the error that it couldn't find VolGroup00. But it recommended that I could consider using the command: vgreduce --removemissing. I did the following:
vgreduce --test --removemissing VolGroup00 vgreduce --removemissing VolGroup00 vgreduce --test --removemissing VolGroup00
Luckily, my existing data in the Ubuntu volume group is un-destroyed.
Now vgscan says this:
root@weasel:/# vgscan Reading all physical volumes. This may take a while... /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error Found volume group "VolGroup00" using metadata type lvm2 Found volume group "Ubuntu" using metadata type lvm2
- initialize /dev/sdb1?
pvcreate -M2 /dev/sdb
Unfortunately, I got this back:
root@weasel:/# pvcreate -M2 /dev/sdb Device /dev/sdb not found (or ignored by filtering).
But pvscan says this:
root@weasel:/# pvscan /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error PV /dev/sda5 VG Ubuntu lvm2 [232.64 GB / 0 free] PV /dev/sdb1 lvm2 [232.88 GB] Total: 2 [465.53 GB] / in use: 1 [232.64 GB] / in no VG: 1 [232.88 GB]
So I guess I don't need to pvcreate anything. But how do I go from here to having useable mountpoint?
I did a vgcreate like so:
root@weasel:/# vgcreate vg00 /dev/sdb1 /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error Volume group "vg00" successfully created root@weasel:/# pvscan /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error PV /dev/sdb1 VG vg00 lvm2 [232.88 GB / 232.88 GB free] PV /dev/sda5 VG Ubuntu lvm2 [232.64 GB / 0 free] Total: 2 [465.53 GB] / in use: 2 [465.53 GB] / in no VG: 0 [0 ]
I then used vgdisplay to find the Total PE of vg00 (59618). I then fed this to lvcreate like so:
root@weasel:/# lvcreate -l 59618 vg00 -n big /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error Logical volume "big" created
I hope that I/O error goes away after a reboot or something...
Now lvscan says this:
root@weasel:/# lvscan /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error ACTIVE '/dev/vg00/big' [232.88 GB] inherit ACTIVE '/dev/Ubuntu/root' [226.77 GB] inherit ACTIVE '/dev/Ubuntu/swap_1' [5.88 GB] inherit
I formatted this logical volume like this:
root@weasel:/# mkfs -t ext3 /dev/vg00/big
mke2fs 1.38 (30-Jun-2005)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
30539776 inodes, 61048832 blocks
3052441 blocks (5.00%) reserved for the super user
First data block=0
1864 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 32 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
Then I could copy the line for my root partition in fstab like this:
/dev/mapper/Ubuntu-root / ext3 defaults,errors=remount-ro 0 1 /dev/mapper/vg00-big /big ext3 defaults,errors=remount-ro 0 1
and then issue the command "mount /big" and it worked.
- I no longer intend to assign this new disk to the existing volume group. Instead, I'm going to make a different mount point and filesystem for editing video. Hopefully, this will allow me to partition my disk failures.
kernel update causes disk failure
Perhaps it was just the reboot following the kernel update, but my /big filesystem disappeared. I didn't notice a problem until my root filesystem filled up when my daily synch between root and /big turned into a synch between root and root, causing root to fill up.
I tried wiggling the wires and unplugging the disk that hosts the failing filesystem. But that caused the system to be unbootable. Ugh. (My guess here is that the BIOS renumbered the disks when it saw that there was a change and tried to boot from the wrong disk. I had to go into the BIOS utility to change the disk boot priority to fix this.)
I had an old Ubuntu install CD on hand that let me boot the machine and get to the point of editing the LVM configuration, which was probably unnecessary. Luckily, I didn't make any substantive changes. Eventually, I found that tweaking the LVM config was not getting me anywhere and I went to the BIOS to get things working again. At least I got it to boot.
Now I'm at the point of having a bootable system with both disks recognized by the BIOS, but the non-root disk is not visible to the operating system.
df -k shows only filesystems from /dev/sda. The one big filesystem on /dev/sdb is missing. My fstab curiously did not have an entry for the /big filesystem. Perhaps Ubuntu helpfully removed it for me. I just added it back and now all is well again.
However, many reboots convinced comcast to give me a new IP address. So now I have to reconfigure for the new IP number. I need to make a wiki page on how to do that.
Can't mount second disk
For a time I had my backup disk (/big) disconnected and commented out of fstab. When I plugged it back in and uncommented the fstab entry, I found that I couldn't mount the disk.
When I try to mount it I get this:
root@weasel:~# mount /big
mount: wrong fs type, bad option, bad superblock on /dev/mapper/vg00-big,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
In /var/log/syslog, I see this:
Jun 22 22:51:17 weasel kernel: [ 1427.167819] EXT3-fs error (device dm-0): ext3_check_descriptors: Block bitmap for group 0 not in group (block 3858766779)! Jun 22 22:51:17 weasel kernel: [ 1427.178580] EXT3-fs: group descriptors corrupted!
This appears to be working to recover the corruption:
root@weasel:/dev/vg00# fsck /dev/vg00/big fsck 1.40.8 (13-Mar-2008) e2fsck 1.40.8 (13-Mar-2008) fsck.ext3: Group descriptors look bad... trying backup blocks... ext3 recovery flag is clear, but journal has data. Recovery flag not set in backup superblock, so running journal anyway. /dev/vg00/big: recovering journal Pass 1: Checking inodes, blocks, and sizes
It is taking a pretty long time. I should probably walk away so I'm not tempted to interrupt it or poke at it.
Eventually, it started to ask permission to fix group numbers, free inodes, etc.
About 1800 groups needed to be repaired. In the end it looked like this:
Free inodes count wrong for group #1855 (16383, counted=16384). Fix<y>? yes Free inodes count wrong for group #1860 (16381, counted=16382). Fix<y>? yes Free inodes count wrong (30459026, counted=30443879). Fix<y>? yes /dev/vg00/big: ***** FILE SYSTEM WAS MODIFIED ***** /dev/vg00/big: 95897/30539776 files (4.2% non-contiguous), 50861505/61048832 blocks