Activating second disk on ubuntu

I have two disks installed on my ubuntu dapper machine. Each is 250GB. But only one of them appears to be mounted and usable. This has been the case since my initial install of Ubuntu on this machine. I'm not sure how I confused the installer so badly. But I clearly did not understand the subtleties of LVM.

Collected data

root@weasel:/var/log# df -kh
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/Ubuntu-root
                      224G   87G  125G  42% /
varrun               1007M  208K 1007M   1% /var/run
varlock              1007M  4.0K 1007M   1% /var/lock
udev                 1007M  108K 1007M   1% /dev
devshm               1007M     0 1007M   0% /dev/shm
lrm                  1007M   22M  986M   3% /lib/modules/2.6.15-26-amd64-k8/volatile
/dev/sda1             228M  102M  115M  48% /boot

Logical volume manager (LVM) has something to do with this.

root@weasel:/var/log# fdisk -l

Disk /dev/sda: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          31      248976   83  Linux
/dev/sda2              32       30401   243947025    5  Extended
/dev/sda5              32       30401   243946993+  8e  Linux LVM

Disk /dev/sdb: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1       30401   244196001   8e  Linux LVM

The end of the LVM Howto has some common tasks and how to do them. I noticed the pvdisplay command:

root@weasel:/etc/lvm# pvdisplay
  /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'Y67lyo-mEdN-F0un-yPLl-HCTa-hWLC-mCentd'.
  --- Physical volume ---
  PV Name               unknown device
  VG Name               VolGroup00
  PV Size               232.78 GB / not usable 0
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              7449
  Free PE               0
  Allocated PE          7449
  PV UUID               Y67lyo-mEdN-F0un-yPLl-HCTa-hWLC-mCentd

  --- Physical volume ---
  PV Name               /dev/sdb1
  VG Name               VolGroup00
  PV Size               232.88 GB / not usable 0
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              7452
  Free PE               2
  Allocated PE          7450
  PV UUID               mh0UHf-UYhT-NOj8-Ddlv-3NVL-hKBW-enhOXu

  --- Physical volume ---
  PV Name               /dev/sda5
  VG Name               Ubuntu
  PV Size               232.64 GB / not usable 0
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              59557
  Free PE               0
  Allocated PE          59557
  PV UUID               hzY3In-XHIh-RGwS-1ilj-nL6P-Ajts-KmySkw

There's a volume group display command as well:

root@weasel:/etc/lvm# vgdisplay
  /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error
  /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'Y67lyo-mEdN-F0un-yPLl-HCTa-hWLC-mCentd'.
  Couldn't find all physical volumes for volume group VolGroup00.
  /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error
  /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'Y67lyo-mEdN-F0un-yPLl-HCTa-hWLC-mCentd'.
  Couldn't find all physical volumes for volume group VolGroup00.
  Volume group "VolGroup00" doesn't exist
  --- Volume group ---
  VG Name               Ubuntu
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               232.64 GB
  PE Size               4.00 MB
  Total PE              59557
  Alloc PE / Size       59557 / 232.64 GB
  Free  PE / Size       0 / 0
  VG UUID               85vnO6-FtRR-SZnS-s0UQ-aFNf-OY5X-ZiwMm0

Physical volume scan (pvscan) shows this:

root@weasel:/etc/lvm# pvscan
  /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error
  /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'Y67lyo-mEdN-F0un-yPLl-HCTa-hWLC-mCentd'.
  PV unknown device   VG VolGroup00   lvm2 [232.78 GB / 0    free]
  PV /dev/sdb1        VG VolGroup00   lvm2 [232.88 GB / 64.00 MB free]
  PV /dev/sda5        VG Ubuntu       lvm2 [232.64 GB / 0    free]
  Total: 3 [698.30 GB] / in use: 3 [698.30 GB] / in no VG: 0 [0   ]

Analysis

It looks to me like I have two volume groups named VolGroup00 and Ubuntu. The Ubuntu volume group is working, but the VolGroup00 is not. Device /dev/sda5 is associated with Ubuntu and /dev/sdb1 is associated with VolGroup00.

The cunning plan

I think the steps toward a working configuration are these:

• remove logical volumes in VolGroup00?

lvremove /dev/???

• remove volume group VolGroup00

vgchange -a n VolGroup00
vgremove VolGroup00

The above commands failed with the error that it couldn't find VolGroup00. But it recommended that I could consider using the command: vgreduce --removemissing. I did the following:

vgreduce --test --removemissing VolGroup00
vgreduce --removemissing VolGroup00
vgreduce --test --removemissing VolGroup00

Luckily, my existing data in the Ubuntu volume group is un-destroyed.

Now vgscan says this:

root@weasel:/# vgscan
  Reading all physical volumes.  This may take a while...
  /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error
  Found volume group "VolGroup00" using metadata type lvm2
  Found volume group "Ubuntu" using metadata type lvm2

• initialize /dev/sdb1?

pvcreate -M2 /dev/sdb

Unfortunately, I got this back:

root@weasel:/# pvcreate -M2 /dev/sdb
  Device /dev/sdb not found (or ignored by filtering).

But pvscan says this:

root@weasel:/# pvscan
  /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error
  PV /dev/sda5   VG Ubuntu   lvm2 [232.64 GB / 0    free]
  PV /dev/sdb1               lvm2 [232.88 GB]
  Total: 2 [465.53 GB] / in use: 1 [232.64 GB] / in no VG: 1 [232.88 GB]

So I guess I don't need to pvcreate anything. But how do I go from here to having useable mountpoint?

I did a vgcreate like so:

root@weasel:/# vgcreate vg00 /dev/sdb1
  /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error
  /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error
  Volume group "vg00" successfully created
root@weasel:/# pvscan
  /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error
  PV /dev/sdb1   VG vg00     lvm2 [232.88 GB / 232.88 GB free]
  PV /dev/sda5   VG Ubuntu   lvm2 [232.64 GB / 0    free]
  Total: 2 [465.53 GB] / in use: 2 [465.53 GB] / in no VG: 0 [0   ]

I then used vgdisplay to find the Total PE of vg00 (59618). I then fed this to lvcreate like so:

root@weasel:/# lvcreate -l 59618 vg00 -n big
  /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error
  Logical volume "big" created

I hope that I/O error goes away after a reboot or something...

Now lvscan says this:

root@weasel:/# lvscan
  /dev/evms/lvm2/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error
  ACTIVE            '/dev/vg00/big' [232.88 GB] inherit
  ACTIVE            '/dev/Ubuntu/root' [226.77 GB] inherit
  ACTIVE            '/dev/Ubuntu/swap_1' [5.88 GB] inherit

I formatted this logical volume like this:

root@weasel:/# mkfs -t ext3 /dev/vg00/big
mke2fs 1.38 (30-Jun-2005)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
30539776 inodes, 61048832 blocks
3052441 blocks (5.00%) reserved for the super user
First data block=0
1864 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 32 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Then I could copy the line for my root partition in fstab like this:

/dev/mapper/Ubuntu-root /               ext3    defaults,errors=remount-ro 0    1
/dev/mapper/vg00-big /big           ext3    defaults,errors=remount-ro 0       1

and then issue the command "mount /big" and it worked.

• I no longer intend to assign this new disk to the existing volume group. Instead, I'm going to make a different mount point and filesystem for editing video. Hopefully, this will allow me to partition my disk failures.

kernel update causes disk failure

Perhaps it was just the reboot following the kernel update, but my /big filesystem disappeared. I didn't notice a problem until my root filesystem filled up when my daily synch between root and /big turned into a synch between root and root, causing root to fill up.

I tried wiggling the wires and unplugging the disk that hosts the failing filesystem. But that caused the system to be unbootable. Ugh. (My guess here is that the BIOS renumbered the disks when it saw that there was a change and tried to boot from the wrong disk. I had to go into the BIOS utility to change the disk boot priority to fix this.)

I had an old Ubuntu install CD on hand that let me boot the machine and get to the point of editing the LVM configuration, which was probably unnecessary. Luckily, I didn't make any substantive changes. Eventually, I found that tweaking the LVM config was not getting me anywhere and I went to the BIOS to get things working again. At least I got it to boot.

Now I'm at the point of having a bootable system with both disks recognized by the BIOS, but the non-root disk is not visible to the operating system.

df -k shows only filesystems from /dev/sda. The one big filesystem on /dev/sdb is missing. My fstab curiously did not have an entry for the /big filesystem. Perhaps Ubuntu helpfully removed it for me. I just added it back and now all is well again.

However, many reboots convinced comcast to give me a new IP address. So now I have to reconfigure for the new IP number. I need to make a wiki page on how to do that.

Can't mount second disk

For a time I had my backup disk (/big) disconnected and commented out of fstab. When I plugged it back in and uncommented the fstab entry, I found that I couldn't mount the disk.

When I try to mount it I get this:

root@weasel:~# mount /big
mount: wrong fs type, bad option, bad superblock on /dev/mapper/vg00-big,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

In /var/log/syslog, I see this:

Jun 22 22:51:17 weasel kernel: [ 1427.167819] EXT3-fs error (device dm-0): ext3_check_descriptors: Block bitmap for group 0 not in group (block 3858766779)!
Jun 22 22:51:17 weasel kernel: [ 1427.178580] EXT3-fs: group descriptors corrupted!

This appears to be working to recover the corruption:

root@weasel:/dev/vg00# fsck  /dev/vg00/big
fsck 1.40.8 (13-Mar-2008)
e2fsck 1.40.8 (13-Mar-2008)
fsck.ext3: Group descriptors look bad... trying backup blocks...
ext3 recovery flag is clear, but journal has data.
Recovery flag not set in backup superblock, so running journal anyway.
/dev/vg00/big: recovering journal
Pass 1: Checking inodes, blocks, and sizes

It is taking a pretty long time. I should probably walk away so I'm not tempted to interrupt it or poke at it.

Eventually, it started to ask permission to fix group numbers, free inodes, etc.

About 1800 groups needed to be repaired. In the end it looked like this:

Free inodes count wrong for group #1855 (16383, counted=16384).
Fix<y>? yes

Free inodes count wrong for group #1860 (16381, counted=16382).
Fix<y>? yes

Free inodes count wrong (30459026, counted=30443879).
Fix<y>? yes


/dev/vg00/big: ***** FILE SYSTEM WAS MODIFIED *****
/dev/vg00/big: 95897/30539776 files (4.2% non-contiguous), 50861505/61048832 blocks

Now I get this:

root@weasel:/dev/vg00# fsck  /dev/vg00/big
fsck 1.40.8 (13-Mar-2008)
e2fsck 1.40.8 (13-Mar-2008)
/dev/vg00/big: clean, 95897/30539776 files, 50861505/61048832 blocks

Yay.