Drobo
The drobo showed up yesterday. The unboxing was as cool as with an Apple product. And it is really easy to setup and start using... as long as you are plugging it into a Mac or PC. The instructions don't say anything about how to get it to work on Linux. Luckily I found this via google:
[root@192.168.1.1]# lshw [root@192.168.1.1]# /sbin/mke2fs -j -i 262144 -L Drobo -m 0 -O sparse_super,^resize_inode -q /dev/sdc [root@192.168.1.1]# mkdir /drobo [root@192.168.1.1]# mount -t ext3 /dev/sdc /drobo update# ls -l /dev/disk/by-uuid (copy the uuid for insertion into fstab) [root@192.168.1.1]# vi /etc/fstab UUID=7fcc72fe-0884-4d66-b4f3-962901875650 /drobo ext3 defaults 0 0
That was all I needed to get me going.
I thought I had a bunch of old hard drives lying around that I would be able to wedge into drobo until I could afford to buy real drives, but none of them are SATA. Darn. Another bummer is that my server has a firewire plug that is heart-shaped and resembles a mac firewire, but the firewire plug on drobo is square. So I think I'm stuck with USB. My first speed test writing to drobo shows 14.8MB/sec. That feels pretty lame.
Another weird thing is that the drobo shows the full capacity of the array, not the actual capacity. That is, I put two 1TB drives in it and it looks like this with df -h:
/dev/sde 2.0T 155G 1.9T 8% /drobo
I expected drobo to report the capacity of the array to be about 1TB instead of 2TB.
I added a few more drives to the drobo to give it the capacity to hold our vacation videos.
Interestingly, after adding two 1TB drives, drobo still reports the same capacity to df. Even more reason not to trust df numbers for drobo.
Contents
- 1 drobo performance on usb 2.0
- 2 native internal disk performance on the same machine
- 3 drobo over firewire 400 (1394a)
- 4 drobo and the 2TB limit
- 5 iostat while copying from one side of drobo to the other
- 6 looking for a good test
- 7 drobo drives
- 8 blkid
- 9 drobom info
- 10 upgrade drobo
- 11 replace drobo
- 12 disk full on drobo
- 13 disk failure while rebuilding raid
- 14 replaced disk #2
- 15 drobo recovers
- 16 firmware upgrade?
- 17 drobo performance on 32-bit firewire
drobo performance on usb 2.0
But still the drobo is just too slow.
Drobo claims to have these throughput stats:
Max Sustained Transfer Rate: FireWire 800: Up to 52MB/s reads and 34MB/s writes USB 2.0: Up to 30MB/s reads and 24MB/s writes
How slow is it for me? Bonnie says this:
rday@weasel:/drobo/bonnie$ bonnie
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
weasel 6G 249 74 1019 0 500 0 876 63 1111 0 58.3 2
Latency 1516ms 21811ms 1552ms 76164us 261ms 727ms
Version 1.96 ------Sequential Create------ --------Random Create--------
weasel -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 1387 6 +++++ +++ 1815 4 1615 4 +++++ +++ 1811 4
Latency 63932us 1261us 725us 2761us 2696us 121us
I care most about block reads and writes. Those come out at 1,111 and 1,019 K/sec.
Hmm, what units is bonnie reporting? I'll guess bytes since bits would be silly. So I'm getting 1MB/sec on drobo. Bleck. That's a lot worse than the 24MB/sec that drobo advertises. I wonder what I'm doing wrong? I'm getting slightly faster reads than writes, which seems reasonable.
native internal disk performance on the same machine
Compare that to bonnie running on an internal drive on the same machine:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
weasel 6G 347 98 33362 24 17006 10 1311 95 46774 13 115.1 7
Latency 83296us 2552ms 1958ms 97333us 412ms 1399ms
Version 1.96 ------Sequential Create------ --------Random Create--------
weasel -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 2238 86 +++++ +++ +++++ +++ 2691 97 +++++ +++ 6854 97
Latency 35917us 410us 319us 53088us 119us 874us
That means block reads and writes at 46 and 33 MB/sec. About 30 times better. And curiously close to what drobo claims to be able to do. So I have to presume that the "block sequential read" that bonnie is measuring is a different metric than the "max sustained transfer rate" that drobo is measuring.
But if I just need to find the right firewire cable to switch off of USB, then I'm a dummy for using USB all this time.
Wikipedia says that USB 2.0 generally performs at 240Mbit/sec and 1394 does 800Mbit/sec with less CPU load. I've got to find that cable.
Hopefully, I'll be able to post bonnie stats for drobo over firewire soon.
drobo over firewire 400 (1394a)
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
weasel 6G 265 98 16951 14 10546 6 961 97 28571 8 123.6 7
Latency 110ms 6345ms 537ms 67886us 230ms 796ms
Version 1.96 ------Sequential Create------ --------Random Create--------
weasel -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 8777 28 +++++ +++ 16891 46 14137 43 +++++ +++ 15867 47
Latency 87774us 12285us 13658us 13962us 11168us 13427us
So let's see, that means 17MB/sec block writes and 28MB/sec block reads. Wow, that's much better. And if I wanted to invest in a $25 1394b card, I could probably double that. Well, I'd have to buy another $37 cable too. So maybe I'll save up for awhile.
Watching movies now seems much better. I only saw one stutter on a dvd served by the drobo and that was while the cache was still filling right after starting VLC. Now I just need to watch more movies and gather much more data.
drobo and the 2TB limit
I only just noticed that my second generation regular drobo has a 2TB limit on the LUNSIZE and that when I filled all four bays with 1TB drives, it didn't just give me (4TB - raid overhead) usable storage space. It gave me 2 x (2TB - raid overhead) of usable storage space. That is, I now have two devices, not one big pot to put all my files in. Bummer. And according to this, it is not advisable to use LVM to wrap the devices up into a single logical unit. I find that hard to believe, but I'm not immediately prepared to test my theory.
Maybe I can live with two medium-sized buckets for my data.
iostat while copying from one side of drobo to the other
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 2.00 20.80 22.40 104 112 dm-0 5.40 20.80 22.40 104 112 dm-1 0.00 0.00 0.00 0 0 sdb 0.00 0.00 0.00 0 0 sdc 0.00 0.00 0.00 0 0 dm-2 0.00 0.00 0.00 0 0 dm-3 0.00 0.00 0.00 0 0 dm-4 0.00 0.00 0.00 0 0 sdd 56.80 13633.60 0.00 68168 0 sde 18.80 0.00 9593.60 0 47968 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 1.80 6.00 2.40 1.00 33.60 56.00 26.35 0.05 14.71 4.71 1.60 dm-0 0.00 0.00 4.20 7.00 33.60 56.00 8.00 0.09 7.68 1.43 1.60 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 2.40 0.60 79.80 1.40 19126.40 16.00 235.74 2.00 24.70 12.29 99.80 sde 0.00 1633.20 0.00 13.80 0.00 13176.00 954.78 13.92 1008.41 30.00 41.40
The near 100% utilization of sdd tells me that reads are the bottleneck. And the number of blocks that have been read in a 5-second interval was 13633.60 with a standard block size of 4KB means that drobo is reading at a rate of 55.84MB in 5 seconds or 11.17MB/sec. That's a bummer when you are trying to move several hundred gigabytes.
looking for a good test
scp read from drobo, write to /tmp
rday@weasel:/drobo/dvds$ scp Zombieland.iso localhost:/tmp The authenticity of host 'localhost (127.0.0.1)' can't be established. RSA key fingerprint is be:7f:1e:a6:2a:e3:01:1e:5f:25:2d:9c:6a:0f:ef:10. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'localhost' (RSA) to the list of known hosts. Enter passphrase for key '/home/rday/.ssh/id_dsa': Zombieland.iso 66% 5331MB 22.6MB/s 01:59 ETA
timed cp from drobo to /tmp
rday@weasel:/drobo/dvds$ /usr/bin/time -v cp Zombieland.iso /tmp Command being timed: "cp Zombieland.iso /tmp" User time (seconds): 0.11 System time (seconds): 81.54 Percent of CPU this job got: 23% Elapsed (wall clock) time (h:mm:ss or m:ss): 5:51.06 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 992 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 1 Minor (reclaiming a frame) page faults: 313 Voluntary context switches: 36004 Involuntary context switches: 2110 Swaps: 0 File system inputs: 16434904 File system outputs: 16439552 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0
Total bytes of this file: 8417050624 Total seconds: 351.06 Bytes/sec: 23,976,102
timed cp from drobo to /dev/null
I guess the write to a spinning disk still has some cost and is not completely parallelized with the read.
rday@weasel:/drobo/dvds$ /usr/bin/time -v cp Zombieland.iso /dev/null Command being timed: "cp Zombieland.iso /dev/null" User time (seconds): 0.10 System time (seconds): 17.98 Percent of CPU this job got: 5% Elapsed (wall clock) time (h:mm:ss or m:ss): 5:02.58 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 992 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 1 Minor (reclaiming a frame) page faults: 313 Voluntary context switches: 34180 Involuntary context switches: 292 Swaps: 0 File system inputs: 16434432 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0
Total bytes of this file: 8417050624 Total seconds: 302.58 Bytes/sec: 27,817,604
dd from /dev/zero to drobo
rday@weasel:/drobo/dvds$ /usr/bin/time -v dd if=/dev/zero of=./test bs=16k count=163840 163840+0 records in 163840+0 records out 2684354560 bytes (2.7 GB) copied, 213.715 s, 12.6 MB/s Command being timed: "dd if=/dev/zero of=./test bs=16k count=163840" User time (seconds): 0.08 System time (seconds): 17.57 Percent of CPU this job got: 8% Elapsed (wall clock) time (h:mm:ss or m:ss): 3:33.88 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 924 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 2 Minor (reclaiming a frame) page faults: 289 Voluntary context switches: 12677 Involuntary context switches: 243 Swaps: 0 File system inputs: 192 File system outputs: 5242880 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0
drobo drives
Drobo contains four bays, numbered from the top.
| drive | barcode | model | capacity |
|---|---|---|---|
| drive 1 | W1E6M94B | ST2000DM001-1CH164 | 2TB |
| drive 2 | Z340NNB2 | ST2000DM001-1CH164 | 2TB |
| drive 3 | 9VP8QLAS | ST31000528AS | 1TB |
| drive 4 | 9VP8Q9KL | ST31000528AS | 1TB |
blkid
/dev/sdd: LABEL="drobo" UUID="7fcc72fe-0884-4d66-b4f3-962901875650" SEC_TYPE="ext2" TYPE="ext3" /dev/sde: LABEL="Drobo2" UUID="1fa7b76d-795c-4d3f-acc8-3d83b2782f95" SEC_TYPE="ext2" TYPE="ext3"
drobom info
rday@weasel:/drobo/dvds$ sudo drobom info
---------------------------------------------------------
Info about Drobo Name: Drobo disk pack Devices: /dev/sdd:/dev/sde
---------------------------------------------------------
query config result:
(4, 16, 2199023185408)
max lun size is: 2199023185408
query capacity result:
(94213595136, 2895427313664, 2989640908800, 0)
Physical space... used: 2895427313664 free: 94213595136 Total: 2989640908800
query protocol version result:
(0, 11)
query settings result:
(1388526915, 8, 'Drobo disk pack')
Drobo time is Tue Dec 31 13:55:15 2013
query slotinfo result: number of slots: 4
[(0, 1000204886016, 0, 'red', 'ST31000520AS', 'ST31000520AS'), (1, 1000204886016, 0, 'green', 'ST31000520AS', 'ST31000520AS'), (2, 1000204886016, 0, 'green', 'ST31000528AS', 'ST31000528AS'), (3, 1000204886016, 0, 'green', 'ST31000528AS', 'ST31000528AS')]
query firmware result:
(1, 252, 21110, 14, 6, 'Aug 31 2009,18:12:03', 'ArmMarvell', '1.3.5', ['NO_AUTO_REBOOT', 'NO_FAT32_FORMAT', 'USED_CAPACITY_FROM_HOST', 'DISKPACKSTATUS', 'ENCRYPT_NOHEADER', 'CMD_STATUS_QUERIABLE', 'VARIABLE_LUN_SIZE_1_16', 'PARTITION_LUN_GPT_MBR', 'FAT32_FORMAT_VOLNAME', 'SUPPORTS_DROBOSHARE', 'SUPPORTS_NEW_LUNINFO2', 'feature x0800', 'feature x2000 '])
drobo says firmware revision: 1 . 252 ( 21110 ) was built: Aug 31 2009,18:12:03
query status result:
(['Red alert'], 0)
query options result:
{'RedThreshold': 95, 'YellowThreshold': 85}
query luninfo result:
(0, 2199023185920, 2895427313664, 'No Partitions', ['FAT32'])
(0, 0, 0, 'No Partitions', ['NTFS'])
---------------------------------------------------------
upgrade drobo
Third generation drobos go on sale soon.
From http://www.engadget.com/2014/03/26/drobo-gen-3/?ncid=rss_truncated
Pre-orders begin from today to ship at the end of April, with the base model setting you back $350 and running all the way to $1,450 if you want a 16TB unit. Unless, that is, you already own a first-or-second-gen Drobo, in which case you'll get a $50 discount if you order before June 9th.
replace drobo
- QNAP
- Synology
- DIY
disk full on drobo
# the drobo was alerting as if it were full but df says it is not root@weasel:~# drobom status /dev/sdd:/dev/sde /drobo:/drobo2 Drobo disk pack 98% full - (['Red alert', 'Yellow alert', 'No redundancy', 'Relay out in progress', 'no estimate yet '], 2738) root@weasel:~# df -h Filesystem Size Used Avail Use% Mounted on ... /dev/sdd 2.0T 1.6T 446G 79% /drobo /dev/sde 2.0T 778G 1.3T 39% /drobo2 # most troubling, the red led is lighted on the top drive # all four disks were 1TB, but the top one complained about being too full # so I swapped in a 2TB disk in that slot # now it says Relay out in progress and all the leds are blinking green/yellow. # the case says that means "don't remove any more drives" # we'll see how long it takes to rebuild consistency # started at about 2:30pm Friday Apr 25 2014 # Saturday at 9:38am there has been no change in drobo status /dev/sdd:/dev/sde /drobo:/drobo2 Drobo disk pack 98% full - (['Red alert', 'Yellow alert', 'No redundancy', 'Relay out in progress', 'no estimate yet '], 1766) # reads and writes to /drobo2 seem to work because I have a git repo there # that is still working # LEDs are still flashing green/yellow for all drives
disk failure while rebuilding raid
The LED on disk #2 went red after a few days of rebuilding raid consistency. That's after disk #1 had been upgraded from 1TB to 2TB. The drobo then went into a state where all the disk LEDs were off and a single status LED was solid red. Rebooting the drobo in this state would cause the disk LEDs to turn green for each disk except for disk #2 which was solid red. After about a minute or two, the disk LEDs go off and the single status LED at the bottom goes solid red.
I have the drobo unmounted and physically disconnected from the server in case that allows the mirrors to rebuild faster.
replaced disk #2
On Friday night (5/2/2014) I upgraded disk #2 from 1TB to 2TB. Hot swapping the disks did not change the drobo state: still all disk LEDs off and a single red status LED. So I rebooted drobo which put it into the state of all disk LEDs flashing green/yellow and the red status LED being off.
drobo recovers
At sometime before Sunday afternoon (5/4/2014), the drobo stopped flashing green/yellow and had a single orange status LED. After a reboot, it came up all green.
root@weasel:~# drobom status
/dev/sdd:/dev/sde /drobo:/drobo2 Drobo disk pack 73% full - ([], 0)
root@weasel:~# drobom info
---------------------------------------------------------
Info about Drobo Name: Drobo disk pack Devices: /dev/sdd:/dev/sde
---------------------------------------------------------
query config result:
(4, 16, 2199023185408)
max lun size is: 2199023185408
query capacity result:
(1041050513408, 2947999977472, 3989050490880, 0)
Physical space... used: 2947999977472 free: 1041050513408 Total: 3989050490880
query protocol version result:
(0, 11)
query settings result:
(1398412569, 8, 'Drobo disk pack')
Drobo time is Fri Apr 25 00:56:09 2014
query slotinfo result: number of slots: 4
[(0, 2000398934016, 0, 'green', 'ST2000DM001-1CH164', 'ST2000DM001-1CH1'), (1, 2000398934016, 0, 'green', 'ST2000DM001-1CH164', 'ST2000DM001-1CH1'), (2, 1000204886016, 0, 'green', 'ST31000528AS', 'ST31000528AS'), (3, 1000204886016, 0, 'green', 'ST31000528AS', 'ST31000528AS')]
query firmware result:
(1, 252, 21110, 14, 6, 'Aug 31 2009,18:12:03', 'ArmMarvell', '1.3.5', ['NO_AUTO_REBOOT', 'NO_FAT32_FORMAT', 'USED_CAPACITY_FROM_HOST', 'DISKPACKSTATUS', 'ENCRYPT_NOHEADER', 'CMD_STATUS_QUERIABLE', 'VARIABLE_LUN_SIZE_1_16', 'PARTITION_LUN_GPT_MBR', 'FAT32_FORMAT_VOLNAME', 'SUPPORTS_DROBOSHARE', 'SUPPORTS_NEW_LUNINFO2', 'feature x0800', 'feature x2000 '])
drobo says firmware revision: 1 . 252 ( 21110 ) was built: Aug 31 2009,18:12:03
query status result:
([], 0)
query options result:
{'RedThreshold': 95, 'YellowThreshold': 85}
query luninfo result:
(0, 2199023185920, 2947999977472, 'No Partitions', ['FAT32'])
(0, 0, 0, 'No Partitions', ['NTFS'])
---------------------------------------------------------
Bah, the drobo 2TB limit on this model is more real than I hoped. This drobo has four slots. The first two slots are filled with 2TB drives, the second two slots are filled with 1TB drives. So /dev/sdd should have 4TB and /dev/sde should have 2TB. But this is what is reported to the OS:
/dev/sdd 2.0T 1.6T 442G 79% /drobo /dev/sde 2.0T 778G 1.3T 39% /drobo2
So I'm wasting 2TB with this config.
firmware upgrade?
The free option is to look for a firmware upgrade and hope for the best. Otherwise, I'll be spending some money. This product was end-of-lifed a few years ago in 2010: http://support.drobo.com/app/answers/detail/a_id/594/ It looks like the latest firmware is 1.3.8 and I'm running 1.3.5. http://support.drobo.com/app/answers/detail/a_id/598/kw/software/r_id/100004
I think my best way forward is to leave it alone until it breaks and then shop for some different hardware. I like the idea of hotswapping drives and not incurring downtime. But in practice for the last several years that has not been valuable. The value of the drobo has been its portability: connecting to any laptop over usb and serving movies while we are on the road or at the beach. It is fast enough to serve a movie without stutter, but copying a dvd to it seems to take a lot longer than is necessary.
drobo performance on 32-bit firewire
standard spinning hdd drive for comparison
# hdparm -Tt /dev/sda /dev/sda: Timing cached reads: 334 MB in 2.01 seconds = 166.49 MB/sec Timing buffered disk reads: 182 MB in 3.03 seconds = 60.09 MB/sec # dd if=/dev/zero of=temp bs=8k count=10k 10240+0 records in 10240+0 records out 83886080 bytes (84 MB) copied, 0.45355 s, 185 MB/s
drobo side one
# hdparm -Tt /dev/sdd /dev/sdd: Timing cached reads: 370 MB in 2.01 seconds = 184.06 MB/sec Timing buffered disk reads: 56 MB in 3.07 seconds = 18.22 MB/sec # dd if=/dev/zero of=temp bs=8k count=10k 10240+0 records in 10240+0 records out 83886080 bytes (84 MB) copied, 0.489644 s, 171 MB/s
drobo side two
root@weasel:/drobo# hdparm -Tt /dev/sde /dev/sde: Timing cached reads: 556 MB in 2.00 seconds = 277.86 MB/sec Timing buffered disk reads: 68 MB in 3.07 seconds = 22.12 MB/sec # dd if=/dev/zero of=temp bs=8k count=10k 10240+0 records in 10240+0 records out 83886080 bytes (84 MB) copied, 10.957 s, 7.7 MB/s
