Drobo

From finninday
Jump to navigation Jump to search

The drobo showed up yesterday. The unboxing was as cool as with an Apple product. And it is really easy to setup and start using... as long as you are plugging it into a Mac or PC. The instructions don't say anything about how to get it to work on Linux. Luckily I found this via google: 

[root@192.168.1.1]# lshw
[root@192.168.1.1]# /sbin/mke2fs -j -i 262144 -L Drobo -m 0 -O sparse_super,^resize_inode -q /dev/sdc
[root@192.168.1.1]# mkdir /drobo
[root@192.168.1.1]# mount -t ext3 /dev/sdc /drobo
update# ls -l /dev/disk/by-uuid
(copy the uuid for insertion into fstab)
[root@192.168.1.1]# vi /etc/fstab
 UUID=7fcc72fe-0884-4d66-b4f3-962901875650   /drobo  ext3    defaults     0 0


That was all I needed to get me going. I thought I had a bunch of old hard drives lying around that I would be able to wedge into drobo until I could afford to buy real drives, but none of them are SATA. Darn. Another bummer is that my server has a firewire plug that is heart-shaped and resembles a mac firewire, but the firewire plug on drobo is square. So I think I'm stuck with USB. My first speed test writing to drobo shows 14.8MB/sec. That feels pretty lame.

Another weird thing is that the drobo shows the full capacity of the array, not the actual capacity. That is, I put two 1TB drives in it and it looks like this with df -h: 

/dev/sde         2.0T   155G   1.9T   8%   /drobo

I expected drobo to report the capacity of the array to be about 1TB instead of 2TB.

I added a few more drives to the drobo to give it the capacity to hold our vacation videos.

Interestingly, after adding two 1TB drives, drobo still reports the same capacity to df. Even more reason not to trust df numbers for drobo.

drobo performance on usb 2.0[edit]

But still the drobo is just too slow.

Drobo claims to have these throughput stats: 

Max Sustained Transfer Rate:
FireWire 800: Up to 52MB/s reads and 34MB/s writes
USB 2.0: Up to 30MB/s reads and 24MB/s writes

How slow is it for me? Bonnie says this: 

rday@weasel:/drobo/bonnie$ bonnie
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
weasel           6G   249  74  1019   0   500   0   876  63  1111   0  58.3   2
Latency              1516ms   21811ms    1552ms   76164us     261ms     727ms
Version  1.96       ------Sequential Create------ --------Random Create--------
weasel              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1387   6 +++++ +++  1815   4  1615   4 +++++ +++  1811   4
Latency             63932us    1261us     725us    2761us    2696us     121us

I care most about block reads and writes. Those come out at 1,111 and 1,019 K/sec.

Hmm, what units is bonnie reporting? I'll guess bytes since bits would be silly. So I'm getting 1MB/sec on drobo. Bleck. That's a lot worse than the 24MB/sec that drobo advertises. I wonder what I'm doing wrong? I'm getting slightly faster reads than writes, which seems reasonable.

native internal disk performance on the same machine[edit]

Compare that to bonnie running on an internal drive on the same machine: 

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
weasel           6G   347  98 33362  24 17006  10  1311  95 46774  13 115.1   7
Latency             83296us    2552ms    1958ms   97333us     412ms    1399ms
Version  1.96       ------Sequential Create------ --------Random Create--------
weasel              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  2238  86 +++++ +++ +++++ +++  2691  97 +++++ +++  6854  97
Latency             35917us     410us     319us   53088us     119us     874us

That means block reads and writes at 46 and 33 MB/sec. About 30 times better. And curiously close to what drobo claims to be able to do. So I have to presume that the "block sequential read" that bonnie is measuring is a different metric than the "max sustained transfer rate" that drobo is measuring.

But if I just need to find the right firewire cable to switch off of USB, then I'm a dummy for using USB all this time.

Wikipedia says that USB 2.0 generally performs at 240Mbit/sec and 1394 does 800Mbit/sec with less CPU load. I've got to find that cable.

Hopefully, I'll be able to post bonnie stats for drobo over firewire soon.

drobo over firewire 400 (1394a)[edit]

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
weasel           6G   265  98 16951  14 10546   6   961  97 28571   8 123.6   7
Latency               110ms    6345ms     537ms   67886us     230ms     796ms
Version  1.96       ------Sequential Create------ --------Random Create--------
weasel              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  8777  28 +++++ +++ 16891  46 14137  43 +++++ +++ 15867  47
Latency             87774us   12285us   13658us   13962us   11168us   13427us

So let's see, that means 17MB/sec block writes and 28MB/sec block reads. Wow, that's much better. And if I wanted to invest in a $25 1394b card, I could probably double that. Well, I'd have to buy another $37 cable too. So maybe I'll save up for awhile.

Watching movies now seems much better. I only saw one stutter on a dvd served by the drobo and that was while the cache was still filling right after starting VLC. Now I just need to watch more movies and gather much more data.

drobo and the 2TB limit[edit]

I only just noticed that my second generation regular drobo has a 2TB limit on the LUNSIZE and that when I filled all four bays with 1TB drives, it didn't just give me (4TB - raid overhead) usable storage space. It gave me 2 x (2TB - raid overhead) of usable storage space. That is, I now have two devices, not one big pot to put all my files in. Bummer. And according to this, it is not advisable to use LVM to wrap the devices up into a single logical unit. I find that hard to believe, but I'm not immediately prepared to test my theory.

Maybe I can live with two medium-sized buckets for my data.

iostat while copying from one side of drobo to the other[edit]

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               2.00        20.80        22.40        104        112
dm-0              5.40        20.80        22.40        104        112
dm-1              0.00         0.00         0.00          0          0
sdb               0.00         0.00         0.00          0          0
sdc               0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0
dm-3              0.00         0.00         0.00          0          0
dm-4              0.00         0.00         0.00          0          0
sdd              56.80     13633.60         0.00      68168          0
sde              18.80         0.00      9593.60          0      47968

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               1.80     6.00    2.40    1.00    33.60    56.00    26.35     0.05   14.71   4.71   1.60
dm-0              0.00     0.00    4.20    7.00    33.60    56.00     8.00     0.09    7.68   1.43   1.60
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-3              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdd               2.40     0.60   79.80    1.40 19126.40    16.00   235.74     2.00   24.70  12.29  99.80
sde               0.00  1633.20    0.00   13.80     0.00 13176.00   954.78    13.92 1008.41  30.00  41.40

The near 100% utilization of sdd tells me that reads are the bottleneck. And the number of blocks that have been read in a 5-second interval was 13633.60 with a standard block size of 4KB means that drobo is reading at a rate of 55.84MB in 5 seconds or 11.17MB/sec. That's a bummer when you are trying to move several hundred gigabytes.

looking for a good test[edit]

scp read from drobo, write to /tmp[edit]

rday@weasel:/drobo/dvds$ scp Zombieland.iso localhost:/tmp
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is be:7f:1e:a6:2a:e3:01:1e:5f:25:2d:9c:6a:0f:ef:10.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Enter passphrase for key '/home/rday/.ssh/id_dsa': 
Zombieland.iso                                    66% 5331MB  22.6MB/s   01:59 ETA

timed cp from drobo to /tmp[edit]

rday@weasel:/drobo/dvds$ /usr/bin/time -v cp Zombieland.iso /tmp
	Command being timed: "cp Zombieland.iso /tmp"
	User time (seconds): 0.11
	System time (seconds): 81.54
	Percent of CPU this job got: 23%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 5:51.06
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 992
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 1
	Minor (reclaiming a frame) page faults: 313
	Voluntary context switches: 36004
	Involuntary context switches: 2110
	Swaps: 0
	File system inputs: 16434904
	File system outputs: 16439552
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Total bytes of this file: 8417050624
Total seconds: 351.06
Bytes/sec: 23,976,102

timed cp from drobo to /dev/null[edit]

I guess the write to a spinning disk still has some cost and is not completely parallelized with the read. 

rday@weasel:/drobo/dvds$ /usr/bin/time -v cp Zombieland.iso /dev/null
	Command being timed: "cp Zombieland.iso /dev/null"
	User time (seconds): 0.10
	System time (seconds): 17.98
	Percent of CPU this job got: 5%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 5:02.58
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 992
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 1
	Minor (reclaiming a frame) page faults: 313
	Voluntary context switches: 34180
	Involuntary context switches: 292
	Swaps: 0
	File system inputs: 16434432
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Total bytes of this file: 8417050624
Total seconds: 302.58
Bytes/sec: 27,817,604

dd from /dev/zero to drobo[edit]

rday@weasel:/drobo/dvds$ /usr/bin/time -v dd if=/dev/zero of=./test bs=16k count=163840
163840+0 records in
163840+0 records out
2684354560 bytes (2.7 GB) copied, 213.715 s, 12.6 MB/s
	Command being timed: "dd if=/dev/zero of=./test bs=16k count=163840"
	User time (seconds): 0.08
	System time (seconds): 17.57
	Percent of CPU this job got: 8%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 3:33.88
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 924
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 2
	Minor (reclaiming a frame) page faults: 289
	Voluntary context switches: 12677
	Involuntary context switches: 243
	Swaps: 0
	File system inputs: 192
	File system outputs: 5242880
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

drobo drives[edit]

Drobo contains four bays, numbered from the top.

drive barcode model capacity
drive 1 W1E6M94B ST2000DM001-1CH164 2TB
drive 2 Z340NNB2 ST2000DM001-1CH164 2TB
drive 3 9VP8QLAS ST31000528AS 1TB
drive 4 9VP8Q9KL ST31000528AS 1TB

blkid[edit]

/dev/sdd: LABEL="drobo" UUID="7fcc72fe-0884-4d66-b4f3-962901875650" SEC_TYPE="ext2" TYPE="ext3" 
/dev/sde: LABEL="Drobo2" UUID="1fa7b76d-795c-4d3f-acc8-3d83b2782f95" SEC_TYPE="ext2" TYPE="ext3"

drobom info[edit]

rday@weasel:/drobo/dvds$ sudo drobom info
---------------------------------------------------------
Info about Drobo     Name: Drobo disk pack       Devices: /dev/sdd:/dev/sde
---------------------------------------------------------
query config result: 
(4, 16, 2199023185408)
max lun size is:  2199023185408

query capacity result:
(94213595136, 2895427313664, 2989640908800, 0)
Physical space... used:  2895427313664  free:  94213595136  Total:  2989640908800

query protocol version result: 
(0, 11)

query settings result:
(1388526915, 8, 'Drobo disk pack')
Drobo time is Tue Dec 31 13:55:15 2013

query slotinfo result:  number of slots: 4
[(0, 1000204886016, 0, 'red', 'ST31000520AS', 'ST31000520AS'), (1, 1000204886016, 0, 'green', 'ST31000520AS', 'ST31000520AS'), (2, 1000204886016, 0, 'green', 'ST31000528AS', 'ST31000528AS'), (3, 1000204886016, 0, 'green', 'ST31000528AS', 'ST31000528AS')]
query firmware result:
(1, 252, 21110, 14, 6, 'Aug 31 2009,18:12:03', 'ArmMarvell', '1.3.5', ['NO_AUTO_REBOOT', 'NO_FAT32_FORMAT', 'USED_CAPACITY_FROM_HOST', 'DISKPACKSTATUS', 'ENCRYPT_NOHEADER', 'CMD_STATUS_QUERIABLE', 'VARIABLE_LUN_SIZE_1_16', 'PARTITION_LUN_GPT_MBR', 'FAT32_FORMAT_VOLNAME', 'SUPPORTS_DROBOSHARE', 'SUPPORTS_NEW_LUNINFO2', 'feature x0800', 'feature x2000 '])
drobo says firmware revision:  1 . 252 ( 21110 ) was built:  Aug 31 2009,18:12:03

query status result:
(['Red alert'], 0)

query options result:
{'RedThreshold': 95, 'YellowThreshold': 85}
query luninfo result:
(0, 2199023185920, 2895427313664, 'No Partitions', ['FAT32'])
(0, 0, 0, 'No Partitions', ['NTFS'])

---------------------------------------------------------

upgrade drobo[edit]

Third generation drobos go on sale soon.

From http://www.engadget.com/2014/03/26/drobo-gen-3/?ncid=rss_truncated

Pre-orders begin from today to ship at the end of April, with the base model setting you back $350 and running all the way to $1,450 if you want a 16TB unit. Unless, that is, you already own a first-or-second-gen Drobo, in which case you'll get a $50 discount if you order before June 9th.

replace drobo[edit]

disk full on drobo[edit]

# the drobo was alerting as if it were full but df says it is not

root@weasel:~# drobom status
/dev/sdd:/dev/sde /drobo:/drobo2 Drobo disk pack 98% full - (['Red alert', 'Yellow alert', 'No redundancy', 'Relay out in progress', 'no estimate yet '], 2738)

root@weasel:~# df -h
Filesystem                     Size  Used Avail Use% Mounted on
...
/dev/sdd                       2.0T  1.6T  446G  79% /drobo
/dev/sde                       2.0T  778G  1.3T  39% /drobo2

# most troubling, the red led is lighted on the top drive
# all four disks were 1TB, but the top one complained about being too full
# so I swapped in a 2TB disk in that slot

# now it says Relay out in progress and all the leds are blinking green/yellow.
# the case says that means "don't remove any more drives"

# we'll see how long it takes to rebuild consistency
# started at about 2:30pm Friday Apr 25 2014

# Saturday at 9:38am there has been no change in drobo status

/dev/sdd:/dev/sde /drobo:/drobo2 Drobo disk pack 98% full - (['Red alert', 'Yellow alert', 'No redundancy', 'Relay out in progress', 'no estimate yet
 '], 1766)

# reads and writes to /drobo2 seem to work because I have a git repo there
# that is still working

# LEDs are still flashing green/yellow for all drives

disk failure while rebuilding raid[edit]

The LED on disk #2 went red after a few days of rebuilding raid consistency. That's after disk #1 had been upgraded from 1TB to 2TB. The drobo then went into a state where all the disk LEDs were off and a single status LED was solid red. Rebooting the drobo in this state would cause the disk LEDs to turn green for each disk except for disk #2 which was solid red. After about a minute or two, the disk LEDs go off and the single status LED at the bottom goes solid red.

I have the drobo unmounted and physically disconnected from the server in case that allows the mirrors to rebuild faster.

replaced disk #2[edit]

On Friday night (5/2/2014) I upgraded disk #2 from 1TB to 2TB. Hot swapping the disks did not change the drobo state: still all disk LEDs off and a single red status LED. So I rebooted drobo which put it into the state of all disk LEDs flashing green/yellow and the red status LED being off.

drobo recovers[edit]

At sometime before Sunday afternoon (5/4/2014), the drobo stopped flashing green/yellow and had a single orange status LED. After a reboot, it came up all green. 

root@weasel:~# drobom status
/dev/sdd:/dev/sde /drobo:/drobo2 Drobo disk pack 73% full - ([], 0)

root@weasel:~# drobom info
---------------------------------------------------------
Info about Drobo     Name: Drobo disk pack       Devices: /dev/sdd:/dev/sde
---------------------------------------------------------
query config result: 
(4, 16, 2199023185408)
max lun size is:  2199023185408

query capacity result:
(1041050513408, 2947999977472, 3989050490880, 0)
Physical space... used:  2947999977472  free:  1041050513408  Total:  3989050490880

query protocol version result: 
(0, 11)

query settings result:
(1398412569, 8, 'Drobo disk pack')
Drobo time is Fri Apr 25 00:56:09 2014

query slotinfo result:  number of slots: 4
[(0, 2000398934016, 0, 'green', 'ST2000DM001-1CH164', 'ST2000DM001-1CH1'), (1, 2000398934016, 0, 'green', 'ST2000DM001-1CH164', 'ST2000DM001-1CH1'), (2, 1000204886016, 0, 'green', 'ST31000528AS', 'ST31000528AS'), (3, 1000204886016, 0, 'green', 'ST31000528AS', 'ST31000528AS')]
query firmware result:
(1, 252, 21110, 14, 6, 'Aug 31 2009,18:12:03', 'ArmMarvell', '1.3.5', ['NO_AUTO_REBOOT', 'NO_FAT32_FORMAT', 'USED_CAPACITY_FROM_HOST', 'DISKPACKSTATUS', 'ENCRYPT_NOHEADER', 'CMD_STATUS_QUERIABLE', 'VARIABLE_LUN_SIZE_1_16', 'PARTITION_LUN_GPT_MBR', 'FAT32_FORMAT_VOLNAME', 'SUPPORTS_DROBOSHARE', 'SUPPORTS_NEW_LUNINFO2', 'feature x0800', 'feature x2000 '])
drobo says firmware revision:  1 . 252 ( 21110 ) was built:  Aug 31 2009,18:12:03

query status result:
([], 0)

query options result:
{'RedThreshold': 95, 'YellowThreshold': 85}
query luninfo result:
(0, 2199023185920, 2947999977472, 'No Partitions', ['FAT32'])
(0, 0, 0, 'No Partitions', ['NTFS'])

---------------------------------------------------------

Bah, the drobo 2TB limit on this model is more real than I hoped. This drobo has four slots. The first two slots are filled with 2TB drives, the second two slots are filled with 1TB drives. So /dev/sdd should have 4TB and /dev/sde should have 2TB. But this is what is reported to the OS: 

/dev/sdd                       2.0T  1.6T  442G  79% /drobo
/dev/sde                       2.0T  778G  1.3T  39% /drobo2

So I'm wasting 2TB with this config.

firmware upgrade?[edit]

The free option is to look for a firmware upgrade and hope for the best. Otherwise, I'll be spending some money. This product was end-of-lifed a few years ago in 2010: http://support.drobo.com/app/answers/detail/a_id/594/ It looks like the latest firmware is 1.3.8 and I'm running 1.3.5. http://support.drobo.com/app/answers/detail/a_id/598/kw/software/r_id/100004

I think my best way forward is to leave it alone until it breaks and then shop for some different hardware. I like the idea of hotswapping drives and not incurring downtime. But in practice for the last several years that has not been valuable. The value of the drobo has been its portability: connecting to any laptop over usb and serving movies while we are on the road or at the beach. It is fast enough to serve a movie without stutter, but copying a dvd to it seems to take a lot longer than is necessary.

drobo performance on 32-bit firewire[edit]

standard spinning hdd drive for comparison[edit]

# hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   334 MB in  2.01 seconds = 166.49 MB/sec
 Timing buffered disk reads: 182 MB in  3.03 seconds =  60.09 MB/sec


# dd if=/dev/zero of=temp bs=8k count=10k
10240+0 records in
10240+0 records out
83886080 bytes (84 MB) copied, 0.45355 s, 185 MB/s

drobo side one[edit]

# hdparm -Tt /dev/sdd

/dev/sdd:
 Timing cached reads:   370 MB in  2.01 seconds = 184.06 MB/sec
 Timing buffered disk reads:  56 MB in  3.07 seconds =  18.22 MB/sec

# dd if=/dev/zero of=temp bs=8k count=10k
10240+0 records in
10240+0 records out
83886080 bytes (84 MB) copied, 0.489644 s, 171 MB/s

drobo side two[edit]

root@weasel:/drobo# hdparm -Tt /dev/sde

/dev/sde:
 Timing cached reads:   556 MB in  2.00 seconds = 277.86 MB/sec
 Timing buffered disk reads:  68 MB in  3.07 seconds =  22.12 MB/sec

# dd if=/dev/zero of=temp bs=8k count=10k
10240+0 records in
10240+0 records out
83886080 bytes (84 MB) copied, 10.957 s, 7.7 MB/s
Retrieved from "https://finninday.net/wiki/index.php?title=Drobo&oldid=5032"