The aim is showing that it is possible to create a low-cost storage, efficient and scalable, using opensource solutions. In the example below, I am using Ceph for scalability and reliability, combined with EnhanceIO to ensure very good performance.

Architecture

The idea was to create a storage with two parts …

Create a disk with an insane size can be fun, but a little hard to remove.

Here's a little trick (use with caution!) to remove a too big image for rm rbd (if image is not initialized, or not fully).

Image format 1 :

$ rbd info rbdbigsize
rbd image 'rbdbigsize':
    size …

An exemple for migrating journal from file on the default location to a dedicated partition on /dev/sde1.

$ apt-get install hdparm

### Stop osd
$ service ceph stop osd.0

### Flush Journal
$ ceph-osd --flush-journal -i 0

### Create symlink to partition
$ rm /var/lib/ceph/osd/ceph-0/journal
$ ln -s /dev/sde1 /var …

6 months since my last post... Firefly is released. http://ceph.com/releases/v0-80-firefly-released/

Ceph Major Release

Depending on the kernel version you may be missing some features required by the cluster (or vice versa, required by client but not available on the cluster). Hence the errors "feature set mismatch".

Some examples of errors that can be encountered :

mon0 192.168.0.1:6789 feature set mismatch …