CephNotes

Some notes about Ceph
Laurent Barbe @SIB

Erasure code on small clusters

Erasure code is rather designed for clusters with a sufficient size. However if you want to use it with a small amount of hosts you can also adapt the crushmap for a better matching distribution to your need.

Here a first example for distributing data with 1 host OR 2 drive fault tolerance with k=4, m=2 on 3 hosts and more.

rule erasure_ruleset {
    ruleset X
    type erasure
    min_size 6
    max_size 6
    step take default
    step choose indep 3 type host
    step choose indep 2 type osd
    step emit
}

Testing it on sample crushmap :

# crushtool --test -i crushmap.bin --rule 1 --show-mappings --x 1 --num-rep 6
CRUSH rule 1 x 1 [0,1,8,7,5,3]

# crushtool --test -i crushmap.bin --tree
ID  WEIGHT  TYPE NAME
-5      4.00000 root default
-1      1.00000         host host-01
0       1.00000                 osd.0   <-- DATA
1       1.00000                 osd.1   <-- DATA
2       1.00000                 osd.2
-2      1.00000         host host-02
3       1.00000                 osd.3   <-- PARITY
4       1.00000                 osd.4
5       1.00000                 osd.5   <-- PARITY
-3      1.00000         host host-03
6       1.00000                 osd.6
7       1.00000                 osd.7   <-- DATA
8       1.00000                 osd.8   <-- DATA
-4      1.00000         host host-04
9       1.00000                 osd.9
10      1.00000                 osd.10
11      1.00000                 osd.11

Here is an other example for distributing data with ( 1 host AND 1 drive ) OR ( 3 drives ) fault tolerance with k=5, m=3 on 4 hosts and more.

rule erasure_ruleset {
    ruleset X
    type erasure
    min_size 8
    max_size 8
    step take default
    step choose indep 4 type host
    step choose indep 2 type osd
    step emit
}

# crushtool --test -i crushmap.bin --rule 2 --show-mappings --x 1 --num-rep 8
CRUSH rule 2 x 1 [0,1,8,7,9,11,5,3]

# crushtool --test -i crushmap.bin --tree
ID      WEIGHT  TYPE NAME
-5      4.00000 root default
-1      1.00000         host host-01
0       1.00000                 osd.0   <-- DATA
1       1.00000                 osd.1   <-- DATA
2       1.00000                 osd.2
-2      1.00000         host host-02
3       1.00000                 osd.3   <-- PARITY
4       1.00000                 osd.4
5       1.00000                 osd.5   <-- PARITY
-3      1.00000         host host-03
6       1.00000                 osd.6
7       1.00000                 osd.7   <-- DATA
8       1.00000                 osd.8   <-- DATA
-4      1.00000         host host-04
9       1.00000                 osd.9   <-- DATA
10      1.00000                 osd.10
11      1.00000                 osd.11  <-- PARITY

And a last example for distributing data with ( 1 host AND 2 drives ) OR ( 4 drives ) fault tolerance with k=8, m=4 on 4 hosts and more.

rule erasure_ruleset {
    ruleset X
    type erasure
    min_size 12
    max_size 12
    step take default
    step choose indep 4 type host
    step choose indep 3 type osd
    step emit
}

An other possibility is to use LRC Erasure code plugin :

http://docs.ceph.com/docs/master/rados/operations/erasure-code-lrc/

Comments