Some notes about Ceph
Laurent Barbe @CCM Benchmark

Test ceph persistant rbd device

Create persistant rbd device

Create block device and map it with /etc/ceph/rbdmap

$ rbd create rbd/myrbd --size=1024
$ echo "rbd/myrbd" >> /etc/ceph/rbdmap
$ service rbdmap reload
[ ok ] Starting RBD Mapping: rbd/myrbd.
[ ok ] Mounting all filesystems...done.

View rbd mapped :

$ rbd showmapped
id pool image snap device    
1  rbd  myrbd -    /dev/rbd1

Create FS and mount :

$ mkfs.xfs /dev/rbd/rbd/myrbd 
log stripe unit (4194304 bytes) is too large (maximum is 256KiB)
log stripe unit adjusted to 32KiB
meta-data=/dev/rbd/rbd/myrbd     isize=256    agcount=9, agsize=31744 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=1024   swidth=1024 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

$ mkdir -p /mnt/myrbd
$ blkid | grep rbd1
/dev/rbd1: UUID="a07e969e-bb1a-4921-9171-82cf7a737a69" TYPE="xfs"
$ echo "UUID=a07e969e-bb1a-4921-9171-82cf7a737a69 /mnt/myrbd xfs defaults 0 0" >> /etc/fstab
$ mount -a

Check :

$ mount | grep rbd1
/dev/rbd1 on /mnt/myrbd type xfs (rw,relatime,attr2,inode64,sunit=8192,swidth=8192,noquota)

Test snapshot

$ touch /mnt/myrbd/v1

Make snapshot :

$ sync && xfs_freeze -f /mnt/
$ rbd snap create rbd/myrbd@snap1
$ xfs_freeze -u /mnt/

Change a file :

$ mv /mnt/myrbd/v1 /mnt/myrbd/v2

Mount snapshot in RO :

$ mkdir -p /mnt/myrbd@snap1
$ rbd map rbd/myrbd@snap1
$ mount -t xfs -o ro,norecovery,nouuid "/dev/rbd/rbd/myrbd@snap1" "/mnt/myrbd@snap1"
$ ls "/mnt/myrbd"
total 0


$ ls "/mnt/myrbd@snap1"
total 0

Nothing ??? Something went wrong with the sync ?

Try again :

$ sync && xfs_freeze -f /mnt/
$ rbd snap create rbd/myrbd@snap2
$ xfs_freeze -u /mnt/
$ mkdir -p /mnt/myrbd@snap2
$ rbd map rbd/myrbd@snap2
$ mount -t xfs -o ro,norecovery,nouuid "/dev/rbd/rbd/myrbd@snap2" "/mnt/myrbd@snap2"

Move again the file.

$ mv /mnt/myrbd/v2 /mnt/myrbd/v3
$ ls /mnt/myrbd@snap2
total 0
$ ls /mnt/myrbd
total 0

All right.

Stop rbdmap (will remove all rbd mapped device)

$ service rbdmap remove

Remove line added in /etc/ceph/rbdmap

Remove myrbd :

$ rbd snap purge rbd/myrbd
Removing all snapshots: 100% complete...done.
$ rbd rm rbd/myrbd
Removing image: 100% complete...done.