Depending on the kernel version you may be missing some features required by the cluster (or vice versa, required by client but not available on the cluster). Hence the errors "feature set mismatch".
Some examples of errors that can be encountered :
mon0 192.168.0.1:6789 feature set mismatch, my XXXXXX < server's XXXXXX, missing 2040000
--> Upgrade kernel client up to 3.9 or set tunables to legacy : ceph osd crush tunables legacy
mon0 192.168.0.1:6789 feature set mismatch, my XXXXXX < server's XXXXXX, missing 40000000
--> Upgrade kernel client up to 3.9 or unset hashpspool : ceph osd pool set rbd hashpspool false
mon0 192.168.0.1:6789 feature set mismatch, my XXXXXX < server's XXXXXX, missing 800000000
--> Remove cache pool and reload monitors or upgrade kernel client up to 3.14
mon0 192.168.0.1:6789 feature set mismatch, my XXXXXX < server's XXXXXX, missing 1000000000
--> Upgrade kernel client up to 3.14
mon0 192.168.0.1:6789 feature set mismatch, my 4a042a42 < server's 2004a042a42, missing 20000000000
--> Upgrade kernel client up to 3.15 or disable tunable 3 features
CEPH_FEATURE Table and Kernel Version
You can find the feature missing in that table :
For exemple, missing 2040000 means that CRUSH_TUNABLES (40000) and CRUSH_TUNABLES2 (2000000) is missing on kernel client.
'R':required, 'S':support, '-X-' feature is new since this version
| Feature | BIT | OCT | 3.8 | 3.9 | 3.10 | 3.14 | 3.15 | 3.18 | 4.1 | 4.5 | 4.6 | 4.20 | 6.9 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NOSRCADDR | 1 | 2 | R | R | R | R | R | R | R | R | R | - | - |
| SUBSCRIBE2 | 4 | 10 | -R- | - | - | ||||||||
| RECONNECT_SEQ | 6 | 40 | -R- | R | R | R | R | R | R | - | - | ||
| PGID64 | 9 | 200 | R | R | R | R | R | R | R | R | - | - | |
| PGPOOL3 | 11 | 800 | R | R | R | R | R | R | R | R | - | - | |
| OSDENC | 13 | 2000 | R | R | R | R | R | R | R | R | - | - | |
| CRUSH_TUNABLES | 18 | 40000 | S | S | S | S | S | S | S | S | S | S | S |
| MSG_AUTH | 23 | 800000 | -S- | S | S | S | S | S | |||||
| CRUSH_TUNABLES2 | 25 | 2000000 | S | S | S | S | S | S | S | S | S | S | |
| REPLY_CREATE_INODE | 27 | 8000000 | S | S | S | S | S | S | S | S | S | S | |
| OSDHASHPSPOOL | 30 | 40000000 | S | S | S | S | S | S | S | S | S | S | |
| OSD_CACHEPOOL | 35 | 800000000 | -S- | S | S | S | S | S | S | S | |||
| CRUSH_V2 | 36 | 1000000000 | -S- | S | S | S | S | S | S | S | |||
| EXPORT_PEER | 37 | 2000000000 | -S- | S | S | S | S | S | S | S | |||
| OSD_ERASURE_CODES*** | 38 | 4000000000 | |||||||||||
| OSDMAP_ENC | 39 | 8000000000 | -S- | S | S | S | S | S | S | ||||
| CRUSH_TUNABLES3 | 41 | 20000000000 | -S- | S | S | S | S | S | S | ||||
| OSD_PRIMARY_AFFINITY | 41* | 20000000000 | -S- | S | S | S | S | S | S | ||||
| CRUSH_V4 **** | 48 | 1000000000000 | -S- | S | S | S | S | ||||||
| CRUSH_TUNABLES5 | 58 | 200000000000000 | -S- | S | S | S | |||||||
| NEW_OSDOPREPLY_ENCODING | 58* | 200000000000000 | -S- | S | S | S | |||||||
| FS_FILE_LAYOUT_V2 | 58* | 200000000000000 | -S- | S | S | S | |||||||
| FS_BTIME | 59 | ||||||||||||
| FS_CHANGE_ATTR | 59* | ||||||||||||
| MSG_ADDR2 | 59* | S | |||||||||||
| OSD_RECOVERY_DELETES | 60 | ||||||||||||
| CEPHX_V2 | 61* | S |
(*** Note that CEPH_FEATURE_OSD_ERASURE_CODES is no more required for client since firefly)
(**** staw2 : http://tracker.ceph.com/issues/11364)
Table for RBD format 2 features :
For example :
rbd: image rbd01: image uses unsupported features: 0x40
--> 0x40 = 64 : Disable journaling feature (use snapshot-based replication or the nbd kernel module if you want to use rbd-mirroring).
| RBD format 2 Feature | BIT | HEXA | Kernel Support since |
|---|---|---|---|
| layering | 1 | 0x1 | 4.0 |
| striping | 2 | 0x2 | 4.0 |
| exclusive-lock | 4 | 0x4 | 4.9 |
| object-map | 8 | 0x8 | 5.3 |
| fast-diff | 16 | 0x10 | 5.3 |
| deep-flatten | 32 | 0x20 | 5.1 |
| journaling | 64 | 0x40 | |
| data-pool | 128 | 0x80 | 5.0 |
| operation | 256 | 0x100 | 5.0 |
| migrating | 512 | 0x200 | |
| non-primary | 1024 | 0x400 |