CephNotes

Some notes about Ceph
Laurent Barbe @CCM Benchmark

Placement_pools on Rados-GW

The purpose of this test is to map a RadosGw Bucket to a specific Ceph pool. For exemple, if using a fast pool with ssd and a low pool for archive…

1
2
   standard_bucket datas  --> .rgw.buckets        (default pool)
   specific_bucket datas  --> .rgw.buckets.custom

First, we create a pool .rgw.buckets.custom, with, for example, some specific parameters (different size and different ruleset in crushmap) :

1
2
3
4
5
6
7
8
$ ceph osd pool create .rgw.buckets.custom 64 64
pool '.rgw.buckets.custom' created

$ ceph osd pool set .rgw.buckets.custom size 2
set pool 59 size to 2

$ ceph osd pool set .rgw.buckets.custom crush_ruleset 6
set pool 59 crush_ruleset to 6

Then, we need to configure a specific placement_targets in region map and zone. For next step, you need to have a running config of rados-gw…

1
2
$ radosgw-admin region get > region.conf.json
$ vim region.conf.json            # Add an entry in placement_targets
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{ "name": "default",
  "api_name": "",
  "is_master": "true",
  "endpoints": [],
  "master_zone": "",
  "zones": [
        { "name": "default",
          "endpoints": [],
          "log_meta": "false",
          "log_data": "false"}],
  "placement_targets": [
        { "name": "default-placement",
          "tags": []},
        { "name": "custom-placement",
          "tags": []}],
  "default_placement": "default-placement"}
1
2
$ radosgw-admin region set < region.conf.json
....
1
2
$ radosgw-admin zone get > zone.conf.json
$ vim zone.conf.json            # Add an entry in placement_pools with key "custom-placement"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
{ "domain_root": ".rgw",
  "control_pool": ".rgw.control",
  "gc_pool": ".rgw.gc",
  "log_pool": ".log",
  "intent_log_pool": ".intent-log",
  "usage_log_pool": ".usage",
  "user_keys_pool": ".users",
  "user_email_pool": ".users.email",
  "user_swift_pool": ".users.swift",
  "user_uid_pool": ".users.uid",
  "system_key": { "access_key": "",
      "secret_key": ""},
  "placement_pools": [
        { "key": "default-placement",
          "val": { "index_pool": ".rgw.buckets.index",
              "data_pool": ".rgw.buckets",
              "data_extra_pool": ".rgw.buckets.extra"}},
        { "key": "custom-placement",
          "val": { "index_pool": ".rgw.buckets.index",
              "data_pool": ".rgw.buckets.custom",
              "data_extra_pool": ".rgw.buckets.extra"}}]}
1
2
3
$ radosgw-admin zone set <zone.conf.json
2014-11-25 18:03:23.894153 7f728c0f2780  0 couldn't find old data placement pools config, setting up new ones for the zone
.....
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
$ radosgw-admin regionmap update
{ "regions": [
        { "key": "default",
          "val": { "name": "default",
              "api_name": "",
              "is_master": "true",
              "endpoints": [],
              "master_zone": "",
              "zones": [
                    { "name": "default",
                      "endpoints": [],
                      "log_meta": "false",
                      "log_data": "false"}],
              "placement_targets": [
                    { "name": "custom-placement",
                      "tags": []},
                    { "name": "default-placement",
                      "tags": []}],
              "default_placement": "default-placement"}}],
  "master_region": "default",
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1}}

$ /etc/init.d/radosgw reload
Reloading ...

To configure s3cmd for RadosGW you can have a look here : http://lollyrock.com/articles/s3cmd-with-radosgw/

Now we can test bucket creation :

1
2
3
4
5
6
7
8
9
$ s3cmd mb s3://custombucket --bucket-location=custom-placement
Bucket 'custombucket' created

$ touch "file_on_custom_pool"

$ s3cmd put file_on_custom_pool s3://custombucket
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
file_on_custom_pool -> s3://custombucket/file_on_custom_pool  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

Verify :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ radosgw-admin bucket stats --bucket=custombucket
{ "bucket": "custombucket",
  "pool": ".rgw.buckets.custom",
  "index_pool": ".rgw.buckets.index",
  "id": "default.240909.1",
  "marker": "default.240909.1",
  "owner": "testuser",
  "ver": 1,
  "master_ver": 0,
  "mtime": 1417016078,
  "max_marker": "",
  "usage": {},
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1}}

Pool var is set on “.rgw.buckets.custom”.

1
2
$ rados -p .rgw.buckets.custom ls
default.241071.1_file_on_custom_pool

It’s here !

Data placement pool is define in this order :

  1. from the request (“bucket location”)
  2. from user (“default_placement” : see with radosgw-admin metadata get user:<uid>)
  3. from region map (“default_placement”)

Comments