CephNoteshttps://cephnotes.ksperis.com/2024-02-15T22:30:00+01:00Some notes about Ceph<br>Laurent Barbe @SIBTest Ceph NVMe-oF2024-02-15T22:30:00+01:002024-02-15T22:30:00+01:00Laurent Barbetag:cephnotes.ksperis.com,2024-02-15:/blog/2024/02/15/ceph-nvmeof-vmware/<p>A quick test of NVMe over Fabrics (NVMe/TCP) and VMware...</p>
<p><img alt="NVMe over Fabrics Ceph VMWare" src="https://cephnotes.ksperis.com/images/nvmeof-illustration1.png"></p>
<h2>Create a RBD volume</h2>
<div class="highlight"><pre><span></span><code><span class="c1"># Create a pool "nvmeof_pool01" and a rbd image inside :</span>
ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>nvmeof_pool01
rbd<span class="w"> </span>pool<span class="w"> </span>init<span class="w"> </span>nvmeof_pool01
rbd<span class="w"> </span>-p<span class="w"> </span>nvmeof_pool01<span class="w"> </span>create<span class="w"> </span>nvme_image<span class="w"> </span>--size<span class="w"> </span>50G
<span class="c1"># Deploy nvmeof instance (version 1.0.0)</span>
ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set …</span></code></pre></div><p>A quick test of NVMe over Fabrics (NVMe/TCP) and VMware...</p>
<p><img alt="NVMe over Fabrics Ceph VMWare" src="https://cephnotes.ksperis.com/images/nvmeof-illustration1.png"></p>
<h2>Create a RBD volume</h2>
<div class="highlight"><pre><span></span><code><span class="c1"># Create a pool "nvmeof_pool01" and a rbd image inside :</span>
ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>nvmeof_pool01
rbd<span class="w"> </span>pool<span class="w"> </span>init<span class="w"> </span>nvmeof_pool01
rbd<span class="w"> </span>-p<span class="w"> </span>nvmeof_pool01<span class="w"> </span>create<span class="w"> </span>nvme_image<span class="w"> </span>--size<span class="w"> </span>50G
<span class="c1"># Deploy nvmeof instance (version 1.0.0)</span>
ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mgr<span class="w"> </span>mgr/cephadm/container_image_nvmeof<span class="w"> </span>quay.io/ceph/nvmeof:1.0.0
ceph<span class="w"> </span>orch<span class="w"> </span>apply<span class="w"> </span>nvmeof<span class="w"> </span>nvmeof_pool01<span class="w"> </span>--placement<span class="o">=</span><span class="s2">"ceph01"</span>
</code></pre></div>
<h2>Configure NVME Subsystem</h2>
<div class="highlight"><pre><span></span><code><span class="c1"># Create an alias to simplify the following commands</span>
<span class="c1"># I use the same version of nvmeof-cli:1.0.0, 192.168.10.11 is the host ceph01</span>
<span class="c1"># I run those commands directly on ceph host but it could be on any other with podman installed</span>
<span class="nb">alias</span><span class="w"> </span>nvmeof-cli<span class="o">=</span><span class="s1">'podman run -it quay.io/ceph/nvmeof-cli:1.0.0 --server-address 192.168.10.11 --server-port 5500'</span>
<span class="c1"># Create the subsystem, the namespace corresponding the rbd image, and the listener</span>
nvmeof-cli<span class="w"> </span>subsystem<span class="w"> </span>add<span class="w"> </span>--subsystem<span class="w"> </span>nqn.2016-06.io.spdk:ceph
nvmeof-cli<span class="w"> </span>namespace<span class="w"> </span>add<span class="w"> </span>--subsystem<span class="w"> </span>nqn.2016-06.io.spdk:ceph<span class="w"> </span>--rbd-pool<span class="w"> </span>nvmeof_pool01<span class="w"> </span>--rbd-image<span class="w"> </span>nvme_image
nvmeof-cli<span class="w"> </span>listener<span class="w"> </span>add<span class="w"> </span>--subsystem<span class="w"> </span>nqn.2016-06.io.spdk:ceph<span class="w"> </span>--gateway-name<span class="w"> </span>client.nvmeof.nvmeof_pool01.ceph01.htrgqe<span class="w"> </span>--traddr<span class="w"> </span><span class="m">192</span>.168.10.11<span class="w"> </span>--trsvcid<span class="w"> </span><span class="m">4420</span>
<span class="c1"># Allow all clients (to allow specific esxi clients, retreive NQN address with "esxcli nvme info get")</span>
nvmeof-cli<span class="w"> </span>host<span class="w"> </span>add<span class="w"> </span>--subsystem<span class="w"> </span>nqn.2016-06.io.spdk:ceph<span class="w"> </span>--host<span class="w"> </span><span class="s2">"*"</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="c1"># List all subsystems</span>
nvmeof-cli<span class="w"> </span>subsystem<span class="w"> </span>list
╒═══════════╤════════════════════════════╤════════════╤═══════════════════╤══════════════════════╤══════════════════╤═════════════╕
│<span class="w"> </span>Subtype<span class="w"> </span>│<span class="w"> </span>NQN<span class="w"> </span>│<span class="w"> </span>HA<span class="w"> </span>State<span class="w"> </span>│<span class="w"> </span>Serial<span class="w"> </span>Number<span class="w"> </span>│<span class="w"> </span>Model<span class="w"> </span>Number<span class="w"> </span>│<span class="w"> </span>Controller<span class="w"> </span>IDs<span class="w"> </span>│<span class="w"> </span>Namespace<span class="w"> </span>│
│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>Count<span class="w"> </span>│
╞═══════════╪════════════════════════════╪════════════╪═══════════════════╪══════════════════════╪══════════════════╪═════════════╡
│<span class="w"> </span>NVMe<span class="w"> </span>│<span class="w"> </span>nqn.2016-06.io.spdk:ceph<span class="w"> </span>│<span class="w"> </span>disabled<span class="w"> </span>│<span class="w"> </span>SPDK4675192209286<span class="w"> </span>│<span class="w"> </span>SPDK<span class="w"> </span>bdev<span class="w"> </span>Controller<span class="w"> </span>│<span class="w"> </span><span class="m">1</span>-65519<span class="w"> </span>│<span class="w"> </span><span class="m">1</span><span class="w"> </span>│
╘═══════════╧════════════════════════════╧════════════╧═══════════════════╧══════════════════════╧══════════════════╧═════════════╛
<span class="c1"># List namespace</span>
nvmeof-cli<span class="w"> </span>namespace<span class="w"> </span>list<span class="w"> </span>--subsystem<span class="w"> </span>nqn.2016-06.io.spdk:ceph
╒════════╤════════════════════════╤═══════════════╤════════════╤═════════╤═════════╤═════════════════════╤═════════════╤═══════════╤═══════════╤════════════╤═════════════╕
│<span class="w"> </span>NSID<span class="w"> </span>│<span class="w"> </span>Bdev<span class="w"> </span>│<span class="w"> </span>RBD<span class="w"> </span>│<span class="w"> </span>RBD<span class="w"> </span>│<span class="w"> </span>Image<span class="w"> </span>│<span class="w"> </span>Block<span class="w"> </span>│<span class="w"> </span>UUID<span class="w"> </span>│<span class="w"> </span>Load<span class="w"> </span>│<span class="w"> </span>R/W<span class="w"> </span>IOs<span class="w"> </span>│<span class="w"> </span>R/W<span class="w"> </span>MBs<span class="w"> </span>│<span class="w"> </span>Read<span class="w"> </span>MBs<span class="w"> </span>│<span class="w"> </span>Write<span class="w"> </span>MBs<span class="w"> </span>│
│<span class="w"> </span>│<span class="w"> </span>Name<span class="w"> </span>│<span class="w"> </span>Pool<span class="w"> </span>│<span class="w"> </span>Image<span class="w"> </span>│<span class="w"> </span>Size<span class="w"> </span>│<span class="w"> </span>Size<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>Balancing<span class="w"> </span>│<span class="w"> </span>per<span class="w"> </span>│<span class="w"> </span>per<span class="w"> </span>│<span class="w"> </span>per<span class="w"> </span>│<span class="w"> </span>per<span class="w"> </span>│
│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>Group<span class="w"> </span>│<span class="w"> </span>second<span class="w"> </span>│<span class="w"> </span>second<span class="w"> </span>│<span class="w"> </span>second<span class="w"> </span>│<span class="w"> </span>second<span class="w"> </span>│
╞════════╪════════════════════════╪═══════════════╪════════════╪═════════╪═════════╪═════════════════════╪═════════════╪═══════════╪═══════════╪════════════╪═════════════╡
│<span class="w"> </span><span class="m">1</span><span class="w"> </span>│<span class="w"> </span>bdev_df4877b5-79b8-<span class="w"> </span>│<span class="w"> </span>nvmeof_pool01<span class="w"> </span>│<span class="w"> </span>nvme_image<span class="w"> </span>│<span class="w"> </span><span class="m">50</span><span class="w"> </span>GiB<span class="w"> </span>│<span class="w"> </span><span class="m">512</span><span class="w"> </span>B<span class="w"> </span>│<span class="w"> </span>df4877b5-79b8-41e0-<span class="w"> </span>│<span class="w"> </span><n/a><span class="w"> </span>│<span class="w"> </span>unlimited<span class="w"> </span>│<span class="w"> </span>unlimited<span class="w"> </span>│<span class="w"> </span>unlimited<span class="w"> </span>│<span class="w"> </span>unlimited<span class="w"> </span>│
│<span class="w"> </span>│<span class="w"> </span>41e0-95f1-2c9aa222a513<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>95f1-2c9aa222a513<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│
╘════════╧════════════════════════╧═══════════════╧════════════╧═════════╧═════════╧═════════════════════╧═════════════╧═══════════╧═══════════╧════════════╧═════════════╛
<span class="c1"># List hosts allowed</span>
nvmeof-cli<span class="w"> </span>host<span class="w"> </span>list<span class="w"> </span>--subsystem<span class="w"> </span>nqn.2016-06.io.spdk:ceph
╒════════════╕
│<span class="w"> </span>Host<span class="w"> </span>NQN<span class="w"> </span>│
╞════════════╡
│<span class="w"> </span>Any<span class="w"> </span>host<span class="w"> </span>│
╘════════════╛
</code></pre></div>
<h2>Configure NVME on ESXi</h2>
<p>In this case I use a single network. A dedicated and redundant network with multiple initiators would be better...</p>
<p>This is done on all ESXi hosts :</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span>root@esxi01:~<span class="o">]</span><span class="w"> </span>esxcli<span class="w"> </span>nvme<span class="w"> </span>fabrics<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>--protocol<span class="w"> </span>TCP<span class="w"> </span>--device<span class="w"> </span>vmnic0
<span class="o">[</span>root@esxi01:~<span class="o">]</span><span class="w"> </span>esxcli<span class="w"> </span>network<span class="w"> </span>ip<span class="w"> </span>interface<span class="w"> </span>tag<span class="w"> </span>add<span class="w"> </span>--interface-name<span class="w"> </span>vmk0<span class="w"> </span>--tagname<span class="w"> </span>NVMeTCP
</code></pre></div>
<p>Test discovery :</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span>root@esxi01:~<span class="o">]</span><span class="w"> </span>esxcli<span class="w"> </span>nvme<span class="w"> </span>fabrics<span class="w"> </span>discover<span class="w"> </span>-a<span class="w"> </span>vmhba65<span class="w"> </span>-i<span class="w"> </span><span class="m">192</span>.168.10.11<span class="w"> </span>-p<span class="w"> </span><span class="m">8009</span>
Transport<span class="w"> </span>Type<span class="w"> </span>Address<span class="w"> </span>Family<span class="w"> </span>Subsystem<span class="w"> </span>Type<span class="w"> </span>Controller<span class="w"> </span>ID<span class="w"> </span>Admin<span class="w"> </span>Queue<span class="w"> </span>Max<span class="w"> </span>Size<span class="w"> </span>Transport<span class="w"> </span>Address<span class="w"> </span>Transport<span class="w"> </span>Service<span class="w"> </span>ID<span class="w"> </span>Subsystem<span class="w"> </span>NQN<span class="w"> </span>Connected
--------------<span class="w"> </span>--------------<span class="w"> </span>--------------<span class="w"> </span>-------------<span class="w"> </span>--------------------<span class="w"> </span>-----------------<span class="w"> </span>--------------------<span class="w"> </span>--------------------------<span class="w"> </span>---------
TCP<span class="w"> </span>IPv4<span class="w"> </span>NVM<span class="w"> </span><span class="m">65535</span><span class="w"> </span><span class="m">128</span><span class="w"> </span><span class="m">192</span>.168.10.11<span class="w"> </span><span class="m">4420</span><span class="w"> </span>nqn.2016-06.io.spdk:ceph<span class="w"> </span><span class="nb">true</span>
</code></pre></div>
<p>Connect and list namespaces :</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span>root@esxi01:~<span class="o">]</span><span class="w"> </span>esxcli<span class="w"> </span>nvme<span class="w"> </span>fabrics<span class="w"> </span>connect<span class="w"> </span>-a<span class="w"> </span>vmhba65<span class="w"> </span>-i<span class="w"> </span><span class="m">192</span>.168.10.11<span class="w"> </span>-p<span class="w"> </span><span class="m">4420</span><span class="w"> </span>-s<span class="w"> </span>nqn.2016-06.io.spdk:ceph
<span class="o">[</span>root@esxi01:~<span class="o">]</span><span class="w"> </span>esxcli<span class="w"> </span>nvme<span class="w"> </span>namespace<span class="w"> </span>list
Name<span class="w"> </span>Controller<span class="w"> </span>Number<span class="w"> </span>Namespace<span class="w"> </span>ID<span class="w"> </span>Block<span class="w"> </span>Size<span class="w"> </span>Capacity<span class="w"> </span><span class="k">in</span><span class="w"> </span>MB
------------------------------------<span class="w"> </span>-----------------<span class="w"> </span>------------<span class="w"> </span>----------<span class="w"> </span>--------------
eui.df4877b579b841e095f12c9aa222a513<span class="w"> </span><span class="m">256</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">512</span><span class="w"> </span><span class="m">51200</span>
</code></pre></div>
<p>Add the datastore from VSphere client :</p>
<p><img alt="nvmeof new datastore" src="https://cephnotes.ksperis.com/images/nvmeof-new-datastore2.png"></p>
<p><img alt="nvmeof new datastore" src="https://cephnotes.ksperis.com/images/nvmeof-new-datastore3.png"></p>
<p>This promises interesting things...</p>
<h2>References :</h2>
<p>Ceph NVMEOF Github : <a href="https://github.com/ceph/ceph-nvmeof">https://github.com/ceph/ceph-nvmeof</a></p>
<p>IBM Ceph Docs NVME-oF Tech Preview : <a href="https://www.ibm.com/docs/en/storage-ceph/7?topic=devices-ceph-nvme-gateway-technology-preview">https://www.ibm.com/docs/.../devices-ceph-nvme-gateway-technology-preview</a></p>
<p>RedHat Doc : Configuring NVMe over fabrics using NVMe/TCP <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/managing_storage_devices/configuring-nvme-over-fabrics-using-nvme-tcp_managing-storage-devices">https://access.redhat.com/documentation/.../configuring-nvme-over-fabrics...</a></p>
<p>Red Hat Ceph Storage: A use case study into ESXi host attach over NVMeTCP : <a href="https://community.ibm.com/community/user/storage/blogs/ashish-jagdale/2023/12/11/red-hat-ceph-storage-esxi-nvmetcp">https://community.ibm.com/.../red-hat-ceph-storage-esxi-nvmetcp</a></p>
<p>VMWare : Configuring NVMe over TCP on ESXi : <a href="https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-storage/GUID-5F776E6E-62B1-445D-854C-BEA689DD4C92.html">https://docs.vmware.com/...</a></p>
<p>ESXI nvme commands : <a href="https://vdc-repo.vmware.com/vmwb-repository/dcr-public/2f4dc74e-ff3a-4c9f-9682-d1300bad5dba/9f234484-2879-4c07-bcf6-5475bc37f4ab/namespace/esxcli_nvme.html">https://vdc-repo.vmware.com/.../esxcli_nvme.html</a></p>
<p>NVM Express - TCP Transport Specification : <a href="https://nvmexpress.org/wp-content/uploads/NVM-Express-TCP-Transport-Specification-1.0d-2023.12.27-Ratified.pdf">hhttps://nvmexpress.org/.../NVM-Express-TCP-Transport-Specification-1.0d-2023.12.27-Ratified.pdf</a></p>RBD journal offloading2023-03-17T15:30:00+01:002023-03-18T22:35:00+01:00Laurent Barbetag:cephnotes.ksperis.com,2023-03-17:/blog/2023/03/17/rbd-journal-offloading/<p>If you are using rbd <strong>journaling</strong> feature (for example for rbd mirroring), in some cases it could be interesting to offloading journaling on specific pool. For example if your rbd pool is on hdd drives and you also have ssd or nvme.</p>
<p><img alt="external rbd journal" src="https://cephnotes.ksperis.com/images/2023-03-17-journal.png"></p>
<p>To change rbd journaling pool, there are 2 …</p><p>If you are using rbd <strong>journaling</strong> feature (for example for rbd mirroring), in some cases it could be interesting to offloading journaling on specific pool. For example if your rbd pool is on hdd drives and you also have ssd or nvme.</p>
<p><img alt="external rbd journal" src="https://cephnotes.ksperis.com/images/2023-03-17-journal.png"></p>
<p>To change rbd journaling pool, there are 2 ways :</p>
<ul>
<li>
<p>Cluster wide : you need to add the option <code>rbd_journal_pool</code> in your <code>ceph.conf</code> or with <code>ceph config</code>, this option cannot be changed dynamiquely, it must be applied on all OSDs after restarting all of them. If your cluster is alreading deployed and uses journaling, it could be difficult to update this configuration. </p>
</li>
<li>
<p>Per image : with the <code>--journal-pool</code> option</p>
</li>
</ul>
<h2>Example of rbd image with mirroring</h2>
<p>Have a look to an example rbd image with default journaling storage :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>info<span class="w"> </span>rbd-pve-mirrored/vm-1517-disk-0
rbd<span class="w"> </span>image<span class="w"> </span><span class="s1">'vm-1517-disk-0'</span>:
<span class="w"> </span>size<span class="w"> </span><span class="m">5</span><span class="w"> </span>GiB<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">1280</span><span class="w"> </span>objects
<span class="w"> </span>order<span class="w"> </span><span class="m">22</span><span class="w"> </span><span class="o">(</span><span class="m">4</span><span class="w"> </span>MiB<span class="w"> </span>objects<span class="o">)</span>
<span class="w"> </span>snapshot_count:<span class="w"> </span><span class="m">0</span>
<span class="w"> </span>id:<span class="w"> </span>378952d8c4cf0c
<span class="w"> </span>block_name_prefix:<span class="w"> </span>rbd_data.378952d8c4cf0c
<span class="w"> </span>format:<span class="w"> </span><span class="m">2</span>
<span class="w"> </span>features:<span class="w"> </span>layering,<span class="w"> </span>exclusive-lock,<span class="w"> </span>object-map,<span class="w"> </span>fast-diff,<span class="w"> </span>deep-flatten,<span class="w"> </span>journaling
<span class="w"> </span>op_features:
<span class="w"> </span>flags:
<span class="w"> </span>create_timestamp:<span class="w"> </span>Mon<span class="w"> </span>Mar<span class="w"> </span><span class="m">6</span><span class="w"> </span><span class="m">11</span>:14:12<span class="w"> </span><span class="m">2023</span>
<span class="w"> </span>access_timestamp:<span class="w"> </span>Mon<span class="w"> </span>Mar<span class="w"> </span><span class="m">13</span><span class="w"> </span><span class="m">08</span>:43:01<span class="w"> </span><span class="m">2023</span>
<span class="w"> </span>modify_timestamp:<span class="w"> </span>Mon<span class="w"> </span>Mar<span class="w"> </span><span class="m">13</span><span class="w"> </span><span class="m">09</span>:13:37<span class="w"> </span><span class="m">2023</span>
<span class="w"> </span>journal:<span class="w"> </span>378952d8c4cf0c<span class="w"> </span><span class="c1"># <---</span>
<span class="w"> </span>mirroring<span class="w"> </span>state:<span class="w"> </span>enabled
<span class="w"> </span>mirroring<span class="w"> </span>mode:<span class="w"> </span>journal<span class="w"> </span><span class="c1"># <---</span>
<span class="w"> </span>mirroring<span class="w"> </span>global<span class="w"> </span>id:<span class="w"> </span>9b968a20-91a5-47a0-85af-84d3d79111b8
<span class="w"> </span>mirroring<span class="w"> </span>primary:<span class="w"> </span><span class="nb">true</span>
</code></pre></div>
<p>We can see mirroring mode journal <code>mirroring mode: journal</code> and journal oid <code>journal: 378952d8c4cf0c</code>.
To get more detail use <code>journal info</code>:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>journal<span class="w"> </span>info<span class="w"> </span>--pool<span class="w"> </span>rbd-pve-mirrored<span class="w"> </span>--image<span class="w"> </span>vm-1517-disk-0
rbd<span class="w"> </span>journal<span class="w"> </span><span class="s1">'378952d8c4cf0c'</span>:
<span class="w"> </span>header_oid:<span class="w"> </span>journal.378952d8c4cf0c<span class="w"> </span><span class="c1"># <--- journal header</span>
<span class="w"> </span>object_oid_prefix:<span class="w"> </span>journal_data.51.378952d8c4cf0c.<span class="w"> </span><span class="c1"># <--- journal data</span>
<span class="w"> </span>order:<span class="w"> </span><span class="m">24</span><span class="w"> </span><span class="o">(</span><span class="m">16</span><span class="w"> </span>MiB<span class="w"> </span>objects<span class="o">)</span>
<span class="w"> </span>splay_width:<span class="w"> </span><span class="m">4</span>
</code></pre></div>
<p>With <code>rados ls</code>, you can see thoses objects in the current rbd pool (journal.378952d8c4cf0c and journal_data.51.378952d8c4cf0c.)</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>rbd-pve-mirrored<span class="w"> </span>ls<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^journal_data.51.378952d8c4cf0c.'</span>
journal_data.51.378952d8c4cf0c.2
journal_data.51.378952d8c4cf0c.1
journal_data.51.378952d8c4cf0c.0
...
</code></pre></div>
<h2>Migrate the journal</h2>
<p>In case of mirroring the operation need to be done on the primary side.</p>
<p>Create a new pool to store journals, e.g. on nvme replicated crush rule</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>rbd-journals<span class="w"> </span><span class="m">32</span><span class="w"> </span><span class="m">32</span><span class="w"> </span>nvme-replicated
</code></pre></div>
<p>Disable the mirroring on the image
! disabling mirroring will delete replicated versions of the image</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>mirror<span class="w"> </span>image<span class="w"> </span>disable<span class="w"> </span>rbd-pve-mirrored/vm-1517-disk-0
</code></pre></div>
<p>Check that journaling feature has been disabled, or run rbd feature disable rbd-pve-mirrored/vm-1517-disk-0 journaling :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>info<span class="w"> </span>rbd-pve-mirrored/vm-1517-disk-0<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>features
<span class="w"> </span>features:<span class="w"> </span>layering,<span class="w"> </span>exclusive-lock,<span class="w"> </span>object-map,<span class="w"> </span>fast-diff,<span class="w"> </span>deep-flatten
<span class="w"> </span>op_features:
</code></pre></div>
<p>Now we can enable journaling feature with the specific journal pool</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>feature<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>rbd-pve-mirrored/vm-1517-disk-0<span class="w"> </span>journaling<span class="w"> </span>--journal-pool<span class="w"> </span>rbd-journals
</code></pre></div>
<p>Check the journal details :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>journal<span class="w"> </span>info<span class="w"> </span>--pool<span class="w"> </span>rbd-pve-mirrored<span class="w"> </span>--image<span class="w"> </span>vm-1517-disk-0
rbd<span class="w"> </span>journal<span class="w"> </span><span class="s1">'378952d8c4cf0c'</span>:
<span class="w"> </span>header_oid:<span class="w"> </span>journal.378952d8c4cf0c
<span class="w"> </span>object_oid_prefix:<span class="w"> </span>journal_data.51.378952d8c4cf0c.
<span class="w"> </span>order:<span class="w"> </span><span class="m">24</span><span class="w"> </span><span class="o">(</span><span class="m">16</span><span class="w"> </span>MiB<span class="w"> </span>objects<span class="o">)</span>
<span class="w"> </span>splay_width:<span class="w"> </span><span class="m">4</span>
<span class="w"> </span>object_pool:<span class="w"> </span>rbd-journals<span class="w"> </span><span class="c1"># <--- We can see the custom pool</span>
</code></pre></div>
<p>Re-enable mirroring in journaling mode :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>mirror<span class="w"> </span>image<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>rbd-pve-mirrored/vm-1517-disk-0<span class="w"> </span>journal
</code></pre></div>
<p>See objects on the new rbd-journals pool :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>rbd-journals<span class="w"> </span>ls<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^journal_data.51.378952d8c4cf0c.'</span>
journal_data.51.378952d8c4cf0c.2
journal_data.51.378952d8c4cf0c.1
journal_data.51.378952d8c4cf0c.0
</code></pre></div>How many mouvement when I add a replica ?2017-07-28T15:15:46+02:002017-07-28T15:15:46+02:00Laurent Barbetag:cephnotes.ksperis.com,2017-07-28:/blog/2017/07/28/how-many-mouvement-when-i-add-a-replica/<p>Make a simple simulation !</p>
<p>Use your own crushmap :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>getcrushmap<span class="w"> </span>-o<span class="w"> </span>crushmap
got<span class="w"> </span>crush<span class="w"> </span>map<span class="w"> </span>from<span class="w"> </span>osdmap<span class="w"> </span>epoch<span class="w"> </span><span class="m">28673</span>
</code></pre></div>
<p>Or create a sample clushmap :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>crushtool<span class="w"> </span>--outfn<span class="w"> </span>crushmap<span class="w"> </span>--build<span class="w"> </span>--num_osds<span class="w"> </span><span class="m">36</span><span class="w"> </span>host<span class="w"> </span>straw<span class="w"> </span><span class="m">12</span><span class="w"> </span>root<span class="w"> </span>straw<span class="w"> </span><span class="m">0</span>
<span class="m">2017</span>-07-28<span class="w"> </span><span class="m">15</span>:01:16.240974<span class="w"> </span>7f4dda123760<span class="w"> </span><span class="m">1</span><span class="w"> </span>
ID<span class="w"> </span>WEIGHT<span class="w"> </span>TYPE<span class="w"> </span>NAME
-4 …</code></pre></div><p>Make a simple simulation !</p>
<p>Use your own crushmap :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>getcrushmap<span class="w"> </span>-o<span class="w"> </span>crushmap
got<span class="w"> </span>crush<span class="w"> </span>map<span class="w"> </span>from<span class="w"> </span>osdmap<span class="w"> </span>epoch<span class="w"> </span><span class="m">28673</span>
</code></pre></div>
<p>Or create a sample clushmap :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>crushtool<span class="w"> </span>--outfn<span class="w"> </span>crushmap<span class="w"> </span>--build<span class="w"> </span>--num_osds<span class="w"> </span><span class="m">36</span><span class="w"> </span>host<span class="w"> </span>straw<span class="w"> </span><span class="m">12</span><span class="w"> </span>root<span class="w"> </span>straw<span class="w"> </span><span class="m">0</span>
<span class="m">2017</span>-07-28<span class="w"> </span><span class="m">15</span>:01:16.240974<span class="w"> </span>7f4dda123760<span class="w"> </span><span class="m">1</span><span class="w"> </span>
ID<span class="w"> </span>WEIGHT<span class="w"> </span>TYPE<span class="w"> </span>NAME
-4<span class="w"> </span><span class="m">36</span>.00000<span class="w"> </span>root<span class="w"> </span>root
-1<span class="w"> </span><span class="m">12</span>.00000<span class="w"> </span>host<span class="w"> </span>host0
<span class="m">0</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.0
<span class="m">1</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.1
<span class="m">2</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.2
<span class="m">3</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.3
<span class="m">4</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.4
<span class="m">5</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.5
<span class="m">6</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.6
<span class="m">7</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.7
<span class="m">8</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.8
<span class="m">9</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.9
<span class="m">10</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.10
<span class="m">11</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.11
-2<span class="w"> </span><span class="m">12</span>.00000<span class="w"> </span>host<span class="w"> </span>host1
<span class="m">12</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.12
<span class="m">13</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.13
<span class="m">14</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.14
<span class="m">15</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.15
<span class="m">16</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.16
<span class="m">17</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.17
<span class="m">18</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.18
<span class="m">19</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.19
<span class="m">20</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.20
<span class="m">21</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.21
<span class="m">22</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.22
<span class="m">23</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.23
-3<span class="w"> </span><span class="m">12</span>.00000<span class="w"> </span>host<span class="w"> </span>host2
<span class="m">24</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.24
<span class="m">25</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.25
<span class="m">26</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.26
<span class="m">27</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.27
<span class="m">28</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.28
<span class="m">29</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.29
<span class="m">30</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.30
<span class="m">31</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.31
<span class="m">32</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.32
<span class="m">33</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.33
<span class="m">34</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.34
<span class="m">35</span><span class="w"> </span><span class="m">1</span>.00000<span class="w"> </span>osd.35
</code></pre></div>
<p>Simulate the rule 0 or your own :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>crushtool<span class="w"> </span>--test<span class="w"> </span>-i<span class="w"> </span>crushmap<span class="w"> </span>--rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>--show-mappings<span class="w"> </span>--min-x<span class="w"> </span><span class="m">0</span><span class="w"> </span>--max-x<span class="w"> </span><span class="m">10</span><span class="w"> </span>--num-rep<span class="w"> </span><span class="m">2</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">[</span><span class="m">0</span>,12<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">[</span><span class="m">5</span>,24<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="o">[</span><span class="m">9</span>,14<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="o">[</span><span class="m">30</span>,11<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">4</span><span class="w"> </span><span class="o">[</span><span class="m">20</span>,10<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">5</span><span class="w"> </span><span class="o">[</span><span class="m">28</span>,0<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">6</span><span class="w"> </span><span class="o">[</span><span class="m">6</span>,34<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">7</span><span class="w"> </span><span class="o">[</span><span class="m">19</span>,31<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">8</span><span class="w"> </span><span class="o">[</span><span class="m">17</span>,26<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">9</span><span class="w"> </span><span class="o">[</span><span class="m">9</span>,20<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="o">[</span><span class="m">10</span>,33<span class="o">]</span>
crushtool<span class="w"> </span>--test<span class="w"> </span>-i<span class="w"> </span>crushmap<span class="w"> </span>--rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>--show-mappings<span class="w"> </span>--min-x<span class="w"> </span><span class="m">0</span><span class="w"> </span>--max-x<span class="w"> </span><span class="m">10</span><span class="w"> </span>--num-rep<span class="w"> </span><span class="m">3</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">[</span><span class="m">0</span>,12,32<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">[</span><span class="m">5</span>,24,20<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="o">[</span><span class="m">9</span>,14,28<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="o">[</span><span class="m">30</span>,11,13<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">4</span><span class="w"> </span><span class="o">[</span><span class="m">20</span>,10,31<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">5</span><span class="w"> </span><span class="o">[</span><span class="m">28</span>,0,12<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">6</span><span class="w"> </span><span class="o">[</span><span class="m">6</span>,34,14<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">7</span><span class="w"> </span><span class="o">[</span><span class="m">19</span>,31,6<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">8</span><span class="w"> </span><span class="o">[</span><span class="m">17</span>,26,5<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">9</span><span class="w"> </span><span class="o">[</span><span class="m">9</span>,20,30<span class="o">]</span>
CRUSH<span class="w"> </span>rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>x<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="o">[</span><span class="m">10</span>,33,12<span class="o">]</span>
</code></pre></div>
<p>In general it's going well. But in some cases it could be better to test.</p>Dealing with some osd timeouts2017-03-03T11:55:58+01:002017-03-03T11:55:58+01:00Laurent Barbetag:cephnotes.ksperis.com,2017-03-03:/blog/2017/03/03/dealing-with-some-osd-timeouts/<p>In some cases, some operations may take a little longer to be processed by the osd. And the operation may fail, or even make the OSD to suicide.
There are many parameters for these timeouts. Some examples :</p>
<h2>Thread suicide timed out</h2>
<div class="highlight"><pre><span></span><code><span class="n">heartbeat_map</span><span class="w"> </span><span class="n">is_healthy</span><span class="w"> </span><span class="s1">'OSD::osd_op_tp thread 0x7f1ee3ca7700'</span><span class="w"> </span><span class="n">had</span><span class="w"> </span><span class="n">suicide</span><span class="w"> </span><span class="n">timed …</span></code></pre></div><p>In some cases, some operations may take a little longer to be processed by the osd. And the operation may fail, or even make the OSD to suicide.
There are many parameters for these timeouts. Some examples :</p>
<h2>Thread suicide timed out</h2>
<div class="highlight"><pre><span></span><code><span class="n">heartbeat_map</span><span class="w"> </span><span class="n">is_healthy</span><span class="w"> </span><span class="s1">'OSD::osd_op_tp thread 0x7f1ee3ca7700'</span><span class="w"> </span><span class="n">had</span><span class="w"> </span><span class="n">suicide</span><span class="w"> </span><span class="n">timed</span><span class="w"> </span><span class="n">out</span><span class="w"> </span><span class="n">after</span><span class="w"> </span><span class="mi">150</span>
<span class="n">common</span><span class="o">/</span><span class="n">HeartbeatMap</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="w"> </span><span class="n">In</span><span class="w"> </span><span class="n">function</span><span class="w"> </span><span class="s1">'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)'</span><span class="w"> </span><span class="n">thread</span><span class="w"> </span><span class="mi">7</span><span class="n">f1f0c2a3700</span><span class="w"> </span><span class="n">time</span><span class="w"> </span><span class="mi">2017</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">03</span><span class="w"> </span><span class="mi">11</span><span class="p">:</span><span class="mi">03</span><span class="p">:</span><span class="mf">46.550118</span>
<span class="n">common</span><span class="o">/</span><span class="n">HeartbeatMap</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="w"> </span><span class="mi">79</span><span class="p">:</span><span class="w"> </span><span class="n">FAILED</span><span class="w"> </span><span class="nb">assert</span><span class="p">(</span><span class="mi">0</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">"hit suicide timeout"</span><span class="p">)</span>
</code></pre></div>
<p>In ceph.conf :</p>
<div class="highlight"><pre><span></span><code><span class="k">[osd]</span>
<span class="na">osd_op_thread_suicide_timeout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">900</span>
</code></pre></div>
<h2>Operation thread timeout</h2>
<div class="highlight"><pre><span></span><code>heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fd306416700' had timed out after 15
</code></pre></div>
<div class="highlight"><pre><span></span><code>ceph tell osd.XX injectargs --osd-op-thread-timeout 90
(default value is 15s)
</code></pre></div>
<h2>Recovery thread timout</h2>
<div class="highlight"><pre><span></span><code>heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7f4c2edab700' had timed out after 30
</code></pre></div>
<div class="highlight"><pre><span></span><code>ceph tell osd.XX injectargs --osd-recovery-thread-timeout 180
(default value is 30s)
</code></pre></div>
<p>For more details, please refer to ceph documentation :</p>
<p>http://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/</p>Erasure code on small clusters2017-01-27T16:47:51+01:002017-01-27T16:47:51+01:00Laurent Barbetag:cephnotes.ksperis.com,2017-01-27:/blog/2017/01/27/erasure-code-on-small-clusters/<p>Erasure code is rather designed for clusters with a sufficient size. However if you want to use it with a small amount of hosts you can also adapt the crushmap for a better matching distribution to your need.</p>
<p>Here a first example for distributing data with 1 host OR 2 …</p><p>Erasure code is rather designed for clusters with a sufficient size. However if you want to use it with a small amount of hosts you can also adapt the crushmap for a better matching distribution to your need.</p>
<p>Here a first example for distributing data with 1 host OR 2 drive fault tolerance with k=4, m=2 on 3 hosts and more.</p>
<div class="highlight"><pre><span></span><code><span class="nx">rule</span><span class="w"> </span><span class="nx">erasure_ruleset</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">ruleset</span><span class="w"> </span><span class="nx">X</span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nx">erasure</span>
<span class="w"> </span><span class="nx">min_size</span><span class="w"> </span><span class="mi">6</span>
<span class="w"> </span><span class="nx">max_size</span><span class="w"> </span><span class="mi">6</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">take</span><span class="w"> </span><span class="k">default</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">choose</span><span class="w"> </span><span class="nx">indep</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nx">host</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">choose</span><span class="w"> </span><span class="nx">indep</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nx">osd</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">emit</span>
<span class="p">}</span>
</code></pre></div>
<!-- more -->
<p>Testing it on sample crushmap :</p>
<div class="highlight"><pre><span></span><code><span class="c1"># crushtool --test -i crushmap.bin --rule 1 --show-mappings --x 1 --num-rep 6</span>
<span class="n">CRUSH</span><span class="w"> </span><span class="n">rule</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">8</span><span class="p">,</span><span class="mi">7</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">3</span><span class="p">]</span>
<span class="c1"># crushtool --test -i crushmap.bin --tree</span>
<span class="n">ID</span><span class="w"> </span><span class="n">WEIGHT</span><span class="w"> </span><span class="n">TYPE</span><span class="w"> </span><span class="n">NAME</span>
<span class="o">-</span><span class="mi">5</span><span class="w"> </span><span class="mf">4.00000</span><span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="n">default</span>
<span class="o">-</span><span class="mi">1</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">host</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">01</span>
<span class="mi">0</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">0</span><span class="w"> </span><span class="o"><--</span><span class="w"> </span><span class="n">DATA</span>
<span class="mi">1</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">1</span><span class="w"> </span><span class="o"><--</span><span class="w"> </span><span class="n">DATA</span>
<span class="mi">2</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">2</span>
<span class="o">-</span><span class="mi">2</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">host</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">02</span>
<span class="mi">3</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">3</span><span class="w"> </span><span class="o"><--</span><span class="w"> </span><span class="n">PARITY</span>
<span class="mi">4</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">4</span>
<span class="mi">5</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">5</span><span class="w"> </span><span class="o"><--</span><span class="w"> </span><span class="n">PARITY</span>
<span class="o">-</span><span class="mi">3</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">host</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">03</span>
<span class="mi">6</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">6</span>
<span class="mi">7</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">7</span><span class="w"> </span><span class="o"><--</span><span class="w"> </span><span class="n">DATA</span>
<span class="mi">8</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">8</span><span class="w"> </span><span class="o"><--</span><span class="w"> </span><span class="n">DATA</span>
<span class="o">-</span><span class="mi">4</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">host</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">04</span>
<span class="mi">9</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">9</span>
<span class="mi">10</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">10</span>
<span class="mi">11</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">11</span>
</code></pre></div>
<p>Here is an other example for distributing data with ( 1 host AND 1 drive ) OR ( 3 drives ) fault tolerance with k=5, m=3 on 4 hosts and more.</p>
<div class="highlight"><pre><span></span><code><span class="nx">rule</span><span class="w"> </span><span class="nx">erasure_ruleset</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">ruleset</span><span class="w"> </span><span class="nx">X</span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nx">erasure</span>
<span class="w"> </span><span class="nx">min_size</span><span class="w"> </span><span class="mi">8</span>
<span class="w"> </span><span class="nx">max_size</span><span class="w"> </span><span class="mi">8</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">take</span><span class="w"> </span><span class="k">default</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">choose</span><span class="w"> </span><span class="nx">indep</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nx">host</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">choose</span><span class="w"> </span><span class="nx">indep</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nx">osd</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">emit</span>
<span class="p">}</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="c1"># crushtool --test -i crushmap.bin --rule 2 --show-mappings --x 1 --num-rep 8</span>
<span class="n">CRUSH</span><span class="w"> </span><span class="n">rule</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">8</span><span class="p">,</span><span class="mi">7</span><span class="p">,</span><span class="mi">9</span><span class="p">,</span><span class="mi">11</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">3</span><span class="p">]</span>
<span class="c1"># crushtool --test -i crushmap.bin --tree</span>
<span class="n">ID</span><span class="w"> </span><span class="n">WEIGHT</span><span class="w"> </span><span class="n">TYPE</span><span class="w"> </span><span class="n">NAME</span>
<span class="o">-</span><span class="mi">5</span><span class="w"> </span><span class="mf">4.00000</span><span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="n">default</span>
<span class="o">-</span><span class="mi">1</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">host</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">01</span>
<span class="mi">0</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">0</span><span class="w"> </span><span class="o"><--</span><span class="w"> </span><span class="n">DATA</span>
<span class="mi">1</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">1</span><span class="w"> </span><span class="o"><--</span><span class="w"> </span><span class="n">DATA</span>
<span class="mi">2</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">2</span>
<span class="o">-</span><span class="mi">2</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">host</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">02</span>
<span class="mi">3</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">3</span><span class="w"> </span><span class="o"><--</span><span class="w"> </span><span class="n">PARITY</span>
<span class="mi">4</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">4</span>
<span class="mi">5</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">5</span><span class="w"> </span><span class="o"><--</span><span class="w"> </span><span class="n">PARITY</span>
<span class="o">-</span><span class="mi">3</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">host</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">03</span>
<span class="mi">6</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">6</span>
<span class="mi">7</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">7</span><span class="w"> </span><span class="o"><--</span><span class="w"> </span><span class="n">DATA</span>
<span class="mi">8</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">8</span><span class="w"> </span><span class="o"><--</span><span class="w"> </span><span class="n">DATA</span>
<span class="o">-</span><span class="mi">4</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">host</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">04</span>
<span class="mi">9</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">9</span><span class="w"> </span><span class="o"><--</span><span class="w"> </span><span class="n">DATA</span>
<span class="mi">10</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">10</span>
<span class="mi">11</span><span class="w"> </span><span class="mf">1.00000</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">11</span><span class="w"> </span><span class="o"><--</span><span class="w"> </span><span class="n">PARITY</span>
</code></pre></div>
<p>And a last example for distributing data with ( 1 host AND 2 drives ) OR ( 4 drives ) fault tolerance with k=8, m=4 on 4 hosts and more.</p>
<div class="highlight"><pre><span></span><code><span class="nx">rule</span><span class="w"> </span><span class="nx">erasure_ruleset</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">ruleset</span><span class="w"> </span><span class="nx">X</span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nx">erasure</span>
<span class="w"> </span><span class="nx">min_size</span><span class="w"> </span><span class="mi">12</span>
<span class="w"> </span><span class="nx">max_size</span><span class="w"> </span><span class="mi">12</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">take</span><span class="w"> </span><span class="k">default</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">choose</span><span class="w"> </span><span class="nx">indep</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nx">host</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">choose</span><span class="w"> </span><span class="nx">indep</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nx">osd</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">emit</span>
<span class="p">}</span>
</code></pre></div>
<p>An other possibility is to use LRC Erasure code plugin :</p>
<p>http://docs.ceph.com/docs/master/rados/operations/erasure-code-lrc/</p>Crushmap for 2 DC2017-01-23T10:04:14+01:002023-11-06T13:40:00+01:00Laurent Barbetag:cephnotes.ksperis.com,2017-01-23:/blog/2017/01/23/crushmap-for-2-dc/<p>Update 2021: Since Pacific version, there is a specific operating mode for the monitors in case of an stretched cluster. See : <a href="https://docs.ceph.com/en/latest/rados/operations/stretch-mode/">https://docs.ceph.com/en/latest/rados/operations/stretch-mode/</a>
For more informations, check out <a href="https://archive.fosdem.org/2020/schedule/event/sds_ceph_stretch_clusters/attachments/slides/3990/export/events/attachments/sds_ceph_stretch_clusters/slides/3990/2020_02_02_Stretch_Clusters.pdf">Greg's talk on Fostem</a></p>
<p>An example of crushmap for 2 Datacenter replication with 2 or …</p><p>Update 2021: Since Pacific version, there is a specific operating mode for the monitors in case of an stretched cluster. See : <a href="https://docs.ceph.com/en/latest/rados/operations/stretch-mode/">https://docs.ceph.com/en/latest/rados/operations/stretch-mode/</a>
For more informations, check out <a href="https://archive.fosdem.org/2020/schedule/event/sds_ceph_stretch_clusters/attachments/slides/3990/export/events/attachments/sds_ceph_stretch_clusters/slides/3990/2020_02_02_Stretch_Clusters.pdf">Greg's talk on Fostem</a></p>
<p>An example of crushmap for 2 Datacenter replication with 2 or 3 replica :</p>
<div class="highlight"><pre><span></span><code><span class="nx">rule</span><span class="w"> </span><span class="nx">replicated_ruleset</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">ruleset</span><span class="w"> </span><span class="nx">X</span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nx">replicated</span>
<span class="w"> </span><span class="nx">min_size</span><span class="w"> </span><span class="mi">2</span>
<span class="w"> </span><span class="nx">max_size</span><span class="w"> </span><span class="mi">3</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">take</span><span class="w"> </span><span class="k">default</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">choose</span><span class="w"> </span><span class="nx">firstn</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nx">datacenter</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">chooseleaf</span><span class="w"> </span><span class="nx">firstn</span><span class="w"> </span><span class="o">-</span><span class="mi">1</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nx">host</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">emit</span>
<span class="p">}</span>
</code></pre></div>
<p>This working well with pool size=2 (not recommended!) or 3.
If you set pool size more than 3 (and increase the max_size in crush), be careful : you will have n-1 replica on one side and only one on the other datacenter.</p>
<p>If you want to be able to write data even when one of the datacenters is inaccessible, pool min_size should be set at 1 even if size is set to 3. In this case, pay attention to the monitors location.</p>
<!-- more -->
<p>Other posts about crushmap : <a href="http://cephnotes.ksperis.com/tag/crushmap.html">http://cephnotes.ksperis.com/tag/crushmap.html</a></p>Change log level on the fly to Ceph daemons2017-01-20T11:55:53+01:002017-01-20T11:55:53+01:00Laurent Barbetag:cephnotes.ksperis.com,2017-01-20:/blog/2017/01/20/change-log-level-on-the-fly-to-ceph-daemons/<p>Aaahhh full disk this morning.
Sometimes the logs can go crazy, and the files can quickly reach several gigabytes.</p>
<p>Show debug option (on host) :</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Look at log file</span>
<span class="n">tail</span><span class="w"> </span><span class="o">-</span><span class="n">n</span><span class="w"> </span><span class="mi">1000</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">ceph</span><span class="o">-</span><span class="n">osd</span><span class="o">.</span><span class="mf">33.</span><span class="n">log</span>
<span class="c1"># Check debug levels</span>
<span class="n">ceph</span><span class="w"> </span><span class="n">daemon</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">33</span><span class="w"> </span><span class="n">config</span><span class="w"> </span><span class="n">show</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">grep</span><span class="w"> </span><span class="s1">'"debug_'</span>
<span class="w"> </span><span class="s2">"debug_none …</span></code></pre></div><p>Aaahhh full disk this morning.
Sometimes the logs can go crazy, and the files can quickly reach several gigabytes.</p>
<p>Show debug option (on host) :</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Look at log file</span>
<span class="n">tail</span><span class="w"> </span><span class="o">-</span><span class="n">n</span><span class="w"> </span><span class="mi">1000</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">ceph</span><span class="o">-</span><span class="n">osd</span><span class="o">.</span><span class="mf">33.</span><span class="n">log</span>
<span class="c1"># Check debug levels</span>
<span class="n">ceph</span><span class="w"> </span><span class="n">daemon</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">33</span><span class="w"> </span><span class="n">config</span><span class="w"> </span><span class="n">show</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">grep</span><span class="w"> </span><span class="s1">'"debug_'</span>
<span class="w"> </span><span class="s2">"debug_none"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0\/5"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"debug_lockdep"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0\/1"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"debug_context"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0\/1"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"debug_crush"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1\/1"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"debug_mds"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1\/5"</span><span class="p">,</span>
<span class="w"> </span><span class="o">...</span>
<span class="w"> </span><span class="s2">"debug_filestore"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1\/5"</span><span class="p">,</span>
<span class="w"> </span><span class="o">...</span>
</code></pre></div>
<p>In my case it was about filestore, so "ceph tell" is my friend to apply the new value to the whole cluster (on admin host) :</p>
<div class="highlight"><pre><span></span><code>ceph tell osd.* injectargs --debug-filestore 0/5
</code></pre></div>
<p>Now you can remove the log file on reopen it :</p>
<div class="highlight"><pre><span></span><code><span class="n">rm</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">ceph</span><span class="o">-</span><span class="n">osd</span><span class="o">.</span><span class="mf">33.</span><span class="n">log</span>
<span class="n">ceph</span><span class="w"> </span><span class="n">daemon</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">33</span><span class="w"> </span><span class="nb">log</span><span class="w"> </span><span class="n">reopen</span>
</code></pre></div>
<p>Then it will remain to be added in the ceph.conf file (on each osd hosts) :</p>
<div class="highlight"><pre><span></span><code><span class="k">[osd]</span>
<span class="w"> </span><span class="na">debug filestore</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">0/5</span>
</code></pre></div>
<!-- more -->
<p>For more information from ceph documentation :
http://docs.ceph.com/docs/master/rados/troubleshooting/log-and-debug/</p>Main new features in the latest versions of ceph2017-01-06T14:19:08+01:002017-01-06T14:19:08+01:00Laurent Barbetag:cephnotes.ksperis.com,2017-01-06:/blog/2017/01/06/main-new-features-in-the-latest-versions-of-ceph/<p>It's always pleasant to see how fast new features appear in Ceph. :)</p>
<p>Here is a non-exhaustive list of some of theme on the latest releases :</p>
<h2>Kraken (October 2016)</h2>
<ul>
<li>BlueStore declared as stable</li>
<li>AsyncMessenger</li>
<li>RGW : metadata indexing via Elasticseasrch, index resharding, compression</li>
<li>S3 bucket lifecycle API, RGW Export NFS version 3 …</li></ul><p>It's always pleasant to see how fast new features appear in Ceph. :)</p>
<p>Here is a non-exhaustive list of some of theme on the latest releases :</p>
<h2>Kraken (October 2016)</h2>
<ul>
<li>BlueStore declared as stable</li>
<li>AsyncMessenger</li>
<li>RGW : metadata indexing via Elasticseasrch, index resharding, compression</li>
<li>S3 bucket lifecycle API, RGW Export NFS version 3 throw Ganesha</li>
<li>Rados support overwrites on erasure-coded pools / RBD on erasure coded pool (experimental)</li>
</ul>
<h2>Jewel (April 2016)</h2>
<ul>
<li>CephFS declared as stable</li>
<li>RGW multisite rearchitected (Allow active/active configuration)</li>
<li>AWS4 compatibility</li>
<li>RBD mirroring</li>
<li>BlueStore (experimental)</li>
</ul>
<!-- more -->
<h2>Infernalis (November 2015)</h2>
<ul>
<li>Erasure coding declared as stable and support many new features</li>
<li>New features for Swift API (Object expiration,...)</li>
<li>Systemd</li>
</ul>
<h2>Hammer (April 2015)</h2>
<ul>
<li>RGW object versioning, bucket sharding</li>
<li>Crush straw2</li>
</ul>
<h2>Giant (October 2014)</h2>
<ul>
<li>LRC erasure code</li>
<li>CephFS journal recovery, diagnostic tools</li>
</ul>
<h2>Firefly (May 2014)</h2>
<ul>
<li>Erasure coding</li>
<li>Cache tiering</li>
<li>Key/value OSD backend</li>
<li>Standalone radosgw (with civetweb)</li>
</ul>
<p>Maybe this will make you want to upgrade your cluster.</p>
<p>http://docs.ceph.com/docs/master/releases/</p>Check OSD version2016-05-26T14:45:04+02:002016-05-26T14:45:04+02:00Laurent Barbetag:cephnotes.ksperis.com,2016-05-26:/blog/2016/05/26/check-osd-version/<p>Occasionally it may be useful to check the version of the OSD on the entire cluster :</p>
<div class="highlight"><pre><span></span><code>ceph<span class="w"> </span>tell<span class="w"> </span>osd.*<span class="w"> </span>version
</code></pre></div>
<!--more-->
<div class="highlight"><pre><span></span><code>osd.0: {
"version": "ceph version 0.94.5-224-g4051bc2 (4051bc2a5e4313ac0f6236d7a34ed5fb4a1d9ea2)"
}
osd.1: {
"version": "ceph version 0.94.5-224-g4051bc2 (4051bc2a5e4313ac0f6236d7a34ed5fb4a1d9ea2)"
}
osd.2: {
"version": "ceph version 0.94.5-224-g4051bc2 (4051bc2a5e4313ac0f6236d7a34ed5fb4a1d9ea2)"
}
osd.3: {
"version …</code></pre></div><p>Occasionally it may be useful to check the version of the OSD on the entire cluster :</p>
<div class="highlight"><pre><span></span><code>ceph<span class="w"> </span>tell<span class="w"> </span>osd.*<span class="w"> </span>version
</code></pre></div>
<!--more-->
<div class="highlight"><pre><span></span><code>osd.0: {
"version": "ceph version 0.94.5-224-g4051bc2 (4051bc2a5e4313ac0f6236d7a34ed5fb4a1d9ea2)"
}
osd.1: {
"version": "ceph version 0.94.5-224-g4051bc2 (4051bc2a5e4313ac0f6236d7a34ed5fb4a1d9ea2)"
}
osd.2: {
"version": "ceph version 0.94.5-224-g4051bc2 (4051bc2a5e4313ac0f6236d7a34ed5fb4a1d9ea2)"
}
osd.3: {
"version": "ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)"
}
osd.4: {
"version": "ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)"
}
osd.5: {
"version": "ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)"
}
osd.6: {
"version": "ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)"
}
...
</code></pre></div>Find the OSD location2016-05-24T19:24:59+02:002016-05-24T19:24:59+02:00Laurent Barbetag:cephnotes.ksperis.com,2016-05-24:/blog/2016/05/24/find-the-osd-location/<p>Of course, the simplest way is using the command <code>ceph osd tree</code>.</p>
<p>Note that, if an osd is down, you can see "last address" in <code>ceph health detail</code> :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>health<span class="w"> </span>detail
...
osd.37<span class="w"> </span>is<span class="w"> </span>down<span class="w"> </span>since<span class="w"> </span>epoch<span class="w"> </span><span class="m">16952</span>,<span class="w"> </span>last<span class="w"> </span>address<span class="w"> </span><span class="m">172</span>.16.4.68:6804/628
</code></pre></div>
<p>Also, you can use …</p><p>Of course, the simplest way is using the command <code>ceph osd tree</code>.</p>
<p>Note that, if an osd is down, you can see "last address" in <code>ceph health detail</code> :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>health<span class="w"> </span>detail
...
osd.37<span class="w"> </span>is<span class="w"> </span>down<span class="w"> </span>since<span class="w"> </span>epoch<span class="w"> </span><span class="m">16952</span>,<span class="w"> </span>last<span class="w"> </span>address<span class="w"> </span><span class="m">172</span>.16.4.68:6804/628
</code></pre></div>
<p>Also, you can use:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>find<span class="w"> </span><span class="m">37</span>
<span class="o">{</span>
<span class="w"> </span><span class="s2">"osd"</span>:<span class="w"> </span><span class="m">37</span>,
<span class="w"> </span><span class="s2">"ip"</span>:<span class="w"> </span><span class="s2">"172.16.4.68:6804\/636"</span>,
<span class="w"> </span><span class="s2">"crush_location"</span>:<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="s2">"datacenter"</span>:<span class="w"> </span><span class="s2">"pa2.ssdr"</span>,
<span class="w"> </span><span class="s2">"host"</span>:<span class="w"> </span><span class="s2">"lxc-ceph-main-front-osd-03.ssdr"</span>,
<span class="w"> </span><span class="s2">"physical-host"</span>:<span class="w"> </span><span class="s2">"store-front-03.ssdr"</span>,
<span class="w"> </span><span class="s2">"rack"</span>:<span class="w"> </span><span class="s2">"pa2-104.ssdr"</span>,
<span class="w"> </span><span class="s2">"root"</span>:<span class="w"> </span><span class="s2">"ssdr"</span>
<span class="w"> </span><span class="o">}</span>
<span class="o">}</span>
</code></pre></div>
<p>To get partition UUID, you can use <code>ceph osd dump</code> (see at the end of the line) :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>dump<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>^osd.37
osd.37<span class="w"> </span>down<span class="w"> </span>out<span class="w"> </span>weight<span class="w"> </span><span class="m">0</span><span class="w"> </span>up_from<span class="w"> </span><span class="m">56847</span><span class="w"> </span>up_thru<span class="w"> </span><span class="m">57230</span><span class="w"> </span>down_at<span class="w"> </span><span class="m">57538</span><span class="w"> </span>last_clean_interval<span class="w"> </span><span class="o">[</span><span class="m">56640</span>,56844<span class="o">)</span><span class="w"> </span><span class="m">172</span>.16.4.72:6801/16852<span class="w"> </span><span class="m">172</span>.17.2.37:6801/16852<span class="w"> </span><span class="m">172</span>.17.2.37:6804/16852<span class="w"> </span><span class="m">172</span>.16.4.72:6804/16852<span class="w"> </span>exists<span class="w"> </span>d7ab9ac1-c68c-4594-b25e-48d3a7cfd182
$<span class="w"> </span>ssh<span class="w"> </span><span class="m">172</span>.17.2.37<span class="w"> </span><span class="p">|</span><span class="w"> </span>blkid<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>d7ab9ac1-c68c-4594-b25e-48d3a7cfd182
/dev/sdg1:<span class="w"> </span><span class="nv">UUID</span><span class="o">=</span><span class="s2">"98594f17-eae5-45f8-9e90-cd25a8f89442"</span><span class="w"> </span><span class="nv">TYPE</span><span class="o">=</span><span class="s2">"xfs"</span><span class="w"> </span><span class="nv">PARTLABEL</span><span class="o">=</span><span class="s2">"ceph data"</span><span class="w"> </span><span class="nv">PARTUUID</span><span class="o">=</span><span class="s2">"d7ab9ac1-c68c-4594-b25e-48d3a7cfd182"</span>
<span class="c1">#(Depending on how the partitions are created, PARTUUID label is not necessarily present.)</span>
</code></pre></div>LXC 2.0.0 First support for Ceph RBD2016-04-14T14:10:12+02:002016-04-14T14:10:12+02:00Laurent Barbetag:cephnotes.ksperis.com,2016-04-14:/blog/2016/04/14/lxc-2-dot-0-0-first-support-for-ceph-rbd/<p>FYI, the first RBD support has been added to LXC commands.</p>
<h2>Example :</h2>
<div class="highlight"><pre><span></span><code><span class="c1"># Install LXC 2.0.0 (ubuntu) :</span>
$<span class="w"> </span>add-apt-repository<span class="w"> </span>ppa:ubuntu-lxc/lxc-stable
$<span class="w"> </span>apt-get<span class="w"> </span>update
$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>lxc
<span class="c1"># Add a ceph pool for lxc bloc devices :</span>
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>lxc<span class="w"> </span><span class="m">64</span><span class="w"> </span><span class="m">64</span>
<span class="c1"># To create the container, you only need to …</span></code></pre></div><p>FYI, the first RBD support has been added to LXC commands.</p>
<h2>Example :</h2>
<div class="highlight"><pre><span></span><code><span class="c1"># Install LXC 2.0.0 (ubuntu) :</span>
$<span class="w"> </span>add-apt-repository<span class="w"> </span>ppa:ubuntu-lxc/lxc-stable
$<span class="w"> </span>apt-get<span class="w"> </span>update
$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>lxc
<span class="c1"># Add a ceph pool for lxc bloc devices :</span>
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>lxc<span class="w"> </span><span class="m">64</span><span class="w"> </span><span class="m">64</span>
<span class="c1"># To create the container, you only need to specify "rbd" backingstore :</span>
$<span class="w"> </span>lxc-create<span class="w"> </span>-n<span class="w"> </span>ctn1<span class="w"> </span>-B<span class="w"> </span>rbd<span class="w"> </span>-t<span class="w"> </span>debian
/dev/rbd0
debootstrap<span class="w"> </span>est<span class="w"> </span>/usr/sbin/debootstrap
Checking<span class="w"> </span>cache<span class="w"> </span>download<span class="w"> </span><span class="k">in</span><span class="w"> </span>/var/cache/lxc/debian/rootfs-jessie-amd64<span class="w"> </span>...<span class="w"> </span>
Copying<span class="w"> </span>rootfs<span class="w"> </span>to<span class="w"> </span>/usr/lib/x86_64-linux-gnu/lxc...
Generation<span class="w"> </span>complete.
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>showmapped<span class="w"> </span>
id<span class="w"> </span>pool<span class="w"> </span>image<span class="w"> </span>snap<span class="w"> </span>device<span class="w"> </span>
<span class="m">0</span><span class="w"> </span>lxc<span class="w"> </span>ctn1<span class="w"> </span>-<span class="w"> </span>/dev/rbd0
$<span class="w"> </span>rbd<span class="w"> </span>-p<span class="w"> </span>lxc<span class="w"> </span>info<span class="w"> </span>ctn1
rbd<span class="w"> </span>image<span class="w"> </span><span class="s1">'ctn1'</span>:
<span class="w"> </span>size<span class="w"> </span><span class="m">1024</span><span class="w"> </span>MB<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">256</span><span class="w"> </span>objects
<span class="w"> </span>order<span class="w"> </span><span class="m">22</span><span class="w"> </span><span class="o">(</span><span class="m">4096</span><span class="w"> </span>kB<span class="w"> </span>objects<span class="o">)</span>
<span class="w"> </span>block_name_prefix:<span class="w"> </span>rb.0.1217d.74b0dc51
<span class="w"> </span>format:<span class="w"> </span><span class="m">1</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>lxc-start<span class="w"> </span>-n<span class="w"> </span>ctn1
$<span class="w"> </span>lxc-attach<span class="w"> </span>-n<span class="w"> </span>ctn1
ctn1$<span class="w"> </span>mount<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">' / '</span>
/dev/rbd/lxc/ctn1<span class="w"> </span>on<span class="w"> </span>/<span class="w"> </span><span class="nb">type</span><span class="w"> </span>ext3<span class="w"> </span><span class="o">(</span>rw,relatime,stripe<span class="o">=</span><span class="m">1024</span>,data<span class="o">=</span>ordered<span class="o">)</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>lxc-destroy<span class="w"> </span>-n<span class="w"> </span>ctn1
Removing<span class="w"> </span>image:<span class="w"> </span><span class="m">100</span>%<span class="w"> </span>complete...done.
Destroyed<span class="w"> </span>container<span class="w"> </span>ctn1
</code></pre></div>
<!-- more -->
<h2>Some options :</h2>
<div class="highlight"><pre><span></span><code>--rbdpool POOL : will create the blockdevice in the pool named POOL, rather than the default, which is 'lxc'.
--rbdname RBDNAME : will create a blockdevice named RBDNAME rather than the default, which is the container name.
--fssize : Create a RBD of size SIZE * unit U (bBkKmMgGtT)
</code></pre></div>
<p>For example :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>lxc-create<span class="w"> </span>-n<span class="w"> </span>ctn2<span class="w"> </span>-B<span class="w"> </span>rbd<span class="w"> </span>-t<span class="w"> </span>debian<span class="w"> </span>--rbdpool<span class="o">=</span>rbd<span class="w"> </span>--rbdname<span class="o">=</span>lxc-ctn2<span class="w"> </span>--fssize<span class="o">=</span>2G
</code></pre></div>
<h2>What is not yet done:</h2>
<ul>
<li>Persistence on reboot: (RBD can be optionally added to rbdmap file)</li>
<li>Snapshots</li>
<li>Params for authentication (user, keyring...)</li>
<li>Other rbd options (format, ...)</li>
</ul>
<p>The release annoucement :</p>
<p>https://linuxcontainers.org/fr/lxc/news/#lxc-200-release-announcement-6th-of-april-2016</p>Downgrade LSI 9207 to P19 firmware2015-08-13T10:48:31+02:002015-08-13T10:48:31+02:00Laurent Barbetag:cephnotes.ksperis.com,2015-08-13:/blog/2015/08/13/downgrade-lsi-9207-to-p19-firmware/<p>After numerous problems encountered with the P20 firmware on this card model, here are the steps I followed to flash in P19 Version.</p>
<p>Since, no more problems :)</p>
<p>The model of the card is a LSI 9207-8i (SAS2308 controler) with IT FW:</p>
<div class="highlight"><pre><span></span><code>lspci | grep LSI
01:00.0 Serial Attached SCSI …</code></pre></div><p>After numerous problems encountered with the P20 firmware on this card model, here are the steps I followed to flash in P19 Version.</p>
<p>Since, no more problems :)</p>
<p>The model of the card is a LSI 9207-8i (SAS2308 controler) with IT FW:</p>
<div class="highlight"><pre><span></span><code>lspci | grep LSI
01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
</code></pre></div>
<!--more-->
<p>I use :</p>
<ul>
<li>
<p>Virtual Floppy Drive : http://sourceforge.net/projects/vfd/</p>
</li>
<li>
<p>Freedos : http://www.fdos.org/bootdisks/autogen/FDOEM.144.gz</p>
</li>
<li>
<p>Installer P20 for UEFI : http://www.avagotech.com/products/server-storage/host-bus-adapters/sas-9207-8i#downloads</p>
</li>
<li>
<p>9207_8i_Package_P19_IR_IT_Firmware_BIOS_for_MSDOS_Windows : http://www.avagotech.com/products/server-storage/host-bus-adapters/sas-9207-8i#downloads</p>
</li>
</ul>
<p>With VFD I create an new 2.88 MB floppy that contain :</p>
<div class="highlight"><pre><span></span><code>KERNEL.SYS from Freedos
sys.com from Freedos
sas2flash.efi from Installer P20 for UEFI
mptsas2.rom from 9207_8i_Package_P19_IR_IT_Firmware_BIOS_for_MSDOS_Windows
9207-8.bin from 9207_8i_Package_P19_IR_IT_Firmware_BIOS_for_MSDOS_Windows (IT firmware in my case)
</code></pre></div>
<p><img alt="Downgrade LSI" src="https://cephnotes.ksperis.com/images/lsi_P19_downgrade-0.png"></p>
<p>Virtual Storage plugin on Supermicro to connect Drive B :</p>
<p><img alt="Downgrade LSI" src="https://cephnotes.ksperis.com/images/lsi_P19_downgrade-1.png"></p>
<p>Reboot server and press F11 on startup to access boot menu and start with Built-in EFI Shell.</p>
<p><img alt="Downgrade LSI" src="https://cephnotes.ksperis.com/images/lsi_P19_downgrade-2.png"></p>
<p>Go into floppy drive (fs0:, or fs1:, fs2:, ...) and verify content :</p>
<p><img alt="Downgrade LSI" src="https://cephnotes.ksperis.com/images/lsi_P19_downgrade-3.png"></p>
<p>Before starting to flash, verify your card with <code>sas2flash -list</code>. If you have more than one controler you can specify the controler with <code>-c controler_id</code> for the next commands.</p>
<p>Next, you will need to erase all the content of the card (to avoid error like "Cannot downgrade NVDATA version...")</p>
<div class="highlight"><pre><span></span><code>sas2flash -o -e 6
</code></pre></div>
<p><img alt="Downgrade LSI" src="https://cephnotes.ksperis.com/images/lsi_P19_downgrade-4.png"></p>
<p>Flash with the new firmware and bios :</p>
<div class="highlight"><pre><span></span><code>sas2flash -f 9207-8.bin -b mptsas2.rom
</code></pre></div>
<p><img alt="Downgrade LSI" src="https://cephnotes.ksperis.com/images/lsi_P19_downgrade-5.png">
<img alt="Downgrade LSI" src="https://cephnotes.ksperis.com/images/lsi_P19_downgrade-6.png"></p>
<p>Verify the new firmware version :</p>
<p><img alt="Downgrade LSI" src="https://cephnotes.ksperis.com/images/lsi_P19_downgrade-7.png"></p>
<p>To see the firmware version on OS :</p>
<div class="highlight"><pre><span></span><code><span class="err">#</span><span class="w"> </span><span class="nx">sas2ircu</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nx">DISPLAY</span>
<span class="nx">LSI</span><span class="w"> </span><span class="nx">Corporation</span><span class="w"> </span><span class="nx">SAS2</span><span class="w"> </span><span class="nx">IR</span><span class="w"> </span><span class="nx">Configuration</span><span class="w"> </span><span class="nx">Utility</span><span class="p">.</span>
<span class="nx">Version</span><span class="w"> </span><span class="m m-Double">16.00.00.00</span><span class="w"> </span><span class="p">(</span><span class="m m-Double">2013.03.01</span><span class="p">)</span><span class="w"> </span>
<span class="nx">Copyright</span><span class="w"> </span><span class="p">(</span><span class="nx">c</span><span class="p">)</span><span class="w"> </span><span class="mi">2009</span><span class="o">-</span><span class="mi">2013</span><span class="w"> </span><span class="nx">LSI</span><span class="w"> </span><span class="nx">Corporation</span><span class="p">.</span><span class="w"> </span><span class="nx">All</span><span class="w"> </span><span class="nx">rights</span><span class="w"> </span><span class="nx">reserved</span><span class="p">.</span>
<span class="nx">Read</span><span class="w"> </span><span class="nx">configuration</span><span class="w"> </span><span class="nx">has</span><span class="w"> </span><span class="nx">been</span><span class="w"> </span><span class="nx">initiated</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">controller</span><span class="w"> </span><span class="mi">0</span>
<span class="o">------------------------------------------------------------------------</span>
<span class="nx">Controller</span><span class="w"> </span><span class="nx">information</span>
<span class="o">------------------------------------------------------------------------</span>
<span class="w"> </span><span class="nx">Controller</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="nx">SAS2308_2</span>
<span class="w"> </span><span class="nx">BIOS</span><span class="w"> </span><span class="nx">version</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="m m-Double">7.37.00.00</span>
<span class="w"> </span><span class="nx">Firmware</span><span class="w"> </span><span class="nx">version</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="m m-Double">19.00.00.00</span>
<span class="w"> </span><span class="nx">Channel</span><span class="w"> </span><span class="nx">description</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="nx">Serial</span><span class="w"> </span><span class="nx">Attached</span><span class="w"> </span><span class="nx">SCSI</span>
<span class="w"> </span><span class="nx">Initiator</span><span class="w"> </span><span class="nx">ID</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="nx">Maximum</span><span class="w"> </span><span class="nx">physical</span><span class="w"> </span><span class="nx">devices</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">1023</span>
<span class="w"> </span><span class="nx">Concurrent</span><span class="w"> </span><span class="nx">commands</span><span class="w"> </span><span class="nx">supported</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">10240</span>
<span class="w"> </span><span class="nx">Slot</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">16</span>
<span class="w"> </span><span class="nx">Segment</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="nx">Bus</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">1</span>
<span class="w"> </span><span class="nx">Device</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="nx">Function</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="nx">RAID</span><span class="w"> </span><span class="nx">Support</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="nx">No</span>
<span class="o">...</span>
</code></pre></div>
<p>SAS2Flash Utility Guide : http://www.lsi.com/sep/Documents/oracle/files/SAS2_Flash_Utility_Software_Ref_Guide.pdf</p>
<p>Thread "Lots of trouble with LSI 9207-8e HBAs - Issue caused by LSI P20 92XX HBA Firmware" https://community.nexenta.com/thread/1053</p>Get OMAP key/value size2015-06-25T16:40:15+02:002015-06-25T16:40:15+02:00Laurent Barbetag:cephnotes.ksperis.com,2015-06-25:/blog/2015/06/25/get-omap-key-slash-value-size/<p>List the total size of all keys for each object on a pool.</p>
<div class="highlight"><pre><span></span><code>object size_keys(kB) size_values(kB) total(kB) nr_keys nr_values
meta.log.44 0 1 1 0 10
data_log.78 0 56419 56419 0 406841
meta.log.36 0 1 1 0 10
data_log.71 0 56758 56758 …</code></pre></div><p>List the total size of all keys for each object on a pool.</p>
<div class="highlight"><pre><span></span><code>object size_keys(kB) size_values(kB) total(kB) nr_keys nr_values
meta.log.44 0 1 1 0 10
data_log.78 0 56419 56419 0 406841
meta.log.36 0 1 1 0 10
data_log.71 0 56758 56758 0 409426
data_log.111 0 56519 56519 0 405909
...
</code></pre></div>
<!--more-->
<p>This is based on the command <code>rados listomapvals</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nt">pool</span><span class="o">=</span><span class="s1">'.rgw.buckets.index'</span>
<span class="nt">list</span><span class="o">=</span><span class="err">`</span><span class="nt">rados</span><span class="w"> </span><span class="nt">-p</span><span class="w"> </span><span class="o">$</span><span class="nt">pool</span><span class="w"> </span><span class="nt">ls</span><span class="err">`</span>
<span class="o">(</span>
<span class="nt">echo</span><span class="w"> </span><span class="s2">"object size_keys(kB) size_values(kB) total(kB) nr_keys nr_values"</span>
<span class="nt">for</span><span class="w"> </span><span class="nt">obj</span><span class="w"> </span><span class="nt">in</span><span class="w"> </span><span class="o">$</span><span class="nt">list</span><span class="o">;</span><span class="w"> </span><span class="nt">do</span>
<span class="nt">echo</span><span class="w"> </span><span class="nt">-en</span><span class="w"> </span><span class="s2">"$obj "</span><span class="o">;</span>
<span class="nt">rados</span><span class="w"> </span><span class="nt">-p</span><span class="w"> </span><span class="o">$</span><span class="nt">pool</span><span class="w"> </span><span class="nt">listomapvals</span><span class="w"> </span><span class="o">$</span><span class="nt">obj</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nt">awk</span><span class="w"> </span><span class="s1">'</span>
<span class="s1">/^key: \(.* bytes\)/ { sizekey+= substr($2, 2, length($2)); nbkeys++ }</span>
<span class="s1">/^value: \(.* bytes\)/ { sizevalue+= substr($2, 2, length($2)); nbvalues++ }</span>
<span class="s1">END { printf("%i %i %i %i %i\n", sizekey/1024, sizevalue/1024, (sizekey+sizevalue)/1024, nbkey, nbvalues) }</span>
<span class="s1">'</span>
<span class="nt">done</span>
<span class="o">)</span>
</code></pre></div>
<p>The method is not really optimal, so it may take some time. But it helps to have an idea.</p>
<p>You can add <code>| column -t</code> at the end of the command line for better display :</p>
<div class="highlight"><pre><span></span><code><span class="nf">object</span><span class="w"> </span><span class="no">size_keys</span><span class="p">(</span><span class="no">kB</span><span class="p">)</span><span class="w"> </span><span class="no">size_values</span><span class="p">(</span><span class="no">kB</span><span class="p">)</span><span class="w"> </span><span class="no">total</span><span class="p">(</span><span class="no">kB</span><span class="p">)</span><span class="w"> </span><span class="no">nr_keys</span><span class="w"> </span><span class="no">nr_values</span>
<span class="na">.dir.default.1970130.1.250</span><span class="w"> </span><span class="mi">8863</span><span class="w"> </span><span class="mi">75592</span><span class="w"> </span><span class="mi">84455</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">538692</span>
<span class="na">.dir.default.1977514.4.161</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">55</span><span class="w"> </span><span class="mi">61</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">405</span>
<span class="na">.dir.default.1977550.10.98</span><span class="w"> </span><span class="mi">12</span><span class="w"> </span><span class="mi">83</span><span class="w"> </span><span class="mi">95</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">692</span>
<span class="na">.dir.default.1977550.5.83</span><span class="w"> </span><span class="mi">47</span><span class="w"> </span><span class="mi">434</span><span class="w"> </span><span class="mi">482</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">3074</span>
<span class="na">.dir.default.1977550.11.34</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">15</span>
<span class="na">.dir.default.1970130.1.69</span><span class="w"> </span><span class="mi">9147</span><span class="w"> </span><span class="mi">78077</span><span class="w"> </span><span class="mi">87224</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">556235</span>
<span class="na">.dir.default.1977550.8.116</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">3</span>
<span class="na">.dir.default.1977550.3.49</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">36</span>
<span class="na">.dir.default.1930743.19.0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="na">.dir.default.1985483.1.40</span><span class="w"> </span><span class="mi">204</span><span class="w"> </span><span class="mi">1868</span><span class="w"> </span><span class="mi">2073</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">13261</span>
<span class="na">.dir.default.1977514.7.77</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">66</span>
<span class="na">.dir.default.1977514.2.74</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">66</span>
<span class="na">.dir.default.1977550.8.113</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">15</span>
<span class="na">.dir.default.1977550.4.121</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="mi">40</span><span class="w"> </span><span class="mi">47</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">351</span>
<span class="na">.dir.default.1977514.8.122</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">6</span>
<span class="na">.dir.default.1985483.2.148</span><span class="w"> </span><span class="mi">578</span><span class="w"> </span><span class="mi">5423</span><span class="w"> </span><span class="mi">6001</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">38583</span>
<span class="na">...</span>
</code></pre></div>
<p>http://ceph.com/docs/master/man/8/rados/#pool-specific-commands</p>The kernel 4.1 is out2015-06-22T11:16:21+02:002015-06-22T11:16:21+02:00Laurent Barbetag:cephnotes.ksperis.com,2015-06-22:/blog/2015/06/22/the-kernel-4-dot-1-is-out/<p>This kernel version support all features for Hammer, in particular straw v2.</p>
<p>https://www.kernel.org/</p>
<!--more-->
<p>The main changes in this version:</p>
<div class="highlight"><pre><span></span><code><span class="n">rbd</span><span class="o">:</span><span class="w"> </span><span class="n">rbd_wq</span><span class="w"> </span><span class="n">comment</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">obsolete</span>
<span class="n">libceph</span><span class="o">:</span><span class="w"> </span><span class="n">announce</span><span class="w"> </span><span class="n">support</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">straw2</span><span class="w"> </span><span class="n">buckets</span>
<span class="n">crush</span><span class="o">:</span><span class="w"> </span><span class="n">straw2</span><span class="w"> </span><span class="n">bucket</span><span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">an</span><span class="w"> </span><span class="n">efficient</span><span class="w"> </span><span class="mi">64</span><span class="o">-</span><span class="n">bit</span><span class="w"> </span><span class="n">crush_ln</span><span class="o">()</span>
<span class="n">crush</span><span class="o">:</span><span class="w"> </span><span class="n">ensuring</span><span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">most</span><span class="w"> </span><span class="n">num</span><span class="o">-</span><span class="n">rep</span><span class="w"> </span><span class="n">osds …</span></code></pre></div><p>This kernel version support all features for Hammer, in particular straw v2.</p>
<p>https://www.kernel.org/</p>
<!--more-->
<p>The main changes in this version:</p>
<div class="highlight"><pre><span></span><code><span class="n">rbd</span><span class="o">:</span><span class="w"> </span><span class="n">rbd_wq</span><span class="w"> </span><span class="n">comment</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">obsolete</span>
<span class="n">libceph</span><span class="o">:</span><span class="w"> </span><span class="n">announce</span><span class="w"> </span><span class="n">support</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">straw2</span><span class="w"> </span><span class="n">buckets</span>
<span class="n">crush</span><span class="o">:</span><span class="w"> </span><span class="n">straw2</span><span class="w"> </span><span class="n">bucket</span><span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">an</span><span class="w"> </span><span class="n">efficient</span><span class="w"> </span><span class="mi">64</span><span class="o">-</span><span class="n">bit</span><span class="w"> </span><span class="n">crush_ln</span><span class="o">()</span>
<span class="n">crush</span><span class="o">:</span><span class="w"> </span><span class="n">ensuring</span><span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">most</span><span class="w"> </span><span class="n">num</span><span class="o">-</span><span class="n">rep</span><span class="w"> </span><span class="n">osds</span><span class="w"> </span><span class="n">are</span><span class="w"> </span><span class="n">selected</span>
<span class="n">crush</span><span class="o">:</span><span class="w"> </span><span class="n">drop</span><span class="w"> </span><span class="n">unnecessary</span><span class="w"> </span><span class="k">include</span><span class="w"> </span><span class="n">from</span><span class="w"> </span><span class="n">mapper</span><span class="o">.</span><span class="na">c</span>
<span class="n">ceph</span><span class="o">:</span><span class="w"> </span><span class="n">fix</span><span class="w"> </span><span class="n">uninline</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="kd">function</span>
<span class="n">ceph</span><span class="o">:</span><span class="w"> </span><span class="n">rename</span><span class="w"> </span><span class="n">snapshot</span><span class="w"> </span><span class="n">support</span>
<span class="n">ceph</span><span class="o">:</span><span class="w"> </span><span class="n">fix</span><span class="w"> </span><span class="kc">null</span><span class="w"> </span><span class="n">pointer</span><span class="w"> </span><span class="n">dereference</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">send_mds_reconnect</span><span class="o">()</span>
<span class="n">ceph</span><span class="o">:</span><span class="w"> </span><span class="n">hold</span><span class="w"> </span><span class="n">on</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">exclusive</span><span class="w"> </span><span class="n">caps</span><span class="w"> </span><span class="n">on</span><span class="w"> </span><span class="n">complete</span><span class="w"> </span><span class="n">directories</span>
<span class="n">libceph</span><span class="o">:</span><span class="w"> </span><span class="n">simplify</span><span class="w"> </span><span class="n">our</span><span class="w"> </span><span class="n">debugfs</span><span class="w"> </span><span class="n">attr</span><span class="w"> </span><span class="n">macro</span>
<span class="n">ceph</span><span class="o">:</span><span class="w"> </span><span class="n">show</span><span class="w"> </span><span class="n">non</span><span class="o">-</span><span class="k">default</span><span class="w"> </span><span class="n">options</span><span class="w"> </span><span class="n">only</span>
<span class="n">libceph</span><span class="o">:</span><span class="w"> </span><span class="n">expose</span><span class="w"> </span><span class="n">client</span><span class="w"> </span><span class="n">options</span><span class="w"> </span><span class="n">through</span><span class="w"> </span><span class="n">debugfs</span>
<span class="n">libceph</span><span class="o">,</span><span class="w"> </span><span class="n">ceph</span><span class="o">:</span><span class="w"> </span><span class="n">split</span><span class="w"> </span><span class="n">ceph_show_options</span><span class="o">()</span>
<span class="n">rbd</span><span class="o">:</span><span class="w"> </span><span class="n">mark</span><span class="w"> </span><span class="n">block</span><span class="w"> </span><span class="n">queue</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">non</span><span class="o">-</span><span class="n">rotational</span>
<span class="n">libceph</span><span class="o">:</span><span class="w"> </span><span class="n">don</span><span class="s1">'t overwrite specific con error msgs</span>
<span class="s1">ceph: cleanup unsafe requests when reconnecting is denied</span>
<span class="s1">ceph: don'</span><span class="n">t</span><span class="w"> </span><span class="n">zero</span><span class="w"> </span><span class="n">i_wrbuffer_ref</span><span class="w"> </span><span class="n">when</span><span class="w"> </span><span class="n">reconnecting</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">denied</span>
<span class="n">ceph</span><span class="o">:</span><span class="w"> </span><span class="n">don</span><span class="err">'</span><span class="n">t</span><span class="w"> </span><span class="n">mark</span><span class="w"> </span><span class="n">dirty</span><span class="w"> </span><span class="n">caps</span><span class="w"> </span><span class="n">when</span><span class="w"> </span><span class="n">there</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">no</span><span class="w"> </span><span class="n">auth</span><span class="w"> </span><span class="n">cap</span>
<span class="n">ceph</span><span class="o">:</span><span class="w"> </span><span class="n">keep</span><span class="w"> </span><span class="n">i_snap_realm</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="n">there</span><span class="w"> </span><span class="n">are</span><span class="w"> </span><span class="n">writers</span>
<span class="n">libceph</span><span class="o">:</span><span class="w"> </span><span class="n">osdmap</span><span class="o">.</span><span class="na">h</span><span class="o">:</span><span class="w"> </span><span class="n">Add</span><span class="w"> </span><span class="n">missing</span><span class="w"> </span><span class="n">format</span><span class="w"> </span><span class="n">newlines</span>
</code></pre></div>
<p>https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=1204c464458e9837320a326a9fce550e3c5ef5de</p>
<p>http://cephnotes.ksperis.com/blog/2014/01/21/feature-set-mismatch-error-on-ceph-kernel-client</p>Add support of curl_multi_wait for RadosGW on Debian Wheezy2015-06-18T17:42:39+02:002015-06-18T17:42:39+02:00Laurent Barbetag:cephnotes.ksperis.com,2015-06-18:/blog/2015/06/18/add-support-of-curl-multi-wait-for-radosgw-on-debian-wheezy/<div class="highlight"><pre><span></span><code><span class="n">WARNING</span><span class="o">:</span><span class="w"> </span><span class="n">libcurl</span><span class="w"> </span><span class="n">doesn</span><span class="err">'</span><span class="n">t</span><span class="w"> </span><span class="n">support</span><span class="w"> </span><span class="n">curl_multi_wait</span><span class="o">()</span>
<span class="n">WARNING</span><span class="o">:</span><span class="w"> </span><span class="n">cross</span><span class="w"> </span><span class="n">zone</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="n">transfer</span><span class="w"> </span><span class="n">performance</span><span class="w"> </span><span class="n">may</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">affected</span>
</code></pre></div>
<p>If you have already been confronted to this error at startup of RadosGW, the problem is the version of libcurl used.
To enable support of curl_multi_wait, you will need to compile radosgw with libcurl >= 7 …</p><div class="highlight"><pre><span></span><code><span class="n">WARNING</span><span class="o">:</span><span class="w"> </span><span class="n">libcurl</span><span class="w"> </span><span class="n">doesn</span><span class="err">'</span><span class="n">t</span><span class="w"> </span><span class="n">support</span><span class="w"> </span><span class="n">curl_multi_wait</span><span class="o">()</span>
<span class="n">WARNING</span><span class="o">:</span><span class="w"> </span><span class="n">cross</span><span class="w"> </span><span class="n">zone</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="n">transfer</span><span class="w"> </span><span class="n">performance</span><span class="w"> </span><span class="n">may</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">affected</span>
</code></pre></div>
<p>If you have already been confronted to this error at startup of RadosGW, the problem is the version of libcurl used.
To enable support of curl_multi_wait, you will need to compile radosgw with libcurl >= 7.28.0 :
http://curl.haxx.se/libcurl/c/curl_multi_wait.html</p>
<!-- more -->
<p>On debian wheezy, you can use ceph-extras repo which contains libcurl 7.29.0 to recompile ceph packages :</p>
<div class="highlight"><pre><span></span><code><span class="err">#</span><span class="w"> </span><span class="nx">apt</span><span class="o">-</span><span class="nx">cache</span><span class="w"> </span><span class="nx">policy</span><span class="w"> </span><span class="nx">libcurl4</span><span class="o">-</span><span class="nx">gnutls</span><span class="o">-</span><span class="nx">dev</span>
<span class="nx">libcurl4</span><span class="o">-</span><span class="nx">gnutls</span><span class="o">-</span><span class="nx">dev</span><span class="p">:</span>
<span class="w"> </span><span class="nx">Installed</span><span class="p">:</span><span class="w"> </span><span class="p">(</span><span class="nx">none</span><span class="p">)</span>
<span class="w"> </span><span class="nx">CandiDate</span><span class="p">:</span><span class="w"> </span><span class="m m-Double">7.26.0</span><span class="o">-</span><span class="mi">1</span><span class="o">+</span><span class="nx">wheezy13</span>
<span class="err">#</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">deb</span><span class="w"> </span><span class="nx">http</span><span class="p">:</span><span class="c1">//ceph.com/packages/ceph-extras/debian wheezy main | tee /etc/apt/sources.list.d/ceph-extras.list</span>
<span class="err">#</span><span class="w"> </span><span class="nx">apt</span><span class="o">-</span><span class="nx">get</span><span class="w"> </span><span class="nx">update</span>
<span class="err">#</span><span class="w"> </span><span class="nx">apt</span><span class="o">-</span><span class="nx">cache</span><span class="w"> </span><span class="nx">policy</span><span class="w"> </span><span class="nx">libcurl4</span><span class="o">-</span><span class="nx">gnutls</span><span class="o">-</span><span class="nx">dev</span>
<span class="nx">libcurl4</span><span class="o">-</span><span class="nx">gnutls</span><span class="o">-</span><span class="nx">dev</span><span class="p">:</span>
<span class="w"> </span><span class="nx">Installed</span><span class="p">:</span><span class="w"> </span><span class="p">(</span><span class="nx">none</span><span class="p">)</span>
<span class="w"> </span><span class="nx">CandiDate</span><span class="p">:</span><span class="w"> </span><span class="m m-Double">7.29.0</span><span class="o">-</span><span class="mi">1</span><span class="o">~</span><span class="nx">bpo70</span><span class="o">+</span><span class="mi">1</span><span class="p">.</span><span class="nx">ceph</span>
</code></pre></div>
<p>Retrieve Ceph repo on Github (in this example, I use hammer version) :</p>
<div class="highlight"><pre><span></span><code># apt-get install git build-essential automake
# git clone --recursive https://github.com/ceph/ceph.git -b hammer
# cd ceph
</code></pre></div>
<p>Install dependencies and build packages (without libbabeltrace-ctf-dev libbabeltrace-dev, here we not need...)</p>
<div class="highlight"><pre><span></span><code><span class="c1"># apt-get install autoconf automake autotools-dev libbz2-dev cryptsetup-bin debhelper default-jdk gdisk javahelper junit4 libaio-dev libatomic-ops-dev libblkid-dev libboost-dev libboost-program-options-dev libboost-system-dev libboost-thread-dev libcurl4-gnutls-dev libedit-dev libexpat1-dev libfcgi-dev libfuse-dev libgoogle-perftools-dev libkeyutils-dev libleveldb-dev libnss3-dev libsnappy-dev liblttng-ust-dev libtool libudev-dev libxml2-dev parted pkg-config python-nose python-virtualenv sdparm uuid-dev uuid-runtime xfslibs-dev xfsprogs xmlstarlet yasm zlib1g-dev</span>
<span class="c1"># dpkg-buildpackage -d</span>
</code></pre></div>
<p>On RadosGW host, you will need to add "ceph-extras" repo (for libcurl) and install radosgw packages and dependencies :</p>
<div class="highlight"><pre><span></span><code><span class="err">#</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">deb</span><span class="w"> </span><span class="nx">http</span><span class="p">:</span><span class="c1">//ceph.com/packages/ceph-extras/debian wheezy main | tee /etc/apt/sources.list.d/ceph-extras.list</span>
<span class="err">#</span><span class="w"> </span><span class="nx">apt</span><span class="o">-</span><span class="nx">get</span><span class="w"> </span><span class="nx">update</span>
<span class="err">#</span><span class="w"> </span><span class="nx">dpkg</span><span class="w"> </span><span class="o">-</span><span class="nx">i</span><span class="w"> </span><span class="nx">ceph</span><span class="o">-</span><span class="nx">common_</span><span class="o">*</span><span class="p">.</span><span class="nx">deb</span><span class="w"> </span><span class="nx">librbd1_</span><span class="o">*</span><span class="p">.</span><span class="nx">deb</span><span class="w"> </span><span class="nx">python</span><span class="o">-</span><span class="nx">cephfs_</span><span class="o">*</span><span class="p">.</span><span class="nx">deb</span><span class="w"> </span><span class="nx">python</span><span class="o">-</span><span class="nx">rbd_</span><span class="o">*</span><span class="p">.</span><span class="nx">deb</span><span class="w"> </span><span class="nx">librados2_</span><span class="o">*</span><span class="p">.</span><span class="nx">deb</span><span class="w"> </span><span class="nx">python</span><span class="o">-</span><span class="nx">ceph_</span><span class="o">*</span><span class="p">.</span><span class="nx">deb</span><span class="w"> </span><span class="nx">python</span><span class="o">-</span><span class="nx">rados_</span><span class="o">*</span><span class="p">.</span><span class="nx">deb</span><span class="w"> </span><span class="nx">radosgw_</span><span class="o">*</span><span class="p">.</span><span class="nx">deb</span>
</code></pre></div>Intel 520 SSD journal2015-05-19T09:54:39+02:002015-05-19T09:54:39+02:00Laurent Barbetag:cephnotes.ksperis.com,2015-05-19:/blog/2015/05/19/intel-520-ssd-journal/<p>A quick check of my Intel 520 SSD that running since 2 years on a small cluster.</p>
<div class="highlight"><pre><span></span><code><span class="n">smartctl</span><span class="w"> </span><span class="o">-</span><span class="n">a</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sda</span>
<span class="o">===</span><span class="w"> </span><span class="n">START</span><span class="w"> </span><span class="n">OF</span><span class="w"> </span><span class="n">INFORMATION</span><span class="w"> </span><span class="n">SECTION</span><span class="w"> </span><span class="o">===</span>
<span class="n">Model</span><span class="w"> </span><span class="n">Family</span><span class="p">:</span><span class="w"> </span><span class="n">Intel</span><span class="w"> </span><span class="mi">520</span><span class="w"> </span><span class="n">Series</span><span class="w"> </span><span class="n">SSDs</span>
<span class="n">Device</span><span class="w"> </span><span class="n">Model</span><span class="p">:</span><span class="w"> </span><span class="n">INTEL</span><span class="w"> </span><span class="n">SSDSC2CW060A3</span>
<span class="n">Serial</span><span class="w"> </span><span class="n">Number</span><span class="p">:</span><span class="w"> </span><span class="n">CVCV305200NB060AGN</span>
<span class="n">LU</span><span class="w"> </span><span class="n">WWN</span><span class="w"> </span><span class="n">Device</span><span class="w"> </span><span class="n">Id</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">001517</span><span class="w"> </span><span class="mi">8</span><span class="n">f36af9db</span>
<span class="n">Firmware</span><span class="w"> </span><span class="n">Version</span><span class="p">:</span><span class="w"> </span><span class="mi">400</span><span class="n">i …</span></code></pre></div><p>A quick check of my Intel 520 SSD that running since 2 years on a small cluster.</p>
<div class="highlight"><pre><span></span><code><span class="n">smartctl</span><span class="w"> </span><span class="o">-</span><span class="n">a</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sda</span>
<span class="o">===</span><span class="w"> </span><span class="n">START</span><span class="w"> </span><span class="n">OF</span><span class="w"> </span><span class="n">INFORMATION</span><span class="w"> </span><span class="n">SECTION</span><span class="w"> </span><span class="o">===</span>
<span class="n">Model</span><span class="w"> </span><span class="n">Family</span><span class="p">:</span><span class="w"> </span><span class="n">Intel</span><span class="w"> </span><span class="mi">520</span><span class="w"> </span><span class="n">Series</span><span class="w"> </span><span class="n">SSDs</span>
<span class="n">Device</span><span class="w"> </span><span class="n">Model</span><span class="p">:</span><span class="w"> </span><span class="n">INTEL</span><span class="w"> </span><span class="n">SSDSC2CW060A3</span>
<span class="n">Serial</span><span class="w"> </span><span class="n">Number</span><span class="p">:</span><span class="w"> </span><span class="n">CVCV305200NB060AGN</span>
<span class="n">LU</span><span class="w"> </span><span class="n">WWN</span><span class="w"> </span><span class="n">Device</span><span class="w"> </span><span class="n">Id</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">001517</span><span class="w"> </span><span class="mi">8</span><span class="n">f36af9db</span>
<span class="n">Firmware</span><span class="w"> </span><span class="n">Version</span><span class="p">:</span><span class="w"> </span><span class="mi">400</span><span class="n">i</span>
<span class="n">User</span><span class="w"> </span><span class="n">Capacity</span><span class="p">:</span><span class="w"> </span><span class="mi">60</span><span class="w"> </span><span class="mi">022</span><span class="w"> </span><span class="mi">480</span><span class="w"> </span><span class="mi">896</span><span class="w"> </span><span class="n">bytes</span><span class="w"> </span><span class="p">[</span><span class="mi">60</span><span class="p">,</span><span class="mi">0</span><span class="w"> </span><span class="n">GB</span><span class="p">]</span>
<span class="n">Sector</span><span class="w"> </span><span class="n">Size</span><span class="p">:</span><span class="w"> </span><span class="mi">512</span><span class="w"> </span><span class="n">bytes</span><span class="w"> </span><span class="n">logical</span><span class="o">/</span><span class="n">physical</span>
<span class="n">ID</span><span class="c1"># ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE</span>
<span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="n">Reallocated_Sector_Ct</span><span class="w"> </span><span class="mh">0x0032</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Old_age</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="n">Power_On_Hours_and_Msec</span><span class="w"> </span><span class="mh">0x0032</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Old_age</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">910315</span><span class="n">h</span><span class="o">+</span><span class="mi">05</span><span class="n">m</span><span class="o">+</span><span class="mf">29.420</span><span class="n">s</span>
<span class="w"> </span><span class="mi">12</span><span class="w"> </span><span class="n">Power_Cycle_Count</span><span class="w"> </span><span class="mh">0x0032</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Old_age</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">13</span>
<span class="mi">170</span><span class="w"> </span><span class="n">Available_Reservd_Space</span><span class="w"> </span><span class="mh">0x0033</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">010</span><span class="w"> </span><span class="n">Pre</span><span class="o">-</span><span class="n">fail</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">0</span>
<span class="mi">171</span><span class="w"> </span><span class="n">Program_Fail_Count</span><span class="w"> </span><span class="mh">0x0032</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Old_age</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">0</span>
<span class="mi">172</span><span class="w"> </span><span class="n">Erase_Fail_Count</span><span class="w"> </span><span class="mh">0x0032</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Old_age</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">0</span>
<span class="mi">174</span><span class="w"> </span><span class="n">Unexpect_Power_Loss_Ct</span><span class="w"> </span><span class="mh">0x0032</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Old_age</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">13</span>
<span class="mi">184</span><span class="w"> </span><span class="n">End</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">End_Error</span><span class="w"> </span><span class="mh">0x0033</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">090</span><span class="w"> </span><span class="n">Pre</span><span class="o">-</span><span class="n">fail</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">0</span>
<span class="mi">187</span><span class="w"> </span><span class="n">Uncorrectable_Error_Cnt</span><span class="w"> </span><span class="mh">0x000f</span><span class="w"> </span><span class="mi">117</span><span class="w"> </span><span class="mi">117</span><span class="w"> </span><span class="mi">050</span><span class="w"> </span><span class="n">Pre</span><span class="o">-</span><span class="n">fail</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">153797776</span>
<span class="mi">192</span><span class="w"> </span><span class="n">Power</span><span class="o">-</span><span class="n">Off_Retract_Count</span><span class="w"> </span><span class="mh">0x0032</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Old_age</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">13</span>
<span class="mi">225</span><span class="w"> </span><span class="n">Host_Writes_32MiB</span><span class="w"> </span><span class="mh">0x0032</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Old_age</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1367528</span>
<span class="mi">226</span><span class="w"> </span><span class="n">Workld_Media_Wear_Indic</span><span class="w"> </span><span class="mh">0x0032</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Old_age</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">65535</span>
<span class="mi">227</span><span class="w"> </span><span class="n">Workld_Host_Reads_Perc</span><span class="w"> </span><span class="mh">0x0032</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Old_age</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">3</span>
<span class="mi">228</span><span class="w"> </span><span class="n">Workload_Minutes</span><span class="w"> </span><span class="mh">0x0032</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Old_age</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">65535</span>
<span class="mi">232</span><span class="w"> </span><span class="n">Available_Reservd_Space</span><span class="w"> </span><span class="mh">0x0033</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">010</span><span class="w"> </span><span class="n">Pre</span><span class="o">-</span><span class="n">fail</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">0</span>
<span class="mi">233</span><span class="w"> </span><span class="n">Media_Wearout_Indicator</span><span class="w"> </span><span class="mh">0x0032</span><span class="w"> </span><span class="mi">093</span><span class="w"> </span><span class="mi">093</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Old_age</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">0</span>
<span class="mi">241</span><span class="w"> </span><span class="n">Host_Writes_32MiB</span><span class="w"> </span><span class="mh">0x0032</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Old_age</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1367528</span>
<span class="mi">242</span><span class="w"> </span><span class="n">Host_Reads_32MiB</span><span class="w"> </span><span class="mh">0x0032</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Old_age</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">56808</span>
<span class="mi">249</span><span class="w"> </span><span class="n">NAND_Writes_1GiB</span><span class="w"> </span><span class="mh">0x0013</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="mi">000</span><span class="w"> </span><span class="n">Pre</span><span class="o">-</span><span class="n">fail</span><span class="w"> </span><span class="n">Always</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">33624</span>
</code></pre></div>
<h2>9 - Power on hours count</h2>
<p>Cluster started since 2 years.</p>
<h2>170 Available_Reservd_Space</h2>
<p>100%</p>
<h2>174 - Unexpected power loss</h2>
<p>13 => Due to power loss on cluster. Everything has always well restarted. :)</p>
<h2>187 - Uncorrectable error count</h2>
<p>? Limit Ok</p>
<h2>233 Media Wearout Indicator</h2>
<p>093 => progressively decrease, I do not know if it's completely reliable, but it is usually a good indicator.</p>
<h2>241 - Host Writes 32MiB</h2>
<p>1367528 => 42 Tb written by host
This correspond to 60 GB per days for 3 osd. This seems normal.</p>
<h2>249 - NAND Writes 1GiB</h2>
<p>33624 => 33 Tb written on Nand
write amplification = 0.79 That is pretty good.</p>
<p>The drive is a 60.0 GB. This make each LBA written about 560 times.</p>
<p>For clusters with a little more load, Intel DC S3700 models remains my favorite, but in my case the Intel 520 do very well their job.</p>RadosGW big index2015-05-12T16:16:56+02:002015-05-12T16:16:56+02:00Laurent Barbetag:cephnotes.ksperis.com,2015-05-12:/blog/2015/05/12/radosgw-big-index/<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>.default.rgw.buckets.index<span class="w"> </span>listomapkeys<span class="w"> </span>.dir.default.1970130.1<span class="w"> </span><span class="p">|</span><span class="w"> </span>wc<span class="w"> </span>-l
<span class="m">166768275</span>
</code></pre></div>
<p>With each key containing between 100 and 250 bytes, this make a very big object for rados (several GB)... Especially when migrating it from an OSD to another (this will lock all writes), moreover, the OSD …</p><div class="highlight"><pre><span></span><code>$<span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>.default.rgw.buckets.index<span class="w"> </span>listomapkeys<span class="w"> </span>.dir.default.1970130.1<span class="w"> </span><span class="p">|</span><span class="w"> </span>wc<span class="w"> </span>-l
<span class="m">166768275</span>
</code></pre></div>
<p>With each key containing between 100 and 250 bytes, this make a very big object for rados (several GB)... Especially when migrating it from an OSD to another (this will lock all writes), moreover, the OSD containing this object will use a lot of memory ...</p>
<p>Since the hammer release it is possible to shard the bucket index. However, you can not shard an existing one but you can setup it for new buckets.
This is a very good thing for the scalability.</p>
<!-- more -->
<h2>Setting up index max shards</h2>
<p>You can specify the default number of shards for new buckets :</p>
<ul>
<li>Per zone, in regionmap :</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="err">$</span><span class="w"> </span><span class="err">radosgw</span><span class="mi">-</span><span class="err">admi</span><span class="kc">n</span><span class="w"> </span><span class="err">regio</span><span class="kc">n</span><span class="w"> </span><span class="err">ge</span><span class="kc">t</span><span class="w"> </span>
<span class="err">...</span>
<span class="nt">"zones"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"default"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"endpoints"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"http:\/\/storage.example.com:80\/"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"log_meta"</span><span class="p">:</span><span class="w"> </span><span class="s2">"true"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"log_data"</span><span class="p">:</span><span class="w"> </span><span class="s2">"true"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"bucket_index_max_shards"</span><span class="p">:</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="err"><===</span>
<span class="w"> </span><span class="p">},</span>
<span class="err">...</span>
</code></pre></div>
<ul>
<li>In in radosgw section in ceph.conf (this override the per zone value)</li>
</ul>
<div class="highlight"><pre><span></span><code>...
[client.radosgw.gateway]
rgw bucket index max shards = 8
....
</code></pre></div>
<h2>Verification :</h2>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>radosgw-admin<span class="w"> </span>metadata<span class="w"> </span>get<span class="w"> </span>bucket:mybucket<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>bucket_id
<span class="w"> </span><span class="s2">"bucket_id"</span>:<span class="w"> </span><span class="s2">"default.1970130.1"</span>
$<span class="w"> </span>radosgw-admin<span class="w"> </span>metadata<span class="w"> </span>get<span class="w"> </span>bucket.instance:mybucket:default.1970130.1<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>num_shards
<span class="w"> </span><span class="s2">"num_shards"</span>:<span class="w"> </span><span class="m">8</span>,
$<span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>.rgw.buckets.index<span class="w"> </span>ls<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>default.1970130.1
.dir.default.1970130.1.0
.dir.default.1970130.1.1
.dir.default.1970130.1.2
.dir.default.1970130.1.3
.dir.default.1970130.1.4
.dir.default.1970130.1.5
.dir.default.1970130.1.6
.dir.default.1970130.1.7
</code></pre></div>
<h2>Bucket listing impact :</h2>
<p>A simple test with ~200k objects in a bucket :</p>
<table>
<thead>
<tr>
<th style="text-align: left;">num_shard</th>
<th style="text-align: left;">time (s)</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;">0</td>
<td style="text-align: left;">25</td>
</tr>
<tr>
<td style="text-align: left;">8</td>
<td style="text-align: left;">36</td>
</tr>
<tr>
<td style="text-align: left;">128</td>
<td style="text-align: left;">109</td>
</tr>
</tbody>
</table>
<p>So, do not use buckets with thousands of shards if you do not need it, because the bucket listing will become very slow...</p>
<p>Link to the blueprint :</p>
<p>https://wiki.ceph.com/Planning/Blueprints/Hammer/rgw%3A_bucket_index_scalability</p>OpenVZ: kernel 3.10 with rbd module2015-05-04T12:06:19+02:002015-05-04T12:06:19+02:00Laurent Barbetag:cephnotes.ksperis.com,2015-05-04:/blog/2015/05/04/openvz-kernel-3-dot-10-with-rbd-module/<p>3.X Kernel for OpenVZ is out and it is compiled with rbd module:</p>
<div class="highlight"><pre><span></span><code><span class="n">root</span><span class="err">@</span><span class="n">debian</span><span class="p">:</span><span class="o">~</span><span class="c1"># uname -a</span>
<span class="n">Linux</span><span class="w"> </span><span class="n">debian</span><span class="w"> </span><span class="mf">3.10</span><span class="o">.</span><span class="mi">0</span><span class="o">-</span><span class="mi">3</span><span class="o">-</span><span class="n">pve</span><span class="w"> </span><span class="c1">#1 SMP Thu Jun 12 13:50:49 CEST 2014 x86_64 GNU/Linux</span>
<span class="n">root</span><span class="err">@</span><span class="n">debian</span><span class="p">:</span><span class="o">~</span><span class="c1"># modinfo rbd</span>
<span class="n">filename</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">modules</span><span class="o">/</span><span class="mf">3.10</span><span class="o">.</span><span class="mi">0</span><span class="o">-</span><span class="mi">3</span><span class="o">-</span><span class="n">pve …</span></code></pre></div><p>3.X Kernel for OpenVZ is out and it is compiled with rbd module:</p>
<div class="highlight"><pre><span></span><code><span class="n">root</span><span class="err">@</span><span class="n">debian</span><span class="p">:</span><span class="o">~</span><span class="c1"># uname -a</span>
<span class="n">Linux</span><span class="w"> </span><span class="n">debian</span><span class="w"> </span><span class="mf">3.10</span><span class="o">.</span><span class="mi">0</span><span class="o">-</span><span class="mi">3</span><span class="o">-</span><span class="n">pve</span><span class="w"> </span><span class="c1">#1 SMP Thu Jun 12 13:50:49 CEST 2014 x86_64 GNU/Linux</span>
<span class="n">root</span><span class="err">@</span><span class="n">debian</span><span class="p">:</span><span class="o">~</span><span class="c1"># modinfo rbd</span>
<span class="n">filename</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">modules</span><span class="o">/</span><span class="mf">3.10</span><span class="o">.</span><span class="mi">0</span><span class="o">-</span><span class="mi">3</span><span class="o">-</span><span class="n">pve</span><span class="o">/</span><span class="n">kernel</span><span class="o">/</span><span class="n">drivers</span><span class="o">/</span><span class="n">block</span><span class="o">/</span><span class="n">rbd</span><span class="o">.</span><span class="n">ko</span>
<span class="n">license</span><span class="p">:</span><span class="w"> </span><span class="n">GPL</span>
<span class="n">author</span><span class="p">:</span><span class="w"> </span><span class="n">Jeff</span><span class="w"> </span><span class="n">Garzik</span><span class="w"> </span><span class="o"><</span><span class="n">jeff</span><span class="err">@</span><span class="n">garzik</span><span class="o">.</span><span class="n">org</span><span class="o">></span>
<span class="n">description</span><span class="p">:</span><span class="w"> </span><span class="n">rados</span><span class="w"> </span><span class="n">block</span><span class="w"> </span><span class="n">device</span>
<span class="n">author</span><span class="p">:</span><span class="w"> </span><span class="n">Yehuda</span><span class="w"> </span><span class="n">Sadeh</span><span class="w"> </span><span class="o"><</span><span class="n">yehuda</span><span class="err">@</span><span class="n">hq</span><span class="o">.</span><span class="n">newdream</span><span class="o">.</span><span class="n">net</span><span class="o">></span>
<span class="n">author</span><span class="p">:</span><span class="w"> </span><span class="n">Sage</span><span class="w"> </span><span class="n">Weil</span><span class="w"> </span><span class="o"><</span><span class="n">sage</span><span class="err">@</span><span class="n">newdream</span><span class="o">.</span><span class="n">net</span><span class="o">></span>
<span class="n">srcversion</span><span class="p">:</span><span class="w"> </span><span class="n">F459625E3E9943C5880D8BE</span>
<span class="n">depends</span><span class="p">:</span><span class="w"> </span><span class="n">libceph</span>
<span class="n">intree</span><span class="p">:</span><span class="w"> </span><span class="n">Y</span>
<span class="n">vermagic</span><span class="p">:</span><span class="w"> </span><span class="mf">3.10</span><span class="o">.</span><span class="mi">0</span><span class="o">-</span><span class="mi">3</span><span class="o">-</span><span class="n">pve</span><span class="w"> </span><span class="n">SMP</span><span class="w"> </span><span class="n">mod_unload</span><span class="w"> </span><span class="n">modversions</span>
</code></pre></div>
<p>There will be new things to test ...</p>
<p>By default, no CephFS module.</p>
<p>The announcement of the publication of the source code :
http://lists.openvz.org/pipermail/announce/2015-April/000579.html</p>Ceph Pool Migration2015-04-15T16:23:59+02:002015-04-15T16:23:59+02:00Laurent Barbetag:cephnotes.ksperis.com,2015-04-15:/blog/2015/04/15/ceph-pool-migration/<p>You have probably already be faced to migrate all objects from a pool to another, especially to change parameters that can not be modified on pool. For example, to migrate from a replicated pool to an EC pool, change EC profile, or to reduce the number of PGs...
There are …</p><p>You have probably already be faced to migrate all objects from a pool to another, especially to change parameters that can not be modified on pool. For example, to migrate from a replicated pool to an EC pool, change EC profile, or to reduce the number of PGs...
There are different methods, depending on the contents of the pool (RBD, objects), size...</p>
<h1>The simple way</h1>
<p>The simplest and safest method to copy all objects with the "rados cppool" command.
However, it need to have read only access to the pool during the copy.</p>
<p>For example for migrating to an EC pool :</p>
<div class="highlight"><pre><span></span><code><span class="nv">pool</span><span class="o">=</span>testpool
ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span><span class="nv">$pool</span>.new<span class="w"> </span><span class="m">4096</span><span class="w"> </span><span class="m">4096</span><span class="w"> </span>erasure<span class="w"> </span>default
rados<span class="w"> </span>cppool<span class="w"> </span><span class="nv">$pool</span><span class="w"> </span><span class="nv">$pool</span>.new
ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>rename<span class="w"> </span><span class="nv">$pool</span><span class="w"> </span><span class="nv">$pool</span>.old
ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>rename<span class="w"> </span><span class="nv">$pool</span>.new<span class="w"> </span><span class="nv">$pool</span>
</code></pre></div>
<p>But it does not work in all cases. For example with EC pools : "error copying pool testpool => newpool: (95) Operation not supported".</p>
<h1>Using Cache Tier</h1>
<p><strong>This must to be used with caution, make tests before using it on a cluster in production. It worked for my needs, but I can not say that it works in all cases.</strong></p>
<p>I find this method interesting method, because it allows transparent operation, reduce downtime and avoid to duplicate all data. The principle is simple: use the cache tier, but in reverse order.</p>
<p>At the begning, we have 2 pools : the current "testpool", and the new one "newpool"</p>
<p><img alt="Ceph Pool Migration" src="https://cephnotes.ksperis.com/images/pool_migration-1.png"></p>
<h2>Setup cache tier</h2>
<p>Configure the existing pool as cache pool :</p>
<div class="highlight"><pre><span></span><code>ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>add<span class="w"> </span>newpool<span class="w"> </span>testpool<span class="w"> </span>--force-nonempty
ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>cache-mode<span class="w"> </span>testpool<span class="w"> </span>forward
</code></pre></div>
<p>In <code>ceph osd dump</code> you should see something like that :
--> pool 58 'testpool' replicated size 3 .... tier_of 80 </p>
<p>Now, all new objects will be create on new pool :</p>
<p><img alt="Ceph Pool Migration" src="https://cephnotes.ksperis.com/images/pool_migration-2.png"></p>
<p>Now we can force to move all objects to new pool :</p>
<div class="highlight"><pre><span></span><code>rados<span class="w"> </span>-p<span class="w"> </span>testpool<span class="w"> </span>cache-flush-evict-all
</code></pre></div>
<p><img alt="Ceph Pool Migration" src="https://cephnotes.ksperis.com/images/pool_migration-3.png"></p>
<h2>Switch all clients to the new pool</h2>
<p>(You can also do this step earlier. For example, just after the cache pool creation.)
Until all the data has not been flushed to the new pool you need to specify an overlay to search objects on old pool :</p>
<div class="highlight"><pre><span></span><code>ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>set-overlay<span class="w"> </span>newpool<span class="w"> </span>testpool
</code></pre></div>
<p>In <code>ceph osd dump</code> you should see something like that :
--> pool 80 'newpool' replicated size 3 .... tiers 58 read_tier 58 write_tier 58</p>
<p>With overlay, all operation will be forwarded to the old testpool :</p>
<p><img alt="Ceph Pool Migration" src="https://cephnotes.ksperis.com/images/pool_migration-4.png"></p>
<p>Now you can switch all the clients to access objects on the new pool.</p>
<h2>Finish</h2>
<p>When all data is migrate, you can remove overlay and old "cache" pool :</p>
<div class="highlight"><pre><span></span><code>ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>remove-overlay<span class="w"> </span>newpool
ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>remove<span class="w"> </span>newpool<span class="w"> </span>testpool
</code></pre></div>
<p><img alt="Ceph Pool Migration" src="https://cephnotes.ksperis.com/images/pool_migration-5.png"></p>
<!-- more -->
<h2>In-use object</h2>
<p>During eviction you can find some error :
....
rb.0.59189e.2ae8944a.000000000001 <br>
rb.0.59189e.2ae8944a.000000000023 <br>
rb.0.59189e.2ae8944a.000000000006 <br>
testrbd.rbd
failed to evict testrbd.rbd: (16) Device or resource busy
rb.0.59189e.2ae8944a.000000000000 <br>
rb.0.59189e.2ae8944a.000000000026 <br>
...</p>
<p>List watcher on object can help :</p>
<div class="highlight"><pre><span></span><code>rados<span class="w"> </span>-p<span class="w"> </span>testpool<span class="w"> </span>listwatchers<span class="w"> </span>testrbd.rbd
<span class="nv">watcher</span><span class="o">=</span><span class="m">10</span>.20.6.39:0/3318181122<span class="w"> </span>client.5520194<span class="w"> </span><span class="nv">cookie</span><span class="o">=</span><span class="m">1</span>
</code></pre></div>
<h1>Using Rados Export/Import</h1>
<p>For this, you need to use a temporary local directory.</p>
<div class="highlight"><pre><span></span><code>rados<span class="w"> </span><span class="nb">export</span><span class="w"> </span>--create<span class="w"> </span>testpool<span class="w"> </span>tmp_dir
<span class="o">[</span>exported<span class="o">]</span><span class="w"> </span>rb.0.4975.2ae8944a.000000002391
<span class="o">[</span>exported<span class="o">]</span><span class="w"> </span>rb.0.4975.2ae8944a.000000004abc
<span class="o">[</span>exported<span class="o">]</span><span class="w"> </span>rb.0.4975.2ae8944a.0000000018ce
...
rados<span class="w"> </span>import<span class="w"> </span>tmp_dir<span class="w"> </span>newpool
<span class="c1"># Stop All IO</span>
<span class="c1"># And redo a sync of modified objects</span>
rados<span class="w"> </span><span class="nb">export</span><span class="w"> </span>--workers<span class="w"> </span><span class="m">5</span><span class="w"> </span>testpool<span class="w"> </span>tmp_dir
rados<span class="w"> </span>import<span class="w"> </span>--workers<span class="w"> </span><span class="m">5</span><span class="w"> </span>tmp_dir<span class="w"> </span>newpool
</code></pre></div>RadosGW: Simple replication example2015-03-13T16:20:29+01:002015-03-13T16:20:29+01:00Laurent Barbetag:cephnotes.ksperis.com,2015-03-13:/blog/2015/03/13/radosgw-simple-replication-example/<p>This is a simple example of federated gateways config to make an asynchonous replication between two Ceph clusters.</p>
<p><img alt="RadosGW replication example" src="https://cephnotes.ksperis.com/images/simple_example_radosgw-agent.png"></p>
<!-- more -->
<p>(Update Nov. 2016 : Since Jewel version, radosgw-agent is no more needed and active-active replication between zone is now supported. See here : http://docs.ceph.com/docs/jewel/radosgw/multisite/)</p>
<p>( The configuration below …</p><p>This is a simple example of federated gateways config to make an asynchonous replication between two Ceph clusters.</p>
<p><img alt="RadosGW replication example" src="https://cephnotes.ksperis.com/images/simple_example_radosgw-agent.png"></p>
<!-- more -->
<p>(Update Nov. 2016 : Since Jewel version, radosgw-agent is no more needed and active-active replication between zone is now supported. See here : http://docs.ceph.com/docs/jewel/radosgw/multisite/)</p>
<p>( The configuration below is using radosgw-agent and is based on Ceph documentation :
http://docs.ceph.com/docs/hammer/radosgw/federated-config/ )</p>
<p>Here I use only one region ("default") and two zones ("main" and "fallback"), one for each cluster.</p>
<p>Note that in this example, I use 3 placement targets (default, hot, cold) that correspond respectively on pool .main.rgw.buckets, .main.rgw.hot.buckets, .main.rgw.cold.buckets.
Be carefull to replace the tags {MAIN_USER_ACCESS}, {MAIN_USER_SECRET}, {FALLBACK_USER_ACESS}, {FALLBACK_USER_SECRET} by corresponding values.</p>
<p>First I created region and zones files, that will be require on the 2 clusters :</p>
<p>The region file "region.conf.json" :</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"default"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"api_name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"default"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"is_master"</span><span class="p">:</span><span class="w"> </span><span class="s2">"true"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"endpoints"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"http:\/\/s3.mydomain.com:80\/"</span><span class="p">],</span>
<span class="w"> </span><span class="nt">"master_zone"</span><span class="p">:</span><span class="w"> </span><span class="s2">"main"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"zones"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"main"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"endpoints"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"http:\/\/s3.mydomain.com:80\/"</span><span class="p">],</span>
<span class="w"> </span><span class="nt">"log_meta"</span><span class="p">:</span><span class="w"> </span><span class="s2">"true"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"log_data"</span><span class="p">:</span><span class="w"> </span><span class="s2">"true"</span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"fallback"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"endpoints"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"http:\/\/s3-fallback.mydomain.com:80\/"</span><span class="p">],</span>
<span class="w"> </span><span class="nt">"log_meta"</span><span class="p">:</span><span class="w"> </span><span class="s2">"true"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"log_data"</span><span class="p">:</span><span class="w"> </span><span class="s2">"true"</span><span class="p">}],</span>
<span class="w"> </span><span class="nt">"placement_targets"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"default-placement"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"tags"</span><span class="p">:</span><span class="w"> </span><span class="p">[]},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"cold-placement"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"tags"</span><span class="p">:</span><span class="w"> </span><span class="p">[]},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hot-placement"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"tags"</span><span class="p">:</span><span class="w"> </span><span class="p">[]}],</span>
<span class="w"> </span><span class="nt">"default_placement"</span><span class="p">:</span><span class="w"> </span><span class="s2">"default-placement"</span><span class="p">}</span>
</code></pre></div>
<p>a zone file "zone-main.conf.json" :</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"> </span><span class="nt">"domain_root"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.domain.rgw"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"control_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.rgw.control"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"gc_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.rgw.gc"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"log_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.log"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"intent_log_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.intent-log"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"usage_log_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.usage"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"user_keys_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.users"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"user_email_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.users.email"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"user_swift_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.users.swift"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"user_uid_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.users.uid"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"system_key"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"access_key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"{MAIN_USER_ACCESS}"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"secret_key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"{MAIN_USER_SECRET}"</span><span class="p">},</span>
<span class="w"> </span><span class="nt">"placement_pools"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"default-placement"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"val"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"index_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.rgw.buckets.index"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.rgw.buckets"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_extra_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.rgw.buckets.extra"</span><span class="p">}},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"cold-placement"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"val"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"index_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.rgw.buckets.index"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.rgw.cold.buckets"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_extra_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.rgw.buckets.extra"</span><span class="p">}},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hot-placement"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"val"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"index_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.rgw.buckets.index"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.rgw.hot.buckets"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_extra_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".main.rgw.buckets.extra"</span><span class="p">}}]}</span>
</code></pre></div>
<p>And a zone file "zone-fallback.conf.json" :</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"> </span><span class="nt">"domain_root"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.domain.rgw"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"control_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.rgw.control"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"gc_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.rgw.gc"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"log_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.log"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"intent_log_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.intent-log"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"usage_log_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.usage"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"user_keys_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.users"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"user_email_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.users.email"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"user_swift_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.users.swift"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"user_uid_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.users.uid"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"system_key"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"access_key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"{FALLBACK_USER_ACESS}"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"secret_key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"{FALLBACK_USER_SECRET}"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">"placement_pools"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"default-placement"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"val"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"index_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.rgw.buckets.index"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.rgw.buckets"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_extra_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.rgw.buckets.extra"</span><span class="p">}},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"cold-placement"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"val"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"index_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.rgw.buckets.index"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.rgw.cold.buckets"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_extra_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.rgw.buckets.extra"</span><span class="p">}},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hot-placement"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"val"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"index_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.rgw.buckets.index"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.rgw.hot.buckets"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_extra_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".fallback.rgw.buckets.extra"</span><span class="p">}}]}</span>
</code></pre></div>
<h2>On first cluster (MAIN)</h2>
<p>I created the pools :</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.rgw.root<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.rgw.root<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.domain.rgw<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.rgw.control<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.rgw.gc<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.rgw.buckets<span class="w"> </span><span class="m">512</span><span class="w"> </span><span class="m">512</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.rgw.hot.buckets<span class="w"> </span><span class="m">512</span><span class="w"> </span><span class="m">512</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.rgw.cold.buckets<span class="w"> </span><span class="m">512</span><span class="w"> </span><span class="m">512</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.rgw.buckets.index<span class="w"> </span><span class="m">32</span><span class="w"> </span><span class="m">32</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.rgw.buckets.extra<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.log<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.intent-log<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.usage<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.users<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.users.email<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.users.swift<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.main.users.uid<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
</code></pre></div>
<p>I configured region, zone, and add system users :</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>radosgw-admin<span class="w"> </span>region<span class="w"> </span><span class="nb">set</span><span class="w"> </span>--name<span class="w"> </span>client.radosgw.main<span class="w"> </span><<span class="w"> </span>region.conf.json
<span class="w"> </span>radosgw-admin<span class="w"> </span>zone<span class="w"> </span><span class="nb">set</span><span class="w"> </span>--rgw-zone<span class="o">=</span>main<span class="w"> </span>--name<span class="w"> </span>client.radosgw.main<span class="w"> </span><<span class="w"> </span>zone-main.conf.json
<span class="w"> </span>radosgw-admin<span class="w"> </span>zone<span class="w"> </span><span class="nb">set</span><span class="w"> </span>--rgw-zone<span class="o">=</span>fallback<span class="w"> </span>--name<span class="w"> </span>client.radosgw.main<span class="w"> </span><<span class="w"> </span>zone-fallback.conf.json
<span class="w"> </span>radosgw-admin<span class="w"> </span>regionmap<span class="w"> </span>update<span class="w"> </span>--name<span class="w"> </span>client.radosgw.main
<span class="w"> </span>radosgw-admin<span class="w"> </span>user<span class="w"> </span>create<span class="w"> </span>--uid<span class="o">=</span><span class="s2">"main"</span><span class="w"> </span>--display-name<span class="o">=</span><span class="s2">"Zone main"</span><span class="w"> </span>--name<span class="w"> </span>client.radosgw.main<span class="w"> </span>--system<span class="w"> </span>--access-key<span class="o">={</span>MAIN_USER_ACCESS<span class="o">}</span><span class="w"> </span>--secret<span class="o">={</span>MAIN_USER_SECRET<span class="o">}</span>
<span class="w"> </span>radosgw-admin<span class="w"> </span>user<span class="w"> </span>create<span class="w"> </span>--uid<span class="o">=</span><span class="s2">"fallback"</span><span class="w"> </span>--display-name<span class="o">=</span><span class="s2">"Zone fallback"</span><span class="w"> </span>--name<span class="w"> </span>client.radosgw.main<span class="w"> </span>--system<span class="w"> </span>--access-key<span class="o">={</span>FALLBACK_USER_ACESS<span class="o">}</span><span class="w"> </span>--secret<span class="o">={</span>FALLBACK_USER_SECRET<span class="o">}</span>
</code></pre></div>
<p>Setup RadosGW Config in ceph.conf on cluster MAIN :</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="p">[</span><span class="n">client</span><span class="o">.</span><span class="n">radosgw</span><span class="o">.</span><span class="n">main</span><span class="p">]</span>
<span class="w"> </span><span class="n">host</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ceph</span><span class="o">-</span><span class="n">main</span><span class="o">-</span><span class="n">radosgw</span><span class="o">-</span><span class="mi">01</span>
<span class="w"> </span><span class="n">rgw</span><span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">default</span>
<span class="w"> </span><span class="n">rgw</span><span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="n">pool</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">.</span><span class="n">rgw</span><span class="o">.</span><span class="n">root</span>
<span class="w"> </span><span class="n">rgw</span><span class="w"> </span><span class="n">zone</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">main</span>
<span class="w"> </span><span class="n">rgw</span><span class="w"> </span><span class="n">zone</span><span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="n">pool</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">.</span><span class="n">main</span><span class="o">.</span><span class="n">rgw</span><span class="o">.</span><span class="n">root</span>
<span class="w"> </span><span class="n">rgw</span><span class="w"> </span><span class="n">frontends</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"civetweb port=80"</span>
<span class="w"> </span><span class="n">rgw</span><span class="w"> </span><span class="n">dns</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">s3</span><span class="o">.</span><span class="n">mydomain</span><span class="o">.</span><span class="n">com</span>
<span class="w"> </span><span class="n">keyring</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">ceph</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">radosgw</span><span class="o">.</span><span class="n">keyring</span>
<span class="w"> </span><span class="n">rgw_socket_path</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">run</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">radosgw</span><span class="o">.</span><span class="n">sock</span>
</code></pre></div>
<p>I needed to create keyring for [client.radosgw.main] in /etc/ceph/ceph.client.radosgw.keyring, see documentation.</p>
<p>Then, start/restart radosgw for cluster MAIN.</p>
<h2>On the other Ceph cluster (FALLBACK)</h2>
<p>I created the pools :</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.rgw.root<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.rgw.root<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.domain.rgw<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.rgw.control<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.rgw.gc<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.rgw.buckets<span class="w"> </span><span class="m">512</span><span class="w"> </span><span class="m">512</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.rgw.hot.buckets<span class="w"> </span><span class="m">512</span><span class="w"> </span><span class="m">512</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.rgw.cold.buckets<span class="w"> </span><span class="m">512</span><span class="w"> </span><span class="m">512</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.rgw.buckets.index<span class="w"> </span><span class="m">32</span><span class="w"> </span><span class="m">32</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.rgw.buckets.extra<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.log<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.intent-log<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.usage<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.users<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.users.email<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.users.swift<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.fallback.users.uid<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">16</span>
</code></pre></div>
<p>I configured region, zone, and add system users :</p>
<div class="highlight"><pre><span></span><code><span class="n">radosgw</span><span class="o">-</span><span class="n">admin</span><span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="n">set</span><span class="w"> </span><span class="o">--</span><span class="n">name</span><span class="w"> </span><span class="n">client</span><span class="p">.</span><span class="n">radosgw</span><span class="p">.</span><span class="n">fallback</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">region</span><span class="p">.</span><span class="n">conf</span><span class="p">.</span><span class="n">json</span>
<span class="n">radosgw</span><span class="o">-</span><span class="n">admin</span><span class="w"> </span><span class="n">zone</span><span class="w"> </span><span class="n">set</span><span class="w"> </span><span class="o">--</span><span class="n">rgw</span><span class="o">-</span><span class="n">zone</span><span class="o">=</span><span class="n">fallback</span><span class="w"> </span><span class="o">--</span><span class="n">name</span><span class="w"> </span><span class="n">client</span><span class="p">.</span><span class="n">radosgw</span><span class="p">.</span><span class="n">fallback</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">zone</span><span class="o">-</span><span class="n">fallback</span><span class="p">.</span><span class="n">conf</span><span class="p">.</span><span class="n">json</span>
<span class="n">radosgw</span><span class="o">-</span><span class="n">admin</span><span class="w"> </span><span class="n">zone</span><span class="w"> </span><span class="n">set</span><span class="w"> </span><span class="o">--</span><span class="n">rgw</span><span class="o">-</span><span class="n">zone</span><span class="o">=</span><span class="n">main</span><span class="w"> </span><span class="o">--</span><span class="n">name</span><span class="w"> </span><span class="n">client</span><span class="p">.</span><span class="n">radosgw</span><span class="p">.</span><span class="n">fallback</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">zone</span><span class="o">-</span><span class="n">main</span><span class="p">.</span><span class="n">conf</span><span class="p">.</span><span class="n">json</span>
<span class="n">radosgw</span><span class="o">-</span><span class="n">admin</span><span class="w"> </span><span class="n">regionmap</span><span class="w"> </span><span class="n">update</span><span class="w"> </span><span class="o">--</span><span class="n">name</span><span class="w"> </span><span class="n">client</span><span class="p">.</span><span class="n">radosgw</span><span class="p">.</span><span class="n">fallback</span>
<span class="n">radosgw</span><span class="o">-</span><span class="n">admin</span><span class="w"> </span><span class="n">user</span><span class="w"> </span><span class="n">create</span><span class="w"> </span><span class="o">--</span><span class="n">uid</span><span class="o">=</span><span class="s">"fallback"</span><span class="w"> </span><span class="o">--</span><span class="n">display</span><span class="o">-</span><span class="n">name</span><span class="o">=</span><span class="s">"Zone fallback"</span><span class="w"> </span><span class="o">--</span><span class="n">name</span><span class="w"> </span><span class="n">client</span><span class="p">.</span><span class="n">radosgw</span><span class="p">.</span><span class="n">fallback</span><span class="w"> </span><span class="o">--</span><span class="n">system</span><span class="w"> </span><span class="o">--</span><span class="n">access</span><span class="o">-</span><span class="n">key</span><span class="o">=</span><span class="p">{</span><span class="n">FALLBACK_USER_ACESS</span><span class="p">}</span><span class="w"> </span><span class="o">--</span><span class="n">secret</span><span class="o">=</span><span class="p">{</span><span class="n">FALLBACK_USER_SECRET</span><span class="p">}</span>
<span class="n">radosgw</span><span class="o">-</span><span class="n">admin</span><span class="w"> </span><span class="n">user</span><span class="w"> </span><span class="n">create</span><span class="w"> </span><span class="o">--</span><span class="n">uid</span><span class="o">=</span><span class="s">"main"</span><span class="w"> </span><span class="o">--</span><span class="n">display</span><span class="o">-</span><span class="n">name</span><span class="o">=</span><span class="s">"Zone main"</span><span class="w"> </span><span class="o">--</span><span class="n">name</span><span class="w"> </span><span class="n">client</span><span class="p">.</span><span class="n">radosgw</span><span class="p">.</span><span class="n">fallback</span><span class="w"> </span><span class="o">--</span><span class="n">system</span><span class="w"> </span><span class="o">--</span><span class="n">access</span><span class="o">-</span><span class="n">key</span><span class="o">=</span><span class="p">{</span><span class="n">MAIN_USER_ACCESS</span><span class="p">}</span><span class="w"> </span><span class="o">--</span><span class="n">secret</span><span class="o">=</span><span class="p">{</span><span class="n">MAIN_USER_SECRET</span><span class="p">}</span>
</code></pre></div>
<p>Setup RadosGW Config in ceph.conf on cluster FALLBACK :</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="p">[</span><span class="n">client</span><span class="o">.</span><span class="n">radosgw</span><span class="o">.</span><span class="n">fallback</span><span class="p">]</span>
<span class="w"> </span><span class="n">host</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ceph</span><span class="o">-</span><span class="n">fallback</span><span class="o">-</span><span class="n">radosgw</span><span class="o">-</span><span class="mi">01</span>
<span class="w"> </span><span class="n">rgw</span><span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">default</span>
<span class="w"> </span><span class="n">rgw</span><span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="n">pool</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">.</span><span class="n">rgw</span><span class="o">.</span><span class="n">root</span>
<span class="w"> </span><span class="n">rgw</span><span class="w"> </span><span class="n">zone</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">fallback</span>
<span class="w"> </span><span class="n">rgw</span><span class="w"> </span><span class="n">zone</span><span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="n">pool</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">.</span><span class="n">fallback</span><span class="o">.</span><span class="n">rgw</span><span class="o">.</span><span class="n">root</span>
<span class="w"> </span><span class="n">rgw</span><span class="w"> </span><span class="n">frontends</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"civetweb port=80"</span>
<span class="w"> </span><span class="n">rgw</span><span class="w"> </span><span class="n">dns</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">s3</span><span class="o">-</span><span class="n">fallback</span><span class="o">.</span><span class="n">mydomain</span><span class="o">.</span><span class="n">com</span>
<span class="w"> </span><span class="n">keyring</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">ceph</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">radosgw</span><span class="o">.</span><span class="n">keyring</span>
<span class="w"> </span><span class="n">rgw_socket_path</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">run</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">radosgw</span><span class="o">.</span><span class="n">sock</span>
</code></pre></div>
<p>Also, I needed to create keyring for [client.radosgw.fallback] in /etc/ceph/ceph.client.radosgw.keyring and start radosgw for cluster FALLBACK.</p>
<h2>Finally setup the RadosGW Agent</h2>
<p>/etc/ceph/radosgw-agent/default.conf :</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="n">src_zone</span><span class="p">:</span><span class="w"> </span><span class="n">main</span>
<span class="w"> </span><span class="n">source</span><span class="p">:</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">s3</span><span class="o">.</span><span class="n">mydomain</span><span class="o">.</span><span class="n">com</span><span class="p">:</span><span class="mi">80</span>
<span class="w"> </span><span class="n">src_access_key</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="n">MAIN_USER_ACCESS</span><span class="p">}</span>
<span class="w"> </span><span class="n">src_secret_key</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="n">MAIN_USER_SECRET</span><span class="p">}</span>
<span class="w"> </span><span class="n">dest_zone</span><span class="p">:</span><span class="w"> </span><span class="n">fallback</span>
<span class="w"> </span><span class="n">destination</span><span class="p">:</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">s3</span><span class="o">-</span><span class="n">fallback</span><span class="o">.</span><span class="n">mydomain</span><span class="o">.</span><span class="n">com</span><span class="p">:</span><span class="mi">80</span>
<span class="w"> </span><span class="n">dest_access_key</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="n">FALLBACK_USER_ACESS</span><span class="p">}</span>
<span class="w"> </span><span class="n">dest_secret_key</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="n">FALLBACK_USER_SECRET</span><span class="p">}</span>
<span class="w"> </span><span class="n">log_file</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">radosgw</span><span class="o">/</span><span class="n">radosgw</span><span class="o">-</span><span class="n">sync</span><span class="o">.</span><span class="n">log</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="w"> </span>/etc/init.d/radosgw-agent<span class="w"> </span>start
</code></pre></div>
<p>After that, he still has a little suspense ...
Then I try to create a bucket with data on s3.mydomain.com and verify that, it's well synchronized.</p>
<p>for debug, you can enable logs on the RadosGW on each side, and start radosgw-agent with radosgw-agent -v -c /etc/ceph/radosgw-agent/default.conf</p>
<p>These steps work for me. The establishment is sometimes not obvious. Whenever I setup a sync it rarely works the first time, but it always ends up running.</p>get the number of placement groups per osd2015-02-23T17:53:51+01:002015-02-23T17:53:51+01:00Laurent Barbetag:cephnotes.ksperis.com,2015-02-23:/blog/2015/02/23/get-the-number-of-placement-groups-per-osd/<p>Get the PG distribution per osd in command line :</p>
<div class="highlight"><pre><span></span><code><span class="n">pool</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">SUM</span><span class="w"> </span>
<span class="o">------------------------------------------------</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">10</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">84</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">102</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">11</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">76</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">95</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">12</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">56</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">67</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">20</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">107</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">122</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">13</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">73</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">82 …</span></code></pre></div><p>Get the PG distribution per osd in command line :</p>
<div class="highlight"><pre><span></span><code><span class="n">pool</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">SUM</span><span class="w"> </span>
<span class="o">------------------------------------------------</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">10</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">84</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">102</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">11</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">76</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">95</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">12</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">56</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">67</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">20</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">107</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">122</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">13</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">73</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">82</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">21</span><span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="mi">110</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">139</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">14</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">85</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">94</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">15</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">87</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">105</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">22</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">87</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">104</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">23</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="mi">87</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">117</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">16</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="mi">102</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">123</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">17</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">99</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">114</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">18</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">103</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">115</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">19</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="mi">112</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">133</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">0</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">72</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">87</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">1</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">83</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">99</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">74</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">83</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">3</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">61</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">76</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">4</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">76</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">86</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">78</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">93</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">6</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">78</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">85</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">7</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">88</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">97</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">8</span><span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="mi">91</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">118</span>
<span class="n">osd</span><span class="o">.</span><span class="mi">9</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">79</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">96</span>
<span class="o">------------------------------------------------</span>
<span class="n">SUM</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="mi">128</span><span class="w"> </span><span class="mi">128</span><span class="w"> </span><span class="mi">128</span><span class="w"> </span><span class="mi">2048</span><span class="w"> </span><span class="o">|</span>
</code></pre></div>
<!--more-->
<p>The command :</p>
<div class="highlight"><pre><span></span><code><span class="n">ceph</span><span class="w"> </span><span class="n">pg</span><span class="w"> </span><span class="k">dump</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">awk</span><span class="w"> </span><span class="s1">'</span>
<span class="s1">BEGIN { IGNORECASE = 1 }</span>
<span class="s1"> /^PG_STAT/ { col=1; while($col!="UP") {col++}; col++ }</span>
<span class="s1"> /^[0-9a-f]+\.[0-9a-f]+/ { match($0,/^[0-9a-f]+/); pool=substr($0, RSTART, RLENGTH); poollist[pool]=0;</span>
<span class="s1"> up=$col; i=0; RSTART=0; RLENGTH=0; delete osds; while(match(up,/[0-9]+/)>0) { osds[++i]=substr(up,RSTART,RLENGTH); up = substr(up, RSTART+RLENGTH) }</span>
<span class="s1"> for(i in osds) {array[osds[i],pool]++; osdlist[osds[i]];}</span>
<span class="s1">}</span>
<span class="s1">END {</span>
<span class="s1"> printf("\n");</span>
<span class="s1"> printf("pool :\t"); for (i in poollist) printf("%s\t",i); printf("| SUM \n");</span>
<span class="s1"> for (i in poollist) printf("--------"); printf("----------------\n");</span>
<span class="s1"> for (i in osdlist) { printf("osd.%i\t", i); sum=0;</span>
<span class="s1"> for (j in poollist) { printf("%i\t", array[i,j]); sum+=array[i,j]; sumpool[j]+=array[i,j] }; printf("| %i\n",sum) }</span>
<span class="s1"> for (i in poollist) printf("--------"); printf("----------------\n");</span>
<span class="s1"> printf("SUM :\t"); for (i in poollist) printf("%s\t",sumpool[i]); printf("|\n");</span>
<span class="s1">}'</span>
</code></pre></div>
<p>Copy-paste should work directly.
The sum at bottom of the table must match to (pg_num x size).</p>
<p>You can find pg_num recommandations here :
http://ceph.com/docs/master/rados/operations/placement-groups/</p>
<p>Also, a pg_num Calculator is avaible here :
http://ceph.com/pgcalc</p>
<p>To view the number of pg per osd :
http://cephnotes.ksperis.com/blog/2015/02/23/get-the-number-of-placement-groups-per-osd/</p>CRUSHMAP : Example of a hierarchical cluster map2015-02-02T22:10:43+01:002015-02-02T22:10:43+01:00Laurent Barbetag:cephnotes.ksperis.com,2015-02-02:/blog/2015/02/02/crushmap-example-of-a-hierarchical-cluster-map/<p>It is not always easy to know how to organize your data in the Crushmap, especially when trying to distribute the data geographically while separating different types of discs, eg SATA, SAS and SSD.
Let's see what we can imagine as Crushmap hierarchy.</p>
<p>Take a simple example of a distribution …</p><p>It is not always easy to know how to organize your data in the Crushmap, especially when trying to distribute the data geographically while separating different types of discs, eg SATA, SAS and SSD.
Let's see what we can imagine as Crushmap hierarchy.</p>
<p>Take a simple example of a distribution on two datacenters.
<img alt="Crushmap Tree" src="https://cephnotes.ksperis.com/images/crushmap-tree-1.png">
(Model 1.1)</p>
<!--more-->
<p>With the introduction of cache pools we can easily imagine adding ssd drives to our cluster. Take the example of ssd added on new hosts. We then left with two types of disks to manage. In a hierarchy that only describing the physical division of the cluster, we would end up with this type of hierarchy:
<img alt="Crushmap Tree" src="https://cephnotes.ksperis.com/images/crushmap-tree-2.png">
(Model 1.2)</p>
<p>However, we soon realized that this configuration does not allow to separate the types of discs for use in a specific pool.</p>
<p>To separate these discs and organize Crushmap, the simplest method is still to duplicate the tree from the root.
Thus we get two points entered "default" (that could be rename "hdd") and "ssd".
<img alt="Crushmap Tree" src="https://cephnotes.ksperis.com/images/crushmap-tree-3.1.png">
An other example with hdd and ssd mixed on same hardware (you need to split each host) :
<img alt="Crushmap Tree" src="https://cephnotes.ksperis.com/images/crushmap-tree-3.2.png">
(Model 1.3)</p>
<p>The problem is that we have segmented all the cluster by drive type. It was therefore no more entry point into our tree to select any disk in "dc-1" or in "dc-2".
For example, we can no longer define a rule to store data on a specific data center regardless of the type of disc.</p>
<p>What we can do is add other entry points to the root level. For example, to permit access all drives :
<img alt="Crushmap Tree" src="https://cephnotes.ksperis.com/images/crushmap-tree-4.png">
(Model 1.4)</p>
<p>If we wants to keep a certain logic in the tree, it is also possible to add more levels, some more "logical" for exemple for disks types, and other that represent physical distribution.
They can be placed wherever we want and named as we wish. For example, here I added the level "pool" that one could also have called "type" or otherwise.
<img alt="Crushmap Tree" src="https://cephnotes.ksperis.com/images/crushmap-tree-5.1.png">
<img alt="Crushmap Tree" src="https://cephnotes.ksperis.com/images/crushmap-tree-5.2.png">
(Model 1.5)</p>
<p>Ok, it works, but it is difficult to read. Moreover, it becomes unreadable when the SSD and HDD are mixed on the same hosts as it involves duplicating each host.
Also, there is no more physical data placement logic. We can try to insert a level between HOST and OSD:
<img alt="Crushmap Tree" src="https://cephnotes.ksperis.com/images/crushmap-tree-6.png">
(Model 1.6)</p>
<p>Ok, it may be advantageous in the case of a small cluster, or in the case where there is no need for other levels.
Let's try something else, we can also try using another organization, such as separating the osd in different levels and use that in the specific rules.
For examble, have <code>step chooseleaf firstn 5 type host-sata</code> to select sata drive, and <code>step chooseleaf firstn 5 type host-ssd</code> to select ssd drive.
<img alt="Crushmap Tree" src="https://cephnotes.ksperis.com/images/crushmap-tree-7.png">
(Model 1.7)</p>
<p>But this do NOT WORK. Indeed, the algorithm will try to take an OSD in each branch of the tree. If no OSD is found, it will try again to retrace. But this operation is quite random, and you can easily end up with insufficient replications.</p>
<p>Make the test of this with this crushmap :</p>
<ul>
<li>
<p>1 rule for select one ssd on each DC</p>
</li>
<li>
<p>1 rule for select one sata on each DC</p>
</li>
<li>
<p>1 rule for select one sata on 2 diffrents hosts</p>
</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="c1"># begin crush map</span>
<span class="w"> </span><span class="n">tunable</span><span class="w"> </span><span class="n">choose_local_tries</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">tunable</span><span class="w"> </span><span class="n">choose_local_fallback_tries</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">tunable</span><span class="w"> </span><span class="n">choose_total_tries</span><span class="w"> </span><span class="mi">50</span>
<span class="w"> </span><span class="n">tunable</span><span class="w"> </span><span class="n">chooseleaf_descend_once</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">tunable</span><span class="w"> </span><span class="n">chooseleaf_vary_r</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="c1"># devices</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">0</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">1</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">2</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">3</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">4</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">5</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">6</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">7</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">8</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">9</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">10</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">11</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">11</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">12</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">12</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">13</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">13</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">14</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">14</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">15</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">15</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">16</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">16</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">17</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">17</span>
<span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">18</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">18</span>
<span class="w"> </span><span class="c1"># types</span>
<span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">osd</span>
<span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="n">ssd</span>
<span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="n">sata</span>
<span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">datacenter</span>
<span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">root</span>
<span class="w"> </span><span class="c1"># buckets</span>
<span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="n">sata</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">01</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">alg</span><span class="w"> </span><span class="n">straw</span>
<span class="w"> </span><span class="nb">hash</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">1</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">2</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">3</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="n">sata</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">02</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">alg</span><span class="w"> </span><span class="n">straw</span>
<span class="w"> </span><span class="nb">hash</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">4</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">5</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">6</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="n">sata</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">03</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">alg</span><span class="w"> </span><span class="n">straw</span>
<span class="w"> </span><span class="nb">hash</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">7</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">8</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">9</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="n">sata</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">alg</span><span class="w"> </span><span class="n">straw</span>
<span class="w"> </span><span class="nb">hash</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">10</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">11</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">12</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="n">ssd</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">05</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">alg</span><span class="w"> </span><span class="n">straw</span>
<span class="w"> </span><span class="nb">hash</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">13</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">14</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">15</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="n">ssd</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">06</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">alg</span><span class="w"> </span><span class="n">straw</span>
<span class="w"> </span><span class="nb">hash</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">16</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">17</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">18</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">datacenter</span><span class="w"> </span><span class="n">dc1</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">alg</span><span class="w"> </span><span class="n">straw</span>
<span class="w"> </span><span class="nb">hash</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">01</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">02</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">05</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">datacenter</span><span class="w"> </span><span class="n">dc2</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">alg</span><span class="w"> </span><span class="n">straw</span>
<span class="w"> </span><span class="nb">hash</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">03</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="mi">06</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="n">default</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">alg</span><span class="w"> </span><span class="n">straw</span>
<span class="w"> </span><span class="nb">hash</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">dc1</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">dc2</span><span class="w"> </span><span class="n">weight</span><span class="w"> </span><span class="mf">1.000</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="c1"># rules</span>
<span class="w"> </span><span class="n">rule</span><span class="w"> </span><span class="n">sata</span><span class="o">-</span><span class="n">rep_2dc</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">ruleset</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="n">replicated</span>
<span class="w"> </span><span class="n">min_size</span><span class="w"> </span><span class="mi">2</span>
<span class="w"> </span><span class="n">max_size</span><span class="w"> </span><span class="mi">2</span>
<span class="w"> </span><span class="n">step</span><span class="w"> </span><span class="n">take</span><span class="w"> </span><span class="n">default</span>
<span class="w"> </span><span class="n">step</span><span class="w"> </span><span class="n">choose</span><span class="w"> </span><span class="n">firstn</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="n">datacenter</span>
<span class="w"> </span><span class="n">step</span><span class="w"> </span><span class="n">chooseleaf</span><span class="w"> </span><span class="n">firstn</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="n">sata</span>
<span class="w"> </span><span class="n">step</span><span class="w"> </span><span class="n">emit</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">rule</span><span class="w"> </span><span class="n">ssd</span><span class="o">-</span><span class="n">rep_2dc</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">ruleset</span><span class="w"> </span><span class="mi">1</span>
<span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="n">replicated</span>
<span class="w"> </span><span class="n">min_size</span><span class="w"> </span><span class="mi">2</span>
<span class="w"> </span><span class="n">max_size</span><span class="w"> </span><span class="mi">2</span>
<span class="w"> </span><span class="n">step</span><span class="w"> </span><span class="n">take</span><span class="w"> </span><span class="n">default</span>
<span class="w"> </span><span class="n">step</span><span class="w"> </span><span class="n">choose</span><span class="w"> </span><span class="n">firstn</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="n">datacenter</span>
<span class="w"> </span><span class="n">step</span><span class="w"> </span><span class="n">chooseleaf</span><span class="w"> </span><span class="n">firstn</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="n">ssd</span>
<span class="w"> </span><span class="n">step</span><span class="w"> </span><span class="n">emit</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">rule</span><span class="w"> </span><span class="n">sata</span><span class="o">-</span><span class="n">all</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">ruleset</span><span class="w"> </span><span class="mi">2</span>
<span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="n">replicated</span>
<span class="w"> </span><span class="n">min_size</span><span class="w"> </span><span class="mi">2</span>
<span class="w"> </span><span class="n">max_size</span><span class="w"> </span><span class="mi">2</span>
<span class="w"> </span><span class="n">step</span><span class="w"> </span><span class="n">take</span><span class="w"> </span><span class="n">default</span>
<span class="w"> </span><span class="n">step</span><span class="w"> </span><span class="n">chooseleaf</span><span class="w"> </span><span class="n">firstn</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="n">host</span><span class="o">-</span><span class="n">sata</span>
<span class="w"> </span><span class="n">step</span><span class="w"> </span><span class="n">emit</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="c1"># end crush map</span>
</code></pre></div>
<p>To test the placement, we can use those commands :</p>
<div class="highlight"><pre><span></span><code><span class="n">crushtool</span><span class="w"> </span><span class="o">-</span><span class="n">c</span><span class="w"> </span><span class="n">crushmap</span><span class="o">.</span><span class="n">txt</span><span class="w"> </span><span class="o">-</span><span class="n">o</span><span class="w"> </span><span class="n">crushmap</span><span class="o">-</span><span class="n">new</span><span class="o">.</span><span class="n">bin</span>
<span class="n">crushtool</span><span class="w"> </span><span class="o">--</span><span class="n">test</span><span class="w"> </span><span class="o">-</span><span class="n">i</span><span class="w"> </span><span class="n">crushmap</span><span class="o">-</span><span class="n">new</span><span class="o">.</span><span class="n">bin</span><span class="w"> </span><span class="o">--</span><span class="n">show</span><span class="o">-</span><span class="n">utilization</span><span class="w"> </span><span class="o">--</span><span class="n">rule</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="o">--</span><span class="n">num</span><span class="o">-</span><span class="n">rep</span><span class="o">=</span><span class="mi">2</span>
<span class="n">crushtool</span><span class="w"> </span><span class="o">--</span><span class="n">test</span><span class="w"> </span><span class="o">-</span><span class="n">i</span><span class="w"> </span><span class="n">crushmap</span><span class="o">-</span><span class="n">new</span><span class="o">.</span><span class="n">bin</span><span class="w"> </span><span class="o">--</span><span class="n">show</span><span class="o">-</span><span class="n">choose</span><span class="o">-</span><span class="n">tries</span><span class="w"> </span><span class="o">--</span><span class="n">rule</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="o">--</span><span class="n">num</span><span class="o">-</span><span class="n">rep</span><span class="o">=</span><span class="mi">2</span>
</code></pre></div>
<p>Check the utilization :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>crushtool<span class="w"> </span>--test<span class="w"> </span>-i<span class="w"> </span>crushmap-new.bin<span class="w"> </span>--show-utilization<span class="w"> </span>--num-rep<span class="o">=</span><span class="m">2</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>^rule
rule<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">(</span>sata-rep_2dc<span class="o">)</span>,<span class="w"> </span><span class="nv">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>..1023,<span class="w"> </span><span class="nv">numrep</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">2</span>..2
rule<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">(</span>sata-rep_2dc<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>:<span class="w"> </span><span class="m">117</span>/1024
rule<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">(</span>sata-rep_2dc<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">1</span>:<span class="w"> </span><span class="m">448</span>/1024
rule<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">(</span>sata-rep_2dc<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">2</span>:<span class="w"> </span><span class="m">459</span>/1024
rule<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">(</span>ssd-rep_2dc<span class="o">)</span>,<span class="w"> </span><span class="nv">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>..1023,<span class="w"> </span><span class="nv">numrep</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">2</span>..2
rule<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">(</span>ssd-rep_2dc<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>:<span class="w"> </span><span class="m">459</span>/1024
rule<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">(</span>ssd-rep_2dc<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">1</span>:<span class="w"> </span><span class="m">448</span>/1024
rule<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">(</span>ssd-rep_2dc<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">2</span>:<span class="w"> </span><span class="m">117</span>/1024
rule<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="o">(</span>sata-all<span class="o">)</span>,<span class="w"> </span><span class="nv">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>..1023,<span class="w"> </span><span class="nv">numrep</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">2</span>..2
rule<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="o">(</span>sata-all<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>:<span class="w"> </span><span class="m">113</span>/1024
rule<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="o">(</span>sata-all<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">1</span>:<span class="w"> </span><span class="m">519</span>/1024
rule<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="o">(</span>sata-all<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">2</span>:<span class="w"> </span><span class="m">392</span>/1024
</code></pre></div>
<p>For all the rules, the number of replication is insufficient for a part of the sample. Particularly for drives in a minor amount (in that case ssd).
Looking at the number of retry to chooose an osd, we see that it will be useless to increase the "choose_total_tries" which is sufficient :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>crushtool<span class="w"> </span>--test<span class="w"> </span>-i<span class="w"> </span>crushmap-new.bin<span class="w"> </span>--show-choose-tries<span class="w"> </span>--rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>--num-rep<span class="o">=</span><span class="m">2</span>
<span class="w"> </span><span class="m">0</span>:<span class="w"> </span><span class="m">4298</span>
<span class="w"> </span><span class="m">1</span>:<span class="w"> </span><span class="m">226</span>
<span class="w"> </span><span class="m">2</span>:<span class="w"> </span><span class="m">130</span>
<span class="w"> </span><span class="m">3</span>:<span class="w"> </span><span class="m">59</span>
<span class="w"> </span><span class="m">4</span>:<span class="w"> </span><span class="m">38</span>
<span class="w"> </span><span class="m">5</span>:<span class="w"> </span><span class="m">11</span>
<span class="w"> </span><span class="m">6</span>:<span class="w"> </span><span class="m">10</span>
<span class="w"> </span><span class="m">7</span>:<span class="w"> </span><span class="m">3</span>
<span class="w"> </span><span class="m">8</span>:<span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">9</span>:<span class="w"> </span><span class="m">2</span>
<span class="m">10</span>:<span class="w"> </span><span class="m">1</span>
<span class="m">11</span>:<span class="w"> </span><span class="m">0</span>
<span class="m">12</span>:<span class="w"> </span><span class="m">2</span>
$<span class="w"> </span>crushtool<span class="w"> </span>--test<span class="w"> </span>-i<span class="w"> </span>crushmap-new.bin<span class="w"> </span>--show-choose-tries<span class="w"> </span>--rule<span class="w"> </span><span class="m">1</span><span class="w"> </span>--num-rep<span class="o">=</span><span class="m">2</span>
<span class="w"> </span><span class="m">0</span>:<span class="w"> </span><span class="m">2930</span>
<span class="w"> </span><span class="m">1</span>:<span class="w"> </span><span class="m">226</span>
<span class="w"> </span><span class="m">2</span>:<span class="w"> </span><span class="m">130</span>
<span class="w"> </span><span class="m">3</span>:<span class="w"> </span><span class="m">59</span>
<span class="w"> </span><span class="m">4</span>:<span class="w"> </span><span class="m">38</span>
<span class="w"> </span><span class="m">5</span>:<span class="w"> </span><span class="m">11</span>
<span class="w"> </span><span class="m">6</span>:<span class="w"> </span><span class="m">10</span>
<span class="w"> </span><span class="m">7</span>:<span class="w"> </span><span class="m">3</span>
<span class="w"> </span><span class="m">8</span>:<span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">9</span>:<span class="w"> </span><span class="m">2</span>
<span class="m">10</span>:<span class="w"> </span><span class="m">1</span>
<span class="m">11</span>:<span class="w"> </span><span class="m">0</span>
<span class="m">12</span>:<span class="w"> </span><span class="m">2</span>
$<span class="w"> </span>crushtool<span class="w"> </span>--test<span class="w"> </span>-i<span class="w"> </span>crushmap-new.bin<span class="w"> </span>--show-choose-tries<span class="w"> </span>--rule<span class="w"> </span><span class="m">2</span><span class="w"> </span>--num-rep<span class="o">=</span><span class="m">2</span>
<span class="w"> </span><span class="m">0</span>:<span class="w"> </span><span class="m">2542</span>
<span class="w"> </span><span class="m">1</span>:<span class="w"> </span><span class="m">52</span>
<span class="w"> </span><span class="m">2</span>:<span class="w"> </span><span class="m">12</span>
</code></pre></div>
<p>We can test to increase the number of osd for testing (It's not very pretty...) :</p>
<p>in sata-rep_2dc : <code>step chooseleaf firstn 5 type host-sata</code></p>
<p>in ssd-rep_2dc : <code>step chooseleaf firstn 5 type host-ssd</code></p>
<p>in sata-all : <code>step chooseleaf firstn 15 type host-sata</code></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>crushtool<span class="w"> </span>--test<span class="w"> </span>-i<span class="w"> </span>crushmap-new.bin<span class="w"> </span>--show-utilization<span class="w"> </span>--num-rep<span class="o">=</span><span class="m">2</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>^rule
rule<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">(</span>sata-rep_2dc<span class="o">)</span>,<span class="w"> </span><span class="nv">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>..1023,<span class="w"> </span><span class="nv">numrep</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">2</span>..2
rule<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">(</span>sata-rep_2dc<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">1</span>:<span class="w"> </span><span class="m">1</span>/1024
rule<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">(</span>sata-rep_2dc<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">2</span>:<span class="w"> </span><span class="m">1023</span>/1024
rule<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">(</span>ssd-rep_2dc<span class="o">)</span>,<span class="w"> </span><span class="nv">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>..1023,<span class="w"> </span><span class="nv">numrep</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">2</span>..2
rule<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">(</span>ssd-rep_2dc<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>:<span class="w"> </span><span class="m">20</span>/1024
rule<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">(</span>ssd-rep_2dc<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">1</span>:<span class="w"> </span><span class="m">247</span>/1024
rule<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">(</span>ssd-rep_2dc<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">2</span>:<span class="w"> </span><span class="m">757</span>/1024
rule<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="o">(</span>sata-all<span class="o">)</span>,<span class="w"> </span><span class="nv">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>..1023,<span class="w"> </span><span class="nv">numrep</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">2</span>..2
rule<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="o">(</span>sata-all<span class="o">)</span><span class="w"> </span>num_rep<span class="w"> </span><span class="m">2</span><span class="w"> </span>result<span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">2</span>:<span class="w"> </span><span class="m">1024</span>/1024
</code></pre></div>
<p>It's better, we see that for the rule "sata-all" it works pretty well, by cons, when there are fewer disk, the number of replications is always not correct.
The idea of this distribution was attractive, but quickly realizes that this can not work.</p>
<p>If people have explored this way, or have examples of advanced CRUSHMAP, I encourage you to share them. I'm curious of all that can be done with this. Meanwhile, the best is yet to make it simple to suit your needs. In most cases, the 1.3 model will be perfect.</p>
<p>More details on CRUSH algorithm :
http://ceph.com/papers/weil-crush-sc06.pdf</p>Multiple clusters on the same hardware: OSD isolation with LXC2015-01-29T08:00:41+01:002015-01-29T08:00:41+01:00Laurent Barbetag:cephnotes.ksperis.com,2015-01-29:/blog/2015/01/29/multiple-clusters-on-the-same-hardware-osd-isolation-with-lxc/<p>Ceph makes it easy to create multiple cluster on the same hardware with the naming of clusters. If you want a better insolation you can use LXC, for example to allow a different version of Ceph between your clusters.</p>
<p><img alt="Ceph LXC" src="https://cephnotes.ksperis.com/images/img019.png"></p>
<p>For this you will need access to the physical disks from …</p><p>Ceph makes it easy to create multiple cluster on the same hardware with the naming of clusters. If you want a better insolation you can use LXC, for example to allow a different version of Ceph between your clusters.</p>
<p><img alt="Ceph LXC" src="https://cephnotes.ksperis.com/images/img019.png"></p>
<p>For this you will need access to the physical disks from the container. You just allow access to the device with cgroup and create the device with mknod :</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Retrieve the major and minor number for a device :</span>
<span class="o">$</span><span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="o">-</span><span class="n">l</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sda5</span>
<span class="n">brw</span><span class="o">-</span><span class="n">rw</span><span class="o">---</span><span class="n">T</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="n">disk</span><span class="w"> </span><span class="mi">8</span><span class="p">,</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="n">janv</span><span class="o">.</span><span class="w"> </span><span class="mi">26</span><span class="w"> </span><span class="mi">18</span><span class="p">:</span><span class="mi">47</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sda5</span>
<span class="o">$</span><span class="w"> </span><span class="n">mknod</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">lxc</span><span class="o">/</span><span class="n">container</span><span class="o">-</span><span class="n">cluster1</span><span class="o">/</span><span class="n">rootfs</span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sda5</span><span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">5</span>
<span class="o">$</span><span class="w"> </span><span class="n">echo</span><span class="w"> </span><span class="s2">"lxc.cgroup.devices.allow = b 8:7 rwm"</span><span class="w"> </span><span class="o">>></span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">lxc</span><span class="o">/</span><span class="n">container</span><span class="o">-</span><span class="n">cluster1</span><span class="o">/</span><span class="n">config</span>
</code></pre></div>
<!--more-->
<p>However, if you want to use "ceph-deploy" you may be faced to many problems for the deployment of OSD. Indeed, tools needs to access access to all devices to work correctly. It will therefore also need to create all partition on physical host and add a line like that: <code>lxc.cgroup.devices.allow = b 8:* r</code> in lxc config. Also, udev is not active in the container, so it will need to create all devices or make a bind mount for /dev. <code>mount -o bind /dev /var/lib/lxc/container-cluster1/rootfs/dev/</code>. Then, udev rules will not run.... The easiest way remains the manual deployment for this part or use for example ceph-ansible which will certainly allows to be more flexible.</p>Replace Apache by Civetweb on the RadosGW2015-01-27T09:43:19+01:002015-01-27T09:43:19+01:00Laurent Barbetag:cephnotes.ksperis.com,2015-01-27:/blog/2015/01/27/replace-apache-by-civetweb-on-the-radosgw/<p>Since Firefly you can test the use of the lightweight web client Civetweb instead of Apache.
To activate it, it's very simple, there's nothing to install again, simply add this line to your ceph.conf:</p>
<div class="highlight"><pre><span></span><code><span class="k">[client.radosgw.gateway]</span>
<span class="na">rgw frontends</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"civetweb port=80"</span>
<span class="na">...</span>
</code></pre></div>
<p>If you have already installed apache, remember …</p><p>Since Firefly you can test the use of the lightweight web client Civetweb instead of Apache.
To activate it, it's very simple, there's nothing to install again, simply add this line to your ceph.conf:</p>
<div class="highlight"><pre><span></span><code><span class="k">[client.radosgw.gateway]</span>
<span class="na">rgw frontends</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"civetweb port=80"</span>
<span class="na">...</span>
</code></pre></div>
<p>If you have already installed apache, remember to stop it before activating civetweb, or it must not listen on the same port.</p>
<p>Then :</p>
<div class="highlight"><pre><span></span><code>/etc/init.d/radosgw restart
</code></pre></div>Difference between 'ceph osd reweight' and 'ceph osd crush reweight'2014-12-23T10:03:06+01:002014-12-23T10:03:06+01:00Laurent Barbetag:cephnotes.ksperis.com,2014-12-23:/blog/2014/12/23/difference-between-ceph-osd-reweight-and-ceph-osd-crush-reweight/<p>From Gregory and Craig in mailing list...</p>
<p>{% blockquote Gregory Farnum http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-June/040961.html %}
"ceph osd crush reweight" sets the CRUSH weight of the OSD. This
weight is an arbitrary value (generally the size of the disk in TB or
something) and controls how …</p><p>From Gregory and Craig in mailing list...</p>
<p>{% blockquote Gregory Farnum http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-June/040961.html %}
"ceph osd crush reweight" sets the CRUSH weight of the OSD. This
weight is an arbitrary value (generally the size of the disk in TB or
something) and controls how much data the system tries to allocate to
the OSD.</p>
<p _="%" endblockquote>"ceph osd reweight" sets an override weight on the OSD. This value is
in the range 0 to 1, and forces CRUSH to re-place (1-weight) of the
data that would otherwise live on this drive. It does <em>not</em> change the
weights assigned to the buckets above the OSD, and is a corrective
measure in case the normal CRUSH distribution isn't working out quite
right. (For instance, if one of your OSDs is at 90% and the others are
at 50%, you could reduce this weight to try and compensate for it.)</p>
<p>{% blockquote Craig Lewis http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-June/040967.html %}
Note that 'ceph osd reweight' is not a persistent setting. When an OSD
gets marked out, the osd weight will be set to 0. When it gets marked in
again, the weight will be changed to 1.</p>
<p _="%" endblockquote>Because of this 'ceph osd reweight' is a temporary solution. You should
only use it to keep your cluster running while you're ordering more
hardware.</p>
<p>http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-June/040961.html</p>
<!-- more -->
<p>I asked myself when one of my osd was marked down (on my old cluster in Cuttlefish...) and I noticed that only the drive of the local machine seemed to fill. Something that seems normal since the weight of the host had not changed in crushmap.</p>
<h2>Testing</h2>
<p>Testing on simple cluster (Giant), with this crushmap :</p>
<div class="highlight"><pre><span></span><code><span class="nx">ruleset</span><span class="w"> </span><span class="mi">0</span>
<span class="k">type</span><span class="w"> </span><span class="nx">replicated</span>
<span class="nx">min_size</span><span class="w"> </span><span class="mi">1</span>
<span class="nx">max_size</span><span class="w"> </span><span class="mi">10</span>
<span class="nx">step</span><span class="w"> </span><span class="nx">take</span><span class="w"> </span><span class="k">default</span>
<span class="nx">step</span><span class="w"> </span><span class="nx">chooseleaf</span><span class="w"> </span><span class="nx">firstn</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nx">host</span>
<span class="w"> </span><span class="nx">step</span><span class="w"> </span><span class="nx">emit</span>
</code></pre></div>
<p>Take the example of the 8 pgs on pool 3 :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^3.'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{print $1,$15;}'</span>
dumped<span class="w"> </span>all<span class="w"> </span><span class="k">in</span><span class="w"> </span>format<span class="w"> </span>plain
<span class="m">3</span>.4<span class="w"> </span><span class="o">[</span><span class="m">0</span>,2<span class="o">]</span>
<span class="m">3</span>.5<span class="w"> </span><span class="o">[</span><span class="m">4</span>,1<span class="o">]</span>
<span class="m">3</span>.6<span class="w"> </span><span class="o">[</span><span class="m">2</span>,0<span class="o">]</span>
<span class="m">3</span>.7<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span>
<span class="m">3</span>.0<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span>
<span class="m">3</span>.1<span class="w"> </span><span class="o">[</span><span class="m">0</span>,2<span class="o">]</span>
<span class="m">3</span>.2<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span>
<span class="m">3</span>.3<span class="w"> </span><span class="o">[</span><span class="m">2</span>,4<span class="o">]</span>
</code></pre></div>
<p>Now I try ceph osd out :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>out<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="c1"># This is equivalent to "ceph osd reweight 0 0"</span>
marked<span class="w"> </span>out<span class="w"> </span>osd.0.<span class="w"> </span>
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>tree
<span class="c1"># id weight type name up/down reweight</span>
-1<span class="w"> </span><span class="m">0</span>.2<span class="w"> </span>root<span class="w"> </span>default
-2<span class="w"> </span><span class="m">0</span>.09998<span class="w"> </span>host<span class="w"> </span>ceph-01
<span class="m">0</span><span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>osd.0<span class="w"> </span>up<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="c1"># <-- reweight has set to "0"</span>
<span class="m">4</span><span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>osd.4<span class="w"> </span>up<span class="w"> </span><span class="m">1</span><span class="w"> </span>
-3<span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>host<span class="w"> </span>ceph-02
<span class="m">1</span><span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>osd.1<span class="w"> </span>up<span class="w"> </span><span class="m">1</span><span class="w"> </span>
-4<span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>host<span class="w"> </span>ceph-03
<span class="m">2</span><span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>osd.2<span class="w"> </span>up<span class="w"> </span><span class="m">1</span><span class="w"> </span>
$<span class="w"> </span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^3.'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{print $1,$15;}'</span>
dumped<span class="w"> </span>all<span class="w"> </span><span class="k">in</span><span class="w"> </span>format<span class="w"> </span>plain
<span class="m">3</span>.4<span class="w"> </span><span class="o">[</span><span class="m">2</span>,4<span class="o">]</span><span class="w"> </span><span class="c1"># <-- [0,2] (move pg on osd.4)</span>
<span class="m">3</span>.5<span class="w"> </span><span class="o">[</span><span class="m">4</span>,1<span class="o">]</span>
<span class="m">3</span>.6<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span><span class="w"> </span><span class="c1"># <-- [2,0] (move pg on osd.1)</span>
<span class="m">3</span>.7<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span>
<span class="m">3</span>.0<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span>
<span class="m">3</span>.1<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span><span class="w"> </span><span class="c1"># <-- [0,2] (move pg on osd.1)</span>
<span class="m">3</span>.2<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span>
<span class="m">3</span>.3<span class="w"> </span><span class="o">[</span><span class="m">2</span>,4<span class="o">]</span>
</code></pre></div>
<p>Now I try ceph osd CRUSH out :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>crush<span class="w"> </span>reweight<span class="w"> </span>osd.0<span class="w"> </span><span class="m">0</span>
reweighted<span class="w"> </span>item<span class="w"> </span>id<span class="w"> </span><span class="m">0</span><span class="w"> </span>name<span class="w"> </span><span class="s1">'osd.0'</span><span class="w"> </span>to<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="k">in</span><span class="w"> </span>crush<span class="w"> </span>map
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>tree
<span class="c1"># id weight type name up/down reweight</span>
-1<span class="w"> </span><span class="m">0</span>.15<span class="w"> </span>root<span class="w"> </span>default
-2<span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>host<span class="w"> </span>ceph-01<span class="w"> </span><span class="c1"># <-- the weight of the host changed</span>
<span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span>osd.0<span class="w"> </span>up<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="c1"># <-- crush weight is set to "0"</span>
<span class="m">4</span><span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>osd.4<span class="w"> </span>up<span class="w"> </span><span class="m">1</span><span class="w"> </span>
-3<span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>host<span class="w"> </span>ceph-02
<span class="m">1</span><span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>osd.1<span class="w"> </span>up<span class="w"> </span><span class="m">1</span><span class="w"> </span>
-4<span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>host<span class="w"> </span>ceph-03
<span class="m">2</span><span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>osd.2<span class="w"> </span>up<span class="w"> </span><span class="m">1</span><span class="w"> </span>
$<span class="w"> </span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^3.'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{print $1,$15;}'</span>
dumped<span class="w"> </span>all<span class="w"> </span><span class="k">in</span><span class="w"> </span>format<span class="w"> </span>plain
<span class="m">3</span>.4<span class="w"> </span><span class="o">[</span><span class="m">4</span>,2<span class="o">]</span><span class="w"> </span><span class="c1"># <-- [0,2] (move pg on osd.4)</span>
<span class="m">3</span>.5<span class="w"> </span><span class="o">[</span><span class="m">4</span>,1<span class="o">]</span>
<span class="m">3</span>.6<span class="w"> </span><span class="o">[</span><span class="m">2</span>,4<span class="o">]</span><span class="w"> </span><span class="c1"># <-- [2,0] (move pg on osd.4)</span>
<span class="m">3</span>.7<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span>
<span class="m">3</span>.0<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span>
<span class="m">3</span>.1<span class="w"> </span><span class="o">[</span><span class="m">4</span>,2<span class="o">]</span><span class="w"> </span><span class="c1"># <-- [0,2] (move pg on osd.4)</span>
<span class="m">3</span>.2<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span>
<span class="m">3</span>.3<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span>
</code></pre></div>
<p>This does not seem very logical because the weight assigned to the bucket "host ceph-01" is still higher than the others. This would probably be different with more PG...</p>
<h2>Trying with more pgs</h2>
<div class="highlight"><pre><span></span><code><span class="c1"># Add more pg on my testpool</span>
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span>testpool<span class="w"> </span>pg_num<span class="w"> </span><span class="m">128</span>
<span class="nb">set</span><span class="w"> </span>pool<span class="w"> </span><span class="m">3</span><span class="w"> </span>pg_num<span class="w"> </span>to<span class="w"> </span><span class="m">128</span>
<span class="c1"># Check repartition</span>
$<span class="w"> </span><span class="k">for</span><span class="w"> </span>i<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">4</span><span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"osd.</span><span class="nv">$i</span><span class="s2">=</span><span class="k">$(</span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span><span class="m">2</span>>/dev/null<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^3.'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{print $15;}'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="nv">$i</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>wc<span class="w"> </span>-l<span class="k">)</span><span class="s2"> pgs"</span><span class="p">;</span><span class="w"> </span><span class="k">done</span>
osd.0<span class="o">=</span><span class="m">48</span><span class="w"> </span>pgs
osd.1<span class="o">=</span><span class="m">78</span><span class="w"> </span>pgs
osd.2<span class="o">=</span><span class="m">77</span><span class="w"> </span>pgs
osd.4<span class="o">=</span><span class="m">53</span><span class="w"> </span>pgs
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>reweight<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>
reweighted<span class="w"> </span>osd.0<span class="w"> </span>to<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">(</span><span class="m">802</span><span class="o">)</span>
$<span class="w"> </span><span class="k">for</span><span class="w"> </span>i<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">4</span><span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"osd.</span><span class="nv">$i</span><span class="s2">=</span><span class="k">$(</span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span><span class="m">2</span>>/dev/null<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^3.'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{print $15;}'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="nv">$i</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>wc<span class="w"> </span>-l<span class="k">)</span><span class="s2"> pgs"</span><span class="p">;</span><span class="w"> </span><span class="k">done</span>
osd.0<span class="o">=</span><span class="m">0</span><span class="w"> </span>pgs
osd.1<span class="o">=</span><span class="m">96</span><span class="w"> </span>pgs
osd.2<span class="o">=</span><span class="m">97</span><span class="w"> </span>pgs
osd.4<span class="o">=</span><span class="m">63</span><span class="w"> </span>pgs
</code></pre></div>
<p>The distribution seems fair. Why in the same case, with Cuttlefish, distribution is not the same ?</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>reweight<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1</span>
reweighted<span class="w"> </span>osd.0<span class="w"> </span>to<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">(</span><span class="m">802</span><span class="o">)</span>
$<span class="w"> </span><span class="k">for</span><span class="w"> </span>i<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">4</span><span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"osd.</span><span class="nv">$i</span><span class="s2">=</span><span class="k">$(</span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span><span class="m">2</span>>/dev/null<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^3.'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{print $15;}'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="nv">$i</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>wc<span class="w"> </span>-l<span class="k">)</span><span class="s2"> pgs"</span><span class="p">;</span><span class="w"> </span><span class="k">done</span>
osd.0<span class="o">=</span><span class="m">0</span><span class="w"> </span>pgs
osd.1<span class="o">=</span><span class="m">96</span><span class="w"> </span>pgs
osd.2<span class="o">=</span><span class="m">97</span><span class="w"> </span>pgs
osd.4<span class="o">=</span><span class="m">63</span><span class="w"> </span>pgs
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>crush<span class="w"> </span>reweight<span class="w"> </span>osd.0<span class="w"> </span><span class="m">0</span>
reweighted<span class="w"> </span>osd.0<span class="w"> </span>to<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">(</span><span class="m">802</span><span class="o">)</span>
$<span class="w"> </span><span class="k">for</span><span class="w"> </span>i<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">4</span><span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"osd.</span><span class="nv">$i</span><span class="s2">=</span><span class="k">$(</span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span><span class="m">2</span>>/dev/null<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^3.'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{print $15;}'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="nv">$i</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>wc<span class="w"> </span>-l<span class="k">)</span><span class="s2"> pgs"</span><span class="p">;</span><span class="w"> </span><span class="k">done</span>
osd.0<span class="o">=</span><span class="m">0</span><span class="w"> </span>pgs
osd.1<span class="o">=</span><span class="m">87</span><span class="w"> </span>pgs
osd.2<span class="o">=</span><span class="m">88</span><span class="w"> </span>pgs
osd.4<span class="o">=</span><span class="m">81</span><span class="w"> </span>pgs
</code></pre></div>
<p>With crush reweight, everything is normal.</p>
<h2>Trying with crush legacy</h2>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>crush<span class="w"> </span>tunables<span class="w"> </span>legacy
adjusted<span class="w"> </span>tunables<span class="w"> </span>profile<span class="w"> </span>to<span class="w"> </span>legacy
root@ceph-01:~/ceph-deploy#<span class="w"> </span><span class="k">for</span><span class="w"> </span>i<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">4</span><span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"osd.</span><span class="nv">$i</span><span class="s2">=</span><span class="k">$(</span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span><span class="m">2</span>>/dev/null<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^3.'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{print $15;}'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="nv">$i</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>wc<span class="w"> </span>-l<span class="k">)</span><span class="s2"> pgs"</span><span class="p">;</span><span class="w"> </span><span class="k">done</span>
osd.0<span class="o">=</span><span class="m">0</span><span class="w"> </span>pgs
osd.1<span class="o">=</span><span class="m">87</span><span class="w"> </span>pgs
osd.2<span class="o">=</span><span class="m">88</span><span class="w"> </span>pgs
osd.4<span class="o">=</span><span class="m">81</span><span class="w"> </span>pgs
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>crush<span class="w"> </span>reweight<span class="w"> </span>osd.0<span class="w"> </span><span class="m">0</span>.04999
reweighted<span class="w"> </span>item<span class="w"> </span>id<span class="w"> </span><span class="m">0</span><span class="w"> </span>name<span class="w"> </span><span class="s1">'osd.0'</span><span class="w"> </span>to<span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span><span class="k">in</span><span class="w"> </span>crush<span class="w"> </span>map
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>tree
<span class="c1"># id weight type name up/down reweight</span>
-1<span class="w"> </span><span class="m">0</span>.2<span class="w"> </span>root<span class="w"> </span>default
-2<span class="w"> </span><span class="m">0</span>.09998<span class="w"> </span>host<span class="w"> </span>ceph-01
<span class="m">0</span><span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>osd.0<span class="w"> </span>up<span class="w"> </span><span class="m">0</span><span class="w"> </span>
<span class="m">4</span><span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>osd.4<span class="w"> </span>up<span class="w"> </span><span class="m">1</span><span class="w"> </span>
-3<span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>host<span class="w"> </span>ceph-02
<span class="m">1</span><span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>osd.1<span class="w"> </span>up<span class="w"> </span><span class="m">1</span><span class="w"> </span>
-4<span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>host<span class="w"> </span>ceph-03
<span class="m">2</span><span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>osd.2<span class="w"> </span>up<span class="w"> </span><span class="m">1</span><span class="w"> </span>
$<span class="w"> </span><span class="k">for</span><span class="w"> </span>i<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">4</span><span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"osd.</span><span class="nv">$i</span><span class="s2">=</span><span class="k">$(</span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span><span class="m">2</span>>/dev/null<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^3.'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{print $15;}'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="nv">$i</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>wc<span class="w"> </span>-l<span class="k">)</span><span class="s2"> pgs"</span><span class="p">;</span><span class="w"> </span><span class="k">done</span>
osd.0<span class="o">=</span><span class="m">0</span><span class="w"> </span>pgs
osd.1<span class="o">=</span><span class="m">78</span><span class="w"> </span>pgs
osd.2<span class="o">=</span><span class="m">77</span><span class="w"> </span>pgs
osd.4<span class="o">=</span><span class="m">101</span><span class="w"> </span>pgs<span class="w"> </span><span class="c1"># <--- All pg from osd.0 and osd.4 is here when using legacy value (on host ceph-01)</span>
</code></pre></div>
<p>So, it is an evolution of the distribution algorithm to prefer a more global distribution when OSD is marked down (instead of distributing preferably by proximity). Indeed the old distribution can cause problems when there is not a lot of OSD by host, and that they are nearly full.</p>
<p _="%" endblockquote>{% blockquote Ceph Docs http://ceph.com/docs/master/rados/operations/crush-map/#impact-of-legacy-values %}
When some OSDs are marked out, the data tends to get redistributed to nearby OSDs instead of across the entire hierarchy.</p>
<p>To view the number of pg per osd :</p>
<p>http://cephnotes.ksperis.com/blog/2015/02/23/get-the-number-of-placement-groups-per-osd/</p>Use trim/discard with rbd kernel client (since kernel 3.18)2014-12-18T17:57:13+01:002014-12-18T17:57:13+01:00Laurent Barbetag:cephnotes.ksperis.com,2014-12-18:/blog/2014/12/18/use-discard-with-krbd-client-since-kernel-3-dot-18/<p>Realtime :</p>
<div class="highlight"><pre><span></span><code>mount<span class="w"> </span>-o<span class="w"> </span>discard<span class="w"> </span>/dev/rbd0<span class="w"> </span>/mnt/myrbd
</code></pre></div>
<p>Using batch :</p>
<div class="highlight"><pre><span></span><code>fstrim<span class="w"> </span>/mnt/myrbd
</code></pre></div>
<!-- more -->
<h2>Test</h2>
<p>The empty FS :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>create<span class="w"> </span>rbd/myrbd<span class="w"> </span>--size<span class="o">=</span><span class="m">20480</span>
$<span class="w"> </span>mkfs.xfs<span class="w"> </span>/dev/rbd0
$<span class="w"> </span>rbd<span class="w"> </span>diff<span class="w"> </span>rbd/myrbd<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{ SUM += $2 } END { print SUM/1024/1024 " MB" }'</span>
<span class="m">14</span>.4062<span class="w"> </span>MB
</code></pre></div>
<p>With a big file... :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mount<span class="w"> </span>/dev/rbd0 …</code></pre></div><p>Realtime :</p>
<div class="highlight"><pre><span></span><code>mount<span class="w"> </span>-o<span class="w"> </span>discard<span class="w"> </span>/dev/rbd0<span class="w"> </span>/mnt/myrbd
</code></pre></div>
<p>Using batch :</p>
<div class="highlight"><pre><span></span><code>fstrim<span class="w"> </span>/mnt/myrbd
</code></pre></div>
<!-- more -->
<h2>Test</h2>
<p>The empty FS :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>create<span class="w"> </span>rbd/myrbd<span class="w"> </span>--size<span class="o">=</span><span class="m">20480</span>
$<span class="w"> </span>mkfs.xfs<span class="w"> </span>/dev/rbd0
$<span class="w"> </span>rbd<span class="w"> </span>diff<span class="w"> </span>rbd/myrbd<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{ SUM += $2 } END { print SUM/1024/1024 " MB" }'</span>
<span class="m">14</span>.4062<span class="w"> </span>MB
</code></pre></div>
<p>With a big file... :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mount<span class="w"> </span>/dev/rbd0<span class="w"> </span>/mnt/myrbd
$<span class="w"> </span>dd<span class="w"> </span><span class="k">if</span><span class="o">=</span>/dev/zero<span class="w"> </span><span class="nv">of</span><span class="o">=</span>/mnt/myrbd/testfile<span class="w"> </span><span class="nv">bs</span><span class="o">=</span>1M<span class="w"> </span><span class="nv">count</span><span class="o">=</span><span class="m">1024</span>
$<span class="w"> </span>rbd<span class="w"> </span>diff<span class="w"> </span>rbd/myrbd<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{ SUM += $2 } END { print SUM/1024/1024 " MB" }'</span>
<span class="m">1038</span>.41<span class="w"> </span>MB
</code></pre></div>
<p>When the big file has been removed (with file system not mount with "discard") :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rm<span class="w"> </span>/mnt/myrbd/testfile
$<span class="w"> </span>rbd<span class="w"> </span>diff<span class="w"> </span>rbd/myrbd<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{ SUM += $2 } END { print SUM/1024/1024 " MB" }'</span>
<span class="m">1038</span>.41<span class="w"> </span>MB
</code></pre></div>
<p>Launch FS trim :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>fstrim<span class="w"> </span>/mnt/myrbd
$<span class="w"> </span>rbd<span class="w"> </span>diff<span class="w"> </span>rbd/myrbd<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{ SUM += $2 } END { print SUM/1024/1024 " MB" }'</span>
<span class="m">10</span>.6406<span class="w"> </span>MB
</code></pre></div>
<h2>Benchmark discard option</h2>
<p>Without "discard" :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mount<span class="w"> </span>/dev/rbd0<span class="w"> </span>/mnt/rbd0
$<span class="w"> </span>mkdir<span class="w"> </span>testdir<span class="p">;</span><span class="w"> </span><span class="nb">cd</span><span class="w"> </span>testdir
$<span class="w"> </span>dd<span class="w"> </span><span class="k">if</span><span class="o">=</span>/dev/zero<span class="w"> </span><span class="nv">of</span><span class="o">=</span>mainfile<span class="w"> </span><span class="nv">bs</span><span class="o">=</span>1M<span class="w"> </span><span class="nv">count</span><span class="o">=</span><span class="m">200</span>
$<span class="w"> </span>split<span class="w"> </span>-b<span class="w"> </span><span class="m">4048</span><span class="w"> </span>-a<span class="w"> </span><span class="m">7</span><span class="w"> </span>mainfile<span class="p">;</span><span class="w"> </span>sync<span class="w"> </span><span class="c1"># 4k file / ~51k files</span>
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>..
$<span class="w"> </span><span class="nb">time</span><span class="w"> </span>rm<span class="w"> </span>-rf<span class="w"> </span>testdir<span class="p">;</span><span class="w"> </span><span class="nb">time</span><span class="w"> </span>sync
real<span class="w"> </span>0m2.780s
user<span class="w"> </span>0m0.096s
sys<span class="w"> </span>0m2.632s
real<span class="w"> </span>0m0.130s
user<span class="w"> </span>0m0.004s
sys<span class="w"> </span>0m0.016s
<span class="c1"># total: < 3s</span>
</code></pre></div>
<p>With "discard" :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mount<span class="w"> </span>-o<span class="w"> </span>discard<span class="w"> </span>/dev/rbd1<span class="w"> </span>/mnt/rbd1
$<span class="w"> </span>mkdir<span class="w"> </span>testdir<span class="p">;</span><span class="w"> </span><span class="nb">cd</span><span class="w"> </span>testdir
$<span class="w"> </span>dd<span class="w"> </span><span class="k">if</span><span class="o">=</span>/dev/zero<span class="w"> </span><span class="nv">of</span><span class="o">=</span>mainfile<span class="w"> </span><span class="nv">bs</span><span class="o">=</span>1M<span class="w"> </span><span class="nv">count</span><span class="o">=</span><span class="m">200</span>
$<span class="w"> </span>split<span class="w"> </span>-b<span class="w"> </span><span class="m">4048</span><span class="w"> </span>-a<span class="w"> </span><span class="m">7</span><span class="w"> </span>mainfile<span class="p">;</span><span class="w"> </span>sync<span class="w"> </span><span class="c1"># 4k file / ~51k files</span>
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>..
$<span class="w"> </span><span class="nb">time</span><span class="w"> </span>rm<span class="w"> </span>-rf<span class="w"> </span>testdir<span class="p">;</span><span class="w"> </span><span class="nb">time</span><span class="w"> </span>sync
real<span class="w"> </span>1m51.471s
user<span class="w"> </span>0m0.104s
sys<span class="w"> </span>0m2.084s
real<span class="w"> </span>0m47.262s
user<span class="w"> </span>0m0.000s
sys<span class="w"> </span>0m0.008s
<span class="c1"># total: ~1m56</span>
</code></pre></div>
<p>This is on 2 osd without ssd journal...</p>
<p>In the case of intensive use of the file system, with many small file, it may be more advantageous to use fstrim, for example once a day.</p>
<p>(Ceph osd support trim for block device since 0.46)</p>placement_pools on Rados-GW2014-11-28T22:41:04+01:002014-11-28T22:41:04+01:00Laurent Barbetag:cephnotes.ksperis.com,2014-11-28:/blog/2014/11/28/placement-pools-on-rados-gw/<p>The purpose of this test is to map a RadosGw Bucket to a specific Ceph pool. For exemple, if using a fast pool with ssd and a low pool for archive...</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="n">standard_bucket</span><span class="w"> </span><span class="n">datas</span><span class="w"> </span><span class="o">--></span><span class="w"> </span><span class="p">.</span><span class="n">rgw</span><span class="p">.</span><span class="n">buckets</span><span class="w"> </span><span class="p">(</span><span class="n">default</span><span class="w"> </span><span class="n">pool</span><span class="p">)</span>
<span class="w"> </span><span class="n">specific_bucket</span><span class="w"> </span><span class="n">datas</span><span class="w"> </span><span class="o">--></span><span class="w"> </span><span class="p">.</span><span class="n">rgw</span><span class="p">.</span><span class="n">buckets</span><span class="p">.</span><span class="n">custom</span>
</code></pre></div>
<!-- more -->
<p>First, we create a pool .rgw.buckets.custom …</p><p>The purpose of this test is to map a RadosGw Bucket to a specific Ceph pool. For exemple, if using a fast pool with ssd and a low pool for archive...</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="n">standard_bucket</span><span class="w"> </span><span class="n">datas</span><span class="w"> </span><span class="o">--></span><span class="w"> </span><span class="p">.</span><span class="n">rgw</span><span class="p">.</span><span class="n">buckets</span><span class="w"> </span><span class="p">(</span><span class="n">default</span><span class="w"> </span><span class="n">pool</span><span class="p">)</span>
<span class="w"> </span><span class="n">specific_bucket</span><span class="w"> </span><span class="n">datas</span><span class="w"> </span><span class="o">--></span><span class="w"> </span><span class="p">.</span><span class="n">rgw</span><span class="p">.</span><span class="n">buckets</span><span class="p">.</span><span class="n">custom</span>
</code></pre></div>
<!-- more -->
<p>First, we create a pool .rgw.buckets.custom, with, for example, some specific parameters (different size and different ruleset in crushmap) :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>.rgw.buckets.custom<span class="w"> </span><span class="m">64</span><span class="w"> </span><span class="m">64</span>
pool<span class="w"> </span><span class="s1">'.rgw.buckets.custom'</span><span class="w"> </span>created
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span>.rgw.buckets.custom<span class="w"> </span>size<span class="w"> </span><span class="m">2</span>
<span class="nb">set</span><span class="w"> </span>pool<span class="w"> </span><span class="m">59</span><span class="w"> </span>size<span class="w"> </span>to<span class="w"> </span><span class="m">2</span>
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span>.rgw.buckets.custom<span class="w"> </span>crush_ruleset<span class="w"> </span><span class="m">6</span>
<span class="nb">set</span><span class="w"> </span>pool<span class="w"> </span><span class="m">59</span><span class="w"> </span>crush_ruleset<span class="w"> </span>to<span class="w"> </span><span class="m">6</span>
</code></pre></div>
<p>Then, we need to configure a specific placement_targets in region map and zone.
For next step, you need to have a running config of rados-gw...</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>radosgw-admin<span class="w"> </span>region<span class="w"> </span>get<span class="w"> </span>><span class="w"> </span>region.conf.json
$<span class="w"> </span>vim<span class="w"> </span>region.conf.json<span class="w"> </span><span class="c1"># Add an entry in placement_targets</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"default"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"api_name"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"is_master"</span><span class="p">:</span><span class="w"> </span><span class="s2">"true"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"endpoints"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span>
<span class="w"> </span><span class="nt">"master_zone"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"zones"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"default"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"endpoints"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span>
<span class="w"> </span><span class="nt">"log_meta"</span><span class="p">:</span><span class="w"> </span><span class="s2">"false"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"log_data"</span><span class="p">:</span><span class="w"> </span><span class="s2">"false"</span><span class="p">}],</span>
<span class="w"> </span><span class="nt">"placement_targets"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"default-placement"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"tags"</span><span class="p">:</span><span class="w"> </span><span class="p">[]},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"custom-placement"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"tags"</span><span class="p">:</span><span class="w"> </span><span class="p">[]}],</span>
<span class="w"> </span><span class="nt">"default_placement"</span><span class="p">:</span><span class="w"> </span><span class="s2">"default-placement"</span><span class="p">}</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>radosgw-admin<span class="w"> </span>region<span class="w"> </span><span class="nb">set</span><span class="w"> </span><<span class="w"> </span>region.conf.json
....
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>radosgw-admin<span class="w"> </span>zone<span class="w"> </span>get<span class="w"> </span>><span class="w"> </span>zone.conf.json
$<span class="w"> </span>vim<span class="w"> </span>zone.conf.json<span class="w"> </span><span class="c1"># Add an entry in placement_pools with key "custom-placement"</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"> </span><span class="nt">"domain_root"</span><span class="p">:</span><span class="w"> </span><span class="s2">".rgw"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"control_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".rgw.control"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"gc_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".rgw.gc"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"log_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".log"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"intent_log_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".intent-log"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"usage_log_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".usage"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"user_keys_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".users"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"user_email_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".users.email"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"user_swift_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".users.swift"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"user_uid_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".users.uid"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"system_key"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"access_key"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"secret_key"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">},</span>
<span class="w"> </span><span class="nt">"placement_pools"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"default-placement"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"val"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"index_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".rgw.buckets.index"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".rgw.buckets"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_extra_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".rgw.buckets.extra"</span><span class="p">}},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"custom-placement"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"val"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"index_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".rgw.buckets.index"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".rgw.buckets.custom"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"data_extra_pool"</span><span class="p">:</span><span class="w"> </span><span class="s2">".rgw.buckets.extra"</span><span class="p">}}]}</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>radosgw-admin<span class="w"> </span>zone<span class="w"> </span><span class="nb">set</span><span class="w"> </span><zone.conf.json
<span class="m">2014</span>-11-25<span class="w"> </span><span class="m">18</span>:03:23.894153<span class="w"> </span>7f728c0f2780<span class="w"> </span><span class="m">0</span><span class="w"> </span>couldn<span class="err">'</span>t<span class="w"> </span>find<span class="w"> </span>old<span class="w"> </span>data<span class="w"> </span>placement<span class="w"> </span>pools<span class="w"> </span>config,<span class="w"> </span>setting<span class="w"> </span>up<span class="w"> </span>new<span class="w"> </span>ones<span class="w"> </span><span class="k">for</span><span class="w"> </span>the<span class="w"> </span>zone
.....
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>radosgw-admin<span class="w"> </span>regionmap<span class="w"> </span>update
<span class="o">{</span><span class="w"> </span><span class="s2">"regions"</span>:<span class="w"> </span><span class="o">[</span>
<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"key"</span>:<span class="w"> </span><span class="s2">"default"</span>,
<span class="w"> </span><span class="s2">"val"</span>:<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"name"</span>:<span class="w"> </span><span class="s2">"default"</span>,
<span class="w"> </span><span class="s2">"api_name"</span>:<span class="w"> </span><span class="s2">""</span>,
<span class="w"> </span><span class="s2">"is_master"</span>:<span class="w"> </span><span class="s2">"true"</span>,
<span class="w"> </span><span class="s2">"endpoints"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"master_zone"</span>:<span class="w"> </span><span class="s2">""</span>,
<span class="w"> </span><span class="s2">"zones"</span>:<span class="w"> </span><span class="o">[</span>
<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"name"</span>:<span class="w"> </span><span class="s2">"default"</span>,
<span class="w"> </span><span class="s2">"endpoints"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"log_meta"</span>:<span class="w"> </span><span class="s2">"false"</span>,
<span class="w"> </span><span class="s2">"log_data"</span>:<span class="w"> </span><span class="s2">"false"</span><span class="o">}]</span>,
<span class="w"> </span><span class="s2">"placement_targets"</span>:<span class="w"> </span><span class="o">[</span>
<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"name"</span>:<span class="w"> </span><span class="s2">"custom-placement"</span>,
<span class="w"> </span><span class="s2">"tags"</span>:<span class="w"> </span><span class="o">[]}</span>,
<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"name"</span>:<span class="w"> </span><span class="s2">"default-placement"</span>,
<span class="w"> </span><span class="s2">"tags"</span>:<span class="w"> </span><span class="o">[]}]</span>,
<span class="w"> </span><span class="s2">"default_placement"</span>:<span class="w"> </span><span class="s2">"default-placement"</span><span class="o">}}]</span>,
<span class="w"> </span><span class="s2">"master_region"</span>:<span class="w"> </span><span class="s2">"default"</span>,
<span class="w"> </span><span class="s2">"bucket_quota"</span>:<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"enabled"</span>:<span class="w"> </span>false,
<span class="w"> </span><span class="s2">"max_size_kb"</span>:<span class="w"> </span>-1,
<span class="w"> </span><span class="s2">"max_objects"</span>:<span class="w"> </span>-1<span class="o">}</span>,
<span class="w"> </span><span class="s2">"user_quota"</span>:<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"enabled"</span>:<span class="w"> </span>false,
<span class="w"> </span><span class="s2">"max_size_kb"</span>:<span class="w"> </span>-1,
<span class="w"> </span><span class="s2">"max_objects"</span>:<span class="w"> </span>-1<span class="o">}}</span>
$<span class="w"> </span>/etc/init.d/radosgw<span class="w"> </span>reload
Reloading<span class="w"> </span>...
</code></pre></div>
<p>To configure s3cmd for RadosGW you can have a look here :
http://lollyrock.com/articles/s3cmd-with-radosgw/</p>
<p>Now we can test bucket creation :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>s3cmd<span class="w"> </span>mb<span class="w"> </span>s3://custombucket<span class="w"> </span>--bucket-location<span class="o">=</span>custom-placement
Bucket<span class="w"> </span><span class="s1">'custombucket'</span><span class="w"> </span>created
$<span class="w"> </span>touch<span class="w"> </span><span class="s2">"file_on_custom_pool"</span>
$<span class="w"> </span>s3cmd<span class="w"> </span>put<span class="w"> </span>file_on_custom_pool<span class="w"> </span>s3://custombucket
WARNING:<span class="w"> </span>Module<span class="w"> </span>python-magic<span class="w"> </span>is<span class="w"> </span>not<span class="w"> </span>available.<span class="w"> </span>Guessing<span class="w"> </span>MIME<span class="w"> </span>types<span class="w"> </span>based<span class="w"> </span>on<span class="w"> </span>file<span class="w"> </span>extensions.
file_on_custom_pool<span class="w"> </span>-><span class="w"> </span>s3://custombucket/file_on_custom_pool<span class="w"> </span><span class="o">[</span><span class="m">1</span><span class="w"> </span>of<span class="w"> </span><span class="m">1</span><span class="o">]</span>
<span class="w"> </span><span class="m">0</span><span class="w"> </span>of<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>%<span class="w"> </span><span class="k">in</span><span class="w"> </span>0s<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span>B/s<span class="w"> </span><span class="k">done</span>
</code></pre></div>
<p>Verify :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>radosgw-admin<span class="w"> </span>bucket<span class="w"> </span>stats<span class="w"> </span>--bucket<span class="o">=</span>custombucket
<span class="o">{</span><span class="w"> </span><span class="s2">"bucket"</span>:<span class="w"> </span><span class="s2">"custombucket"</span>,
<span class="w"> </span><span class="s2">"pool"</span>:<span class="w"> </span><span class="s2">".rgw.buckets.custom"</span>,
<span class="w"> </span><span class="s2">"index_pool"</span>:<span class="w"> </span><span class="s2">".rgw.buckets.index"</span>,
<span class="w"> </span><span class="s2">"id"</span>:<span class="w"> </span><span class="s2">"default.240909.1"</span>,
<span class="w"> </span><span class="s2">"marker"</span>:<span class="w"> </span><span class="s2">"default.240909.1"</span>,
<span class="w"> </span><span class="s2">"owner"</span>:<span class="w"> </span><span class="s2">"testuser"</span>,
<span class="w"> </span><span class="s2">"ver"</span>:<span class="w"> </span><span class="m">1</span>,
<span class="w"> </span><span class="s2">"master_ver"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"mtime"</span>:<span class="w"> </span><span class="m">1417016078</span>,
<span class="w"> </span><span class="s2">"max_marker"</span>:<span class="w"> </span><span class="s2">""</span>,
<span class="w"> </span><span class="s2">"usage"</span>:<span class="w"> </span><span class="o">{}</span>,
<span class="w"> </span><span class="s2">"bucket_quota"</span>:<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"enabled"</span>:<span class="w"> </span>false,
<span class="w"> </span><span class="s2">"max_size_kb"</span>:<span class="w"> </span>-1,
<span class="w"> </span><span class="s2">"max_objects"</span>:<span class="w"> </span>-1<span class="o">}}</span>
</code></pre></div>
<p>Pool var is set on ".rgw.buckets.custom".</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>.rgw.buckets.custom<span class="w"> </span>ls
default.241071.1_file_on_custom_pool
</code></pre></div>
<p>It's here !</p>
<p>Data placement pool is define in this order : </p>
<ol>
<li>from the request ("bucket location")</li>
<li>from user ("default_placement" : see with <code>radosgw-admin metadata get user:<uid></code>)</li>
<li>from region map ("default_placement")</li>
</ol>Ceph RBD with LXC containers2014-11-17T11:15:19+01:002014-11-17T11:15:19+01:00Laurent Barbetag:cephnotes.ksperis.com,2014-11-17:/blog/2014/11/17/ceph-rbd-with-lxc-containers/<p _="%" endblockquote>{% blockquote LXC 2.0.0 First Support for Ceph RBD : http://cephnotes.ksperis.com/blog/2016/04/14/lxc-2-dot-0-0-first-support-for-ceph-rbd %}
Update on Apr 14th, 2016:</p>
<p>A simple way to secure your data with containers is to use a distributed storage such as Ceph for LXC root storage.</p>
<p>For exemple :</p>
<div class="highlight"><pre><span></span><code><span class="c1"># lxc-create -n …</span></code></pre></div><p _="%" endblockquote>{% blockquote LXC 2.0.0 First Support for Ceph RBD : http://cephnotes.ksperis.com/blog/2016/04/14/lxc-2-dot-0-0-first-support-for-ceph-rbd %}
Update on Apr 14th, 2016:</p>
<p>A simple way to secure your data with containers is to use a distributed storage such as Ceph for LXC root storage.</p>
<p>For exemple :</p>
<div class="highlight"><pre><span></span><code><span class="c1"># lxc-create -n mycontainer -t debian -B rbd --pool rbd --rbd mycontainer --fstype ext4 --fssize 500</span>
</code></pre></div>
<!-- more -->
<div class="highlight"><pre><span></span><code><span class="n">mke2fs</span><span class="w"> </span><span class="mf">1.42</span><span class="o">.</span><span class="mi">5</span><span class="w"> </span><span class="p">(</span><span class="mi">29</span><span class="o">-</span><span class="n">Jul</span><span class="o">-</span><span class="mi">2012</span><span class="p">)</span>
<span class="n">Filesystem</span><span class="w"> </span><span class="n">label</span><span class="o">=</span>
<span class="n">OS</span><span class="w"> </span><span class="n">type</span><span class="p">:</span><span class="w"> </span><span class="n">Linux</span>
<span class="n">Block</span><span class="w"> </span><span class="n">size</span><span class="o">=</span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="nb">log</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="n">Fragment</span><span class="w"> </span><span class="n">size</span><span class="o">=</span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="nb">log</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="n">Stride</span><span class="o">=</span><span class="mi">4096</span><span class="w"> </span><span class="n">blocks</span><span class="p">,</span><span class="w"> </span><span class="n">Stripe</span><span class="w"> </span><span class="n">width</span><span class="o">=</span><span class="mi">4096</span><span class="w"> </span><span class="n">blocks</span>
<span class="mi">128016</span><span class="w"> </span><span class="n">inodes</span><span class="p">,</span><span class="w"> </span><span class="mi">512000</span><span class="w"> </span><span class="n">blocks</span>
<span class="mi">25600</span><span class="w"> </span><span class="n">blocks</span><span class="w"> </span><span class="p">(</span><span class="mf">5.00</span><span class="o">%</span><span class="p">)</span><span class="w"> </span><span class="n">reserved</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">super</span><span class="w"> </span><span class="n">user</span>
<span class="n">First</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="n">block</span><span class="o">=</span><span class="mi">1</span>
<span class="n">Maximum</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">blocks</span><span class="o">=</span><span class="mi">67633152</span>
<span class="mi">63</span><span class="w"> </span><span class="n">block</span><span class="w"> </span><span class="n">groups</span>
<span class="mi">8192</span><span class="w"> </span><span class="n">blocks</span><span class="w"> </span><span class="n">per</span><span class="w"> </span><span class="n">group</span><span class="p">,</span><span class="w"> </span><span class="mi">8192</span><span class="w"> </span><span class="n">fragments</span><span class="w"> </span><span class="n">per</span><span class="w"> </span><span class="n">group</span>
<span class="mi">2032</span><span class="w"> </span><span class="n">inodes</span><span class="w"> </span><span class="n">per</span><span class="w"> </span><span class="n">group</span>
<span class="n">Superblock</span><span class="w"> </span><span class="n">backups</span><span class="w"> </span><span class="n">stored</span><span class="w"> </span><span class="n">on</span><span class="w"> </span><span class="n">blocks</span><span class="p">:</span><span class="w"> </span>
<span class="w"> </span><span class="mi">8193</span><span class="p">,</span><span class="w"> </span><span class="mi">24577</span><span class="p">,</span><span class="w"> </span><span class="mi">40961</span><span class="p">,</span><span class="w"> </span><span class="mi">57345</span><span class="p">,</span><span class="w"> </span><span class="mi">73729</span><span class="p">,</span><span class="w"> </span><span class="mi">204801</span><span class="p">,</span><span class="w"> </span><span class="mi">221185</span><span class="p">,</span><span class="w"> </span><span class="mi">401409</span>
<span class="n">Allocating</span><span class="w"> </span><span class="n">group</span><span class="w"> </span><span class="n">tables</span><span class="p">:</span><span class="w"> </span><span class="n">done</span><span class="w"> </span>
<span class="n">Writing</span><span class="w"> </span><span class="n">inode</span><span class="w"> </span><span class="n">tables</span><span class="p">:</span><span class="w"> </span><span class="n">done</span><span class="w"> </span>
<span class="n">Creating</span><span class="w"> </span><span class="n">journal</span><span class="w"> </span><span class="p">(</span><span class="mi">8192</span><span class="w"> </span><span class="n">blocks</span><span class="p">):</span><span class="w"> </span><span class="n">done</span>
<span class="n">Writing</span><span class="w"> </span><span class="n">superblocks</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">accounting</span><span class="w"> </span><span class="n">information</span><span class="p">:</span><span class="w"> </span><span class="n">done</span><span class="w"> </span>
<span class="n">Note</span><span class="p">:</span><span class="w"> </span><span class="n">Usually</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">template</span><span class="w"> </span><span class="n">option</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">called</span><span class="w"> </span><span class="n">with</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">configuration</span>
<span class="n">file</span><span class="w"> </span><span class="n">option</span><span class="w"> </span><span class="n">too</span><span class="p">,</span><span class="w"> </span><span class="n">mostly</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">configure</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">network</span><span class="o">.</span>
<span class="n">For</span><span class="w"> </span><span class="n">more</span><span class="w"> </span><span class="n">information</span><span class="w"> </span><span class="n">look</span><span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">lxc</span><span class="o">.</span><span class="n">conf</span><span class="w"> </span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
<span class="n">debootstrap</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">sbin</span><span class="o">/</span><span class="n">debootstrap</span>
<span class="n">Checking</span><span class="w"> </span><span class="n">cache</span><span class="w"> </span><span class="n">download</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">cache</span><span class="o">/</span><span class="n">lxc</span><span class="o">/</span><span class="n">debian</span><span class="o">/</span><span class="n">rootfs</span><span class="o">-</span><span class="n">wheezy</span><span class="o">-</span><span class="n">amd64</span><span class="w"> </span><span class="o">...</span><span class="w"> </span>
<span class="n">Copying</span><span class="w"> </span><span class="n">rootfs</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">lxc</span><span class="o">/</span><span class="n">mycontainer</span><span class="o">/</span><span class="n">rootfs</span><span class="o">...</span><span class="n">Generating</span><span class="w"> </span><span class="n">locales</span><span class="w"> </span><span class="p">(</span><span class="n">this</span><span class="w"> </span><span class="n">might</span><span class="w"> </span><span class="n">take</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="k">while</span><span class="p">)</span><span class="o">...</span>
<span class="w"> </span><span class="n">en_US</span><span class="o">.</span><span class="n">UTF</span><span class="o">-</span><span class="mf">8.</span><span class="o">..</span><span class="w"> </span><span class="n">done</span>
<span class="n">Generation</span><span class="w"> </span><span class="n">complete</span><span class="o">.</span>
<span class="n">update</span><span class="o">-</span><span class="n">rc</span><span class="o">.</span><span class="n">d</span><span class="p">:</span><span class="w"> </span><span class="n">using</span><span class="w"> </span><span class="n">dependency</span><span class="w"> </span><span class="n">based</span><span class="w"> </span><span class="n">boot</span><span class="w"> </span><span class="n">sequencing</span>
<span class="n">update</span><span class="o">-</span><span class="n">rc</span><span class="o">.</span><span class="n">d</span><span class="p">:</span><span class="w"> </span><span class="n">using</span><span class="w"> </span><span class="n">dependency</span><span class="w"> </span><span class="n">based</span><span class="w"> </span><span class="n">boot</span><span class="w"> </span><span class="n">sequencing</span>
<span class="n">update</span><span class="o">-</span><span class="n">rc</span><span class="o">.</span><span class="n">d</span><span class="p">:</span><span class="w"> </span><span class="n">using</span><span class="w"> </span><span class="n">dependency</span><span class="w"> </span><span class="n">based</span><span class="w"> </span><span class="n">boot</span><span class="w"> </span><span class="n">sequencing</span>
<span class="n">update</span><span class="o">-</span><span class="n">rc</span><span class="o">.</span><span class="n">d</span><span class="p">:</span><span class="w"> </span><span class="n">using</span><span class="w"> </span><span class="n">dependency</span><span class="w"> </span><span class="n">based</span><span class="w"> </span><span class="n">boot</span><span class="w"> </span><span class="n">sequencing</span>
<span class="n">Current</span><span class="w"> </span><span class="n">default</span><span class="w"> </span><span class="n">time</span><span class="w"> </span><span class="n">zone</span><span class="p">:</span><span class="w"> </span><span class="s1">'America/New_York'</span>
<span class="n">Local</span><span class="w"> </span><span class="n">time</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">now</span><span class="p">:</span><span class="w"> </span><span class="n">Tue</span><span class="w"> </span><span class="n">Nov</span><span class="w"> </span><span class="mi">18</span><span class="w"> </span><span class="mi">09</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">16</span><span class="w"> </span><span class="n">EST</span><span class="w"> </span><span class="mf">2014.</span>
<span class="n">Universal</span><span class="w"> </span><span class="n">Time</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">now</span><span class="p">:</span><span class="w"> </span><span class="n">Tue</span><span class="w"> </span><span class="n">Nov</span><span class="w"> </span><span class="mi">18</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">16</span><span class="w"> </span><span class="n">UTC</span><span class="w"> </span><span class="mf">2014.</span>
<span class="n">Root</span><span class="w"> </span><span class="n">password</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="s1">'root'</span><span class="p">,</span><span class="w"> </span><span class="n">please</span><span class="w"> </span><span class="n">change</span><span class="w"> </span><span class="o">!</span>
<span class="s1">'debian'</span><span class="w"> </span><span class="n">template</span><span class="w"> </span><span class="n">installed</span>
<span class="s1">'mycontainer'</span><span class="w"> </span><span class="n">created</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="c1"># mount | grep mycontainer</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">rbd1</span><span class="w"> </span><span class="n">on</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">lxc</span><span class="o">/</span><span class="n">mycontainer</span><span class="o">/</span><span class="n">rootfs</span><span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="n">ext4</span><span class="w"> </span><span class="p">(</span><span class="n">rw</span><span class="p">,</span><span class="n">relatime</span><span class="p">,</span><span class="n">stripe</span><span class="o">=</span><span class="mi">4096</span><span class="p">,</span><span class="n">data</span><span class="o">=</span><span class="n">ordered</span><span class="p">)</span>
</code></pre></div>
<p>Diff file for lxc-create :</p>
<div class="highlight"><pre><span></span><code><span class="c1"># diff -u /usr/bin/lxc-create.orig /usr/bin/lxc-create</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="gd">--- /usr/bin/lxc-create.orig 2014-11-17 04:16:41.181942000 -0500</span>
<span class="gi">+++ /usr/bin/lxc-create 2014-11-17 04:35:27.225942000 -0500</span>
<span class="gu">@@ -24,6 +24,7 @@</span>
<span class="w"> </span> echo "usage: lxc-create -n <name> [-f configuration] [-t template] [-h] [fsopts] -- [template_options]"
<span class="w"> </span> echo " fsopts: -B none"
<span class="w"> </span> echo " fsopts: -B lvm [--lvname lvname] [--vgname vgname] [--fstype fstype] [--fssize fssize]"
<span class="gi">+ echo " fsopts: -B rbd [--pool poolname] [--rbd rbd] [--fstype fstype] [--fssize fssize]"</span>
<span class="w"> </span> echo " fsopts: -B btrfs"
<span class="w"> </span> echo " flag is not necessary, if possible btrfs support will be used"
<span class="w"> </span># echo " fsopts: -B union [--uniontype overlayfs]"
<span class="gu">@@ -64,7 +65,7 @@</span>
<span class="w"> </span>}
<span class="w"> </span>shortoptions='hn:f:t:B:'
<span class="gd">-longoptions='help,name:,config:,template:,backingstore:,fstype:,lvname:,vgname:,fssize:'</span>
<span class="gi">+longoptions='help,name:,config:,template:,backingstore:,fstype:,lvname:,vgname:,pool:,rbd:,fssize:'</span>
<span class="w"> </span>localstatedir=/var
<span class="w"> </span>lxc_path=${localstatedir}/lib/lxc
<span class="w"> </span>bindir=/usr/bin
<span class="gu">@@ -119,6 +120,16 @@</span>
<span class="w"> </span> vgname=$1
<span class="w"> </span> shift
<span class="w"> </span> ;;
<span class="gi">+ --pool)</span>
<span class="gi">+ shift</span>
<span class="gi">+ pool=$1</span>
<span class="gi">+ shift</span>
<span class="gi">+ ;;</span>
<span class="gi">+ --rbd)</span>
<span class="gi">+ shift</span>
<span class="gi">+ rbd=$1</span>
<span class="gi">+ shift</span>
<span class="gi">+ ;;</span>
<span class="w"> </span> --fstype)
<span class="w"> </span> shift
<span class="w"> </span> fstype=$1
<span class="gu">@@ -161,7 +172,7 @@</span>
<span class="w"> </span>fi
<span class="w"> </span>case "$backingstore" in
<span class="gd">- lvm|none|btrfs|_unset) :;;</span>
<span class="gi">+ lvm|rbd|none|btrfs|_unset) :;;</span>
<span class="w"> </span> *) echo "'$backingstore' is not known ('none', 'lvm', 'btrfs')"
<span class="w"> </span> usage
<span class="w"> </span> exit 1
<span class="gu">@@ -216,6 +227,13 @@</span>
<span class="w"> </span> echo "please delete it (using \"lvremove $rootdev\") and try again"
<span class="w"> </span> exit 1
<span class="w"> </span> fi
<span class="gi">+elif [ "$backingstore" = "rbd" ]; then</span>
<span class="gi">+ which rbd > /dev/null</span>
<span class="gi">+ if [ $? -ne 0 ]; then</span>
<span class="gi">+ echo "rbd command not found. Please install ceph-common package"</span>
<span class="gi">+ exit 1</span>
<span class="gi">+ fi</span>
<span class="gi">+ rootdev=/dev/rbd/$pool/$rbd</span>
<span class="w"> </span>elif [ "$backingstore" = "btrfs" ]; then
<span class="w"> </span> mkdir "$lxc_path/$lxc_name"
<span class="w"> </span> if ! out=$(btrfs subvolume create "$rootfs" 2>&1); then
<span class="gu">@@ -257,6 +275,14 @@</span>
<span class="w"> </span> mkfs -t $fstype $rootdev || exit 1
<span class="w"> </span> mount -t $fstype $rootdev $rootfs
<span class="w"> </span>fi
<span class="gi">+</span>
<span class="gi">+if [ $backingstore = "rbd" ]; then</span>
<span class="gi">+ [ -d "$rootfs" ] || mkdir $rootfs</span>
<span class="gi">+ rbd create $pool/$rbd --size=$fssize || exit 1</span>
<span class="gi">+ rbd map $pool/$rbd || exit 1</span>
<span class="gi">+ mkfs -t $fstype $rootdev || exit 1</span>
<span class="gi">+ mount -t $fstype $rootdev $rootfs</span>
<span class="gi">+fi</span>
<span class="w"> </span>if [ ! -z $lxc_template ]; then
</code></pre></div>
<p>If you want to make persistent after reboot, you must add rbd in /etc/ceph/rbdmap and add line in fstab.</p>OpenNebula 4.8 with Ceph Support on Debian Wheezy2014-11-14T16:57:10+01:002014-11-14T16:57:10+01:00Laurent Barbetag:cephnotes.ksperis.com,2014-11-14:/blog/2014/11/14/opennebula-4-dot-8-with-ceph-support-on-debian-wheezy/<p>A quick howto to install OpenNebula 4.8 with support for Ceph on Debian Wheezy.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>onedatastore<span class="w"> </span>show<span class="w"> </span>cephds
DATASTORE<span class="w"> </span><span class="m">101</span><span class="w"> </span>INFORMATION<span class="w"> </span>
ID<span class="w"> </span>:<span class="w"> </span><span class="m">101</span><span class="w"> </span>
NAME<span class="w"> </span>:<span class="w"> </span>cephds<span class="w"> </span>
USER<span class="w"> </span>:<span class="w"> </span>oneadmin<span class="w"> </span>
GROUP<span class="w"> </span>:<span class="w"> </span>oneadmin<span class="w"> </span>
CLUSTER<span class="w"> </span>:<span class="w"> </span>-<span class="w"> </span>
TYPE<span class="w"> </span>:<span class="w"> </span>IMAGE<span class="w"> </span>
DS_MAD<span class="w"> </span>:<span class="w"> </span>ceph<span class="w"> </span>
TM_MAD<span class="w"> </span>:<span class="w"> </span>ceph<span class="w"> </span>
BASE<span class="w"> </span>PATH<span class="w"> </span>:<span class="w"> </span>/var/lib/one//datastores/101
DISK_TYPE<span class="w"> </span>:<span class="w"> </span>RBD<span class="w"> </span>
</code></pre></div>
<!-- more -->
<h1>OpenNebula Installation</h1>
<h2>OpenNebula Frontend</h2>
<p>Install …</p><p>A quick howto to install OpenNebula 4.8 with support for Ceph on Debian Wheezy.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>onedatastore<span class="w"> </span>show<span class="w"> </span>cephds
DATASTORE<span class="w"> </span><span class="m">101</span><span class="w"> </span>INFORMATION<span class="w"> </span>
ID<span class="w"> </span>:<span class="w"> </span><span class="m">101</span><span class="w"> </span>
NAME<span class="w"> </span>:<span class="w"> </span>cephds<span class="w"> </span>
USER<span class="w"> </span>:<span class="w"> </span>oneadmin<span class="w"> </span>
GROUP<span class="w"> </span>:<span class="w"> </span>oneadmin<span class="w"> </span>
CLUSTER<span class="w"> </span>:<span class="w"> </span>-<span class="w"> </span>
TYPE<span class="w"> </span>:<span class="w"> </span>IMAGE<span class="w"> </span>
DS_MAD<span class="w"> </span>:<span class="w"> </span>ceph<span class="w"> </span>
TM_MAD<span class="w"> </span>:<span class="w"> </span>ceph<span class="w"> </span>
BASE<span class="w"> </span>PATH<span class="w"> </span>:<span class="w"> </span>/var/lib/one//datastores/101
DISK_TYPE<span class="w"> </span>:<span class="w"> </span>RBD<span class="w"> </span>
</code></pre></div>
<!-- more -->
<h1>OpenNebula Installation</h1>
<h2>OpenNebula Frontend</h2>
<p>Install OpenNebula repo</p>
<div class="highlight"><pre><span></span><code><span class="n">wget</span><span class="w"> </span><span class="o">-</span><span class="n">q</span><span class="w"> </span><span class="o">-</span><span class="n">O</span><span class="o">-</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">downloads</span><span class="o">.</span><span class="n">opennebula</span><span class="o">.</span><span class="n">org</span><span class="o">/</span><span class="n">repo</span><span class="o">/</span><span class="n">Ubuntu</span><span class="o">/</span><span class="n">repo</span><span class="o">.</span><span class="n">key</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">apt</span><span class="o">-</span><span class="n">key</span><span class="w"> </span><span class="n">add</span><span class="w"> </span><span class="o">-</span>
<span class="n">echo</span><span class="w"> </span><span class="s2">"deb http://downloads.opennebula.org/repo/4.8/Debian/7/ stable opennebula"</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">apt</span><span class="o">/</span><span class="n">sources</span><span class="o">.</span><span class="n">list</span><span class="o">.</span><span class="n">d</span><span class="o">/</span><span class="n">opennebula</span><span class="o">.</span><span class="n">list</span>
</code></pre></div>
<p>Download packages</p>
<div class="highlight"><pre><span></span><code>apt-get update
apt-get install opennebula opennebula-sunstone nfs-kernel-server
</code></pre></div>
<p>Configure and start the service</p>
<div class="highlight"><pre><span></span><code>sed -i -e 's/:host: 127.0.0.1/:host: 0.0.0.0/g' /etc/one/sunstone-server.conf
/etc/init.d/opennebula-sunstone restart
</code></pre></div>
<p>Export NFS</p>
<div class="highlight"><pre><span></span><code><span class="n">echo</span><span class="w"> </span><span class="s2">"/var/lib/one/ *(rw,sync,no_subtree_check,root_squash)"</span><span class="w"> </span><span class="o">>></span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">exports</span>
<span class="n">service</span><span class="w"> </span><span class="n">nfs</span><span class="o">-</span><span class="n">kernel</span><span class="o">-</span><span class="n">server</span><span class="w"> </span><span class="n">restart</span>
</code></pre></div>
<p>Configure SSH Public Key</p>
<div class="highlight"><pre><span></span><code>su - oneadmin
$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
$ cat << EOT > ~/.ssh/config
Host *
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
EOT
$ chmod 600 ~/.ssh/config
</code></pre></div>
<h2>OpenNebula Compute Nodes</h2>
<p>Install OpenNebula repo</p>
<div class="highlight"><pre><span></span><code><span class="n">wget</span><span class="w"> </span><span class="o">-</span><span class="n">q</span><span class="w"> </span><span class="o">-</span><span class="n">O</span><span class="o">-</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">downloads</span><span class="o">.</span><span class="n">opennebula</span><span class="o">.</span><span class="n">org</span><span class="o">/</span><span class="n">repo</span><span class="o">/</span><span class="n">Ubuntu</span><span class="o">/</span><span class="n">repo</span><span class="o">.</span><span class="n">key</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">apt</span><span class="o">-</span><span class="n">key</span><span class="w"> </span><span class="n">add</span><span class="w"> </span><span class="o">-</span>
<span class="n">echo</span><span class="w"> </span><span class="s2">"deb http://downloads.opennebula.org/repo/4.8/Debian/7/ stable opennebula"</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">apt</span><span class="o">/</span><span class="n">sources</span><span class="o">.</span><span class="n">list</span><span class="o">.</span><span class="n">d</span><span class="o">/</span><span class="n">opennebula</span><span class="o">.</span><span class="n">list</span>
</code></pre></div>
<p>Download packages</p>
<div class="highlight"><pre><span></span><code>apt-get update
apt-get install opennebula-node nfs-common bridge-utils
</code></pre></div>
<p>Network config</p>
<div class="highlight"><pre><span></span><code>vim /etc/network/interfaces
...
bridge_ports eth0
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off
</code></pre></div>
<p>Fstab</p>
<div class="highlight"><pre><span></span><code><span class="n">vim</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">fstab</span>
<span class="mf">10.2</span><span class="o">.</span><span class="mf">0.130</span><span class="p">:</span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">one</span><span class="o">/</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">one</span><span class="o">/</span><span class="w"> </span><span class="n">nfs</span><span class="w"> </span><span class="n">soft</span><span class="p">,</span><span class="n">intr</span><span class="p">,</span><span class="n">rsize</span><span class="o">=</span><span class="mi">8192</span><span class="p">,</span><span class="n">wsize</span><span class="o">=</span><span class="mi">8192</span><span class="p">,</span><span class="n">noauto</span>
<span class="n">mount</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">one</span><span class="o">/</span>
</code></pre></div>
<p>Qemu config</p>
<div class="highlight"><pre><span></span><code>cat << EOT > /etc/libvirt/qemu.conf
user = "oneadmin"
group = "oneadmin"
dynamic_ownership = 0
EOT
</code></pre></div>
<h1>Ceph configuration</h1>
<p>I suppose that you have already a Ceph cluster running.</p>
<p>Now create a pool "one" for Opennebula and create auth key.</p>
<div class="highlight"><pre><span></span><code>ceph osd pool create one 128 128
ceph osd pool set one crush_ruleset 2
ceph auth get-or-create client.one mon 'allow r' osd 'allow rwx pool=one'
[client.one]
key = AQCfTjVUeOPqIhAAiCAiBIgYd85fuMFT0dXVpA==
</code></pre></div>
<h2>Add Ceph firefly for support rbd on libvirt + qemu</h2>
<p>Before, you need compile rbd support for libvirt and qemu. You can have a look to this post : http://cephnotes.ksperis.com/blog/2013/09/12/using-ceph-rbd-with-libvirt-on-debian-wheezy (If you do not want to compile qemu you can directluy download those packages here : http://ksperis.com/files/qemu-kvm_1.1.2+dfsg-6_amd64.deb, http://ksperis.com/files/qemu-kvm_1.1.2+dfsg-6_amd64.deb)</p>
<p>On each node, add Ceph firefly repo, and reinstall libvirt and qemu packages</p>
<div class="highlight"><pre><span></span><code><span class="nt">apt-get</span><span class="w"> </span><span class="nt">install</span><span class="w"> </span><span class="nt">lsb-release</span>
<span class="nt">wget</span><span class="w"> </span><span class="nt">-q</span><span class="w"> </span><span class="nt">-O-</span><span class="w"> </span><span class="s1">'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc'</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nt">sudo</span><span class="w"> </span><span class="nt">apt-key</span><span class="w"> </span><span class="nt">add</span><span class="w"> </span><span class="nt">-</span>
<span class="nt">echo</span><span class="w"> </span><span class="nt">deb</span><span class="w"> </span><span class="nt">http</span><span class="o">://</span><span class="nt">ceph</span><span class="p">.</span><span class="nc">com</span><span class="o">/</span><span class="nt">debian-firefly</span><span class="o">/</span><span class="w"> </span><span class="o">$(</span><span class="nt">lsb_release</span><span class="w"> </span><span class="nt">-sc</span><span class="o">)</span><span class="w"> </span><span class="nt">main</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nt">sudo</span><span class="w"> </span><span class="nt">tee</span><span class="w"> </span><span class="o">/</span><span class="nt">etc</span><span class="o">/</span><span class="nt">apt</span><span class="o">/</span><span class="nt">sources</span><span class="p">.</span><span class="nc">list</span><span class="p">.</span><span class="nc">d</span><span class="o">/</span><span class="nt">ceph</span><span class="p">.</span><span class="nc">list</span>
<span class="nt">apt-get</span><span class="w"> </span><span class="nt">update</span>
<span class="nt">apt-get</span><span class="w"> </span><span class="nt">install</span><span class="w"> </span><span class="nt">librados2</span><span class="w"> </span><span class="nt">librbd1</span>
<span class="nt">dpkg</span><span class="w"> </span><span class="nt">-i</span><span class="w"> </span><span class="nt">qemu-kvm_1</span><span class="p">.</span><span class="nc">1</span><span class="p">.</span><span class="nc">2</span><span class="o">+</span><span class="nt">dfsg-6</span><span class="o">+</span><span class="nt">deb7u4_amd64</span><span class="p">.</span><span class="nc">deb</span><span class="w"> </span><span class="nt">qemu-utils_1</span><span class="p">.</span><span class="nc">1</span><span class="p">.</span><span class="nc">2</span><span class="o">+</span><span class="nt">dfsg-6a</span><span class="o">+</span><span class="nt">deb7u4_amd64</span><span class="p">.</span><span class="nc">deb</span>
</code></pre></div>
<p>Create the secret for libvirt :</p>
<div class="highlight"><pre><span></span><code>echo<span class="w"> </span>"
<span class="nt"><secret</span><span class="w"> </span><span class="na">ephemeral=</span><span class="s">'no'</span><span class="w"> </span><span class="na">private=</span><span class="s">'no'</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><usage</span><span class="w"> </span><span class="na">type=</span><span class="s">'ceph'</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><name></span>client.one<span class="w"> </span>secret<span class="nt"></name></span>
<span class="w"> </span><span class="nt"></usage></span>
<span class="nt"></secret></span>"<span class="w"> </span>><span class="w"> </span>secret.xml
service<span class="w"> </span>libvirt<span class="w"> </span>reload
</code></pre></div>
<p>Add ceph datastore on OpenNebula</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>su<span class="w"> </span>-<span class="w"> </span>oneadmin
$<span class="w"> </span>vim<span class="w"> </span>ceph.one
<span class="nv">NAME</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>cephds
<span class="nv">DS_MAD</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>ceph
<span class="nv">TM_MAD</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>ceph
<span class="nv">DISK_TYPE</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>RBD
<span class="nv">POOL_NAME</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>one
<span class="nv">BRIDGE_LIST</span><span class="w"> </span><span class="o">=</span><span class="s2">"onenode1 onenode2 onenode3"</span>
<span class="nv">CEPH_HOST</span><span class="w"> </span><span class="o">=</span><span class="s2">"192.168.0.1:6789 192.168.0.2:6789 192.168.0.3:6789"</span>
<span class="nv">CEPH_SECRET</span><span class="w"> </span><span class="o">=</span><span class="s2">"26a8b4d7-eb24-bf85-396d-fbf0x252e402"</span>
<span class="nv">CEPH_USER</span><span class="w"> </span><span class="o">=</span><span class="s2">"one"</span>
$<span class="w"> </span>onedatastore<span class="w"> </span>create<span class="w"> </span>ceph.one
ID:<span class="w"> </span><span class="m">101</span>
</code></pre></div>
<p>Example config :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>onedatastore<span class="w"> </span>show<span class="w"> </span>cephds
DATASTORE<span class="w"> </span><span class="m">101</span><span class="w"> </span>INFORMATION<span class="w"> </span>
ID<span class="w"> </span>:<span class="w"> </span><span class="m">101</span><span class="w"> </span>
NAME<span class="w"> </span>:<span class="w"> </span>cephds<span class="w"> </span>
USER<span class="w"> </span>:<span class="w"> </span>oneadmin<span class="w"> </span>
GROUP<span class="w"> </span>:<span class="w"> </span>oneadmin<span class="w"> </span>
CLUSTER<span class="w"> </span>:<span class="w"> </span>-<span class="w"> </span>
TYPE<span class="w"> </span>:<span class="w"> </span>IMAGE<span class="w"> </span>
DS_MAD<span class="w"> </span>:<span class="w"> </span>ceph<span class="w"> </span>
TM_MAD<span class="w"> </span>:<span class="w"> </span>ceph<span class="w"> </span>
BASE<span class="w"> </span>PATH<span class="w"> </span>:<span class="w"> </span>/var/lib/one//datastores/101
DISK_TYPE<span class="w"> </span>:<span class="w"> </span>RBD
DATASTORE<span class="w"> </span>CAPACITY<span class="w"> </span>
TOTAL:<span class="w"> </span>:<span class="w"> </span><span class="m">15</span>.9T<span class="w"> </span>
FREE:<span class="w"> </span>:<span class="w"> </span><span class="m">13</span>.4T<span class="w"> </span>
USED:<span class="w"> </span>:<span class="w"> </span><span class="m">2</span>.5T<span class="w"> </span>
LIMIT:<span class="w"> </span>:<span class="w"> </span>-
PERMISSIONS<span class="w"> </span>
OWNER<span class="w"> </span>:<span class="w"> </span>um-<span class="w"> </span>
GROUP<span class="w"> </span>:<span class="w"> </span>u--<span class="w"> </span>
OTHER<span class="w"> </span>:<span class="w"> </span>---
DATASTORE<span class="w"> </span>TEMPLATE<span class="w"> </span>
<span class="nv">BASE_PATH</span><span class="o">=</span><span class="s2">"/var/lib/one//datastores/"</span>
<span class="nv">BRIDGE_LIST</span><span class="o">=</span><span class="s2">"onenode1 onenode2 onenode3"</span>
<span class="nv">CEPH_HOST</span><span class="o">=</span><span class="s2">"192.168.0.1:6789 192.168.0.2:6789 192.168.0.3:6789"</span>
<span class="nv">CEPH_SECRET</span><span class="o">=</span><span class="s2">"26a8b4d7-eb24-bf85-396d-fbf0x252e402"</span>
<span class="nv">CEPH_USER</span><span class="o">=</span><span class="s2">"one"</span>
<span class="nv">CLONE_TARGET</span><span class="o">=</span><span class="s2">"SELF"</span>
<span class="nv">DISK_TYPE</span><span class="o">=</span><span class="s2">"RBD"</span>
<span class="nv">DS_MAD</span><span class="o">=</span><span class="s2">"ceph"</span>
<span class="nv">LN_TARGET</span><span class="o">=</span><span class="s2">"NONE"</span>
<span class="nv">POOL_NAME</span><span class="o">=</span><span class="s2">"one"</span>
<span class="nv">TM_MAD</span><span class="o">=</span><span class="s2">"ceph"</span>
<span class="nv">TYPE</span><span class="o">=</span><span class="s2">"IMAGE_DS"</span>
IMAGES<span class="w"> </span>
<span class="m">9</span><span class="w"> </span>
<span class="m">10</span><span class="w"> </span>
<span class="m">11</span><span class="w"> </span>
<span class="m">12</span><span class="w"> </span>
<span class="m">16</span><span class="w"> </span>
<span class="m">21</span><span class="w"> </span>
<span class="m">24</span><span class="w"> </span>
<span class="m">27</span><span class="w"> </span>
<span class="m">28</span><span class="w"> </span>
<span class="m">29</span><span class="w"> </span>
<span class="m">31</span><span class="w"> </span>
<span class="m">32</span>
</code></pre></div>
<p>Try import image in new datastore :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>oneimage<span class="w"> </span>create
<span class="w"> </span>--name<span class="w"> </span><span class="s2">"CentOS-6.5_x86_64"</span>
<span class="w"> </span>--path<span class="w"> </span><span class="s2">"http://appliances.c12g.com/CentOS-6.5/centos6.5.qcow2.gz"</span>
<span class="w"> </span>--driver<span class="w"> </span>qcow2
<span class="w"> </span>--datastore<span class="w"> </span>cephds
</code></pre></div>
<p>Datastore view in Sunstone :</p>
<p><img alt="Opennebula 4.8 Ceph" src="https://cephnotes.ksperis.com/images/img018.png"></p>
<p>More details here :</p>
<p>http://ceph.com/community/ceph-support-in-opennebula-4-0/</p>
<p>http://archives.opennebula.org/documentation:archives:rel4.0:ceph_ds</p>Remove pool without name2014-10-29T14:46:38+01:002014-10-29T14:46:38+01:00Laurent Barbetag:cephnotes.ksperis.com,2014-10-29:/blog/2014/10/29/remove-pool-without-name/<p>For exemple :</p>
<div class="highlight"><pre><span></span><code><span class="c1"># rados lspools</span>
<span class="nf">data</span>
<span class="nf">metadata</span>
<span class="nf">rbd</span>
<span class="w"> </span><span class="err"><----</span><span class="w"> </span><span class="err">?????</span>
<span class="na">.eu.rgw.root</span>
<span class="na">.eu-west-1.domain.rgw</span>
<span class="na">.eu-west-1.rgw.root</span>
<span class="na">.eu-west-1.rgw.control</span>
<span class="na">.eu-west-1.rgw.gc</span>
<span class="na">.eu-west-1.rgw.buckets.index</span>
<span class="na">.eu-west-1.rgw.buckets</span>
<span class="na">.eu-west-1.log</span>
<span class="c1"># ceph osd dump | grep "pool 4 "</span>
<span class="nf">pool</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="err">''</span><span class="w"> </span><span class="no">replicated</span><span class="w"> </span><span class="no">size</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="no">min_size</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="no">crush_ruleset</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="no">object_hash</span><span class="w"> </span><span class="no">rjenkins</span><span class="w"> </span><span class="no">pg_num</span><span class="w"> </span><span class="mi">8 …</span></code></pre></div><p>For exemple :</p>
<div class="highlight"><pre><span></span><code><span class="c1"># rados lspools</span>
<span class="nf">data</span>
<span class="nf">metadata</span>
<span class="nf">rbd</span>
<span class="w"> </span><span class="err"><----</span><span class="w"> </span><span class="err">?????</span>
<span class="na">.eu.rgw.root</span>
<span class="na">.eu-west-1.domain.rgw</span>
<span class="na">.eu-west-1.rgw.root</span>
<span class="na">.eu-west-1.rgw.control</span>
<span class="na">.eu-west-1.rgw.gc</span>
<span class="na">.eu-west-1.rgw.buckets.index</span>
<span class="na">.eu-west-1.rgw.buckets</span>
<span class="na">.eu-west-1.log</span>
<span class="c1"># ceph osd dump | grep "pool 4 "</span>
<span class="nf">pool</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="err">''</span><span class="w"> </span><span class="no">replicated</span><span class="w"> </span><span class="no">size</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="no">min_size</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="no">crush_ruleset</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="no">object_hash</span><span class="w"> </span><span class="no">rjenkins</span><span class="w"> </span><span class="no">pg_num</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="no">pgp_num</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="no">last_change</span><span class="w"> </span><span class="mi">1668</span><span class="w"> </span><span class="no">stripe_width</span><span class="w"> </span><span class="mi">0</span>
<span class="c1"># rados rmpool "" "" --yes-i-really-really-mean-it</span>
<span class="nf">successfully</span><span class="w"> </span><span class="no">deleted</span><span class="w"> </span><span class="no">pool</span>
</code></pre></div>Ceph Node.js bindings for librados2014-08-25T09:57:01+02:002014-08-25T09:57:01+02:00Laurent Barbetag:cephnotes.ksperis.com,2014-08-25:/blog/2014/08/25/ceph-node-dot-js-bindings-for-librados/<div class="highlight"><pre><span></span><code><span class="kd">var</span><span class="w"> </span><span class="nx">cluster</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">rados</span><span class="p">.</span><span class="nx">Rados</span><span class="p">(</span><span class="w"> </span><span class="s2">"ceph"</span><span class="p">,</span><span class="w"> </span><span class="s2">"client.admin"</span><span class="p">,</span><span class="w"> </span><span class="s2">"/etc/ceph/ceph.conf"</span><span class="p">);</span>
<span class="nx">cluster</span><span class="p">.</span><span class="nx">connect</span><span class="p">();</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">ioctx</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">rados</span><span class="p">.</span><span class="nx">Ioctx</span><span class="p">(</span><span class="nx">cluster</span><span class="p">,</span><span class="w"> </span><span class="s2">"data"</span><span class="p">);</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">aio_write</span><span class="p">(</span><span class="s2">"testfile2"</span><span class="p">,</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">Buffer</span><span class="p">(</span><span class="s2">"1234567879ABCD"</span><span class="p">),</span><span class="w"> </span><span class="mf">14</span><span class="p">,</span><span class="w"> </span><span class="mf">0</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="nx">err</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">...</span>
</code></pre></div>
<!-- more -->
<p>To my knowledge, there is not yet any wrapper Node.js for librados. (I guess …</p><div class="highlight"><pre><span></span><code><span class="kd">var</span><span class="w"> </span><span class="nx">cluster</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">rados</span><span class="p">.</span><span class="nx">Rados</span><span class="p">(</span><span class="w"> </span><span class="s2">"ceph"</span><span class="p">,</span><span class="w"> </span><span class="s2">"client.admin"</span><span class="p">,</span><span class="w"> </span><span class="s2">"/etc/ceph/ceph.conf"</span><span class="p">);</span>
<span class="nx">cluster</span><span class="p">.</span><span class="nx">connect</span><span class="p">();</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">ioctx</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">rados</span><span class="p">.</span><span class="nx">Ioctx</span><span class="p">(</span><span class="nx">cluster</span><span class="p">,</span><span class="w"> </span><span class="s2">"data"</span><span class="p">);</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">aio_write</span><span class="p">(</span><span class="s2">"testfile2"</span><span class="p">,</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">Buffer</span><span class="p">(</span><span class="s2">"1234567879ABCD"</span><span class="p">),</span><span class="w"> </span><span class="mf">14</span><span class="p">,</span><span class="w"> </span><span class="mf">0</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="nx">err</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">...</span>
</code></pre></div>
<!-- more -->
<p>To my knowledge, there is not yet any wrapper Node.js for librados. (I guess that Alexandre has not found again the code it had begun. http://tracker.ceph.com/issues/4230). So, I started to make a draft of a plugin (when I have some time). For now I am not using it, but it allows me to discover Node. If people are interested this is here :</p>
<p>https://github.com/ksperis/node-rados</p>
<p>(suggestions are welcome, especially on the Error Handling, the use of libuv, buffers / strings, and everything else...)</p>
<p>All is not yet implemented, but the basic functions are present.</p>
<p>For example (example.js file in repo):</p>
<div class="highlight"><pre><span></span><code><span class="kd">var</span><span class="w"> </span><span class="nx">rados</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">require</span><span class="p">(</span><span class="s1">'./build/Release/rados'</span><span class="p">);</span>
<span class="c1">// EXEMPLE FILE</span>
<span class="c1">//==================================</span>
<span class="c1">// Connect to cluster</span>
<span class="c1">//==================================</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">cluster</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">rados</span><span class="p">.</span><span class="nx">Rados</span><span class="p">(</span><span class="w"> </span><span class="s2">"ceph"</span><span class="p">,</span><span class="w"> </span><span class="s2">"client.admin"</span><span class="p">,</span><span class="w"> </span><span class="s2">"/etc/ceph/ceph.conf"</span><span class="p">);</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">cluster</span><span class="p">.</span><span class="nx">connect</span><span class="p">();</span>
<span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// On connection error</span>
<span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">"Error "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">err</span><span class="p">);</span>
<span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="nx">err</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// Print cluster FSID, pools</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="w"> </span><span class="s2">"fsid : "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">cluster</span><span class="p">.</span><span class="nx">get_fsid</span><span class="p">()</span><span class="w"> </span><span class="p">);</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="w"> </span><span class="s2">"ls pools : "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">cluster</span><span class="p">.</span><span class="nx">pool_list</span><span class="p">()</span><span class="w"> </span><span class="p">);</span>
<span class="c1">//==================================</span>
<span class="c1">// Create IOCTX</span>
<span class="c1">//==================================</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">ioctx</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">rados</span><span class="p">.</span><span class="nx">Ioctx</span><span class="p">(</span><span class="nx">cluster</span><span class="p">,</span><span class="w"> </span><span class="s2">"data"</span><span class="p">);</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">" --- RUN Sync Write / Read --- "</span><span class="p">);</span>
<span class="c1">// Sync write_full</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">write_full</span><span class="p">(</span><span class="s2">"testfile1"</span><span class="p">,</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">Buffer</span><span class="p">(</span><span class="s2">"01234567ABCDEF"</span><span class="p">));</span>
<span class="c1">// Sync Read</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="w"> </span><span class="s2">"Read data : "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span>
<span class="w"> </span><span class="nx">ioctx</span><span class="p">.</span><span class="nx">read</span><span class="p">(</span><span class="s2">"testfile1"</span><span class="p">,</span><span class="w"> </span><span class="nx">ioctx</span><span class="p">.</span><span class="nx">stat</span><span class="p">(</span><span class="s2">"testfile1"</span><span class="p">).</span><span class="nx">psize</span><span class="p">).</span><span class="nx">toString</span><span class="p">()</span><span class="w"> </span><span class="p">);</span>
<span class="c1">// Remove</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">remove</span><span class="p">(</span><span class="s2">"testfile1"</span><span class="p">);</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">" --- RUN ASync Write / Read --- "</span><span class="p">);</span>
<span class="c1">// ASync write_full</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">aio_write</span><span class="p">(</span><span class="s2">"testfile2"</span><span class="p">,</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">Buffer</span><span class="p">(</span><span class="s2">"1234567879ABCD"</span><span class="p">),</span><span class="w"> </span><span class="mf">14</span><span class="p">,</span><span class="w"> </span><span class="mf">0</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="nx">err</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="nx">ioctx</span><span class="p">.</span><span class="nx">aio_read</span><span class="p">(</span><span class="s2">"testfile2"</span><span class="p">,</span><span class="w"> </span><span class="mf">14</span><span class="p">,</span><span class="w"> </span><span class="mf">0</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="nx">data</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="nx">err</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">"[async callback] data = "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">data</span><span class="p">.</span><span class="nx">toString</span><span class="p">());</span>
<span class="w"> </span><span class="p">});</span>
<span class="p">});</span>
<span class="c1">//==================================</span>
<span class="c1">// Read / Write Attributes</span>
<span class="c1">//==================================</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">" --- RUN Attributes Write / Read --- "</span><span class="p">);</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">setxattr</span><span class="p">(</span><span class="s2">"testfile3"</span><span class="p">,</span><span class="w"> </span><span class="s2">"attr1"</span><span class="p">,</span><span class="w"> </span><span class="s2">"first attr"</span><span class="p">);</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">setxattr</span><span class="p">(</span><span class="s2">"testfile3"</span><span class="p">,</span><span class="w"> </span><span class="s2">"attr2"</span><span class="p">,</span><span class="w"> </span><span class="s2">"second attr"</span><span class="p">);</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">setxattr</span><span class="p">(</span><span class="s2">"testfile3"</span><span class="p">,</span><span class="w"> </span><span class="s2">"attr3"</span><span class="p">,</span><span class="w"> </span><span class="s2">"last attr value"</span><span class="p">);</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">attrs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">ioctx</span><span class="p">.</span><span class="nx">getxattrs</span><span class="p">(</span><span class="s2">"testfile3"</span><span class="p">);</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">"testfile3 xattr = %j"</span><span class="p">,</span><span class="w"> </span><span class="nx">attrs</span><span class="p">);</span>
<span class="c1">// destroy ioctx and close cluster after aio_flush</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">aio_flush_async</span><span class="p">(</span><span class="kd">function</span><span class="w"> </span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">ioctx</span><span class="p">.</span><span class="nx">destroy</span><span class="p">();</span>
<span class="w"> </span><span class="nx">cluster</span><span class="p">.</span><span class="nx">shutdown</span><span class="p">();</span>
<span class="p">});</span>
<span class="nx">process</span><span class="p">.</span><span class="nx">exit</span><span class="p">(</span><span class="nx">code</span><span class="o">=</span><span class="mf">0</span><span class="p">)</span>
<span class="c1">// OTHER EXEMPLES</span>
<span class="c1">// Read Sync file in chunks</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">file</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"testfile"</span><span class="p">;</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">fileSize</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">ioctx</span><span class="p">.</span><span class="nx">stat</span><span class="p">(</span><span class="nx">file</span><span class="p">).</span><span class="nx">psize</span><span class="p">,</span>
<span class="w"> </span><span class="nx">chunkSize</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mf">512</span><span class="p">,</span>
<span class="w"> </span><span class="nx">bytesRead</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mf">0</span><span class="p">;</span>
<span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="nx">bytesRead</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="nx">fileSize</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">((</span><span class="nx">bytesRead</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">chunkSize</span><span class="p">)</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="nx">fileSize</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">chunkSize</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="nx">fileSize</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nx">bytesRead</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">buffer</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">ioctx</span><span class="p">.</span><span class="nx">read</span><span class="p">(</span><span class="nx">file</span><span class="p">,</span><span class="w"> </span><span class="nx">chunkSize</span><span class="p">,</span><span class="w"> </span><span class="nx">bytesRead</span><span class="p">);</span>
<span class="w"> </span><span class="nx">bytesRead</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="nx">chunkSize</span><span class="p">;</span>
<span class="w"> </span><span class="nx">process</span><span class="p">.</span><span class="nx">stdout</span><span class="p">.</span><span class="nx">write</span><span class="p">(</span><span class="nx">buffer</span><span class="p">.</span><span class="nx">toString</span><span class="p">());</span>
<span class="p">}</span>
<span class="c1">// Read Async file in chunks</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">file</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"testfile"</span><span class="p">;</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">fileSize</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">ioctx</span><span class="p">.</span><span class="nx">stat</span><span class="p">(</span><span class="nx">file</span><span class="p">).</span><span class="nx">psize</span><span class="p">,</span>
<span class="w"> </span><span class="nx">chunkSize</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mf">512</span><span class="p">,</span>
<span class="w"> </span><span class="nx">bytesRead</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mf">0</span><span class="p">;</span>
<span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="nx">bytesRead</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="nx">fileSize</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">((</span><span class="nx">bytesRead</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">chunkSize</span><span class="p">)</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="nx">fileSize</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">chunkSize</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="nx">fileSize</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nx">bytesRead</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="nx">ioctx</span><span class="p">.</span><span class="nx">aio_read</span><span class="p">(</span><span class="nx">file</span><span class="p">,</span><span class="w"> </span><span class="nx">chunkSize</span><span class="p">,</span><span class="w"> </span><span class="nx">bytesRead</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="nx">data</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">process</span><span class="p">.</span><span class="nx">stdout</span><span class="p">.</span><span class="nx">write</span><span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">toString</span><span class="p">());</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="nx">bytesRead</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="nx">chunkSize</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// Use snapshot</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">write_full</span><span class="p">(</span><span class="s2">"testfile10"</span><span class="p">,</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">Buffer</span><span class="p">(</span><span class="s2">"version1"</span><span class="p">));</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">snap_create</span><span class="p">(</span><span class="s2">"snaptest1"</span><span class="p">);</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">write_full</span><span class="p">(</span><span class="s2">"testfile10"</span><span class="p">,</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">Buffer</span><span class="p">(</span><span class="s2">"version2"</span><span class="p">));</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">snap_create</span><span class="p">(</span><span class="s2">"snaptest2"</span><span class="p">);</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">write_full</span><span class="p">(</span><span class="s2">"testfile10"</span><span class="p">,</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">Buffer</span><span class="p">(</span><span class="s2">"version3"</span><span class="p">));</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">snap_create</span><span class="p">(</span><span class="s2">"snaptest3"</span><span class="p">);</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">snap_rollback</span><span class="p">(</span><span class="s2">"testfile10"</span><span class="p">,</span><span class="w"> </span><span class="s2">"snaptest2"</span><span class="p">);</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">ioctx</span><span class="p">.</span><span class="nx">read</span><span class="p">(</span><span class="s2">"testfile10"</span><span class="p">).</span><span class="nx">toString</span><span class="p">());</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">snap_remove</span><span class="p">(</span><span class="s2">"snaptest1"</span><span class="p">);</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">snap_remove</span><span class="p">(</span><span class="s2">"snaptest2"</span><span class="p">);</span>
<span class="nx">ioctx</span><span class="p">.</span><span class="nx">snap_remove</span><span class="p">(</span><span class="s2">"snaptest3"</span><span class="p">);</span>
</code></pre></div>Ceph Primary Affinity2014-08-20T09:36:13+02:002014-08-20T09:36:13+02:00Laurent Barbetag:cephnotes.ksperis.com,2014-08-20:/blog/2014/08/20/ceph-primary-affinity/<p>This option allows you to answer a fairly constant worry in the case of heterogeneous cluster. Indeed, all the discs do not have the same performance or not the same ratio performance / size.
With this option it is possible to reduce the load on a disk without reducing the amount …</p><p>This option allows you to answer a fairly constant worry in the case of heterogeneous cluster. Indeed, all the discs do not have the same performance or not the same ratio performance / size.
With this option it is possible to reduce the load on a disk without reducing the amount of data it contains.
Furthermore, the option is easy to modify because it does not result in data migration. Only preference between primary / secondary will be modified and propagated to clients.</p>
<!-- more -->
<p>Before playing with cluster options and tune crushmap, remember to verify that your client is compatible with those options.
{% blockquote %} You must enable 'mon osd allow primary affinity = true' on the mons before you can adjust primary-affinity. note that older clients will no longer be able to communicate with the cluster. {% endblockquote %}
( For client kernel module you can have a look to http://cephnotes.ksperis.com/blog/2014/01/21/feature-set-mismatch-error-on-ceph-kernel-client. )</p>
<p>Look if the monitor has the primary affinity option:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># ceph --admin-daemon /var/run/ceph/ceph-mon.*.asok config show | grep 'primary_affinity'</span>
<span class="w"> </span><span class="s2">"mon_osd_allow_primary_affinity"</span><span class="p">:</span><span class="w"> </span><span class="s2">"false"</span><span class="p">,</span>
</code></pre></div>
<p>Edit ceph.conf dans add in section [mon]:</p>
<div class="highlight"><pre><span></span><code>mon osd allow primary affinity = true
</code></pre></div>
<p>Reload mon and test.
We look how many pg is primary on osd.0, and how many is secondary :</p>
<div class="highlight"><pre><span></span><code># ceph pg dump | grep active+clean | egrep "\[0," | wc -l
100
# ceph pg dump | grep active+clean | egrep ",0\]" | wc -l
80
</code></pre></div>
<p>Try to change primary affinity :</p>
<div class="highlight"><pre><span></span><code># ceph osd primary-affinity osd.0 0.5
set osd.0 primary-affinity to 0.5 (8327682)
# ceph pg dump | grep active+clean | egrep "\[0," | wc -l
48
# ceph pg dump | grep active+clean | egrep ",0\]" | wc -l
132
# ceph osd primary-affinity osd.0 0
set osd.0 primary-affinity to 0 (802)
# ceph pg dump | grep active+clean | egrep "\[0," | wc -l
0
# ceph pg dump | grep active+clean | egrep ",0\]" | wc -l
180
</code></pre></div>
<p>Now there will be no more reading on this OSD.</p>Scallable thumbnaling service with Thumbor and Ceph2014-08-18T10:02:40+02:002014-08-18T10:02:40+02:00Laurent Barbetag:cephnotes.ksperis.com,2014-08-18:/blog/2014/08/18/scallable-thumbnaling-service-with-thumbor-and-ceph/<p>An example of using the python-ceph library for thumbnailing service.</p>
<!-- more -->
<p>Thumbor is an opensource tool for thumbnail generation developed by globo.</p>
<p>The tool allows to make a number of operations (crop, resize, filters ...) directly via the URL. It also uses face detection algos or points of interest to enhance Crop …</p><p>An example of using the python-ceph library for thumbnailing service.</p>
<!-- more -->
<p>Thumbor is an opensource tool for thumbnail generation developed by globo.</p>
<p>The tool allows to make a number of operations (crop, resize, filters ...) directly via the URL. It also uses face detection algos or points of interest to enhance Crop or Auto Resize. Original images are imported via an url or directly using Restful API.</p>
<p>For interfacing with Ceph, two methods are possible:
- Use the S3 API (via Rados GW)
- Use librados and python wrapper for librados.</p>
<p>In this example, it is the second case that I present because it optimizes latency between the application and the storage cluster.</p>
<p><img alt="Scallable thumbnaling" src="https://cephnotes.ksperis.com/images/img015.png"></p>
<h2>Installation :</h2>
<p>Thumbor installation :</p>
<div class="highlight"><pre><span></span><code>apt-get install libjpeg-dev libpng-dev libtiff-dev libjasper-dev libcurl4-gnutls-dev libwebp-dev
apt-get install python-dev python-pycurl python-opencv
apt-get install python-pip
pip install thumbor
</code></pre></div>
<p>thumbor_ceph storage plugin installation :</p>
<div class="highlight"><pre><span></span><code>apt-get install python-ceph
pip install thumbor_ceph
</code></pre></div>
<h2>Configuration :</h2>
<p>Generate initial thumbor config file :</p>
<div class="highlight"><pre><span></span><code>thumbor-config > /etc/thumbor.conf
</code></pre></div>
<p>For more configuration options:
https://github.com/thumbor/thumbor/wiki/Configuration</p>
<p>Populate /etc/ceph/ with ceph.conf and admin keyring</p>
<p>Create a ceph pool for thumbor :</p>
<div class="highlight"><pre><span></span><code>ceph osd pool create thumbor 128 128
</code></pre></div>
<p>Section File Storage in /etc/thumbor.conf</p>
<div class="highlight"><pre><span></span><code><span class="gu">##</span>############################### File Storage #################################
STORAGE = 'thumbor_ceph.storages.ceph_storage'
CEPH_STORAGE_POOL = 'thumbor'
</code></pre></div>
<p>Section Upload</p>
<div class="highlight"><pre><span></span><code><span class="c1">#################################### Upload ####################################</span>
<span class="n">UPLOAD_PHOTO_STORAGE</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'thumbor_ceph.storages.ceph_storage'</span>
</code></pre></div>
<p>Section Result Storage</p>
<div class="highlight"><pre><span></span><code><span class="gu">##</span>############################## Result Storage ################################
RESULT_STORAGE = 'thumbor_ceph.result_storages.ceph_storage'
CEPH_RESULT_STORAGE_POOL = 'thumbor'
</code></pre></div>
<h2>Launch thumbor</h2>
<div class="highlight"><pre><span></span><code># thumbor -l debug
2014-05-27 10:27:24 thumbor:DEBUG INIT RADOS Storage (ID:c3eb3343-d06c-438f-ae49-cf998468824d)
2014-05-27 10:27:24 thumbor:DEBUG INIT RADOS Result Storage (ID:c3eb3343-d06c-438f-ae49-cf998468824d)
2014-05-27 10:27:24 root:DEBUG thumbor running at 0.0.0.0:8888
</code></pre></div>
<h2>Usage</h2>
<p>Now you can easily resize ceph logo <img alt="Ceph logo" src="https://cephnotes.ksperis.com/images/img016.png"> from this images <img alt="Ceph logo" src="https://cephnotes.ksperis.com/images/img017.png"> with this url :</p>
<div class="highlight"><pre><span></span><code>http://localhost:8888/unsafe/0x0:36x36/http://ceph.com/wp-content/themes/ceph/images/logo.png
</code></pre></div>
<p>https://github.com/thumbor/thumbor/wiki/Usage</p>
<h2>More</h2>
<ul>
<li>
<p>https://github.com/thumbor/thumbor</p>
</li>
<li>
<p>http://www.dadoune.com/blog/best-thumbnailing-solution-set-up-thumbor-on-aws/</p>
</li>
<li>http://99designs.com/tech-blog/blog/2013/07/01/thumbnailing-with-thumbor/</li>
<li>http://tech.yipit.com/2013/01/03/how-yipit-scales-thumbnailing-with-thumbor-and-cloudfront/</li>
<li>http://heynemann.logdown.com/posts/152158-scaling-thumbor</li>
</ul>RBD Replication2014-08-12T09:33:43+02:002014-08-12T09:33:43+02:00Laurent Barbetag:cephnotes.ksperis.com,2014-08-12:/blog/2014/08/12/rbd-replication/<p>A simple exemple to make Replication for RBD.</p>
<!-- more -->
<p>Based on this post from scuttlemonkey : http://ceph.com/dev-notes/incremental-snapshots-with-rbd/, here is a sample script to synchronize rbd image on a remote cluster (eg for backups).
In the example below, the sync is made to an "archive" pool on the same …</p><p>A simple exemple to make Replication for RBD.</p>
<!-- more -->
<p>Based on this post from scuttlemonkey : http://ceph.com/dev-notes/incremental-snapshots-with-rbd/, here is a sample script to synchronize rbd image on a remote cluster (eg for backups).
In the example below, the sync is made to an "archive" pool on the same cluster.
(For remote host, you need to use ssh key.)</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nv">pool</span><span class="o">=</span><span class="s1">'rbd'</span>
<span class="nv">pooldest</span><span class="o">=</span><span class="s1">'archive'</span>
<span class="nv">rbd</span><span class="o">=</span><span class="s2">"myrbd"</span>
<span class="nv">destination_host</span><span class="o">=</span><span class="s2">"127.0.0.1"</span>
<span class="nv">snapname</span><span class="o">=</span><span class="s1">'rbd-sync-'</span>
<span class="c1"># Retreive last synced id</span>
<span class="nv">expr</span><span class="o">=</span><span class="s2">" </span><span class="nv">$snapname</span><span class="s2">\([[:digit:]]\+\)"</span>
<span class="k">if</span><span class="w"> </span>rbd<span class="w"> </span>info<span class="w"> </span><span class="nv">$pool</span>/<span class="nv">$rbd</span><span class="w"> </span>>/dev/null<span class="w"> </span><span class="m">2</span>><span class="p">&</span><span class="m">1</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span>rbd<span class="w"> </span>snap<span class="w"> </span>ls<span class="w"> </span><span class="nv">$pool</span>/<span class="nv">$rbd</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s2">"</span><span class="nv">$expr</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="s2">"s/.*</span><span class="nv">$expr</span><span class="s2">.*/\1/g"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sort<span class="w"> </span>-n<span class="w"> </span>><span class="w"> </span>/tmp/rbd-sync-snaplistlocal
<span class="k">else</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"no image </span><span class="nv">$pool</span><span class="s2">/</span><span class="nv">$rbd</span><span class="s2">"</span>
<span class="w"> </span><span class="k">return</span>
<span class="k">fi</span>
<span class="k">if</span><span class="w"> </span>ssh<span class="w"> </span><span class="nv">$destination_host</span><span class="w"> </span>rbd<span class="w"> </span>info<span class="w"> </span><span class="nv">$pooldest</span>/<span class="nv">$rbd</span><span class="w"> </span>>/dev/null<span class="w"> </span><span class="m">2</span>><span class="p">&</span><span class="m">1</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span>ssh<span class="w"> </span><span class="nv">$destination_host</span><span class="w"> </span>rbd<span class="w"> </span>snap<span class="w"> </span>ls<span class="w"> </span><span class="nv">$pooldest</span>/<span class="nv">$rbd</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s2">"</span><span class="nv">$expr</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="s2">"s/.*</span><span class="nv">$expr</span><span class="s2">.*/\1/g"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sort<span class="w"> </span>-n<span class="w"> </span>><span class="w"> </span>/tmp/rbd-sync-snaplistremote
<span class="k">else</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span>><span class="w"> </span>/tmp/rbd-sync-snaplistremote
<span class="k">fi</span>
<span class="nv">syncid</span><span class="o">=</span><span class="k">$(</span>comm<span class="w"> </span>-12<span class="w"> </span>/tmp/rbd-sync-snaplistlocal<span class="w"> </span>/tmp/rbd-sync-snaplistremote<span class="w"> </span><span class="p">|</span><span class="w"> </span>tail<span class="w"> </span>-n1<span class="k">)</span>
<span class="nv">lastid</span><span class="o">=</span><span class="k">$(</span>cat<span class="w"> </span>/tmp/rbd-sync-snaplistlocal<span class="w"> </span>/tmp/rbd-sync-snaplistremote<span class="w"> </span><span class="p">|</span><span class="w"> </span>sort<span class="w"> </span>-n<span class="w"> </span><span class="p">|</span><span class="w"> </span>tail<span class="w"> </span>-n1<span class="k">)</span>
<span class="nv">nextid</span><span class="o">=</span><span class="k">$((</span><span class="nv">$lastid</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="m">1</span><span class="k">))</span>
<span class="c1"># Initial sync</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"</span><span class="nv">$syncid</span><span class="s2">"</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Initial sync with id </span><span class="nv">$nextid</span><span class="s2">"</span>
<span class="w"> </span>rbd<span class="w"> </span>snap<span class="w"> </span>create<span class="w"> </span><span class="nv">$pool</span>/<span class="nv">$rbd</span>@<span class="nv">$snapname$nextid</span>
<span class="w"> </span>rbd<span class="w"> </span><span class="nb">export</span><span class="w"> </span>--no-progress<span class="w"> </span><span class="nv">$pool</span>/<span class="nv">$rbd</span>@<span class="nv">$snapname$nextid</span><span class="w"> </span>-<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="p">|</span><span class="w"> </span>ssh<span class="w"> </span><span class="nv">$destination_host</span><span class="w"> </span>rbd<span class="w"> </span>import<span class="w"> </span>--image-format<span class="w"> </span><span class="m">2</span><span class="w"> </span>-<span class="w"> </span><span class="nv">$pooldest</span>/<span class="nv">$rbd</span>
<span class="w"> </span>ssh<span class="w"> </span><span class="nv">$destination_host</span><span class="w"> </span>rbd<span class="w"> </span>snap<span class="w"> </span>create<span class="w"> </span><span class="nv">$pooldest</span>/<span class="nv">$rbd</span>@<span class="nv">$snapname$nextid</span>
<span class="c1"># Incremental sync</span>
<span class="k">else</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Found synced id : </span><span class="nv">$syncid</span><span class="s2">"</span>
<span class="w"> </span>rbd<span class="w"> </span>snap<span class="w"> </span>create<span class="w"> </span><span class="nv">$pool</span>/<span class="nv">$rbd</span>@<span class="nv">$snapname$nextid</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Sync </span><span class="nv">$syncid</span><span class="s2"> -> </span><span class="nv">$nextid</span><span class="s2">"</span>
<span class="w"> </span>rbd<span class="w"> </span>export-diff<span class="w"> </span>--no-progress<span class="w"> </span>--from-snap<span class="w"> </span><span class="nv">$snapname$syncid</span><span class="w"> </span><span class="nv">$pool</span>/<span class="nv">$rbd</span>@<span class="nv">$snapname$nextid</span><span class="w"> </span>-<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="p">|</span><span class="w"> </span>ssh<span class="w"> </span><span class="nv">$destination_host</span><span class="w"> </span>rbd<span class="w"> </span>import-diff<span class="w"> </span>-<span class="w"> </span><span class="nv">$pooldest</span>/<span class="nv">$rbd</span>
<span class="k">fi</span>
</code></pre></div>
<p>An other exemple : https://www.rapide.nl/blog/item/ceph_-_rbd_replication</p>Low Cost Scale-Out Nas for the Office2014-08-07T22:30:00+02:002014-08-07T22:30:00+02:00Laurent Barbetag:cephnotes.ksperis.com,2014-08-07:/blog/2014/08/07/low-cost-scale-out-nas-for-the-office/<p>The aim is showing that it is possible to create a low-cost storage, efficient and scalable, using opensource solutions.
In the example below, I am using <a href="http://ceph.com/">Ceph</a> for scalability and reliability, combined with <a href="https://github.com/stec-inc/EnhanceIO/">EnhanceIO</a> to ensure very good performance.</p>
<!-- more -->
<h2>Architecture</h2>
<p>The idea was to create a storage with two parts …</p><p>The aim is showing that it is possible to create a low-cost storage, efficient and scalable, using opensource solutions.
In the example below, I am using <a href="http://ceph.com/">Ceph</a> for scalability and reliability, combined with <a href="https://github.com/stec-inc/EnhanceIO/">EnhanceIO</a> to ensure very good performance.</p>
<!-- more -->
<h2>Architecture</h2>
<p>The idea was to create a storage with two parts: the storage itself (backing store) and a large cache to keep good performance on all data actually used.
In fact, the volume needs may be important, but in the context of use for office, the data really used every day is only a portion of this storage.
In my case, I intend to use Ceph deployed on low-cost hardware to ensure a scalable and reliable conformatble volume, based on small servers with large SATA drives. Data access will be via <a href="http://www.samba.org/">Samba</a> shares on a slightly more powerful machine with an SAS Array to create a big cache. (Since Firefly, it would be more interesting to use ceph cache tiering)</p>
<p><img alt="low-cost-scale-out-nas-for-the-office" src="https://cephnotes.ksperis.com/images/img014.png"></p>
<h2>Hardware</h2>
<h4>Ceph Part (Backing Store)</h4>
<p>The material you choose should match the requirements of both performance and cost. To keep prices extremely competitive machines chosen here are based on Supermicro hardware without additional disk controller. Initially, the ceph storage is composed of 5 machines: store-01-02 store, store-03-04 store, store-05. Each node is build with simple CPU Core 2, 4 GB Memory, 1 SSD Drive (Intel 520 60GB) for system and Ceph journal and 3 SATA Drive of 3 TO (Seagate CS 3TB) with no specific Drive controller (Using onbord controller).</p>
<h4>Samba / EnhanceIO Part (File Server and Cache)</h4>
<p>The cache must be on the same server as the file server. In my case, I use Dell server with 10 sas drive 10k in Raid10 for the cache.</p>
<h4>Network</h4>
<p>I do not provide for a specific network, just a gigabit switch with vlan to separate public / private network for Ceph OSD and interface bonding for file server.</p>
<p>The system is currently used everyday since more than one year. And it works perfectly.</p>
<h1>Installation</h1>
<p>Here is some detail of the installation (from what I remember or I noted):</p>
<h2>Ceph Cluster</h2>
<p>The current configuration is running on Ceph Cuttlefish (v0.61) and Debian Wheezy. (Installation made in June 2013.)</p>
<h4>Debian install and partitionning</h4>
<p>The operating system is installed on the SSD with 3 others partitions for osd journal.</p>
<p><img alt="low-cost-scale-out-nas-for-the-office" src="https://cephnotes.ksperis.com/images/img001.png"></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>vim<span class="w"> </span>sudo<span class="w"> </span>sysstat<span class="w"> </span>ntp<span class="w"> </span>smartmontools
$<span class="w"> </span>vim<span class="w"> </span>/etc/default/smartmontools
$<span class="w"> </span>vim<span class="w"> </span>/etc/smartd.conf
</code></pre></div>
<h4>Kernel upgrade</h4>
<p>The use of the kernel version 3.12 available in backports seems improve memory footprint on the osd node.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"deb http://ftp.fr.debian.org/debian wheezy-backports main"</span><span class="w"> </span>>><span class="w"> </span>/etc/apt/sources.list
$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>-t<span class="w"> </span>wheezy-backports<span class="w"> </span>linux-image-3.12-0.bpo.1-amd64
</code></pre></div>
<h4>Ceph Installation</h4>
<p>( Based on the official documentation: http://ceph.com/docs/master/start/ )</p>
<p>On each server create a ceph user</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>useradd<span class="w"> </span>-d<span class="w"> </span>/home/ceph<span class="w"> </span>-m<span class="w"> </span>ceph
$<span class="w"> </span>passwd<span class="w"> </span>ceph
$<span class="w"> </span>vim<span class="w"> </span><span class="o">[[</span>/etc/hosts<span class="o">]]</span>
<span class="m">192</span>.168.0.1<span class="w"> </span>store-b1-01
<span class="m">192</span>.168.0.2<span class="w"> </span>store-b1-02
<span class="m">192</span>.168.0.3<span class="w"> </span>store-b1-03
<span class="m">192</span>.168.0.4<span class="w"> </span>store-b1-04
<span class="m">192</span>.168.0.5<span class="w"> </span>store-b1-05
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"ceph ALL = (root) NOPASSWD:ALL"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sudo<span class="w"> </span>tee<span class="w"> </span>/etc/sudoers.d/ceph
$<span class="w"> </span>chmod<span class="w"> </span><span class="m">0440</span><span class="w"> </span>/etc/sudoers.d/ceph
</code></pre></div>
<p>On store-b1-01 (deployment server)</p>
<p>Create a new key for ssh authentication :
$ ssh-keygen</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nv">cluster</span><span class="o">=</span><span class="s2">"store-b1-01 store-b1-02 store-b1-03 store-b1-04 store-b1-05"</span>
$<span class="w"> </span><span class="k">for</span><span class="w"> </span>i<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="nv">$cluster</span><span class="p">;</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span>ssh-copy-id<span class="w"> </span>ceph@<span class="nv">$i</span>
<span class="w"> </span><span class="k">done</span>
$<span class="w"> </span>vim<span class="w"> </span>/root/.ssh/config
Host<span class="w"> </span>store*
<span class="w"> </span>User<span class="w"> </span>ceph
</code></pre></div>
<p>Install ceph-deploy and its dependencies</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>wget<span class="w"> </span>-q<span class="w"> </span>-O-<span class="w"> </span><span class="s1">'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sudo<span class="w"> </span>apt-key<span class="w"> </span>add<span class="w"> </span>-
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>deb<span class="w"> </span>http://eu.ceph.com/debian-cuttlefish/<span class="w"> </span><span class="k">$(</span>lsb_release<span class="w"> </span>-sc<span class="k">)</span><span class="w"> </span>main<span class="w"> </span><span class="p">|</span><span class="w"> </span>sudo<span class="w"> </span>tee<span class="w"> </span>/etc/apt/sources.list.d/ceph.list
$<span class="w"> </span>apt-get<span class="w"> </span>update
$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>python-pkg-resources<span class="w"> </span>python-setuptools<span class="w"> </span>ceph-deploy<span class="w"> </span>collectd
$<span class="w"> </span>curl<span class="w"> </span>http://python-distribute.org/distribute_setup.py<span class="w"> </span><span class="p">|</span><span class="w"> </span>python
$<span class="w"> </span>easy_install<span class="w"> </span>pushy
</code></pre></div>
<p>Install ceph on cluster</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph-deploy<span class="w"> </span>install<span class="w"> </span><span class="nv">$cluster</span>
$<span class="w"> </span>ceph-deploy<span class="w"> </span>new<span class="w"> </span>store-b1-01<span class="w"> </span>store-b1-02<span class="w"> </span>store-b1-03
$<span class="w"> </span>vim<span class="w"> </span>ceph.conf
$<span class="w"> </span>ceph-deploy<span class="w"> </span>mon<span class="w"> </span>create<span class="w"> </span>store-b1-01<span class="w"> </span>store-b1-02<span class="w"> </span>store-b1-03
</code></pre></div>
<p>Cluster deployment</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph-deploy<span class="w"> </span>gatherkeys<span class="w"> </span>store-b1-01
$<span class="w"> </span>ceph-deploy<span class="w"> </span>osd<span class="w"> </span>create<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-01:/dev/sdb:/dev/sda5<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-01:/dev/sdc:/dev/sda6<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-01:/dev/sdd:/dev/sda7<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-02:/dev/sdb:/dev/sda5<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-02:/dev/sdc:/dev/sda6<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-02:/dev/sdd:/dev/sda7<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-03:/dev/sdb:/dev/sda5<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-03:/dev/sdc:/dev/sda6<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-03:/dev/sdd:/dev/sda7<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-04:/dev/sdb:/dev/sda5<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-04:/dev/sdc:/dev/sda6<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-04:/dev/sdd:/dev/sda7<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-05:/dev/sdb:/dev/sda5<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-05:/dev/sdc:/dev/sda6<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>store-b1-05:/dev/sdd:/dev/sda7
</code></pre></div>
<p>Add in fstab:</p>
<div class="highlight"><pre><span></span><code><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb1</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">osd</span><span class="o">/</span><span class="n">ceph</span><span class="o">-</span><span class="mi">0</span><span class="w"> </span><span class="n">xfs</span><span class="w"> </span><span class="n">inode64</span><span class="p">,</span><span class="n">noatime</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdc1</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">osd</span><span class="o">/</span><span class="n">ceph</span><span class="o">-</span><span class="mi">1</span><span class="w"> </span><span class="n">xfs</span><span class="w"> </span><span class="n">inode64</span><span class="p">,</span><span class="n">noatime</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdd1</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">osd</span><span class="o">/</span><span class="n">ceph</span><span class="o">-</span><span class="mi">2</span><span class="w"> </span><span class="n">xfs</span><span class="w"> </span><span class="n">inode64</span><span class="p">,</span><span class="n">noatime</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
</code></pre></div>
<p>One can check that all osd have been created, and check the cluster status :
$ ceph osd tree</p>
<div class="highlight"><pre><span></span><code><span class="err">#</span><span class="w"> </span><span class="nx">id</span><span class="w"> </span><span class="nx">weight</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nx">name</span><span class="w"> </span><span class="nx">up</span><span class="o">/</span><span class="nx">down</span><span class="w"> </span><span class="nx">reweight</span>
<span class="o">-</span><span class="mi">1</span><span class="w"> </span><span class="m m-Double">41.23</span><span class="w"> </span><span class="nx">root</span><span class="w"> </span><span class="k">default</span>
<span class="o">-</span><span class="mi">8</span><span class="w"> </span><span class="m m-Double">41.23</span><span class="w"> </span><span class="nx">datacenter</span><span class="w"> </span><span class="nx">DC1</span>
<span class="o">-</span><span class="mi">7</span><span class="w"> </span><span class="m m-Double">41.23</span><span class="w"> </span><span class="nx">rack</span><span class="w"> </span><span class="nx">b1</span>
<span class="o">-</span><span class="mi">2</span><span class="w"> </span><span class="m m-Double">8.24</span><span class="w"> </span><span class="nx">host</span><span class="w"> </span><span class="nx">store</span><span class="o">-</span><span class="nx">b1</span><span class="o">-</span><span class="mi">01</span>
<span class="mi">0</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.0</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="mi">1</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.1</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="mi">2</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.2</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="o">-</span><span class="mi">3</span><span class="w"> </span><span class="m m-Double">8.24</span><span class="w"> </span><span class="nx">host</span><span class="w"> </span><span class="nx">store</span><span class="o">-</span><span class="nx">b1</span><span class="o">-</span><span class="mi">02</span>
<span class="mi">3</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.3</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="mi">4</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.4</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="mi">5</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.5</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="o">-</span><span class="mi">4</span><span class="w"> </span><span class="m m-Double">8.24</span><span class="w"> </span><span class="nx">host</span><span class="w"> </span><span class="nx">store</span><span class="o">-</span><span class="nx">b1</span><span class="o">-</span><span class="mi">03</span>
<span class="mi">6</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.6</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="mi">7</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.7</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="mi">8</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.8</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="o">-</span><span class="mi">5</span><span class="w"> </span><span class="m m-Double">8.24</span><span class="w"> </span><span class="nx">host</span><span class="w"> </span><span class="nx">store</span><span class="o">-</span><span class="nx">b1</span><span class="o">-</span><span class="mi">04</span>
<span class="mi">9</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.9</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="mi">10</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.10</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="mi">11</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.11</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="o">-</span><span class="mi">6</span><span class="w"> </span><span class="m m-Double">8.24</span><span class="w"> </span><span class="nx">host</span><span class="w"> </span><span class="nx">store</span><span class="o">-</span><span class="nx">b1</span><span class="o">-</span><span class="mi">05</span>
<span class="mi">12</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.12</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="mi">13</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.13</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span>
<span class="mi">14</span><span class="w"> </span><span class="m m-Double">2.65</span><span class="w"> </span><span class="nx">osd</span><span class="m m-Double">.14</span><span class="w"> </span><span class="nx">up</span><span class="w"> </span><span class="mi">1</span>
</code></pre></div>
<h2>File Server</h2>
<h4>Debian interface bonding</h4>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>ifenslave
$<span class="w"> </span>vim<span class="w"> </span>/etc/network/interface
auto<span class="w"> </span>bond0
iface<span class="w"> </span>bond0<span class="w"> </span>inet<span class="w"> </span>static
<span class="w"> </span>address<span class="w"> </span><span class="m">192</span>.168.0.12
<span class="w"> </span>netmask<span class="w"> </span><span class="m">255</span>.255.0.0
<span class="w"> </span>gateway<span class="w"> </span><span class="m">192</span>.168.0.1
<span class="w"> </span>slaves<span class="w"> </span>eth0<span class="w"> </span>eth1
<span class="w"> </span>bond-mode<span class="w"> </span><span class="m">802</span>.3ad
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"alias bond0 bonding</span>
<span class="s2">options bonding mode=4 miimon=100 lacp_rate=1"</span><span class="w"> </span>><span class="w"> </span>vim<span class="w"> </span>/etc/modprobe.d/bonding.conf
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"bonding"</span><span class="w"> </span>>><span class="w"> </span>/etc/modules
</code></pre></div>
<h4>Kernel Version</h4>
<p>The kernel version is important for using KRBD. I recommend to use at least kernel 3.10.26 or later.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>debconf-utils<span class="w"> </span>dpkg-dev<span class="w"> </span>debhelper<span class="w"> </span>build-essential<span class="w"> </span>kernel-package<span class="w"> </span>libncurses5-dev
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>/usr/src/
$<span class="w"> </span>wget<span class="w"> </span>http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.6.11.tar.bz2
$<span class="w"> </span>tar<span class="w"> </span>xjf<span class="w"> </span>linux-3.6.11.tar.bz2
</code></pre></div>
<h4>RBD</h4>
<p>Create rbd volume (Format 1) :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>create<span class="w"> </span>datashare/share1<span class="w"> </span>--image-format<span class="o">=</span><span class="m">1</span><span class="w"> </span>--size<span class="o">=</span><span class="m">1048576</span>
$<span class="w"> </span>mkdir<span class="w"> </span>/share1
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"/dev/rbd/datashare/share1 /share1 xfs _netdev,barrier=0,nodiratime 0 0"</span><span class="w"> </span>>><span class="w"> </span>/etc/fstab
</code></pre></div>
<h4>EnhanceIO</h4>
<p>The choice of cache mechanism is focused on "EnhanceIO" because it allows to enable or disable cache while a source volume is being used. This is particularly useful when we want to resize a volume without interrupting service.</p>
<p>Build EnhanceIO :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>build-essential<span class="w"> </span>dkms
$<span class="w"> </span>git<span class="w"> </span>clone<span class="w"> </span>https://github.com/stec-inc/EnhanceIO.git
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>EnhanceIO/
$<span class="w"> </span>wget<span class="w"> </span>http://ftp.de.debian.org/debian/pool/main/e/enhanceio/enhanceio_0+git20130620-3.debian.tar.xz
$<span class="w"> </span>tar<span class="w"> </span>xJf<span class="w"> </span>enhanceio_0+git20130620-3.debian.tar.xz
$<span class="w"> </span>dpkg-buildpackage<span class="w"> </span>-rfakeroot<span class="w"> </span>-b
$<span class="w"> </span>dpkg<span class="w"> </span>-i<span class="w"> </span>../*.deb
</code></pre></div>
<p>Create cache :</p>
<p>For exemple in write-though :
(/dev/sdb2 is a local partition dedicate for cache)</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>eio_cli<span class="w"> </span>create<span class="w"> </span>-d<span class="w"> </span>/dev/rbd1<span class="w"> </span>-s<span class="w"> </span>/dev/sdb2<span class="w"> </span>-p<span class="w"> </span>lru<span class="w"> </span>-m<span class="w"> </span>wt<span class="w"> </span>-b<span class="w"> </span><span class="m">4096</span><span class="w"> </span>-c<span class="w"> </span>share1
</code></pre></div>
<p>If you want to use write-back cache, you can protect the file system to mount before the cache by using a symbolic link in the udev script. ( https://github.com/ksperis/EnhanceIO/commit/954e167fdb580d514747512ce2bd1c9c29a77418 )</p>
<h4>Samba</h4>
<p>Packages installation</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"deb http://ftp.sernet.de/pub/samba/3.6/debian wheezy main"</span><span class="w"> </span>>><span class="w"> </span>/etc/apt/sources.list
$<span class="w"> </span>apt-get<span class="w"> </span>update
$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>sernet-samba<span class="w"> </span>sernet-winbind<span class="w"> </span>xfsprogs<span class="w"> </span>krb5-user<span class="w"> </span>acl<span class="w"> </span>attr
</code></pre></div>
<p>Update startup script</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>vim<span class="w"> </span>/etc/init.d/samba
<span class="c1"># Should-Start: slapd cups rbdmap</span>
<span class="c1"># Should-Stop: slapd cups rbdmap</span>
$<span class="w"> </span>insserv<span class="w"> </span>-d<span class="w"> </span>samba
</code></pre></div>
<p>Configure and join the domain ( https://help.ubuntu.com/community/ActiveDirectoryWinbindHowto )</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>vi<span class="w"> </span>/etc/krb5.conf
...
$<span class="w"> </span>kinit<span class="w"> </span>Administrator@AD.MYDOMAIN.COM
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="n">$</span><span class="w"> </span><span class="n">vim</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">samba</span><span class="o">/</span><span class="n">smb</span><span class="p">.</span><span class="n">conf</span>
<span class="p">[</span><span class="n">global</span><span class="p">]</span>
<span class="w"> </span><span class="n">workgroup</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">MYDOMAIN</span>
<span class="w"> </span><span class="n">realm</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">AD</span><span class="p">.</span><span class="n">MYDOMAIN</span><span class="p">.</span><span class="n">COM</span>
<span class="w"> </span><span class="n">netbios</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">MYNAS</span>
<span class="w"> </span><span class="n">wins</span><span class="w"> </span><span class="n">server</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mf">192.168.0.4</span>
<span class="w"> </span><span class="n">server</span><span class="w"> </span><span class="n">string</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">%</span><span class="n">h</span><span class="w"> </span><span class="n">server</span>
<span class="w"> </span><span class="n">dns</span><span class="w"> </span><span class="n">proxy</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">no</span>
<span class="w"> </span><span class="n">log</span><span class="w"> </span><span class="n">file</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="n">var</span><span class="o">/</span><span class="n">log</span><span class="o">/</span><span class="n">samba</span><span class="o">/</span><span class="n">log</span><span class="p">.</span><span class="o">%</span><span class="n">m</span>
<span class="w"> </span><span class="n">log</span><span class="w"> </span><span class="n">level</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span>
<span class="w"> </span><span class="n">max</span><span class="w"> </span><span class="n">log</span><span class="w"> </span><span class="n">size</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1000</span>
<span class="w"> </span><span class="n">syslog</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">panic</span><span class="w"> </span><span class="n">action</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">share</span><span class="o">/</span><span class="n">samba</span><span class="o">/</span><span class="n">panic</span><span class="o">-</span><span class="n">action</span><span class="w"> </span><span class="o">%</span><span class="n">d</span>
<span class="w"> </span><span class="n">security</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ADS</span>
<span class="w"> </span><span class="n">winbind</span><span class="w"> </span><span class="n">separator</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">+</span>
<span class="w"> </span><span class="n">client</span><span class="w"> </span><span class="n">use</span><span class="w"> </span><span class="n">spnego</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="n">winbind</span><span class="w"> </span><span class="n">use</span><span class="w"> </span><span class="k">default</span><span class="w"> </span><span class="n">domain</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="n">domain</span><span class="w"> </span><span class="n">master</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">no</span>
<span class="w"> </span><span class="n">local</span><span class="w"> </span><span class="n">master</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">no</span>
<span class="w"> </span><span class="n">preferred</span><span class="w"> </span><span class="n">master</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">no</span>
<span class="w"> </span><span class="n">encrypt</span><span class="w"> </span><span class="n">passwords</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">true</span>
<span class="w"> </span><span class="n">passdb</span><span class="w"> </span><span class="n">backend</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tdbsam</span>
<span class="w"> </span><span class="n">obey</span><span class="w"> </span><span class="n">pam</span><span class="w"> </span><span class="n">restrictions</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="n">unix</span><span class="w"> </span><span class="n">password</span><span class="w"> </span><span class="n">sync</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="n">passwd</span><span class="w"> </span><span class="n">program</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">passwd</span><span class="w"> </span><span class="o">%</span><span class="n">u</span>
<span class="w"> </span><span class="n">passwd</span><span class="w"> </span><span class="n">chat</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">*</span><span class="n">Enter</span><span class="err">\</span><span class="n">snew</span><span class="err">\</span><span class="n">s</span><span class="o">*</span><span class="err">\</span><span class="n">spassword</span><span class="o">:*</span><span class="w"> </span><span class="o">%</span><span class="n">n</span><span class="err">\</span><span class="n">n</span><span class="w"> </span><span class="o">*</span><span class="n">Retype</span><span class="err">\</span><span class="n">snew</span><span class="err">\</span><span class="n">s</span><span class="o">*</span><span class="err">\</span><span class="n">spassword</span><span class="o">:*</span><span class="w"> </span><span class="o">%</span><span class="n">n</span><span class="err">\</span><span class="n">n</span><span class="w"> </span><span class="o">*</span><span class="n">password</span><span class="err">\</span><span class="n">supdated</span><span class="err">\</span><span class="n">ssuccessfully</span><span class="o">*</span><span class="w"> </span><span class="p">.</span>
<span class="w"> </span><span class="n">pam</span><span class="w"> </span><span class="n">password</span><span class="w"> </span><span class="n">change</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="n">idmap</span><span class="w"> </span><span class="n">uid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">10000-20000</span>
<span class="w"> </span><span class="n">idmap</span><span class="w"> </span><span class="n">gid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">10000-20000</span>
<span class="w"> </span><span class="n">template</span><span class="w"> </span><span class="n">shell</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">bash</span>
<span class="w"> </span><span class="n">template</span><span class="w"> </span><span class="n">homedir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="n">share4</span><span class="o">/</span><span class="n">home</span><span class="o">/%</span><span class="n">D</span><span class="o">/%</span><span class="n">U</span>
<span class="w"> </span><span class="n">winbind</span><span class="w"> </span><span class="k">enum</span><span class="w"> </span><span class="n">groups</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="n">winbind</span><span class="w"> </span><span class="k">enum</span><span class="w"> </span><span class="n">users</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="n">map</span><span class="w"> </span><span class="n">acl</span><span class="w"> </span><span class="n">inherit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="n">vfs</span><span class="w"> </span><span class="n">objects</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">acl_xattr</span><span class="w"> </span><span class="n">recycle</span><span class="w"> </span><span class="n">shadow_copy2</span>
<span class="w"> </span><span class="nl">recycle</span><span class="p">:</span><span class="n">repository</span><span class="w"> </span><span class="o">=</span><span class="p">.</span><span class="n">recycle</span><span class="o">/%</span><span class="n">u</span><span class="w"> </span>
<span class="w"> </span><span class="nl">recycle</span><span class="p">:</span><span class="n">keeptree</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="nl">recycle</span><span class="p">:</span><span class="n">exclude</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">*</span><span class="p">.</span><span class="n">tmp</span>
<span class="w"> </span><span class="nl">recycle</span><span class="p">:</span><span class="n">touch</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="nl">shadow</span><span class="p">:</span><span class="n">snapdir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">.</span><span class="n">snapshots</span>
<span class="w"> </span><span class="nl">shadow</span><span class="p">:</span><span class="n">sort</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">desc</span>
<span class="w"> </span><span class="n">ea</span><span class="w"> </span><span class="n">support</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="n">map</span><span class="w"> </span><span class="n">hidden</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">no</span>
<span class="w"> </span><span class="n">map</span><span class="w"> </span><span class="n">system</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">no</span>
<span class="w"> </span><span class="n">map</span><span class="w"> </span><span class="n">archive</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">no</span>
<span class="w"> </span><span class="n">map</span><span class="w"> </span><span class="k">readonly</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">no</span>
<span class="w"> </span><span class="n">store</span><span class="w"> </span><span class="n">dos</span><span class="w"> </span><span class="n">attributes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="n">load</span><span class="w"> </span><span class="n">printers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">no</span>
<span class="w"> </span><span class="n">printing</span><span class="o">=</span><span class="n">bsd</span>
<span class="w"> </span><span class="n">printcap</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">null</span>
<span class="w"> </span><span class="n">disable</span><span class="w"> </span><span class="n">spoolss</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="n">guest</span><span class="w"> </span><span class="n">account</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">invité</span>
<span class="w"> </span><span class="n">map</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">guest</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bad</span><span class="w"> </span><span class="n">user</span>
<span class="n">$</span><span class="o">================================================================</span>
<span class="p">[</span><span class="n">share0</span><span class="p">]</span>
<span class="w"> </span><span class="n">comment</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">My</span><span class="w"> </span><span class="n">first</span><span class="w"> </span><span class="n">share</span>
<span class="w"> </span><span class="n">path</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="n">share0</span>
<span class="w"> </span><span class="n">writable</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="n">valid</span><span class="w"> </span><span class="n">users</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">@"MYDOMAIN+Domain Admins"</span><span class="w"> </span><span class="s">"MYDOMAIN+laurent"</span>
<span class="p">[</span><span class="n">share1</span><span class="p">]</span>
<span class="w"> </span><span class="n">comment</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Other</span><span class="w"> </span><span class="n">share</span>
<span class="w"> </span><span class="n">path</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="n">share1</span>
<span class="w"> </span><span class="n">writable</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span>
<span class="w"> </span><span class="n">valid</span><span class="w"> </span><span class="n">users</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">@"MYDOMAIN+Domain Admins"</span><span class="w"> </span><span class="s">"MYDOMAIN+laurent"</span>
<span class="p">....</span>
<span class="n">$</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">init</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">samba</span><span class="w"> </span><span class="n">restart</span>
<span class="n">$</span><span class="w"> </span><span class="n">net</span><span class="w"> </span><span class="n">join</span><span class="w"> </span><span class="o">-</span><span class="n">U</span><span class="w"> </span><span class="n">Administrator</span>
<span class="n">$</span><span class="w"> </span><span class="n">wbinfo</span><span class="w"> </span><span class="o">-</span><span class="n">u</span>
<span class="n">$</span><span class="w"> </span><span class="n">wbinfo</span><span class="w"> </span><span class="o">-</span><span class="n">g</span>
<span class="n">$</span><span class="w"> </span><span class="n">vi</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">nsswitch</span><span class="p">.</span><span class="n">conf</span>
<span class="nl">passwd</span><span class="p">:</span><span class="w"> </span><span class="n">compat</span><span class="w"> </span><span class="n">winbind</span>
<span class="nl">group</span><span class="p">:</span><span class="w"> </span><span class="n">compat</span><span class="w"> </span><span class="n">winbind</span>
</code></pre></div>
<h4>Virtual Shadow Copy</h4>
<p>Using this script :
https://github.com/ksperis/autosnap-rbd-shadow-copy</p>
<h4>TimeMachine Backup</h4>
<p>Fisrt, create rbd timemachine and mount to /mnt/timemachine with noatime and uquota if you want to use per user quota (Do not add to the autosnap script.)</p>
<p>Install avahi</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>avahi-daemon<span class="w"> </span>avahi-utils
$<span class="w"> </span>vim<span class="w"> </span>/etc/avahi/services/smb.service
<span class="cp"><?xml version="1.0" standalone='no'?></span>
<span class="cp"><!DOCTYPE service-group SYSTEM "avahi-service.dtd"></span>
<span class="nt"><service-group></span>
<span class="w"> </span><span class="nt"><name</span><span class="w"> </span><span class="na">replace-wildcards=</span><span class="s">"yes"</span><span class="nt">></span>%h<span class="w"> </span>(File<span class="w"> </span>Server)<span class="nt"></name></span>
<span class="w"> </span><span class="nt"><service></span>
<span class="w"> </span><span class="nt"><type></span>_smb._tcp<span class="nt"></type></span>
<span class="w"> </span><span class="nt"><port></span>445<span class="nt"></port></span>
<span class="w"> </span><span class="nt"></service></span>
<span class="w"> </span><span class="nt"><service></span>
<span class="w"> </span><span class="nt"><type></span>_device-info._tcp<span class="nt"></type></span>
<span class="w"> </span><span class="nt"><port></span>0<span class="nt"></port></span>
<span class="w"> </span><span class="nt"><txt-record></span>model=RackMac<span class="nt"></txt-record></span>
<span class="w"> </span><span class="nt"></service></span>
<span class="nt"></service-group></span>
$<span class="w"> </span>vim<span class="w"> </span>/etc/avahi/services/timemachine.service
<span class="cp"><?xml version="1.0" standalone="no"?></span>
<span class="cp"><!DOCTYPE service-group SYSTEM "avahi-service.dtd"></span>
<span class="nt"><service-group></span>
<span class="w"> </span><span class="nt"><name</span><span class="w"> </span><span class="na">replace-wildcards=</span><span class="s">"yes"</span><span class="nt">></span>%h<span class="w"> </span>(Time<span class="w"> </span>Machine)<span class="nt"></name></span>
<span class="w"> </span><span class="nt"><service></span>
<span class="w"> </span><span class="nt"><type></span>_afpovertcp._tcp<span class="nt"></type></span>
<span class="w"> </span><span class="nt"><port></span>548<span class="nt"></port></span>
<span class="w"> </span><span class="nt"></service></span>
<span class="w"> </span><span class="nt"><service></span>
<span class="w"> </span><span class="nt"><type></span>_device-info._tcp<span class="nt"></type></span>
<span class="w"> </span><span class="nt"><port></span>0<span class="nt"></port></span>
<span class="w"> </span><span class="nt"><txt-record></span>model=TimeCapsule<span class="nt"></txt-record></span>
<span class="w"> </span><span class="nt"></service></span>
<span class="w"> </span><span class="nt"><service></span>
<span class="w"> </span><span class="nt"><type></span>_adisk._tcp<span class="nt"></type></span>
<span class="w"> </span><span class="nt"><port></span>9<span class="nt"></port></span>
<span class="w"> </span><span class="nt"><txt-record></span>sys=waMA=00:1d:09:63:87:e0,adVF=0x100<span class="nt"></txt-record></span>
<span class="w"> </span><span class="nt"><txt-record></span>dk0=adVF=0x83,adVN=TimeMachine<span class="nt"></txt-record></span>
<span class="w"> </span><span class="nt"></service></span>
<span class="nt"></service-group></span>
</code></pre></div>
<p>Install netatalk and configure like this :</p>
<div class="highlight"><pre><span></span><code><span class="n">$</span><span class="w"> </span><span class="n">apt</span><span class="o">-</span><span class="n">get</span><span class="w"> </span><span class="n">install</span><span class="w"> </span><span class="n">netatalk</span>
<span class="n">$</span><span class="w"> </span><span class="n">vim</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">netatalk</span><span class="o">/</span><span class="n">afpd</span><span class="p">.</span><span class="n">conf</span>
<span class="o">-</span><span class="w"> </span><span class="o">-</span><span class="n">tcp</span><span class="w"> </span><span class="o">-</span><span class="n">noddp</span><span class="w"> </span><span class="o">-</span><span class="n">nozeroconf</span><span class="w"> </span><span class="o">-</span><span class="n">uamlist</span><span class="w"> </span><span class="n">uams_dhx</span><span class="p">.</span><span class="n">so</span><span class="p">,</span><span class="n">uams_dhx2</span><span class="p">.</span><span class="n">so</span><span class="w"> </span><span class="o">-</span><span class="n">nosavepassword</span><span class="w"> </span><span class="o">-</span><span class="n">setuplog</span><span class="w"> </span><span class="s">"default log_info /var/log/afpd.log"</span>
<span class="n">$</span><span class="w"> </span><span class="n">vim</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">netatalk</span><span class="o">/</span><span class="n">AppleVolumes</span><span class="p">.</span><span class="k">default</span><span class="w"> </span><span class="p">(</span><span class="n">Remove</span><span class="w"> </span><span class="n">home</span><span class="w"> </span><span class="n">directory</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">Add</span><span class="w"> </span><span class="o">:</span><span class="p">)</span>
<span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">timemachine</span><span class="o">/</span><span class="w"> </span><span class="s">"TimeMachine"</span><span class="w"> </span><span class="n">cnidscheme</span><span class="o">:</span><span class="n">dbd</span><span class="w"> </span><span class="n">options</span><span class="o">:</span><span class="n">usedots</span><span class="p">,</span><span class="n">upriv</span><span class="p">,</span><span class="n">tm</span><span class="w"> </span><span class="n">allow</span><span class="o">:</span><span class="s">@"MYDOMAIN+Domain Users"</span>
<span class="n">$</span><span class="w"> </span><span class="n">vim</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">pam</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">netatalk</span>
<span class="n">auth</span><span class="w"> </span><span class="n">required</span><span class="w"> </span><span class="n">pam_winbind</span><span class="p">.</span><span class="n">so</span>
<span class="n">account</span><span class="w"> </span><span class="n">required</span><span class="w"> </span><span class="n">pam_winbind</span><span class="p">.</span><span class="n">so</span>
<span class="n">session</span><span class="w"> </span><span class="n">required</span><span class="w"> </span><span class="n">pam_unix</span><span class="p">.</span><span class="n">so</span>
</code></pre></div>
<p>Setting quota for a user : (Do not use soft limits, because it will not be recognized by timemachine.)</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>xfs_quota<span class="w"> </span>-x<span class="w"> </span>-c<span class="w"> </span><span class="s1">'limit bhard=1024g user1'</span><span class="w"> </span>/mnt/timemachine
</code></pre></div>
<h2>Benchmark</h2>
<p><img alt="Benchmark Ceph Office" src="/images/img013.png"></p>
<p>( For sequential IO, bandwidth is limited by the network card on the client side. )</p>Remove big RBD image2014-07-04T16:38:16+02:002014-07-04T16:38:16+02:00Laurent Barbetag:cephnotes.ksperis.com,2014-07-04:/blog/2014/07/04/remove-big-rbd-image/<p>Create a disk with an insane size can be fun, but a little hard to remove.</p>
<p><img alt="Big RBD image" src="https://cephnotes.ksperis.com/images/img012.png"></p>
<!--more-->
<p>Here's a little trick (use with caution!) to remove a too big image for <code>rm rbd</code> (if image is not initialized, or not fully).</p>
<h2>Image format 1 :</h2>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>info<span class="w"> </span>rbdbigsize
rbd<span class="w"> </span>image<span class="w"> </span><span class="s1">'rbdbigsize'</span>:
<span class="w"> </span>size …</code></pre></div><p>Create a disk with an insane size can be fun, but a little hard to remove.</p>
<p><img alt="Big RBD image" src="https://cephnotes.ksperis.com/images/img012.png"></p>
<!--more-->
<p>Here's a little trick (use with caution!) to remove a too big image for <code>rm rbd</code> (if image is not initialized, or not fully).</p>
<h2>Image format 1 :</h2>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>info<span class="w"> </span>rbdbigsize
rbd<span class="w"> </span>image<span class="w"> </span><span class="s1">'rbdbigsize'</span>:
<span class="w"> </span>size<span class="w"> </span><span class="m">1862</span><span class="w"> </span>PB<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">500000000000</span><span class="w"> </span>objects
<span class="w"> </span>order<span class="w"> </span><span class="m">22</span><span class="w"> </span><span class="o">(</span><span class="m">4096</span><span class="w"> </span>kB<span class="w"> </span>objects<span class="o">)</span>
<span class="w"> </span>block_name_prefix:<span class="w"> </span>rb.0.47aa.74b0dc51
<span class="w"> </span>format:<span class="w"> </span><span class="m">1</span>
</code></pre></div>
<p>Remove the header file :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbdbigsize.rbd
</code></pre></div>
<p>Remove all rbd data :
(This may take some time depending on the space already used on the blockdevice.)</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>ls<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^rb.0.47aa.74b0dc51'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>xargs<span class="w"> </span>-n<span class="w"> </span><span class="m">200</span><span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>rm
</code></pre></div>
<p>Finally, remove from rbd list :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbdbigsize
<span class="m">2014</span>-06-12<span class="w"> </span><span class="m">16</span>:27:20.838596<span class="w"> </span>7fd86dff3760<span class="w"> </span>-1<span class="w"> </span>librbd::ImageCtx:<span class="w"> </span>error<span class="w"> </span>finding<span class="w"> </span>header:<span class="w"> </span><span class="o">(</span><span class="m">2</span><span class="o">)</span><span class="w"> </span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory
Removing<span class="w"> </span>image:<span class="w"> </span><span class="m">100</span>%<span class="w"> </span>complete...done.
</code></pre></div>
<h2>Image format 2 :</h2>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>info<span class="w"> </span>rbdbigsize
rbd<span class="w"> </span>image<span class="w"> </span><span class="s1">'rbdbigsize'</span>:
<span class="w"> </span>size<span class="w"> </span><span class="m">1024</span><span class="w"> </span>TB<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">268435456</span><span class="w"> </span>objects
<span class="w"> </span>order<span class="w"> </span><span class="m">22</span><span class="w"> </span><span class="o">(</span><span class="m">4096</span><span class="w"> </span>kB<span class="w"> </span>objects<span class="o">)</span>
<span class="w"> </span>block_name_prefix:<span class="w"> </span>rbd_data.81d12ae8944a
<span class="w"> </span>format:<span class="w"> </span><span class="m">2</span>
<span class="w"> </span>features:<span class="w"> </span>layering
</code></pre></div>
<p>Remove header</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbd_id.rbdbigsize<span class="w"> </span>
$<span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbd_header.81d12ae8944a
</code></pre></div>
<p>Remove all rbd data :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>ls<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^rbd_data.81d12ae8944a.'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>xargs<span class="w"> </span>-n<span class="w"> </span><span class="m">200</span><span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>rm
</code></pre></div>
<p>Remove from rbd list :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbdbigsize
<span class="m">2014</span>-06-12<span class="w"> </span><span class="m">17</span>:37:24.483587<span class="w"> </span>7f413506c760<span class="w"> </span>-1<span class="w"> </span>librbd::ImageCtx:<span class="w"> </span>error<span class="w"> </span>finding<span class="w"> </span>header:<span class="w"> </span><span class="o">(</span><span class="m">2</span><span class="o">)</span><span class="w"> </span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory
Removing<span class="w"> </span>image:<span class="w"> </span><span class="m">100</span>%<span class="w"> </span>complete...done.
</code></pre></div>
<h2>Time comparison :</h2>
<p>Example of delete time for blockdevice of 1PB just formatted.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>create<span class="w"> </span>rbdbigsize<span class="w"> </span>--size<span class="o">=</span><span class="m">1073741824</span><span class="w"> </span>--image-format<span class="o">=</span><span class="m">2</span>
$<span class="w"> </span>rbd<span class="w"> </span>map<span class="w"> </span>rbdbigsize
$<span class="w"> </span>mkfs.xfs<span class="w"> </span>/dev/rbd0
$<span class="w"> </span>rbd<span class="w"> </span>unmap<span class="w"> </span>/dev/rbd0
$<span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>ls<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^rbd_data.81d12ae8944a.'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>wc<span class="w"> </span>-l
<span class="m">1536</span>
</code></pre></div>
<p>The block device is composed solely of 1536 block and can be quickly removed via <code>rados rm</code> :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">time</span><span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>ls<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^rbd_data.81d12ae8944a.'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>xargs<span class="w"> </span>-n<span class="w"> </span><span class="m">200</span><span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>rm
real<span class="w"> </span>0m10.073s
user<span class="w"> </span>0m1.244s
sys<span class="w"> </span>0m0.228s
</code></pre></div>
<p>To compare:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">time</span><span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbdbigsize
Removing<span class="w"> </span>image:<span class="w"> </span><span class="m">100</span>%<span class="w"> </span>complete...done.
real<span class="w"> </span>1123m5.409s
user<span class="w"> </span>686m19.924s
sys<span class="w"> </span>229m56.596s
</code></pre></div>
<p><img alt="Remove big RBD image" src="https://cephnotes.ksperis.com/images/img010.png"></p>
<h2>Fully initialized image :</h2>
<p>Of course, deleting a fully initialized image is faster with simple "rbd rm". :)</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>create<span class="w"> </span>rbdbigsize<span class="w"> </span>--size<span class="w"> </span><span class="m">1048576</span>
$<span class="w"> </span>rbd<span class="w"> </span>map<span class="w"> </span>rbdbigsize<span class="p">;</span><span class="w"> </span>dd<span class="w"> </span><span class="k">if</span><span class="o">=</span>/dev/zero<span class="w"> </span><span class="nv">of</span><span class="o">=</span>/dev/rbd0<span class="p">;</span><span class="w"> </span>rbd<span class="w"> </span>unmap<span class="w"> </span>/dev/rbd0<span class="p">;</span>
$<span class="w"> </span><span class="nb">time</span><span class="w"> </span><span class="o">(</span>rados<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>ls<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="sb">`</span>rbd<span class="w"> </span>info<span class="w"> </span>rbdbigsize<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>block_name_prefix<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>-o<span class="w"> </span><span class="s1">'rb.*$'</span><span class="sb">`</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>xargs<span class="w"> </span>-n<span class="w"> </span><span class="m">200</span><span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>rm<span class="p">;</span><span class="w"> </span>rados<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbdbigsize.rbd<span class="p">;</span><span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbdbigsize<span class="o">)</span>
real<span class="w"> </span>143m6.373s
user<span class="w"> </span>2m49.100s
sys<span class="w"> </span>0m22.080s
$<span class="w"> </span>rbd<span class="w"> </span>create<span class="w"> </span>rbdbigsize<span class="w"> </span>--size<span class="w"> </span><span class="m">1048576</span>
$<span class="w"> </span>rbd<span class="w"> </span>map<span class="w"> </span>rbdbigsize<span class="p">;</span><span class="w"> </span>dd<span class="w"> </span><span class="k">if</span><span class="o">=</span>/dev/zero<span class="w"> </span><span class="nv">of</span><span class="o">=</span>/dev/rbd0<span class="p">;</span><span class="w"> </span>rbd<span class="w"> </span>unmap<span class="w"> </span>/dev/rbd0<span class="p">;</span>
$<span class="w"> </span><span class="nb">time</span><span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbdbigsize
real<span class="w"> </span>88m44.121s
user<span class="w"> </span>1m5.464s
sys<span class="w"> </span>0m25.244s
</code></pre></div>
<p><img alt="Remove big RBD image" src="https://cephnotes.ksperis.com/images/img011.png"></p>Ceph Journal Migration2014-06-29T15:55:42+02:002014-06-29T15:55:42+02:00Laurent Barbetag:cephnotes.ksperis.com,2014-06-29:/blog/2014/06/29/ceph-journal-migration/<p>An exemple for migrating journal from file on the default location to a dedicated partition on /dev/sde1.</p>
<!-- more -->
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>hdparm
<span class="c1">### Stop osd</span>
$<span class="w"> </span>service<span class="w"> </span>ceph<span class="w"> </span>stop<span class="w"> </span>osd.0
<span class="c1">### Flush Journal</span>
$<span class="w"> </span>ceph-osd<span class="w"> </span>--flush-journal<span class="w"> </span>-i<span class="w"> </span><span class="m">0</span>
<span class="c1">### Create symlink to partition</span>
$<span class="w"> </span>rm<span class="w"> </span>/var/lib/ceph/osd/ceph-0/journal
$<span class="w"> </span>ln<span class="w"> </span>-s<span class="w"> </span>/dev/sde1<span class="w"> </span>/var …</code></pre></div><p>An exemple for migrating journal from file on the default location to a dedicated partition on /dev/sde1.</p>
<!-- more -->
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>hdparm
<span class="c1">### Stop osd</span>
$<span class="w"> </span>service<span class="w"> </span>ceph<span class="w"> </span>stop<span class="w"> </span>osd.0
<span class="c1">### Flush Journal</span>
$<span class="w"> </span>ceph-osd<span class="w"> </span>--flush-journal<span class="w"> </span>-i<span class="w"> </span><span class="m">0</span>
<span class="c1">### Create symlink to partition</span>
$<span class="w"> </span>rm<span class="w"> </span>/var/lib/ceph/osd/ceph-0/journal
$<span class="w"> </span>ln<span class="w"> </span>-s<span class="w"> </span>/dev/sde1<span class="w"> </span>/var/lib/ceph/osd/ceph-0/journal
<span class="c1">### Create new journal</span>
$<span class="w"> </span>ceph-osd<span class="w"> </span>--mkjournal<span class="w"> </span>-i<span class="w"> </span><span class="m">0</span>
$<span class="w"> </span>service<span class="w"> </span>ceph<span class="w"> </span>start<span class="w"> </span>osd.0
</code></pre></div>Ceph Versions2014-06-27T14:27:47+02:002014-06-27T14:27:47+02:00Laurent Barbetag:cephnotes.ksperis.com,2014-06-27:/blog/2014/06/27/ceph-versions/<p>6 months since my last post... Firefly is released.
http://ceph.com/releases/v0-80-firefly-released/</p>
<!--more-->
<h2>Ceph Major Release</h2>
<table>
<thead>
<tr>
<th style="text-align: left;">Version</th>
<th style="text-align: center;">Date</th>
<th style="text-align: left;">Some new features</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;">0.87 - Giant</td>
<td style="text-align: center;">11/2014</td>
<td style="text-align: left;">http://ceph.com/uncategorized/v0-87-giant-released/</td>
</tr>
<tr>
<td style="text-align: left;">0.80 - Firefly</td>
<td style="text-align: center;">07/2014</td>
<td style="text-align: left;">erasure coding, cache tiering</td>
</tr>
<tr>
<td style="text-align: left;">0.72 - Emperor</td>
<td style="text-align: center;">11/2013</td>
<td style="text-align: left;">multi-datacenter replication RGW, improved …</td></tr></tbody></table><p>6 months since my last post... Firefly is released.
http://ceph.com/releases/v0-80-firefly-released/</p>
<!--more-->
<h2>Ceph Major Release</h2>
<table>
<thead>
<tr>
<th style="text-align: left;">Version</th>
<th style="text-align: center;">Date</th>
<th style="text-align: left;">Some new features</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;">0.87 - Giant</td>
<td style="text-align: center;">11/2014</td>
<td style="text-align: left;">http://ceph.com/uncategorized/v0-87-giant-released/</td>
</tr>
<tr>
<td style="text-align: left;">0.80 - Firefly</td>
<td style="text-align: center;">07/2014</td>
<td style="text-align: left;">erasure coding, cache tiering</td>
</tr>
<tr>
<td style="text-align: left;">0.72 - Emperor</td>
<td style="text-align: center;">11/2013</td>
<td style="text-align: left;">multi-datacenter replication RGW, improved usability & performance</td>
</tr>
<tr>
<td style="text-align: left;">0.67 - Dumpling</td>
<td style="text-align: center;">08/2013</td>
<td style="text-align: left;">global namespace and region, REST Api for cluster management</td>
</tr>
<tr>
<td style="text-align: left;">0.61 - Cuttlefish</td>
<td style="text-align: center;">08/2013</td>
<td style="text-align: left;">ceph-deploy</td>
</tr>
<tr>
<td style="text-align: left;">0.56 - Bobtail</td>
<td style="text-align: center;">01/2013</td>
<td style="text-align: left;">improve stability / performance</td>
</tr>
<tr>
<td style="text-align: left;">0.48 - Argonaut</td>
<td style="text-align: center;">07/2012</td>
<td style="text-align: left;">improve stability / performance</td>
</tr>
</tbody>
</table>
<p>All Ceph Releases : http://ceph.com/category/releases/</p>
<h2>Packages versions included in Debian/Ubuntu distribution</h2>
<table>
<thead>
<tr>
<th style="text-align: left;">Distributions</th>
<th style="text-align: left;">Ceph Version</th>
<th style="text-align: left;">Linux Kernel</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;">Ubuntu Utopic 14.10</td>
<td style="text-align: left;">0.80 (Firefly)</td>
<td style="text-align: left;">3.15</td>
</tr>
<tr>
<td style="text-align: left;">Ubuntu Trusty 14.04LTS</td>
<td style="text-align: left;">0.79 - 0.80 (trusty-updates)</td>
<td style="text-align: left;">3.13</td>
</tr>
<tr>
<td style="text-align: left;">Ubuntu Saucy 13.10</td>
<td style="text-align: left;">0.67 (Dumpling)</td>
<td style="text-align: left;">3.11</td>
</tr>
<tr>
<td style="text-align: left;">Ubuntu Raring 13.04</td>
<td style="text-align: left;">0.56 (Bobtail)</td>
<td style="text-align: left;">3.8</td>
</tr>
<tr>
<td style="text-align: left;">Ubuntu Precise 12.04LTS</td>
<td style="text-align: left;">0.41</td>
<td style="text-align: left;">3.2</td>
</tr>
<tr>
<td style="text-align: left;">Debian Jessie</td>
<td style="text-align: left;">0.80 (Firefly)</td>
<td style="text-align: left;">3.14</td>
</tr>
<tr>
<td style="text-align: left;">Debian Wheezy [backports]</td>
<td style="text-align: left;"> 0.80 (Firefly)</td>
<td style="text-align: left;">3.2.41</td>
</tr>
<tr>
<td style="text-align: left;">Debian Wheezy</td>
<td style="text-align: left;"> -</td>
<td style="text-align: left;">3.2.41</td>
</tr>
<tr>
<td style="text-align: left;">Debian Squeeze</td>
<td style="text-align: left;"> -</td>
<td style="text-align: left;">2.6.32</td>
</tr>
</tbody>
</table>
<p>For the latest versions for each distribution: http://ceph.com/resources/downloads/</p>Feature set mismatch error on Ceph kernel client2014-01-21T09:40:53+01:002014-01-21T09:40:53+01:00Laurent Barbetag:cephnotes.ksperis.com,2014-01-21:/blog/2014/01/21/feature-set-mismatch-error-on-ceph-kernel-client/<p>Depending on the kernel version you may be missing some features required by the cluster (or vice versa, required by client but not available on the cluster). Hence the errors "feature set mismatch".</p>
<!-- more -->
<p>Some examples of errors that can be encountered :</p>
<div class="highlight"><pre><span></span><code>mon0 192.168.0.1:6789 feature set mismatch …</code></pre></div><p>Depending on the kernel version you may be missing some features required by the cluster (or vice versa, required by client but not available on the cluster). Hence the errors "feature set mismatch".</p>
<!-- more -->
<p>Some examples of errors that can be encountered :</p>
<div class="highlight"><pre><span></span><code>mon0 192.168.0.1:6789 feature set mismatch, my XXXXXX < server's XXXXXX, missing 2040000
</code></pre></div>
<p>--> Upgrade kernel client up to 3.9 or set tunables to legacy : <code>ceph osd crush tunables legacy</code></p>
<div class="highlight"><pre><span></span><code>mon0 192.168.0.1:6789 feature set mismatch, my XXXXXX < server's XXXXXX, missing 40000000
</code></pre></div>
<p>--> Upgrade kernel client up to 3.9 or unset hashpspool : <code>ceph osd pool set rbd hashpspool false</code></p>
<div class="highlight"><pre><span></span><code>mon0 192.168.0.1:6789 feature set mismatch, my XXXXXX < server's XXXXXX, missing 800000000
</code></pre></div>
<p>--> Remove cache pool and reload monitors or upgrade kernel client up to 3.14</p>
<div class="highlight"><pre><span></span><code>mon0 192.168.0.1:6789 feature set mismatch, my XXXXXX < server's XXXXXX, missing 1000000000
</code></pre></div>
<p>--> Upgrade kernel client up to 3.14</p>
<div class="highlight"><pre><span></span><code>mon0 192.168.0.1:6789 feature set mismatch, my 4a042a42 < server's 2004a042a42, missing 20000000000
</code></pre></div>
<p>--> Upgrade kernel client up to 3.15 or disable tunable 3 features</p>
<h2>CEPH_FEATURE Table and Kernel Version</h2>
<p>You can find the feature missing in that table :</p>
<p>For exemple, <code>missing 2040000</code> means that CEPH_FEATURE_CRUSH_TUNABLES (40000) and CEPH_FEATURE_CRUSH_TUNABLES2 (2000000) is missing on kernel client.</p>
<p>'R':required, 'S':support, '-X-' feature is new since this version</p>
<table>
<thead>
<tr>
<th style="text-align: left;">Feature</th>
<th style="text-align: center;">BIT</th>
<th style="text-align: right;">OCT</th>
<th style="text-align: center;">3.8</th>
<th style="text-align: center;">3.9</th>
<th style="text-align: center;">3.10</th>
<th style="text-align: center;">3.14</th>
<th style="text-align: center;">3.15</th>
<th style="text-align: center;">3.18</th>
<th style="text-align: center;">4.1</th>
<th style="text-align: center;">4.5</th>
<th style="text-align: center;">4.6</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;">CEPH_FEATURE_NOSRCADDR</td>
<td style="text-align: center;">1</td>
<td style="text-align: right;">2</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_SUBSCRIBE2</td>
<td style="text-align: center;">4</td>
<td style="text-align: right;">10</td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;">-R-</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_RECONNECT_SEQ</td>
<td style="text-align: center;">6</td>
<td style="text-align: right;">40</td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;">-R-</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_PGID64</td>
<td style="text-align: center;">9</td>
<td style="text-align: right;">200</td>
<td style="text-align: center;"></td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_PGPOOL3</td>
<td style="text-align: center;">11</td>
<td style="text-align: right;">800</td>
<td style="text-align: center;"></td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_OSDENC</td>
<td style="text-align: center;">13</td>
<td style="text-align: right;">2000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_CRUSH_TUNABLES</td>
<td style="text-align: center;">18</td>
<td style="text-align: right;">40000</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_MSG_AUTH</td>
<td style="text-align: center;">23</td>
<td style="text-align: right;">800000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;">-S-</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_CRUSH_TUNABLES2</td>
<td style="text-align: center;">25</td>
<td style="text-align: right;">2000000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_REPLY_CREATE_INODE</td>
<td style="text-align: center;">27</td>
<td style="text-align: right;">8000000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_OSDHASHPSPOOL</td>
<td style="text-align: center;">30</td>
<td style="text-align: right;">40000000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_OSD_CACHEPOOL</td>
<td style="text-align: center;">35</td>
<td style="text-align: right;">800000000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;">-S-</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_CRUSH_V2</td>
<td style="text-align: center;">36</td>
<td style="text-align: right;">1000000000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;">-S-</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_EXPORT_PEER</td>
<td style="text-align: center;">37</td>
<td style="text-align: right;">2000000000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;">-S-</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_OSD_ERASURE_CODES***</td>
<td style="text-align: center;">38</td>
<td style="text-align: right;">4000000000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_OSDMAP_ENC</td>
<td style="text-align: center;">39</td>
<td style="text-align: right;">8000000000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;">-S-</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_CRUSH_TUNABLES3</td>
<td style="text-align: center;">41</td>
<td style="text-align: right;">20000000000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;">-S-</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_OSD_PRIMARY_AFFINITY</td>
<td style="text-align: center;">41*</td>
<td style="text-align: right;">20000000000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;">-S-</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_CRUSH_V4 ****</td>
<td style="text-align: center;">48</td>
<td style="text-align: right;">1000000000000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;">-S-</td>
<td style="text-align: center;">S</td>
<td style="text-align: center;">S</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_CRUSH_TUNABLES5</td>
<td style="text-align: center;">58</td>
<td style="text-align: right;">200000000000000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;">-S-</td>
<td style="text-align: center;">S</td>
</tr>
<tr>
<td style="text-align: left;">CEPH_FEATURE_NEW_OSDOPREPLY_ENCODING</td>
<td style="text-align: center;">58*</td>
<td style="text-align: right;">400000000000000</td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;"></td>
<td style="text-align: center;">-S-</td>
<td style="text-align: center;">S</td>
</tr>
</tbody>
</table>
<p>(*** Note that CEPH_FEATURE_OSD_ERASURE_CODES is no more required for client since firefly)</p>
<p>(**** staw2 : http://tracker.ceph.com/issues/11364)</p>Set tunables optimal on Ceph crushmap2014-01-16T15:05:12+01:002014-01-16T15:05:12+01:00Laurent Barbetag:cephnotes.ksperis.com,2014-01-16:/blog/2014/01/16/set-tunables-optimal-on-ceph-crushmap/<p>Il you just upgrade to Ceph 0.75, you can see this message :</p>
<p>$ ceph health
HEALTH_WARN crush map has non-optimal tunables</p>
<p>For more explanation, everything is explained in the documentation:
http://ceph.com/docs/master/rados/operations/crush-map/#warning-when-tunables-are-non-optimal</p>
<!-- more -->
<p>Before doing this, read the documentation !</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>crush<span class="w"> </span>tunables<span class="w"> </span>optimal …</code></pre></div><p>Il you just upgrade to Ceph 0.75, you can see this message :</p>
<p>$ ceph health
HEALTH_WARN crush map has non-optimal tunables</p>
<p>For more explanation, everything is explained in the documentation:
http://ceph.com/docs/master/rados/operations/crush-map/#warning-when-tunables-are-non-optimal</p>
<!-- more -->
<p>Before doing this, read the documentation !</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>crush<span class="w"> </span>tunables<span class="w"> </span>optimal
</code></pre></div>
<p>Verify the change in the crushmap :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>getcrushmap<span class="w"> </span>-o<span class="w"> </span>crushmap_optimal.bin
got<span class="w"> </span>crush<span class="w"> </span>map<span class="w"> </span>from<span class="w"> </span>osdmap<span class="w"> </span>epoch<span class="w"> </span><span class="m">186</span>
$<span class="w"> </span>crushtool<span class="w"> </span>-d<span class="w"> </span>crushmap_optimal.bin<span class="w"> </span>-o<span class="w"> </span>crushmap_optimal.txt
$<span class="w"> </span>head<span class="w"> </span>-n6<span class="w"> </span>crushmap_optimal.txt
<span class="c1"># begin crush map</span>
tunable<span class="w"> </span>choose_local_tries<span class="w"> </span><span class="m">0</span>
tunable<span class="w"> </span>choose_local_fallback_tries<span class="w"> </span><span class="m">0</span>
tunable<span class="w"> </span>choose_total_tries<span class="w"> </span><span class="m">50</span>
tunable<span class="w"> </span>chooseleaf_descend_once<span class="w"> </span><span class="m">1</span>
</code></pre></div>
<p>For more details about these options, you can look at the docmention :
http://ceph.com/docs/master/rados/operations/crush-map/#crush-tunables</p>
<p _="%" endblockquote>{% blockquote %}
- choose_local_tries: Number of local retries. Legacy value is 2, optimal value is 0.
- choose_local_fallback_tries: Legacy value is 5, optimal value is 0.
- choose_total_tries: Total number of attempts to choose an item. Legacy value was 19, subsequent testing indicates that a value of 50 is more appropriate for typical clusters. For extremely large clusters, a larger value might be necessary.
- chooseleaf_descend_once: Whether a recursive chooseleaf attempt will retry, or only try once and allow the original placement to retry. Legacy default is 0, optimal value is 1.</p>
<p>Note that, before update the crushmap you must ensure that clients are compatible with these options. For example, an incompatible kernel will cause "feature set mismatch" in kernel logs.
If you dont want to enable optimal tunables, you can disable this warning with the option <code>mon warn on legacy crush tunables = false</code> in [mon] section in ceph.conf.</p>RBD client kernel 3.10.262014-01-10T09:29:32+01:002014-01-10T09:29:32+01:00Laurent Barbetag:cephnotes.ksperis.com,2014-01-10:/blog/2014/01/10/rbd-client-kernel-3-dot-10-dot-26/<p>I take opportunity to the new kernel release 3.10.26 to advise the person using the client rbd kernel update to this version. Indeed a number of fixes have been applied to this version making use of rbd kernel particularly stable.</p>
<!--more-->
<p>http://kernel.org</p>
<p>Patches applied :</p>
<div class="highlight"><pre><span></span><code><span class="n">Josh</span><span class="w"> </span><span class="n">Durgin</span><span class="w"> </span><span class="o"><</span><span class="n">josh …</span></code></pre></div><p>I take opportunity to the new kernel release 3.10.26 to advise the person using the client rbd kernel update to this version. Indeed a number of fixes have been applied to this version making use of rbd kernel particularly stable.</p>
<!--more-->
<p>http://kernel.org</p>
<p>Patches applied :</p>
<div class="highlight"><pre><span></span><code><span class="n">Josh</span><span class="w"> </span><span class="n">Durgin</span><span class="w"> </span><span class="o"><</span><span class="n">josh</span><span class="o">.</span><span class="n">durgin</span><span class="err">@</span><span class="n">inktank</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">rbd</span><span class="p">:</span><span class="w"> </span><span class="n">fix</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="n">handling</span><span class="w"> </span><span class="n">from</span><span class="w"> </span><span class="n">rbd_snap_name</span><span class="p">()</span>
<span class="n">Josh</span><span class="w"> </span><span class="n">Durgin</span><span class="w"> </span><span class="o"><</span><span class="n">josh</span><span class="o">.</span><span class="n">durgin</span><span class="err">@</span><span class="n">inktank</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">rbd</span><span class="p">:</span><span class="w"> </span><span class="n">ignore</span><span class="w"> </span><span class="n">unmapped</span><span class="w"> </span><span class="n">snapshots</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">no</span><span class="w"> </span><span class="n">longer</span><span class="w"> </span><span class="n">exist</span>
<span class="n">Josh</span><span class="w"> </span><span class="n">Durgin</span><span class="w"> </span><span class="o"><</span><span class="n">josh</span><span class="o">.</span><span class="n">durgin</span><span class="err">@</span><span class="n">inktank</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">rbd</span><span class="p">:</span><span class="w"> </span><span class="n">fix</span><span class="w"> </span><span class="n">use</span><span class="o">-</span><span class="n">after</span><span class="w"> </span><span class="n">free</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">rbd_dev</span><span class="o">-></span><span class="n">disk</span>
<span class="n">Josh</span><span class="w"> </span><span class="n">Durgin</span><span class="w"> </span><span class="o"><</span><span class="n">josh</span><span class="o">.</span><span class="n">durgin</span><span class="err">@</span><span class="n">inktank</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">rbd</span><span class="p">:</span><span class="w"> </span><span class="n">make</span><span class="w"> </span><span class="n">rbd_obj_notify_ack</span><span class="p">()</span><span class="w"> </span><span class="n">synchronous</span>
<span class="n">Josh</span><span class="w"> </span><span class="n">Durgin</span><span class="w"> </span><span class="o"><</span><span class="n">josh</span><span class="o">.</span><span class="n">durgin</span><span class="err">@</span><span class="n">inktank</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">rbd</span><span class="p">:</span><span class="w"> </span><span class="n">complete</span><span class="w"> </span><span class="n">notifies</span><span class="w"> </span><span class="n">before</span><span class="w"> </span><span class="n">cleaning</span><span class="w"> </span><span class="n">up</span><span class="w"> </span><span class="n">osd_client</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">rbd_dev</span>
<span class="n">Josh</span><span class="w"> </span><span class="n">Durgin</span><span class="w"> </span><span class="o"><</span><span class="n">josh</span><span class="o">.</span><span class="n">durgin</span><span class="err">@</span><span class="n">inktank</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">libceph</span><span class="p">:</span><span class="w"> </span><span class="n">add</span><span class="w"> </span><span class="n">function</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">ensure</span><span class="w"> </span><span class="n">notifies</span><span class="w"> </span><span class="n">are</span><span class="w"> </span><span class="n">complete</span>
<span class="n">Josh</span><span class="w"> </span><span class="n">Durgin</span><span class="w"> </span><span class="o"><</span><span class="n">josh</span><span class="o">.</span><span class="n">durgin</span><span class="err">@</span><span class="n">inktank</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">rbd</span><span class="p">:</span><span class="w"> </span><span class="n">fix</span><span class="w"> </span><span class="nb nb-Type">null</span><span class="w"> </span><span class="n">dereference</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">dout</span>
<span class="n">Josh</span><span class="w"> </span><span class="n">Durgin</span><span class="w"> </span><span class="o"><</span><span class="n">josh</span><span class="o">.</span><span class="n">durgin</span><span class="err">@</span><span class="n">inktank</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">rbd</span><span class="p">:</span><span class="w"> </span><span class="n">fix</span><span class="w"> </span><span class="n">buffer</span><span class="w"> </span><span class="n">size</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">writes</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">images</span><span class="w"> </span><span class="n">with</span><span class="w"> </span><span class="n">snapshots</span>
<span class="n">majianpeng</span><span class="w"> </span><span class="o"><</span><span class="n">majianpeng</span><span class="err">@</span><span class="n">gmail</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">ceph</span><span class="p">:</span><span class="w"> </span><span class="n">allow</span><span class="w"> </span><span class="n">sync_read</span><span class="o">/</span><span class="n">write</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">partial</span><span class="w"> </span><span class="n">successed</span><span class="w"> </span><span class="n">size</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">read</span><span class="o">/</span><span class="n">write</span><span class="o">.</span>
<span class="n">majianpeng</span><span class="w"> </span><span class="o"><</span><span class="n">majianpeng</span><span class="err">@</span><span class="n">gmail</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">ceph</span><span class="p">:</span><span class="w"> </span><span class="n">fix</span><span class="w"> </span><span class="n">bugs</span><span class="w"> </span><span class="n">about</span><span class="w"> </span><span class="n">handling</span><span class="w"> </span><span class="n">short</span><span class="o">-</span><span class="n">read</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">sync</span><span class="w"> </span><span class="n">read</span><span class="w"> </span><span class="n">mode</span><span class="o">.</span>
<span class="n">Dan</span><span class="w"> </span><span class="n">Carpenter</span><span class="w"> </span><span class="o"><</span><span class="n">dan</span><span class="o">.</span><span class="n">carpenter</span><span class="err">@</span><span class="n">oracle</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">libceph</span><span class="p">:</span><span class="w"> </span><span class="n">create_singlethread_workqueue</span><span class="p">()</span><span class="w"> </span><span class="n">doesn</span><span class="s1">'t return ERR_PTRs</span>
<span class="n">Dan</span><span class="w"> </span><span class="n">Carpenter</span><span class="w"> </span><span class="o"><</span><span class="n">dan</span><span class="o">.</span><span class="n">carpenter</span><span class="err">@</span><span class="n">oracle</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">libceph</span><span class="p">:</span><span class="w"> </span><span class="n">potential</span><span class="w"> </span><span class="n">NULL</span><span class="w"> </span><span class="n">dereference</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">ceph_osdc_handle_map</span><span class="p">()</span>
<span class="n">Dan</span><span class="w"> </span><span class="n">Carpenter</span><span class="w"> </span><span class="o"><</span><span class="n">dan</span><span class="o">.</span><span class="n">carpenter</span><span class="err">@</span><span class="n">oracle</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">libceph</span><span class="p">:</span><span class="w"> </span><span class="n">fix</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="n">handling</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">handle_reply</span><span class="p">()</span>
<span class="n">majianpeng</span><span class="w"> </span><span class="o"><</span><span class="n">majianpeng</span><span class="err">@</span><span class="n">gmail</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">ceph</span><span class="p">:</span><span class="w"> </span><span class="n">Add</span><span class="w"> </span><span class="n">check</span><span class="w"> </span><span class="n">returned</span><span class="w"> </span><span class="n">value</span><span class="w"> </span><span class="n">on</span><span class="w"> </span><span class="k">func</span><span class="w"> </span><span class="n">ceph_calc_ceph_pg</span><span class="o">.</span>
<span class="n">Dan</span><span class="w"> </span><span class="n">Carpenter</span><span class="w"> </span><span class="o"><</span><span class="n">dan</span><span class="o">.</span><span class="n">carpenter</span><span class="err">@</span><span class="n">oracle</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">ceph</span><span class="p">:</span><span class="w"> </span><span class="n">cleanup</span><span class="w"> </span><span class="n">types</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">striped_read</span><span class="p">()</span>
<span class="n">Nathaniel</span><span class="w"> </span><span class="n">Yazdani</span><span class="w"> </span><span class="o"><</span><span class="n">n1ght</span><span class="o">.</span><span class="mi">4</span><span class="n">nd</span><span class="o">.</span><span class="n">d4y</span><span class="err">@</span><span class="n">gmail</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">ceph</span><span class="p">:</span><span class="w"> </span><span class="n">fix</span><span class="w"> </span><span class="nb nb-Type">null</span><span class="w"> </span><span class="n">pointer</span><span class="w"> </span><span class="n">dereference</span>
<span class="n">Yan</span><span class="p">,</span><span class="w"> </span><span class="n">Zheng</span><span class="w"> </span><span class="o"><</span><span class="n">zheng</span><span class="o">.</span><span class="n">z</span><span class="o">.</span><span class="n">yan</span><span class="err">@</span><span class="n">intel</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">libceph</span><span class="p">:</span><span class="w"> </span><span class="n">call</span><span class="w"> </span><span class="n">r_unsafe_callback</span><span class="w"> </span><span class="n">when</span><span class="w"> </span><span class="n">unsafe</span><span class="w"> </span><span class="n">reply</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">received</span>
<span class="n">Sasha</span><span class="w"> </span><span class="n">Levin</span><span class="w"> </span><span class="o"><</span><span class="n">sasha</span><span class="o">.</span><span class="n">levin</span><span class="err">@</span><span class="n">oracle</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">ceph</span><span class="p">:</span><span class="w"> </span><span class="n">avoid</span><span class="w"> </span><span class="n">accessing</span><span class="w"> </span><span class="n">invalid</span><span class="w"> </span><span class="n">memory</span>
<span class="n">majianpeng</span><span class="w"> </span><span class="o"><</span><span class="n">majianpeng</span><span class="err">@</span><span class="n">gmail</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">ceph</span><span class="p">:</span><span class="w"> </span><span class="n">Free</span><span class="w"> </span><span class="n">mdsc</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">alloc</span><span class="w"> </span><span class="n">mdsc</span><span class="o">-></span><span class="n">mdsmap</span><span class="w"> </span><span class="n">failed</span><span class="o">.</span>
<span class="n">Sage</span><span class="w"> </span><span class="n">Weil</span><span class="w"> </span><span class="o"><</span><span class="n">sage</span><span class="err">@</span><span class="n">inktank</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">rbd</span><span class="p">:</span><span class="w"> </span><span class="n">fix</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">couple</span><span class="w"> </span><span class="n">warnings</span>
<span class="n">Yan</span><span class="p">,</span><span class="w"> </span><span class="n">Zheng</span><span class="w"> </span><span class="o"><</span><span class="n">zheng</span><span class="o">.</span><span class="n">z</span><span class="o">.</span><span class="n">yan</span><span class="err">@</span><span class="n">intel</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">libceph</span><span class="p">:</span><span class="w"> </span><span class="n">fix</span><span class="w"> </span><span class="n">truncate</span><span class="w"> </span><span class="n">size</span><span class="w"> </span><span class="n">calculation</span>
<span class="n">Yan</span><span class="p">,</span><span class="w"> </span><span class="n">Zheng</span><span class="w"> </span><span class="o"><</span><span class="n">zheng</span><span class="o">.</span><span class="n">z</span><span class="o">.</span><span class="n">yan</span><span class="err">@</span><span class="n">intel</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">libceph</span><span class="p">:</span><span class="w"> </span><span class="n">fix</span><span class="w"> </span><span class="n">safe</span><span class="w"> </span><span class="n">completion</span>
<span class="n">Alex</span><span class="w"> </span><span class="n">Elder</span><span class="w"> </span><span class="o"><</span><span class="n">elder</span><span class="err">@</span><span class="n">inktank</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">rbd</span><span class="p">:</span><span class="w"> </span><span class="n">protect</span><span class="w"> </span><span class="n">against</span><span class="w"> </span><span class="n">concurrent</span><span class="w"> </span><span class="n">unmaps</span>
<span class="n">Alex</span><span class="w"> </span><span class="n">Elder</span><span class="w"> </span><span class="o"><</span><span class="n">elder</span><span class="err">@</span><span class="n">inktank</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">rbd</span><span class="p">:</span><span class="w"> </span><span class="n">set</span><span class="w"> </span><span class="n">removing</span><span class="w"> </span><span class="n">flag</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="n">holding</span><span class="w"> </span><span class="n">list</span><span class="w"> </span><span class="n">lock</span>
<span class="n">Alex</span><span class="w"> </span><span class="n">Elder</span><span class="w"> </span><span class="o"><</span><span class="n">elder</span><span class="err">@</span><span class="n">inktank</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">rbd</span><span class="p">:</span><span class="w"> </span><span class="n">flush</span><span class="w"> </span><span class="n">dcache</span><span class="w"> </span><span class="n">after</span><span class="w"> </span><span class="n">zeroing</span><span class="w"> </span><span class="n">page</span><span class="w"> </span><span class="n">data</span>
<span class="n">Alex</span><span class="w"> </span><span class="n">Elder</span><span class="w"> </span><span class="o"><</span><span class="n">elder</span><span class="err">@</span><span class="n">inktank</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">libceph</span><span class="p">:</span><span class="w"> </span><span class="n">add</span><span class="w"> </span><span class="n">lingering</span><span class="w"> </span><span class="n">request</span><span class="w"> </span><span class="n">reference</span><span class="w"> </span><span class="n">when</span><span class="w"> </span><span class="n">registered</span>
<span class="n">Emil</span><span class="w"> </span><span class="n">Goode</span><span class="w"> </span><span class="o"><</span><span class="n">emilgoode</span><span class="err">@</span><span class="n">gmail</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
<span class="w"> </span><span class="n">ceph</span><span class="p">:</span><span class="w"> </span><span class="n">improve</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="n">handling</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">ceph_mdsmap_decode</span>
</code></pre></div>
<p>Thank you to the developers.</p>Map rbd kernel without install ceph-common2014-01-09T11:03:55+01:002014-01-09T11:03:55+01:00Laurent Barbetag:cephnotes.ksperis.com,2014-01-09:/blog/2014/01/09/map-rbd-kernel-without-install-ceph-common/<p>It is not mandatory to install Ceph binaries (especially ceph-common) to be able to map a Rbd blocdevice you can also use the kernel path /sys/bus/rbd provided by rbd module.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>modprobe<span class="w"> </span>rbd
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"192.168.0.1,192.168.0.2,192.168.0.3 name=admin,secret …</span></code></pre></div><p>It is not mandatory to install Ceph binaries (especially ceph-common) to be able to map a Rbd blocdevice you can also use the kernel path /sys/bus/rbd provided by rbd module.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>modprobe<span class="w"> </span>rbd
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"192.168.0.1,192.168.0.2,192.168.0.3 name=admin,secret=AQBG5SlSmLELKBAA6sKvuJyRWUmFl2R5E1ukTw== rbd testrbd"</span><span class="w"> </span>><span class="w"> </span>/sys/bus/rbd/add
</code></pre></div>
<!--more-->
<h1>Startup rc script</h1>
<p>For map rbd on startup and unmap on shutdown you can use this modified rc script <a href="https://cephnotes.ksperis.com/downloads/rbdmap">http://cephnotes.ksperis.com/downloads/rbdmap</a> </p>
<p>You do not need ceph-common :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-cache<span class="w"> </span>policy<span class="w"> </span>ceph-common
ceph-common:
<span class="w"> </span>Installed:<span class="w"> </span><span class="o">(</span>none<span class="o">)</span>
<span class="w"> </span>CandiDate:<span class="w"> </span><span class="m">0</span>.67.4-0ubuntu2
<span class="w"> </span>Version<span class="w"> </span>table:
<span class="w"> </span><span class="m">0</span>.67.4-0ubuntu2<span class="w"> </span><span class="m">0</span>
<span class="w"> </span><span class="m">500</span><span class="w"> </span>http://fr.archive.ubuntu.com/ubuntu/<span class="w"> </span>saucy/main<span class="w"> </span>amd64<span class="w"> </span>Packages
</code></pre></div>
<p>You need to have a ceph.conf file with mon_host var :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>egrep<span class="w"> </span><span class="s1">'mon[ _]host'</span><span class="w"> </span>/etc/ceph/ceph.conf
<span class="nv">mon_host</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">192</span>.168.0.1,192.168.0.2,192.168.0.3
</code></pre></div>
<p>And /etc/ceph/rbdmap file with this format :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>/etc/ceph/rbdmap
rbd/testrbd<span class="w"> </span><span class="nv">id</span><span class="o">=</span>admin,secret<span class="o">=</span><span class="nv">AQBG5SlSmLELKBAA6sKvuJyRWUmFl2R5E1ukTw</span><span class="o">==</span>
</code></pre></div>
<p>Be carefull tu use "id=" and "secret=" (not user, key or keyfile...)
Then, download this modify rc script :</p>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="w"> </span><span class="n">wget</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">cephnotes</span><span class="o">.</span><span class="n">ksperis</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">downloads</span><span class="o">/</span><span class="n">rbdmap</span><span class="w"> </span><span class="o">-</span><span class="n">O</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">init</span><span class="o">.</span><span class="n">d</span><span class="o">/</span><span class="n">rbdmap</span>
<span class="o">$</span><span class="w"> </span><span class="n">chmod</span><span class="w"> </span><span class="o">+</span><span class="n">x</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">init</span><span class="o">.</span><span class="n">d</span><span class="o">/</span><span class="n">rbdmap</span>
<span class="o">$</span><span class="w"> </span><span class="n">update</span><span class="o">-</span><span class="n">rc</span><span class="o">.</span><span class="n">d</span><span class="w"> </span><span class="n">rbdmap</span><span class="w"> </span><span class="n">defaults</span>
</code></pre></div>
<p>Test :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>/etc/init.d/rbdmap<span class="w"> </span>start
*<span class="w"> </span>Starting<span class="w"> </span>RBD<span class="w"> </span>Mapping<span class="w"> </span><span class="o">[</span><span class="w"> </span>OK<span class="w"> </span><span class="o">]</span>
*<span class="w"> </span>Mounting<span class="w"> </span>all<span class="w"> </span>filesystems...<span class="w"> </span><span class="o">[</span><span class="w"> </span>OK<span class="w"> </span><span class="o">]</span>
$<span class="w"> </span>ls<span class="w"> </span>/dev/rbd/*
testrbd
$<span class="w"> </span>/etc/init.d/rbdmap<span class="w"> </span>stop
*<span class="w"> </span>Stopping<span class="w"> </span>RBD<span class="w"> </span>Mapping<span class="w"> </span><span class="o">[</span><span class="w"> </span>OK<span class="w"> </span><span class="o">]</span>
$<span class="w"> </span>ls<span class="w"> </span>/dev/rbd/*
ls:<span class="w"> </span>cannot<span class="w"> </span>access<span class="w"> </span>/dev/rbd/*:<span class="w"> </span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory
</code></pre></div>Ceph osd reweight2013-12-09T20:30:00+01:002013-12-09T20:30:00+01:00Laurent Barbetag:cephnotes.ksperis.com,2013-12-09:/blog/2013/12/09/ceph-osd-reweight/<div class="highlight"><pre><span></span><code>ceph health
HEALTH_WARN 1 near full osd(s)
</code></pre></div>
<p>Arrhh, Trying to optimize a little weight given to the OSD.
Rebalancing load between osd seems to be easy, but do not always go as we would like...</p>
<!--more-->
<h1>Increase osd weight</h1>
<p>Before operation get the map of Placement Groups.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>pg<span class="w"> </span>dump …</code></pre></div><div class="highlight"><pre><span></span><code>ceph health
HEALTH_WARN 1 near full osd(s)
</code></pre></div>
<p>Arrhh, Trying to optimize a little weight given to the OSD.
Rebalancing load between osd seems to be easy, but do not always go as we would like...</p>
<!--more-->
<h1>Increase osd weight</h1>
<p>Before operation get the map of Placement Groups.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span>><span class="w"> </span>/tmp/pg_dump.1
</code></pre></div>
<p>Let's go slowly, we will increase the weight of osd.13 with a step of 0.05.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>tree<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>osd.13
<span class="m">13</span><span class="w"> </span><span class="m">3</span><span class="w"> </span>osd.13<span class="w"> </span>up<span class="w"> </span><span class="m">1</span>
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>crush<span class="w"> </span>reweight<span class="w"> </span>osd.13<span class="w"> </span><span class="m">3</span>.05
reweighted<span class="w"> </span>item<span class="w"> </span>id<span class="w"> </span><span class="m">13</span><span class="w"> </span>name<span class="w"> </span><span class="s1">'osd.13'</span><span class="w"> </span>to<span class="w"> </span><span class="m">3</span>.05<span class="w"> </span><span class="k">in</span><span class="w"> </span>crush<span class="w"> </span>map
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>tree<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>osd.13
<span class="m">13</span><span class="w"> </span><span class="m">3</span>.05<span class="w"> </span>osd.13<span class="w"> </span>up<span class="w"> </span><span class="m">1</span>
</code></pre></div>
<p>The new weight has been changed in the crushmap. Look at what is happening in the cluster.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>health<span class="w"> </span>details
HEALTH_WARN<span class="w"> </span><span class="m">2</span><span class="w"> </span>pgs<span class="w"> </span>backfilling<span class="p">;</span><span class="w"> </span><span class="m">2</span><span class="w"> </span>pgs<span class="w"> </span>stuck<span class="w"> </span>unclean<span class="p">;</span><span class="w"> </span>recovery<span class="w"> </span><span class="m">16884</span>/9154554<span class="w"> </span>degraded<span class="w"> </span><span class="o">(</span><span class="m">0</span>.184%<span class="o">)</span>
pg<span class="w"> </span><span class="m">3</span>.183<span class="w"> </span>is<span class="w"> </span>stuck<span class="w"> </span>unclean<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="m">434</span>.029986,<span class="w"> </span>current<span class="w"> </span>state<span class="w"> </span>active+remapped+backfilling,<span class="w"> </span>last<span class="w"> </span>acting<span class="w"> </span><span class="o">[</span><span class="m">1</span>,13,5<span class="o">]</span>
pg<span class="w"> </span><span class="m">3</span>.83<span class="w"> </span>is<span class="w"> </span>stuck<span class="w"> </span>unclean<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="m">2479</span>.504088,<span class="w"> </span>current<span class="w"> </span>state<span class="w"> </span>active+remapped+backfilling,<span class="w"> </span>last<span class="w"> </span>acting<span class="w"> </span><span class="o">[</span><span class="m">5</span>,13,12<span class="o">]</span>
pg<span class="w"> </span><span class="m">3</span>.183<span class="w"> </span>is<span class="w"> </span>active+remapped+backfilling,<span class="w"> </span>acting<span class="w"> </span><span class="o">[</span><span class="m">1</span>,13,5<span class="o">]</span>
pg<span class="w"> </span><span class="m">3</span>.83<span class="w"> </span>is<span class="w"> </span>active+remapped+backfilling,<span class="w"> </span>acting<span class="w"> </span><span class="o">[</span><span class="m">5</span>,13,12<span class="o">]</span>
recovery<span class="w"> </span><span class="m">16884</span>/9154554<span class="w"> </span>degraded<span class="w"> </span><span class="o">(</span><span class="m">0</span>.184%<span class="o">)</span>
</code></pre></div>
<p>Well, pg 3.183 and 3.83 is in active+remapped+backfilling state :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>pg<span class="w"> </span>map<span class="w"> </span><span class="m">3</span>.183
osdmap<span class="w"> </span>e4588<span class="w"> </span>pg<span class="w"> </span><span class="m">3</span>.183<span class="w"> </span><span class="o">(</span><span class="m">3</span>.183<span class="o">)</span><span class="w"> </span>-><span class="w"> </span>up<span class="w"> </span><span class="o">[</span><span class="m">1</span>,13<span class="o">]</span><span class="w"> </span>acting<span class="w"> </span><span class="o">[</span><span class="m">1</span>,13,5<span class="o">]</span>
$<span class="w"> </span>ceph<span class="w"> </span>pg<span class="w"> </span>map<span class="w"> </span><span class="m">3</span>.83
osdmap<span class="w"> </span>e4588<span class="w"> </span>pg<span class="w"> </span><span class="m">3</span>.83<span class="w"> </span><span class="o">(</span><span class="m">3</span>.83<span class="o">)</span><span class="w"> </span>-><span class="w"> </span>up<span class="w"> </span><span class="o">[</span><span class="m">13</span>,5<span class="o">]</span><span class="w"> </span>acting<span class="w"> </span><span class="o">[</span><span class="m">5</span>,13,12<span class="o">]</span>
</code></pre></div>
<p>In this case, we can see that osd with id 13 has been added for this two placement groups. Pg 3.183 and 3.83 will respectively remove from osd 5 and 12.
If we have a look on osd bandwidth, we can see those transfert osd.1 --> osd.13 and osd.5 --> osd.13 :</p>
<p><img alt="Ceph osd reweight" src="https://cephnotes.ksperis.com/images/img003.png">
<img alt="Ceph osd reweight" src="https://cephnotes.ksperis.com/images/img004.png">
<img alt="Ceph osd reweight" src="https://cephnotes.ksperis.com/images/img005.png"></p>
<p>OSD 1 and 5 are primary for pg 3.183 and 3.83 (see acting table) and OSD 13 is writing.</p>
<p>I wait that cluster has finished. Then,</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span>><span class="w"> </span>/tmp/pg_dump.3
</code></pre></div>
<p>Let us look at the change.</p>
<div class="highlight"><pre><span></span><code><span class="gh">#</span> Old map
$ egrep '^(3.183|3.83)' /tmp/pg_dump.1 | awk '{print $1,$9,$14,$15}'
3.183 active+clean [1,5] [1,5]
3.83 active+clean [12,5] [12,5]
<span class="gh">#</span> New map
$ egrep '^(3.183|3.83)' /tmp/pg_dump.3 | awk '{print $1,$9,$14,$15}'
3.183 active+clean [1,13] [1,13]
3.83 active+clean [13,5] [13,5]
</code></pre></div>
<p>So, for pg 3.183 and 3.83, osd 5 and 12 will be replace by osd13</p>
<h1>Deacrese osd weight</h1>
<p>Same as above, but this time to reduce the weight for the osd in "near full ratio".</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span>><span class="w"> </span>/tmp/pg_dump.4
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>tree<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>osd.7
<span class="m">7</span><span class="w"> </span><span class="m">2</span>.65<span class="w"> </span>osd.7<span class="w"> </span>up<span class="w"> </span><span class="m">1</span>
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>crush<span class="w"> </span>reweight<span class="w"> </span>osd.7<span class="w"> </span><span class="m">2</span>.6
reweighted<span class="w"> </span>item<span class="w"> </span>id<span class="w"> </span><span class="m">7</span><span class="w"> </span>name<span class="w"> </span><span class="s1">'osd.7'</span><span class="w"> </span>to<span class="w"> </span><span class="m">2</span>.6<span class="w"> </span><span class="k">in</span><span class="w"> </span>crush<span class="w"> </span>map
$<span class="w"> </span>ceph<span class="w"> </span>health<span class="w"> </span>detail
HEALTH_WARN<span class="w"> </span><span class="m">2</span><span class="w"> </span>pgs<span class="w"> </span>backfilling<span class="p">;</span><span class="w"> </span><span class="m">2</span><span class="w"> </span>pgs<span class="w"> </span>stuck<span class="w"> </span>unclean<span class="p">;</span><span class="w"> </span>recovery<span class="w"> </span><span class="m">17117</span>/9160466<span class="w"> </span>degraded<span class="w"> </span><span class="o">(</span><span class="m">0</span>.187%<span class="o">)</span>
pg<span class="w"> </span><span class="m">3</span>.ca<span class="w"> </span>is<span class="w"> </span>stuck<span class="w"> </span>unclean<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="m">1097</span>.132237,<span class="w"> </span>current<span class="w"> </span>state<span class="w"> </span>active+remapped+backfilling,<span class="w"> </span>last<span class="w"> </span>acting<span class="w"> </span><span class="o">[</span><span class="m">4</span>,6,7<span class="o">]</span>
pg<span class="w"> </span><span class="m">3</span>.143<span class="w"> </span>is<span class="w"> </span>stuck<span class="w"> </span>unclean<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="m">1097</span>.456265,<span class="w"> </span>current<span class="w"> </span>state<span class="w"> </span>active+remapped+backfilling,<span class="w"> </span>last<span class="w"> </span>acting<span class="w"> </span><span class="o">[</span><span class="m">12</span>,6,7<span class="o">]</span>
pg<span class="w"> </span><span class="m">3</span>.143<span class="w"> </span>is<span class="w"> </span>active+remapped+backfilling,<span class="w"> </span>acting<span class="w"> </span><span class="o">[</span><span class="m">12</span>,6,7<span class="o">]</span>
pg<span class="w"> </span><span class="m">3</span>.ca<span class="w"> </span>is<span class="w"> </span>active+remapped+backfilling,<span class="w"> </span>acting<span class="w"> </span><span class="o">[</span><span class="m">4</span>,6,7<span class="o">]</span>
recovery<span class="w"> </span><span class="m">17117</span>/9160466<span class="w"> </span>degraded<span class="w"> </span><span class="o">(</span><span class="m">0</span>.187%<span class="o">)</span>
</code></pre></div>
<p>On the osd bandwidth, we can see those transfert osd.4 --> osd.6 and osd.12 --> osd.6 :</p>
<p><img alt="Ceph osd reweight" src="https://cephnotes.ksperis.com/images/img006.png">
<img alt="Ceph osd reweight" src="https://cephnotes.ksperis.com/images/img007.png">
<img alt="Ceph osd reweight" src="https://cephnotes.ksperis.com/images/img008.png"></p>
<p>OSD 4 and 12 are primary for pg 3.143 and 3.ca (see acting table) and OSD 6 is writing.
The OSD 7 will be released from both PG who will both be added to the OSD 6.
In my case, the osd 7 has no reading because it is only as replica for both pgs.</p>
<div class="highlight"><pre><span></span><code><span class="gh">#</span> Before
$ egrep '^(3.ca|3.143)' /tmp/pg_dump.3 | awk '{print $1,$9,$14,$15}'
3.143 active+clean [12,7] [12,7]
3.ca active+clean [4,7] [4,7]
<span class="gh">#</span> After
$ ceph pg dump > /tmp/pg_dump.5
$ egrep '^(3.ca|3.143)' /tmp/pg_dump.5 | awk '{print $1,$9,$14,$15}'
3.143 active+clean [12,6] [12,6]
3.ca active+clean [4,6] [4,6]
</code></pre></div>
<p>Well, obviously, the data are not really carried on the desired osd and now it is also too full.
I think it will take a little time for something well balanced.</p>
<h1>Using crushtool</h1>
<p>A good idea could be to test using <code>crushtool</code> with otpion --show-utilization.</p>
<p>Before retrieve current crushmap :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>getcrushmap<span class="w"> </span>-o<span class="w"> </span>crushmap.bin
</code></pre></div>
<p>You can show utilisation for a specific pool and rep size :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>dump<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'^pool 0'</span>
pool<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="s1">'data'</span><span class="w"> </span>rep<span class="w"> </span>size<span class="w"> </span><span class="m">2</span><span class="w"> </span>min_size<span class="w"> </span><span class="m">1</span><span class="w"> </span>crush_ruleset<span class="w"> </span><span class="m">0</span><span class="w"> </span>object_hash<span class="w"> </span>rjenkins<span class="w"> </span>pg_num<span class="w"> </span><span class="m">64</span><span class="w"> </span>pgp_num<span class="w"> </span><span class="m">64</span><span class="w"> </span>last_change<span class="w"> </span><span class="m">1</span><span class="w"> </span>owner<span class="w"> </span><span class="m">0</span><span class="w"> </span>
$<span class="w"> </span>crushtool<span class="w"> </span>--test<span class="w"> </span>-i<span class="w"> </span>crushmap.bin<span class="w"> </span>--show-utilization<span class="w"> </span>--rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>--num-rep<span class="o">=</span><span class="m">2</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">0</span>:<span class="w"> </span><span class="m">123</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">1</span>:<span class="w"> </span><span class="m">145</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">2</span>:<span class="w"> </span><span class="m">125</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">3</span>:<span class="w"> </span><span class="m">121</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">4</span>:<span class="w"> </span><span class="m">139</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">5</span>:<span class="w"> </span><span class="m">133</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">6</span>:<span class="w"> </span><span class="m">129</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">7</span>:<span class="w"> </span><span class="m">142</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">8</span>:<span class="w"> </span><span class="m">146</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">9</span>:<span class="w"> </span><span class="m">139</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">10</span>:<span class="w"> </span><span class="m">146</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">11</span>:<span class="w"> </span><span class="m">143</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">12</span>:<span class="w"> </span><span class="m">129</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">13</span>:<span class="w"> </span><span class="m">136</span>
<span class="w"> </span>device<span class="w"> </span><span class="m">14</span>:<span class="w"> </span><span class="m">152</span>
</code></pre></div>
<p>Make modification, and test with new weight :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>crushtool<span class="w"> </span>-d<span class="w"> </span>crushmap.bin<span class="w"> </span>-o<span class="w"> </span>crushmap.txt
<span class="c1"># edit crushmap.txt</span>
$<span class="w"> </span>crushtool<span class="w"> </span>-c<span class="w"> </span>crushmap.txt<span class="w"> </span>-o<span class="w"> </span>crushmap-new.bin
$<span class="w"> </span>crushtool<span class="w"> </span>--test<span class="w"> </span>-i<span class="w"> </span>crushmap-new.bin<span class="w"> </span>--show-utilization<span class="w"> </span>--rule<span class="w"> </span><span class="m">0</span><span class="w"> </span>--num-rep<span class="o">=</span><span class="m">2</span>
</code></pre></div>
<p>If all goes well, reimport crushmap :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>setcrushmap<span class="w"> </span>-i<span class="w"> </span>crushmap-new.bin
</code></pre></div>
<p>http://ceph.com/docs/master/man/8/crushtool/</p>
<h1>osd reweight-by-utilization</h1>
<p>Also, you can use <code>ceph osd reweight-by-utilization</code>.</p>
<p>http://ceph.com/docs/master/rados/operations/control/#osd-subsystem</p>
<h1>Other</h1>
<p>To view the number of pg per osd :</p>
<p>http://cephnotes.ksperis.com/blog/2015/02/23/get-the-number-of-placement-groups-per-osd/</p>Using ceph rbd with libvirt on Debian Wheezy2013-09-12T16:44:00+02:002013-09-12T16:44:00+02:00Laurent Barbetag:cephnotes.ksperis.com,2013-09-12:/blog/2013/09/12/using-ceph-rbd-with-libvirt-on-debian-wheezy/<p>Howto add support of rbd device on debian wheezy.</p>
<!-- more -->
<h1>libvirt</h1>
<p>Since wheezy, libvirt supports rbd device.</p>
<h1>qemu-kvm</h1>
<p>If you do not add the support in qemu-kvm you may have an error like this:</p>
<div class="highlight"><pre><span></span><code><span class="n">error</span><span class="o">:</span><span class="w"> </span><span class="n">Failed</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">start</span><span class="w"> </span><span class="n">domain</span><span class="w"> </span><span class="n">ubuntu</span>
<span class="n">error</span><span class="o">:</span><span class="w"> </span><span class="kd">internal</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="n">process</span><span class="w"> </span><span class="n">exited</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="n">connecting</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">monitor</span><span class="o">:</span><span class="w"> </span><span class="n">char</span><span class="w"> </span><span class="n">device …</span></code></pre></div><p>Howto add support of rbd device on debian wheezy.</p>
<!-- more -->
<h1>libvirt</h1>
<p>Since wheezy, libvirt supports rbd device.</p>
<h1>qemu-kvm</h1>
<p>If you do not add the support in qemu-kvm you may have an error like this:</p>
<div class="highlight"><pre><span></span><code><span class="n">error</span><span class="o">:</span><span class="w"> </span><span class="n">Failed</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">start</span><span class="w"> </span><span class="n">domain</span><span class="w"> </span><span class="n">ubuntu</span>
<span class="n">error</span><span class="o">:</span><span class="w"> </span><span class="kd">internal</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="n">process</span><span class="w"> </span><span class="n">exited</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="n">connecting</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">monitor</span><span class="o">:</span><span class="w"> </span><span class="n">char</span><span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="n">redirected</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="sr">/dev/pts/</span><span class="mi">2</span>
<span class="n">kvm</span><span class="o">:</span><span class="w"> </span><span class="o">-</span><span class="n">drive</span><span class="w"> </span><span class="n">file</span><span class="o">=</span><span class="n">rbd</span><span class="o">:</span><span class="n">vmimages</span><span class="sr">/ubuntu:id=vmimages:key=AQAfwTFScNKmLxAAo9abfSLct78rfIQRzQnS5A==:auth_supported=cephx\;none:mon_host=192.168.0.100\:6789\;192.168.0.101\:6789\;192.168.0.102\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback: could not open disk image rbd:vmimages/</span><span class="n">ubuntu</span><span class="o">:</span><span class="n">id</span><span class="o">=</span><span class="n">vmimages</span><span class="o">:</span><span class="n">key</span><span class="o">=</span><span class="n">AQAfwTFScNKmLxAAo9abfSLct78rfIQRzQnS5A</span><span class="o">==:</span><span class="n">auth_supported</span><span class="o">=</span><span class="n">cephx</span><span class="o">\;</span><span class="n">none</span><span class="o">:</span><span class="n">mon_host</span><span class="o">=</span><span class="mf">192.168</span><span class="o">.</span><span class="mf">0.100</span><span class="o">\:</span><span class="mi">6789</span><span class="o">\;</span><span class="mf">192.168</span><span class="o">.</span><span class="mf">0.101</span><span class="o">\:</span><span class="mi">6789</span><span class="o">\;</span><span class="mf">192.168</span><span class="o">.</span><span class="mf">0.102</span><span class="o">\:</span><span class="mi">6789</span><span class="o">:</span><span class="w"> </span><span class="n">No</span><span class="w"> </span><span class="n">such</span><span class="w"> </span><span class="n">file</span><span class="w"> </span><span class="n">or</span><span class="w"> </span><span class="n">directory</span>
</code></pre></div>
<h2>install ceph dependencies</h2>
<div class="highlight"><pre><span></span><code><span class="nt">apt-get</span><span class="w"> </span><span class="nt">install</span><span class="w"> </span><span class="nt">lsb-release</span>
<span class="nt">wget</span><span class="w"> </span><span class="nt">-q</span><span class="w"> </span><span class="nt">-O-</span><span class="w"> </span><span class="s1">'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc'</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nt">sudo</span><span class="w"> </span><span class="nt">apt-key</span><span class="w"> </span><span class="nt">add</span><span class="w"> </span><span class="nt">-</span>
<span class="nt">echo</span><span class="w"> </span><span class="nt">deb</span><span class="w"> </span><span class="nt">http</span><span class="o">://</span><span class="nt">ceph</span><span class="p">.</span><span class="nc">com</span><span class="o">/</span><span class="nt">debian-firefly</span><span class="o">/</span><span class="w"> </span><span class="o">$(</span><span class="nt">lsb_release</span><span class="w"> </span><span class="nt">-sc</span><span class="o">)</span><span class="w"> </span><span class="nt">main</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nt">sudo</span><span class="w"> </span><span class="nt">tee</span><span class="w"> </span><span class="o">/</span><span class="nt">etc</span><span class="o">/</span><span class="nt">apt</span><span class="o">/</span><span class="nt">sources</span><span class="p">.</span><span class="nc">list</span><span class="p">.</span><span class="nc">d</span><span class="o">/</span><span class="nt">ceph</span><span class="p">.</span><span class="nc">list</span>
<span class="nt">apt-get</span><span class="w"> </span><span class="nt">update</span>
<span class="nt">apt-get</span><span class="w"> </span><span class="nt">install</span><span class="w"> </span><span class="nt">librbd-dev</span><span class="w"> </span><span class="nt">librados-dev</span><span class="w"> </span><span class="nt">build-essential</span>
</code></pre></div>
<h2>build kvm</h2>
<div class="highlight"><pre><span></span><code>mkdir -p sources/qemu-kvm
cd sources/qemu-kvm
apt-get source qemu-kvm
apt-get build-dep qemu-kvm
cd qemu-kvm-1.1.2+dfsg/
vim debian/optional-features
</code></pre></div>
<p>Uncomment line <code># --enable-rbd</code>
and compile :</p>
<div class="highlight"><pre><span></span><code><span class="nx">dpkg</span><span class="o">-</span><span class="nx">buildpackage</span><span class="w"> </span><span class="o">-</span><span class="nx">rfakeroot</span><span class="w"> </span><span class="o">-</span><span class="nx">b</span>
<span class="nx">cd</span><span class="w"> </span><span class="p">..</span>
</code></pre></div>
<p>Installation :</p>
<div class="highlight"><pre><span></span><code>apt-get install librados2 librbd1
dpkg -i kvm_1.1.2+dfsg-6_amd64.deb qemu-kvm_1.1.2+dfsg-6_amd64.deb
</code></pre></div>
<p>Becareful on upgrade !
You can also hold package version :</p>
<div class="highlight"><pre><span></span><code>echo "kvm hold" | dpkg --set-selections
echo "qemu-kvm hold" | dpkg --set-selections
</code></pre></div>
<h1>qemu (for util like qemu-img)</h1>
<p>(Only if needed)
You also need to add support in debian rules : "--enable-rbd"</p>
<div class="highlight"><pre><span></span><code><span class="nx">mkdir</span><span class="w"> </span><span class="o">-</span><span class="nx">p</span><span class="w"> </span><span class="nx">sources</span><span class="o">/</span><span class="nx">qemu</span>
<span class="nx">cd</span><span class="w"> </span><span class="nx">sources</span><span class="o">/</span><span class="nx">qemu</span>
<span class="nx">apt</span><span class="o">-</span><span class="nx">get</span><span class="w"> </span><span class="nx">source</span><span class="w"> </span><span class="nx">qemu</span>
<span class="nx">apt</span><span class="o">-</span><span class="nx">get</span><span class="w"> </span><span class="nx">build</span><span class="o">-</span><span class="nx">dep</span><span class="w"> </span><span class="nx">qemu</span>
<span class="nx">cd</span><span class="w"> </span><span class="nx">qemu</span><span class="o">-</span><span class="m m-Double">1.1.2</span><span class="o">+</span><span class="nx">dfsg</span><span class="o">/</span>
<span class="nx">vim</span><span class="w"> </span><span class="nx">debian</span><span class="o">/</span><span class="nx">rules</span><span class="w"> </span><span class="o">+</span><span class="mi">82</span>
<span class="nx">dpkg</span><span class="o">-</span><span class="nx">buildpackage</span><span class="w"> </span><span class="o">-</span><span class="nx">rfakeroot</span><span class="w"> </span><span class="o">-</span><span class="nx">b</span>
<span class="nx">cd</span><span class="w"> </span><span class="p">..</span>
</code></pre></div>
<h1>Use</h1>
<p>Create a pool and associated key on ceph admin host :</p>
<div class="highlight"><pre><span></span><code>ceph osd pool create vmimages 100 100
ceph auth get-or-create client.vmimages mon 'allow r' osd 'allow rwx pool=vmimages'
</code></pre></div>
<p>keep the key returned.</p>
<p>Create a secret file for libvirt for this user :</p>
<div class="highlight"><pre><span></span><code>echo<span class="w"> </span>"
<span class="w"> </span><span class="nt"><secret</span><span class="w"> </span><span class="na">ephemeral=</span><span class="s">'no'</span><span class="w"> </span><span class="na">private=</span><span class="s">'no'</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><usage</span><span class="w"> </span><span class="na">type=</span><span class="s">'ceph'</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><name></span>client.vmimages<span class="w"> </span>secret<span class="nt"></name></span>
<span class="w"> </span><span class="nt"></usage></span>
<span class="nt"></secret></span>"<span class="w"> </span>><span class="w"> </span>secret.xml
virsh<span class="w"> </span>secret-define<span class="w"> </span>--file<span class="w"> </span>secret.xml
</code></pre></div>
<p>Define the key value :
first uuid is the value return by the previous command
follow by the key return by <code>ceph auth get-or-create</code></p>
<div class="highlight"><pre><span></span><code>virsh secret-set-value 76e3a541-b997-58ac-f7bd-77dd7d4347cb AQAREH1QkNDNCBaac03ZICi/CePnRDS+vGyrqQ==
</code></pre></div>
<p>Create a device:</p>
<p>(on ceph admin host if you do not want to specify the user, the key, and monitors)
with rbd : <code>rbd create vmimages/ubuntu-newdrive --size=2048</code>
with qemu (you need to add support for that) : <code>qemu-img create -f rbd rbd:vmimages/ubuntu-newdrive 2G</code></p>
<p>Attach on existing host (even if running) :</p>
<p>(192.168.0.100, 192.168.0.101, 192.168.0.102 is the mon host.)</p>
<div class="highlight"><pre><span></span><code>echo<span class="w"> </span>"
<span class="w"> </span><span class="nt"><disk</span><span class="w"> </span><span class="na">type=</span><span class="s">'network'</span><span class="w"> </span><span class="na">device=</span><span class="s">'disk'</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><driver</span><span class="w"> </span><span class="na">name=</span><span class="s">'qemu'</span><span class="w"> </span><span class="na">type=</span><span class="s">'raw'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"><auth</span><span class="w"> </span><span class="na">username=</span><span class="s">'vmimages'</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><secret</span><span class="w"> </span><span class="na">type=</span><span class="s">'ceph'</span><span class="w"> </span><span class="na">uuid=</span><span class="s">'76e3a541-b997-58ac-f7bd-77dd7d4347cb'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"></auth></span>
<span class="w"> </span><span class="nt"><source</span><span class="w"> </span><span class="na">protocol=</span><span class="s">'rbd'</span><span class="w"> </span><span class="na">name=</span><span class="s">'vmimages/ubuntu-newdrive'</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><host</span><span class="w"> </span><span class="na">name=</span><span class="s">'192.168.0.100'</span><span class="w"> </span><span class="na">port=</span><span class="s">'6789'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"><host</span><span class="w"> </span><span class="na">name=</span><span class="s">'192.168.0.101'</span><span class="w"> </span><span class="na">port=</span><span class="s">'6789'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"><host</span><span class="w"> </span><span class="na">name=</span><span class="s">'192.168.0.102'</span><span class="w"> </span><span class="na">port=</span><span class="s">'6789'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"></source></span>
<span class="w"> </span><span class="nt"><target</span><span class="w"> </span><span class="na">dev=</span><span class="s">'vdz'</span><span class="w"> </span><span class="na">bus=</span><span class="s">'virtio'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"></disk></span>
"<span class="w"> </span>><span class="w"> </span>device.xml
virsh<span class="w"> </span>attach-device<span class="w"> </span>ubuntu<span class="w"> </span>device.xml<span class="w"> </span>--persistent
</code></pre></div>Object difference between RBD format1 and format22013-09-10T18:58:00+02:002013-09-10T18:58:00+02:00Laurent Barbetag:cephnotes.ksperis.com,2013-09-10:/blog/2013/09/10/object-difference-between-rbd-format1-and-format2/<p>Lets take a look how rbd object are store on rados and the difference between format 1 and format 2.</p>
<!-- more -->
<h1>Format 1</h1>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>create<span class="w"> </span>myrbd<span class="w"> </span>--size<span class="o">=</span><span class="m">10</span><span class="w"> </span>
$<span class="w"> </span>rados<span class="w"> </span>ls<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>
myrbd.rbd
rbd_directory
$<span class="w"> </span>rbd<span class="w"> </span>map<span class="w"> </span>myrbd
$<span class="w"> </span>dd<span class="w"> </span><span class="k">if</span><span class="o">=</span>/dev/zero<span class="w"> </span><span class="nv">of</span><span class="o">=</span>/dev/rbd/rbd/myrbd
$<span class="w"> </span>rados<span class="w"> </span>ls<span class="w"> </span>-p<span class="w"> </span>rbd …</code></pre></div><p>Lets take a look how rbd object are store on rados and the difference between format 1 and format 2.</p>
<!-- more -->
<h1>Format 1</h1>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>create<span class="w"> </span>myrbd<span class="w"> </span>--size<span class="o">=</span><span class="m">10</span><span class="w"> </span>
$<span class="w"> </span>rados<span class="w"> </span>ls<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>
myrbd.rbd
rbd_directory
$<span class="w"> </span>rbd<span class="w"> </span>map<span class="w"> </span>myrbd
$<span class="w"> </span>dd<span class="w"> </span><span class="k">if</span><span class="o">=</span>/dev/zero<span class="w"> </span><span class="nv">of</span><span class="o">=</span>/dev/rbd/rbd/myrbd
$<span class="w"> </span>rados<span class="w"> </span>ls<span class="w"> </span>-p<span class="w"> </span>rbd
myrbd.rbd
rb.0.1286.74b0dc51.000000000000
rb.0.1286.74b0dc51.000000000001
rb.0.1286.74b0dc51.000000000002
rbd_directory
</code></pre></div>
<ol>
<li><em>$image_name</em>.rbd : Contain the id of the image ( rb.0.1286.74b0dc51 )</li>
<li><em>$rbd_id</em>.<em>$fragment</em> : Raw data image</li>
<li>rbd_directory : Rbd image list for the current pool</li>
</ol>
<h1>Format 2</h1>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>create<span class="w"> </span>myrbd<span class="w"> </span>--size<span class="o">=</span><span class="m">10</span><span class="w"> </span>--image-format<span class="o">=</span><span class="m">2</span>
$<span class="w"> </span>rados<span class="w"> </span>ls<span class="w"> </span>-p<span class="w"> </span>rbd<span class="w"> </span>
rbd_directory
rbd_header.134a74b0dc51
rbd_id.myrbd
$<span class="w"> </span>rbd<span class="w"> </span>map<span class="w"> </span>myrbd
$<span class="w"> </span>dd<span class="w"> </span><span class="k">if</span><span class="o">=</span>/dev/zero<span class="w"> </span><span class="nv">of</span><span class="o">=</span>/dev/rbd/rbd/myrbd
$<span class="w"> </span>rados<span class="w"> </span>ls<span class="w"> </span>-p<span class="w"> </span>rbd
rbd_data.134a74b0dc51.0000000000000000
rbd_data.134a74b0dc51.0000000000000001
rbd_data.134a74b0dc51.0000000000000002
rbd_directory
rbd_header.134a74b0dc51
rbd_id.myrbd
</code></pre></div>
<ol>
<li>rbd_data.<em>$rbd_id</em>.<em>$fragment</em> : Raw data image</li>
<li>rbd_directory : Rbd image list for the current pool</li>
<li>rbd_header.<em>$rbd_id</em> : Metadata about the image</li>
<li>rbd_id.<em>$image_name</em> : Contain the id of the image ( 134a74b0dc51 )</li>
</ol>Mon failed to start2013-08-29T17:05:00+02:002013-08-29T17:05:00+02:00Laurent Barbetag:cephnotes.ksperis.com,2013-08-29:/blog/2013/08/29/mon-failed-to-start/<p>Some common problems when adding a monitor to an existing cluster, for example if config is not found :</p>
<!--more-->
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="o">$</span><span class="w"> </span><span class="n">service</span><span class="w"> </span><span class="n">ceph</span><span class="w"> </span><span class="n">start</span><span class="w"> </span><span class="n">mon</span><span class="o">.</span><span class="n">ceph</span><span class="o">-</span><span class="mi">03</span>
<span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">init</span><span class="o">.</span><span class="n">d</span><span class="o">/</span><span class="n">ceph</span><span class="p">:</span><span class="w"> </span><span class="n">mon</span><span class="o">.</span><span class="n">ceph</span><span class="o">-</span><span class="mi">03</span><span class="w"> </span><span class="ow">not</span><span class="w"> </span><span class="n">found</span><span class="w"> </span><span class="p">(</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">ceph</span><span class="o">.</span><span class="n">conf</span><span class="w"> </span><span class="n">defines</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">2</span><span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">ceph</span><span class="w"> </span><span class="n">defines</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">2</span><span class="p">)</span>
</code></pre></div>
<p>If you do not …</p><p>Some common problems when adding a monitor to an existing cluster, for example if config is not found :</p>
<!--more-->
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="o">$</span><span class="w"> </span><span class="n">service</span><span class="w"> </span><span class="n">ceph</span><span class="w"> </span><span class="n">start</span><span class="w"> </span><span class="n">mon</span><span class="o">.</span><span class="n">ceph</span><span class="o">-</span><span class="mi">03</span>
<span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">init</span><span class="o">.</span><span class="n">d</span><span class="o">/</span><span class="n">ceph</span><span class="p">:</span><span class="w"> </span><span class="n">mon</span><span class="o">.</span><span class="n">ceph</span><span class="o">-</span><span class="mi">03</span><span class="w"> </span><span class="ow">not</span><span class="w"> </span><span class="n">found</span><span class="w"> </span><span class="p">(</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">ceph</span><span class="o">.</span><span class="n">conf</span><span class="w"> </span><span class="n">defines</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">2</span><span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">ceph</span><span class="w"> </span><span class="n">defines</span><span class="w"> </span><span class="n">osd</span><span class="o">.</span><span class="mi">2</span><span class="p">)</span>
</code></pre></div>
<p>If you do not want to specify a section mon.ceph-03 in ceph.conf, you need to have a file <em>sysvinit</em> in /var/lib/ceph/mon/ceph-ceph-03/</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>-l<span class="w"> </span>/var/lib/ceph/mon/ceph-ceph-03/
total<span class="w"> </span><span class="m">8</span>
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">77</span><span class="w"> </span>août<span class="w"> </span><span class="m">29</span><span class="w"> </span><span class="m">16</span>:56<span class="w"> </span>keyring
drwxr-xr-x<span class="w"> </span><span class="m">2</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">4096</span><span class="w"> </span>août<span class="w"> </span><span class="m">29</span><span class="w"> </span><span class="m">17</span>:03<span class="w"> </span>store.db
</code></pre></div>
<p>Just create the file, then it should start :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>touch<span class="w"> </span>/var/lib/ceph/mon/ceph-ceph-03/sysvinit
$<span class="w"> </span>service<span class="w"> </span>ceph<span class="w"> </span>start<span class="w"> </span>mon.ceph-03
<span class="o">===</span><span class="w"> </span>mon.ceph-03<span class="w"> </span><span class="o">===</span><span class="w"> </span>
Starting<span class="w"> </span>Ceph<span class="w"> </span>mon.ceph-03<span class="w"> </span>on<span class="w"> </span>ceph-03...
failed:<span class="w"> </span><span class="s1">'ulimit -n 32768; /usr/bin/ceph-mon -i ceph-03 --pid-file /var/run/ceph/mon.ceph-03.pid -c /etc/ceph/ceph.conf '</span>
Starting<span class="w"> </span>ceph-create-keys<span class="w"> </span>on<span class="w"> </span>ceph-03...
</code></pre></div>
<p>Next error on starting monitor, if you have a look to log you can see :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tail<span class="w"> </span>-f<span class="w"> </span>ceph-mon.ceph-03.log
mon.ceph-03<span class="w"> </span>does<span class="w"> </span>not<span class="w"> </span>exist<span class="w"> </span><span class="k">in</span><span class="w"> </span>monmap,<span class="w"> </span>will<span class="w"> </span>attempt<span class="w"> </span>to<span class="w"> </span>join<span class="w"> </span>an<span class="w"> </span>existing<span class="w"> </span>cluster
no<span class="w"> </span>public_addr<span class="w"> </span>or<span class="w"> </span>public_network<span class="w"> </span>specified,<span class="w"> </span>and<span class="w"> </span>mon.ceph-03<span class="w"> </span>not<span class="w"> </span>present<span class="w"> </span><span class="k">in</span><span class="w"> </span>monmap<span class="w"> </span>or<span class="w"> </span>ceph.conf
</code></pre></div>
<p>You shoud verify, that you do not have ceph-create-keys process that hang, if so you can kill it :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ps<span class="w"> </span>aux<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>create-keys
root<span class="w"> </span><span class="m">1317</span><span class="w"> </span><span class="m">0</span>.1<span class="w"> </span><span class="m">1</span>.4<span class="w"> </span><span class="m">36616</span><span class="w"> </span><span class="m">7168</span><span class="w"> </span>pts/0<span class="w"> </span>S<span class="w"> </span><span class="m">17</span>:13<span class="w"> </span><span class="m">0</span>:00<span class="w"> </span>/usr/bin/python<span class="w"> </span>/usr/sbin/ceph-create-keys<span class="w"> </span>-i<span class="w"> </span>ceph-03
$<span class="w"> </span><span class="nb">kill</span><span class="w"> </span><span class="m">1317</span>
</code></pre></div>
<p>Verify that you have defined this mon on the current monmap</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>mon<span class="w"> </span>dump
dumped<span class="w"> </span>monmap<span class="w"> </span>epoch<span class="w"> </span><span class="m">6</span>
epoch<span class="w"> </span><span class="m">6</span>
fsid<span class="w"> </span>e0506c4d-e86a-40a8-8306-4856f9ccb989
last_changed<span class="w"> </span><span class="m">2013</span>-08-29<span class="w"> </span><span class="m">16</span>:58:06.145127
created<span class="w"> </span><span class="m">0</span>.000000
<span class="m">0</span>:<span class="w"> </span><span class="m">10</span>.2.4.10:6789/0<span class="w"> </span>mon.ceph-01
<span class="m">1</span>:<span class="w"> </span><span class="m">10</span>.2.4.11:6789/0<span class="w"> </span>mon.ceph-02
<span class="m">2</span>:<span class="w"> </span><span class="m">10</span>.2.4.12:6789/0<span class="w"> </span>mon.ceph-03
</code></pre></div>
<p>You need to retrieve the current monmap and add if for this node : </p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>mon<span class="w"> </span>getmap<span class="w"> </span>-o<span class="w"> </span>/tmp/monmap
<span class="m">2013</span>-08-29<span class="w"> </span><span class="m">17</span>:36:36.204257<span class="w"> </span>7f641a54d700<span class="w"> </span><span class="m">0</span><span class="w"> </span>--<span class="w"> </span>:/1005682<span class="w"> </span>>><span class="w"> </span><span class="m">10</span>.2.4.12:6789/0<span class="w"> </span>pipe<span class="o">(</span>0x2283400<span class="w"> </span><span class="nv">sd</span><span class="o">=</span><span class="m">4</span><span class="w"> </span>:0<span class="w"> </span><span class="nv">s</span><span class="o">=</span><span class="m">1</span><span class="w"> </span><span class="nv">pgs</span><span class="o">=</span><span class="m">0</span><span class="w"> </span><span class="nv">cs</span><span class="o">=</span><span class="m">0</span><span class="w"> </span><span class="nv">l</span><span class="o">=</span><span class="m">1</span><span class="w"> </span><span class="nv">c</span><span class="o">=</span>0x2283660<span class="o">)</span>.fault
got<span class="w"> </span>latest<span class="w"> </span>monmap
$<span class="w"> </span>ceph-mon<span class="w"> </span>-i<span class="w"> </span>ceph-03<span class="w"> </span>--inject-monmap<span class="w"> </span>/tmp/monmap
</code></pre></div>
<p>Try again :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>service<span class="w"> </span>ceph<span class="w"> </span>start<span class="w"> </span>mon.ceph-03
<span class="o">===</span><span class="w"> </span>mon.ceph-03<span class="w"> </span><span class="o">===</span><span class="w"> </span>
Starting<span class="w"> </span>Ceph<span class="w"> </span>mon.ceph-03<span class="w"> </span>on<span class="w"> </span>ceph-03...
Starting<span class="w"> </span>ceph-create-keys<span class="w"> </span>on<span class="w"> </span>ceph-03...
</code></pre></div>
<p>It's seems working fine. You can verify the state of monitor quorum :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>mon<span class="w"> </span>stat
e6:<span class="w"> </span><span class="m">3</span><span class="w"> </span>mons<span class="w"> </span>at<span class="w"> </span><span class="o">{</span>ceph-01<span class="o">=</span><span class="m">10</span>.2.4.10:6789/0,ceph-02<span class="o">=</span><span class="m">10</span>.2.4.11:6789/0,ceph-03<span class="o">=</span><span class="m">10</span>.2.4.12:6789/0<span class="o">}</span>,<span class="w"> </span>election<span class="w"> </span>epoch<span class="w"> </span><span class="m">1466</span>,<span class="w"> </span>quorum<span class="w"> </span><span class="m">0</span>,1,2<span class="w"> </span>ceph-01,ceph-02,ceph-03
</code></pre></div>
<p>For more information, have a look to the documentation :
http://ceph.com/docs/master/rados/operations/add-or-rm-mons/</p>RBD Image Real Size2013-08-28T14:51:00+02:002013-08-28T14:51:00+02:00Laurent Barbetag:cephnotes.ksperis.com,2013-08-28:/blog/2013/08/28/rbd-image-real-size/<p>To get the real size used by a rbd image :</p>
<div class="highlight"><pre><span></span><code><span class="n">rbd</span> <span class="n">diff</span> <span class="o">$</span><span class="n">POOL</span><span class="o">/$</span><span class="n">IMAGE</span> <span class="p">|</span> <span class="n">awk</span> <span class="s">'{ SUM += $2 } END { print SUM/1024/1024 " MB" }'</span>
</code></pre></div>
<!--more-->
<p>For exemple :</p>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="n">rbd</span> <span class="n">info</span> <span class="n">myrbd</span>
<span class="n">rbd</span> <span class="n">image</span> <span class="s">'myrbd'</span><span class="p">:</span>
<span class="n">size</span> <span class="mi">2048</span> <span class="n">MB</span> <span class="n">in</span> <span class="mi">512</span> <span class="n">objects</span>
<span class="n">order</span> <span class="mi">22</span> <span class="p">(</span><span class="mi">4096</span> <span class="n">KB</span> <span class="n">objects</span><span class="p">)</span>
<span class="n">block_name_prefix</span><span class="p">:</span> <span class="n">rb</span><span class="p">.</span><span class="mf">0.2</span><span class="n">c6a</span><span class="p">.</span><span class="mi">238</span><span class="n">e1f29</span>
<span class="n">format …</span></code></pre></div><p>To get the real size used by a rbd image :</p>
<div class="highlight"><pre><span></span><code><span class="n">rbd</span> <span class="n">diff</span> <span class="o">$</span><span class="n">POOL</span><span class="o">/$</span><span class="n">IMAGE</span> <span class="p">|</span> <span class="n">awk</span> <span class="s">'{ SUM += $2 } END { print SUM/1024/1024 " MB" }'</span>
</code></pre></div>
<!--more-->
<p>For exemple :</p>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="n">rbd</span> <span class="n">info</span> <span class="n">myrbd</span>
<span class="n">rbd</span> <span class="n">image</span> <span class="s">'myrbd'</span><span class="p">:</span>
<span class="n">size</span> <span class="mi">2048</span> <span class="n">MB</span> <span class="n">in</span> <span class="mi">512</span> <span class="n">objects</span>
<span class="n">order</span> <span class="mi">22</span> <span class="p">(</span><span class="mi">4096</span> <span class="n">KB</span> <span class="n">objects</span><span class="p">)</span>
<span class="n">block_name_prefix</span><span class="p">:</span> <span class="n">rb</span><span class="p">.</span><span class="mf">0.2</span><span class="n">c6a</span><span class="p">.</span><span class="mi">238</span><span class="n">e1f29</span>
<span class="n">format</span><span class="p">:</span> <span class="mi">1</span>
<span class="o">$</span> <span class="n">rbd</span> <span class="n">diff</span> <span class="n">myrbd</span> <span class="p">|</span> <span class="n">awk</span> <span class="s">'{ SUM += $2 } END { print SUM/1024/1024 " MB" }'</span>
<span class="mf">14.2812</span> <span class="n">MB</span>
</code></pre></div>
<p>From Josh in Ceph User mailing list : http://permalink.gmane.org/gmane.comp.file-systems.ceph.user/3684</p>Deep scrub distribution2013-08-27T19:30:00+02:002013-08-27T19:30:00+02:00Laurent Barbetag:cephnotes.ksperis.com,2013-08-27:/blog/2013/08/27/deep-scrub-distribution/<p>To verify the integrity of data, Ceph uses a mechanism called deep scrubbing which browse all your data once per week for each placement group.
This can be the cause of overload when all osd running deep scrubbing at the same time. </p>
<p>You can easly see if a deep scrub …</p><p>To verify the integrity of data, Ceph uses a mechanism called deep scrubbing which browse all your data once per week for each placement group.
This can be the cause of overload when all osd running deep scrubbing at the same time. </p>
<p>You can easly see if a deep scrub is current running (and how many) with ceph status <code>`ceph -w`</code>.
But the most interesting is to see the distribution over a week to anticipate the load :</p>
<h3>Get the distribution for the week</h3>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="k">for</span><span class="w"> </span>date<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="sb">`</span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>active<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{print $20}'</span><span class="sb">`</span><span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span>date<span class="w"> </span>+%A<span class="w"> </span>-d<span class="w"> </span><span class="nv">$date</span><span class="p">;</span><span class="w"> </span><span class="k">done</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sort<span class="w"> </span><span class="p">|</span><span class="w"> </span>uniq<span class="w"> </span>-c
<span class="w"> </span><span class="m">239</span><span class="w"> </span>monday
<span class="w"> </span><span class="m">41</span><span class="w"> </span>tuesday
<span class="w"> </span><span class="m">2</span><span class="w"> </span>saturday
<span class="w"> </span><span class="m">410</span><span class="w"> </span>sunday
</code></pre></div>
<p>We can easy see when are done the deep scrub (in my case sunday and monday).</p>
<!--more-->
<p>The default interval for deep scrubbing is every week but it could be tunable with parameter 'osd deep scrub interval' :
http://ceph.com/docs/master/rados/configuration/osd-config-ref/#scrubbing</p>
<h3>Distribution per hours</h3>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="k">for</span><span class="w"> </span>date<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="sb">`</span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>active<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{print $21}'</span><span class="sb">`</span><span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span>date<span class="w"> </span>+%H<span class="w"> </span>-d<span class="w"> </span><span class="nv">$date</span><span class="p">;</span><span class="w"> </span><span class="k">done</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sort<span class="w"> </span><span class="p">|</span><span class="w"> </span>uniq<span class="w"> </span>-c
<span class="w"> </span><span class="m">11</span><span class="w"> </span><span class="m">00</span>
<span class="w"> </span><span class="m">26</span><span class="w"> </span><span class="m">01</span>
<span class="w"> </span><span class="m">34</span><span class="w"> </span><span class="m">02</span>
<span class="w"> </span><span class="m">37</span><span class="w"> </span><span class="m">03</span>
<span class="w"> </span><span class="m">26</span><span class="w"> </span><span class="m">04</span>
<span class="w"> </span><span class="m">25</span><span class="w"> </span><span class="m">05</span>
<span class="w"> </span><span class="m">14</span><span class="w"> </span><span class="m">06</span>
<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">08</span>
<span class="w"> </span><span class="m">28</span><span class="w"> </span><span class="m">09</span>
<span class="w"> </span><span class="m">49</span><span class="w"> </span><span class="m">10</span>
<span class="w"> </span><span class="m">52</span><span class="w"> </span><span class="m">11</span>
<span class="w"> </span><span class="m">46</span><span class="w"> </span><span class="m">12</span>
<span class="w"> </span><span class="m">37</span><span class="w"> </span><span class="m">13</span>
<span class="w"> </span><span class="m">31</span><span class="w"> </span><span class="m">14</span>
<span class="w"> </span><span class="m">29</span><span class="w"> </span><span class="m">15</span>
<span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">16</span>
<span class="w"> </span><span class="m">34</span><span class="w"> </span><span class="m">17</span>
<span class="w"> </span><span class="m">33</span><span class="w"> </span><span class="m">18</span>
<span class="w"> </span><span class="m">27</span><span class="w"> </span><span class="m">19</span>
<span class="w"> </span><span class="m">33</span><span class="w"> </span><span class="m">20</span>
<span class="w"> </span><span class="m">27</span><span class="w"> </span><span class="m">21</span>
<span class="w"> </span><span class="m">25</span><span class="w"> </span><span class="m">22</span>
<span class="w"> </span><span class="m">22</span><span class="w"> </span><span class="m">23</span>
</code></pre></div>Ceph OSD : Where is my data ?2013-08-20T23:28:00+02:002013-08-20T23:28:00+02:00Laurent Barbetag:cephnotes.ksperis.com,2013-08-20:/blog/2013/08/20/ceph-osd-where-is-my-data/<p>The purpose is to verify where my data is stored on the Ceph cluster.</p>
<!-- more -->
<p>For this, I have just create a minimal cluster with 3 osd :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph-deploy<span class="w"> </span>osd<span class="w"> </span>create<span class="w"> </span>ceph-01:/dev/sdb<span class="w"> </span>ceph-02:/dev/sdb<span class="w"> </span>ceph-03:/dev/sdb
</code></pre></div>
<p>Where is my osd directory on ceph-01 ?</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mount<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>ceph
/dev/sdb1 …</code></pre></div><p>The purpose is to verify where my data is stored on the Ceph cluster.</p>
<!-- more -->
<p>For this, I have just create a minimal cluster with 3 osd :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph-deploy<span class="w"> </span>osd<span class="w"> </span>create<span class="w"> </span>ceph-01:/dev/sdb<span class="w"> </span>ceph-02:/dev/sdb<span class="w"> </span>ceph-03:/dev/sdb
</code></pre></div>
<p>Where is my osd directory on ceph-01 ?</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mount<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>ceph
/dev/sdb1<span class="w"> </span>on<span class="w"> </span>/var/lib/ceph/osd/ceph-0<span class="w"> </span><span class="nb">type</span><span class="w"> </span>xfs<span class="w"> </span><span class="o">(</span>rw,noatime,attr2,delaylog,noquota<span class="o">)</span>
</code></pre></div>
<p>The directory content :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>/var/lib/ceph/osd/ceph-0<span class="p">;</span><span class="w"> </span>ls<span class="w"> </span>-l
total<span class="w"> </span><span class="m">52</span>
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">487</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>activate.monmap
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">3</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>active
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">37</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>ceph_fsid
drwxr-xr-x<span class="w"> </span><span class="m">133</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">8192</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:18<span class="w"> </span>current
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">37</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>fsid
lrwxrwxrwx<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">58</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>journal<span class="w"> </span>-><span class="w"> </span>/dev/disk/by-partuuid/37180b7e-fe5d-4b53-8693-12a8c1f52ec9
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">37</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>journal_uuid
-rw-------<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">56</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>keyring
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">21</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>magic
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">6</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>ready
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">4</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>store_version
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">0</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>sysvinit
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">2</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>whoami
$<span class="w"> </span>du<span class="w"> </span>-hs<span class="w"> </span>*
<span class="m">4</span>,0K<span class="w"> </span>activate.monmap<span class="w"> </span>→<span class="w"> </span>The<span class="w"> </span>current<span class="w"> </span>monmap
<span class="m">4</span>,0K<span class="w"> </span>active<span class="w"> </span>→<span class="w"> </span><span class="s2">"ok"</span>
<span class="m">4</span>,0K<span class="w"> </span>ceph_fsid<span class="w"> </span>→<span class="w"> </span>cluster<span class="w"> </span>fsid<span class="w"> </span><span class="o">(</span>same<span class="w"> </span><span class="k">return</span><span class="w"> </span>by<span class="w"> </span><span class="s1">'ceph fsid'</span><span class="o">)</span>
<span class="m">2</span>,1M<span class="w"> </span>current
<span class="m">4</span>,0K<span class="w"> </span>fsid<span class="w"> </span>→<span class="w"> </span>id<span class="w"> </span><span class="k">for</span><span class="w"> </span>this<span class="w"> </span>osd
<span class="m">0</span><span class="w"> </span>journal<span class="w"> </span>→<span class="w"> </span>symlink<span class="w"> </span>to<span class="w"> </span>journal<span class="w"> </span>partition
<span class="m">4</span>,0K<span class="w"> </span>journal_uuid
<span class="m">4</span>,0K<span class="w"> </span>keyring<span class="w"> </span>→<span class="w"> </span>the<span class="w"> </span>key
<span class="m">4</span>,0K<span class="w"> </span>magic<span class="w"> </span>→<span class="w"> </span><span class="s2">"ceph osd volume v026"</span>
<span class="m">4</span>,0K<span class="w"> </span>ready<span class="w"> </span>→<span class="w"> </span><span class="s2">"ready"</span>
<span class="m">4</span>,0K<span class="w"> </span>store_version<span class="w"> </span>
<span class="m">0</span><span class="w"> </span>sysvinit
<span class="m">4</span>,0K<span class="w"> </span>whoami<span class="w"> </span>→<span class="w"> </span>id<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>osd
</code></pre></div>
<p>The data are store in the directory "current" :
It contains some file and many _head file :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>current<span class="p">;</span><span class="w"> </span>ls<span class="w"> </span>-l<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>-v<span class="w"> </span>head
total<span class="w"> </span><span class="m">20</span>
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">5</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:18<span class="w"> </span>commit_op_seq
drwxr-xr-x<span class="w"> </span><span class="m">2</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">12288</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:18<span class="w"> </span>meta
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">0</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>nosnap
drwxr-xr-x<span class="w"> </span><span class="m">2</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">111</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>omap
</code></pre></div>
<p>In omap directory :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>omap<span class="p">;</span><span class="w"> </span>ls<span class="w"> </span>-l
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">150</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span><span class="m">000007</span>.sst
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">2031616</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:18<span class="w"> </span><span class="m">000010</span>.log<span class="w"> </span>
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">16</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>CURRENT
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">0</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>LOCK
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">172</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>LOG
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">309</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>LOG.old
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">65536</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>MANIFEST-000009
</code></pre></div>
<p>In meta directory :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>../meta<span class="p">;</span><span class="w"> </span>ls<span class="w"> </span>-l
total<span class="w"> </span><span class="m">940</span>
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">710</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>inc<span class="se">\u</span>osdmap.10__0_F4E9C003__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">958</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>inc<span class="se">\u</span>osdmap.1__0_B65F4306__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">722</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>inc<span class="se">\u</span>osdmap.11__0_F4E9C1D3__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">152</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>inc<span class="se">\u</span>osdmap.12__0_F4E9C163__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">153</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>inc<span class="se">\u</span>osdmap.2__0_B65F40D6__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">574</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>inc<span class="se">\u</span>osdmap.3__0_B65F4066__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">153</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>inc<span class="se">\u</span>osdmap.4__0_B65F4136__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">722</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>inc<span class="se">\u</span>osdmap.5__0_B65F46C6__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">136</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>inc<span class="se">\u</span>osdmap.6__0_B65F4796__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">642</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>inc<span class="se">\u</span>osdmap.7__0_B65F4726__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">153</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>inc<span class="se">\u</span>osdmap.8__0_B65F44F6__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">722</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>inc<span class="se">\u</span>osdmap.9__0_B65F4586__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">0</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>infos__head_16EF7597__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">2870</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>osdmap.10__0_6417091C__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">830</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>osdmap.1__0_FD6E49B1__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">2870</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>osdmap.11__0_64170EAC__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">2870</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>osdmap.12__0_64170E7C__none<span class="w"> </span>→<span class="w"> </span>current<span class="w"> </span>osdmap
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">1442</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>osdmap.2__0_FD6E4941__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">1510</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>osdmap.3__0_FD6E4E11__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">2122</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>osdmap.4__0_FD6E4FA1__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">2122</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>osdmap.5__0_FD6E4F71__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">2122</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>osdmap.6__0_FD6E4C01__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">2190</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>osdmap.7__0_FD6E4DD1__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">2802</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>osdmap.8__0_FD6E4D61__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">2802</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>osdmap.9__0_FD6E4231__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">354</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:14<span class="w"> </span>osd<span class="se">\u</span>superblock__0_23C2FCDE__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">0</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>pglog<span class="se">\u</span><span class="m">0</span>.0__0_103B076E__none<span class="w"> </span>→<span class="w"> </span>Log<span class="w"> </span><span class="k">for</span><span class="w"> </span>each<span class="w"> </span>pg
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">0</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>pglog<span class="se">\u</span><span class="m">0</span>.1__0_103B043E__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">0</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>pglog<span class="se">\u</span><span class="m">0</span>.11__0_5172C9DB__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">0</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>pglog<span class="se">\u</span><span class="m">0</span>.13__0_5172CE3B__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">0</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:13<span class="w"> </span>pglog<span class="se">\u</span><span class="m">0</span>.15__0_5172CC9B__none
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">0</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:13<span class="w"> </span>pglog<span class="se">\u</span><span class="m">0</span>.16__0_5172CC2B__none
............
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>root<span class="w"> </span>root<span class="w"> </span><span class="m">0</span><span class="w"> </span>août<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">12</span>:12<span class="w"> </span>snapmapper__0_A468EC03__noneosd
</code></pre></div>
<p>Try decompiling crush map from osdmap :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>stat
e12:<span class="w"> </span><span class="m">3</span><span class="w"> </span>osds:<span class="w"> </span><span class="m">3</span><span class="w"> </span>up,<span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="k">in</span>
$<span class="w"> </span>osdmaptool<span class="w"> </span>osdmap.12__0_64170E7C__none<span class="w"> </span>--export-crush<span class="w"> </span>/tmp/crushmap.bin
osdmaptool:<span class="w"> </span>osdmap<span class="w"> </span>file<span class="w"> </span><span class="s1">'osdmap.12__0_64170E7C__none'</span>
osdmaptool:<span class="w"> </span>exported<span class="w"> </span>crush<span class="w"> </span>map<span class="w"> </span>to<span class="w"> </span>/tmp/crushmap.bin
$<span class="w"> </span>crushtool<span class="w"> </span>-d<span class="w"> </span>/tmp/crushmap.bin<span class="w"> </span>-o<span class="w"> </span>/tmp/crushmap.txt
$<span class="w"> </span>cat<span class="w"> </span>/tmp/crushmap.txt
<span class="c1"># begin crush map</span>
<span class="c1"># devices</span>
device<span class="w"> </span><span class="m">0</span><span class="w"> </span>osd.0
device<span class="w"> </span><span class="m">1</span><span class="w"> </span>osd.1
device<span class="w"> </span><span class="m">2</span><span class="w"> </span>osd.2
<span class="c1"># types</span>
<span class="nb">type</span><span class="w"> </span><span class="m">0</span><span class="w"> </span>osd
<span class="nb">type</span><span class="w"> </span><span class="m">1</span><span class="w"> </span>host
<span class="nb">type</span><span class="w"> </span><span class="m">2</span><span class="w"> </span>rack
<span class="nb">type</span><span class="w"> </span><span class="m">3</span><span class="w"> </span>row
<span class="nb">type</span><span class="w"> </span><span class="m">4</span><span class="w"> </span>room
<span class="nb">type</span><span class="w"> </span><span class="m">5</span><span class="w"> </span>datacenter
<span class="nb">type</span><span class="w"> </span><span class="m">6</span><span class="w"> </span>root
<span class="c1"># buckets</span>
host<span class="w"> </span>ceph-01<span class="w"> </span><span class="o">{</span>
<span class="w"> </span>id<span class="w"> </span>-2<span class="w"> </span><span class="c1"># do not change unnecessarily</span>
<span class="w"> </span><span class="c1"># weight 0.050</span>
<span class="w"> </span>alg<span class="w"> </span>straw
<span class="w"> </span><span class="nb">hash</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="c1"># rjenkins1</span>
<span class="w"> </span>item<span class="w"> </span>osd.0<span class="w"> </span>weight<span class="w"> </span><span class="m">0</span>.050
<span class="o">}</span>
host<span class="w"> </span>ceph-02<span class="w"> </span><span class="o">{</span>
<span class="w"> </span>id<span class="w"> </span>-3<span class="w"> </span><span class="c1"># do not change unnecessarily</span>
<span class="w"> </span><span class="c1"># weight 0.050</span>
<span class="w"> </span>alg<span class="w"> </span>straw
<span class="w"> </span><span class="nb">hash</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="c1"># rjenkins1</span>
<span class="w"> </span>item<span class="w"> </span>osd.1<span class="w"> </span>weight<span class="w"> </span><span class="m">0</span>.050
<span class="o">}</span>
host<span class="w"> </span>ceph-03<span class="w"> </span><span class="o">{</span>
<span class="w"> </span>id<span class="w"> </span>-4<span class="w"> </span><span class="c1"># do not change unnecessarily</span>
<span class="w"> </span><span class="c1"># weight 0.050</span>
<span class="w"> </span>alg<span class="w"> </span>straw
<span class="w"> </span><span class="nb">hash</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="c1"># rjenkins1</span>
<span class="w"> </span>item<span class="w"> </span>osd.2<span class="w"> </span>weight<span class="w"> </span><span class="m">0</span>.050
<span class="o">}</span>
root<span class="w"> </span>default<span class="w"> </span><span class="o">{</span>
<span class="w"> </span>id<span class="w"> </span>-1<span class="w"> </span><span class="c1"># do not change unnecessarily</span>
<span class="w"> </span><span class="c1"># weight 0.150</span>
<span class="w"> </span>alg<span class="w"> </span>straw
<span class="w"> </span><span class="nb">hash</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="c1"># rjenkins1</span>
<span class="w"> </span>item<span class="w"> </span>ceph-01<span class="w"> </span>weight<span class="w"> </span><span class="m">0</span>.050
<span class="w"> </span>item<span class="w"> </span>ceph-02<span class="w"> </span>weight<span class="w"> </span><span class="m">0</span>.050
<span class="w"> </span>item<span class="w"> </span>ceph-03<span class="w"> </span>weight<span class="w"> </span><span class="m">0</span>.050
<span class="o">}</span>
...
<span class="c1"># end crush map</span>
</code></pre></div>
<p>Ok it's what I expect. :)</p>
<p>The cluster is empty : </p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>find<span class="w"> </span>*_head<span class="w"> </span>-type<span class="w"> </span>f<span class="w"> </span><span class="p">|</span><span class="w"> </span>wc<span class="w"> </span>-l
<span class="m">0</span>
</code></pre></div>
<p>The directory list correspond to the 'ceph pg dump'</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="k">for</span><span class="w"> </span>dir<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="sb">`</span><span class="w"> </span>ceph<span class="w"> </span>pg<span class="w"> </span>dump<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'\[0,'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>cut<span class="w"> </span>-f1<span class="w"> </span><span class="sb">`</span><span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>-d<span class="w"> </span><span class="nv">$dir_head</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span>exist<span class="p">;</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span>nok<span class="p">;</span><span class="w"> </span><span class="k">fi</span><span class="p">;</span><span class="w"> </span><span class="k">done</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sort<span class="w"> </span><span class="p">|</span><span class="w"> </span>uniq<span class="w"> </span>-c
dumped<span class="w"> </span>all<span class="w"> </span><span class="k">in</span><span class="w"> </span>format<span class="w"> </span>plain
<span class="w"> </span><span class="m">69</span><span class="w"> </span>exist
</code></pre></div>
<p>To get all stats for a specific pg :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>pg<span class="w"> </span><span class="m">0</span>.1<span class="w"> </span>query
<span class="o">{</span><span class="w"> </span><span class="s2">"state"</span>:<span class="w"> </span><span class="s2">"active+clean"</span>,
<span class="w"> </span><span class="s2">"epoch"</span>:<span class="w"> </span><span class="m">12</span>,
<span class="w"> </span><span class="s2">"up"</span>:<span class="w"> </span><span class="o">[</span>
<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="m">1</span><span class="o">]</span>,
<span class="w"> </span><span class="s2">"acting"</span>:<span class="w"> </span><span class="o">[</span>
<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="m">1</span><span class="o">]</span>,
<span class="w"> </span><span class="s2">"info"</span>:<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"pgid"</span>:<span class="w"> </span><span class="s2">"0.1"</span>,
<span class="w"> </span><span class="s2">"last_update"</span>:<span class="w"> </span><span class="s2">"0'0"</span>,
<span class="w"> </span><span class="s2">"last_complete"</span>:<span class="w"> </span><span class="s2">"0'0"</span>,
<span class="w"> </span><span class="s2">"log_tail"</span>:<span class="w"> </span><span class="s2">"0'0"</span>,
<span class="w"> </span><span class="s2">"last_backfill"</span>:<span class="w"> </span><span class="s2">"MAX"</span>,
<span class="w"> </span><span class="s2">"purged_snaps"</span>:<span class="w"> </span><span class="s2">"[]"</span>,
<span class="w"> </span><span class="s2">"history"</span>:<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"epoch_created"</span>:<span class="w"> </span><span class="m">1</span>,
<span class="w"> </span><span class="s2">"last_epoch_started"</span>:<span class="w"> </span><span class="m">12</span>,
<span class="w"> </span><span class="s2">"last_epoch_clean"</span>:<span class="w"> </span><span class="m">12</span>,
<span class="w"> </span><span class="s2">"last_epoch_split"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"same_up_since"</span>:<span class="w"> </span><span class="m">9</span>,
<span class="w"> </span><span class="s2">"same_interval_since"</span>:<span class="w"> </span><span class="m">9</span>,
<span class="w"> </span><span class="s2">"same_primary_since"</span>:<span class="w"> </span><span class="m">5</span>,
<span class="w"> </span><span class="s2">"last_scrub"</span>:<span class="w"> </span><span class="s2">"0'0"</span>,
<span class="w"> </span><span class="s2">"last_scrub_stamp"</span>:<span class="w"> </span><span class="s2">"2013-08-20 12:12:37.851559"</span>,
<span class="w"> </span><span class="s2">"last_deep_scrub"</span>:<span class="w"> </span><span class="s2">"0'0"</span>,
<span class="w"> </span><span class="s2">"last_deep_scrub_stamp"</span>:<span class="w"> </span><span class="s2">"2013-08-20 12:12:37.851559"</span>,
<span class="w"> </span><span class="s2">"last_clean_scrub_stamp"</span>:<span class="w"> </span><span class="s2">"0.000000"</span><span class="o">}</span>,
<span class="w"> </span><span class="s2">"stats"</span>:<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"version"</span>:<span class="w"> </span><span class="s2">"0'0"</span>,
<span class="w"> </span><span class="s2">"reported_seq"</span>:<span class="w"> </span><span class="s2">"12"</span>,
<span class="w"> </span><span class="s2">"reported_epoch"</span>:<span class="w"> </span><span class="s2">"12"</span>,
<span class="w"> </span><span class="s2">"state"</span>:<span class="w"> </span><span class="s2">"active+clean"</span>,
<span class="w"> </span><span class="s2">"last_fresh"</span>:<span class="w"> </span><span class="s2">"2013-08-20 12:16:22.709534"</span>,
<span class="w"> </span><span class="s2">"last_change"</span>:<span class="w"> </span><span class="s2">"2013-08-20 12:16:22.105099"</span>,
<span class="w"> </span><span class="s2">"last_active"</span>:<span class="w"> </span><span class="s2">"2013-08-20 12:16:22.709534"</span>,
<span class="w"> </span><span class="s2">"last_clean"</span>:<span class="w"> </span><span class="s2">"2013-08-20 12:16:22.709534"</span>,
<span class="w"> </span><span class="s2">"last_became_active"</span>:<span class="w"> </span><span class="s2">"0.000000"</span>,
<span class="w"> </span><span class="s2">"last_unstale"</span>:<span class="w"> </span><span class="s2">"2013-08-20 12:16:22.709534"</span>,
<span class="w"> </span><span class="s2">"mapping_epoch"</span>:<span class="w"> </span><span class="m">5</span>,
<span class="w"> </span><span class="s2">"log_start"</span>:<span class="w"> </span><span class="s2">"0'0"</span>,
<span class="w"> </span><span class="s2">"ondisk_log_start"</span>:<span class="w"> </span><span class="s2">"0'0"</span>,
<span class="w"> </span><span class="s2">"created"</span>:<span class="w"> </span><span class="m">1</span>,
<span class="w"> </span><span class="s2">"last_epoch_clean"</span>:<span class="w"> </span><span class="m">12</span>,
<span class="w"> </span><span class="s2">"parent"</span>:<span class="w"> </span><span class="s2">"0.0"</span>,
<span class="w"> </span><span class="s2">"parent_split_bits"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"last_scrub"</span>:<span class="w"> </span><span class="s2">"0'0"</span>,
<span class="w"> </span><span class="s2">"last_scrub_stamp"</span>:<span class="w"> </span><span class="s2">"2013-08-20 12:12:37.851559"</span>,
<span class="w"> </span><span class="s2">"last_deep_scrub"</span>:<span class="w"> </span><span class="s2">"0'0"</span>,
<span class="w"> </span><span class="s2">"last_deep_scrub_stamp"</span>:<span class="w"> </span><span class="s2">"2013-08-20 12:12:37.851559"</span>,
<span class="w"> </span><span class="s2">"last_clean_scrub_stamp"</span>:<span class="w"> </span><span class="s2">"0.000000"</span>,
<span class="w"> </span><span class="s2">"log_size"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"ondisk_log_size"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"stats_invalid"</span>:<span class="w"> </span><span class="s2">"0"</span>,
<span class="w"> </span><span class="s2">"stat_sum"</span>:<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"num_bytes"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_objects"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_object_clones"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_object_copies"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_objects_missing_on_primary"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_objects_degraded"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_objects_unfound"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_read"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_read_kb"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_write"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_write_kb"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_scrub_errors"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_shallow_scrub_errors"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_deep_scrub_errors"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_objects_recovered"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_bytes_recovered"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"num_keys_recovered"</span>:<span class="w"> </span><span class="m">0</span><span class="o">}</span>,
<span class="w"> </span><span class="s2">"stat_cat_sum"</span>:<span class="w"> </span><span class="o">{}</span>,
<span class="w"> </span><span class="s2">"up"</span>:<span class="w"> </span><span class="o">[</span>
<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="m">1</span><span class="o">]</span>,
<span class="w"> </span><span class="s2">"acting"</span>:<span class="w"> </span><span class="o">[</span>
<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="m">1</span><span class="o">]}</span>,
<span class="w"> </span><span class="s2">"empty"</span>:<span class="w"> </span><span class="m">1</span>,
<span class="w"> </span><span class="s2">"dne"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"incomplete"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"last_epoch_started"</span>:<span class="w"> </span><span class="m">12</span><span class="o">}</span>,
<span class="w"> </span><span class="s2">"recovery_state"</span>:<span class="w"> </span><span class="o">[</span>
<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"name"</span>:<span class="w"> </span><span class="s2">"Started\/Primary\/Active"</span>,
<span class="w"> </span><span class="s2">"enter_time"</span>:<span class="w"> </span><span class="s2">"2013-08-20 12:15:30.102250"</span>,
<span class="w"> </span><span class="s2">"might_have_unfound"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"recovery_progress"</span>:<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"backfill_target"</span>:<span class="w"> </span>-1,
<span class="w"> </span><span class="s2">"waiting_on_backfill"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"backfill_pos"</span>:<span class="w"> </span><span class="s2">"0\/\/0\/\/-1"</span>,
<span class="w"> </span><span class="s2">"backfill_info"</span>:<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"begin"</span>:<span class="w"> </span><span class="s2">"0\/\/0\/\/-1"</span>,
<span class="w"> </span><span class="s2">"end"</span>:<span class="w"> </span><span class="s2">"0\/\/0\/\/-1"</span>,
<span class="w"> </span><span class="s2">"objects"</span>:<span class="w"> </span><span class="o">[]}</span>,
<span class="w"> </span><span class="s2">"peer_backfill_info"</span>:<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"begin"</span>:<span class="w"> </span><span class="s2">"0\/\/0\/\/-1"</span>,
<span class="w"> </span><span class="s2">"end"</span>:<span class="w"> </span><span class="s2">"0\/\/0\/\/-1"</span>,
<span class="w"> </span><span class="s2">"objects"</span>:<span class="w"> </span><span class="o">[]}</span>,
<span class="w"> </span><span class="s2">"backfills_in_flight"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"pull_from_peer"</span>:<span class="w"> </span><span class="o">[]</span>,
<span class="w"> </span><span class="s2">"pushing"</span>:<span class="w"> </span><span class="o">[]}</span>,
<span class="w"> </span><span class="s2">"scrub"</span>:<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"scrubber.epoch_start"</span>:<span class="w"> </span><span class="s2">"0"</span>,
<span class="w"> </span><span class="s2">"scrubber.active"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"scrubber.block_writes"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"scrubber.finalizing"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"scrubber.waiting_on"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"scrubber.waiting_on_whom"</span>:<span class="w"> </span><span class="o">[]}}</span>,
<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"name"</span>:<span class="w"> </span><span class="s2">"Started"</span>,
<span class="w"> </span><span class="s2">"enter_time"</span>:<span class="w"> </span><span class="s2">"2013-08-20 12:14:51.501628"</span><span class="o">}]}</span>
</code></pre></div>
<h3>Retrieve an object on the cluster</h3>
<p>In this test we create a standard pool (pgnum=8 and repli=2) </p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rados<span class="w"> </span>mkpool<span class="w"> </span>testpool
$<span class="w"> </span>wget<span class="w"> </span>-q<span class="w"> </span>http://ceph.com/docs/master/_static/logo.png
$<span class="w"> </span>md5sum<span class="w"> </span>logo.png
4c7c15e856737efc0d2d71abde3c6b28<span class="w"> </span>logo.png
$<span class="w"> </span>rados<span class="w"> </span>put<span class="w"> </span>-p<span class="w"> </span>testpool<span class="w"> </span>logo.png<span class="w"> </span>logo.png
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>map<span class="w"> </span>testpool<span class="w"> </span>logo.png
osdmap<span class="w"> </span>e14<span class="w"> </span>pool<span class="w"> </span><span class="s1">'testpool'</span><span class="w"> </span><span class="o">(</span><span class="m">3</span><span class="o">)</span><span class="w"> </span>object<span class="w"> </span><span class="s1">'logo.png'</span><span class="w"> </span>-><span class="w"> </span>pg<span class="w"> </span><span class="m">3</span>.9e17671a<span class="w"> </span><span class="o">(</span><span class="m">3</span>.2<span class="o">)</span><span class="w"> </span>-><span class="w"> </span>up<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span><span class="w"> </span>acting<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span>
</code></pre></div>
<p>My Ceph logo is on pg 3.2 (main on osd.2 and replica on osd.1)</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>tree
<span class="c1"># id weight type name up/down reweight</span>
-1<span class="w"> </span><span class="m">0</span>.15<span class="w"> </span>root<span class="w"> </span>default
-2<span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>host<span class="w"> </span>ceph-01
<span class="m">0</span><span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>osd.0<span class="w"> </span>up<span class="w"> </span><span class="m">1</span><span class="w"> </span>
-3<span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>host<span class="w"> </span>ceph-02
<span class="m">1</span><span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>osd.1<span class="w"> </span>up<span class="w"> </span><span class="m">1</span><span class="w"> </span>
-4<span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>host<span class="w"> </span>ceph-03
<span class="m">2</span><span class="w"> </span><span class="m">0</span>.04999<span class="w"> </span>osd.2<span class="w"> </span>up<span class="w"> </span><span class="m">1</span>
</code></pre></div>
<p>And osd.2 is on ceph-03 :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>/var/lib/ceph/osd/ceph-2/current/3.2_head/
$<span class="w"> </span>ls
logo.png__head_9E17671A__3
$<span class="w"> </span>md5sum<span class="w"> </span>logo.png__head_9E17671A__3
4c7c15e856737efc0d2d71abde3c6b28<span class="w"> </span>logo.png__head_9E17671A__3
</code></pre></div>
<p>It exactly the same :)</p>
<h3>Import RBD</h3>
<p>Same thing, but testing as a block device.</p>
<div class="highlight"><pre><span></span><code><span class="err">$</span> <span class="n">rbd</span> <span class="kn">import</span> <span class="nn">logo.png</span> <span class="n">testpool</span><span class="o">/</span><span class="n">logo</span><span class="o">.</span><span class="n">png</span>
<span class="n">Importing</span> <span class="n">image</span><span class="p">:</span> <span class="mi">100</span><span class="o">%</span> <span class="n">complete</span><span class="o">...</span><span class="n">done</span><span class="o">.</span>
<span class="err">$</span> <span class="n">rbd</span> <span class="n">info</span> <span class="n">testpool</span><span class="o">/</span><span class="n">logo</span><span class="o">.</span><span class="n">png</span>
<span class="n">rbd</span> <span class="n">image</span> <span class="s1">'logo.png'</span><span class="p">:</span>
<span class="n">size</span> <span class="mi">3898</span> <span class="nb">bytes</span> <span class="ow">in</span> <span class="mi">1</span> <span class="n">objects</span>
<span class="n">order</span> <span class="mi">22</span> <span class="p">(</span><span class="mi">4096</span> <span class="n">KB</span> <span class="n">objects</span><span class="p">)</span>
<span class="n">block_name_prefix</span><span class="p">:</span> <span class="n">rb</span><span class="mf">.0.1048.2</span><span class="n">ae8944a</span>
<span class="nb">format</span><span class="p">:</span> <span class="mi">1</span>
</code></pre></div>
<p>Only one object.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rados<span class="w"> </span>ls<span class="w"> </span>-p<span class="w"> </span>testpool
logo.png
rb.0.1048.2ae8944a.000000000000
rbd_directory
logo.png.rbd
$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>map<span class="w"> </span>testpool<span class="w"> </span>logo.png.rbd
osdmap<span class="w"> </span>e14<span class="w"> </span>pool<span class="w"> </span><span class="s1">'testpool'</span><span class="w"> </span><span class="o">(</span><span class="m">3</span><span class="o">)</span><span class="w"> </span>object<span class="w"> </span><span class="s1">'logo.png.rbd'</span><span class="w"> </span>-><span class="w"> </span>pg<span class="w"> </span><span class="m">3</span>.d592352c<span class="w"> </span><span class="o">(</span><span class="m">3</span>.4<span class="o">)</span><span class="w"> </span>-><span class="w"> </span>up<span class="w"> </span><span class="o">[</span><span class="m">0</span>,2<span class="o">]</span><span class="w"> </span>acting<span class="w"> </span><span class="o">[</span><span class="m">0</span>,2<span class="o">]</span>
</code></pre></div>
<p>Let's go.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>/var/lib/ceph/osd/ceph-0/current/3.4_head/
$<span class="w"> </span>cat<span class="w"> </span>logo.png.rbd__head_D592352C__3
<span class="o"><<<</span><span class="w"> </span>Rados<span class="w"> </span>Block<span class="w"> </span>Device<span class="w"> </span>Image<span class="w"> </span>>>>
rb.0.1048.2ae8944aRBD001.005:
</code></pre></div>
<p>Here we can retrieve the block name prefix of the rbd 'rb.0.1048.2ae8944a' :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>map<span class="w"> </span>testpool<span class="w"> </span>rb.0.1048.2ae8944a.000000000000
osdmap<span class="w"> </span>e14<span class="w"> </span>pool<span class="w"> </span><span class="s1">'testpool'</span><span class="w"> </span><span class="o">(</span><span class="m">3</span><span class="o">)</span><span class="w"> </span>object<span class="w"> </span><span class="s1">'rb.0.1048.2ae8944a.000000000000'</span><span class="w"> </span>-><span class="w"> </span>pg<span class="w"> </span><span class="m">3</span>.d512078b<span class="w"> </span><span class="o">(</span><span class="m">3</span>.3<span class="o">)</span><span class="w"> </span>-><span class="w"> </span>up<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span><span class="w"> </span>acting<span class="w"> </span><span class="o">[</span><span class="m">2</span>,1<span class="o">]</span>
</code></pre></div>
<p>On ceph-03 :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>/var/lib/ceph/osd/ceph-2/current/3.3_head
$<span class="w"> </span>md5sum<span class="w"> </span>rb.0.1048.2ae8944a.000000000000__head_D512078B__3
4c7c15e856737efc0d2d71abde3c6b28<span class="w"> </span>rb.0.1048.2ae8944a.000000000000__head_D512078B__3
</code></pre></div>
<p>We retrieve the file unchanged because it is not split :)</p>
<h3>Try RBD snapshot</h3>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="w"> </span><span class="n">rbd</span><span class="w"> </span><span class="n">snap</span><span class="w"> </span><span class="n">create</span><span class="w"> </span><span class="n">testpool</span><span class="o">/</span><span class="n">logo</span><span class="o">.</span><span class="n">png</span><span class="err">@</span><span class="n">snap1</span>
<span class="o">$</span><span class="w"> </span><span class="n">rbd</span><span class="w"> </span><span class="n">snap</span><span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="n">testpool</span><span class="o">/</span><span class="n">logo</span><span class="o">.</span><span class="n">png</span>
<span class="n">SNAPID</span><span class="w"> </span><span class="n">NAME</span><span class="w"> </span><span class="n">SIZE</span><span class="w"> </span>
<span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">snap1</span><span class="w"> </span><span class="mi">3898</span><span class="w"> </span><span class="n">bytes</span>
<span class="o">$</span><span class="w"> </span><span class="n">echo</span><span class="w"> </span><span class="s2">"testpool/logo.png"</span><span class="w"> </span><span class="o">>></span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">rbdmap</span>
<span class="o">$</span><span class="w"> </span><span class="n">service</span><span class="w"> </span><span class="n">rbdmap</span><span class="w"> </span><span class="n">reload</span>
<span class="p">[</span><span class="w"> </span><span class="n">ok</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="n">Starting</span><span class="w"> </span><span class="n">RBD</span><span class="w"> </span><span class="n">Mapping</span><span class="p">:</span><span class="w"> </span><span class="n">testpool</span><span class="o">/</span><span class="n">logo</span><span class="o">.</span><span class="n">png</span><span class="o">.</span>
<span class="p">[</span><span class="w"> </span><span class="n">ok</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="n">Mounting</span><span class="w"> </span><span class="n">all</span><span class="w"> </span><span class="n">filesystems</span><span class="o">...</span><span class="n">done</span><span class="o">.</span>
<span class="o">$</span><span class="w"> </span><span class="n">dd</span><span class="w"> </span><span class="k">if</span><span class="o">=/</span><span class="n">dev</span><span class="o">/</span><span class="n">zero</span><span class="w"> </span><span class="n">of</span><span class="o">=/</span><span class="n">dev</span><span class="o">/</span><span class="n">rbd</span><span class="o">/</span><span class="n">testpool</span><span class="o">/</span><span class="n">logo</span><span class="o">.</span><span class="n">png</span><span class="w"> </span>
<span class="n">dd</span><span class="p">:</span><span class="w"> </span><span class="err">é</span><span class="n">criture</span><span class="w"> </span><span class="n">vers</span><span class="w"> </span><span class="err">«</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">rbd</span><span class="o">/</span><span class="n">testpool</span><span class="o">/</span><span class="n">logo</span><span class="o">.</span><span class="n">png</span><span class="w"> </span><span class="err">»</span><span class="p">:</span><span class="w"> </span><span class="n">Aucun</span><span class="w"> </span><span class="n">espace</span><span class="w"> </span><span class="n">disponible</span><span class="w"> </span><span class="n">sur</span><span class="w"> </span><span class="n">le</span><span class="w"> </span><span class="n">périphérique</span>
<span class="mi">8</span><span class="o">+</span><span class="mi">0</span><span class="w"> </span><span class="n">enregistrements</span><span class="w"> </span><span class="n">lus</span>
<span class="mi">7</span><span class="o">+</span><span class="mi">0</span><span class="w"> </span><span class="n">enregistrements</span><span class="w"> </span><span class="err">é</span><span class="n">crits</span>
<span class="mi">3584</span><span class="w"> </span><span class="n">octets</span><span class="w"> </span><span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="mi">6</span><span class="w"> </span><span class="n">kB</span><span class="p">)</span><span class="w"> </span><span class="n">copiés</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="mi">285823</span><span class="w"> </span><span class="n">s</span><span class="p">,</span><span class="w"> </span><span class="mi">12</span><span class="p">,</span><span class="mi">5</span><span class="w"> </span><span class="n">kB</span><span class="o">/</span><span class="n">s</span>
<span class="o">$</span><span class="w"> </span><span class="n">ceph</span><span class="w"> </span><span class="n">osd</span><span class="w"> </span><span class="n">map</span><span class="w"> </span><span class="n">testpool</span><span class="w"> </span><span class="n">rb</span><span class="o">.</span><span class="mf">0.1048</span><span class="o">.</span><span class="mi">2</span><span class="n">ae8944a</span><span class="o">.</span><span class="mi">000000000000</span>
<span class="n">osdmap</span><span class="w"> </span><span class="n">e15</span><span class="w"> </span><span class="n">pool</span><span class="w"> </span><span class="s1">'testpool'</span><span class="w"> </span><span class="p">(</span><span class="mi">3</span><span class="p">)</span><span class="w"> </span><span class="n">object</span><span class="w"> </span><span class="s1">'rb.0.1048.2ae8944a.000000000000'</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">pg</span><span class="w"> </span><span class="mf">3.</span><span class="n">d512078b</span><span class="w"> </span><span class="p">(</span><span class="mf">3.3</span><span class="p">)</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">up</span><span class="w"> </span><span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">]</span><span class="w"> </span><span class="n">acting</span><span class="w"> </span><span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">]</span>
</code></pre></div>
<p>It's the same place on ceph-03 :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>/var/lib/ceph/osd/ceph-2/current/3.3_head
$<span class="w"> </span>md5sum<span class="w"> </span>*
4c7c15e856737efc0d2d71abde3c6b28<span class="w"> </span>rb.0.1048.2ae8944a.000000000000__2_D512078B__3
dd99129a16764a6727d3314b501e9c23<span class="w"> </span>rb.0.1048.2ae8944a.000000000000__head_D512078B__3
</code></pre></div>
<p>We can notice that file containing <em>2</em> (snap id 2) contain original data.
And a new file has been created for the current data : <em>head</em></p>
<p>For next tests, I will try with stripped files, rbd format 2 and snap on pool.</p>Samba shadow_copy and Ceph RBD2013-08-09T22:42:00+02:002013-08-09T22:42:00+02:00Laurent Barbetag:cephnotes.ksperis.com,2013-08-09:/blog/2013/08/09/samba-shadow-copy-and-ceph-rbd/<p>I add script to create snapshot on rbd for use with samba shadow_copy2.
For more details go on https://github.com/ksperis/autosnap-rbd-shadow-copy</p>
<!-- more -->
<h2>How to use :</h2>
<p>Before you need to have ceph cluster running and samba installed.</p>
<p>Verify admin access to the ceph cluster : (should not return error)</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>ls …</code></pre></div><p>I add script to create snapshot on rbd for use with samba shadow_copy2.
For more details go on https://github.com/ksperis/autosnap-rbd-shadow-copy</p>
<!-- more -->
<h2>How to use :</h2>
<p>Before you need to have ceph cluster running and samba installed.</p>
<p>Verify admin access to the ceph cluster : (should not return error)</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>ls
</code></pre></div>
<p>Get the script :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mkdir<span class="w"> </span>-p<span class="w"> </span>/etc/ceph/scripts/
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>/etc/ceph/scripts/
$<span class="w"> </span>wget<span class="w"> </span>https://raw.github.com/ksperis/autosnap-rbd-shadow-copy/master/autosnap.conf
$<span class="w"> </span>wget<span class="w"> </span>https://raw.github.com/ksperis/autosnap-rbd-shadow-copy/master/autosnap.sh
$<span class="w"> </span>chmod<span class="w"> </span>+x<span class="w"> </span>autosnap.sh
</code></pre></div>
<p>Create a block device :</p>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="w"> </span><span class="n">rbd</span><span class="w"> </span><span class="n">create</span><span class="w"> </span><span class="n">myshare</span><span class="w"> </span><span class="o">--</span><span class="n">size</span><span class="o">=</span><span class="mi">1024</span>
<span class="o">$</span><span class="w"> </span><span class="n">echo</span><span class="w"> </span><span class="s2">"myshare"</span><span class="w"> </span><span class="o">>></span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">rbdmap</span>
<span class="o">$</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">init</span><span class="o">.</span><span class="n">d</span><span class="o">/</span><span class="n">rbdmap</span><span class="w"> </span><span class="n">reload</span>
<span class="p">[</span><span class="w"> </span><span class="n">ok</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="n">Starting</span><span class="w"> </span><span class="n">RBD</span><span class="w"> </span><span class="n">Mapping</span><span class="p">:</span><span class="w"> </span><span class="n">rbd</span><span class="o">/</span><span class="n">myshare</span><span class="o">.</span>
<span class="p">[</span><span class="w"> </span><span class="n">ok</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="n">Mounting</span><span class="w"> </span><span class="n">all</span><span class="w"> </span><span class="n">filesystems</span><span class="o">...</span><span class="n">done</span><span class="o">.</span>
</code></pre></div>
<p>Format the block device :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mkfs.xfs<span class="w"> </span>/dev/rbd/rbd/myshare
log<span class="w"> </span>stripe<span class="w"> </span>unit<span class="w"> </span><span class="o">(</span><span class="m">4194304</span><span class="w"> </span>bytes<span class="o">)</span><span class="w"> </span>is<span class="w"> </span>too<span class="w"> </span>large<span class="w"> </span><span class="o">(</span>maximum<span class="w"> </span>is<span class="w"> </span>256KiB<span class="o">)</span>
log<span class="w"> </span>stripe<span class="w"> </span>unit<span class="w"> </span>adjusted<span class="w"> </span>to<span class="w"> </span>32KiB
meta-data<span class="o">=</span>/dev/rbd/rbd/myshare<span class="w"> </span><span class="nv">isize</span><span class="o">=</span><span class="m">256</span><span class="w"> </span><span class="nv">agcount</span><span class="o">=</span><span class="m">9</span>,<span class="w"> </span><span class="nv">agsize</span><span class="o">=</span><span class="m">31744</span><span class="w"> </span><span class="nv">blks</span>
<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">sectsz</span><span class="o">=</span><span class="m">512</span><span class="w"> </span><span class="nv">attr</span><span class="o">=</span><span class="m">2</span>,<span class="w"> </span><span class="nv">projid32bit</span><span class="o">=</span><span class="m">0</span>
<span class="nv">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">bsize</span><span class="o">=</span><span class="m">4096</span><span class="w"> </span><span class="nv">blocks</span><span class="o">=</span><span class="m">262144</span>,<span class="w"> </span><span class="nv">imaxpct</span><span class="o">=</span><span class="nv">25</span>
<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">sunit</span><span class="o">=</span><span class="m">1024</span><span class="w"> </span><span class="nv">swidth</span><span class="o">=</span><span class="m">1024</span><span class="w"> </span>blks
<span class="nv">naming</span><span class="w"> </span><span class="o">=</span>version<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="nv">bsize</span><span class="o">=</span><span class="m">4096</span><span class="w"> </span>ascii-ci<span class="o">=</span><span class="m">0</span>
<span class="nv">log</span><span class="w"> </span><span class="o">=</span>internal<span class="w"> </span>log<span class="w"> </span><span class="nv">bsize</span><span class="o">=</span><span class="m">4096</span><span class="w"> </span><span class="nv">blocks</span><span class="o">=</span><span class="m">2560</span>,<span class="w"> </span><span class="nv">version</span><span class="o">=</span><span class="nv">2</span>
<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">sectsz</span><span class="o">=</span><span class="m">512</span><span class="w"> </span><span class="nv">sunit</span><span class="o">=</span><span class="m">8</span><span class="w"> </span>blks,<span class="w"> </span>lazy-count<span class="o">=</span><span class="m">1</span>
<span class="nv">realtime</span><span class="w"> </span><span class="o">=</span>none<span class="w"> </span><span class="nv">extsz</span><span class="o">=</span><span class="m">4096</span><span class="w"> </span><span class="nv">blocks</span><span class="o">=</span><span class="m">0</span>,<span class="w"> </span><span class="nv">rtextents</span><span class="o">=</span><span class="m">0</span>
</code></pre></div>
<p>Mount the share :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mkdir<span class="w"> </span>/myshare
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"/dev/rbd/rbd/myshare /myshare xfs defaults 0 0"</span><span class="w"> </span>>><span class="w"> </span>/etc/fstab
$<span class="w"> </span>mount<span class="w"> </span>/myshare
</code></pre></div>
<p>Add this section in your /etc/samba/smb.conf :</p>
<div class="highlight"><pre><span></span><code><span class="k">[myshare]</span>
<span class="w"> </span><span class="na">path</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/myshare</span>
<span class="w"> </span><span class="na">writable</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">yes</span>
<span class="w"> </span><span class="na">vfs objects</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">shadow_copy2</span>
<span class="w"> </span><span class="na">shadow</span><span class="o">:</span><span class="s">snapdir = .snapshots</span>
<span class="w"> </span><span class="na">shadow</span><span class="o">:</span><span class="s">sort = desc</span>
</code></pre></div>
<p>Reload samba</p>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">init</span><span class="o">.</span><span class="n">d</span><span class="o">/</span><span class="n">samba</span><span class="w"> </span><span class="n">reload</span>
</code></pre></div>
<p>Create snapshot directory and run the script :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mkdir<span class="w"> </span>-p<span class="w"> </span>/myshare/.snapshots
$<span class="w"> </span>/etc/ceph/scripts/autosnap.sh
*<span class="w"> </span>Create<span class="w"> </span>snapshot<span class="w"> </span><span class="k">for</span><span class="w"> </span>myshare:<span class="w"> </span>@GMT-2013.08.09-10.16.10-autosnap
synced,<span class="w"> </span>no<span class="w"> </span>cache,<span class="w"> </span>snapshot<span class="w"> </span>created.
*<span class="w"> </span>Shadow<span class="w"> </span>Copy<span class="w"> </span>to<span class="w"> </span>mount<span class="w"> </span><span class="k">for</span><span class="w"> </span>rbd/myshare<span class="w"> </span>:
GMT-2013.08.09-10.14.44
</code></pre></div>
<p>Verify that the first snapshot is correctly mount :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mount<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>myshare
/dev/rbd1<span class="w"> </span>on<span class="w"> </span>/myshare<span class="w"> </span><span class="nb">type</span><span class="w"> </span>xfs<span class="w"> </span><span class="o">(</span>rw,relatime,attr2,inode64,sunit<span class="o">=</span><span class="m">8192</span>,swidth<span class="o">=</span><span class="m">8192</span>,noquota<span class="o">)</span>
/dev/rbd2<span class="w"> </span>on<span class="w"> </span>/myshare/.snapshots/@GMT-2013.08.09-10.14.44<span class="w"> </span><span class="nb">type</span><span class="w"> </span>xfs<span class="w"> </span><span class="o">(</span>ro,relatime,nouuid,norecovery,attr2,inode64,sunit<span class="o">=</span><span class="m">8192</span>,swidth<span class="o">=</span><span class="m">8192</span>,noquota<span class="o">)</span>
</code></pre></div>
<p>Also, you can add this on crontab to run everyday the script :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"00 0 * * * root /bin/bash /etc/ceph/scripts/autosnap.sh"</span><span class="w"> </span>>><span class="w"> </span>/etc/crontab
</code></pre></div>Test ceph persistant rbd device2013-08-03T23:40:00+02:002013-08-03T23:40:00+02:00Laurent Barbetag:cephnotes.ksperis.com,2013-08-03:/blog/2013/08/03/test-ceph-persistant-rbd-device/<h2>Create persistant rbd device</h2>
<p>Create block device and map it with /etc/ceph/rbdmap</p>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="w"> </span><span class="n">rbd</span><span class="w"> </span><span class="n">create</span><span class="w"> </span><span class="n">rbd</span><span class="o">/</span><span class="n">myrbd</span><span class="w"> </span><span class="o">--</span><span class="n">size</span><span class="o">=</span><span class="mi">1024</span>
<span class="o">$</span><span class="w"> </span><span class="n">echo</span><span class="w"> </span><span class="s2">"rbd/myrbd"</span><span class="w"> </span><span class="o">>></span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">rbdmap</span>
<span class="o">$</span><span class="w"> </span><span class="n">service</span><span class="w"> </span><span class="n">rbdmap</span><span class="w"> </span><span class="n">reload</span>
<span class="p">[</span><span class="w"> </span><span class="n">ok</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="n">Starting</span><span class="w"> </span><span class="n">RBD</span><span class="w"> </span><span class="n">Mapping</span><span class="p">:</span><span class="w"> </span><span class="n">rbd</span><span class="o">/</span><span class="n">myrbd</span><span class="o">.</span>
<span class="p">[</span><span class="w"> </span><span class="n">ok</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="n">Mounting</span><span class="w"> </span><span class="n">all</span><span class="w"> </span><span class="n">filesystems</span><span class="o">...</span><span class="n">done</span><span class="o">.</span>
</code></pre></div>
<p>View rbd mapped :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>showmapped
id<span class="w"> </span>pool<span class="w"> </span>image<span class="w"> </span>snap<span class="w"> </span>device …</code></pre></div><h2>Create persistant rbd device</h2>
<p>Create block device and map it with /etc/ceph/rbdmap</p>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="w"> </span><span class="n">rbd</span><span class="w"> </span><span class="n">create</span><span class="w"> </span><span class="n">rbd</span><span class="o">/</span><span class="n">myrbd</span><span class="w"> </span><span class="o">--</span><span class="n">size</span><span class="o">=</span><span class="mi">1024</span>
<span class="o">$</span><span class="w"> </span><span class="n">echo</span><span class="w"> </span><span class="s2">"rbd/myrbd"</span><span class="w"> </span><span class="o">>></span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">rbdmap</span>
<span class="o">$</span><span class="w"> </span><span class="n">service</span><span class="w"> </span><span class="n">rbdmap</span><span class="w"> </span><span class="n">reload</span>
<span class="p">[</span><span class="w"> </span><span class="n">ok</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="n">Starting</span><span class="w"> </span><span class="n">RBD</span><span class="w"> </span><span class="n">Mapping</span><span class="p">:</span><span class="w"> </span><span class="n">rbd</span><span class="o">/</span><span class="n">myrbd</span><span class="o">.</span>
<span class="p">[</span><span class="w"> </span><span class="n">ok</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="n">Mounting</span><span class="w"> </span><span class="n">all</span><span class="w"> </span><span class="n">filesystems</span><span class="o">...</span><span class="n">done</span><span class="o">.</span>
</code></pre></div>
<p>View rbd mapped :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>showmapped
id<span class="w"> </span>pool<span class="w"> </span>image<span class="w"> </span>snap<span class="w"> </span>device<span class="w"> </span>
<span class="m">1</span><span class="w"> </span>rbd<span class="w"> </span>myrbd<span class="w"> </span>-<span class="w"> </span>/dev/rbd1
</code></pre></div>
<!-- more -->
<p>Create FS and mount :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mkfs.xfs<span class="w"> </span>/dev/rbd/rbd/myrbd<span class="w"> </span>
log<span class="w"> </span>stripe<span class="w"> </span>unit<span class="w"> </span><span class="o">(</span><span class="m">4194304</span><span class="w"> </span>bytes<span class="o">)</span><span class="w"> </span>is<span class="w"> </span>too<span class="w"> </span>large<span class="w"> </span><span class="o">(</span>maximum<span class="w"> </span>is<span class="w"> </span>256KiB<span class="o">)</span>
log<span class="w"> </span>stripe<span class="w"> </span>unit<span class="w"> </span>adjusted<span class="w"> </span>to<span class="w"> </span>32KiB
meta-data<span class="o">=</span>/dev/rbd/rbd/myrbd<span class="w"> </span><span class="nv">isize</span><span class="o">=</span><span class="m">256</span><span class="w"> </span><span class="nv">agcount</span><span class="o">=</span><span class="m">9</span>,<span class="w"> </span><span class="nv">agsize</span><span class="o">=</span><span class="m">31744</span><span class="w"> </span><span class="nv">blks</span>
<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">sectsz</span><span class="o">=</span><span class="m">512</span><span class="w"> </span><span class="nv">attr</span><span class="o">=</span><span class="m">2</span>,<span class="w"> </span><span class="nv">projid32bit</span><span class="o">=</span><span class="m">0</span>
<span class="nv">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">bsize</span><span class="o">=</span><span class="m">4096</span><span class="w"> </span><span class="nv">blocks</span><span class="o">=</span><span class="m">262144</span>,<span class="w"> </span><span class="nv">imaxpct</span><span class="o">=</span><span class="nv">25</span>
<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">sunit</span><span class="o">=</span><span class="m">1024</span><span class="w"> </span><span class="nv">swidth</span><span class="o">=</span><span class="m">1024</span><span class="w"> </span>blks
<span class="nv">naming</span><span class="w"> </span><span class="o">=</span>version<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="nv">bsize</span><span class="o">=</span><span class="m">4096</span><span class="w"> </span>ascii-ci<span class="o">=</span><span class="m">0</span>
<span class="nv">log</span><span class="w"> </span><span class="o">=</span>internal<span class="w"> </span>log<span class="w"> </span><span class="nv">bsize</span><span class="o">=</span><span class="m">4096</span><span class="w"> </span><span class="nv">blocks</span><span class="o">=</span><span class="m">2560</span>,<span class="w"> </span><span class="nv">version</span><span class="o">=</span><span class="nv">2</span>
<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">sectsz</span><span class="o">=</span><span class="m">512</span><span class="w"> </span><span class="nv">sunit</span><span class="o">=</span><span class="m">8</span><span class="w"> </span>blks,<span class="w"> </span>lazy-count<span class="o">=</span><span class="m">1</span>
<span class="nv">realtime</span><span class="w"> </span><span class="o">=</span>none<span class="w"> </span><span class="nv">extsz</span><span class="o">=</span><span class="m">4096</span><span class="w"> </span><span class="nv">blocks</span><span class="o">=</span><span class="m">0</span>,<span class="w"> </span><span class="nv">rtextents</span><span class="o">=</span><span class="m">0</span>
$<span class="w"> </span>mkdir<span class="w"> </span>-p<span class="w"> </span>/mnt/myrbd
$<span class="w"> </span>blkid<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>rbd1
/dev/rbd1:<span class="w"> </span><span class="nv">UUID</span><span class="o">=</span><span class="s2">"a07e969e-bb1a-4921-9171-82cf7a737a69"</span><span class="w"> </span><span class="nv">TYPE</span><span class="o">=</span><span class="s2">"xfs"</span>
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"UUID=a07e969e-bb1a-4921-9171-82cf7a737a69 /mnt/myrbd xfs defaults 0 0"</span><span class="w"> </span>>><span class="w"> </span>/etc/fstab
$<span class="w"> </span>mount<span class="w"> </span>-a
</code></pre></div>
<p>Check :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mount<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>rbd1
/dev/rbd1<span class="w"> </span>on<span class="w"> </span>/mnt/myrbd<span class="w"> </span><span class="nb">type</span><span class="w"> </span>xfs<span class="w"> </span><span class="o">(</span>rw,relatime,attr2,inode64,sunit<span class="o">=</span><span class="m">8192</span>,swidth<span class="o">=</span><span class="m">8192</span>,noquota<span class="o">)</span>
</code></pre></div>
<h2>Test snapshot</h2>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>touch<span class="w"> </span>/mnt/myrbd/v1
</code></pre></div>
<p>Make snapshot :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sync<span class="w"> </span><span class="o">&&</span><span class="w"> </span>xfs_freeze<span class="w"> </span>-f<span class="w"> </span>/mnt/
$<span class="w"> </span>rbd<span class="w"> </span>snap<span class="w"> </span>create<span class="w"> </span>rbd/myrbd@snap1
$<span class="w"> </span>xfs_freeze<span class="w"> </span>-u<span class="w"> </span>/mnt/
</code></pre></div>
<p>Change a file :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mv<span class="w"> </span>/mnt/myrbd/v1<span class="w"> </span>/mnt/myrbd/v2
</code></pre></div>
<p>Mount snapshot in RO :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mkdir<span class="w"> </span>-p<span class="w"> </span>/mnt/myrbd@snap1
$<span class="w"> </span>rbd<span class="w"> </span>map<span class="w"> </span>rbd/myrbd@snap1
$<span class="w"> </span>mount<span class="w"> </span>-t<span class="w"> </span>xfs<span class="w"> </span>-o<span class="w"> </span>ro,norecovery,nouuid<span class="w"> </span><span class="s2">"/dev/rbd/rbd/myrbd@snap1"</span><span class="w"> </span><span class="s2">"/mnt/myrbd@snap1"</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span><span class="s2">"/mnt/myrbd"</span>
total<span class="w"> </span><span class="m">0</span>
v2
</code></pre></div>
<p>OK.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span><span class="s2">"/mnt/myrbd@snap1"</span>
total<span class="w"> </span><span class="m">0</span>
</code></pre></div>
<p>Nothing ??? Something went wrong with the sync ?</p>
<p>Try again :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sync<span class="w"> </span><span class="o">&&</span><span class="w"> </span>xfs_freeze<span class="w"> </span>-f<span class="w"> </span>/mnt/
$<span class="w"> </span>rbd<span class="w"> </span>snap<span class="w"> </span>create<span class="w"> </span>rbd/myrbd@snap2
$<span class="w"> </span>xfs_freeze<span class="w"> </span>-u<span class="w"> </span>/mnt/
$<span class="w"> </span>mkdir<span class="w"> </span>-p<span class="w"> </span>/mnt/myrbd@snap2
$<span class="w"> </span>rbd<span class="w"> </span>map<span class="w"> </span>rbd/myrbd@snap2
$<span class="w"> </span>mount<span class="w"> </span>-t<span class="w"> </span>xfs<span class="w"> </span>-o<span class="w"> </span>ro,norecovery,nouuid<span class="w"> </span><span class="s2">"/dev/rbd/rbd/myrbd@snap2"</span><span class="w"> </span><span class="s2">"/mnt/myrbd@snap2"</span>
</code></pre></div>
<p>Move again the file.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mv<span class="w"> </span>/mnt/myrbd/v2<span class="w"> </span>/mnt/myrbd/v3
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>/mnt/myrbd@snap2
total<span class="w"> </span><span class="m">0</span>
v2
$<span class="w"> </span>ls<span class="w"> </span>/mnt/myrbd
total<span class="w"> </span><span class="m">0</span>
v3
</code></pre></div>
<p>All right.</p>
<p>Stop rbdmap (will remove all rbd mapped device)</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>service<span class="w"> </span>rbdmap<span class="w"> </span>remove
</code></pre></div>
<p>Remove line added in /etc/ceph/rbdmap</p>
<p>Remove myrbd :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>snap<span class="w"> </span>purge<span class="w"> </span>rbd/myrbd
Removing<span class="w"> </span>all<span class="w"> </span>snapshots:<span class="w"> </span><span class="m">100</span>%<span class="w"> </span>complete...done.
$<span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbd/myrbd
Removing<span class="w"> </span>image:<span class="w"> </span><span class="m">100</span>%<span class="w"> </span>complete...done.
</code></pre></div>Don't forget unmap before remove rbd2013-08-02T20:56:00+02:002013-08-02T20:56:00+02:00Laurent Barbetag:cephnotes.ksperis.com,2013-08-02:/blog/2013/08/02/dont-forget-unmap-before-remove-rbd/<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbd/myrbd
Removing<span class="w"> </span>image:<span class="w"> </span><span class="m">99</span>%<span class="w"> </span>complete...failed.2013-08-02<span class="w"> </span><span class="m">14</span>:07:17.530470<span class="w"> </span>7f3ba2692760<span class="w"> </span>-1<span class="w"> </span>librbd:<span class="w"> </span>error<span class="w"> </span>removing<span class="w"> </span>header:<span class="w"> </span><span class="o">(</span><span class="m">16</span><span class="o">)</span><span class="w"> </span>Device<span class="w"> </span>or<span class="w"> </span>resource<span class="w"> </span>busy
rbd:<span class="w"> </span>error:<span class="w"> </span>image<span class="w"> </span>still<span class="w"> </span>has<span class="w"> </span>watchers
This<span class="w"> </span>means<span class="w"> </span>the<span class="w"> </span>image<span class="w"> </span>is<span class="w"> </span>still<span class="w"> </span>open<span class="w"> </span>or<span class="w"> </span>the<span class="w"> </span>client<span class="w"> </span>using<span class="w"> </span>it<span class="w"> </span>crashed.<span class="w"> </span>Try<span class="w"> </span>again<span class="w"> </span>after<span class="w"> </span>closing/unmapping<span class="w"> </span>it …</code></pre></div><div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbd/myrbd
Removing<span class="w"> </span>image:<span class="w"> </span><span class="m">99</span>%<span class="w"> </span>complete...failed.2013-08-02<span class="w"> </span><span class="m">14</span>:07:17.530470<span class="w"> </span>7f3ba2692760<span class="w"> </span>-1<span class="w"> </span>librbd:<span class="w"> </span>error<span class="w"> </span>removing<span class="w"> </span>header:<span class="w"> </span><span class="o">(</span><span class="m">16</span><span class="o">)</span><span class="w"> </span>Device<span class="w"> </span>or<span class="w"> </span>resource<span class="w"> </span>busy
rbd:<span class="w"> </span>error:<span class="w"> </span>image<span class="w"> </span>still<span class="w"> </span>has<span class="w"> </span>watchers
This<span class="w"> </span>means<span class="w"> </span>the<span class="w"> </span>image<span class="w"> </span>is<span class="w"> </span>still<span class="w"> </span>open<span class="w"> </span>or<span class="w"> </span>the<span class="w"> </span>client<span class="w"> </span>using<span class="w"> </span>it<span class="w"> </span>crashed.<span class="w"> </span>Try<span class="w"> </span>again<span class="w"> </span>after<span class="w"> </span>closing/unmapping<span class="w"> </span>it<span class="w"> </span>or<span class="w"> </span>waiting<span class="w"> </span>30s<span class="w"> </span><span class="k">for</span><span class="w"> </span>the<span class="w"> </span>crashed<span class="w"> </span>client<span class="w"> </span>to<span class="w"> </span>timeout.
</code></pre></div>
<!-- more -->
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>showmapped
id<span class="w"> </span>pool<span class="w"> </span>image<span class="w"> </span>snap<span class="w"> </span>device<span class="w"> </span>
<span class="m">1</span><span class="w"> </span>rbd<span class="w"> </span>myrbd<span class="w"> </span>-<span class="w"> </span>/dev/rbd1
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>service<span class="w"> </span>rbdmap<span class="w"> </span>stop
<span class="o">[</span><span class="w"> </span>ok<span class="w"> </span><span class="o">]</span><span class="w"> </span>Stopping<span class="w"> </span>RBD<span class="w"> </span>Mapping:<span class="w"> </span>/dev/rbd1
$<span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbd/myrbd
Removing<span class="w"> </span>image:<span class="w"> </span><span class="m">100</span>%<span class="w"> </span>complete...done.
$<span class="w"> </span>vim<span class="w"> </span>/etc/ceph/rbdmap<span class="w"> </span>
</code></pre></div>
<p>To know who is using rbd device you can use listwatchers :</p>
<h2>Since Hammer</h2>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>status<span class="w"> </span>myrbd
Watchers:
<span class="w"> </span><span class="nv">watcher</span><span class="o">=</span><span class="m">10</span>.2.0.131:0/1013964<span class="w"> </span>client.34453<span class="w"> </span><span class="nv">cookie</span><span class="o">=</span><span class="m">1</span>
</code></pre></div>
<h2>Before Hammer</h2>
<div class="highlight"><pre><span></span><code><span class="gh">#</span> Image format 1 :
<span class="gh">#</span> ----------------
$ rados -p rbd listwatchers myrbd.rbd
watcher=10.2.0.131:0/1013964 client.34453 cookie=1
<span class="gh">#</span> Image format 2 :
<span class="gh">#</span> ----------------
$ rbd info myrbd
rbd image 'myrbd':
size 1024 TB in 268435456 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.82072ae8944a
format: 2
features: layering
$ rados -p rbd listwatchers rbd_header.82072ae8944a
watcher=10.2.0.131:0/1013964 client.34453 cookie=1
</code></pre></div>Convert RBD to format v22013-07-30T23:40:00+02:002013-07-30T23:40:00+02:00Laurent Barbetag:cephnotes.ksperis.com,2013-07-30:/blog/2013/07/30/convert-rbd-to-format-v2/<h2>Simple Import / Export</h2>
<p>Don't forget to stop IO before sync and unmap rbd before rename.</p>
<div class="highlight"><pre><span></span><code><span class="err">$</span> <span class="n">rbd</span> <span class="n">export</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd</span> <span class="o">-</span> <span class="o">|</span> <span class="n">rbd</span> <span class="kn">import</span> <span class="o">--</span><span class="n">image</span><span class="o">-</span><span class="nb">format</span> <span class="mi">2</span> <span class="o">-</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd_v2</span>
<span class="err">$</span> <span class="n">rbd</span> <span class="n">mv</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd_old</span>
<span class="err">$</span> <span class="n">rbd</span> <span class="n">mv</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd_v2</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd</span>
</code></pre></div>
<!-- more -->
<p>Check :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>info<span class="w"> </span>rbd/myrbd
$<span class="w"> </span>rbd<span class="w"> </span>image<span class="w"> </span><span class="s1">'myrbd'</span>:
<span class="w"> </span>size<span class="w"> </span><span class="m">102400</span><span class="w"> </span>KB<span class="w"> </span><span class="k">in …</span></code></pre></div><h2>Simple Import / Export</h2>
<p>Don't forget to stop IO before sync and unmap rbd before rename.</p>
<div class="highlight"><pre><span></span><code><span class="err">$</span> <span class="n">rbd</span> <span class="n">export</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd</span> <span class="o">-</span> <span class="o">|</span> <span class="n">rbd</span> <span class="kn">import</span> <span class="o">--</span><span class="n">image</span><span class="o">-</span><span class="nb">format</span> <span class="mi">2</span> <span class="o">-</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd_v2</span>
<span class="err">$</span> <span class="n">rbd</span> <span class="n">mv</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd_old</span>
<span class="err">$</span> <span class="n">rbd</span> <span class="n">mv</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd_v2</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd</span>
</code></pre></div>
<!-- more -->
<p>Check :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>info<span class="w"> </span>rbd/myrbd
$<span class="w"> </span>rbd<span class="w"> </span>image<span class="w"> </span><span class="s1">'myrbd'</span>:
<span class="w"> </span>size<span class="w"> </span><span class="m">102400</span><span class="w"> </span>KB<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">25</span><span class="w"> </span>objects
<span class="w"> </span>order<span class="w"> </span><span class="m">22</span><span class="w"> </span><span class="o">(</span><span class="m">4096</span><span class="w"> </span>KB<span class="w"> </span>objects<span class="o">)</span>
<span class="w"> </span>block_name_prefix:<span class="w"> </span>rbd_data.24f72ae8944a
<span class="w"> </span>format:<span class="w"> </span><span class="m">2</span>
<span class="w"> </span>features:<span class="w"> </span>layering
</code></pre></div>
<h2>Using DIFF</h2>
<p>For the first sync, you don't need to stop IO on device.
Make snap, and sync until this snapshot.</p>
<div class="highlight"><pre><span></span><code><span class="err">$</span> <span class="n">rbd</span> <span class="n">snap</span> <span class="n">create</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd</span><span class="nd">@snap1</span>
<span class="err">$</span> <span class="n">rbd</span> <span class="n">export</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd</span><span class="nd">@snap1</span> <span class="o">-</span> <span class="o">|</span> <span class="n">rbd</span> <span class="kn">import</span> <span class="o">--</span><span class="n">image</span><span class="o">-</span><span class="nb">format</span> <span class="mi">2</span> <span class="o">-</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd_v2</span>
<span class="err">$</span> <span class="n">rbd</span> <span class="n">snap</span> <span class="n">create</span> <span class="n">rbd</span><span class="o">/</span><span class="n">myrbd_v2</span><span class="nd">@snap1</span>
</code></pre></div>
<p>Check snapshot :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>snap<span class="w"> </span>ls<span class="w"> </span>rbd/myrbd_v2
</code></pre></div>
<p>Sync diff :
Stop IO before last sync</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>snap<span class="w"> </span>create<span class="w"> </span>rbd/myrbd@snap2
$<span class="w"> </span>rbd<span class="w"> </span>export-diff<span class="w"> </span>--from-snap<span class="w"> </span>snap1<span class="w"> </span>rbd/myrbd@snap2<span class="w"> </span>-<span class="w"> </span><span class="p">|</span><span class="w"> </span>rbd<span class="w"> </span>import-diff<span class="w"> </span>-<span class="w"> </span>rbd/myrbd_v2
</code></pre></div>Remove snapshot before rbd2013-07-30T23:28:00+02:002013-07-30T23:28:00+02:00Laurent Barbetag:cephnotes.ksperis.com,2013-07-30:/blog/2013/07/30/remove-snapshot-before-rbd/<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbd/myrbd
<span class="m">2013</span>-07-30<span class="w"> </span><span class="m">14</span>:10:13.341184<span class="w"> </span>7f9e11922760<span class="w"> </span>-1<span class="w"> </span>librbd:<span class="w"> </span>image<span class="w"> </span>has<span class="w"> </span>snapshots<span class="w"> </span>-<span class="w"> </span>not<span class="w"> </span>removing
Removing<span class="w"> </span>image:<span class="w"> </span><span class="m">0</span>%<span class="w"> </span>complete...failed.
rbd:<span class="w"> </span>image<span class="w"> </span>has<span class="w"> </span>snapshots<span class="w"> </span>-<span class="w"> </span>these<span class="w"> </span>must<span class="w"> </span>be<span class="w"> </span>deleted<span class="w"> </span>with<span class="w"> </span><span class="s1">'rbd snap purge'</span><span class="w"> </span>before<span class="w"> </span>the<span class="w"> </span>image<span class="w"> </span>can<span class="w"> </span>be<span class="w"> </span>removed.
$<span class="w"> </span>rbd<span class="w"> </span>snap<span class="w"> </span>ls<span class="w"> </span>rbd/myrbd
SNAPID<span class="w"> </span>NAME<span class="w"> </span>SIZE<span class="w"> </span>
<span class="w"> </span><span class="m">10 …</span></code></pre></div><div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbd/myrbd
<span class="m">2013</span>-07-30<span class="w"> </span><span class="m">14</span>:10:13.341184<span class="w"> </span>7f9e11922760<span class="w"> </span>-1<span class="w"> </span>librbd:<span class="w"> </span>image<span class="w"> </span>has<span class="w"> </span>snapshots<span class="w"> </span>-<span class="w"> </span>not<span class="w"> </span>removing
Removing<span class="w"> </span>image:<span class="w"> </span><span class="m">0</span>%<span class="w"> </span>complete...failed.
rbd:<span class="w"> </span>image<span class="w"> </span>has<span class="w"> </span>snapshots<span class="w"> </span>-<span class="w"> </span>these<span class="w"> </span>must<span class="w"> </span>be<span class="w"> </span>deleted<span class="w"> </span>with<span class="w"> </span><span class="s1">'rbd snap purge'</span><span class="w"> </span>before<span class="w"> </span>the<span class="w"> </span>image<span class="w"> </span>can<span class="w"> </span>be<span class="w"> </span>removed.
$<span class="w"> </span>rbd<span class="w"> </span>snap<span class="w"> </span>ls<span class="w"> </span>rbd/myrbd
SNAPID<span class="w"> </span>NAME<span class="w"> </span>SIZE<span class="w"> </span>
<span class="w"> </span><span class="m">10</span><span class="w"> </span>snap1<span class="w"> </span><span class="m">102400</span><span class="w"> </span>KB<span class="w"> </span>
<span class="w"> </span><span class="m">12</span><span class="w"> </span>snap2<span class="w"> </span><span class="m">102400</span><span class="w"> </span>KB<span class="w"> </span>
$<span class="w"> </span>rbd<span class="w"> </span>snap<span class="w"> </span>purge<span class="w"> </span>rbd/myrbd
Removing<span class="w"> </span>all<span class="w"> </span>snapshots:<span class="w"> </span><span class="m">100</span>%<span class="w"> </span>complete...done.
$<span class="w"> </span>rbd<span class="w"> </span>rm<span class="w"> </span>rbd/myrbd
Removing<span class="w"> </span>image:<span class="w"> </span><span class="m">100</span>%<span class="w"> </span>complete...done.
</code></pre></div>Using ceph-deploy2013-07-30T23:25:00+02:002013-07-30T23:25:00+02:00Laurent Barbetag:cephnotes.ksperis.com,2013-07-30:/blog/2013/07/30/using-ceph-deploy/<h2>Install the ceph cluster</h2>
<h3>On each node :</h3>
<p>create a user "ceph" and configure sudo for nopassword :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>useradd<span class="w"> </span>-d<span class="w"> </span>/home/ceph<span class="w"> </span>-m<span class="w"> </span>ceph
$<span class="w"> </span>passwd<span class="w"> </span>ceph
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"ceph ALL = (root) NOPASSWD:ALL"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sudo<span class="w"> </span>tee<span class="w"> </span>/etc/sudoers.d/ceph
$<span class="w"> </span>chmod<span class="w"> </span><span class="m">0440</span><span class="w"> </span>/etc/sudoers.d/ceph
</code></pre></div>
<p>Update hosts file</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>vim<span class="w"> </span>/etc/hosts
<span class="m">192</span>.168 …</code></pre></div><h2>Install the ceph cluster</h2>
<h3>On each node :</h3>
<p>create a user "ceph" and configure sudo for nopassword :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>useradd<span class="w"> </span>-d<span class="w"> </span>/home/ceph<span class="w"> </span>-m<span class="w"> </span>ceph
$<span class="w"> </span>passwd<span class="w"> </span>ceph
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"ceph ALL = (root) NOPASSWD:ALL"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sudo<span class="w"> </span>tee<span class="w"> </span>/etc/sudoers.d/ceph
$<span class="w"> </span>chmod<span class="w"> </span><span class="m">0440</span><span class="w"> </span>/etc/sudoers.d/ceph
</code></pre></div>
<p>Update hosts file</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>vim<span class="w"> </span>/etc/hosts
<span class="m">192</span>.168.0.100<span class="w"> </span>cephnode-01<span class="w"> </span>cephnode-01.local
<span class="m">192</span>.168.0.101<span class="w"> </span>cephnode-02<span class="w"> </span>cephnode-02.local
<span class="m">192</span>.168.0.102<span class="w"> </span>cephnode-03<span class="w"> </span>cephnode-03.local
<span class="m">192</span>.168.0.103<span class="w"> </span>cephnode-04<span class="w"> </span>cephnode-04.local
<span class="m">192</span>.168.0.104<span class="w"> </span>cephnode-05<span class="w"> </span>cephnode-05.local
</code></pre></div>
<h3>On admin server</h3>
<p>(for me on cephnode-01)</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ssh-keygen
</code></pre></div>
<p>Deploy the key on each node</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nv">cluster</span><span class="o">=</span><span class="s2">"cephnode-01 cephnode-02 cephnode-03 cephnode-04 cephnode-05"</span>
$<span class="w"> </span><span class="k">for</span><span class="w"> </span>i<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="nv">$cluster</span><span class="p">;</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span>ssh-copy-id<span class="w"> </span>ceph@<span class="nv">$i</span>
<span class="w"> </span><span class="k">done</span>
</code></pre></div>
<p>Add option on ssh config to connect with ceph user :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>vim<span class="w"> </span>/root/.ssh/config
Host<span class="w"> </span>ceph*
User<span class="w"> </span>ceph
</code></pre></div>
<p>Install ceph-deploy (Dumpling Version)</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>wget<span class="w"> </span>-q<span class="w"> </span>-O-<span class="w"> </span><span class="s1">'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sudo<span class="w"> </span>apt-key<span class="w"> </span>add<span class="w"> </span>-
$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>deb<span class="w"> </span>http://ceph.com/debian-dumpling/<span class="w"> </span><span class="k">$(</span>lsb_release<span class="w"> </span>-sc<span class="k">)</span><span class="w"> </span>main<span class="w"> </span><span class="p">|</span><span class="w"> </span>sudo<span class="w"> </span>tee<span class="w"> </span>/etc/apt/sources.list.d/ceph.list
$<span class="w"> </span>apt-get<span class="w"> </span>update
$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>python-pkg-resources<span class="w"> </span>python-setuptools<span class="w"> </span>ceph-deploy
</code></pre></div>
<!-- more -->
<p>Install ceph on the cluster :</p>
<p>Before you need to create partition on ssd device (if use seperate journal)</p>
<p>For my exemple, I use :
sda 1: system partition
2: swap
5: osd journal (10 GB)
6: osd journal (10 GB)
7: osd journal (10 GB)
sdb : osd
sdc : osd
sdd : osd</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mkdir<span class="w"> </span>ceph-deploy<span class="p">;</span><span class="w"> </span><span class="nb">cd</span><span class="w"> </span>ceph-deploy
$<span class="w"> </span>ceph-deploy<span class="w"> </span>install<span class="w"> </span><span class="nv">$cluster</span>
$<span class="w"> </span>ceph-deploy<span class="w"> </span>new<span class="w"> </span>cephnode-01<span class="w"> </span>cephnode-02<span class="w"> </span>cephnode-03
$<span class="w"> </span>ceph-deploy<span class="w"> </span>--overwrite-conf<span class="w"> </span>mon<span class="w"> </span>create<span class="w"> </span>cephnode-01<span class="w"> </span>cephnode-02<span class="w"> </span>cephnode-03
$<span class="w"> </span>ceph-deploy<span class="w"> </span>gatherkeys<span class="w"> </span>cephnode-01
$<span class="w"> </span>ceph-deploy<span class="w"> </span>osd<span class="w"> </span>create<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-01:/dev/sdb:/dev/sda5<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-01:/dev/sdc:/dev/sda6<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-01:/dev/sdd:/dev/sda7<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-02:/dev/sdb:/dev/sda5<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-02:/dev/sdc:/dev/sda6<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-02:/dev/sdd:/dev/sda7<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-03:/dev/sdb:/dev/sda5<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-03:/dev/sdc:/dev/sda6<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-03:/dev/sdd:/dev/sda7<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-04:/dev/sdb:/dev/sda5<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-04:/dev/sdc:/dev/sda6<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-04:/dev/sdd:/dev/sda7<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-05:/dev/sdb:/dev/sda5<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-05:/dev/sdc:/dev/sda6<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>cephnode-05:/dev/sdd:/dev/sda7
</code></pre></div>
<h2>Destroy cluster and remove all data</h2>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ceph-deploy<span class="w"> </span>purgedata<span class="w"> </span><span class="nv">$cluster</span>
$<span class="w"> </span>ceph-deploy<span class="w"> </span>purge<span class="w"> </span><span class="nv">$cluster</span>
$<span class="w"> </span><span class="k">for</span><span class="w"> </span>host<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="nv">$cluster</span>
<span class="w"> </span><span class="k">do</span>
<span class="w"> </span>ssh<span class="w"> </span><span class="nv">$host</span><span class="w"> </span><span class="s"><<EOF</span>
<span class="s"> sudo dd if=/dev/zero of=/dev/sdb bs=1M count=100</span>
<span class="s"> sudo dd if=/dev/zero of=/dev/sdc bs=1M count=100</span>
<span class="s"> sudo dd if=/dev/zero of=/dev/sdd bs=1M count=100</span>
<span class="s"> sudo sgdisk -g --clear /dev/sdb</span>
<span class="s"> sudo sgdisk -g --clear /dev/sdc</span>
<span class="s"> sudo sgdisk -g --clear /dev/sdd</span>
<span class="s"> EOF</span>
<span class="w"> </span><span class="k">done</span>
</code></pre></div>Ceph RBD Online Resize2013-06-11T14:14:00+02:002013-06-11T14:14:00+02:00Laurent Barbetag:cephnotes.ksperis.com,2013-06-11:/blog/2013/06/11/ceph-rbd-online-resize/<h2>Extend rbd drive with libvirt and XFS</h2>
<p>First, resize the device on the physical host.</p>
<p>Get the current size :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>qemu-img<span class="w"> </span>info<span class="w"> </span>-f<span class="w"> </span>rbd<span class="w"> </span><span class="s2">"rbd:rbd/myrbd"</span>
</code></pre></div>
<p>Be careful, you must specify a bigger size, shrink a volume is destructive for the FS.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>qemu-img<span class="w"> </span>resize<span class="w"> </span>-f<span class="w"> </span>rbd<span class="w"> </span><span class="s2">"rbd:rbd/myrbd"</span><span class="w"> </span>600G …</code></pre></div><h2>Extend rbd drive with libvirt and XFS</h2>
<p>First, resize the device on the physical host.</p>
<p>Get the current size :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>qemu-img<span class="w"> </span>info<span class="w"> </span>-f<span class="w"> </span>rbd<span class="w"> </span><span class="s2">"rbd:rbd/myrbd"</span>
</code></pre></div>
<p>Be careful, you must specify a bigger size, shrink a volume is destructive for the FS.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>qemu-img<span class="w"> </span>resize<span class="w"> </span>-f<span class="w"> </span>rbd<span class="w"> </span><span class="s2">"rbd:rbd/myrbd"</span><span class="w"> </span>600G
</code></pre></div>
<!-- more -->
<p>List device define for myVM :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>virsh<span class="w"> </span>domblklist<span class="w"> </span>myVM
</code></pre></div>
<p>Resize libvirt blockdevice :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>virsh<span class="w"> </span>blockresize<span class="w"> </span>--domain<span class="w"> </span>myVM<span class="w"> </span>--path<span class="w"> </span>vdb<span class="w"> </span>--size<span class="w"> </span>600G
$<span class="w"> </span>rbd<span class="w"> </span>info<span class="w"> </span>rbd/myrb
</code></pre></div>
<p>Extend xfs on guest :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>xfs_growfs<span class="w"> </span>/mnt/rbd/myrbd
</code></pre></div>
<h2>Extend rbd with kernel module</h2>
<p>You need at least kernel 3.10 on ceph client to support resizing.
For previous version look at http://dachary.org/?p=2179</p>
<p>Get current size :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>info<span class="w"> </span>rbd/myrbd
</code></pre></div>
<p>Just do :</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rbd<span class="w"> </span>resize<span class="w"> </span>rbd/myrbd<span class="w"> </span>--size<span class="w"> </span><span class="m">600000</span>
$<span class="w"> </span>xfs_growfs<span class="w"> </span>/mnt/rbd/myrbd
</code></pre></div>
<p>Also, since cuttlefish you can't shrink a bloc device without specify additional option (–allow-shrink)</p>