{"id":5211,"date":"2020-03-14T23:47:13","date_gmt":"2020-03-14T20:47:13","guid":{"rendered":"https:\/\/kifarunix.com\/?p=5211"},"modified":"2023-12-08T19:06:20","modified_gmt":"2023-12-08T16:06:20","slug":"configure-and-use-ceph-block-device-on-linux-clients","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/configure-and-use-ceph-block-device-on-linux-clients\/","title":{"rendered":"Configure and Use Ceph Block Device on Linux Clients"},"content":{"rendered":"\n<p>In this guide, you will learn how to configure and use Ceph block device on Linux clients. Ceph provides various interfaces through which clients can access storage. Such interfaces include the Ceph Object Storage (for object storage), Ceph File System (for distributed posix-compliant filesystem) and the RADOS Block Device (RBD) for (block-based storage).<\/p>\n\n\n\n<p><em>Ceph RBD (RADOS Block Device) block storage stripes virtual disks over objects within a Ceph storage cluster, distributing data and workload across all available devices for extreme scalability and performance. RBD disk images are thinly provisioned, support both read-only snapshots and writable clones, and can be asynchronously mirrored to remote Ceph clusters in other data centers for disaster recovery or backup, making Ceph RBD the leading choice for block storage in public\/private cloud and virtualization environments<\/em>.<\/p>\n\n\n\n<p>RBD integrates well to&nbsp;KVMs&nbsp;such as&nbsp;<a href=\"https:\/\/docs.ceph.com\/en\/latest\/rbd\/qemu-rbd\/\" target=\"_blank\" rel=\"noopener\">QEMU<\/a>, and cloud-based computing systems like&nbsp;<a href=\"https:\/\/docs.ceph.com\/en\/latest\/rbd\/rbd-openstack\" target=\"_blank\" rel=\"noopener\">OpenStack<\/a>&nbsp;and&nbsp;<a href=\"https:\/\/docs.ceph.com\/en\/latest\/rbd\/rbd-cloudstack\" target=\"_blank\" rel=\"noopener\">CloudStack<\/a>&nbsp;that rely on libvirt and QEMU to integrate with Ceph block devices.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#configure-and-use-ceph-block-device-on-linux-clients\">Configure and Use Ceph Block Device on Linux Clients<\/a><ul><li><a href=\"#deploy-ceph-storage-cluster\">Deploy Ceph Storage Cluster<\/a><\/li><li><a href=\"#setup-linux-client-for-ceph-block-device-storage-use\">Setup Linux Client for Ceph Block Device Storage Use<\/a><ul><li><a href=\"#copy-ssh-keys-to-linux-client\">Copy SSH Keys to Linux Client<\/a><\/li><li><a href=\"#install-ceph-packages-on-the-client\">Install Ceph Packages on the Client<\/a><\/li><li><a href=\"#configure-ceph-on-linux-client\">Configure Ceph on Linux Client<\/a><\/li><\/ul><\/li><li><a href=\"#create-block-device-pools\">Create Block Device Pools<\/a><ul><li><a href=\"#ensure-cluster-is-in-good-health\">Ensure Cluster is in Good Health<\/a><\/li><li><a href=\"#create-ceph-block-device-pool-on-admin-node\">Create Ceph Block Device Pool on Admin Node<\/a><\/li><li><a href=\"#initialize-block-device-osd-pool-on-admin-node\">Initialize Block Device OSD Pool on Admin Node<\/a><\/li><li><a href=\"#enable-rados-block-device-application-for-osd-pool\">Enable RADOS Block Device Application for OSD Pool<\/a><\/li><\/ul><\/li><li><a href=\"#creating-block-device-images-on-client\">Creating Block Device Images on Client<\/a><\/li><li><a href=\"#mapping-images-to-block-devices\">Mapping Images to Block Devices<\/a><\/li><li><a href=\"#create-file-system-on-ceph-block-device\">Create FileSystem on Ceph Block Device<\/a><\/li><li><a href=\"#mounting-ceph-block-device-on-linux-client\">Mounting Ceph Block Device on Linux Client<\/a><\/li><li><a href=\"#check-the-image-status-on-ceph-dashboard\">Check the Image Status on Ceph Dashboard<\/a><\/li><li><a href=\"#other-tutorials\">Other Tutorials<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"configure-and-use-ceph-block-device-on-linux-clients\">Configure and Use Ceph Block Device on Linux Clients<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploy-ceph-storage-cluster\">Deploy Ceph Storage Cluster<\/h3>\n\n\n\n<p>Before you can proceed, ensure that you have a running Ceph storage cluster.<\/p>\n\n\n\n<p>In our previous guide, we learnt how to deploy a three node Ceph Storage Cluster. Check the links below;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-setup-ceph-storage-cluster-on-ubuntu-2204\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install and Setup Ceph Storage Cluster on Ubuntu<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"setup-linux-client-for-ceph-block-device-storage-use\">Setup Linux Client for Ceph Block Device Storage Use<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"copy-ssh-keys-to-linux-client\">Copy SSH Keys to Linux Client<\/h4>\n\n\n\n<p>Copy the Ceph SSH key generated by the bootstrap command to Ceph block device Linux client. Ensure Root Login is permitted on the client.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ssh-copy-id -f -i \/etc\/ceph\/ceph.pub root@linux-client-ip-or-hostname<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-ceph-packages-on-the-client\">Install Ceph Packages on the Client<\/h4>\n\n\n\n<p>Login to the client and install Ceph command line tools.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ssh root@@linux-client-ip-or-hostname<\/code><\/pre>\n\n\n\n<p>Install the Ceph client packages;<\/p>\n\n\n\n<p>Ubuntu\/Debian;<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>sudo apt install apt-transport-https \\\n\tca-certificates \\\n\tcurl \\\n\tgnupg-agent \\\n\tsoftware-properties-common -y\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>wget -q -O- 'https:\/\/download.ceph.com\/keys\/release.asc' | \\\ngpg --dearmor -o \/etc\/apt\/trusted.gpg.d\/cephadm.gpg<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>echo deb https:\/\/download.ceph.com\/debian-reef\/ $(lsb_release -sc) main \\\n&gt; \/etc\/apt\/sources.list.d\/cephadm.list<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>apt update<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>apt install ceph-common<\/code><\/pre>\n\n\n\n<p>CentOS\/RHEL based distros;<\/p>\n\n\n\n<p>Get the <a href=\"https:\/\/docs.ceph.com\/en\/latest\/releases\/\" target=\"_blank\" rel=\"noreferrer noopener\">release version<\/a> of the Ceph you are running. For example, I am running Ceph Reef, 18.2.0.<\/p>\n\n\n\n<p>Install the EPEL repository;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dnf install epel-release<\/code><\/pre>\n\n\n\n<p>Install CEPH repository. Replace {ceph-release} with the CEPH release version, e.g 18.2.0. Similarly, replace {distro} with your OS release number.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>cat > \/etc\/yum.repos.d\/ceph.repo &lt;&lt; EOL\n[ceph]\nname=Ceph packages for $basearch\nbaseurl=https:\/\/download.ceph.com\/rpm-{ceph-release}\/{distro}\/$basearch\nenabled=1\npriority=2\ngpgcheck=1\ngpgkey=https:\/\/download.ceph.com\/keys\/release.asc\nEOL\n<\/code><\/pre>\n\n\n\n<p>For example, you can get the release number using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"el$(grep VERSION_ID \/etc\/os-release | cut -d= -f2 | awk -F. '{print $1}' | tr -d '\"')\"<\/code><\/pre>\n\n\n\n<p>You RHEL Ceph repos should look like below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat \/etc\/yum.repos.d\/ceph.repo<\/code><\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code>[ceph]\nname=Ceph packages for $basearch\nbaseurl=https:\/\/download.ceph.com\/rpm-18.2.0\/el9\/$basearch\nenabled=1\npriority=2\ngpgcheck=1\ngpgkey=https:\/\/download.ceph.com\/keys\/release.asc\n<\/code><\/pre>\n\n\n\n<p>Next, install the Ceph packages;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dnf install ceph-common -y<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-ceph-on-linux-client\">Configure Ceph on Linux Client<\/h4>\n\n\n\n<p>Simply copy the Ceph configuration files from the Ceph monitor or ceph admin node.<\/p>\n\n\n\n<p>Similarly, you need to copy the Ceph client keyring. This is necessary to authenticate the client to the Ceph cluster.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cephadmin@ceph-admin:~$ <strong>sudo scp \/etc\/ceph\/{ceph.conf,ceph.client.admin.keyring} root@linux-client-ip-or-hostname:\/etc\/ceph\/<\/strong><\/code><\/pre>\n\n\n\n<p>These are the contents of the Ceph configuration file.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat \/etc\/ceph\/ceph.conf<\/code><\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code># minimal ceph.conf for 70d227de-83e3-11ee-9dda-ff8b7941e415\n[global]\n\tfsid = 70d227de-83e3-11ee-9dda-ff8b7941e415\n\tmon_host = [v2:192.168.122.240:3300\/0,v1:192.168.122.240:6789\/0] [v2:192.168.122.45:3300\/0,v1:192.168.122.45:6789\/0] [v2:192.168.122.231:3300\/0,v1:192.168.122.231:6789\/0] [v2:192.168.122.49:3300\/0,v1:192.168.122.49:6789\/0]\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>fsid<\/code>: Specifies the Universally Unique Identifier (UUID) for the Ceph cluster. In this case, it&#8217;s <code>70d227de-83e3-11ee-9dda-ff8b7941e415<\/code>.<\/li>\n\n\n\n<li><code>mon_host<\/code>: Lists the monitor (MON) nodes and their addresses in the cluster. Each entry consists of monitor IP addresses in both <code>v2<\/code> and <code>v1<\/code> formats, along with port numbers.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"create-block-device-pools\">Create Block Device Pools<\/h3>\n\n\n\n<p>In order to use the Ceph block device on your clients, you need to create a pool for the RADOS Block Device (RBD) and initialize it.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A pool is a logical group for storing objects. They manage placement groups, replicas and the CRUSH rule for the pool.<\/li>\n\n\n\n<li>A placement group is a fragment of logical object pool that places objects as a group into OSDs. Ceph client calculates which placement group an object should be stored.<\/li>\n<\/ul>\n\n\n\n<p>There are two types of Ceph OSD pools;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>replicated pool or<\/li>\n\n\n\n<li>erasure-coded pool<\/li>\n<\/ul>\n\n\n\n<p>In a replicated type of a pool, which is usually the default type of a pool when you create one, data is copied from the primary OSD across other multiple OSDs in the cluster. By default, Ceph creates two replicas of an object (a total of three copies, or a pool size of 3). Thus, replicated pool type requires more raw storage.<\/p>\n\n\n\n<p>On the other hand, in erasure-coded pool type, Ceph uses erasure code algorithms to break object data into two types of chunks that are written across different OSDs;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>data chunks, also know as, data blocks. The data chunks are denoted by <strong>k<\/strong>.<\/li>\n\n\n\n<li>parity chunks, also known as parity blocks. They are denoted by <strong>m.<\/strong> If a drive fails or becomes corrupted, the parity blocks are used to rebuild the data and as such, they specify how many OSDs that be allowed to fail without causing data loss.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"ensure-cluster-is-in-good-health\">Ensure Cluster is in Good Health<\/h4>\n\n\n\n<p>Before you can proceed further, ensure the cluster is in good health,&nbsp;<code>active&nbsp;+&nbsp;clean<\/code>&nbsp;state.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph -s<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>  cluster:\n    id:     70d227de-83e3-11ee-9dda-ff8b7941e415\n    <strong>health: HEALTH_OK<\/strong>\n \n  services:\n    mon: 4 daemons, quorum ceph-admin,ceph-mon,ceph-osd1,ceph-osd2 (age 21h)\n    mgr: ceph-admin.ykkdly(active, since 22h), standbys: ceph-mon.grwzmv\n    osd: 3 osds: 3 up (since 21h), 3 in (since 21h)\n \n  data:\n    pools:   1 pools, 1 pgs\n    objects: 2 objects, 449 KiB\n    usage:   80 MiB used, 300 GiB \/ 300 GiB avail\n    <strong>pgs:     1 active+clean<\/strong>\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-ceph-block-device-pool-on-admin-node\">Create Ceph Block Device Pool on Admin Node<\/h4>\n\n\n\n<p>Ceph creates a default erasure code profile when initializing a cluster and it provides the same level of redundancy as two copies in a replicated pool. However, it uses 25% less storage capacity. The default profiles define&nbsp;<strong>k=2<\/strong>&nbsp;and&nbsp;<strong>m=2<\/strong>, meaning Ceph will spread the object data over four OSDs (<strong>k+m=4<\/strong>) and Ceph can lose one of those OSDs without losing data.<\/p>\n\n\n\n<p>The erasure-coded pools reduce the amount of disk space required to ensure data durability but it is computationally a bit more expensive than replication.<\/p>\n\n\n\n<p>To create Ceph pool use the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph osd pool create {pool-name} pg_num pgp_num<\/code><\/pre>\n\n\n\n<p>Where:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code><strong>{pool-name}<\/strong><\/code>&nbsp;is the name of the Ceph pool you are creating. The name&nbsp;<strong>rbd<\/strong>&nbsp;is recommended. We will use the name, <strong><code>glance-images<\/code><\/strong> in this guide.<\/li>\n\n\n\n<li><code><strong>pg_num<\/strong><\/code>&nbsp;is the total number of placement groups for the pool. It determines the number of actual Placement Groups (PGs) that data objects will be divided into. See how to determine the&nbsp;<a href=\"https:\/\/docs.ceph.com\/en\/latest\/rados\/operations\/placement-groups\/#choosing-number-of-placement-groups\" target=\"_blank\" rel=\"noreferrer noopener\">Number of PGs<\/a>.<\/li>\n\n\n\n<li><strong><code>pgp_num<\/code><\/strong>&nbsp;specifies total number of placement groups for placement purposes.&nbsp;<strong>Should be equal to the total number of placement groups<\/strong>.<\/li>\n<\/ul>\n\n\n\n<p>So, let\u2019s see how can calculate the number of PGs. You can use the formula;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Total PGs  = (No. of OSDs x 100)\/pool size<\/code><\/pre>\n\n\n\n<p>Where <strong>pool size<\/strong> refers to:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>number of replicas for replicated pools or<\/li>\n\n\n\n<li>the K+M sum for erasure-coded pools.<\/li>\n<\/ul>\n\n\n\n<p>The value of PGs should be&nbsp;<strong>rounded UP to the nearest power of two<\/strong>&nbsp;(2^x). Rounding up is optional, but it is recommended for CRUSH to evenly balance the number of objects among placement groups.<\/p>\n\n\n\n<p>By default, Ceph creates two replicas of an object (a total of three copies, i.e a pool size of&nbsp;<code>3<\/code>).<\/p>\n\n\n\n<p>To calculate the pool size using the default replica size;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>PGs= (6x100)\/3=200<\/strong><\/code><\/pre>\n\n\n\n<p>This rounded up to power of 2, gives 256 PGs.<\/p>\n\n\n\n<p>You can always get the replica size using the command, <strong><code>ceph osd dump | grep 'replicated size'<\/code><\/strong>.<\/p>\n\n\n\n<p>To calculate total number of PGs using erasure coding chunk size, you need to get the default values of K+M for erasure-coded pools as follows.<\/p>\n\n\n\n<p>Get your erasure coded pool profile;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph osd erasure-code-profile ls<\/code><\/pre>\n\n\n\n<p>Output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>default<\/code><\/pre>\n\n\n\n<p>Next, get the erasure-coded profile information using the command&nbsp;<strong><code>ceph osd erasure-code-profile get &lt;profile&gt;<\/code><\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph osd erasure-code-profile get default<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code><strong>k=2\nm=2<\/strong>\nplugin=jerasure\ntechnique=reed_sol_van\n<\/code><\/pre>\n\n\n\n<p>Now, the sum of K and M is 2+2=4.<\/p>\n\n\n\n<p>Hence, <strong>PGs=3&#215;100\/4 = 300\/4 = 75<\/strong>. The nearest power of 2 here is 2^7 = <strong>128<\/strong>. Hence, our PGs should be 128.<\/p>\n\n\n\n<p>Hence, let&#8217;s create the block device pools;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph osd pool create rbd 128 128<\/code><\/pre>\n\n\n\n<p>You can list OSD pools using the command;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph osd pool ls<\/code><\/pre>\n\n\n\n<p>Or;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph osd lspools<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"initialize-block-device-osd-pool-on-admin-node\">Initialize Block Device OSD Pool on Admin Node<\/h4>\n\n\n\n<p>Once you have created the OSD pool, you can initialize it using the command, <code><strong>rbd pool init &lt;pool-name&gt;<\/strong><\/code>;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo rbd pool init rbd<\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"enable-rados-block-device-application-for-osd-pool\">Enable RADOS Block Device Application for OSD Pool<\/h4>\n\n\n\n<p>Associate the pool created with the respective application to prevent unauthorized types of clients from writing data to the pool. An application can be;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>cephfs<\/code>&nbsp;(Ceph Filesystem).<\/li>\n\n\n\n<li><code>rbd<\/code>&nbsp;(Ceph Block Device).<\/li>\n\n\n\n<li><code>rgw<\/code>&nbsp;(Ceph Object Gateway).<\/li>\n<\/ul>\n\n\n\n<p>To associate the pool created above with RBD, simply execute the command, <strong><code>ceph osd pool application enable &lt;pool> &lt;app> [--yes-i-really-mean-it]<\/code><\/strong>. Replace the name of the pool accordingly.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo ceph osd pool application enable rbd rbd<\/pre>\n\n\n\n<p>Command output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>enabled application 'rbd' on pool 'rbd'<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"creating-block-device-images-on-client\">Creating Block Device Images on Client<\/h3>\n\n\n\n<p>Login to the client to execute the following commands.<\/p>\n\n\n\n<p>You need to create a block device image (<strong>virtual block device<\/strong>) in the Ceph storage cluster before adding it to a node using the command below in a Ceph Client.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">rbd create &lt;image-name&gt; --size &lt;megabytes&gt; --pool &lt;pool-name&gt;<\/pre>\n\n\n\n<p>(For more options, do <strong><code>rbd help create<\/code><\/strong>).<\/p>\n\n\n\n<p>For example, to create a block device image of 10GB in the pool created above, <strong>rbd<\/strong>, simply run the command;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo rbd create disk01 --size 10G --pool rbd<\/pre>\n\n\n\n<p>To list the images in your pool;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo rbd ls -l rbd<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>NAME     SIZE PARENT FMT PROT LOCK \ndisk01   10 GiB        2         <\/code><\/pre>\n\n\n\n<p>To retrieve information about the image created, run the command;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo rbd --image disk01 -p rbd info<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>rbd image 'disk01':\n\tsize 10 GiB in 2560 objects\n\torder 22 (4 MiB objects)\n\tsnapshot_count: 0\n\tid: 5f31272f93d8\n\tblock_name_prefix: rbd_data.5f31272f93d8\n\tformat: 2\n\tfeatures: layering, exclusive-lock, object-map, fast-diff, deep-flatten\n\top_features: \n\tflags: \n\tcreate_timestamp: Thu Nov 16 18:17:08 2023\n\taccess_timestamp: Thu Nov 16 18:17:08 2023\n\tmodify_timestamp: Thu Nov 16 18:17:08 2023\n<\/code><\/pre>\n\n\n\n<p>To remove an image from the pool;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo rbd rm disk01 -p rbd<\/pre>\n\n\n\n<p>To move it to trash for later deletion;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo rbd trash move rbd\/disk01<\/pre>\n\n\n\n<p>To restore image from trash to the pool, obtain the image ID as assigned on the trash store then restore the image using the ID;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo rbd trash list rbd<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">38c8adcf4ca disk01<\/pre>\n\n\n\n<p>Where <strong>rbd<\/strong> is the name of the pool.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo rbd trash restore rbd\/38c8adcf4ca<\/pre>\n\n\n\n<p>To empty image from trash for permanent deletion;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">rbd trash remove rbd\/<strong>38c8adcf4ca<\/strong><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"mapping-images-to-block-devices\">Mapping Images to Block Devices<\/h3>\n\n\n\n<p>After creating an image, you can map it to block devices.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo rbd map disk01 --pool rbd<\/pre>\n\n\n\n<p>You will see an output like;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">\/dev\/rbd0<\/pre>\n\n\n\n<p>To show block device images mapped to kernel modules with the&nbsp;<code>rbd<\/code>&nbsp;command;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo rbd showmapped<\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code>id  pool  namespace  image   snap  device   \n0   rbd              disk02  -     \/dev\/rbd0\n\n<\/code><\/pre>\n\n\n\n<p>To unmap a block device image, use the command, <strong>r<code>bd unmap \/dev\/rbd\/{poolname}\/{imagename}<\/code><\/strong> for example;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>sudo rbd unmap \/dev\/rbd\/rbd\/disk01<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"create-file-system-on-ceph-block-device\">Create FileSystem on Ceph Block Device<\/h3>\n\n\n\n<p>The Ceph mapped block device is now ready. All is left is to create a file system on it and mount it to make it use-able.<\/p>\n\n\n\n<p>For example, to create an XFS file system on it (you can use your preferred filesystem type);<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo mkfs.xfs \/dev\/rbd0 -L cephdisk01<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>meta-data=\/dev\/rbd0              isize=512    agcount=16, agsize=163840 blks\n         =                       sectsz=512   attr=2, projid32bit=1\n         =                       crc=1        finobt=1, sparse=1, rmapbt=0\n         =                       reflink=1    bigtime=0 inobtcount=0\ndata     =                       bsize=4096   blocks=2621440, imaxpct=25\n         =                       sunit=16     swidth=16 blks\nnaming   =version 2              bsize=4096   ascii-ci=0, ftype=1\nlog      =internal log           bsize=4096   blocks=2560, version=2\n         =                       sectsz=512   sunit=16 blks, lazy-count=1\nrealtime =none                   extsz=4096   blocks=0, rtextents=0\nDiscarding blocks...Done.\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"mounting-ceph-block-device-on-linux-client\">Mounting Ceph Block Device on Linux Client<\/h3>\n\n\n\n<p>You can now mount the block device. For example, to mount it under <code><strong>\/media\/ceph<\/strong><\/code> directory;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo mkdir \/media\/ceph<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo mount \/dev\/rbd0 \/media\/ceph<\/pre>\n\n\n\n<p>You can as well mount it as follows;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo mount \/dev\/rbd\/rbd\/disk01 \/media\/ceph\/<\/pre>\n\n\n\n<p>Check mounted Filesystems;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">df -hT -P \/dev\/rbd0<\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code>Filesystem     Type  Size  Used Avail Use% Mounted on\n\/dev\/rbd0      xfs    10G  105M  9.9G   2% \/media\/ceph\n<\/code><\/pre>\n\n\n\n<p>There you go.<\/p>\n\n\n\n<p>If you check the Ceph cluster health;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">ceph --status<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>  cluster:\n    id:     70d227de-83e3-11ee-9dda-ff8b7941e415\n    health: HEALTH_OK\n \n  services:\n    mon: 4 daemons, quorum ceph-admin,ceph-mon,ceph-osd1,ceph-osd2 (age 23h)\n    mgr: ceph-admin.ykkdly(active, since 24h), standbys: ceph-mon.grwzmv\n    osd: 3 osds: 3 up (since 23h), 3 in (since 23h)\n \n  data:\n    pools:   2 pools, 33 pgs\n    objects: 45 objects, 14 MiB\n    usage:   236 MiB used, 300 GiB \/ 300 GiB avail\n    pgs:     33 active+clean\n \n  progress:\n    Global Recovery Event (15s)\n      [===========================.]\n<\/code><\/pre>\n\n\n\n<p>If the Global Recovery Event remains persistent despite the cluster Health OK, then you can try to clear the progress;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph progress clear<\/code><\/pre>\n\n\n\n<p>The check status again;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph -s<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>  cluster:\n    id:     70d227de-83e3-11ee-9dda-ff8b7941e415\n    health: HEALTH_OK\n \n  services:\n    mon: 4 daemons, quorum ceph-admin,ceph-mon,ceph-osd1,ceph-osd2 (age 23h)\n    mgr: ceph-admin.ykkdly(active, since 24h), standbys: ceph-mon.grwzmv\n    osd: 3 osds: 3 up (since 23h), 3 in (since 23h)\n \n  data:\n    pools:   2 pools, 33 pgs\n    objects: 45 objects, 14 MiB\n    usage:   236 MiB used, 300 GiB \/ 300 GiB avail\n    pgs:     33 active+clean\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"check-the-image-status-on-ceph-dashboard\">Check the Image Status on Ceph Dashboard<\/h3>\n\n\n\n<p>Login to Ceph dashboard and check the status of the virtual block devices.<\/p>\n\n\n\n<p>On the dashboard, navigate to <strong>Block &gt; Images<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1882\" height=\"485\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-block-images.png?v=1700165679\" alt=\"\" class=\"wp-image-19288\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-block-images.png?v=1700165679 1882w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-block-images-768x198.png?v=1700165679 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-block-images-1536x396.png?v=1700165679 1536w\" sizes=\"(max-width: 1882px) 100vw, 1882px\" \/><\/figure>\n\n\n\n<p>Click on the drop down button against each disk to expand and view more details.<\/p>\n\n\n\n<p>You can also access the Grafana dashboards (<strong>https:\/\/ceph-admin:3000<\/strong>), Replace the address accordingly.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1385\" height=\"925\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-osd-pool-overview.png?v=1700168629\" alt=\"\" class=\"wp-image-19291\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-osd-pool-overview.png?v=1700168629 1385w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-osd-pool-overview-768x513.png?v=1700168629 768w\" sizes=\"(max-width: 1385px) 100vw, 1385px\" \/><\/figure>\n\n\n\n<p>That marks the end of our guide on how to configure and use Ceph Block Device on Linux.<\/p>\n\n\n\n<p>Reference<\/p>\n\n\n\n<p><a href=\"https:\/\/docs.ceph.com\/en\/latest\/start\/quick-rbd\/\" target=\"_blank\" rel=\"noreferrer noopener\">Block Device Quick Start<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"other-tutorials\">Other Tutorials<\/h3>\n\n\n\n<p><a rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/install-and-setup-glusterfs-on-ubuntu-18-04\/\" target=\"_blank\">Install and Setup GlusterFS on Ubuntu 18.04<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/how-to-install-and-configure-nfs-server-on-rhel-centos-7\/\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">How to Install and Configure NFS Server on RHEL\/CentOS 7<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this guide, you will learn how to configure and use Ceph block device on Linux clients. Ceph provides various interfaces through which clients can<\/p>\n","protected":false},"author":1,"featured_media":11667,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[39,1338,121],"tags":[1345,1350,1347,1348,1340,1346],"class_list":["post-5211","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-storage","category-ceph","category-howtos","tag-ceph-block-device","tag-ceph-block-device-client","tag-ceph-block-device-ubuntu-18-04","tag-ceph-block-storage","tag-ceph-ubuntu-18-04","tag-rados-block-device-ceph","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/5211"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=5211"}],"version-history":[{"count":17,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/5211\/revisions"}],"predecessor-version":[{"id":19491,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/5211\/revisions\/19491"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/11667"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=5211"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=5211"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=5211"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}