{"id":19638,"date":"2023-12-16T16:51:47","date_gmt":"2023-12-16T13:51:47","guid":{"rendered":"https:\/\/kifarunix.com\/?p=19638"},"modified":"2024-03-10T15:22:48","modified_gmt":"2024-03-10T12:22:48","slug":"provision-block-storage-for-kubernetes-on-rook-ceph-cluster","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/provision-block-storage-for-kubernetes-on-rook-ceph-cluster\/","title":{"rendered":"Provision Block Storage for Kubernetes on Rook Ceph Cluster"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1042\" height=\"589\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/block-storage-for-kubernetes-on-rook-ceph-cluster.png\" alt=\"provision Block Storage for Kubernetes on Rook Ceph Cluster\" class=\"wp-image-19664\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/block-storage-for-kubernetes-on-rook-ceph-cluster.png?v=1702734653 1042w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/block-storage-for-kubernetes-on-rook-ceph-cluster-768x434.png?v=1702734653 768w\" sizes=\"(max-width: 1042px) 100vw, 1042px\" \/><\/figure>\n\n\n\n<p>In this tutorial, you will learn how to provision block storage for Kubernetes on Rook ceph cluster. <a href=\"https:\/\/rook.io\/\" target=\"_blank\" rel=\"noreferrer noopener\">Rook<\/a> is an open-source platform for managing and operating a Ceph storage cluster within Kubernetes. Currently, Kubernetes is the de facto standard for automating deployment, scaling, and management of containerized applications. As organizations embrace Kubernetes, the demand for resilient and scalable storage solutions to support stateful workloads becomes paramount. This is where Ceph block storage comes in.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#configure-block-storage-for-kubernetes\">Configure Block Storage for Kubernetes<\/a><ul><li><a href=\"#deply-ceph-storage-cluster-in-kubernetes-using-rook\">Deply Ceph Storage Cluster in Kubernetes using Rook<\/a><\/li><li><a href=\"#configuring-block-storage-for-kubernetes-on-rook-ceph-cluster\">Configuring Block Storage for Kubernetes on Rook Ceph Cluster<\/a><\/li><li><a href=\"#configure-kubernetes-applications-to-use-ceph-block-storage\">Configure Kubernetes Applications to use Ceph Block Storage<\/a><\/li><li><a href=\"#check-block-devices-on-ceph-rbd-pool\">Check Block Devices on Ceph RBD Pool<\/a><\/li><li><a href=\"#verify-block-device-image-usage-on-the-pod\">Verify Block Device Image Usage on the Pod<\/a><\/li><li><a href=\"#manually-create-block-device-image-and-mount-in-a-pod\">Manually Create Block Device Image and Mount in a Pod<\/a><ul><li><a href=\"#create-direct-mount-pod\">Create Direct Mount Pod<\/a><\/li><li><a href=\"#create-block-device-image-of-specific-size\">Create Block Device Image of specific Size<\/a><\/li><li><a href=\"#get-block-device-image-information\">Get Block Device Image Information<\/a><\/li><li><a href=\"#map-block-device-image-to-local-device\">Map Block Device Image to Local Device<\/a><\/li><li><a href=\"#create-filesystem-on-mapped-block-device-image\">Create Filesystem on Mapped Block Device Image<\/a><\/li><li><a href=\"#mount-unmount-block-device-image\">Mount\/Unmount Block Device Image<\/a><\/li><\/ul><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"configure-block-storage-for-kubernetes\">Configure Block Storage for Kubernetes<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deply-ceph-storage-cluster-in-kubernetes-using-rook\">Deply Ceph Storage Cluster in Kubernetes using Rook<\/h3>\n\n\n\n<p>To begin with, ensure you have a Ceph storage cluster running in Kubernetes.<\/p>\n\n\n\n<p>You can check our previous guide on how to deploy Ceph cluster in Kubernetes using Rook by following the link below.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/deploy-ceph-storage-cluster-in-kubernetes-using-rook\" target=\"_blank\" rel=\"noreferrer noopener\">Deploy Ceph Storage Cluster in Kubernetes using Rook<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configuring-block-storage-for-kubernetes-on-rook-ceph-cluster\">Configuring Block Storage for Kubernetes on Rook Ceph Cluster<\/h3>\n\n\n\n<p>Once the cluster is up, you can proceed to provision block storage for use by Kubernetes applications.<\/p>\n\n\n\n<p>Rook provides a manifest called <strong><code>storageclass.yaml<\/code><\/strong> that resides in the RBD directory under <strong><code>~\/rook\/deploy\/examples\/csi\/<\/code><\/strong> directory. This manifest defines various configuration options that can be used to provision persistent block storage volumes for use by Kubernetes application pods.<\/p>\n\n\n\n<p>By default, without command lines, this is how it is defined;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat ~\/rook\/deploy\/examples\/csi\/rbd\/storageclass.yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>apiVersion: ceph.rook.io\/v1\nkind: CephBlockPool\nmetadata:\n  name: replicapool\n  namespace: rook-ceph # namespace:cluster\nspec:\n  failureDomain: host\n  replicated:\n    size: 3\n    requireSafeReplicaSize: true\n---\napiVersion: storage.k8s.io\/v1\nkind: StorageClass\nmetadata:\n  name: rook-ceph-block\nprovisioner: rook-ceph.rbd.csi.ceph.com\nparameters: \n  clusterID: rook-ceph # namespace:cluster\n  pool: replicapool\n  imageFormat: \"2\"\n  imageFeatures: layering\n  csi.storage.k8s.io\/provisioner-secret-name: rook-csi-rbd-provisioner\n  csi.storage.k8s.io\/provisioner-secret-namespace: rook-ceph # namespace:cluster\n  csi.storage.k8s.io\/controller-expand-secret-name: rook-csi-rbd-provisioner\n  csi.storage.k8s.io\/controller-expand-secret-namespace: rook-ceph # namespace:cluster\n  csi.storage.k8s.io\/node-stage-secret-name: rook-csi-rbd-node\n  csi.storage.k8s.io\/node-stage-secret-namespace: rook-ceph # namespace:cluster\n  csi.storage.k8s.io\/fstype: ext4\nallowVolumeExpansion: true\nreclaimPolicy: Delete\n<\/code><\/pre>\n\n\n\n<p>The YAML file defines two resources: a <strong><code>CephBlockPool<\/code><\/strong> and a <strong><code>StorageClass<\/code><\/strong>, both related to block storage in a Rook Ceph cluster. Here&#8217;s a breakdown of each resource:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>CephBlockPool:<\/strong>\n<ul class=\"wp-block-list\">\n<li><code>apiVersion: ceph.rook.io\/v1<\/code>: Specifies the API version for the Rook CephBlockPool resource.<\/li>\n\n\n\n<li><code>kind: CephBlockPool<\/code>: Declares the type of resource as a CephBlockPool.<\/li>\n\n\n\n<li><code>metadata<\/code>: Contains information about the metadata of the CephBlockPool, including its name.<\/li>\n\n\n\n<li><code>spec<\/code>: Defines the specifications for the CephBlockPool.\n<ul class=\"wp-block-list\">\n<li><code>failureDomain: host<\/code>: Specifies the failure domain as &#8220;host,&#8221; indicating that the pool should be distributed across hosts.<\/li>\n\n\n\n<li><code>replicated<\/code>: Indicates that the pool is replicated.\n<ul class=\"wp-block-list\">\n<li><code>size: 3<\/code>: Sets the replication size to 3, meaning data will be stored on three different nodes.<\/li>\n\n\n\n<li><code>requireSafeReplicaSize: true<\/code>: Ensures that the pool won&#8217;t be created unless the desired replica size can be satisfied.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>StorageClass:<\/strong>\n<ul class=\"wp-block-list\">\n<li><code>apiVersion: storage.k8s.io\/v1<\/code>: Specifies the API version for the Kubernetes StorageClass resource.<\/li>\n\n\n\n<li><code>kind: StorageClass<\/code>: Declares the type of resource as a StorageClass.<\/li>\n\n\n\n<li><code>metadata<\/code>: Contains information about the metadata of the StorageClass, including its name.<\/li>\n\n\n\n<li><code>provisioner: rook-ceph.rbd.csi.ceph.com<\/code>: Specifies the Rook Ceph CSI provisioner, where <code>rook-ceph<\/code> is the namespace.<\/li>\n\n\n\n<li><code>parameters<\/code>: Defines parameters for the StorageClass.\n<ul class=\"wp-block-list\">\n<li><code>pool: replicapool<\/code>: Specifies the CephBlockPool to use.<\/li>\n\n\n\n<li><code>imageFormat: \"2\"<\/code>: Sets the RBD image format to version 2.<\/li>\n\n\n\n<li><code>imageFeatures: layering<\/code>: Enables RBD image layering.<\/li>\n\n\n\n<li><code class=\"\">csi.storage.k8s.io\/*secret*<\/code>:&nbsp;These parameters define the location and names of various secrets required by the provisioner and CSI driver,&nbsp;ensuring secure access to cluster credentials and resources.<\/li>\n\n\n\n<li><code>csi.storage.k8s.io\/fstype: ext4<\/code>: Specifies the filesystem type as ext4.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><code>allowVolumeExpansion: true<\/code>: Enables online volume expansion for applications.<\/li>\n\n\n\n<li><code>reclaimPolicy: Delete<\/code>: Specifies the reclaim policy as &#8220;Delete,&#8221; meaning that when a PersistentVolume (PV) using this StorageClass is released, the associated volume and data will be deleted. If you want to the images to exist even after the&nbsp;<code>PersistentVolume<\/code>&nbsp;has been deleted, you can use <strong>Retain<\/strong> policy. Note that you will need to remove them images manually using <strong><code>rbd rm<\/code><\/strong> Ceph command.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<p>Note that failure domain is set to <strong><code>host<\/code><\/strong> and a <strong><code>replica size<\/code><\/strong> of 3. This therefore requires 3 hosts\/nodes with at least 1 OSD per node.<\/p>\n\n\n\n<p>In this demo guide, we will provision the block storage using the default manifest provided. Take note of the namespace. If you are using a namespace other than the default, be sure to make appropriate adjustments.<\/p>\n\n\n\n<p>Verify the cluster status before you can proceed.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec deploy\/rook-ceph-tools -- ceph status<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>  cluster:\n    id:     e1466372-9f01-42af-8ad7-0bfcfa71ef78\n    health: HEALTH_OK\n \n  services:\n    mon: 3 daemons, quorum a,b,c (age 38h)\n    mgr: a(active, since 16m), standbys: b\n    osd: 3 osds: 3 up (since 38h), 3 in (since 38h)\n \n  data:\n    pools:   1 pools, 1 pgs\n    objects: 2 objects, 449 KiB\n    usage:   81 MiB used, 300 GiB \/ 300 GiB avail\n    pgs:     1 active+clean\n<\/code><\/pre>\n\n\n\n<p>Next, proceed to configure block storage for Kubernetes. Start by creating the storage class.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cd ~\/rook\/deploy\/examples<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl create -f csi\/rbd\/storageclass.yaml<\/code><\/pre>\n\n\n\n<p>When executed, the command will now create a Ceph pool called replicapool;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it rook-ceph-tools-564c8446db-xh6qp -- ceph osd pool ls<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>.mgr\nreplicapool\n<\/code><\/pre>\n\n\n\n<p>To get the details of a pool;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it rook-ceph-tools-564c8446db-xh6qp -- ceph osd pool ls detail<\/code><\/pre>\n\n\n\n<p>You can show in pretty json format;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it rook-ceph-tools-564c8446db-xh6qp -- ceph osd pool ls detail -f json-pretty<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\n[\n    {\n        \"pool_id\": 1,\n        \"pool_name\": \".mgr\",\n        \"create_time\": \"2023-12-14T21:08:22.463543+0000\",\n        \"flags\": 1,\n        \"flags_names\": \"hashpspool\",\n        \"type\": 1,\n        \"size\": 3,\n        \"min_size\": 2,\n        \"crush_rule\": 0,\n        \"peering_crush_bucket_count\": 0,\n        \"peering_crush_bucket_target\": 0,\n        \"peering_crush_bucket_barrier\": 0,\n        \"peering_crush_bucket_mandatory_member\": 2147483647,\n        \"object_hash\": 2,\n        \"pg_autoscale_mode\": \"on\",\n        \"pg_num\": 1,\n        \"pg_placement_num\": 1,\n        \"pg_placement_num_target\": 1,\n        \"pg_num_target\": 1,\n        \"pg_num_pending\": 1,\n        \"last_pg_merge_meta\": {\n            \"source_pgid\": \"0.0\",\n            \"ready_epoch\": 0,\n            \"last_epoch_started\": 0,\n            \"last_epoch_clean\": 0,\n            \"source_version\": \"0'0\",\n            \"target_version\": \"0'0\"\n        },\n        \"last_change\": \"16\",\n        \"last_force_op_resend\": \"0\",\n        \"last_force_op_resend_prenautilus\": \"0\",\n        \"last_force_op_resend_preluminous\": \"0\",\n        \"auid\": 0,\n        \"snap_mode\": \"selfmanaged\",\n        \"snap_seq\": 0,\n        \"snap_epoch\": 0,\n        \"pool_snaps\": [],\n        \"removed_snaps\": \"[]\",\n        \"quota_max_bytes\": 0,\n        \"quota_max_objects\": 0,\n        \"tiers\": [],\n        \"tier_of\": -1,\n        \"read_tier\": -1,\n        \"write_tier\": -1,\n        \"cache_mode\": \"none\",\n        \"target_max_bytes\": 0,\n        \"target_max_objects\": 0,\n        \"cache_target_dirty_ratio_micro\": 400000,\n        \"cache_target_dirty_high_ratio_micro\": 600000,\n        \"cache_target_full_ratio_micro\": 800000,\n        \"cache_min_flush_age\": 0,\n        \"cache_min_evict_age\": 0,\n        \"erasure_code_profile\": \"\",\n        \"hit_set_params\": {\n            \"type\": \"none\"\n        },\n        \"hit_set_period\": 0,\n        \"hit_set_count\": 0,\n        \"use_gmt_hitset\": true,\n        \"min_read_recency_for_promote\": 0,\n        \"min_write_recency_for_promote\": 0,\n        \"hit_set_grade_decay_rate\": 0,\n        \"hit_set_search_last_n\": 0,\n        \"grade_table\": [],\n        \"stripe_width\": 0,\n        \"expected_num_objects\": 0,\n        \"fast_read\": false,\n        \"options\": {\n            \"pg_num_max\": 32,\n            \"pg_num_min\": 1\n        },\n        \"application_metadata\": {\n            \"mgr\": {}\n        },\n        \"read_balance\": {\n            \"score_acting\": 3,\n            \"score_stable\": 3,\n            \"optimal_score\": 1,\n            \"raw_score_acting\": 3,\n            \"raw_score_stable\": 3,\n            \"primary_affinity_weighted\": 1,\n            \"average_primary_affinity\": 1,\n            \"average_primary_affinity_weighted\": 1\n        }\n    },\n    {\n        \"pool_id\": 3,\n        \"pool_name\": \"replicapool\",\n        \"create_time\": \"2023-12-16T11:29:26.186490+0000\",\n        \"flags\": 8193,\n        \"flags_names\": \"hashpspool,selfmanaged_snaps\",\n        \"type\": 1,\n        \"size\": 3,\n        \"min_size\": 2,\n        \"crush_rule\": 2,\n        \"peering_crush_bucket_count\": 0,\n        \"peering_crush_bucket_target\": 0,\n        \"peering_crush_bucket_barrier\": 0,\n        \"peering_crush_bucket_mandatory_member\": 2147483647,\n        \"object_hash\": 2,\n        \"pg_autoscale_mode\": \"on\",\n        \"pg_num\": 32,\n        \"pg_placement_num\": 32,\n        \"pg_placement_num_target\": 32,\n        \"pg_num_target\": 32,\n        \"pg_num_pending\": 32,\n        \"last_pg_merge_meta\": {\n            \"source_pgid\": \"0.0\",\n            \"ready_epoch\": 0,\n            \"last_epoch_started\": 0,\n            \"last_epoch_clean\": 0,\n            \"source_version\": \"0'0\",\n            \"target_version\": \"0'0\"\n        },\n        \"last_change\": \"60\",\n        \"last_force_op_resend\": \"0\",\n        \"last_force_op_resend_prenautilus\": \"0\",\n        \"last_force_op_resend_preluminous\": \"58\",\n        \"auid\": 0,\n        \"snap_mode\": \"selfmanaged\",\n        \"snap_seq\": 3,\n        \"snap_epoch\": 55,\n        \"pool_snaps\": [],\n        \"removed_snaps\": \"[]\",\n        \"quota_max_bytes\": 0,\n        \"quota_max_objects\": 0,\n        \"tiers\": [],\n        \"tier_of\": -1,\n        \"read_tier\": -1,\n        \"write_tier\": -1,\n        \"cache_mode\": \"none\",\n        \"target_max_bytes\": 0,\n        \"target_max_objects\": 0,\n        \"cache_target_dirty_ratio_micro\": 400000,\n        \"cache_target_dirty_high_ratio_micro\": 600000,\n        \"cache_target_full_ratio_micro\": 800000,\n        \"cache_min_flush_age\": 0,\n        \"cache_min_evict_age\": 0,\n        \"erasure_code_profile\": \"\",\n        \"hit_set_params\": {\n            \"type\": \"none\"\n        },\n        \"hit_set_period\": 0,\n        \"hit_set_count\": 0,\n        \"use_gmt_hitset\": true,\n        \"min_read_recency_for_promote\": 0,\n        \"min_write_recency_for_promote\": 0,\n        \"hit_set_grade_decay_rate\": 0,\n        \"hit_set_search_last_n\": 0,\n        \"grade_table\": [],\n        \"stripe_width\": 0,\n        \"expected_num_objects\": 0,\n        \"fast_read\": false,\n        \"options\": {},\n        \"application_metadata\": {\n            \"rbd\": {}\n        },\n        \"read_balance\": {\n            \"score_acting\": 1.1299999952316284,\n            \"score_stable\": 1.1299999952316284,\n            \"optimal_score\": 1,\n            \"raw_score_acting\": 1.1299999952316284,\n            \"raw_score_stable\": 1.1299999952316284,\n            \"primary_affinity_weighted\": 1,\n            \"average_primary_affinity\": 1,\n            \"average_primary_affinity_weighted\": 1\n        }\n    }\n]\n<\/code><\/pre>\n\n\n\n<p>After a few, the cluster pool and PGs should be created.<\/p>\n\n\n\n<p>Confirm the status of the cluster;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec deploy\/rook-ceph-tools -- ceph status<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>  cluster:\n    id:     e1466372-9f01-42af-8ad7-0bfcfa71ef78\n    health: HEALTH_OK\n \n  services:\n    mon: 3 daemons, quorum a,b,c (age 38h)\n    mgr: a(active, since 21m), standbys: b\n    osd: 3 osds: 3 up (since 38h), 3 in (since 38h)\n \n  data:\n    pools:   2 pools, 33 pgs\n    objects: 3 objects, 449 KiB\n    usage:   81 MiB used, 300 GiB \/ 300 GiB avail\n    pgs:     33 active+clean\n\n<\/code><\/pre>\n\n\n\n<p>You can also see the details on the Ceph dashboard;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1605\" height=\"784\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/rook-ceph-rbd-pool.png?v=1702717245\" alt=\"Deploy Ceph Storage Cluster in Kubernetes using Rook\" class=\"wp-image-19632\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/rook-ceph-rbd-pool.png?v=1702717245 1605w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/rook-ceph-rbd-pool-768x375.png?v=1702717245 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/rook-ceph-rbd-pool-1536x750.png?v=1702717245 1536w\" sizes=\"(max-width: 1605px) 100vw, 1605px\" \/><\/figure>\n\n\n\n<p>The RBD pools for block devices are now ready for use.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configure-kubernetes-applications-to-use-ceph-block-storage\">Configure Kubernetes Applications to use Ceph Block Storage<\/h3>\n\n\n\n<p>You can then configure your Kubernetes applications to use the provisioned block storage.<\/p>\n\n\n\n<p>Rook ship with sample manifest files for MySQL and WordPress applications that can deploy to demo the use of block storage.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cd ~\/rook\/deploy\/examples<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>cat mysql.yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>apiVersion: v1\nkind: Service\nmetadata:\n  name: wordpress-mysql\n  labels:\n    app: wordpress\nspec:\n  ports:\n    - port: 3306\n  selector:\n    app: wordpress\n    tier: mysql\n  clusterIP: None\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: mysql-pv-claim\n  labels:\n    app: wordpress\nspec:\n  storageClassName: rook-ceph-block\n  accessModes:\n    - ReadWriteOnce\n  resources:\n    requests:\n      storage: 20Gi\n---\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: wordpress-mysql\n  labels:\n    app: wordpress\n    tier: mysql\nspec:\n  selector:\n    matchLabels:\n      app: wordpress\n      tier: mysql\n  strategy:\n    type: Recreate\n  template:\n    metadata:\n      labels:\n        app: wordpress\n        tier: mysql\n    spec:\n      containers:\n        - image: mysql:5.6\n          name: mysql\n          env:\n            - name: MYSQL_ROOT_PASSWORD\n              value: changeme\n          ports:\n            - containerPort: 3306\n              name: mysql\n          volumeMounts:\n            - name: mysql-persistent-storage\n              mountPath: \/var\/lib\/mysql\n      volumes:\n        - name: mysql-persistent-storage\n          persistentVolumeClaim:\n            claimName: mysql-pv-claim\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>cat wordpress.yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>apiVersion: v1\nkind: Service\nmetadata:\n  name: wordpress\n  labels:\n    app: wordpress\nspec:\n  ports:\n    - port: 80\n  selector:\n    app: wordpress\n    tier: frontend\n  type: LoadBalancer\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: wp-pv-claim\n  labels:\n    app: wordpress\nspec:\n  storageClassName: rook-ceph-block\n  accessModes:\n    - ReadWriteOnce\n  resources:\n    requests:\n      storage: 20Gi\n---\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: wordpress\n  labels:\n    app: wordpress\n    tier: frontend\nspec:\n  selector:\n    matchLabels:\n      app: wordpress\n      tier: frontend\n  strategy:\n    type: Recreate\n  template:\n    metadata:\n      labels:\n        app: wordpress\n        tier: frontend\n    spec:\n      containers:\n        - image: wordpress:4.6.1-apache\n          name: wordpress\n          env:\n            - name: WORDPRESS_DB_HOST\n              value: wordpress-mysql\n            - name: WORDPRESS_DB_PASSWORD\n              value: changeme\n          ports:\n            - containerPort: 80\n              name: wordpress\n          volumeMounts:\n            - name: wordpress-persistent-storage\n              mountPath: \/var\/www\/html\n      volumes:\n        - name: wordpress-persistent-storage\n          persistentVolumeClaim:\n            claimName: wp-pv-claim\n<\/code><\/pre>\n\n\n\n<p>The manifest files defines the service, the persistenvolumeclaim and the deployment of each app.<\/p>\n\n\n\n<p>The two apps have been configured with a PersistentVolumeClaim (PVC) that allows the apps to request a specific amount of storage from a storage class, <strong>rook-ceph-block<\/strong>, 20GiB each.<\/p>\n\n\n\n<p>You can list available storage classes using the command;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get storageclass<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME              PROVISIONER                  RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE\nrook-ceph-block   rook-ceph.rbd.csi.ceph.com   Delete          Immediate           true                   20m\n<\/code><\/pre>\n\n\n\n<p>Similarly, the PVC also defines the access modes to be for the block storage. In this case, the access mode defined is <strong><code>ReadWriteOnce<\/code> (RWO)<\/strong>. Which means, only one pod can mount the PV and read\/write data at any given time. Access mode can also be, <strong>ReadWriteMany (RWX)<\/strong>, which means, multiple pods can simultaneously mount and access the PV, reading and writing data, or <strong>ROX (ReadOnlyMany)<\/strong> which is used when multiple Pods need read-only access to the storage.<\/p>\n\n\n\n<p>You can create your own apps and define the required volumes from the block storage!<\/p>\n\n\n\n<p>Using the default MySQL and WordPress, let&#8217;s start them;<\/p>\n\n\n\n<p>Start MySQL app from the roo-ceph namespace;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl create -n rook-ceph -f mysql.yaml <\/code><\/pre>\n\n\n\n<p>After a short while, the Pod should be running;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pods -n rook-ceph | grep mysql<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>wordpress-mysql-688ddf7f77-nwshk                     1\/1     Running     0               56s<\/code><\/pre>\n\n\n\n<p>You can also list pods based on their labels.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n &lt;name-space&gt; get pod -l app=&lt;app-label&gt;<\/code><\/pre>\n\n\n\n<p>For example;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph get pods -l app=wordpress<\/code><\/pre>\n\n\n\n<p>You can get labels using;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n &lt;name-space&gt; get pod --show-labels<\/code><\/pre>\n\n\n\n<p>Get the persistent volume claims;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pvc -n rook-ceph<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE\nmysql-pv-claim   Bound    pvc-9daf6557-5a94-4878-9b5b-9c35342abd38   20Gi       RWO            rook-ceph-block   119s\n<\/code><\/pre>\n\n\n\n<p>Similarly, start WordPress;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl create -n rook-ceph -f wordpress.yaml<\/code><\/pre>\n\n\n\n<p>Once it starts, you can check the PVC;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pvc -n rook-ceph<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE\nmysql-pv-claim   Bound    pvc-9daf6557-5a94-4878-9b5b-9c35342abd38   20Gi       RWO            rook-ceph-block   3m55s\nwp-pv-claim      Bound    pvc-9d962a97-808f-4ca6-8f6d-576e9b302884   20Gi       RWO            rook-ceph-block   49s\n<\/code><\/pre>\n\n\n\n<p>The two apps are now running;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get svc -n rook-ceph<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                                     TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE\n...\n<strong>wordpress                                LoadBalancer   10.107.77.47     &lt;pending>     80:30689\/TCP        7m48s\nwordpress-mysql                          ClusterIP      None             &lt;none>        3306\/TCP            10m<\/strong>\n<\/code><\/pre>\n\n\n\n<p>WordPress is exposed via the Kubernetes cluster nodes&#8217; IPs on port 30689\/TCP.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"check-block-devices-on-ceph-rbd-pool\">Check Block Devices on Ceph RBD Pool<\/h3>\n\n\n\n<p>You can check the block devices on the Ceph pool using the Rook toolbox (<strong><code>rbd -p replicapool ls<\/code><\/strong>) or from the Ceph dashboard;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it rook-ceph-tools-564c8446db-xh6qp -- rbd -p replicapool ls<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>csi-vol-57059db6-edfb-4bf3-b9ae-0a1bd18d76d3\ncsi-vol-ac81f374-705c-460e-9e0e-dc8c03aefed0\n<\/code><\/pre>\n\n\n\n<p>Check the usage size;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it rook-ceph-tools-564c8446db-xh6qp -- rbd -p replicapool du<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                                          PROVISIONED  USED   \ncsi-vol-57059db6-edfb-4bf3-b9ae-0a1bd18d76d3       20 GiB  232 MiB\ncsi-vol-ac81f374-705c-460e-9e0e-dc8c03aefed0       20 GiB  140 MiB\n&lt;TOTAL>                                            40 GiB  372 MiB\n<\/code><\/pre>\n\n\n\n<p>Get block device image information (<strong><code>rbd --image &lt;IMAGE_NAME&gt; info -p &lt;poolname&gt;<\/code><\/strong>).<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it rook-ceph-tools-564c8446db-xh6qp -- rbd --image csi-vol-57059db6-edfb-4bf3-b9ae-0a1bd18d76d3 info -p replicapool<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>rbd image 'csi-vol-57059db6-edfb-4bf3-b9ae-0a1bd18d76d3':\n\tsize 20 GiB in 5120 objects\n\torder 22 (4 MiB objects)\n\tsnapshot_count: 0\n\tid: 1810dae9feeeb\n\tblock_name_prefix: rbd_data.1810dae9feeeb\n\tformat: 2\n\tfeatures: layering\n\top_features: \n\tflags: \n\tcreate_timestamp: Sat Dec 16 12:11:16 2023\n\taccess_timestamp: Sat Dec 16 12:11:16 2023\n\tmodify_timestamp: Sat Dec 16 12:11:16 2023\n<\/code><\/pre>\n\n\n\n<p>So, how can you know which persistent volume belongs to which pod? Well, you can list the available persistent volumes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pv -n rook-ceph<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS      REASON   AGE\npvc-9d962a97-808f-4ca6-8f6d-576e9b302884   20Gi       RWO            Delete           Bound    rook-ceph\/wp-pv-claim      rook-ceph-block            24m\npvc-9daf6557-5a94-4878-9b5b-9c35342abd38   20Gi       RWO            Delete           Bound    rook-ceph\/mysql-pv-claim   rook-ceph-block            27m\n<\/code><\/pre>\n\n\n\n<p>You can describe each one of them;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl describe pv pvc-9d962a97-808f-4ca6-8f6d-576e9b302884<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Name:            pvc-9d962a97-808f-4ca6-8f6d-576e9b302884\nLabels:          &lt;none>\nAnnotations:     pv.kubernetes.io\/provisioned-by: rook-ceph.rbd.csi.ceph.com\n                 volume.kubernetes.io\/provisioner-deletion-secret-name: rook-csi-rbd-provisioner\n                 volume.kubernetes.io\/provisioner-deletion-secret-namespace: rook-ceph\nFinalizers:      [kubernetes.io\/pv-protection]\nStorageClass:    rook-ceph-block\nStatus:          Bound\nClaim:           rook-ceph\/wp-pv-claim\nReclaim Policy:  Delete\nAccess Modes:    RWO\nVolumeMode:      Filesystem\nCapacity:        20Gi\nNode Affinity:   &lt;none>\nMessage:         \nSource:\n    Type:              CSI (a Container Storage Interface (CSI) volume source)\n    Driver:            rook-ceph.rbd.csi.ceph.com\n    FSType:            ext4\n    VolumeHandle:      0001-0009-rook-ceph-0000000000000003-ac81f374-705c-460e-9e0e-dc8c03aefed0\n    ReadOnly:          false\n    VolumeAttributes:      clusterID=rook-ceph\n                           imageFeatures=layering\n                           imageFormat=2\n                           imageName=csi-vol-ac81f374-705c-460e-9e0e-dc8c03aefed0\n                           journalPool=replicapool\n                           pool=replicapool\n                           storage.kubernetes.io\/csiProvisionerIdentity=1702728513475-2507-rook-ceph.rbd.csi.ceph.com\nEvents:                &lt;none>\n<\/code><\/pre>\n\n\n\n<p>Volume attributes will also show the block device image name.<\/p>\n\n\n\n<p>From the dashboard, you can see the images under <strong>Block &gt; Images<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1611\" height=\"502\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/kubernetes-app-ceph-block-images.png?v=1702730820\" alt=\"\" class=\"wp-image-19650\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/kubernetes-app-ceph-block-images.png?v=1702730820 1611w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/kubernetes-app-ceph-block-images-768x239.png?v=1702730820 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/kubernetes-app-ceph-block-images-1536x479.png?v=1702730820 1536w\" sizes=\"(max-width: 1611px) 100vw, 1611px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"verify-block-device-image-usage-on-the-pod\">Verify Block Device Image Usage on the Pod<\/h3>\n\n\n\n<p>You can login to the pod and check the attached block devices;<\/p>\n\n\n\n<p>WordPress for example;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it wordpress-658d46dddd-zpmfh -- bash<\/code><\/pre>\n\n\n\n<p>Check block devices;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>lsblk<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT\nloop1    7:1    0 63.5M  1 loop \n<strong>rbd0   251:0    0   20G  0 disk \/var\/www\/html<\/strong>\nvdb    252:16   0  100G  0 disk \nloop4    7:4    0 40.9M  1 loop \nloop2    7:2    0  112M  1 loop \nloop5    7:5    0 63.9M  1 loop \nvda    252:0    0  100G  0 disk \n|-vda2 252:2    0    2G  0 part \n|-vda3 252:3    0   98G  0 part \n`-vda1 252:1         1M  0 part \nloop3    7:3    0 49.9M  1 loop\n<\/code><\/pre>\n\n\n\n<p>The block device is mapped to the <strong><code>\/var\/www\/html<\/code><\/strong>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"manually-create-block-device-image-and-mount-in-a-pod\">Manually Create Block Device Image and Mount in a Pod<\/h3>\n\n\n\n<p>It is also possible to create a block device image manually in the Kubernetes Ceph cluster and mount it directly on a pod, similar to what we did in our previous guide on how to <a href=\"https:\/\/kifarunix.com\/configure-and-use-ceph-block-device-on-linux-clients\" target=\"_blank\" rel=\"noreferrer noopener\">Configure and Use Ceph Block Device on Linux Clients<\/a>.<\/p>\n\n\n\n<p>This however, is not recommended for production environment, but for testing purposes only. If a pod dies, the volume attached to it dies as well.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-direct-mount-pod\">Create Direct Mount Pod<\/h4>\n\n\n\n<p>So, create and start direct mount pod provided by Rook;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cd ~\/rook\/deploy\/examples<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl create -f direct-mount.yaml<\/code><\/pre>\n\n\n\n<p>This will create and start <strong><code>rook-direct-mount-*<\/code><\/strong> pod.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-block-device-image-of-specific-size\">Create Block Device Image of specific Size<\/h4>\n\n\n\n<p>List the pods to get the name of the direct mount pod and login to it;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it rook-direct-mount-57b6f954d9-lgqkb -- bash<\/code><\/pre>\n\n\n\n<p>You can now run the RBD commands to create the image and mount!<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>rbd create &lt;image-name&gt; --size &lt;megabytes&gt; --pool &lt;pool-name&gt;<\/code><\/pre>\n\n\n\n<p>e.g;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>rbd create disk01 --size 10G --pool replicapool<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"get-block-device-image-information\">Get Block Device Image Information<\/h4>\n\n\n\n<p>To list the images in your pool;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>rbd ls -l replicapool<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                                          SIZE    PARENT  FMT  PROT  LOCK\ncsi-vol-57059db6-edfb-4bf3-b9ae-0a1bd18d76d3  20 GiB            2            \ncsi-vol-ac81f374-705c-460e-9e0e-dc8c03aefed0  20 GiB            2            \ndisk01                                        10 GiB            2\n<\/code><\/pre>\n\n\n\n<p>To retrieve information about the image created, run the command;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>rbd --image disk01 -p replicapool info<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"map-block-device-image-to-local-device\">Map Block Device Image to Local Device<\/h4>\n\n\n\n<p>After creating an image, you can map it to block devices to a local device node accessible within your system\/Pod.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>rbd map disk01 --pool replicapool<\/code><\/pre>\n\n\n\n<p>You will see an output like;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/dev\/rbd1<\/code><\/pre>\n\n\n\n<p>To show block device images mapped to kernel modules with the&nbsp;<code>rbd<\/code>&nbsp;command;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>rbd showmapped<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>id  pool         namespace  image                                         snap  device   \n0   replicapool             csi-vol-57059db6-edfb-4bf3-b9ae-0a1bd18d76d3  -     \/dev\/rbd0\n1   replicapool             disk01                                        -     \/dev\/rbd1\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-filesystem-on-mapped-block-device-image\">Create Filesystem on Mapped Block Device Image<\/h4>\n\n\n\n<p>The Ceph mapped block device is now ready. All is left is to create a file system on it and mount it to make it use-able.<\/p>\n\n\n\n<p>For example, to create an XFS file system on it (you can use your preferred filesystem type);<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>mkfs.xfs \/dev\/rbd1 -L ceph-disk01<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"mount-unmount-block-device-image\">Mount\/Unmount Block Device Image<\/h4>\n\n\n\n<p>You can now mount the block device. For example, to mount it under&nbsp;<code><strong>\/mnt\/ceph<\/strong><\/code>&nbsp;directory;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>mkdir \/mnt\/ceph<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>mount \/dev\/rbd1 \/mnt\/ceph<\/code><\/pre>\n\n\n\n<p>Check mounted Filesystems;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df -hT -P \/dev\/rbd1<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Filesystem     Type  Size  Used Avail Use% Mounted on\n\/dev\/rbd1      xfs    10G  105M  9.9G   2% \/mnt\/ceph\n<\/code><\/pre>\n\n\n\n<p>There you go.<\/p>\n\n\n\n<p>You can now start using the volume.<\/p>\n\n\n\n<p>When done using it, you can unmount and unmap!<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>umount \/dev\/rbd1<\/code><\/pre>\n\n\n\n<p>Unmap;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>rbd unmap \/dev\/rbd1<\/code><\/pre>\n\n\n\n<p>And that pretty summarizes our guide on how to configure block storage for Kubernetes on Rook ceph cluster.<\/p>\n\n\n\n<p>You can also check our guide on how to provision shared filesystem;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/configuring-shared-filesystem-for-kubernetes-on-rook-ceph-storage\/\" target=\"_blank\" rel=\"noreferrer noopener\">Configuring Shared Filesystem for Kubernetes on Rook Ceph Storage<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this tutorial, you will learn how to provision block storage for Kubernetes on Rook ceph cluster. Rook is an open-source platform for managing and<\/p>\n","protected":false},"author":10,"featured_media":19664,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121,1338,1076,1668,39],"tags":[7344,7343],"class_list":["post-19638","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","category-ceph","category-containers","category-kubernetes","category-storage","tag-kubernetes-ceph-block-device","tag-pods-mount-ceph-block-device","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19638"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=19638"}],"version-history":[{"count":21,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19638\/revisions"}],"predecessor-version":[{"id":20897,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19638\/revisions\/20897"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/19664"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=19638"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=19638"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=19638"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}