{"id":19671,"date":"2023-12-21T00:51:47","date_gmt":"2023-12-20T21:51:47","guid":{"rendered":"https:\/\/kifarunix.com\/?p=19671"},"modified":"2024-03-10T15:21:31","modified_gmt":"2024-03-10T12:21:31","slug":"configuring-shared-filesystem-for-kubernetes-on-rook-ceph-storage","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/configuring-shared-filesystem-for-kubernetes-on-rook-ceph-storage\/","title":{"rendered":"Configuring Shared Filesystem for Kubernetes on Rook Ceph Storage"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1043\" height=\"588\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/cephfs-kubernetes.png?v=1703108482\" alt=\"Configuring Shared Filesystem for Kubernetes on Rook Ceph Storage\" class=\"wp-image-19724\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/cephfs-kubernetes.png?v=1703108482 1043w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/cephfs-kubernetes-768x433.png?v=1703108482 768w\" sizes=\"(max-width: 1043px) 100vw, 1043px\" \/><\/figure>\n\n\n\n<p>Welcome to our guide on configuring shared filesystem for Kubernetes on Rook Ceph Storage. In a <a href=\"https:\/\/kubernetes.io\/\" target=\"_blank\" rel=\"noreferrer noopener\">Kubernetes<\/a> cluster, the need for reliable and scalable storage solutions is paramount, especially when it comes to supporting stateful applications. <a href=\"https:\/\/rook.io\/\" target=\"_blank\" rel=\"noreferrer noopener\">Rook<\/a>, an open-source cloud-native storage orchestrator, can be used to deploy Ceph storage cluster in Kubernetes. This offers a robust solution for providing shared storage for Kubernetes workloads.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#configure-shared-filesystem-for-kubernetes-on-rook-ceph-storage\">Configure Shared Filesystem for Kubernetes on Rook Ceph Storage<\/a><ul><li><a href=\"#the-ceph-storage-types\">The Ceph Storage Types<\/a><\/li><li><a href=\"#the-ceph-filesystem-components\">The Ceph Filesystem Components<\/a><\/li><li><a href=\"#deploy-ceph-storage-cluster-in-kubernetes-using-rook\">Deploy Ceph Storage Cluster in Kubernetes using Rook<\/a><\/li><li><a href=\"#configure-shared-filesystem-for-kubernetes\">Configure Shared Filesystem for Kubernetes<\/a><ul><li><a href=\"#ensure-cluster-is-in-healthy-state\">Ensure Cluster is in Healthy State<\/a><\/li><li><a href=\"#create-shared-fileystem-storage-class-on-rook-ceph-cluster\">Create Shared Fileystem StorageClass on Rook Ceph Cluster<\/a><\/li><li><a href=\"#create-the-ceph-filesystem\">Create the Ceph Filesystem<\/a><\/li><\/ul><\/li><li><a href=\"#configure-pods-to-access-the-shared-ceph-fs-storage\">Configure Pods to Access the Shared CephFS Storage<\/a><ul><li><a href=\"#create-a-persistent-volume-claim-pvc-for-each-pod\">Create a PersistentVolumeClaim (PVC) for each Pod<\/a><\/li><li><a href=\"#create-pods-to-use-ceph-fs-persistent-volume\">Create Pods\u00a0to Use CephFS Persistent Volume<\/a><\/li><\/ul><\/li><li><a href=\"#reference\">Reference<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"configure-shared-filesystem-for-kubernetes-on-rook-ceph-storage\">Configure Shared Filesystem for Kubernetes on Rook Ceph Storage<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"the-ceph-storage-types\">The Ceph Storage Types<\/h3>\n\n\n\n<p>Ceph provides several storage types to cater for different use cases and requirements within a distributed storage infrastructure. The primary storage types provided by Ceph include:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Object Storage (RADOS Gateway &#8211; RGW): \n<ul class=\"wp-block-list\">\n<li>Ceph&#8217;s Object Storage is a scalable and distributed storage system that allows users to store and retrieve data using an HTTP RESTful interface.<\/li>\n\n\n\n<li>Commonly accessed via the RADOS Gateway (RGW), and it seamlessly integrates with applications and tools supporting S3 or Swift API.<\/li>\n\n\n\n<li>Ceph Object storage is ideal for handling large amounts of unstructured data like images, videos, and backups.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Block Storage (RBD &#8211; RADOS Block Device):\n<ul class=\"wp-block-list\">\n<li>Provides&nbsp;<strong>raw block device access<\/strong>&nbsp;for applications,&nbsp;similar to physical disks.<\/li>\n\n\n\n<li>Suitable for&nbsp;<strong>stateful applications<\/strong>&nbsp;like databases,&nbsp;virtual machines,&nbsp;and containerized workloads such as Kubernetes persistent volumes.<\/li>\n\n\n\n<li>Offers&nbsp;<strong>high performance and low latency<\/strong>&nbsp;for frequent data access.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>File System (CephFS):\n<ul class=\"wp-block-list\">\n<li>Ceph File System (CephFS) offers a distributedPOSIX-compliant file system that can be mounted on client machines, providing a shared file storage solution for applications. CephFS supports POSIX-compliant file operations, making it suitable for applications that require a traditional file system interface. It is commonly used for shared storage in multi-node environments, including scenarios where multiple pods or virtual machines need access to shared data.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>If you want to learn how to provision block storage, you can check the links below;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/provision-block-storage-for-kubernetes-on-rook-ceph-cluster\/\" target=\"_blank\" rel=\"noreferrer noopener\">Provision Block Storage for Kubernetes on Rook Ceph Cluster<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/configure-and-use-ceph-block-device-on-linux-clients\/\" target=\"_blank\" rel=\"noreferrer noopener\">Configure and Use Ceph Block Device on Linux Clients<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"the-ceph-filesystem-components\">The Ceph Filesystem Components<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1520\" height=\"632\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/Ceph_File_System_components.png\" alt=\"Configuring Shared Filesystem for Kubernetes on Rook Ceph Storage\" class=\"wp-image-19679\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/Ceph_File_System_components.png 1520w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/Ceph_File_System_components-768x319.png 768w\" sizes=\"(max-width: 1520px) 100vw, 1520px\" \/><\/figure><\/div>\n\n\n<p>There are two primary components of the CephFS:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The <strong>MetaData Server (MDS)<\/strong>:\n<ul class=\"wp-block-list\">\n<li>MDS handles file system metadata,&nbsp;including file and directory structures,&nbsp;permissions,&nbsp;and attributes. The metadata includes information such as file names, sizes, timestamps, and access control lists<\/li>\n\n\n\n<li>They handle file system operations like create,&nbsp;delete,&nbsp;rename,&nbsp;and attribute lookup.<\/li>\n\n\n\n<li>CephFS requires at least one Metadata Server daemon (<code>ceph-mds<\/code>) to run.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>The <strong>CephFS Clients<\/strong>:\n<ul class=\"wp-block-list\">\n<li>These are libraries or user programs that interacts with CephFS.<\/li>\n\n\n\n<li>They communicate with the MDS for metadata operations and directly access the OSDs for reading and writing user data.<\/li>\n\n\n\n<li>Examples include the&nbsp;<code class=\"\">ceph-fuse<\/code>&nbsp;client for FUSE mounting and the&nbsp;<code class=\"\">kcephfs<\/code>&nbsp;client for native kernel integration.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>Other components include (as depicted in the screenshot below):<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Ceph File System Library<\/strong> (libcephfs): This is the library that provides a higher-level interface for applications to interact with CephFS.<\/li>\n\n\n\n<li><strong>Ceph RADOS Library<\/strong> (librados): The librados library provides a low-level interface that allows applications to interact with the core Ceph storage cluster.<\/li>\n\n\n\n<li><strong>Ceph OSDs (Object Storage Daemons):<\/strong>&nbsp;These daemons store the actual data chunks (objects) for both CephFS metadata and user data<\/li>\n\n\n\n<li><strong>Ceph Manager<\/strong>: It is responsible for managing and monitoring the Ceph cluster<\/li>\n<\/ul>\n\n\n\n<p>In order to configure a shared storage filesystem for Kubernetes workloads on Rook ceph storage, proceed as follows.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploy-ceph-storage-cluster-in-kubernetes-using-rook\">Deploy Ceph Storage Cluster in Kubernetes using Rook<\/h3>\n\n\n\n<p>In our previous guide, we provided a comprehensive tutorial on how to deploy Ceph storage cluster in Kubernetes using Rook. Check the link below;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/deploy-ceph-storage-cluster-in-kubernetes-using-rook\/\" target=\"_blank\" rel=\"noreferrer noopener\">Deploy Ceph Storage Cluster in Kubernetes using Rook<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configure-shared-filesystem-for-kubernetes\">Configure Shared Filesystem for Kubernetes<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"ensure-cluster-is-in-healthy-state\">Ensure Cluster is in Healthy State<\/h4>\n\n\n\n<p>Before you can proceed to configure Ceph shared filesystem on Rook Ceph storage cluster, ensure that your Ceph cluster is in healthy state.<\/p>\n\n\n\n<p>Note that we are using Rook toolbox pod to execute the Ceph commands;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it rook-ceph-tools-564c8446db-xh6qp -- ceph -s<\/code><\/pre>\n\n\n\n<p>Output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>  cluster:\n    id:     e1466372-9f01-42af-8ad7-0bfcfa71ef78\n    health: HEALTH_OK\n \n  services:\n    mon: 3 daemons, quorum a,b,c (age 2d)\n    mgr: a(active, since 5h), standbys: b\n    osd: 3 osds: 3 up (since 2d), 3 in (since 2d)\n \n  data:\n    pools:   2 pools, 33 pgs\n    objects: 124 objects, 209 MiB\n    usage:   728 MiB used, 299 GiB \/ 300 GiB avail\n    pgs:     33 active+clean\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-shared-fileystem-storage-class-on-rook-ceph-cluster\">Create Shared Fileystem StorageClass on Rook Ceph Cluster<\/h4>\n\n\n\n<p>A <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/storage\/storage-classes\/\" target=\"_blank\" rel=\"noreferrer noopener\">StorageClass<\/a> is an object that defines the different storage configurations, such as volume type, provisioning policies, and parameters for dynamic provisioning of Persistent Volumes (PVs).<\/p>\n\n\n\n<p>Rook ships with a StorageClass manifest file that defines the CephFS storage configuration options.<\/p>\n\n\n\n<p>The manifest is located under the <strong><code>rook\/deploy\/examples\/csi\/cephfs\/<\/code><\/strong> directory.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat ~\/rook\/deploy\/examples\/csi\/cephfs\/<\/code><\/pre>\n\n\n\n<p>There are two types of storageclass manifests file. One for the <strong><code>replicated<\/code><\/strong> pool storage option (storageclass.yaml) and the other for the <strong><code>erasure-coded<\/code><\/strong> pool option (storageclass-ec.yaml). It is recommended to use the default replicated data pool for CephFS.<\/p>\n\n\n\n<p>This is how the replica pool StorageClass is configured <em>(without comment lines)<\/em>;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat ~\/rook\/deploy\/examples\/csi\/cephfs\/storageclass.yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>apiVersion: storage.k8s.io\/v1\nkind: StorageClass\nmetadata:\n  name: rook-cephfs\nprovisioner: rook-ceph.cephfs.csi.ceph.com # driver:namespace:operator\nparameters:\n  clusterID: rook-ceph # namespace:cluster\n  fsName: myfs\n  pool: myfs-replicated\n  csi.storage.k8s.io\/provisioner-secret-name: rook-csi-cephfs-provisioner\n  csi.storage.k8s.io\/provisioner-secret-namespace: rook-ceph # namespace:cluster\n  csi.storage.k8s.io\/controller-expand-secret-name: rook-csi-cephfs-provisioner\n  csi.storage.k8s.io\/controller-expand-secret-namespace: rook-ceph # namespace:cluster\n  csi.storage.k8s.io\/node-stage-secret-name: rook-csi-cephfs-node\n  csi.storage.k8s.io\/node-stage-secret-namespace: rook-ceph # namespace:cluster\nreclaimPolicy: Delete\nallowVolumeExpansion: true\nmountOptions:\n<\/code><\/pre>\n\n\n\n<p>Where:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><code>apiVersion<\/code> and <code>kind<\/code>:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Specifies the API version and kind of the Kubernetes resource. In this case, it&#8217;s a <strong><code>StorageClass<\/code><\/strong> in the <code>storage.k8s.io\/v1<\/code> API version.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><code>metadata<\/code> section:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Defines metadata for the StorageClass. The name of the StorageClass is set to &#8220;rook-cephfs.&#8221;<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>provisioner:<\/strong> <code class=\"\">rook-ceph.cephfs.csi.ceph.com<\/code>\n<ul class=\"wp-block-list\">\n<li>This identifies the CSI provisioner responsible for creating and managing CephFS volumes for your Kubernetes pods.&nbsp;In this case,&nbsp;it specifies the Rook-Ceph CSI provisioner for CephFS.<\/li>\n\n\n\n<li>Rook provides a CSI driver for CephFS,&nbsp;enabling containerized applications to access and utilize persistent storage from your Ceph cluster.<\/li>\n\n\n\n<li><strong>Ensure the namespace used, rook-ceph, matches the namespace of the Rook operator<\/strong>.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><code>parameters<\/code> section:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Specifies parameters specific to Rook CephFS. These parameters include:\n<ul class=\"wp-block-list\">\n<li><code>fsName<\/code>: The name of the CephFS file system, set to &#8220;myfs.&#8221; You can update this to your preferred name.<\/li>\n\n\n\n<li><code>pool<\/code>: The Ceph pool to use for provisioning, set to &#8220;myfs-replicated.&#8221;<\/li>\n\n\n\n<li><code>csi.storage.k8s.io\/provisioner-secret-name<\/code>: The name of the secret containing the provisioner credentials for Rook CephFS.<\/li>\n\n\n\n<li><code>csi.storage.k8s.io\/controller-expand-secret-name<\/code>: The name of the secret containing the credentials for expanding the file system.<\/li>\n\n\n\n<li><code>csi.storage.k8s.io\/node-stage-secret-name<\/code>: The name of the secret containing the credentials for staging on the node.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><code>reclaimPolicy<\/code>:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Specifies the reclaim policy for Persistent Volumes (PVs) created by this StorageClass. In this case, it&#8217;s set to &#8220;Delete,&#8221; meaning that when a PVC (Persistent Volume Claim) using this StorageClass is deleted, the corresponding PV will also be deleted. The other option is <strong>Retain<\/strong>.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><code>allowVolumeExpansion<\/code>:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Enables or disables volume expansion for PVCs created using this StorageClass. It is set to <code>true<\/code>, allowing volume expansion.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><code>mountOptions<\/code>:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Specifies additional mount options for the file system. In this manifest, it&#8217;s left empty, but you can add specific mount options if needed.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>So, run the command below to create the replicated pool CephFS StorageClass;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cd ~\/rook\/deploy\/examples<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl create -f csi\/cephfs\/storageclass.yaml<\/code><\/pre>\n\n\n\n<p>You can list available StorageClasses using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get sc<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME              PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE\nrook-ceph-block   rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true                   20h\nrook-cephfs       rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   39s\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-the-ceph-filesystem\">Create the Ceph Filesystem<\/h4>\n\n\n\n<p>Similarly, Rook ships with two manifests YAML files for creating a replicated or erasure-coded pools, <strong><code>filesystem.yaml<\/code><\/strong> and <strong><code>filesystem-ec.yaml<\/code><\/strong> files respectively. We are using the replicated-pool filesystem in this guide!<\/p>\n\n\n\n<p>The files provide configurations for the pools to be created, (both data and metadata pools) as well as the MDS settings.<\/p>\n\n\n\n<p>This is how the <strong><code>filesystem.yaml<\/code><\/strong> manifest is defined <em>(without comment lines)<\/em>;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat ~\/rook\/deploy\/examples\/filesystem.yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>apiVersion: ceph.rook.io\/v1\nkind: CephFilesystem\nmetadata:\n  name: myfs\n  namespace: rook-ceph # namespace:cluster\nspec:\n  metadataPool:\n    replicated:\n      size: 3\n      requireSafeReplicaSize: true\n    parameters:\n      compression_mode:\n        none\n  dataPools:\n    - name: replicated\n      failureDomain: host\n      replicated:\n        size: 3\n        requireSafeReplicaSize: true\n      parameters:\n        compression_mode:\n          none\n  preserveFilesystemOnDelete: true\n  metadataServer:\n    activeCount: 1\n    activeStandby: true\n    placement:\n      podAntiAffinity:\n        requiredDuringSchedulingIgnoredDuringExecution:\n          - labelSelector:\n              matchExpressions:\n                - key: app\n                  operator: In\n                  values:\n                    - rook-ceph-mds\n            topologyKey: kubernetes.io\/hostname\n        preferredDuringSchedulingIgnoredDuringExecution:\n          - weight: 100\n            podAffinityTerm:\n              labelSelector:\n                matchExpressions:\n                  - key: app\n                    operator: In\n                    values:\n                      - rook-ceph-mds\n              topologyKey: topology.kubernetes.io\/zone\n    priorityClassName: system-cluster-critical\n    livenessProbe:\n      disabled: false\n    startupProbe:\n      disabled: false\n---\napiVersion: ceph.rook.io\/v1\nkind: CephFilesystemSubVolumeGroup\nmetadata:\n  name: myfs-csi # lets keep the svg crd name same as `filesystem name + csi` for the default csi svg\n  namespace: rook-ceph # namespace:cluster\nspec:\n  filesystemName: myfs\n  pinning:\n    distributed: 1\n<\/code><\/pre>\n\n\n\n<p>Where:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><code>apiVersion<\/code> and <code>kind<\/code>:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Specifies the API version and kind of the Kubernetes resource. In this case, it&#8217;s a <strong><code>CephFilesystem<\/code><\/strong>, which represents a Ceph distributed file system managed by Rook.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><code>metadata<\/code> section:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Defines metadata for the StorageClass, for example the name of the StorageClass is set to myfs.&#8221; You can set the name to a different name.<\/li>\n\n\n\n<li>namespace set to the default operator namesace, rook-ceph.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>spec<\/strong>:\n<ul class=\"wp-block-list\">\n<li>defines the desired state of a Kubernetes object.<\/li>\n\n\n\n<li>Ceph File System uses two pools: <strong>Metadata <\/strong>and<strong> data<\/strong> pools.<\/li>\n\n\n\n<li>The metadata pool stores the data of the Ceph Metadata Server (MDS), which generally consists of inodes; that is, the file ownership, permissions, creation date and time, last modified or accessed date and time, parent directory, and so on.<\/li>\n\n\n\n<li>The data pool stores file data. Ceph may store a file as one or more objects, typically representing smaller chunks of file data such as extents.<\/li>\n\n\n\n<li>Hence, <strong><code>metadataPool<\/code> and <code>dataPools<\/code><\/strong> specifies the replication size, the failure domain, the compression mode&#8230;<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><code>preserveFilesystemOnDelete<\/code>:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Set to <code>true<\/code> to preserve the filesystem on deletion, allowing for recovery in case of accidental deletion.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><code>metadataServer<\/code>:<\/strong> configures CephFS Metadata Server (MDS):\n<ul class=\"wp-block-list\">\n<li><strong>activeCount:<\/strong>&nbsp;sets 1 active MDS instance.<\/li>\n\n\n\n<li><strong>activeStandby:<\/strong>&nbsp;enables hot spare MDS for automatic failover.<\/li>\n\n\n\n<li><strong>placement:<\/strong>&nbsp;defines rules for scheduling MDS pods:\n<ul class=\"wp-block-list\">\n<li><strong>podAntiAffinity:<\/strong>&nbsp;avoids scheduling MDS pods on the same node or zone as another MDS.<\/li>\n\n\n\n<li><code><strong>preferredDuringSchedulingIgnoredDuringExecution<\/strong><\/code>: Specifies a preferred rule for scheduling. The weight (here set to 100) indicates the strength of the preference.<\/li>\n\n\n\n<li><code><strong>podAffinityTerm<\/strong><\/code>: Defines the label selector and the topology domain on which the preference is based.<\/li>\n\n\n\n<li><code>labelSelector<\/code>: Specifies that pods with the label &#8220;app: rook-ceph-mds&#8221; are preferred to be scheduled on different nodes.<\/li>\n\n\n\n<li><code>topologyKey<\/code>: : This line specifies that the anti-affinity rule should be based on the <code>topology.kubernetes.io\/zone<\/code> label associated with nodes.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>a <code class=\"\"><strong>priorityClassName<\/strong><\/code> defines a <strong>level of importance for pods<\/strong>. It acts as a way to differentiate and prioritize which pods receive more resources during scheduling and resource allocation. Kubernetes comes with two default priority classes:\n<ul class=\"wp-block-list\">\n<li><strong>system-cluster-critical<\/strong>:&nbsp;This class represents critical system components like etcd and the Kubernetes control plane.&nbsp;Pods using this class are prioritized above everything else and receive resources first.<\/li>\n\n\n\n<li><strong>system-node-critical<\/strong>:&nbsp;This class is for crucial node-level daemons such as kubelet and network plugins.&nbsp;Pods using this class have higher priority than user-defined workloads but lower than system-cluster-critical.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>livenessProbe &amp; startupProbe:<\/strong>&nbsp;enable health checks for MDS pods.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>A CephFilesystemSubVolumeGroup (SVG) is an abstraction within Ceph File System (CephFS) that allows for <strong>fine-grained control over data placement and management<\/strong>. It acts like a logical grouping of CephFS subvolumes that makes it easy to organize data, apply placement policies&#8230;<\/li>\n\n\n\n<li><strong>metadata:<\/strong>\n<ul class=\"wp-block-list\">\n<li><code class=\"\">name<\/code>:&nbsp;Sets the name of the SVG to&nbsp;<code class=\"\">csi<\/code>.&nbsp;This name will be used to identify and reference the group for data placement and management.<\/li>\n\n\n\n<li>namespace: defines the namespace<\/li>\n\n\n\n<li><strong>spec:<\/strong><\/li>\n\n\n\n<li><code class=\"\">filesystemName<\/code>:&nbsp;This field specifies the CephFS filesystem where this SVG belongs.&nbsp;Here,&nbsp;it references the previously defined filesystem named&nbsp;<code class=\"\">myfs<\/code>.<\/li>\n\n\n\n<li><code class=\"\">pinning<\/code>:&nbsp;This section defines the data placement policy for subvolumes within this SVG.\n<ul class=\"wp-block-list\">\n<li><code class=\"\">distributed: 1<\/code>:&nbsp;This configuration instructs Rook to evenly distribute the subvolumes of this SVG across available Ceph Object Storage Daemons (OSDs) in the cluster.&nbsp;The number&nbsp;<code class=\"\">1<\/code>&nbsp;doesn&#8217;t represent a specific distribution algorithm,&nbsp;but rather implies a basic &#8220;spread-the-data-around&#8221; approach.<\/li>\n\n\n\n<li><a href=\"https:\/\/docs.ceph.com\/en\/latest\/cephfs\/fs-volumes\/#pinning-subvolumes-and-subvolume-groups\" target=\"_blank\" rel=\"noreferrer noopener\">Read more about pinning<\/a>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>Read more on the documentation!<\/p>\n\n\n\n<p>Thus, execute the command below to create CephFS (<em>with default settings<\/em>);<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl create -f ~\/rook\/deploy\/examples\/filesystem.yaml<\/code><\/pre>\n\n\n\n<p>Verify the creation of the Ceph filesystem MDS. It was named it as <strong>myfs<\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph get pods | grep myfs<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>rook-ceph-mds-myfs-a-587bb456c-g4g4t                2\/2     Running     0              6m52s\nrook-ceph-mds-myfs-b-fbf99cb49-jpqsh                2\/2     Running     0              6m51s\n<\/code><\/pre>\n\n\n\n<p>Or, check with label;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph get pod -l app=rook-ceph-mds<\/code><\/pre>\n\n\n\n<p>Check the cluster status;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it rook-ceph-tools-564c8446db-xh6qp -- ceph -s<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>  cluster:\n    id:     e1466372-9f01-42af-8ad7-0bfcfa71ef78\n    health: HEALTH_OK\n \n  services:\n    mon: 3 daemons, quorum a,b,c (age 2d)\n    mgr: a(active, since 3h), standbys: b\n    mds: 1\/1 daemons up, 1 hot standby\n    osd: 3 osds: 3 up (since 5d), 3 in (since 5d)\n \n  data:\n    volumes: 1\/1 healthy\n    pools:   4 pools, 81 pgs\n    objects: 146 objects, 209 MiB\n    usage:   796 MiB used, 299 GiB \/ 300 GiB avail\n    pgs:     81 active+clean\n \n  io:\n    client:   853 B\/s rd, 1 op\/s rd, 0 op\/s wr\n\n<\/code><\/pre>\n\n\n\n<p>As you can see, we have 1 active MDS and 1 standby.<\/p>\n\n\n\n<p>You can check the CephFS status from the Rook toolbox;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it rook-ceph-tools-564c8446db-xh6qp -- ceph fs ls -f json-pretty<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>[\n    {\n        \"name\": \"myfs\",\n        \"metadata_pool\": \"myfs-metadata\",\n        \"metadata_pool_id\": 6,\n        \"data_pool_ids\": [\n            7\n        ],\n        \"data_pools\": [\n            \"myfs-replicated\"\n        ]\n    }\n]\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it rook-ceph-tools-564c8446db-xh6qp -- ceph fs status<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>myfs - 0 clients\n====\nRANK      STATE        MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  \n 0        active      myfs-b  Reqs:    0 \/s    12     15     14      0   \n0-s   standby-replay  myfs-a  Evts:    0 \/s     2      5      4      0   \n      POOL         TYPE     USED  AVAIL  \n myfs-metadata   metadata   132k  94.7G  \nmyfs-replicated    data       0   94.7G  \nMDS version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)\n<\/code><\/pre>\n\n\n\n<p>List the hosts, daemons, and processes:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it rook-ceph-tools-564c8446db-xh6qp -- ceph orch ps --daemon_type=mds<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME        HOST      PORTS  STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION    IMAGE ID      \nmds.myfs-a  worker02         running (43m)     0s ago  43m        -        -  <unknown>  8e1c0c287ee0  \nmds.myfs-b  worker03         running (43m)     0s ago  43m        -        -  <unknown>  8e1c0c287ee0\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configure-pods-to-access-the-shared-ceph-fs-storage\">Configure Pods to Access the Shared CephFS Storage<\/h3>\n\n\n\n<p>The CephFS storage is now ready for consumption. Therefore, to demonstrate how you have configure Kubernetes Pods to access the shared CephFS storage, let&#8217;s create two Pods that will access the same shared storage filesystem.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-a-persistent-volume-claim-pvc-for-each-pod\">Create a PersistentVolumeClaim (PVC) for each Pod<\/h4>\n\n\n\n<p>A PersistentVolumeClaim (PVC) is a resource in Kubernetes that allows a Pod to request and use a specific amount of storage from a storage class. Remember we have created a CephFS storage class above named <strong><code>rook-cephfs<\/code><\/strong>;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get sc<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME              PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE\nrook-ceph-block   rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true                   3d21h\nrook-cephfs       rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   4h9m\n<\/code><\/pre>\n\n\n\n<p>A PVC acts as a request for storage and enables dynamic provisioning, allowing pods to access persistent storage that is automatically provisioned and managed by the underlying storage system.<\/p>\n\n\n\n<p>Below is a manifest file to create a sample PVC;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim cephf-test-pods-pvc.yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: cephfs-pvc\nspec:\n  accessModes:\n    - ReadWriteMany\n  storageClassName: rook-cephfs\n  resources:\n    requests:\n      storage: 10Gi\n<\/code><\/pre>\n\n\n\n<p>Where:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><code>apiVersion<\/code> and <code>kind<\/code>:<\/strong> Specify the API version and type of resource, in this case, a PersistentVolumeClaim.<\/li>\n\n\n\n<li><strong><code>metadata<\/code>:<\/strong> Contains metadata for the PVC, including the name (<code>e.g <\/code><strong><code>cephfs-pvc<\/code><\/strong>).<\/li>\n\n\n\n<li><strong><code>spec<\/code>:<\/strong> Defines the specifications for the PVC.\n<ul class=\"wp-block-list\">\n<li><strong><code>accessModes<\/code>:<\/strong> Specifies the <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/storage\/persistent-volumes\/#access-modes\" target=\"_blank\" rel=\"noreferrer noopener\">access mode<\/a> for the volume (<code>ReadWriteMany<\/code>), allowing multiple pods to access the volume simultaneously.<\/li>\n\n\n\n<li><strong><code>storageClassName<\/code>:<\/strong> Specifies the StorageClass (<code>rook-cephfs<\/code>) to use for provisioning the persistent storage.<\/li>\n\n\n\n<li><strong><code>resources.requests.storage: 10Gi<\/code>:<\/strong>&nbsp;Requests a storage capacity of 10 gigabyte for this PVC.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>So, in summary, this PVC is requesting 10 gigabytes of storage with the <code>ReadWriteMany<\/code> access mode from the <code>rook-cephfs<\/code> StorageClass. The storage is expected to be dynamically provisioned by Rook CephFS, allowing multiple pods to mount and access the volume simultaneously.<\/p>\n\n\n\n<p>When you have the PVC manifest file ready, then create it using the command, <strong><code>kubectl create -f &lt;path-to.yaml&gt;<\/code><\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl create -f demo-cephfs-pvc.yaml<\/code><\/pre>\n\n\n\n<p>You can get the status of the PVC;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pvc<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>NAME         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE\ncephfs-pvc   Pending                                      rook-cephfs    6s\n<\/code><\/pre>\n\n\n\n<p>As you can see, the STATUS is Pending.<\/p>\n\n\n\n<p>Status of a PVC can be:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Pending:<\/strong> The PVC is in the process of being provisioned. This status occurs when the storage system is still allocating the requested resources.<\/li>\n\n\n\n<li><strong>Bound:<\/strong> The PVC has been successfully provisioned, and it is now bound to a Persistent Volume (PV). The PV is the actual storage resource in the cluster.<\/li>\n\n\n\n<li><strong>Lost:<\/strong> This status may occur if the connection to the PV is lost, often due to a failure in the underlying storage system or other issues.<\/li>\n\n\n\n<li><strong>Failed:<\/strong> The provisioning process failed, and the PVC couldn&#8217;t be bound to a PV. This status may occur if the requested resources are not available or if there are issues with the storage class.<\/li>\n\n\n\n<li><strong>Released<\/strong>: This status indicates that the PVC is no longer attached to any storage resource either due to manual deletion or under some conditions, StorageClass or reclaim policy releases it. While the PVC is in the &#8220;Released&#8221; state, the associated PV maybe retained, deleted or recycled depending on the <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/storage\/persistent-volumes\/#reclaiming\" target=\"_blank\" rel=\"noreferrer noopener\">reclaim policy<\/a> defined.<\/li>\n<\/ol>\n\n\n\n<p>If you descibe the PVC, you should see that it is being provisioned;<\/p>\n\n\n\n<p><\/p>\n\n\n\n<pre class=\"scroll-box\"><code>Name:          cephfs-pvc\nNamespace:     default\nStorageClass:  rook-cephfs\nStatus:        Pending\nVolume:        \nLabels:        <none>\nAnnotations:   volume.beta.kubernetes.io\/storage-provisioner: rook-ceph.cephfs.csi.ceph.com\n               volume.kubernetes.io\/storage-provisioner: rook-ceph.cephfs.csi.ceph.com\nFinalizers:    [kubernetes.io\/pvc-protection]\nCapacity:      \nAccess Modes:  \nVolumeMode:    Filesystem\nUsed By:       <none>\nEvents:\n  Type    Reason                Age               From                                                                                                             Message\n  ----    ------                ----              ----                                                                                                             -------\n  Normal  Provisioning          47s               rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-fd76b9895-888tm_8ce04e2b-2580-4f72-a856-eaca75629dfa  External provisioner is provisioning volume for claim \"default\/cephfs-pvc\"\n  Normal  ExternalProvisioning  2s (x5 over 47s)  persistentvolume-controller                                                                                      Waiting for a volume to be created either by the external provisioner 'rook-ceph.cephfs.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.\n<\/code><\/pre>\n\n\n\n<p>If you check the status after a short while, it should be bound now!<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pvc<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE\ncephfs-pvc   Bound    pvc-bafde748-ba57-4d50-abd3-45ba07851f1c   10Gi       RWX            rook-cephfs    2m22s\n<\/code><\/pre>\n\n\n\n<p>To get more details of a PVC;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl describe pvc &lt;name-of-the-pvc&gt;<\/code><\/pre>\n\n\n\n<p>E.g;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl describe cephfs-pvc<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>Name:          cephfs-pvc\nNamespace:     default\nStorageClass:  rook-cephfs\nStatus:        Bound\nVolume:        pvc-bafde748-ba57-4d50-abd3-45ba07851f1c\nLabels:        <none>\nAnnotations:   pv.kubernetes.io\/bind-completed: yes\n               pv.kubernetes.io\/bound-by-controller: yes\n               volume.beta.kubernetes.io\/storage-provisioner: rook-ceph.cephfs.csi.ceph.com\n               volume.kubernetes.io\/storage-provisioner: rook-ceph.cephfs.csi.ceph.com\nFinalizers:    [kubernetes.io\/pvc-protection]\nCapacity:      10Gi\nAccess Modes:  RWX\nVolumeMode:    Filesystem\nUsed By:       <none>\nEvents:\n  Type    Reason                 Age                   From                                                                                                             Message\n  ----    ------                 ----                  ----                                                                                                             -------\n  Normal  Provisioning           2m56s                 rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-fd76b9895-888tm_8ce04e2b-2580-4f72-a856-eaca75629dfa  External provisioner is provisioning volume for claim \"default\/cephfs-pvc\"\n  Normal  ExternalProvisioning   41s (x11 over 2m56s)  persistentvolume-controller                                                                                      Waiting for a volume to be created either by the external provisioner 'rook-ceph.cephfs.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.\n  Normal  Provisioning           35s                   rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-fd76b9895-888tm_cfaa8394-7477-4457-a2ab-410afcc95b05  External provisioner is provisioning volume for claim \"default\/cephfs-pvc\"\n  Normal  ProvisioningSucceeded  35s                   rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-fd76b9895-888tm_cfaa8394-7477-4457-a2ab-410afcc95b05  Successfully provisioned volume pvc-bafde748-ba57-4d50-abd3-45ba07851f1c\n<\/code><\/pre>\n\n\n\n<p>StorageClass acts as a template for creating PVs.&nbsp;It defines the type of storage (local disk,&nbsp;network storage,&nbsp;etc.),&nbsp;provisioner to use (CSI drivers,&nbsp;etc.),&nbsp;and other configuration parameters for a PV.<\/p>\n\n\n\n<p>So, if you check, you should now have a PV created for the PVCs created above (StorageClass is <strong>root-cephfs<\/strong>;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pv<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS      REASON   AGE\npvc-9d962a97-808f-4ca6-8f6d-576e9b302884   20Gi       RWO            Delete           Bound    rook-ceph\/wp-pv-claim      rook-ceph-block            4d7h\npvc-9daf6557-5a94-4878-9b5b-9c35342abd38   20Gi       RWO            Delete           Bound    rook-ceph\/mysql-pv-claim   rook-ceph-block            4d7h\n<strong>pvc-bafde748-ba57-4d50-abd3-45ba07851f1c   10Gi       RWX            Delete           Bound    default\/cephfs-pvc         rook-cephfs                84s<\/strong>\n<\/code><\/pre>\n\n\n\n<p>If you noticed, we didn&#8217;t manually create a PersistentVolume (PV). This is because, one of the primary purposes of StorageClasses is to enable dynamic provisioning of Persistent Volumes. When a PVC (Persistent Volume Claim) is created without specifying a particular PV, the StorageClass determines the appropriate volume type and provisions a PV dynamically.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-pods-to-use-ceph-fs-persistent-volume\">Create Pods<a href=\"https:\/\/kubernetes.io\/docs\/tasks\/configure-pod-container\/configure-persistent-volume-storage\/#create-a-pod\" target=\"_blank\" rel=\"noopener\">&nbsp;<\/a>to Use CephFS Persistent Volume<\/h4>\n\n\n\n<p>We have created a PVC that requests a filesystem of 10G volume.<\/p>\n\n\n\n<p>To demonstrate how this shared filesystem volume can be mounted and used by multiple Pods (remember the access mode is RWX), let&#8217;s create two Pods of an Nginx web server with access to the same configuration files stored under the shared filesystem.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim nginx-pv-pods.yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>apiVersion: v1\nkind: Pod\nmetadata:\n  name: nginx-demo-app1\nspec:\n  volumes:\n    - name: cephfs-volume\n      persistentVolumeClaim:\n        claimName: cephfs-pvc\n  containers:\n    - name: nginx-demo-app1\n      image: nginx\n      ports:\n        - containerPort: 80\n          name: \"http-server\"\n      volumeMounts:\n        - mountPath: \"\/usr\/share\/nginx\/html\"\n          name: cephfs-volume\n---  \napiVersion: v1\nkind: Pod\nmetadata:\n  name: nginx-demo-app2\nspec:\n  volumes:\n    - name: cephfs-volume\n      persistentVolumeClaim:\n        claimName: cephfs-pvc\n  containers:\n    - name: nginx-demo-app2\n      image: nginx\n      ports:\n        - containerPort: 80\n          name: \"http-server\"\n      volumeMounts:\n        - mountPath: \"\/usr\/share\/nginx\/html\"\n          name: cephfs-volume\n<\/code><\/pre>\n\n\n\n<p>Where:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><code>apiVersion<\/code> and <code>kind<\/code>:<\/strong> Indicate the API version (v1) and the resource type (Pod).<\/li>\n\n\n\n<li><strong><code>metadata<\/code>:<\/strong> Contains metadata for the Pod, including the name (<code>nginx-demo-app1<\/code>).<\/li>\n\n\n\n<li><strong><code>spec<\/code>:<\/strong> Specifies the specifications for the Pod.\n<ul class=\"wp-block-list\">\n<li><strong><code>volumes<\/code>:<\/strong> Describes the volumes to be mounted. In this case, it defines a volume named <code>cephfs-volume<\/code> that uses the <code>cephfs-pvc<\/code> PVC.\n<ul class=\"wp-block-list\">\n<li><strong><code>name: cephfs-volume<\/code>:<\/strong> The name of the volume.<\/li>\n\n\n\n<li><strong><code>persistentVolumeClaim:<\/code>:<\/strong> Specifies that the volume is backed by a PersistentVolumeClaim.\n<ul class=\"wp-block-list\">\n<li><strong><code>claimName: cephfs-pvc<\/code>:<\/strong> Specifies the name of the PersistentVolumeClaim (<code>cephfs-pvc<\/code>) to use.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><code>containers<\/code>:<\/strong> Describes the containers within the Pod.\n<ul class=\"wp-block-list\">\n<li><strong><code>name: nginx-demo-app1<\/code>:<\/strong> The name of the container.<\/li>\n\n\n\n<li><strong><code>image: nginx<\/code>:<\/strong> Specifies the Docker image to use for the container (NGINX in this case).<\/li>\n\n\n\n<li><strong><code>ports:<\/code>:<\/strong> Specifies the container&#8217;s port configuration.\n<ul class=\"wp-block-list\">\n<li><strong><code>containerPort: 80<\/code>:<\/strong> Exposes port 80 for the NGINX web server.\n<ul class=\"wp-block-list\">\n<li><strong><code>name: \"http-server\"<\/code>:<\/strong> Provides a name for the exposed port.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><code>volumeMounts:<\/code>:<\/strong> Describes the volume mounts for the container.\n<ul class=\"wp-block-list\">\n<li><strong><code>mountPath: \"\/usr\/share\/nginx\/html\"<\/code>:<\/strong> Specifies the path within the container where the CephFS volume will be mounted.<\/li>\n\n\n\n<li><strong><code>name: cephfs-volume<\/code>:<\/strong> Refers to the previously defined volume (<code>cephfs-volume<\/code>).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>Create the Pods;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl create -f nginx-pv-pods.yaml<\/code><\/pre>\n\n\n\n<p>Once created, check the running pods;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pods<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME              READY   STATUS    RESTARTS   AGE\nnginx-demo-app1   1\/1     Running   0          21s\nnginx-demo-app2   1\/1     Running   0          21s\n<\/code><\/pre>\n\n\n\n<p>You can show the details of a Pod;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl describe pod nginx-demo-app1<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Name:             nginx-demo-app1\nNamespace:        default\nPriority:         0\nService Account:  default\nNode:             worker01\/192.168.122.11\nStart Time:       Wed, 20 Dec 2023 20:42:52 +0000\nLabels:           <none>\nAnnotations:      cni.projectcalico.org\/containerID: f0e1cab7f5ad236b39403296cf4c7f4b66a1e5b795e2a5be878f0ae0cf176900\n                  cni.projectcalico.org\/podIP: 10.100.5.35\/32\n                  cni.projectcalico.org\/podIPs: 10.100.5.35\/32\nStatus:           Running\nIP:               10.100.5.35\nIPs:\n  IP:  10.100.5.35\nContainers:\n  nginx-demo-app1:\n    Container ID:   containerd:\/\/619fb7bec4512ef62b4c3d72a3e89f787d3974456cf3a84dbc4db99f0347b264\n    Image:          nginx\n    Image ID:       docker.io\/library\/nginx@sha256:5040a25cc87f100efc43c5c8c2f504c76035441344345c86d435c693758874b7\n    Port:           80\/TCP\n    Host Port:      0\/TCP\n    State:          Running\n      Started:      Wed, 20 Dec 2023 20:43:03 +0000\n    Ready:          True\n    Restart Count:  0\n    Environment:    <none>\n    Mounts:\n      \/usr\/share\/nginx\/html from cephfs-volume (rw)\n      \/var\/run\/secrets\/kubernetes.io\/serviceaccount from kube-api-access-44nj7 (ro)\nConditions:\n  Type              Status\n  Initialized       True \n  Ready             True \n  ContainersReady   True \n  PodScheduled      True \nVolumes:\n  cephfs-volume:\n    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)\n    ClaimName:  cephfs-pvc\n    ReadOnly:   false\n  kube-api-access-44nj7:\n    Type:                    Projected (a volume that contains injected data from multiple sources)\n    TokenExpirationSeconds:  3607\n    ConfigMapName:           kube-root-ca.crt\n    ConfigMapOptional:       <nil>\n    DownwardAPI:             true\nQoS Class:                   BestEffort\nNode-Selectors:              <none>\nTolerations:                 node.kubernetes.io\/not-ready:NoExecute op=Exists for 300s\n                             node.kubernetes.io\/unreachable:NoExecute op=Exists for 300s\nEvents:\n  Type    Reason                  Age   From                     Message\n  ----    ------                  ----  ----                     -------\n  Normal  Scheduled               102s  default-scheduler        Successfully assigned default\/nginx-demo-app1 to worker01\n  Normal  SuccessfulAttachVolume  101s  attachdetach-controller  AttachVolume.Attach succeeded for volume \"pvc-bafde748-ba57-4d50-abd3-45ba07851f1c\"\n  Normal  Pulling                 91s   kubelet                  Pulling image \"nginx\"\n  Normal  Pulled                  91s   kubelet                  Successfully pulled image \"nginx\" in 825ms (825ms including waiting)\n  Normal  Created                 91s   kubelet                  Created container nginx-demo-app1\n  Normal  Started                 91s   kubelet                  Started container nginx-demo-app1\n<\/code><\/pre>\n\n\n\n<p>The Pods are now running.<\/p>\n\n\n\n<p>So, drop into the shell of one of them, say <strong>nginx-demo-app1<\/strong>;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl exec -it nginx-demo-app1 -- bash<\/code><\/pre>\n\n\n\n<p>Check mounted filesystems;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df -hT -t ceph<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>root@nginx-demo-app1:\/# df -hT -t ceph\nFilesystem                                                                                                                                              Type  Size  Used Avail Use% Mounted on\n10.98.191.150:6789,10.110.254.175:6789,10.110.73.98:6789:\/volumes\/csi\/csi-vol-8d10c28b-42b0-4fba-8287-3e1fbb2887fd\/ce77007d-12d4-40a6-8136-81ca642d79dd ceph   10G     0   10G   0% \/usr\/share\/nginx\/html\n\n<\/code><\/pre>\n\n\n\n<p>Check the content of the mount point, <strong><code>\/usr\/share\/nginx\/html<\/code><\/strong>. Remember this is the default root directory for Nginx.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>root@nginx-demo-app1:\/# ls usr\/share\/nginx\/html\/\nroot@nginx-demo-app1:\/# <\/code><\/pre>\n\n\n\n<p>Create a sample web page;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"hello form k8s CephFS test\" &gt; usr\/share\/nginx\/html\/index.html<\/code><\/pre>\n\n\n\n<p>Confirm;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>root@nginx-demo-app1:\/# curl -k http:\/\/localhost\nhello form k8s CephFS test\nroot@nginx-demo-app1:\/# \n<\/code><\/pre>\n\n\n\n<p>Now, login to the second pod and check mounted filesystems as well as if the created file above is available;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl exec -it nginx-demo-app2 -- bash<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>df -hT -t ceph<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>root@nginx-demo-app2:\/# df -hT -t ceph\nFilesystem                                                                                                                                              Type  Size  Used Avail Use% Mounted on\n10.98.191.150:6789,10.110.254.175:6789,10.110.73.98:6789:\/volumes\/csi\/csi-vol-8d10c28b-42b0-4fba-8287-3e1fbb2887fd\/ce77007d-12d4-40a6-8136-81ca642d79dd ceph   10G     0   10G   0% \/usr\/share\/nginx\/html\n\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>root@nginx-demo-app2:\/# ls usr\/share\/nginx\/html\/\nindex.html<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>cat usr\/share\/nginx\/html\/index.html<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>root@nginx-demo-app2:\/# cat usr\/share\/nginx\/html\/index.html\n<strong>hello form k8s CephFS test<\/strong><\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -k http:\/\/localhost<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>root@nginx-demo-app2:\/# curl -k http:\/\/localhost\nhello form k8s CephFS test<\/code><\/pre>\n\n\n\n<p>And that confirms that you have successfully configured a Pod to use storage from a PersistentVolumeClaim.<\/p>\n\n\n\n<p>You can attach multiple apps to your PVC for shared storage!<\/p>\n\n\n\n<p>Clean up!<\/p>\n\n\n\n<p>If you want to clean your shared storage filesystem;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Delete the pods<\/li>\n\n\n\n<li>Delete the PVCs<\/li>\n\n\n\n<li>Delete the PV<\/li>\n<\/ul>\n\n\n\n<p>Depending on your retain policy, the volume will be retained or deleted with data .<\/p>\n\n\n\n<p>That concludes our guide on how to configure shared filesystem for Kubernetes on Rook Ceph Storage.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"reference\">Reference<\/h3>\n\n\n\n<p>Read more on <a href=\"https:\/\/rook.io\/docs\/rook\/latest-release\/Storage-Configuration\/Shared-Filesystem-CephFS\/filesystem-storage\/\" target=\"_blank\" rel=\"noreferrer noopener\">Filesystem Storage Overview<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Welcome to our guide on configuring shared filesystem for Kubernetes on Rook Ceph Storage. In a Kubernetes cluster, the need for reliable and scalable storage<\/p>\n","protected":false},"author":10,"featured_media":19724,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121,1338,1076,1668,39],"tags":[7348,7349,7350],"class_list":["post-19671","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","category-ceph","category-containers","category-kubernetes","category-storage","tag-cephfs-on-kubernetes","tag-rook-ceph-shared-filesystem","tag-shared-filesystem-rook","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19671"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=19671"}],"version-history":[{"count":36,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19671\/revisions"}],"predecessor-version":[{"id":20895,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19671\/revisions\/20895"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/19724"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=19671"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=19671"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=19671"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}