{"id":19586,"date":"2023-12-16T16:39:43","date_gmt":"2023-12-16T13:39:43","guid":{"rendered":"https:\/\/kifarunix.com\/?p=19586"},"modified":"2024-03-10T15:25:39","modified_gmt":"2024-03-10T12:25:39","slug":"deploy-ceph-storage-cluster-in-kubernetes-using-rook","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/deploy-ceph-storage-cluster-in-kubernetes-using-rook\/","title":{"rendered":"Deploy Ceph Storage Cluster in Kubernetes using Rook"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1047\" height=\"590\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/deploy-ceph-in-kubernetes-using-rook.png\" alt=\"Deploy Ceph Storage Cluster in Kubernetes using Rook\" class=\"wp-image-19662\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/deploy-ceph-in-kubernetes-using-rook.png?v=1702733930 1047w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/deploy-ceph-in-kubernetes-using-rook-768x433.png?v=1702733930 768w\" sizes=\"(max-width: 1047px) 100vw, 1047px\" \/><\/figure>\n\n\n\n<p>In this tutorial, you will learn how to deploy Ceph storage cluster in Kubernetes using Rook. <a href=\"https:\/\/rook.io\/docs\/rook\/latest-release\/Getting-Started\/intro\/\" target=\"_blank\" rel=\"noreferrer noopener\">Rook<\/a> is an open-source cloud-native storage orchestrator that provides a platform, framework and support for deploying various distributed storage solutions such as Ceph, NFS, Cassandra, Minio, CockroachDB etc on Kubernetes. This guide will focus on deploying <a href=\"https:\/\/ceph.com\/\" target=\"_blank\" rel=\"noopener\">Ceph<\/a>, a distributed storage system that provides file, block and object storage services, using Rook.<\/p>\n\n\n\n<p>We are using Ubuntu 22.04 LTS server in our cluster.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#deploying-ceph-storage-cluster-in-kubernetes-using-rook\">Deploying Ceph Storage Cluster in Kubernetes using Rook<\/a><ul><li><a href=\"#why-rook-for-ceph-deplyment-on-kubernetes-cluster\">Why Rook for Ceph Deplyment on Kubernetes Cluster?<\/a><\/li><li><a href=\"#rook-components\">Rook Components<\/a><\/li><li><a href=\"#the-architecture\">The Architecture<\/a><\/li><li><a href=\"#prerequisites-for-deploying-ceph-on-k-8-s-using-rook\">Prerequisites for Deploying Ceph on K8S using Rook<\/a><ul><li><a href=\"#a-running-kubernetes-cluster\">A running Kubernetes Cluster<\/a><\/li><li><a href=\"#storage-resources\">Storage Resources:<\/a><\/li><li><a href=\"#lvm-package\">LVM Package<\/a><\/li><\/ul><\/li><li><a href=\"#deploy-the-rook-operator\">Deploy the Rook Operator<\/a><ul><li><a href=\"#clone-current-rook-release-github-repository\">Clone Current Rook Release Github Repository<\/a><\/li><li><a href=\"#deploy-the-rook-operator-1\">Deploy the Rook Operator<\/a><\/li><\/ul><\/li><li><a href=\"#deploying-ceph-storage-cluster-in-kubernetes-using-rook-2\">Deploying Ceph Storage Cluster in Kubernetes using Rook<\/a><\/li><li><a href=\"#check-ceph-cluster-status\">Check Ceph Cluster Status<\/a><\/li><li><a href=\"#enable-rook-ceph-orchestrator-module\">Enable Rook Ceph Orchestrator Module<\/a><\/li><li><a href=\"#rook-ceph-storage-cluster-kubernetes-services\">Rook Ceph Storage Cluster Kubernetes Services<\/a><\/li><li><a href=\"#accessing-rook-kubernetes-ceph-storage-cluster-dashboard\">Accessing Rook Kubernetes Ceph Storage Cluster Dashboard<\/a><\/li><li><a href=\"#create-ceph-storage-pools\">Create Ceph Storage Pools<\/a><\/li><li><a href=\"#reference\">Reference<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"deploying-ceph-storage-cluster-in-kubernetes-using-rook\">Deploying Ceph Storage Cluster in Kubernetes using Rook<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"why-rook-for-ceph-deplyment-on-kubernetes-cluster\">Why Rook for Ceph Deplyment on Kubernetes Cluster?<\/h3>\n\n\n\n<p>Some of the benefits of using Rook to deploy Ceph on Kubernetes cluster:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Task automation:<\/strong>&nbsp;Rook automates deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and&nbsp;resource management for your Ceph storage cluster.<\/li>\n\n\n\n<li><strong>Self-managing, self-scaling, self-healing:<\/strong>&nbsp;Rook ensures that the storage services are always available and performant. It automatically scales the storage up or down as needed and heals any issues that may arise.<\/li>\n\n\n\n<li><strong>Seamless integration with Kubernetes:<\/strong>&nbsp;Rook uses the Kubernetes API and scheduling features to provide a seamless experience for managing your storage. You can use familiar Kubernetes commands and tools to interact with your Rook deployed storage cluster.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"rook-components\">Rook Components<\/h3>\n\n\n\n<p>Rook is made up of several components that work together to manage a storage cluster:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Rook Operator:<\/strong>&nbsp;This is the core component of Rook. It is a Kubernetes operator that is responsible for deploying, configuring, and managing your storage clusters.<\/li>\n\n\n\n<li><strong>Rook agents:<\/strong>&nbsp;These are the daemons that run on each node in the Kubernetes cluster. They are responsible for mounting and unmounting storage devices and for managing the lifecycle of storage pods.<\/li>\n\n\n\n<li><strong>Rook discover<\/strong>: rook-discover is a dedicated containerized component that runs as a pod within your Kubernetes cluster. It is responsible for actively scanning the cluster for existing Ceph daemons such as MONs, OSDs, and MGRs. Rook Discover informs the Rook Agent about the discovered OSDs.<\/li>\n<\/ul>\n\n\n\n<p>Other components include but not limited to;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Ceph daemons:<\/strong>\n<ul class=\"wp-block-list\">\n<li>MONs: Manage the Ceph cluster and store its configuration<\/li>\n\n\n\n<li>OSDs: Store data in the Ceph cluster<\/li>\n\n\n\n<li>MGRs: Provide management services for the Ceph cluster<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Custom Resource Definitions (CRDs):<\/strong>\n<ul class=\"wp-block-list\">\n<li>Define the desired state of the storage cluster<\/li>\n\n\n\n<li>Allow users to configure the storage cluster<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>CSI drivers:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Allow Rook to integrate with the Container Storage Interface (CSI).<\/li>\n\n\n\n<li>There are three CSI drivers integrated with Rook; Ceph RBD, CephFS and NFS (disabled by default). CephFS and RBD drivers are enabled automatically by the Rook operator.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"the-architecture\">The Architecture<\/h3>\n\n\n\n<p>The Rook architecture is depicted by the screenshot below from the Rooks documentation page.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1563\" height=\"1534\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/Rook-High-Level-Architecture.png\" alt=\"\" class=\"wp-image-19614\" style=\"width:820px;height:auto\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/Rook-High-Level-Architecture.png 1563w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/Rook-High-Level-Architecture-768x754.png 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/Rook-High-Level-Architecture-1536x1508.png 1536w\" sizes=\"(max-width: 1563px) 100vw, 1563px\" \/><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\" id=\"prerequisites-for-deploying-ceph-on-k-8-s-using-rook\">Prerequisites for Deploying Ceph on K8S using Rook<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"a-running-kubernetes-cluster\">A running Kubernetes Cluster<\/h4>\n\n\n\n<p>You need to have a Kubernetes cluster up and running. In this guide, we are using a three worker-node Kubernetes cluster. See our guide below on how to deploy a three node Kubernetes cluster on Ubuntu;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/setup-kubernetes-cluster-on-ubuntu\/\" target=\"_blank\" rel=\"noreferrer noopener\">Setup Kubernetes Cluster on Ubuntu 22.04\/20.04<\/a><\/p>\n\n\n\n<p>These are the details of my K8s cluster;<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td><strong>Node<\/strong><\/td><td><strong>Hostname<\/strong><\/td><td><strong>IP Address<\/strong><\/td><td><strong>vCPUs<\/strong><\/td><td><strong>RAM (GB)<\/strong><\/td><td><strong>Storage Resource<\/strong><\/td><td><strong>OS<\/strong><\/td><\/tr><tr><td>Master<\/td><td>master<\/td><td>192.168.122.10<\/td><td>2<\/td><td>8<\/td><td>OS: \/dev\/vda<\/td><td>Ubuntu 22.04<\/td><\/tr><tr><td>Worker 1<\/td><td>worker01<\/td><td>192.168.122.11<\/td><td>2<\/td><td>8<\/td><td>OS: \/dev\/vda, OSD: raw \/dev\/vdb, 100G<\/td><td>Ubuntu 22.04<\/td><\/tr><tr><td>Worker 2<\/td><td>worker02<\/td><td>192.168.122.12<\/td><td>2<\/td><td>8<\/td><td>OS: \/dev\/vda, OSD: raw \/dev\/vdb, 100G<\/td><td>Ubuntu 22.04<\/td><\/tr><tr><td>Worker 3<\/td><td>worker03<\/td><td>192.168.122.13<\/td><td>2<\/td><td>8<\/td><td>OS: \/dev\/vda, OSD: raw \/dev\/vdb, 100G<\/td><td>Ubuntu 22.04<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Cluster nodes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get nodes<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME       STATUS   ROLES           AGE     VERSION\nmaster     Ready    control-plane   13m     v1.28.2\nworker01   Ready    &lt;none>          5m48s   v1.28.2\nworker02   Ready    &lt;none>          5m45s   v1.28.2\nworker03   Ready    &lt;none>          5m42s   v1.28.2\n<\/code><\/pre>\n\n\n\n<p>Note that Kubernetes&nbsp;<strong>v1.22<\/strong>&nbsp;or higher is supported. You can get the version using the <strong><code>kubectl version<\/code><\/strong> command and check the <strong>Server<\/strong> version.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"storage-resources\">Storage Resources:<\/h4>\n\n\n\n<p><strong>At least one<\/strong> of these local storage types is required:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Raw devices (no partitions or formatted filesystems)<\/li>\n\n\n\n<li>Raw partitions (no formatted filesystem)<\/li>\n\n\n\n<li>LVM Logical Volumes (no formatted filesystem)<\/li>\n\n\n\n<li>Persistent Volumes available from a storage class in&nbsp;<code>block<\/code>&nbsp;mode<\/li>\n<\/ul>\n\n\n\n<p>We will be using raw devices with no partitions\/filesystem in this guide.<\/p>\n\n\n\n<p>We have attached raw  block devices each of 100G to each of the worker nodes in the cluster;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>lsblk | grep -v '^loop'<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS\nvda                       252:0    0   100G  0 disk \n\u251c\u2500vda1                    252:1    0     1M  0 part \n\u251c\u2500vda2                    252:2    0     2G  0 part \/boot\n\u2514\u2500vda3                    252:3    0    98G  0 part \n  \u2514\u2500ubuntu--vg-ubuntu--lv 253:0    0    98G  0 lvm  \/var\/lib\/kubelet\/pods\/d447336a-f345-4629-877f-003053e48c1c\/volume-subpaths\/tigera-ca-bundle\/calico-node\/1\n                                                    \/var\/lib\/kubelet\/pods\/8f82a481-64f1-4833-ad88-addc33181c39\/volume-subpaths\/tigera-ca-bundle\/calico-typha\/1\n                                                    \/\n<strong>vdb                       252:16   0   100G  0 disk<\/strong>\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"lvm-package\">LVM Package<\/h4>\n\n\n\n<p>Ceph OSDs have a dependency on LVM in the following scenarios:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>OSDs are created on raw devices or partitions<\/li>\n\n\n\n<li>If encryption is enabled (<code>encryptedDevice: true<\/code>&nbsp;in the cluster CR)<\/li>\n\n\n\n<li>A&nbsp;<code>metadata<\/code>&nbsp;device is specified<\/li>\n<\/ul>\n\n\n\n<p>LVM is not required for OSDs in these scenarios:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Creating OSDs on PVCs using the&nbsp;<code>storageClassDeviceSets<\/code><\/li>\n<\/ul>\n\n\n\n<p>In this guide, since we are using raw block devices for OSDs, then we need to install the LVM package.<\/p>\n\n\n\n<p>Since we are using Ubuntu in our environment, then you can install LVM package as follows;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt install lvm2<\/code><\/pre>\n\n\n\n<p>For any other Linux distro, consult their documentation on how to install the LVM package.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploy-the-rook-operator\">Deploy the Rook Operator<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"clone-current-rook-release-github-repository\">Clone Current Rook Release Github Repository<\/h4>\n\n\n\n<p>Once you have all the prereqs in place, proceed to deploy Rook operator on the cluster node with full access to the <strong><code>kubeconfig<\/code><\/strong>.<\/p>\n\n\n\n<p>So what is kubeconfig? kubeconfig is a an abbreviation for <em>Kubernetes configuration file<\/em> and is:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A YAML file containing the configuration details required to connect to a Kubernetes cluster.<\/li>\n\n\n\n<li>Stores information like:\n<ul class=\"wp-block-list\">\n<li>Cluster server addresses.<\/li>\n\n\n\n<li>User credentials (certificate or token).<\/li>\n\n\n\n<li>Current context (the specific cluster and namespace you want to interact with).<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Used by various tools like&nbsp;<code class=\"\">kubectl<\/code>,&nbsp;the Kubernetes command-line tool,&nbsp;to interact with the cluster.<\/li>\n\n\n\n<li>By default,&nbsp;kubeconfig is stored in the&nbsp;<code class=\"\">.kube\/config<\/code>&nbsp;file within the home directory of your user account.<\/li>\n\n\n\n<li>You can also specify a custom location for the file using the&nbsp;<code class=\"\">--kubeconfig<\/code>&nbsp;flag in kubectl commands.<\/li>\n<\/ul>\n\n\n\n<p>In our cluster, we have our kubeconfig on the control-plane\/worker node.<\/p>\n\n\n\n<p>To deploy Rook operator, you need to clone their Github repository. It is recommended that you clone the <a href=\"https:\/\/github.com\/rook\/rook\/releases\" target=\"_blank\" rel=\"noreferrer noopener\">current Rook release Github repository<\/a>. v1.13.0 is the current release as of this writing.<\/p>\n\n\n\n<p>Install git package on your system. On Ubuntu, you can install git by executing the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt install git<\/code><\/pre>\n\n\n\n<p>You can then clone the current release branch of Rook as follows;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>git clone --single-branch --branch release-1.13 https:\/\/github.com\/rook\/rook.git<\/code><\/pre>\n\n\n\n<p>This clones Rook to <strong><code>rook<\/code><\/strong> directory in the current working directory.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ls -1 rook\/<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>ADOPTERS.md\nbuild\ncmd\nCODE_OF_CONDUCT.md\nCODE-OWNERS\nCONTRIBUTING.md\nDCO\ndeploy\ndesign\nDocumentation\ngo.mod\ngo.sum\nGOVERNANCE.md\nimages\nINSTALL.md\nLICENSE\nMakefile\nmkdocs.yml\nOWNERS.md\nPendingReleaseNotes.md\npkg\nREADME.md\nROADMAP.md\nSECURITY.md\ntests\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"deploy-the-rook-operator-1\">Deploy the Rook Operator<\/h4>\n\n\n\n<p>Next, navigate to the example manifests directory, <strong><code>rook\/deploy\/examples<\/code><\/strong>, and deploy the Rook operator (<em><code>operator.yaml<\/code><\/em>), the CRDs (<strong>Custom Resource Definitions<\/strong>, <em><strong>crds.yaml<\/strong><\/em>) and the common resources (<strong><em><code>common.yaml<\/code><\/em><\/strong>). In Kubernetes, a manifest is a YAML or JSON file that describes the desired state of a Kubernetes object within the cluster. These objects can include deployment, replica set, service&#8230; It includes information like:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Kinds:<\/strong>&nbsp;The type of resource being created (e.g.,&nbsp;Pod,&nbsp;Deployment,&nbsp;Service).<\/li>\n\n\n\n<li><strong>Metadata:<\/strong>&nbsp;Names,&nbsp;labels,&nbsp;annotations,&nbsp;etc.,&nbsp;for identification and configuration.<\/li>\n\n\n\n<li><strong>Specifications:<\/strong>&nbsp;Detailed configuration of the resource,&nbsp;including containers,&nbsp;volumes,&nbsp;network settings,&nbsp;etc.<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>cd ~\/rook\/deploy\/examples<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl create -f crds.yaml -f common.yaml -f operator.yaml<\/code><\/pre>\n\n\n\n<p>Sample command output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>customresourcedefinition.apiextensions.k8s.io\/cephblockpoolradosnamespaces.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephblockpools.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephbucketnotifications.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephbuckettopics.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephclients.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephclusters.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephcosidrivers.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephfilesystemmirrors.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephfilesystems.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephfilesystemsubvolumegroups.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephnfses.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephobjectrealms.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephobjectstores.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephobjectstoreusers.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephobjectzonegroups.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephobjectzones.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/cephrbdmirrors.ceph.rook.io created\ncustomresourcedefinition.apiextensions.k8s.io\/objectbucketclaims.objectbucket.io created\ncustomresourcedefinition.apiextensions.k8s.io\/objectbuckets.objectbucket.io created\nnamespace\/rook-ceph created\nclusterrole.rbac.authorization.k8s.io\/cephfs-csi-nodeplugin created\nclusterrole.rbac.authorization.k8s.io\/cephfs-external-provisioner-runner created\nclusterrole.rbac.authorization.k8s.io\/objectstorage-provisioner-role created\nclusterrole.rbac.authorization.k8s.io\/rbd-csi-nodeplugin created\nclusterrole.rbac.authorization.k8s.io\/rbd-external-provisioner-runner created\nclusterrole.rbac.authorization.k8s.io\/rook-ceph-cluster-mgmt created\nclusterrole.rbac.authorization.k8s.io\/rook-ceph-global created\nclusterrole.rbac.authorization.k8s.io\/rook-ceph-mgr-cluster created\nclusterrole.rbac.authorization.k8s.io\/rook-ceph-mgr-system created\nclusterrole.rbac.authorization.k8s.io\/rook-ceph-object-bucket created\nclusterrole.rbac.authorization.k8s.io\/rook-ceph-osd created\nclusterrole.rbac.authorization.k8s.io\/rook-ceph-system created\nclusterrolebinding.rbac.authorization.k8s.io\/cephfs-csi-nodeplugin-role created\nclusterrolebinding.rbac.authorization.k8s.io\/cephfs-csi-provisioner-role created\nclusterrolebinding.rbac.authorization.k8s.io\/objectstorage-provisioner-role-binding created\nclusterrolebinding.rbac.authorization.k8s.io\/rbd-csi-nodeplugin created\nclusterrolebinding.rbac.authorization.k8s.io\/rbd-csi-provisioner-role created\nclusterrolebinding.rbac.authorization.k8s.io\/rook-ceph-global created\nclusterrolebinding.rbac.authorization.k8s.io\/rook-ceph-mgr-cluster created\nclusterrolebinding.rbac.authorization.k8s.io\/rook-ceph-object-bucket created\nclusterrolebinding.rbac.authorization.k8s.io\/rook-ceph-osd created\nclusterrolebinding.rbac.authorization.k8s.io\/rook-ceph-system created\nrole.rbac.authorization.k8s.io\/cephfs-external-provisioner-cfg created\nrole.rbac.authorization.k8s.io\/rbd-csi-nodeplugin created\nrole.rbac.authorization.k8s.io\/rbd-external-provisioner-cfg created\nrole.rbac.authorization.k8s.io\/rook-ceph-cmd-reporter created\nrole.rbac.authorization.k8s.io\/rook-ceph-mgr created\nrole.rbac.authorization.k8s.io\/rook-ceph-osd created\nrole.rbac.authorization.k8s.io\/rook-ceph-purge-osd created\nrole.rbac.authorization.k8s.io\/rook-ceph-rgw created\nrole.rbac.authorization.k8s.io\/rook-ceph-system created\nrolebinding.rbac.authorization.k8s.io\/cephfs-csi-provisioner-role-cfg created\nrolebinding.rbac.authorization.k8s.io\/rbd-csi-nodeplugin-role-cfg created\nrolebinding.rbac.authorization.k8s.io\/rbd-csi-provisioner-role-cfg created\nrolebinding.rbac.authorization.k8s.io\/rook-ceph-cluster-mgmt created\nrolebinding.rbac.authorization.k8s.io\/rook-ceph-cmd-reporter created\nrolebinding.rbac.authorization.k8s.io\/rook-ceph-mgr created\nrolebinding.rbac.authorization.k8s.io\/rook-ceph-mgr-system created\nrolebinding.rbac.authorization.k8s.io\/rook-ceph-osd created\nrolebinding.rbac.authorization.k8s.io\/rook-ceph-purge-osd created\nrolebinding.rbac.authorization.k8s.io\/rook-ceph-rgw created\nrolebinding.rbac.authorization.k8s.io\/rook-ceph-system created\nserviceaccount\/objectstorage-provisioner created\nserviceaccount\/rook-ceph-cmd-reporter created\nserviceaccount\/rook-ceph-mgr created\nserviceaccount\/rook-ceph-osd created\nserviceaccount\/rook-ceph-purge-osd created\nserviceaccount\/rook-ceph-rgw created\nserviceaccount\/rook-ceph-system created\nserviceaccount\/rook-csi-cephfs-plugin-sa created\nserviceaccount\/rook-csi-cephfs-provisioner-sa created\nserviceaccount\/rook-csi-rbd-plugin-sa created\nserviceaccount\/rook-csi-rbd-provisioner-sa created\nconfigmap\/rook-ceph-operator-config created\ndeployment.apps\/rook-ceph-operator created\n<\/code><\/pre>\n\n\n\n<p>Rook uses a default namespace called <strong><code>rook-ceph<\/code><\/strong>;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get ns<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME               STATUS   AGE\ncalico-apiserver   Active   3h9m\ncalico-system      Active   3h9m\ndefault            Active   3h14m\nkube-node-lease    Active   3h14m\nkube-public        Active   3h14m\nkube-system        Active   3h14m\n<strong>rook-ceph          Active   7s\n<\/strong>tigera-operator    Active   3h10m\n<\/code><\/pre>\n\n\n\n<p>This therefore means that you have to specify the namespace for all subsequent kubectl commands related to Rook. If you want you can set the rook-ceph namespace as your preferred\/default namespace so that you don&#8217;t have to specify the namespace when executing kubectl commands related to Rook by using the command, <strong><code>kubectl config set-context --current --namespace=&lt;insert-rook-default-namespace-name-here&gt;<\/code><\/strong>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploying-ceph-storage-cluster-in-kubernetes-using-rook-2\">Deploying Ceph Storage Cluster in Kubernetes using Rook<\/h3>\n\n\n\n<p>Once the Rook Operator is deployed, you can now create the Ceph cluster.<\/p>\n\n\n\n<p>Before you can proceed to deploy Ceph storage cluster, ensure that the <strong><code>rook-ceph-operator<\/code><\/strong> is in the <code><strong>Running<\/strong><\/code> state.<\/p>\n\n\n\n<p>Note that Rook operator is deployed into <strong><code>rook-ceph<\/code><\/strong> namespace by default.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pod -n rook-ceph<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                                  READY   STATUS    RESTARTS   AGE\nrook-ceph-operator-598b5dd74c-42gh5   1\/1     Running   0          90s\n<\/code><\/pre>\n\n\n\n<p>Rook supports different Ceph cluster settings for various environments. There are different manifests for various cluster environments that Rook ships with. These include;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>cluster.yaml: Cluster settings for a production cluster running on bare metal. Requires at least three worker nodes. Like in our setup.<\/li>\n\n\n\n<li>cluster-on-pvc.yaml: Cluster settings for a production cluster running in a dynamic cloud environment.<\/li>\n\n\n\n<li>cluster-test.yaml: Cluster settings for a test environment such as minikube.<\/li>\n<\/ul>\n\n\n\n<p>Now that the Rook operator is running, run the command below to create the cluster.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl create -f cluster.yaml<\/code><\/pre>\n\n\n\n<p>Output; <code>cephcluster.ceph.rook.io\/rook-ceph created<\/code>.<\/p>\n\n\n\n<p>The cluster will now take a few  to initialize.<\/p>\n\n\n\n<p>You can execute the command below to check the status of the cluster pods;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pod -n rook-ceph<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                                           READY   STATUS    RESTARTS   AGE\ncsi-cephfsplugin-6cdnk                         2\/2     Running   0          11s\ncsi-cephfsplugin-8klf8                         2\/2     Running   0          11s\ncsi-cephfsplugin-dkkdf                         2\/2     Running   0          11s\ncsi-cephfsplugin-provisioner-fd76b9895-888tm   5\/5     Running   0          11s\ncsi-cephfsplugin-provisioner-fd76b9895-gscgb   5\/5     Running   0          11s\ncsi-rbdplugin-gqfcl                            2\/2     Running   0          11s\ncsi-rbdplugin-lkqpm                            2\/2     Running   0          11s\ncsi-rbdplugin-provisioner-75f66b455d-6xlgh     5\/5     Running   0          11s\ncsi-rbdplugin-provisioner-75f66b455d-7w44t     5\/5     Running   0          11s\ncsi-rbdplugin-tjk2r                            2\/2     Running   0          11s\nrook-ceph-mon-a-6b97dfd866-44r2k               1\/2     Running   0          4s\nrook-ceph-operator-598b5dd74c-42gh5            1\/1     Running   0          2m6s\n<\/code><\/pre>\n\n\n\n<p>Note that the number of osd pods that will be created will depend on the number of nodes in the cluster and the number of devices configured. For the default&nbsp;<code>cluster.yaml<\/code>&nbsp;above, one OSD will be created for each available device found on each node.<\/p>\n\n\n\n<p>Once everything is initialized, the pod status may look like as shown in the output below;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                                                 READY   STATUS      RESTARTS   AGE\ncsi-cephfsplugin-6cdnk                               2\/2     Running     0          2m10s\ncsi-cephfsplugin-8klf8                               2\/2     Running     0          2m10s\ncsi-cephfsplugin-dkkdf                               2\/2     Running     0          2m10s\ncsi-cephfsplugin-provisioner-fd76b9895-888tm         5\/5     Running     0          2m10s\ncsi-cephfsplugin-provisioner-fd76b9895-gscgb         5\/5     Running     0          2m10s\ncsi-rbdplugin-gqfcl                                  2\/2     Running     0          2m10s\ncsi-rbdplugin-lkqpm                                  2\/2     Running     0          2m10s\ncsi-rbdplugin-provisioner-75f66b455d-6xlgh           5\/5     Running     0          2m10s\ncsi-rbdplugin-provisioner-75f66b455d-7w44t           5\/5     Running     0          2m10s\ncsi-rbdplugin-tjk2r                                  2\/2     Running     0          2m10s\nrook-ceph-crashcollector-worker01-9585d87f9-96phf    1\/1     Running     0          60s\nrook-ceph-crashcollector-worker02-7549954c4b-d5jch   1\/1     Running     0          49s\nrook-ceph-crashcollector-worker03-675cdbd7f7-pjv7n   1\/1     Running     0          47s\nrook-ceph-exporter-worker01-6c9cf475fc-jxddz         1\/1     Running     0          60s\nrook-ceph-exporter-worker02-6b57f48d4-qsvsr          1\/1     Running     0          46s\nrook-ceph-exporter-worker03-75fb9cc47f-sht56         1\/1     Running     0          45s\nrook-ceph-mgr-a-65f4bb6685-mh4xr                     3\/3     Running     0          80s\nrook-ceph-mgr-b-6648b7fb6-dqtdv                      3\/3     Running     0          79s\nrook-ceph-mon-a-6b97dfd866-44r2k                     2\/2     Running     0          2m3s\nrook-ceph-mon-b-785dcc4874-wkzm4                     2\/2     Running     0          99s\nrook-ceph-mon-c-586576df47-9ctwf                     2\/2     Running     0          90s\nrook-ceph-operator-598b5dd74c-42gh5                  1\/1     Running     0          4m5s\nrook-ceph-osd-0-55f6c88b9c-l8f2w                     2\/2     Running     0          50s\nrook-ceph-osd-1-64c6c74db4-zjfn9                     2\/2     Running     0          49s\nrook-ceph-osd-2-8455dcfb9f-ghrbc                     2\/2     Running     0          47s\nrook-ceph-osd-prepare-worker01-426s7                 0\/1     Completed   0          21s\nrook-ceph-osd-prepare-worker02-trvpb                 0\/1     Completed   0          18s\nrook-ceph-osd-prepare-worker03-b7n6b                 0\/1     Completed   0          15s\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"check-ceph-cluster-status\">Check Ceph Cluster Status<\/h3>\n\n\n\n<p>To check the Ceph cluster status, you can use Rook toolbox. The Rook toolbox is a container with common tools used for rook debugging and testing and can be run in two modes;<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Interactive mode: Start a toolbox pod where you can connect and execute Ceph commands from a shell<\/li>\n\n\n\n<li>One-time job mode: Run a script with Ceph commands and collect the results from the job log<\/li>\n<\/ol>\n\n\n\n<p>We will use the interactive mode in this example guide. Thus, create a toolbox pod;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl create -f toolbox.yaml<\/code><\/pre>\n\n\n\n<p>You can check the current status of the toolbox deployment;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph rollout status deploy\/rook-ceph-tools<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>deployment \"rook-ceph-tools\" successfully rolled out<\/code><\/pre>\n\n\n\n<p>Also;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pod -n rook-ceph | grep tool<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>rook-ceph-tools-564c8446db-xh6qp                     1\/1     Running     0          71s<\/code><\/pre>\n\n\n\n<p>After that, connect to the rook-ceph-tools pod and check Ceph cluster status;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it deploy\/rook-ceph-tools -- ceph -s<\/code><\/pre>\n\n\n\n<p>Or simply just use the name of the tools pod;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it rook-ceph-tools-564c8446db-xh6qp -- ceph -s<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>  cluster:\n    id:     e1466372-9f01-42af-8ad7-0bfcfa71ef78\n    health: HEALTH_OK\n \n  services:\n    mon: 3 daemons, quorum a,b,c (age 21m)\n    mgr: a(active, since 20m), standbys: b\n    osd: 3 osds: 3 up (since 20m), 3 in (since 21m)\n \n  data:\n    pools:   1 pools, 1 pgs\n    objects: 2 objects, 449 KiB\n    usage:   80 MiB used, 300 GiB \/ 300 GiB avail\n    pgs:     1 active+clean\n<\/code><\/pre>\n\n\n\n<p>Or you can login to the pod and execute the commands if you want;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it deploy\/rook-ceph-tools -- bash<\/code><\/pre>\n\n\n\n<p>You can execute any Ceph commands in there.<\/p>\n\n\n\n<p>If you want to remove the toolbox deployment from the <code>rook-ceph<\/code> namespace;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph delete deploy\/rook-ceph-tools<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"enable-rook-ceph-orchestrator-module\">Enable Rook Ceph Orchestrator Module<\/h3>\n\n\n\n<p>The Rook Orchestrator module provides an integration between Ceph\u2019s Orchestrator framework and Rook.&nbsp;It runs in the&nbsp;<code>ceph-mgr<\/code>&nbsp;daemon and implements the Ceph orchestration API by making changes to the Ceph storage cluster in Kubernetes that describe desired cluster state. A Rook cluster\u2019s&nbsp;<code>ceph-mgr<\/code>&nbsp;daemon is running as a Kubernetes pod, and hence, the rook module can connect to the Kubernetes API without any explicit configuration.<\/p>\n\n\n\n<p>While the&nbsp;<code>orchestrator<\/code>&nbsp;Ceph module is enabled by default, the&nbsp;<code>rook<\/code>&nbsp;Ceph module is disabled by default.<\/p>\n\n\n\n<p>You need the Rook toolbox to enable Rook ceph orchestrator module. We have already enabled the Rook toolbox above.<\/p>\n\n\n\n<p>Thus, either login directly to Rook toolbox tool (<code><strong>kubectl -n rook-ceph exec -it deploy\/rook-ceph-tools -- bash<\/strong><\/code>) and execute the Ceph commands to enable the rook orchestrator module or simply execute the ceph commands without logging into the toolbox;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec deploy\/rook-ceph-tools -- ceph mgr module enable rook<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec deploy\/rook-ceph-tools -- ceph orch set backend rook<\/code><\/pre>\n\n\n\n<p>Check the Ceph Orchestrator status;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec deploy\/rook-ceph-tools -- ceph orch status<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Backend: rook\nAvailable: Yes<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"rook-ceph-storage-cluster-kubernetes-services\">Rook Ceph Storage Cluster Kubernetes Services<\/h3>\n\n\n\n<p>Ceph storage cluster Kubernetes services are also created;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get svc -n rook-ceph<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE\nrook-ceph-exporter        ClusterIP   10.99.247.141    <none>        9926\/TCP            34h\nrook-ceph-mgr             ClusterIP   10.96.140.21     <none>        9283\/TCP            34h\nrook-ceph-mgr-dashboard   ClusterIP   10.103.155.135   <none>        8443\/TCP            34h\nrook-ceph-mon-a           ClusterIP   10.110.254.175   <none>        6789\/TCP,3300\/TCP   34h\nrook-ceph-mon-b           ClusterIP   10.110.73.98     <none>        6789\/TCP,3300\/TCP   34h\nrook-ceph-mon-c           ClusterIP   10.98.191.150    <none>        6789\/TCP,3300\/TCP   34h\n<\/code><\/pre>\n\n\n\n<p>All these services are exposed within the Kubernetes cluster and are only accessible within the cluster via their Cluster IPs and ports.<\/p>\n\n\n\n<p>To see more details about each service, you can describe it. For example;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl describe svc &lt;service-name&gt; -n &lt;namespace&gt;<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"accessing-rook-kubernetes-ceph-storage-cluster-dashboard\">Accessing Rook Kubernetes Ceph Storage Cluster Dashboard<\/h3>\n\n\n\n<p>As you can see above, we have a Ceph manager dashboard service, but is only accessible internally within the K8s cluster via the IP, 10.103.155.135, and port 8443 (https). To confirm this, get the description of the service.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl describe svc rook-ceph-mgr-dashboard -n rook-ceph<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Name:              rook-ceph-mgr-dashboard\nNamespace:         rook-ceph\nLabels:            app=rook-ceph-mgr\n                   rook_cluster=rook-ceph\nAnnotations:       &lt;none>\nSelector:          app=rook-ceph-mgr,mgr_role=active,rook_cluster=rook-ceph\nType:              ClusterIP\nIP Family Policy:  SingleStack\nIP Families:       IPv4\nIP:                10.103.155.135\nIPs:               10.103.155.135\nPort:              https-dashboard  8443\/TCP\nTargetPort:        8443\/TCP\nEndpoints:         10.100.30.75:8443\nSession Affinity:  None\nEvents:            &lt;none>\n<\/code><\/pre>\n\n\n\n<p>As you can see, this service is of type <strong><code>ClusterIP<\/code><\/strong>. This, as already mentioned, this exposes the service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster.<\/p>\n\n\n\n<p>How can we externally access Rook Kubernetes Ceph Cluster dashboard?<\/p>\n\n\n\n<p>There are several ways you can use to expose a service externally. One of them is to set the service type to <strong><code>NodePort<\/code><\/strong>. This exposes the service on each Node&#8217;s IP at a static port. You can let the port be defined automatically or manually set it yourself, in the service manifest file.<\/p>\n\n\n\n<p>By default, Rook ships with a service manifest file that you can use to expose Ceph cluster externally via the cluster Node&#8217;s IP. The file is named as <strong><code>dashboard-external-https.yaml<\/code><\/strong> under <strong><code>~\/rook\/deploy\/examples<\/code><\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat ~\/rook\/deploy\/examples\/dashboard-external-https.yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>apiVersion: v1\nkind: Service\nmetadata:\n  name: rook-ceph-mgr-dashboard-external-https\n  namespace: rook-ceph # namespace:cluster\n  labels:\n    app: rook-ceph-mgr\n    rook_cluster: rook-ceph # namespace:cluster\nspec:\n  ports:\n    - name: dashboard\n      port: 8443\n      protocol: TCP\n      targetPort: 8443\n  selector:\n    app: rook-ceph-mgr\n    mgr_role: active\n    rook_cluster: rook-ceph # namespace:cluster\n  sessionAffinity: None\n  type: NodePort\n<\/code><\/pre>\n\n\n\n<p>So, to access the Ceph dashboard externally, create the external dashboard service.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cd ~\/rook\/deploy\/examples<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl create -f dashboard-external-https.yaml<\/code><\/pre>\n\n\n\n<p>Confirm the services;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get svc -n rook-ceph<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE\nrook-ceph-exporter                       ClusterIP   10.99.247.141    &lt;none>        9926\/TCP            35h\nrook-ceph-mgr                            ClusterIP   10.96.140.21     &lt;none>        9283\/TCP            35h\nrook-ceph-mgr-dashboard                  ClusterIP   10.103.155.135   &lt;none>        8443\/TCP            35h\n<strong>rook-ceph-mgr-dashboard-external-https   NodePort    10.105.180.199   &lt;none>        8443:31617\/TCP      25s<\/strong>\nrook-ceph-mon-a                          ClusterIP   10.110.254.175   &lt;none>        6789\/TCP,3300\/TCP   35h\nrook-ceph-mon-b                          ClusterIP   10.110.73.98     &lt;none>        6789\/TCP,3300\/TCP   35h\nrook-ceph-mon-c                          ClusterIP   10.98.191.150    &lt;none>        6789\/TCP,3300\/TCP   35h\n<\/code><\/pre>\n\n\n\n<p>As you can see, the service, rook-ceph-mgr-dashboard-external-https, internal port, <code><strong>8443\/TCP<\/strong><\/code> is mapped to port <strong><code>31617\/TCP<\/code><\/strong> on the each of the cluster host.<\/p>\n\n\n\n<p>Get Kubernetes cluster Nodes IPs using kubectl command;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get nodes -o wide<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME\nmaster     Ready    control-plane   38h   v1.28.2   192.168.122.10   &lt;none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd:\/\/1.6.26\nworker01   Ready    &lt;none>          38h   v1.28.2   192.168.122.11   &lt;none>        Ubuntu 22.04.2 LTS   5.15.0-91-generic   containerd:\/\/1.6.26\nworker02   Ready    &lt;none>          38h   v1.28.2   192.168.122.12   &lt;none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd:\/\/1.6.26\nworker03   Ready    &lt;none>          38h   v1.28.2   192.168.122.13   &lt;none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd:\/\/1.6.26\n<\/code><\/pre>\n\n\n\n<p>So, you can access dashboard on any node IP and port, 31617\/TCP, https (Accept the ssl warning and proceed to dashboard).<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>https:\/\/192.168.122.10:31617<\/strong><\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1598\" height=\"849\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/rook-kubernetes-ceph-dashboard.png?v=1702716140\" alt=\"\" class=\"wp-image-19628\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/rook-kubernetes-ceph-dashboard.png?v=1702716140 1598w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/rook-kubernetes-ceph-dashboard-768x408.png?v=1702716140 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/rook-kubernetes-ceph-dashboard-1536x816.png?v=1702716140 1536w\" sizes=\"(max-width: 1598px) 100vw, 1598px\" \/><\/figure>\n\n\n\n<p>You can get the login credential for the <strong>admin<\/strong> user from the secrets called&nbsp;<code>rook-ceph-dashboard-password<\/code>;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath=\"{&#91;'data']&#91;'password']}\" | base64 --decode &amp;&amp; echo<\/code><\/pre>\n\n\n\n<p>Use the resulting password to login to ceph as user <strong>admin<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1607\" height=\"886\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/rook-ceph-kubernetes-dashboard.png?v=1702725582\" alt=\"\" class=\"wp-image-19643\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/rook-ceph-kubernetes-dashboard.png?v=1702725582 1607w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/rook-ceph-kubernetes-dashboard-768x423.png?v=1702725582 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/12\/rook-ceph-kubernetes-dashboard-1536x847.png?v=1702725582 1536w\" sizes=\"(max-width: 1607px) 100vw, 1607px\" \/><\/figure>\n\n\n\n<p>You can enable Telemetry from the toolbox pod using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n rook-ceph exec -it rook-ceph-tools-564c8446db-xh6qp -- ceph telemetry on --license sharing-1-0<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"create-ceph-storage-pools\">Create Ceph Storage Pools<\/h3>\n\n\n\n<p>The ceph storage cluster is now running, however, without any usable storage pools. Ceph supports three types of storage that can be exposed by Rook:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Block Storage<\/strong>\n<ul class=\"wp-block-list\">\n<li>Represented by Ceph Block Device (RBD)<\/li>\n\n\n\n<li>Provides raw block devices for persistent storage in Kubernetes pods.<\/li>\n\n\n\n<li>Similar to traditional block storage like hard drives or SSDs.<\/li>\n\n\n\n<li>Used for applications requiring low-latency access to raw data,&nbsp;such as databases and file systems.<\/li>\n\n\n\n<li>Check the guide on how to provision Block storage, <a href=\"https:\/\/kifarunix.com\/provision-block-storage-for-kubernetes-on-rook-ceph-cluster\/\" target=\"_blank\" rel=\"noreferrer noopener\">Provision Block Storage for Kubernetes on Rook Ceph Cluster<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Object Storage<\/strong>\n<ul class=\"wp-block-list\">\n<li>Represented by Ceph Object Storage Gateway (RADOS Gateway,&nbsp;RGW)<\/li>\n\n\n\n<li>Offers S3-compatible object storage for unstructured data like images,&nbsp;videos,&nbsp;and backups.<\/li>\n\n\n\n<li>Applications can access objects directly using the S3 API.<\/li>\n\n\n\n<li>Ideal for cloud-native applications and large-scale data management.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Shared Filesystem Storage<\/strong>\n<ul class=\"wp-block-list\">\n<li>Represented by Ceph File System (CephFS)<\/li>\n\n\n\n<li>Provides POSIX-compliant file system accessible from Kubernetes pods.<\/li>\n\n\n\n<li>Similar to traditional file systems like GlusterFS or NFS.<\/li>\n\n\n\n<li>Check the guide <a href=\"https:\/\/kifarunix.com\/configuring-shared-filesystem-for-kubernetes-on-rook-ceph-storage\/\" target=\"_blank\" rel=\"noreferrer noopener\">Configuring Shared Filesystem for Kubernetes on Rook Ceph Storage<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>Rook ships with manifests for all these types of storage just in case you want to configure and use them.<\/p>\n\n\n\n<p>The manifests of these storage types are provided under the directory, <strong><code>~\/rook\/deploy\/examples\/csi\/<\/code><\/strong> for both RBD and CephFS, and under <strong><code>~\/rook\/deploy\/examples\/object.yaml<\/code><\/strong> for Object Storage.<\/p>\n\n\n\n<p>You can proceed to provision the storage via Ceph.<\/p>\n\n\n\n<p>Check our guide below on how to provision block storage for Kubernetes on Rook ceph cluster.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"reference\">Reference<\/h3>\n\n\n\n<p><a href=\"https:\/\/rook.io\/docs\/rook\/latest-release\/Getting-Started\/intro\/\" target=\"_blank\" rel=\"noreferrer noopener\">Rook Ceph Documentation<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this tutorial, you will learn how to deploy Ceph storage cluster in Kubernetes using Rook. Rook is an open-source cloud-native storage orchestrator that provides<\/p>\n","protected":false},"author":10,"featured_media":19662,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121,1338,1076,1668,7338,39],"tags":[7339,7341,7340],"class_list":["post-19586","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","category-ceph","category-containers","category-kubernetes","category-rook","category-storage","tag-deploy-ceph-using-rook","tag-kubernetes-deploy-ceph-cluster","tag-rook-and-ceph-in-kubernetes","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19586"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=19586"}],"version-history":[{"count":50,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19586\/revisions"}],"predecessor-version":[{"id":20899,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19586\/revisions\/20899"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/19662"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=19586"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=19586"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=19586"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}