{"id":19326,"date":"2023-11-21T08:55:38","date_gmt":"2023-11-21T05:55:38","guid":{"rendered":"https:\/\/kifarunix.com\/?p=19326"},"modified":"2024-03-10T14:50:30","modified_gmt":"2024-03-10T11:50:30","slug":"how-to-deploy-ceph-storage-cluster-on-rocky-linux","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/how-to-deploy-ceph-storage-cluster-on-rocky-linux\/","title":{"rendered":"How to Deploy Ceph Storage Cluster on Rocky Linux"},"content":{"rendered":"\n<p>Follow through this post to learn how to deploy Ceph storage cluster on Rocky Linux. Ceph is a scalable distributed storage system designed for cloud infrastructure and web-scale&nbsp;<a href=\"https:\/\/docs.ceph.com\/docs\/master\/glossary\/#term-ceph-object-storage\" target=\"_blank\" rel=\"noreferrer noopener\">object storage<\/a>. It can also be used to provide&nbsp;<a href=\"https:\/\/docs.ceph.com\/docs\/master\/glossary\/#term-ceph-block-device\" target=\"_blank\" rel=\"noreferrer noopener\">Ceph Block Storage<\/a>&nbsp;as well as&nbsp;<a href=\"https:\/\/docs.ceph.com\/docs\/master\/glossary\/#term-ceph-file-system\" target=\"_blank\" rel=\"noreferrer noopener\">Ceph File System storage<\/a>.<\/p>\n\n\n\n<p>As of this writing, Ceph 18 (code named Reef) is the current stable <a href=\"https:\/\/docs.ceph.com\/en\/latest\/releases\/\" target=\"_blank\" rel=\"noreferrer noopener\">release<\/a>.<\/p>\n\n\n\n<p>We are using Rocky Linux 9.2 (Blue Onyx) in this guide.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#deploying-ceph-storage-cluster-on-rocky-linux\">Deploying Ceph Storage Cluster on Rocky Linux<\/a><ul><li><a href=\"#the-ceph-storage-cluster-daemons\">The Ceph Storage Cluster Daemons<\/a><\/li><li><a href=\"#ceph-storage-cluster-deployment-methods\">Ceph Storage Cluster Deployment Methods<\/a><\/li><li><a href=\"#ceph-depolyment-requirements\">Ceph Depolyment Requirements<\/a><\/li><li><a href=\"#prepare-ceph-nodes-for-ceph-storage-cluster-deployment-on-rocky-linux\">Prepare Ceph Nodes for Ceph Storage Cluster Deployment on Rocky Linux<\/a><ul><li><a href=\"#our-ceph-storage-cluster-deployment-architecture\">Our Ceph Storage Cluster Deployment Architecture<\/a><\/li><li><a href=\"#ceph-storage-nodes-hardware-requirements\">Ceph Storage Nodes Hardware Requirements<\/a><\/li><li><a href=\"#storage-devices-for-creating-ceph-os-ds\">Storage Devices for Creating Ceph OSDs<\/a><\/li><\/ul><\/li><li><a href=\"#set-hostnames-and-update-hosts-file\">Set Hostnames and Update Hosts File<\/a><ul><li><a href=\"#set-time-synchronization\">Set Time Synchronization<\/a><\/li><li><a href=\"#install-ssh-server-on-each-node\">Install SSH Server on Each Node<\/a><\/li><li><a href=\"#install-python-3\">Install Python3<\/a><\/li><li><a href=\"#install-docker-ce-on-each-node\">Install Docker CE on Each Node<\/a><\/li><li><a href=\"#enable-root-login-on-other-nodes-from-ceph-admin-node\">Enable Root Login on Other Nodes from Ceph Admin Node<\/a><\/li><\/ul><\/li><li><a href=\"#setup-ceph-storage-cluster-on-rocky-linux\">Setup Ceph Storage Cluster on Rocky Linux<\/a><ul><li><a href=\"#install-cephadm-utility-on-ceph-admin-node\">Install\u00a0cephadm\u00a0Utility on Ceph Admin Node<\/a><\/li><\/ul><\/li><li><a href=\"#initialize-ceph-cluster-monitor-on-ceph-admin-node\">Initialize Ceph Cluster Monitor On Ceph Admin Node<\/a><\/li><li><a href=\"#enable-ceph-cli\">Enable Ceph CLI<\/a><\/li><li><a href=\"#copy-ssh-keys-to-other-ceph-nodes\">Copy SSH Keys to Other Ceph Nodes<\/a><\/li><li><a href=\"#drop-into-ceph-cli\">Drop into Ceph CLI<\/a><\/li><li><a href=\"#add-other-ceph-monitor-nodes-to-ceph-cluster\">Add Other Ceph Monitor Nodes to Ceph Cluster<\/a><\/li><li><a href=\"#add-ceph-osd-nodes-to-ceph-cluster\">Add Ceph OSD Nodes to Ceph Cluster<\/a><\/li><li><a href=\"#list-ceph-cluster-nodes\">List Ceph Cluster Nodes;<\/a><\/li><li><a href=\"#create-ceph-os-ds-from-attached-storage-devices\">Create Ceph OSDs from Attached Storage Devices<\/a><\/li><li><a href=\"#check-ceph-cluster-health\">Check Ceph Cluster Health<\/a><\/li><li><a href=\"#accessing-ceph-admin-web-user-interface\">Accessing Ceph Admin Web User Interface<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"deploying-ceph-storage-cluster-on-rocky-linux\">Deploying Ceph Storage Cluster on Rocky Linux<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"the-ceph-storage-cluster-daemons\">The Ceph Storage Cluster Daemons<\/h3>\n\n\n\n<p>Ceph Storage Cluster&nbsp;is made up of different daemons eas performing specific role.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Ceph Object Storage Daemon (OSD,&nbsp;<\/strong><code><strong>ceph-osd<\/strong><\/code><strong>)<\/strong>\n<ul class=\"wp-block-list\">\n<li>It provides ceph object data store.<\/li>\n\n\n\n<li>It also performs data replication , data recovery, rebalancing and provides storage information to Ceph Monitor.<\/li>\n\n\n\n<li>At least an OSD is required per storage device.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Ceph Monitor (<\/strong><code><strong>ceph-mon<\/strong><\/code><strong>)<\/strong>\n<ul class=\"wp-block-list\">\n<li>It maintains maps of the entire Ceph cluster state including monitor map, manager map, the OSD map, and the CRUSH map.<\/li>\n\n\n\n<li>manages authentication between daemons and clients.<\/li>\n\n\n\n<li>A Ceph cluster must contain a minimum of three running monitors in order to be both redundant and highly-available.&nbsp;<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Ceph Manager<\/strong>&nbsp;(<strong><code>ceph-mgr<\/code><\/strong>)\n<ul class=\"wp-block-list\">\n<li>keeps track of runtime metrics and the current state of the Ceph cluster, including storage utilization, current performance metrics, and system load.<\/li>\n\n\n\n<li>manages and exposes Ceph cluster web dashboard and API.<\/li>\n\n\n\n<li>At least two managers are required for HA.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Ceph Metadata Server (MDS):<\/strong>\n<ul class=\"wp-block-list\">\n<li>Manages metadata for the Ceph File System (CephFS). Coordinates metadata access and ensures consistency across clients.<\/li>\n\n\n\n<li>One or more, depending on the requirements of the CephFS.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>RADOS Gateway (RGW):<\/strong>\n<ul class=\"wp-block-list\">\n<li>Also called \u201cCeph Object Gateway\u201d<\/li>\n\n\n\n<li>is a component of the Ceph storage system that provides object storage services with a RESTful interface. RGW allows applications and users to interact with Ceph storage using industry-standard APIs, such as the S3 (Simple Storage Service) API (compatible with Amazon S3) and the Swift API (compatible with OpenStack Swift).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"ceph-storage-cluster-deployment-methods\">Ceph Storage Cluster Deployment Methods<\/h3>\n\n\n\n<p>There are different methods you can use to deploy Ceph storage cluster.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code><strong>cephadm<\/strong><\/code> leverages container technology (specifically, Docker containers) to deploy and manage Ceph services on a cluster of machines.<\/li>\n\n\n\n<li><strong><code>Rook<\/code><\/strong>&nbsp;deploys and manages Ceph clusters running in Kubernetes, while also enabling management of storage resources and provisioning via Kubernetes APIs.<\/li>\n\n\n\n<li><strong><code>ceph-ansible<\/code><\/strong>&nbsp;deploys and manages Ceph clusters using Ansible.<\/li>\n\n\n\n<li>ceph-salt&nbsp;installs Ceph using Salt and cephadm.<\/li>\n\n\n\n<li>jaas.ai\/ceph-mon&nbsp;installs Ceph using Juju.<\/li>\n\n\n\n<li>Installs Ceph via <a href=\"https:\/\/github.com\/openstack\/puppet-ceph\" target=\"_blank\" rel=\"noreferrer noopener\">Puppet<\/a>.<\/li>\n\n\n\n<li>Ceph can also be&nbsp;<a href=\"https:\/\/docs.ceph.com\/en\/latest\/install\/index_manual\/#install-manual\" target=\"_blank\" rel=\"noopener\">installed manually<\/a>.<\/li>\n<\/ul>\n\n\n\n<p>Use of cephadm and rooks are the recommended methods for deploying Ceph storage cluster.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"ceph-depolyment-requirements\">Ceph Depolyment Requirements<\/h3>\n\n\n\n<p>Depending on the deployment method you choose, there are different requirements for deploying Ceph storage cluster<\/p>\n\n\n\n<p>In this tutorial, we will use cephadm to deploy Ceph storage cluster.<\/p>\n\n\n\n<p>Below are the requirements for deploying Ceph storage cluster via cephadm;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Python 3 (installed by default on Rocky Linux)<\/li>\n\n\n\n<li>Systemd<\/li>\n\n\n\n<li>Podman or <a href=\"#install-docker-ce-on-rocky-linux\">Docker for running containers<\/a> (<strong>we use docker in this setup<\/strong>)<\/li>\n\n\n\n<li><a href=\"https:\/\/kifarunix.com\/deploy-ntp-server-on-rocky-linux-debian-11\/\" target=\"_blank\" rel=\"noreferrer noopener\">Time synchronization (such as chrony or NTP)<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/docs.ceph.com\/en\/latest\/ceph-volume\/lvm\/prepare\/\" target=\"_blank\" rel=\"noreferrer noopener\">LVM2<\/a> for provisioning storage devices. We are using raw devices without any filesystem in this guide.<\/li>\n<\/ul>\n\n\n\n<p>All the required dependencies are installed automatically by the bootstrap process.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"prepare-ceph-nodes-for-ceph-storage-cluster-deployment-on-rocky-linux\">Prepare Ceph Nodes for Ceph Storage Cluster Deployment on Rocky Linux<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"our-ceph-storage-cluster-deployment-architecture\">Our Ceph Storage Cluster Deployment Architecture<\/h4>\n\n\n\n<p>The diagram below depicts our ceph storage cluster deployment architecture. In a typical production environment, you would have at least 3 monitor nodes as well as at least 3 OSDs.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"720\" height=\"696\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-network-diagram-arch.png\" alt=\"\" class=\"wp-image-19354\" title=\"\"><\/figure><\/div>\n\n\n<h4 class=\"wp-block-heading\" id=\"ceph-storage-nodes-hardware-requirements\">Ceph Storage Nodes Hardware Requirements<\/h4>\n\n\n\n<p>Check the <a href=\"https:\/\/docs.ceph.com\/en\/latest\/start\/hardware-recommendations\/\" target=\"_blank\" rel=\"noreferrer noopener\">hardware recommendations page<\/a> for the Ceph storage cluster nodes hardware requirements.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"storage-devices-for-creating-ceph-os-ds\">Storage Devices for Creating Ceph OSDs<\/h4>\n\n\n<div class=\"gb-container gb-container-b2481987\">\n\n<p>Each Ceph OSD node in our architecture above has unallocated raw disk of 50 GB each.<\/p>\n\n<\/div>\n\n\n<pre class=\"wp-block-code\"><code>lsblk<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>NAME        MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS\nvda         252:0    0  50G  0 disk \n\u251c\u2500vda1      252:1    0   1G  0 part \/boot\n\u2514\u2500vda2      252:2    0  49G  0 part \n  \u251c\u2500rl-root 253:0    0  45G  0 lvm  \/\n  \u2514\u2500rl-swap 253:1    0   4G  0 lvm  [SWAP]\nvdb         252:16   0  50G  0 disk \nvdc         252:32   0  50G  0 disk \nvdd         252:48   0  50G  0 disk \n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"set-hostnames-and-update-hosts-file\">Set Hostnames and Update Hosts File<\/h3>\n\n\n\n<p>To begin with, setup up your nodes hostnames;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>hostnamectl set-hostname node01<\/code><\/pre>\n\n\n\n<p><strong>Set the respective hostnames on other nodes.<\/strong><\/p>\n\n\n\n<p>If you are not using DNS for name resolution, then update the hosts file accordingly.<\/p>\n\n\n\n<p>For example, in our setup, each node hosts file should contain the lines below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>less \/etc\/hosts<\/code><\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code>127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4\n::1         localhost localhost.localdomain localhost6 localhost6.localdomain6\n192.168.122.170 node01\n192.168.122.127 node02\n192.168.122.184 node03\n192.168.122.188 node04\n192.168.122.30 node05\n192.168.122.51 node06\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-time-synchronization\">Set Time Synchronization<\/h4>\n\n\n\n<p>Ensure that the time on <strong>all the nodes<\/strong> is synchronized. Thus install Chrony on each and set it up such that all nodes uses the same NTP server. It might already be installed!<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dnf install chrony -y<\/code><\/pre>\n\n\n\n<p>Edit the Chrony configuration and set your NTP server by replacing the NTP server pools with your NTP server address.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/chrony\/chrony.conf<\/code><\/pre>\n\n\n\n<p>Define your NTP Server. Replace <strong>ntp.kifarunix-demo.com<\/strong> with your respective NTP server address. Otherwise, use the global time servers if you don&#8217;t have your own.<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>...\n# pool ntp.ubuntu.com        iburst maxsources 4\n# pool 0.ubuntu.pool.ntp.org iburst maxsources 1\n# pool 1.ubuntu.pool.ntp.org iburst maxsources 1\n# pool 2.ubuntu.pool.ntp.org iburst maxsources 2\n<strong>pool ntp.kifarunix-demo.com iburst<\/strong>\n...\n<\/code><\/pre>\n\n\n\n<p>Restart Chronyd<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl restart chronyd<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-ssh-server-on-each-node\">Install SSH Server on Each Node<\/h4>\n\n\n\n<p>Ceph deployment through cephadm utility requires that an SSH server is installed on all the nodes.<\/p>\n\n\n\n<p>Rocky Linux comes with SSH server already installed. If not, install and start it as follows;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dnf install openssh-server<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl enable --now sshd<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-python-3\">Install Python3<\/h4>\n\n\n\n<p>Python is required to deploy Ceph on Rocky Linux. Python 3 is installed by default on Rocky Linux;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>python -V<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Python 3.9.16<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-docker-ce-on-each-node\">Install Docker CE on Each Node<\/h4>\n\n\n\n<p id=\"install-docker-on-each-osd-nodes\">The cephadm utility is used to bootstrap a Ceph cluster and to manage ceph daemons deployed with systemd and Docker containers.<\/p>\n\n\n\n<p>Thus, <strong>on each Node<\/strong>, install Docker. You can follow the guide below to install Docker on Rocky Linux;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-docker-on-rocky-linux\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Docker on Rocky Linux 8|9<\/a><\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"enable-root-login-on-other-nodes-from-ceph-admin-node\">Enable Root Login on Other Nodes from Ceph Admin Node<\/h4>\n\n\n\n<p>In order to add other nodes to the Ceph cluster using Ceph Admin Node, you will have to use the root user account.<\/p>\n\n\n\n<p>Thus, on the Ceph Monitor, Ceph OSD nodes, enable root login from the Ceph Admin node.<\/p>\n\n\n\n<p>Ensure access to SSH via root is restricted. For now, I will just enable root SSH login.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/ssh\/sshd_config<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>#PermitRootLogin prohibit-password\n<strong>PermitRootLogin yes<\/strong><\/code><\/pre>\n\n\n\n<p>Save and exit the file.<\/p>\n\n\n\n<p>Reload ssh;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl reload sshd<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"setup-ceph-storage-cluster-on-rocky-linux\">Setup Ceph Storage Cluster on Rocky Linux<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-cephadm-utility-on-ceph-admin-node\">Install&nbsp;<code>cephadm<\/code>&nbsp;Utility on Ceph Admin Node<\/h4>\n\n\n\n<p><strong>On the Ceph admin node<\/strong>, you need to install the cephadm utility. In this guide, we are using node01 as ceph admin node.<\/p>\n\n\n\n<p><em><a href=\"https:\/\/docs.ceph.com\/en\/latest\/cephadm\/#cephadm\" target=\"_blank\" rel=\"noopener\">Cephadm<\/a>&nbsp;installs and manages a Ceph cluster using containers and systemd, with tight integration with the CLI and dashboard GUI.<\/em><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><em>cephadm only supports Octopus and newer releases.<\/em><\/li>\n\n\n\n<li><em>cephadm is fully integrated with the new orchestration API and fully supports the new CLI and dashboard features to manage cluster deployment.<\/em><\/li>\n\n\n\n<li><em>cephadm requires container support (podman or docker) and Python 3<\/em>.<\/li>\n<\/ul>\n\n\n\n<p>To install cephadm on Rocky Linux 9;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>CEPH_RELEASE=18.2.0<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -sLO https:\/\/download.ceph.com\/rpm-${CEPH_RELEASE}\/el9\/noarch\/cephadm<\/code><\/pre>\n\n\n\n<p>Make the binary executable;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>chmod +x .\/cephadm<\/code><\/pre>\n\n\n\n<p>Next, execute the commands below to install cephadm for Ceph Reef on Rocky Linux;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>.\/cephadm add-repo --release reef<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>.\/cephadm install<\/code><\/pre>\n\n\n\n<p>Confirm the installation of cephadm command.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>which cephadm<\/code><\/pre>\n\n\n\n<p>You should see the path to the cephadm binary printed, <strong><code>\/usr\/sbin\/cephadm<\/code><\/strong>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"initialize-ceph-cluster-monitor-on-ceph-admin-node\">Initialize Ceph Cluster Monitor On Ceph Admin Node<\/h3>\n\n\n\n<p>Your nodes are now ready to deploy a Ceph storage cluster. As such, it is time to bootstrap the Ceph cluster in order to create the first Ceph monitor daemon on our first node, node01.<\/p>\n\n\n\n<p>Thus, run the command below, substituting the IP address with that of the Ceph admin node, which also doubles up as our ceph monitor node01 in this guide..<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cephadm bootstrap --mon-ip 192.168.122.170<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Verifying podman|docker is present...\nVerifying lvm2 is present...\nVerifying time synchronization is in place...\nUnit chronyd.service is enabled and running\nRepeating the final host check...\ndocker (\/usr\/bin\/docker) is present\nsystemctl is present\nlvcreate is present\nUnit chronyd.service is enabled and running\nHost looks OK\nCluster fsid: b930f228-87e2-11ee-a4e6-52540054ad0e\nVerifying IP 192.168.122.170 port 3300 ...\nVerifying IP 192.168.122.170 port 6789 ...\nMon IP `192.168.122.170` is in CIDR network `192.168.122.0\/24`\nMon IP `192.168.122.170` is in CIDR network `192.168.122.0\/24`\nInternal network (--cluster-network) has not been provided, OSD replication will default to the public_network\nPulling container image quay.io\/ceph\/ceph:v18...\nCeph version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)\nExtracting ceph user uid\/gid from container image...\nCreating initial keys...\nCreating initial monmap...\nCreating mon...\nfirewalld ready\nEnabling firewalld service ceph-mon in current zone...\nWaiting for mon to start...\nWaiting for mon...\nmon is available\nAssimilating anything we can from ceph.conf...\nGenerating new minimal ceph.conf...\nRestarting the monitor...\nSetting mon public_network to 192.168.122.0\/24\nWrote config to \/etc\/ceph\/ceph.conf\nWrote keyring to \/etc\/ceph\/ceph.client.admin.keyring\nCreating mgr...\nVerifying port 9283 ...\nVerifying port 8765 ...\nVerifying port 8443 ...\nfirewalld ready\nEnabling firewalld service ceph in current zone...\nfirewalld ready\nEnabling firewalld port 9283\/tcp in current zone...\nEnabling firewalld port 8765\/tcp in current zone...\nEnabling firewalld port 8443\/tcp in current zone...\nWaiting for mgr to start...\nWaiting for mgr...\nmgr not available, waiting (1\/15)...\nmgr not available, waiting (2\/15)...\nmgr is available\nEnabling cephadm module...\nWaiting for the mgr to restart...\nWaiting for mgr epoch 4...\nmgr epoch 4 is available\nSetting orchestrator backend to cephadm...\nGenerating ssh key...\nWrote public SSH key to \/etc\/ceph\/ceph.pub\nAdding key to root@localhost authorized_keys...\nAdding host node01...\nDeploying mon service with default placement...\nDeploying mgr service with default placement...\nDeploying crash service with default placement...\nDeploying ceph-exporter service with default placement...\nDeploying prometheus service with default placement...\nDeploying grafana service with default placement...\nDeploying node-exporter service with default placement...\nDeploying alertmanager service with default placement...\nEnabling the dashboard module...\nWaiting for the mgr to restart...\nWaiting for mgr epoch 8...\nmgr epoch 8 is available\nGenerating a dashboard self-signed certificate...\nCreating initial admin user...\nFetching dashboard port number...\nfirewalld ready\nCeph Dashboard is now available at:\n\n\t     URL: https:\/\/node01:8443\/\n\t    User: admin\n\tPassword: boyjca2g3a\n\nEnabling client.admin keyring and conf on hosts with \"admin\" label\nSaving cluster configuration to \/var\/lib\/ceph\/b930f228-87e2-11ee-a4e6-52540054ad0e\/config directory\nEnabling autotune for osd_memory_target\nYou can access the Ceph CLI as following in case of multi-cluster or non-default config:\n\n\tsudo \/usr\/sbin\/cephadm shell --fsid b930f228-87e2-11ee-a4e6-52540054ad0e -c \/etc\/ceph\/ceph.conf -k \/etc\/ceph\/ceph.client.admin.keyring\n\nOr, if you are only running a single cluster on this host:\n\n\tsudo \/usr\/sbin\/cephadm shell \n\nPlease consider enabling telemetry to help improve Ceph:\n\n\tceph telemetry on\n\nFor more information see:\n\n\thttps:\/\/docs.ceph.com\/en\/latest\/mgr\/telemetry\/\n\nBootstrap complete.\n<\/code><\/pre>\n\n\n\n<p>According to the <a href=\"https:\/\/docs.ceph.com\/en\/latest\/cephadm\/install\/#running-the-bootstrap-command\" target=\"_blank\" rel=\"noreferrer noopener\">documentation<\/a>, the bootstrap command;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><em>Create a monitor and manager daemon for the new cluster on the localhost.<\/em><\/li>\n\n\n\n<li><em>Generate a new SSH key for the Ceph cluster and add it to the root user\u2019s&nbsp;<code>\/root\/.ssh\/authorized_keys<\/code>&nbsp;file.<\/em><\/li>\n\n\n\n<li><em>Write a copy of the public key to&nbsp;<code>\/etc\/ceph\/ceph.pub<\/code>.<\/em><\/li>\n\n\n\n<li><em>Write a minimal configuration file to&nbsp;<code>\/etc\/ceph\/ceph.conf<\/code>. This file is needed to communicate with the new cluster.<\/em><\/li>\n\n\n\n<li><em>Write a copy of the&nbsp;<code>client.admin<\/code>&nbsp;administrative (privileged!) secret key to&nbsp;<code>\/etc\/ceph\/ceph.client.admin.keyring<\/code>.<\/em><\/li>\n\n\n\n<li><em>Add the&nbsp;<code>_admin<\/code>&nbsp;label to the bootstrap host. By default, any host with this label will (also) get a copy of&nbsp;<code>\/etc\/ceph\/ceph.conf<\/code>&nbsp;and&nbsp;<code>\/etc\/ceph\/ceph.client.admin.keyring<\/code>.<\/em><\/li>\n<\/ul>\n\n\n\n<p>As you can see, we are using Docker container engine, <strong><em>docker (\/usr\/bin\/docker) is present<\/em><\/strong> from the bootstrap command output.<\/p>\n\n\n\n<p>You can see created containers;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker ps<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>CONTAINER ID   IMAGE                                     COMMAND                  CREATED         STATUS         PORTS     NAMES\nb08131564f70   quay.io\/ceph\/ceph-grafana:9.4.7           \"\/bin\/sh -c 'grafana\u2026\"   6 minutes ago   Up 6 minutes             ceph-b930f228-87e2-11ee-a4e6-52540054ad0e-grafana-node01\n53deac7fb865   quay.io\/prometheus\/alertmanager:v0.25.0   \"\/bin\/alertmanager -\u2026\"   6 minutes ago   Up 6 minutes             ceph-b930f228-87e2-11ee-a4e6-52540054ad0e-alertmanager-node01\nc3817b8f196f   quay.io\/prometheus\/prometheus:v2.43.0     \"\/bin\/prometheus --c\u2026\"   6 minutes ago   Up 6 minutes             ceph-b930f228-87e2-11ee-a4e6-52540054ad0e-prometheus-node01\n0e442dd010ab   quay.io\/prometheus\/node-exporter:v1.5.0   \"\/bin\/node_exporter \u2026\"   7 minutes ago   Up 7 minutes             ceph-b930f228-87e2-11ee-a4e6-52540054ad0e-node-exporter-node01\n2e18cdccea4d   quay.io\/ceph\/ceph                         \"\/usr\/bin\/ceph-crash\u2026\"   7 minutes ago   Up 7 minutes             ceph-b930f228-87e2-11ee-a4e6-52540054ad0e-crash-node01\n761bb884af10   quay.io\/ceph\/ceph                         \"\/usr\/bin\/ceph-expor\u2026\"   7 minutes ago   Up 7 minutes             ceph-b930f228-87e2-11ee-a4e6-52540054ad0e-ceph-exporter-node01\nad8e70e8454f   quay.io\/ceph\/ceph:v18                     \"\/usr\/bin\/ceph-mgr -\u2026\"   8 minutes ago   Up 8 minutes             ceph-b930f228-87e2-11ee-a4e6-52540054ad0e-mgr-node01-ywnnwt\ndc0ded459bf9   quay.io\/ceph\/ceph:v18                     \"\/usr\/bin\/ceph-mon -\u2026\"   8 minutes ago   Up 8 minutes             ceph-b930f228-87e2-11ee-a4e6-52540054ad0e-mon-node01\n<\/code><\/pre>\n\n\n\n<p>If you are using Podman, then list the containers using;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>podman ps<\/code><\/pre>\n\n\n\n<p>Similarly, systemd unit files are also created for these containers;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl list-units 'ceph*'<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>  UNIT                                                                   LOAD   ACTIVE SUB     DESCRIPTION                                                       \n  ceph-b930f228-87e2-11ee-a4e6-52540054ad0e@alertmanager.node01.service  loaded active running Ceph alertmanager.node01 for b930f228-87e2-11ee-a4e6-52540054ad0e\n  ceph-b930f228-87e2-11ee-a4e6-52540054ad0e@ceph-exporter.node01.service loaded active running Ceph ceph-exporter.node01 for b930f228-87e2-11ee-a4e6-52540054ad0e\n  ceph-b930f228-87e2-11ee-a4e6-52540054ad0e@crash.node01.service         loaded active running Ceph crash.node01 for b930f228-87e2-11ee-a4e6-52540054ad0e\n  ceph-b930f228-87e2-11ee-a4e6-52540054ad0e@grafana.node01.service       loaded active running Ceph grafana.node01 for b930f228-87e2-11ee-a4e6-52540054ad0e\n  ceph-b930f228-87e2-11ee-a4e6-52540054ad0e@mgr.node01.ywnnwt.service    loaded active running Ceph mgr.node01.ywnnwt for b930f228-87e2-11ee-a4e6-52540054ad0e\n  ceph-b930f228-87e2-11ee-a4e6-52540054ad0e@mon.node01.service           loaded active running Ceph mon.node01 for b930f228-87e2-11ee-a4e6-52540054ad0e\n  ceph-b930f228-87e2-11ee-a4e6-52540054ad0e@node-exporter.node01.service loaded active running Ceph node-exporter.node01 for b930f228-87e2-11ee-a4e6-52540054ad0e\n  ceph-b930f228-87e2-11ee-a4e6-52540054ad0e@prometheus.node01.service    loaded active running Ceph prometheus.node01 for b930f228-87e2-11ee-a4e6-52540054ad0e   \n  ceph-b930f228-87e2-11ee-a4e6-52540054ad0e.target                       loaded active active  Ceph cluster b930f228-87e2-11ee-a4e6-52540054ad0e\n  ceph.target                                                            loaded active active  All Ceph clusters and services\n\nLOAD   = Reflects whether the unit definition was properly loaded.\nACTIVE = The high-level unit activation state, i.e. generalization of SUB.\nSUB    = The low-level unit activation state, values depend on unit type.\n10 loaded units listed. Pass --all to see loaded but inactive units, too.\nTo show all installed unit files use 'systemctl list-unit-files'.\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"enable-ceph-cli\">Enable Ceph CLI<\/h3>\n\n\n\n<p>When bootstrap command completes, a command for accessing Ceph CLI is provided. Execute that command to access Ceph CLI;<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>\/usr\/sbin\/cephadm shell \\\n\t--fsid b930f228-87e2-11ee-a4e6-52540054ad0e \\\n\t-c \/etc\/ceph\/ceph.conf \\\n\t-k \/etc\/ceph\/ceph.client.admin.keyring\n<\/code><\/pre>\n\n\n\n<p>This drops you onto Ceph container CLI;<\/p>\n\n\n\n<p>You can run the ceph commands eg to check the Ceph status;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph -s<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>[ceph: root@node01 \/]# ceph -s\n  cluster:\n    id:     b930f228-87e2-11ee-a4e6-52540054ad0e\n    health: HEALTH_WARN\n            OSD count 0 < osd_pool_default_size 3\n \n  services:\n    mon: 1 daemons, quorum node01 (age 7m)\n    mgr: node01.ywnnwt(active, since 6m)\n    osd: 0 osds: 0 up, 0 in\n \n  data:\n    pools:   0 pools, 0 pgs\n    objects: 0 objects, 0 B\n    usage:   0 B used, 0 B \/ 0 B avail\n    pgs:\n<\/code><\/pre>\n\n\n\n<p>You can exit the ceph CLI by pressing <strong>Ctrl+D<\/strong> or  type <strong>exit<\/strong> and press ENTER.<\/p>\n\n\n\n<p>There are other ways in which you can access the Ceph CLI. For example, you can run Ceph CLI commands using cephadm command.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cephadm shell -- ceph -s<\/code><\/pre>\n\n\n\n<p><em>Or You could install Ceph CLI tools on the host;<\/em><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><em>cephadm add-repo --release <\/em>reef<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code><em>cephadm install ceph-common<\/em><\/code><\/pre>\n\n\n\n<p><em>With this method, then you can just ran the Ceph commands easily;<\/em><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><em>ceph -s<\/em><\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"copy-ssh-keys-to-other-ceph-nodes\">Copy SSH Keys to Other Ceph Nodes<\/h3>\n\n\n\n<p>Copy the SSH key generated by the bootstrap command to all the monitor and OSD nodes root user account. Ensure Root Login is allowed on the Ceph monitor node.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>for i in {02..06}; do sudo ssh-copy-id -f -i \/etc\/ceph\/ceph.pub root@node$i; done<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"drop-into-ceph-cli\">Drop into Ceph CLI<\/h3>\n\n\n\n<p>You can drop into the Ceph CLI to execute the next commands.<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>sudo \/usr\/sbin\/cephadm shell \\\n\t--fsid e5ddd836-87ba-11ee-8755-525400774e0e \\\n\t-c \/etc\/ceph\/ceph.conf \\\n\t-k \/etc\/ceph\/ceph.client.admin.keyring\n<\/code><\/pre>\n\n\n\n<p>Or if you installed the ceph-common package, no need to drop into the cli as you can directly execute the ceph commands from the terminal.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"add-other-ceph-monitor-nodes-to-ceph-cluster\">Add Other Ceph Monitor Nodes to Ceph Cluster<\/h3>\n\n\n\n<p>At this point, we have just provisioned Ceph Admin node only. You can list all the hosts known to the Ceph ochestrator (ceph-mgr) using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host ls<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>HOST        ADDR             LABELS  STATUS  \nceph-admin  192.168.122.170  _admin          \n1 hosts in cluster\n<\/code><\/pre>\n\n\n\n<p>By default, cephadm will deploy 5 monitor daemons on arbitrary hosts. It is recommended to deploy five monitors if there are five or more nodes in your cluster.<\/p>\n\n\n\n<p>To allow deployment of monitor daemons on arbitrary hosts, execute the command below to add the nodes to the cluster;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>for i in {02..03}; do ceph orch host add node$i; done<\/code><\/pre>\n\n\n\n<p>Sample command output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Added host 'node02' with addr '192.168.122.127'\nAdded host 'node03' with addr '192.168.122.184'<\/code><\/pre>\n\n\n\n<p>You should now have three nodes in the cluster.<\/p>\n\n\n\n<p>Next, label the nodes with its role.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host label add node01 mon-mgr-01<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host label add node02 mon-mgr-02<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host label add node03 mon-mgr-03<\/code><\/pre>\n\n\n\n<p>Confirm the labelling;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host ls<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>HOST    ADDR             LABELS             STATUS  \nnode01  192.168.122.170  _admin,mon-mgr-01          \nnode02  192.168.122.127  mon-mgr-02                 \nnode03  192.168.122.184  mon-mgr-03                 \n3 hosts in cluster\n<\/code><\/pre>\n\n\n\n<p>If you want to manually add monitors, then you need to disable automated monitor deployment, run this command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch apply mon --unmanaged<\/code><\/pre>\n\n\n\n<p>To deploy each additional monitor:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch daemon add mon *&lt;host1:ip&gt;<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"add-ceph-osd-nodes-to-ceph-cluster\">Add Ceph OSD Nodes to Ceph Cluster<\/h3>\n\n\n\n<p>Similarly, add the OSD Nodes to the cluster;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host add node04<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host add node05<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host add node06<\/code><\/pre>\n\n\n\n<p>Define their respective labels;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>for i in {04..06}; do ceph orch host label add node$i osd$i; done<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"list-ceph-cluster-nodes\">List Ceph Cluster Nodes;<\/h3>\n\n\n\n<p>You can list the Ceph cluster nodes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host ls<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>HOST    ADDR             LABELS             STATUS  \nnode01  192.168.122.170  _admin,mon-mgr-01          \nnode02  192.168.122.127  mon-mgr-02                 \nnode03  192.168.122.184  mon-mgr-03                 \nnode04  192.168.122.188  osd04                      \nnode05  192.168.122.30   osd05                      \nnode06  192.168.122.51   osd06                      \n6 hosts in cluster\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"create-ceph-os-ds-from-attached-storage-devices\">Create Ceph OSDs from Attached Storage Devices<\/h3>\n\n\n\n<p>In our setup, we have 3 unallocated raw disks of 50G on each OSD node to be used as <a href=\"https:\/\/docs.ceph.com\/en\/reef\/rados\/configuration\/storage-devices\/#bluestore\" target=\"_blank\" rel=\"noreferrer noopener\">bluestore<\/a> for OSD daemons.<\/p>\n\n\n\n<p>You can list the devices that are available on the OSD nodes for creating OSDs using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch device ls<\/code><\/pre>\n\n\n\n<p>A storage device is considered&nbsp;<em>available<\/em>&nbsp;if all of the following conditions are met:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The device must have no partitions.<\/li>\n\n\n\n<li>The device must not have any LVM state.<\/li>\n\n\n\n<li>The device must not be mounted.<\/li>\n\n\n\n<li>The device must not contain a file system.<\/li>\n\n\n\n<li>The device must not contain a Ceph BlueStore OSD.<\/li>\n\n\n\n<li>The device must be larger than 5 GB.<\/li>\n<\/ul>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>HOST    PATH      TYPE  DEVICE ID   SIZE  AVAILABLE  REFRESHED  REJECT REASONS  \nnode04  \/dev\/vdb  hdd              50.0G  Yes        7s ago                     \nnode04  \/dev\/vdc  hdd              50.0G  Yes        7s ago                     \nnode04  \/dev\/vdd  hdd              50.0G  Yes        7s ago                     \nnode05  \/dev\/vdb  hdd              50.0G  Yes        7s ago                     \nnode05  \/dev\/vdc  hdd              50.0G  Yes        7s ago                     \nnode05  \/dev\/vdd  hdd              50.0G  Yes        7s ago                     \nnode06  \/dev\/vdb  hdd              50.0G  Yes        7s ago                     \nnode06  \/dev\/vdc  hdd              50.0G  Yes        7s ago                     \nnode06  \/dev\/vdd  hdd              50.0G  Yes        7s ago \n<\/code><\/pre>\n\n\n\n<p>You can add all the available devices to ceph OSDs at once or just add them one by one.<\/p>\n\n\n\n<p>To attach them all at once;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch apply osd --all-available-devices --method {raw|lvm}<\/code><\/pre>\n\n\n\n<p>Use raw method if you are using raw disks. otherwise, if you are using LVM disks, use lvm method.<\/p>\n\n\n\n<p>Command output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Scheduled osd.all-available-devices update...<\/code><\/pre>\n\n\n\n<p>Note that when you add devices using this approach, then;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If you add new disks to the cluster, they will be automatically used to create new OSDs.<\/li>\n\n\n\n<li>In the event that an OSD is removed, and the LVM physical volume is cleaned, a new OSD will be generated automatically.<\/li>\n<\/ul>\n\n\n\n<p>If you wish to prevent this behavior (i.e., disable the automatic creation of OSDs on available devices), use the 'unmanaged' parameter:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch apply osd --all-available-devices --unmanaged=true<\/code><\/pre>\n\n\n\n<p>To manually create an OSD from a specific device on a specific host:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch daemon add osd &lt;host&gt;:&lt;device-path&gt;<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"check-ceph-cluster-health\">Check Ceph Cluster Health<\/h3>\n\n\n\n<p>To verify the health status of the ceph cluster, simply execute the command&nbsp;<code>ceph s<\/code>&nbsp;on the admin node or even on each OSD node (if you have installed cephadm\/ceph commands there).<\/p>\n\n\n\n<p>To check Ceph cluster health status from the admin node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph -s<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>  cluster:\n    id:     b930f228-87e2-11ee-a4e6-52540054ad0e\n    health: HEALTH_OK\n \n  services:\n    mon: 5 daemons, quorum node01,node02,node03,node06,node05 (age 12m)\n    mgr: node01.ywnnwt(active, since 8h), standbys: node02.tbhoym\n    osd: 9 osds: 9 up (since 8m), 9 in (since 11m)\n \n  data:\n    pools:   1 pools, 1 pgs\n    objects: 2 objects, 449 KiB\n    usage:   640 MiB used, 449 GiB \/ 450 GiB avail\n    pgs:     1 active+clean\n<\/code><\/pre>\n\n\n\n<p>Check OSDs;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph osd tree<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>ID  CLASS  WEIGHT   TYPE NAME        STATUS  REWEIGHT  PRI-AFF\n-1         0.43918  root default                              \n-5         0.14639      host node04                           \n 0    hdd  0.04880          osd.0        up   1.00000  1.00000\n 4    hdd  0.04880          osd.4        up   1.00000  1.00000\n 7    hdd  0.04880          osd.7        up   1.00000  1.00000\n-3         0.14639      host node05                           \n 1    hdd  0.04880          osd.1        up   1.00000  1.00000\n 3    hdd  0.04880          osd.3        up   1.00000  1.00000\n 6    hdd  0.04880          osd.6        up   1.00000  1.00000\n-7         0.14639      host node06                           \n 2    hdd  0.04880          osd.2        up   1.00000  1.00000\n 5    hdd  0.04880          osd.5        up   1.00000  1.00000\n 8    hdd  0.04880          osd.8        up   1.00000  1.00000\n<\/code><\/pre>\n\n\n\n<p>Get a list of Ceph services;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch ps<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                  HOST    PORTS             STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  \nalertmanager.node01   node01  *:9093,9094       running (7h)      3m ago   8h    21.5M        -  0.25.0   c8568f914cd2  8bfdf9400b38  \nceph-exporter.node01  node01                    running (8h)      3m ago   8h    12.9M        -  18.2.0   10237bca3285  761bb884af10  \nceph-exporter.node02  node02                    running (7h)      4m ago   7h    12.3M        -  18.2.0   10237bca3285  60ca13accfae  \nceph-exporter.node03  node03                    running (7h)      4m ago   7h    14.2M        -  18.2.0   10237bca3285  a4af21c6f957  \nceph-exporter.node04  node04                    running (7h)     93s ago   7h    8019k        -  18.2.0   10237bca3285  ea1fd584d83d  \nceph-exporter.node05  node05                    running (7h)     93s ago   7h    13.7M        -  18.2.0   10237bca3285  106d31b6a421  \nceph-exporter.node06  node06                    running (7h)     93s ago   7h    13.7M        -  18.2.0   10237bca3285  270040c7b0e0  \ncrash.node01          node01                    running (8h)      3m ago   8h    7084k        -  18.2.0   10237bca3285  2e18cdccea4d  \ncrash.node02          node02                    running (7h)      4m ago   7h    7088k        -  18.2.0   10237bca3285  db1a922bbe40  \ncrash.node03          node03                    running (7h)      4m ago   7h    7092k        -  18.2.0   10237bca3285  bfbd319fe511  \ncrash.node04          node04                    running (7h)     93s ago   7h    7079k        -  18.2.0   10237bca3285  cb51b3b7de7b  \ncrash.node05          node05                    running (7h)     93s ago   7h    7092k        -  18.2.0   10237bca3285  de2648d5050c  \ncrash.node06          node06                    running (7h)     93s ago   7h    7088k        -  18.2.0   10237bca3285  b3c756c72120  \ngrafana.node01        node01  *:3000            running (8h)      3m ago   8h    91.0M        -  9.4.7    2c41d148cca3  b08131564f70  \nmgr.node01.ywnnwt     node01  *:9283,8765,8443  running (8h)      3m ago   8h     536M        -  18.2.0   10237bca3285  ad8e70e8454f  \nmgr.node02.tbhoym     node02  *:8443,9283,8765  running (7h)      4m ago   7h     432M        -  18.2.0   10237bca3285  96f9acbc47c5  \nmon.node01            node01                    running (8h)      3m ago   8h     187M    2048M  18.2.0   10237bca3285  dc0ded459bf9  \nmon.node02            node02                    running (7h)      4m ago   7h     164M    2048M  18.2.0   10237bca3285  b4a0267b3afd  \nmon.node03            node03                    running (7h)      4m ago   7h     164M    2048M  18.2.0   10237bca3285  f1808fe3d37a  \nmon.node05            node05                    running (7h)     93s ago   7h     164M    2048M  18.2.0   10237bca3285  43debc6ae77a  \nmon.node06            node06                    running (7h)     93s ago   7h     162M    2048M  18.2.0   10237bca3285  8b7e0534ae3c  \nnode-exporter.node01  node01  *:9100            running (8h)      3m ago   8h    16.4M        -  1.5.0    0da6a335fe13  0e442dd010ab  \nnode-exporter.node02  node02  *:9100            running (7h)      4m ago   7h    17.0M        -  1.5.0    0da6a335fe13  05a8d80be5eb  \nnode-exporter.node03  node03  *:9100            running (7h)      4m ago   7h    18.1M        -  1.5.0    0da6a335fe13  3fbaddfa22ce  \nnode-exporter.node04  node04  *:9100            running (7h)     93s ago   7h    17.1M        -  1.5.0    0da6a335fe13  08b082b601ed  \nnode-exporter.node05  node05  *:9100            running (7h)     93s ago   7h    16.1M        -  1.5.0    0da6a335fe13  abf7791cf58e  \nnode-exporter.node06  node06  *:9100            running (7h)     93s ago   7h    16.6M        -  1.5.0    0da6a335fe13  8728079b9664  \nosd.0                 node04                    running (14m)    93s ago  14m    63.1M    4096M  18.2.0   10237bca3285  98e7571ae26a  \nosd.1                 node05                    running (14m)    93s ago  14m    63.8M    4096M  18.2.0   10237bca3285  f7327f928fbb  \nosd.2                 node06                    running (14m)    93s ago  14m    62.1M    4096M  18.2.0   10237bca3285  4407e271ffe9  \nosd.3                 node05                    running (14m)    93s ago  14m    62.2M    4096M  18.2.0   10237bca3285  c59832aabc61  \nosd.4                 node04                    running (14m)    93s ago  14m    59.1M    4096M  18.2.0   10237bca3285  c664ea7b95f5  \nosd.5                 node06                    running (14m)    93s ago  14m    61.8M    4096M  18.2.0   10237bca3285  728a91483f66  \nosd.6                 node05                    running (14m)    93s ago  14m    59.4M    4096M  18.2.0   10237bca3285  5085843764a1  \nosd.7                 node04                    running (14m)    93s ago  14m    64.6M    4096M  18.2.0   10237bca3285  4766da3ce4c6  \nosd.8                 node06                    running (14m)    93s ago  14m    61.9M    4096M  18.2.0   10237bca3285  08398a2a6c26  \nprometheus.node01     node01  *:9095            running (7h)      3m ago   8h    95.5M        -  2.43.0   a07b618ecd1d  3cce2ffb6aba\n<\/code><\/pre>\n\n\n\n<p>Similarly, you can check Ceph service containers created on other nodes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker ps<\/code><\/pre>\n\n\n\n<p>Or list their systemd unit files;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl list-units 'ceph*'<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"accessing-ceph-admin-web-user-interface\">Accessing Ceph Admin Web User Interface<\/h3>\n\n\n\n<p>The bootstrap commands give a url and credentials to use to access the Ceph admin web user interface;<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>Ceph Dashboard is now available at:\n\n\t     URL: https:\/\/node01:8443\/\n\t    User: admin\n\tPassword: boyjca2g3a\n<\/code><\/pre>\n\n\n\n<p>Since we are running multiple nodes, Ceph manager is deployed on two nodes for HA. From the Ceph status command, you can see under services, that one manager node is active while the other one is on standby.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>  services:\n    mon: 5 daemons, quorum node01,node02,node03,node06,node05 (age 12m)\n    <strong>mgr: node01.ywnnwt(active, since 8h), standbys: node02.tbhoym\n<\/strong>    osd: 9 osds: 9 up (since 8m), 9 in (since 11m)<\/code><\/pre>\n\n\n\n<p>Thus, open the browser and navigate to the URL above. Or you can even use the cephadm node IP address if you want..<\/p>\n\n\n\n<p>Open the port on firewall if firewall is running.<\/p>\n\n\n\n<p>Enter the provided credential and reset your admin password and proceed to login to Ceph Admin dashboard.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1392\" height=\"793\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-user-interface.png?v=1700544752\" alt=\"How to Deploy Ceph Storage Cluster on Rocky Linux\" class=\"wp-image-19349\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-user-interface.png?v=1700544752 1392w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-user-interface-768x438.png?v=1700544752 768w\" sizes=\"(max-width: 1392px) 100vw, 1392px\" \/><\/figure>\n\n\n\n<p>If you want, you can activate the <a href=\"https:\/\/docs.ceph.com\/en\/latest\/mgr\/telemetry\/#telemetry-module\" target=\"_blank\" rel=\"noreferrer noopener\">telemetry module<\/a> by clicking <strong>Activate<\/strong> button or just from the CLI;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph telemetry on --license sharing-1-0<\/code><\/pre>\n\n\n\n<p>Telemetry can be disabled at any time with:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph telemetry off<\/code><\/pre>\n\n\n\n<p>Go through other Ceph menu to see more about Ceph.<\/p>\n\n\n\n<p>Ceph Dashboard;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1651\" height=\"905\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-dashboard.png?v=1700545015\" alt=\"\" class=\"wp-image-19350\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-dashboard.png?v=1700545015 1651w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-dashboard-768x421.png?v=1700545015 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-dashboard-1536x842.png?v=1700545015 1536w\" sizes=\"(max-width: 1651px) 100vw, 1651px\" \/><\/figure>\n\n\n\n<p>Under the Cluster Menu, you can see quite more details; hosts, disks, OSDs etc.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1648\" height=\"783\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-nodes.png?v=1700545275\" alt=\"\" class=\"wp-image-19351\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-nodes.png?v=1700545275 1648w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-nodes-768x365.png?v=1700545275 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-nodes-1536x730.png?v=1700545275 1536w\" sizes=\"(max-width: 1648px) 100vw, 1648px\" \/><\/figure>\n\n\n\n<p>There you go. That marks the end of our tutorial on deploying Ceph storage cluster on Rocky Linux.<\/p>\n\n\n\n<p>Reference<\/p>\n\n\n\n<p><a href=\"https:\/\/docs.ceph.com\/en\/latest\/cephadm\/install\/#deploying-a-new-ceph-cluster\" target=\"_blank\" rel=\"noreferrer noopener\">Deploying a new ceph cluster<\/a><\/p>\n\n\n\n<p>Other Tutorials<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/configure-and-use-ceph-block-device-on-linux-clients\/\" target=\"_blank\" rel=\"noreferrer noopener\">Configure and Use Ceph Block Device on Linux Clients<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/how-to-reset-ceph-dashboard-admin-password\/\" target=\"_blank\" rel=\"noreferrer noopener\">How to Reset Ceph Dashboard Admin Password<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Follow through this post to learn how to deploy Ceph storage cluster on Rocky Linux. Ceph is a scalable distributed storage system designed for cloud<\/p>\n","protected":false},"author":10,"featured_media":19284,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121,1338,39],"tags":[7314,7316,7315],"class_list":["post-19326","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","category-ceph","category-storage","tag-ceph-stoirage-cluster-on-rocky-linux","tag-ceph-storage-rocky-linux","tag-rocky-linux-ceph-cluster","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19326"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=19326"}],"version-history":[{"count":9,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19326\/revisions"}],"predecessor-version":[{"id":20874,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19326\/revisions\/20874"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/19284"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=19326"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=19326"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=19326"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}