{"id":19356,"date":"2023-11-21T23:24:36","date_gmt":"2023-11-21T20:24:36","guid":{"rendered":"https:\/\/kifarunix.com\/?p=19356"},"modified":"2024-03-10T11:49:23","modified_gmt":"2024-03-10T08:49:23","slug":"how-to-deploy-ceph-storage-cluster-on-almalinux","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/how-to-deploy-ceph-storage-cluster-on-almalinux\/","title":{"rendered":"How to Deploy Ceph Storage Cluster on AlmaLinux"},"content":{"rendered":"\n<p>Follow through this post to learn how to deploy Ceph storage cluster on AlmaLinux. Ceph is a scalable distributed storage system designed for cloud infrastructure and web-scale&nbsp;<a href=\"https:\/\/docs.ceph.com\/docs\/master\/glossary\/#term-ceph-object-storage\" target=\"_blank\" rel=\"noreferrer noopener\">object storage<\/a>. It can also be used to provide&nbsp;<a href=\"https:\/\/docs.ceph.com\/docs\/master\/glossary\/#term-ceph-block-device\" target=\"_blank\" rel=\"noreferrer noopener\">Ceph Block Storage<\/a>&nbsp;as well as&nbsp;<a href=\"https:\/\/docs.ceph.com\/docs\/master\/glossary\/#term-ceph-file-system\" target=\"_blank\" rel=\"noreferrer noopener\">Ceph File System storage<\/a>.<\/p>\n\n\n\n<p>As of this writing, Ceph 18 (code named Reef) is the current stable <a href=\"https:\/\/docs.ceph.com\/en\/latest\/releases\/\" target=\"_blank\" rel=\"noreferrer noopener\">release<\/a>.<\/p>\n\n\n\n<p>We are using AlmaLinux 9.3 (Shamrock Pampas Cat) in this guide.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#deploying-ceph-storage-cluster-on-alma-linux\">Deploying Ceph Storage Cluster on AlmaLinux<\/a><ul><li><a href=\"#the-ceph-storage-cluster-daemons\">The Ceph Storage Cluster Daemons<\/a><\/li><li><a href=\"#ceph-storage-cluster-deployment-methods\">Ceph Storage Cluster Deployment Methods<\/a><\/li><li><a href=\"#ceph-depolyment-requirements\">Ceph Depolyment Requirements<\/a><\/li><li><a href=\"#prepare-ceph-nodes-for-ceph-storage-cluster-deployment-on-alma-linux\">Prepare Ceph Nodes for Ceph Storage Cluster Deployment on AlmaLinux<\/a><ul><li><a href=\"#our-ceph-storage-cluster-deployment-architecture\">Our Ceph Storage Cluster Deployment Architecture<\/a><\/li><li><a href=\"#ceph-storage-nodes-hardware-requirements\">Ceph Storage Nodes Hardware Requirements<\/a><\/li><li><a href=\"#storage-devices-for-creating-ceph-os-ds\">Storage Devices for Creating Ceph OSDs<\/a><\/li><\/ul><\/li><li><a href=\"#set-hostnames-and-update-hosts-file\">Set Hostnames and Update Hosts File<\/a><ul><li><a href=\"#set-time-synchronization\">Set Time Synchronization<\/a><\/li><li><a href=\"#install-ssh-server-on-each-node\">Install SSH Server on Each Node<\/a><\/li><li><a href=\"#install-python-3\">Install Python3<\/a><\/li><li><a href=\"#install-lvm-2-on-alma-linux\">Install LVM2 on AlmaLinux<\/a><\/li><li><a href=\"#install-docker-ce-on-each-node\">Install Docker CE on Each Node<\/a><\/li><li><a href=\"#enable-root-login-on-other-nodes-from-ceph-admin-node\">Enable Root Login on Other Nodes from Ceph Admin Node<\/a><\/li><\/ul><\/li><li><a href=\"#setup-ceph-storage-cluster-on-alma-linux\">Setup Ceph Storage Cluster on AlmaLinux<\/a><ul><li><a href=\"#install-cephadm-utility-on-ceph-admin-node\">Install\u00a0cephadm\u00a0Utility on Ceph Admin Node<\/a><\/li><\/ul><\/li><li><a href=\"#initialize-ceph-cluster-monitor-on-ceph-admin-node\">Initialize Ceph Cluster Monitor On Ceph Admin Node<\/a><\/li><li><a href=\"#enable-ceph-cli\">Enable Ceph CLI<\/a><\/li><li><a href=\"#copy-ssh-keys-to-other-ceph-nodes\">Copy SSH Keys to Other Ceph Nodes<\/a><\/li><li><a href=\"#drop-into-ceph-cli\">Drop into Ceph CLI<\/a><\/li><li><a href=\"#add-other-ceph-monitor-nodes-to-ceph-cluster\">Add Other Ceph Monitor Nodes to Ceph Cluster<\/a><\/li><li><a href=\"#add-ceph-osd-nodes-to-ceph-cluster\">Add Ceph OSD Nodes to Ceph Cluster<\/a><\/li><li><a href=\"#list-ceph-cluster-nodes\">List Ceph Cluster Nodes;<\/a><\/li><li><a href=\"#create-ceph-os-ds-from-attached-storage-devices\">Create Ceph OSDs from Attached Storage Devices<\/a><\/li><li><a href=\"#check-ceph-cluster-health\">Check Ceph Cluster Health<\/a><\/li><li><a href=\"#accessing-ceph-admin-web-user-interface\">Accessing Ceph Admin Web User Interface<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"deploying-ceph-storage-cluster-on-alma-linux\">Deploying Ceph Storage Cluster on AlmaLinux<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"the-ceph-storage-cluster-daemons\">The Ceph Storage Cluster Daemons<\/h3>\n\n\n\n<p>Ceph Storage Cluster&nbsp;is made up of different daemons eas performing specific role.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Ceph Object Storage Daemon (OSD,&nbsp;<\/strong><code><strong>ceph-osd<\/strong><\/code><strong>)<\/strong>\n<ul class=\"wp-block-list\">\n<li>It provides ceph object data store.<\/li>\n\n\n\n<li>It also performs data replication , data recovery, rebalancing and provides storage information to Ceph Monitor.<\/li>\n\n\n\n<li>At least an OSD is required per storage device.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Ceph Monitor (<\/strong><code><strong>ceph-mon<\/strong><\/code><strong>)<\/strong>\n<ul class=\"wp-block-list\">\n<li>It maintains maps of the entire Ceph cluster state including monitor map, manager map, the OSD map, and the CRUSH map.<\/li>\n\n\n\n<li>manages authentication between daemons and clients.<\/li>\n\n\n\n<li>A Ceph cluster must contain a minimum of three running monitors in order to be both redundant and highly-available.&nbsp;<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Ceph Manager<\/strong>&nbsp;(<strong><code>ceph-mgr<\/code><\/strong>)\n<ul class=\"wp-block-list\">\n<li>keeps track of runtime metrics and the current state of the Ceph cluster, including storage utilization, current performance metrics, and system load.<\/li>\n\n\n\n<li>manages and exposes Ceph cluster web dashboard and API.<\/li>\n\n\n\n<li>At least two managers are required for HA.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Ceph Metadata Server (MDS):<\/strong>\n<ul class=\"wp-block-list\">\n<li>Manages metadata for the Ceph File System (CephFS). Coordinates metadata access and ensures consistency across clients.<\/li>\n\n\n\n<li>One or more, depending on the requirements of the CephFS.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>RADOS Gateway (RGW):<\/strong>\n<ul class=\"wp-block-list\">\n<li>Also called \u201cCeph Object Gateway\u201d<\/li>\n\n\n\n<li>is a component of the Ceph storage system that provides object storage services with a RESTful interface. RGW allows applications and users to interact with Ceph storage using industry-standard APIs, such as the S3 (Simple Storage Service) API (compatible with Amazon S3) and the Swift API (compatible with OpenStack Swift).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"ceph-storage-cluster-deployment-methods\">Ceph Storage Cluster Deployment Methods<\/h3>\n\n\n\n<p>There are different methods you can use to deploy Ceph storage cluster.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code><strong>cephadm<\/strong><\/code> leverages container technology (specifically, Docker containers) to deploy and manage Ceph services on a cluster of machines.<\/li>\n\n\n\n<li><strong><code>Rook<\/code><\/strong>&nbsp;deploys and manages Ceph clusters running in Kubernetes, while also enabling management of storage resources and provisioning via Kubernetes APIs.<\/li>\n\n\n\n<li><strong><code>ceph-ansible<\/code><\/strong>&nbsp;deploys and manages Ceph clusters using Ansible.<\/li>\n\n\n\n<li>ceph-salt&nbsp;installs Ceph using Salt and cephadm.<\/li>\n\n\n\n<li>jaas.ai\/ceph-mon&nbsp;installs Ceph using Juju.<\/li>\n\n\n\n<li>Install Ceph via <a href=\"https:\/\/github.com\/openstack\/puppet-ceph\" target=\"_blank\" rel=\"noreferrer noopener\">Puppet<\/a>.<\/li>\n\n\n\n<li>Ceph can also be&nbsp;<a href=\"https:\/\/docs.ceph.com\/en\/latest\/install\/index_manual\/#install-manual\" target=\"_blank\" rel=\"noopener\">installed manually<\/a>.<\/li>\n<\/ul>\n\n\n\n<p>Use of cephadm and rooks are the recommended methods for deploying Ceph storage cluster.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"ceph-depolyment-requirements\">Ceph Depolyment Requirements<\/h3>\n\n\n\n<p>Depending on the deployment method you choose, there are different requirements for deploying Ceph storage cluster<\/p>\n\n\n\n<p>In this tutorial, we will use cephadm to deploy Ceph storage cluster.<\/p>\n\n\n\n<p>Below are the requirements for deploying Ceph storage cluster via cephadm;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Python 3 (installed by default on AlmaLinux)<\/li>\n\n\n\n<li>Systemd<\/li>\n\n\n\n<li>Podman or <a href=\"#install-docker-ce-on-almalinux\">Docker for running containers<\/a> (<strong>we use docker in this setup<\/strong>)<\/li>\n\n\n\n<li><a href=\"https:\/\/kifarunix.com\/deploy-ntp-server-on-almalinux-debian-11\/\" target=\"_blank\" rel=\"noreferrer noopener\">Time synchronization (such as chrony or NTP)<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/docs.ceph.com\/en\/latest\/ceph-volume\/lvm\/prepare\/\" target=\"_blank\" rel=\"noreferrer noopener\">LVM2<\/a> for provisioning storage devices. We are using raw devices without any filesystem in this guide.<\/li>\n<\/ul>\n\n\n\n<p>All the required dependencies are installed automatically by the bootstrap process.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"prepare-ceph-nodes-for-ceph-storage-cluster-deployment-on-alma-linux\">Prepare Ceph Nodes for Ceph Storage Cluster Deployment on AlmaLinux<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"our-ceph-storage-cluster-deployment-architecture\">Our Ceph Storage Cluster Deployment Architecture<\/h4>\n\n\n\n<p>The diagram below depicts our ceph storage cluster deployment architecture. In a typical production environment, you would have at least 3 monitor nodes as well as at least 3 OSDs.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"747\" height=\"718\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-deployment-arch.png\" alt=\"\" class=\"wp-image-19365\" title=\"\"><\/figure><\/div>\n\n\n<h4 class=\"wp-block-heading\" id=\"ceph-storage-nodes-hardware-requirements\">Ceph Storage Nodes Hardware Requirements<\/h4>\n\n\n\n<p>Check the <a href=\"https:\/\/docs.ceph.com\/en\/latest\/start\/hardware-recommendations\/\" target=\"_blank\" rel=\"noreferrer noopener\">hardware recommendations page<\/a> for the Ceph storage cluster nodes hardware requirements.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"storage-devices-for-creating-ceph-os-ds\">Storage Devices for Creating Ceph OSDs<\/h4>\n\n\n<div class=\"gb-container gb-container-b2481987\">\n\n<p>Each Ceph OSD node in our architecture above has unallocated raw disk of 50 GB each.<\/p>\n\n<\/div>\n\n\n<pre class=\"wp-block-code\"><code>lsblk<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>NAME        MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS\nvda         252:0    0  50G  0 disk \n\u251c\u2500vda1      252:1    0   1G  0 part \/boot\n\u2514\u2500vda2      252:2    0  49G  0 part \n  \u251c\u2500rl-root 253:0    0  45G  0 lvm  \/\n  \u2514\u2500rl-swap 253:1    0   4G  0 lvm  [SWAP]\nvdb         252:16   0  50G  0 disk \nvdc         252:32   0  50G  0 disk \nvdd         252:48   0  50G  0 disk \n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"set-hostnames-and-update-hosts-file\">Set Hostnames and Update Hosts File<\/h3>\n\n\n\n<p>To begin with, setup up your nodes hostnames;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>hostnamectl set-hostname node01<\/code><\/pre>\n\n\n\n<p><strong>Set the respective hostnames on other nodes.<\/strong><\/p>\n\n\n\n<p>If you are not using DNS for name resolution, then update the hosts file accordingly.<\/p>\n\n\n\n<p>For example, in our setup, each node hosts file should contain the lines below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>less \/etc\/hosts<\/code><\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code>127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4\n::1         localhost localhost.localdomain localhost6 localhost6.localdomain6\n192.168.122.240 node01\n192.168.122.55 node02\n192.168.122.229 node03\n192.168.122.152 node04\n192.168.122.241 node05\n192.168.122.116 node06\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-time-synchronization\">Set Time Synchronization<\/h4>\n\n\n\n<p>Ensure that the time on <strong>all the nodes<\/strong> is synchronized. Thus install Chrony on each and set it up such that all nodes uses the same NTP server. It might already be installed!<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dnf install chrony -y<\/code><\/pre>\n\n\n\n<p>Edit the Chrony configuration and set your NTP server by replacing the NTP server pools with your NTP server address.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/chrony\/chrony.conf<\/code><\/pre>\n\n\n\n<p>Define your NTP Server. Replace <strong>ntp.kifarunix-demo.com<\/strong> with your respective NTP server address. Otherwise, use the global time servers if you don&#8217;t have your own.<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>...\n# pool ntp.ubuntu.com        iburst maxsources 4\n# pool 0.ubuntu.pool.ntp.org iburst maxsources 1\n# pool 1.ubuntu.pool.ntp.org iburst maxsources 1\n# pool 2.ubuntu.pool.ntp.org iburst maxsources 2\n<strong>pool ntp.kifarunix-demo.com iburst<\/strong>\n...\n<\/code><\/pre>\n\n\n\n<p>Restart Chronyd<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl restart chronyd<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-ssh-server-on-each-node\">Install SSH Server on Each Node<\/h4>\n\n\n\n<p>Ceph deployment through cephadm utility requires that an SSH server is installed on all the nodes.<\/p>\n\n\n\n<p>AlmaLinux comes with SSH server already installed. If not, install and start it as follows;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dnf install openssh-server<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl enable --now sshd<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-python-3\">Install Python3<\/h4>\n\n\n\n<p>Python is required to deploy Ceph on AlmaLinux. Python 3 is installed by default on AlmaLinux;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>python -V<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Python 3.9.18<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-lvm-2-on-alma-linux\">Install LVM2 on AlmaLinux<\/h4>\n\n\n\n<p>LVM2 is required for Ceph to manage OSDs (Object Storage Devices) on the host.<\/p>\n\n\n\n<p>You can install using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dnf install lvm2 -y<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-docker-ce-on-each-node\">Install Docker CE on Each Node<\/h4>\n\n\n\n<p id=\"install-docker-on-each-osd-nodes\">The cephadm utility is used to bootstrap a Ceph cluster and to manage ceph daemons deployed with systemd and Docker containers.<\/p>\n\n\n\n<p>Thus, <strong>on each Node<\/strong>, install Docker. You can follow the guide below to install Docker on AlmaLinux;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-docker-on-rocky-linux\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Docker on Rocky Linux 8|9<\/a><\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"enable-root-login-on-other-nodes-from-ceph-admin-node\">Enable Root Login on Other Nodes from Ceph Admin Node<\/h4>\n\n\n\n<p>In order to add other nodes to the Ceph cluster using Ceph Admin Node, you will have to use the root user account.<\/p>\n\n\n\n<p>Thus, on the Ceph Monitor, Ceph OSD nodes, enable root login from the Ceph Admin node.<\/p>\n\n\n\n<p>Ensure access to SSH via root is restricted. For now, I will just enable root SSH login.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/ssh\/sshd_config<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>#PermitRootLogin prohibit-password\n<strong>PermitRootLogin yes<\/strong><\/code><\/pre>\n\n\n\n<p>Save and exit the file.<\/p>\n\n\n\n<p>Reload ssh;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl reload sshd<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"setup-ceph-storage-cluster-on-alma-linux\">Setup Ceph Storage Cluster on AlmaLinux<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-cephadm-utility-on-ceph-admin-node\">Install&nbsp;<code>cephadm<\/code>&nbsp;Utility on Ceph Admin Node<\/h4>\n\n\n\n<p><strong>On the Ceph admin node<\/strong>, you need to install the cephadm utility. In this guide, we are using node01 as ceph admin node.<\/p>\n\n\n\n<p><em><a href=\"https:\/\/docs.ceph.com\/en\/latest\/cephadm\/#cephadm\" target=\"_blank\" rel=\"noopener\">Cephadm<\/a>&nbsp;installs and manages a Ceph cluster using containers and systemd, with tight integration with the CLI and dashboard GUI.<\/em><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><em>cephadm only supports Octopus and newer releases.<\/em><\/li>\n\n\n\n<li><em>cephadm is fully integrated with the new orchestration API and fully supports the new CLI and dashboard features to manage cluster deployment.<\/em><\/li>\n\n\n\n<li><em>cephadm requires container support (podman or docker) and Python 3<\/em>.<\/li>\n<\/ul>\n\n\n\n<p>To install cephadm on AlmaLinux 9;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>CEPH_RELEASE=18.2.0<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -sLO https:\/\/download.ceph.com\/rpm-${CEPH_RELEASE}\/el9\/noarch\/cephadm<\/code><\/pre>\n\n\n\n<p>Make the binary executable;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>chmod +x .\/cephadm<\/code><\/pre>\n\n\n\n<p>Next, execute the commands below to install cephadm for Ceph Reef on AlmaLinux;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>.\/cephadm add-repo --release reef<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>.\/cephadm install<\/code><\/pre>\n\n\n\n<p>Confirm the installation of cephadm command.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>which cephadm<\/code><\/pre>\n\n\n\n<p>You should see the path to the cephadm binary printed, <strong><code>\/usr\/sbin\/cephadm<\/code><\/strong>.<\/p>\n\n\n\n<p>The command will also install <strong>podman<\/strong>. So whichever it will find first between it and Docker, it will use it.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"initialize-ceph-cluster-monitor-on-ceph-admin-node\">Initialize Ceph Cluster Monitor On Ceph Admin Node<\/h3>\n\n\n\n<p>Your nodes are now ready to deploy a Ceph storage cluster. As such, it is time to bootstrap the Ceph cluster in order to create the first Ceph monitor daemon on our first node, node01.<\/p>\n\n\n\n<p>Thus, run the command below, substituting the IP address with that of the Ceph admin node, which also doubles up as our ceph monitor node01 in this guide..<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cephadm bootstrap --mon-ip 192.168.122.240<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Creating directory \/etc\/ceph for ceph.conf\nVerifying podman|docker is present...\nVerifying lvm2 is present...\nVerifying time synchronization is in place...\nUnit chronyd.service is enabled and running\nRepeating the final host check...\npodman (\/usr\/bin\/podman) version 4.6.1 is present\nsystemctl is present\nlvcreate is present\nUnit chronyd.service is enabled and running\nHost looks OK\nCluster fsid: f14cb896-889a-11ee-abf4-525400ac8730\nVerifying IP 192.168.122.240 port 3300 ...\nVerifying IP 192.168.122.240 port 6789 ...\nMon IP `192.168.122.240` is in CIDR network `192.168.122.0\/24`\nMon IP `192.168.122.240` is in CIDR network `192.168.122.0\/24`\nInternal network (--cluster-network) has not been provided, OSD replication will default to the public_network\nPulling container image quay.io\/ceph\/ceph:v18...\nCeph version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)\nExtracting ceph user uid\/gid from container image...\nCreating initial keys...\nCreating initial monmap...\nCreating mon...\nWaiting for mon to start...\nWaiting for mon...\nmon is available\nAssimilating anything we can from ceph.conf...\nGenerating new minimal ceph.conf...\nRestarting the monitor...\nSetting mon public_network to 192.168.122.0\/24\nWrote config to \/etc\/ceph\/ceph.conf\nWrote keyring to \/etc\/ceph\/ceph.client.admin.keyring\nCreating mgr...\nVerifying port 9283 ...\nVerifying port 8765 ...\nVerifying port 8443 ...\nWaiting for mgr to start...\nWaiting for mgr...\nmgr not available, waiting (1\/15)...\nmgr not available, waiting (2\/15)...\nmgr not available, waiting (3\/15)...\nmgr not available, waiting (4\/15)...\nmgr is available\nEnabling cephadm module...\nWaiting for the mgr to restart...\nWaiting for mgr epoch 5...\nmgr epoch 5 is available\nSetting orchestrator backend to cephadm...\nGenerating ssh key...\nWrote public SSH key to \/etc\/ceph\/ceph.pub\nAdding key to root@localhost authorized_keys...\nAdding host node01...\nDeploying mon service with default placement...\nDeploying mgr service with default placement...\nDeploying crash service with default placement...\nDeploying ceph-exporter service with default placement...\nDeploying prometheus service with default placement...\nDeploying grafana service with default placement...\nDeploying node-exporter service with default placement...\nDeploying alertmanager service with default placement...\nEnabling the dashboard module...\nWaiting for the mgr to restart...\nWaiting for mgr epoch 9...\nmgr epoch 9 is available\nGenerating a dashboard self-signed certificate...\nCreating initial admin user...\nFetching dashboard port number...\nCeph Dashboard is now available at:\n\n\t     URL: https:\/\/node01:8443\/\n\t    User: admin\n\tPassword: amw3p3yy2e\n\nEnabling client.admin keyring and conf on hosts with \"admin\" label\nSaving cluster configuration to \/var\/lib\/ceph\/f14cb896-889a-11ee-abf4-525400ac8730\/config directory\nEnabling autotune for osd_memory_target\nYou can access the Ceph CLI as following in case of multi-cluster or non-default config:\n\n\tsudo \/usr\/sbin\/cephadm shell --fsid f14cb896-889a-11ee-abf4-525400ac8730 -c \/etc\/ceph\/ceph.conf -k \/etc\/ceph\/ceph.client.admin.keyring\n\nOr, if you are only running a single cluster on this host:\n\n\tsudo \/usr\/sbin\/cephadm shell \n\nPlease consider enabling telemetry to help improve Ceph:\n\n\tceph telemetry on\n\nFor more information see:\n\n\thttps:\/\/docs.ceph.com\/en\/latest\/mgr\/telemetry\/\n<\/code><\/pre>\n\n\n\n<p>According to the <a href=\"https:\/\/docs.ceph.com\/en\/latest\/cephadm\/install\/#running-the-bootstrap-command\" target=\"_blank\" rel=\"noreferrer noopener\">documentation<\/a>, the bootstrap command;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><em>Create a monitor and manager daemon for the new cluster on the localhost.<\/em><\/li>\n\n\n\n<li><em>Generate a new SSH key for the Ceph cluster and add it to the root user\u2019s&nbsp;<code>\/root\/.ssh\/authorized_keys<\/code>&nbsp;file.<\/em><\/li>\n\n\n\n<li><em>Write a copy of the public key to&nbsp;<code>\/etc\/ceph\/ceph.pub<\/code>.<\/em><\/li>\n\n\n\n<li><em>Write a minimal configuration file to&nbsp;<code>\/etc\/ceph\/ceph.conf<\/code>. This file is needed to communicate with the new cluster.<\/em><\/li>\n\n\n\n<li><em>Write a copy of the&nbsp;<code>client.admin<\/code>&nbsp;administrative (privileged!) secret key to&nbsp;<code>\/etc\/ceph\/ceph.client.admin.keyring<\/code>.<\/em><\/li>\n\n\n\n<li><em>Add the&nbsp;<code>_admin<\/code>&nbsp;label to the bootstrap host. By default, any host with this label will (also) get a copy of&nbsp;<code>\/etc\/ceph\/ceph.conf<\/code>&nbsp;and&nbsp;<code>\/etc\/ceph\/ceph.client.admin.keyring<\/code>.<\/em><\/li>\n<\/ul>\n\n\n\n<p>As you can see, cephadm used podman container management tool, <strong><em>podman (\/usr\/bin\/podman) version 4.6.1 is present<\/em><\/strong>, from the bootstrap command output.<\/p>\n\n\n\n<p>You can see created containers;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>podman ps<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>CONTAINER ID  IMAGE                                                                                      COMMAND               CREATED        STATUS        PORTS       NAMES\nb0001264bf0e  quay.io\/ceph\/ceph:v18                                                                      -n mon.node01 -f ...  3 minutes ago  Up 3 minutes              ceph-f14cb896-889a-11ee-abf4-525400ac8730-mon-node01\n673b69975d89  quay.io\/ceph\/ceph:v18                                                                      -n mgr.node01.gvh...  3 minutes ago  Up 3 minutes              ceph-f14cb896-889a-11ee-abf4-525400ac8730-mgr-node01-gvhmuc\n4d4739e59495  quay.io\/ceph\/ceph@sha256:4ce43f6b683448acc5d45ef184be9a44a569cf9c0019d6bb21472523aa52873c  -n client.ceph-ex...  3 minutes ago  Up 3 minutes              ceph-f14cb896-889a-11ee-abf4-525400ac8730-ceph-exporter-node01\n9123d699b6a9  quay.io\/ceph\/ceph@sha256:4ce43f6b683448acc5d45ef184be9a44a569cf9c0019d6bb21472523aa52873c  -n client.crash.n...  3 minutes ago  Up 3 minutes              ceph-f14cb896-889a-11ee-abf4-525400ac8730-crash-node01\n0be1d3e87064  quay.io\/prometheus\/node-exporter:v1.5.0                                                    --no-collector.ti...  3 minutes ago  Up 3 minutes              ceph-f14cb896-889a-11ee-abf4-525400ac8730-node-exporter-node01\n3a6e81492858  quay.io\/prometheus\/prometheus:v2.43.0                                                      --config.file=\/et...  2 minutes ago  Up 2 minutes              ceph-f14cb896-889a-11ee-abf4-525400ac8730-prometheus-node01\nc7f4e180775c  quay.io\/prometheus\/alertmanager:v0.25.0                                                    --cluster.listen-...  2 minutes ago  Up 2 minutes              ceph-f14cb896-889a-11ee-abf4-525400ac8730-alertmanager-node01\n177cc82138c4  quay.io\/ceph\/ceph-grafana:9.4.7                                                            \/bin\/bash             2 minutes ago  Up 2 minutes              ceph-f14cb896-889a-11ee-abf4-525400ac8730-grafana-node01\n<\/code><\/pre>\n\n\n\n<p>If you are using Docker, then list the containers using;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker ps<\/code><\/pre>\n\n\n\n<p>Similarly, systemd unit files are also created for these containers;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl list-units 'ceph*'<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>  UNIT                                                                   LOAD   ACTIVE SUB     DESCRIPTION                                                       \n  ceph-f14cb896-889a-11ee-abf4-525400ac8730@alertmanager.node01.service  loaded active running Ceph alertmanager.node01 for f14cb896-889a-11ee-abf4-525400ac8730\n  ceph-f14cb896-889a-11ee-abf4-525400ac8730@ceph-exporter.node01.service loaded active running Ceph ceph-exporter.node01 for f14cb896-889a-11ee-abf4-525400ac8730\n  ceph-f14cb896-889a-11ee-abf4-525400ac8730@crash.node01.service         loaded active running Ceph crash.node01 for f14cb896-889a-11ee-abf4-525400ac8730\n  ceph-f14cb896-889a-11ee-abf4-525400ac8730@grafana.node01.service       loaded active running Ceph grafana.node01 for f14cb896-889a-11ee-abf4-525400ac8730\n  ceph-f14cb896-889a-11ee-abf4-525400ac8730@mgr.node01.gvhmuc.service    loaded active running Ceph mgr.node01.gvhmuc for f14cb896-889a-11ee-abf4-525400ac8730\n  ceph-f14cb896-889a-11ee-abf4-525400ac8730@mon.node01.service           loaded active running Ceph mon.node01 for f14cb896-889a-11ee-abf4-525400ac8730\n  ceph-f14cb896-889a-11ee-abf4-525400ac8730@node-exporter.node01.service loaded active running Ceph node-exporter.node01 for f14cb896-889a-11ee-abf4-525400ac8730\n  ceph-f14cb896-889a-11ee-abf4-525400ac8730@prometheus.node01.service    loaded active running Ceph prometheus.node01 for f14cb896-889a-11ee-abf4-525400ac8730   \n  ceph-f14cb896-889a-11ee-abf4-525400ac8730.target                       loaded active active  Ceph cluster f14cb896-889a-11ee-abf4-525400ac8730\n  ceph.target                                                            loaded active active  All Ceph clusters and services\n\nLOAD   = Reflects whether the unit definition was properly loaded.\nACTIVE = The high-level unit activation state, i.e. generalization of SUB.\nSUB    = The low-level unit activation state, values depend on unit type.\n10 loaded units listed. Pass --all to see loaded but inactive units, too.\nTo show all installed unit files use 'systemctl list-unit-files'.\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"enable-ceph-cli\">Enable Ceph CLI<\/h3>\n\n\n\n<p>When bootstrap command completes, a command for accessing Ceph CLI is provided. Execute that command to access Ceph CLI;<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>\/usr\/sbin\/cephadm shell \\\n\t--fsid f14cb896-889a-11ee-abf4-525400ac8730 \\\n\t-c \/etc\/ceph\/ceph.conf \\\n\t-k \/etc\/ceph\/ceph.client.admin.keyring\n<\/code><\/pre>\n\n\n\n<p>This drops you onto Ceph container CLI;<\/p>\n\n\n\n<p>You can run the ceph commands eg to check the Ceph status;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph -s<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>  cluster:\n    id:     f14cb896-889a-11ee-abf4-525400ac8730\n    health: HEALTH_WARN\n            OSD count 0 < osd_pool_default_size 3\n \n  services:\n    mon: 1 daemons, quorum node01 (age 6m)\n    mgr: node01.gvhmuc(active, since 4m)\n    osd: 0 osds: 0 up, 0 in\n \n  data:\n    pools:   0 pools, 0 pgs\n    objects: 0 objects, 0 B\n    usage:   0 B used, 0 B \/ 0 B avail\n    pgs: \n<\/code><\/pre>\n\n\n\n<p>You can exit the ceph CLI by pressing <strong>Ctrl+D<\/strong> or  type <strong>exit<\/strong> and press ENTER.<\/p>\n\n\n\n<p>There are other ways in which you can access the Ceph CLI. For example, you can run Ceph CLI commands using cephadm command.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cephadm shell -- ceph -s<\/code><\/pre>\n\n\n\n<p><em>Or You could install Ceph CLI tools on the host;<\/em><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><em>cephadm add-repo --release <\/em>reef<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code><em>cephadm install ceph-common<\/em><\/code><\/pre>\n\n\n\n<p><em>With this method, then you can just ran the Ceph commands easily;<\/em><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><em>ceph -s<\/em><\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"copy-ssh-keys-to-other-ceph-nodes\">Copy SSH Keys to Other Ceph Nodes<\/h3>\n\n\n\n<p>Copy the SSH key generated by the bootstrap command to all the monitor and OSD nodes root user account. Ensure Root Login is allowed on the Ceph monitor node.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>for i in {02..06}; do sudo ssh-copy-id -f -i \/etc\/ceph\/ceph.pub root@node$i; done<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"drop-into-ceph-cli\">Drop into Ceph CLI<\/h3>\n\n\n\n<p>You can drop into the Ceph CLI to execute the next commands.<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>sudo \/usr\/sbin\/cephadm shell \\\n\t--fsid f14cb896-889a-11ee-abf4-525400ac8730 \\\n\t-c \/etc\/ceph\/ceph.conf \\\n\t-k \/etc\/ceph\/ceph.client.admin.keyring\n<\/code><\/pre>\n\n\n\n<p>Or if you installed the ceph-common package, no need to drop into the cli as you can directly execute the ceph commands from the terminal.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"add-other-ceph-monitor-nodes-to-ceph-cluster\">Add Other Ceph Monitor Nodes to Ceph Cluster<\/h3>\n\n\n\n<p>At this point, we have just provisioned Ceph Admin node only. You can list all the hosts known to the Ceph ochestrator (ceph-mgr) using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host ls<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>HOST        ADDR             LABELS  STATUS  \nceph-admin  192.168.122.240  _admin          \n1 hosts in cluster\n<\/code><\/pre>\n\n\n\n<p>By default, cephadm will deploy 5 monitor daemons on arbitrary hosts. It is recommended to deploy five monitors if there are five or more nodes in your cluster.<\/p>\n\n\n\n<p>To allow deployment of monitor daemons on arbitrary hosts, execute the command below to add the nodes to the cluster;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>for i in {02..03}; do ceph orch host add node$i; done<\/code><\/pre>\n\n\n\n<p>Sample command output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Added host 'node02' with addr '192.168.122.55'\nAdded host 'node03' with addr '192.168.122.229'<\/code><\/pre>\n\n\n\n<p>You should now have three nodes in the cluster.<\/p>\n\n\n\n<p>Next, label the nodes with its role.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host label add node01 mon01<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host label add node02 mon02<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host label add node03 mon03<\/code><\/pre>\n\n\n\n<p>Confirm the labelling;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host ls<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>HOST    ADDR             LABELS        STATUS  \nnode01  192.168.122.240  _admin,mon01          \nnode02  192.168.122.55   mon02                 \nnode03  192.168.122.229  mon03                 \n3 hosts in cluster\n<\/code><\/pre>\n\n\n\n<p>If you want to manually add monitors, then you need to disable automated monitor deployment, by running this command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch apply mon --unmanaged<\/code><\/pre>\n\n\n\n<p>To deploy each additional monitor:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch daemon add mon *&lt;host1:ip&gt;<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"add-ceph-osd-nodes-to-ceph-cluster\">Add Ceph OSD Nodes to Ceph Cluster<\/h3>\n\n\n\n<p>Similarly, add the OSD Nodes to the cluster;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host add node04<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host add node05<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host add node06<\/code><\/pre>\n\n\n\n<p>Define their respective labels;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>for i in {04..06}; do ceph orch host label add node$i osd$i; done<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"list-ceph-cluster-nodes\">List Ceph Cluster Nodes;<\/h3>\n\n\n\n<p>You can list the Ceph cluster nodes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch host ls<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>HOST    ADDR             LABELS        STATUS  \nnode01  192.168.122.240  _admin,mon01          \nnode02  192.168.122.55   mon02                 \nnode03  192.168.122.229  mon03                 \nnode04  192.168.122.152  osd04                 \nnode05  192.168.122.241  osd05                 \nnode06  192.168.122.116  osd06                 \n6 hosts in cluster\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"create-ceph-os-ds-from-attached-storage-devices\">Create Ceph OSDs from Attached Storage Devices<\/h3>\n\n\n\n<p>In our setup, we have 3 unallocated raw disks of 50G on each OSD node to be used as <a href=\"https:\/\/docs.ceph.com\/en\/reef\/rados\/configuration\/storage-devices\/#bluestore\" target=\"_blank\" rel=\"noreferrer noopener\">bluestore<\/a> for OSD daemons.<\/p>\n\n\n\n<p>You can list the devices that are available on the OSD nodes for creating OSDs using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch device ls<\/code><\/pre>\n\n\n\n<p>A storage device is considered&nbsp;<em>available<\/em>&nbsp;if all of the following conditions are met:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The device must have no partitions.<\/li>\n\n\n\n<li>The device must not have any LVM state.<\/li>\n\n\n\n<li>The device must not be mounted.<\/li>\n\n\n\n<li>The device must not contain a file system.<\/li>\n\n\n\n<li>The device must not contain a Ceph BlueStore OSD.<\/li>\n\n\n\n<li>The device must be larger than 5 GB.<\/li>\n<\/ul>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>HOST    PATH      TYPE  DEVICE ID   SIZE  AVAILABLE  REFRESHED  REJECT REASONS  \nnode04  \/dev\/vdb  hdd              50.0G  Yes        5s ago                     \nnode04  \/dev\/vdc  hdd              50.0G  Yes        5s ago                     \nnode04  \/dev\/vdd  hdd              50.0G  Yes        5s ago                     \nnode05  \/dev\/vdb  hdd              50.0G  Yes        5s ago                     \nnode05  \/dev\/vdc  hdd              50.0G  Yes        5s ago                     \nnode05  \/dev\/vdd  hdd              50.0G  Yes        5s ago                     \nnode06  \/dev\/vdb  hdd              50.0G  Yes        5s ago                     \nnode06  \/dev\/vdc  hdd              50.0G  Yes        5s ago                     \nnode06  \/dev\/vdd  hdd              50.0G  Yes        5s ago\n<\/code><\/pre>\n\n\n\n<p>You can add all the available devices to ceph OSDs at once or just add them one by one.<\/p>\n\n\n\n<p>To attach them all at once;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch apply osd --all-available-devices --method {raw|lvm}<\/code><\/pre>\n\n\n\n<p>Use raw method if you are using raw disks. otherwise, if you are using LVM disks, use lvm method.<\/p>\n\n\n\n<p>Command output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Scheduled osd.all-available-devices update...<\/code><\/pre>\n\n\n\n<p>Note that when you add devices using this approach, then;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If you add new disks to the cluster, they will be automatically used to create new OSDs.<\/li>\n\n\n\n<li>In the event that an OSD is removed, and the LVM physical volume is cleaned, a new OSD will be generated automatically.<\/li>\n<\/ul>\n\n\n\n<p>If you wish to prevent this behavior (i.e., disable the automatic creation of OSDs on available devices), use the 'unmanaged' parameter:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch apply osd --all-available-devices --unmanaged=true<\/code><\/pre>\n\n\n\n<p>To manually create an OSD from a specific device on a specific host:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch daemon add osd &lt;host&gt;:&lt;device-path&gt;<\/code><\/pre>\n\n\n\n<p><strong>NOTE<\/strong>: The deployment of OSDs with&nbsp;<code>--all-available-devices<\/code>&nbsp;is generally used for smaller clusters. For larger clusters, use the OSD specification file.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"check-ceph-cluster-health\">Check Ceph Cluster Health<\/h3>\n\n\n\n<p>To verify the health status of the ceph cluster, simply execute the command&nbsp;<code>ceph s<\/code>&nbsp;on the admin node or even on each OSD node (if you have installed cephadm\/ceph commands there).<\/p>\n\n\n\n<p>To check Ceph cluster health status from the admin node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph -s<\/code><\/pre>\n\n\n\n<p>After sometime, you should see such an output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>  cluster:\n    id:     f14cb896-889a-11ee-abf4-525400ac8730\n    health: HEALTH_OK\n \n  services:\n    mon: 5 daemons, quorum node01,node02,node03,node04,node06 (age 8m)\n    mgr: node01.gvhmuc(active, since 83m), standbys: node02.gpynef\n    osd: 9 osds: 9 up (since 38s), 9 in (since 74s)\n \n  data:\n    pools:   1 pools, 1 pgs\n    objects: 2 objects, 449 KiB\n    usage:   639 MiB used, 449 GiB \/ 450 GiB avail\n    pgs:     1 active+clean\n<\/code><\/pre>\n\n\n\n<p>Check OSDs;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph osd tree<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>ID  CLASS  WEIGHT   TYPE NAME        STATUS  REWEIGHT  PRI-AFF\n-1         0.43918  root default                              \n-5         0.14639      host node04                           \n 0    hdd  0.04880          osd.0        up   1.00000  1.00000\n 4    hdd  0.04880          osd.4        up   1.00000  1.00000\n 7    hdd  0.04880          osd.7        up   1.00000  1.00000\n-3         0.14639      host node05                           \n 1    hdd  0.04880          osd.1        up   1.00000  1.00000\n 3    hdd  0.04880          osd.3        up   1.00000  1.00000\n 6    hdd  0.04880          osd.6        up   1.00000  1.00000\n-7         0.14639      host node06                           \n 2    hdd  0.04880          osd.2        up   1.00000  1.00000\n 5    hdd  0.04880          osd.5        up   1.00000  1.00000\n 8    hdd  0.04880          osd.8        up   1.00000  1.00000\n<\/code><\/pre>\n\n\n\n<p>Get a list of Ceph services;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph orch ps<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                  HOST    PORTS             STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  \nalertmanager.node01   node01  *:9093,9094       running (18m)     2m ago  88m    16.0M        -  0.25.0   c8568f914cd2  db6532ab83d6  \nceph-exporter.node01  node01                    running (89m)     2m ago  89m    7054k        -  18.2.0   10237bca3285  4d4739e59495  \nceph-exporter.node02  node02                    running (19m)     7m ago  19m    15.9M        -  18.2.0   10237bca3285  86f8b6c06fad  \nceph-exporter.node03  node03                    running (19m)     7m ago  19m    25.4M        -  18.2.0   10237bca3285  3cebac8c8354  \nceph-exporter.node04  node04                    running (14m)     3m ago  14m    25.2M        -  18.2.0   10237bca3285  c6e64e48a377  \nceph-exporter.node05  node05                    running (13m)     3m ago  13m    26.9M        -  18.2.0   10237bca3285  54aceb24e802  \nceph-exporter.node06  node06                    running (13m)     3m ago  13m    27.0M        -  18.2.0   10237bca3285  895f2924d8bc  \ncrash.node01          node01                    running (89m)     2m ago  89m    6665k        -  18.2.0   10237bca3285  9123d699b6a9  \ncrash.node02          node02                    running (19m)     7m ago  19m    7379k        -  18.2.0   10237bca3285  323988bc0599  \ncrash.node03          node03                    running (19m)     7m ago  19m    7348k        -  18.2.0   10237bca3285  de5041df3615  \ncrash.node04          node04                    running (14m)     3m ago  14m    7375k        -  18.2.0   10237bca3285  869911b5699e  \ncrash.node05          node05                    running (13m)     3m ago  13m    7516k        -  18.2.0   10237bca3285  216d5290bf12  \ncrash.node06          node06                    running (13m)     3m ago  13m    7363k        -  18.2.0   10237bca3285  53f443a15f0c  \ngrafana.node01        node01  *:3000            running (88m)     2m ago  88m    82.8M        -  9.4.7    2c41d148cca3  177cc82138c4  \nmgr.node01.gvhmuc     node01  *:9283,8765,8443  running (89m)     2m ago  89m     517M        -  18.2.0   10237bca3285  673b69975d89  \nmgr.node02.gpynef     node02  *:8443,9283,8765  running (19m)     7m ago  19m     437M        -  18.2.0   10237bca3285  d62a305af919  \nmon.node01            node01                    running (89m)     2m ago  89m    72.1M    2048M  18.2.0   10237bca3285  b0001264bf0e  \nmon.node02            node02                    running (19m)     7m ago  19m    49.7M    2048M  18.2.0   10237bca3285  ac9bd2a4215b  \nmon.node03            node03                    running (19m)     7m ago  19m    44.0M    2048M  18.2.0   10237bca3285  a0593be7e384  \nmon.node04            node04                    running (14m)     3m ago  14m    48.5M    2048M  18.2.0   10237bca3285  2cc1fd067cc4  \nmon.node06            node06                    running (13m)     3m ago  13m    45.8M    2048M  18.2.0   10237bca3285  0be533449aa9  \nnode-exporter.node01  node01  *:9100            running (88m)     2m ago  88m    15.3M        -  1.5.0    0da6a335fe13  0be1d3e87064  \nnode-exporter.node02  node02  *:9100            running (19m)     7m ago  19m    16.0M        -  1.5.0    0da6a335fe13  e9df4de91167  \nnode-exporter.node03  node03  *:9100            running (18m)     7m ago  19m    15.5M        -  1.5.0    0da6a335fe13  23d2747cb798  \nnode-exporter.node04  node04  *:9100            running (14m)     3m ago  14m    14.3M        -  1.5.0    0da6a335fe13  b60b52f3340a  \nnode-exporter.node05  node05  *:9100            running (13m)     3m ago  13m    16.3M        -  1.5.0    0da6a335fe13  e0754f0b7925  \nnode-exporter.node06  node06  *:9100            running (13m)     3m ago  13m    15.7M        -  1.5.0    0da6a335fe13  b250510fcff0  \nosd.0                 node04                    running (6m)      3m ago   6m    52.3M    4096M  18.2.0   10237bca3285  cf835e2802b3  \nosd.1                 node05                    running (6m)      3m ago   6m    52.5M    4096M  18.2.0   10237bca3285  35f0f29eef81  \nosd.2                 node06                    running (6m)      3m ago   6m    53.5M    4096M  18.2.0   10237bca3285  624ebe681e44  \nosd.3                 node06                    running (6m)      3m ago   6m    55.1M    4096M  18.2.0   10237bca3285  66b8b4938900  \nosd.4                 node05                    running (6m)      3m ago   6m    58.6M    4096M  18.2.0   10237bca3285  6b1d6052e588  \nosd.5                 node04                    running (6m)      3m ago   6m    59.0M    4096M  18.2.0   10237bca3285  e40e66c1c327  \nosd.6                 node06                    running (6m)      3m ago   6m    58.9M    4096M  18.2.0   10237bca3285  57b06148664c  \nosd.7                 node05                    running (6m)      3m ago   6m    54.4M    4096M  18.2.0   10237bca3285  5a624c61464a  \nosd.8                 node04                    running (6m)      3m ago   6m    53.3M    4096M  18.2.0   10237bca3285  03a0cdaeb9e4  \nprometheus.node01     node01  *:9095            running (13m)     2m ago  88m    68.0M        -  2.43.0   a07b618ecd1d  7d6ae3a65490\n<\/code><\/pre>\n\n\n\n<p>Similarly, you can check Ceph service containers created on other nodes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker ps<\/code><\/pre>\n\n\n\n<p>Or list their systemd unit files;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl list-units 'ceph*'<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"accessing-ceph-admin-web-user-interface\">Accessing Ceph Admin Web User Interface<\/h3>\n\n\n\n<p>The bootstrap commands give a url and credentials to use to access the Ceph admin web user interface;<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>...\nCeph Dashboard is now available at:\n\n\t     URL: https:\/\/node01:8443\/\n\t    User: admin\n\tPassword: amw3p3yy2e\n<\/code><\/pre>\n\n\n\n<p>Since we are running multiple nodes, Ceph manager is deployed on two nodes for HA. From the Ceph status command, <strong><code>ceph -s<\/code><\/strong>, you can see under services, that one manager node is active while the other one is on standby.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>  services:\n    mon: 5 daemons, quorum node01,node02,node03,node04,node06 (age 15m)\n<strong>    mgr: node01.gvhmuc(active, since 90m), standbys: node02.gpynef\n<\/strong>    osd: 9 osds: 9 up (since 7m), 9 in (since 8m)<\/code><\/pre>\n\n\n\n<p>Thus, open the browser and navigate to the URL above. Or you can even use the cephadm node IP address if you want..<\/p>\n\n\n\n<p>Open the port on firewall if firewall is running.<\/p>\n\n\n\n<p>Enter the provided credential and reset your admin password and proceed to login to Ceph Admin dashboard.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1392\" height=\"793\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-user-interface.png?v=1700544752\" alt=\"How to Deploy Ceph Storage Cluster on AlmaLinux\" class=\"wp-image-19349\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-user-interface.png?v=1700544752 1392w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-user-interface-768x438.png?v=1700544752 768w\" sizes=\"(max-width: 1392px) 100vw, 1392px\" \/><\/figure>\n\n\n\n<p>If you want, you can activate the <a href=\"https:\/\/docs.ceph.com\/en\/latest\/mgr\/telemetry\/#telemetry-module\" target=\"_blank\" rel=\"noreferrer noopener\">telemetry module<\/a> by clicking <strong>Activate<\/strong> button or just from the CLI;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph telemetry on --license sharing-1-0<\/code><\/pre>\n\n\n\n<p>Telemetry can be disabled at any time with:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ceph telemetry off<\/code><\/pre>\n\n\n\n<p>Go through other Ceph menu to see more about Ceph.<\/p>\n\n\n\n<p>Ceph Dashboard;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1653\" height=\"893\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-dash-board.png?v=1700597424\" alt=\"\" class=\"wp-image-19359\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-dash-board.png?v=1700597424 1653w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-dash-board-768x415.png?v=1700597424 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-dash-board-1536x830.png?v=1700597424 1536w\" sizes=\"(max-width: 1653px) 100vw, 1653px\" \/><\/figure>\n\n\n\n<p>Under the Cluster Menu, you can see quite more details; hosts, disks, OSDs etc.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1641\" height=\"782\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-nodes.png?v=1700597616\" alt=\"\" class=\"wp-image-19361\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-nodes.png?v=1700597616 1641w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-nodes-768x366.png?v=1700597616 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-nodes-1536x732.png?v=1700597616 1536w\" sizes=\"(max-width: 1641px) 100vw, 1641px\" \/><\/figure>\n\n\n\n<p>You can also monitor Ceph performance via Grafana, <strong><code>https:\/\/manager-IP:3000<\/code><\/strong>.<\/p>\n\n\n\n<p>Sample Ceph cluster dashboards;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1567\" height=\"837\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-grafana-dashboard.png?v=1700597758\" alt=\"\" class=\"wp-image-19362\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-grafana-dashboard.png?v=1700597758 1567w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-grafana-dashboard-768x410.png?v=1700597758 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-grafana-dashboard-1536x820.png?v=1700597758 1536w\" sizes=\"(max-width: 1567px) 100vw, 1567px\" \/><\/figure>\n\n\n\n<p>Cluster;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1645\" height=\"851\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-dashboards.png?v=1700597839\" alt=\"\" class=\"wp-image-19363\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-dashboards.png?v=1700597839 1645w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-dashboards-768x397.png?v=1700597839 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/ceph-cluster-dashboards-1536x795.png?v=1700597839 1536w\" sizes=\"(max-width: 1645px) 100vw, 1645px\" \/><\/figure>\n\n\n\n<p>There you go. That marks the end of our tutorial on how to deploy Ceph storage cluster using cephadm on AlmaLinux.<\/p>\n\n\n\n<p>Reference<\/p>\n\n\n\n<p><a href=\"https:\/\/docs.ceph.com\/en\/latest\/cephadm\/install\/#deploying-a-new-ceph-cluster\" target=\"_blank\" rel=\"noreferrer noopener\">Deploying a new ceph cluster<\/a><\/p>\n\n\n\n<p>Other Tutorials<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/configure-and-use-ceph-block-device-on-linux-clients\/\" target=\"_blank\" rel=\"noreferrer noopener\">Configure and Use Ceph Block Device on Linux Clients<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/how-to-reset-ceph-dashboard-admin-password\/\" target=\"_blank\" rel=\"noreferrer noopener\">How to Reset Ceph Dashboard Admin Password<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Follow through this post to learn how to deploy Ceph storage cluster on AlmaLinux. Ceph is a scalable distributed storage system designed for cloud infrastructure<\/p>\n","protected":false},"author":10,"featured_media":19349,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121,1338,39],"tags":[7317,7318,7320,7319],"class_list":["post-19356","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","category-ceph","category-storage","tag-almalinux-ceph-cluster","tag-ceph-cluster-almalinux","tag-cephadm-almalinux","tag-deploy-ceph-on-almalinux","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19356"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=19356"}],"version-history":[{"count":8,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19356\/revisions"}],"predecessor-version":[{"id":20872,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19356\/revisions\/20872"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/19349"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=19356"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=19356"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=19356"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}