{"id":11648,"date":"2022-02-20T08:48:03","date_gmt":"2022-02-20T05:48:03","guid":{"rendered":"https:\/\/kifarunix.com\/?p=11648"},"modified":"2024-04-01T12:11:59","modified_gmt":"2024-04-01T09:11:59","slug":"install-and-setup-ceph-storage-cluster-on-ubuntu-20-04","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/install-and-setup-ceph-storage-cluster-on-ubuntu-20-04\/","title":{"rendered":"Install and Setup Ceph Storage Cluster on Ubuntu 20.04"},"content":{"rendered":"\n<p>Follow through this post to learn how to install and setup Ceph Storage cluster on Ubuntu 20.04. Ceph is a scalable distributed storage system designed for cloud infrastructure and web-scale&nbsp;<a href=\"https:\/\/docs.ceph.com\/docs\/master\/glossary\/#term-ceph-object-storage\" target=\"_blank\" rel=\"noreferrer noopener\">object storage<\/a>. It can also be used to provide&nbsp;<a href=\"https:\/\/docs.ceph.com\/docs\/master\/glossary\/#term-ceph-block-device\" target=\"_blank\" rel=\"noreferrer noopener\">Ceph Block Storage<\/a>&nbsp;as well as&nbsp;<a href=\"https:\/\/docs.ceph.com\/docs\/master\/glossary\/#term-ceph-file-system\" target=\"_blank\" rel=\"noreferrer noopener\">Ceph File System storage<\/a>.<\/p>\n\n\n\n<p>As of this writing, CEPH Pacific is the current stable <a href=\"https:\/\/docs.ceph.com\/en\/latest\/releases\/\" target=\"_blank\" rel=\"noreferrer noopener\">release<\/a>.<\/p>\n\n\n\n<p>See <a href=\"https:\/\/kifarunix.com\/install-and-setup-ceph-storage-cluster-on-ubuntu-2204\/\" target=\"_blank\" rel=\"noreferrer noopener\">our updated guide on Ubuntu 22.04<\/a>.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#install-and-setup-ceph-storage-cluster-on-ubuntu-20-04\">Install and Setup Ceph Storage Cluster on Ubuntu 20.04<\/a><ul><li><a href=\"#architecture-of-our-deployment\">Architecture of our deployment<\/a><\/li><li><a href=\"#ceph-pacific-depolyment-requirements\">Ceph Pacific Depolyment Requirements<\/a><\/li><\/ul><\/li><li><a href=\"#prepare-ceph-nodes-for-ceph-storage-cluster-deployment-on-ubuntu-20-04\">Prepare Ceph Nodes for Ceph Storage Cluster Deployment on Ubuntu 20.04<\/a><ul><li><a href=\"#attach-storage-disks-to-ceph-osd-nodes\">Attach Storage Disks to Ceph OSD Nodes<\/a><\/li><li><a href=\"#run-system-update\">Run System Update<\/a><\/li><li><a href=\"#set-hostnames-and-update-hosts-file\">Set hostnames and Update Hosts File<\/a><\/li><li><a href=\"#set-time-synchronization\">Set Time Synchronization<\/a><\/li><li><a href=\"#install-ssh-server-on-each-node\">Install SSH Server on Each Node<\/a><\/li><li><a href=\"#install-python-3\">Install Python3<\/a><\/li><li><a href=\"#create-ceph-deployment-user\">Create Ceph Deployment User<\/a><\/li><li><a href=\"#install-docker-on-each-node\">Install Docker on Each Node<\/a><\/li><li><a href=\"#enable-root-login-on-other-nodes\">Enable Root Login on Other Nodes<\/a><\/li><\/ul><\/li><li><a href=\"#setup-ceph-storage-cluster-on-ubuntu-20-04\">Setup Ceph Storage Cluster on Ubuntu 20.04<\/a><ul><li><a href=\"#install-cephadm-utility-on-ceph-admin-node\">Install\u00a0cephadm\u00a0Utility on Ceph Admin Node<\/a><\/li><li><a href=\"#setup-ceph-cluster-monitor\">Setup Ceph Cluster Monitor<\/a><ul><li><a href=\"#initialize-ceph-cluster-monitor\">Initialize Ceph Cluster monitor<\/a><\/li><\/ul><\/li><li><a href=\"#enable-ceph-cli\">Enable Ceph CLI<\/a><\/li><li><a href=\"#add-ceph-monitor-node-to-ceph-cluster\">Add Ceph Monitor Node to Ceph Cluster<\/a><\/li><li><a href=\"#add-ceph-osd-nodes-to-ceph-cluster\">Add Ceph OSD Nodes to Ceph Cluster<\/a><\/li><li><a href=\"#list-ceph-cluster-nodes\">List Ceph Cluster Nodes;<\/a><\/li><li><a href=\"#attach-logical-storage-volumes-to-ceph-osd-nodes\">Attach Logical Storage Volumes to Ceph OSD Nodes<\/a><\/li><li><a href=\"#check-ceph-cluster-health\">Check Ceph Cluster Health<\/a><\/li><li><a href=\"#accessing-ceph-admin-web-user-interface\">Accessing Ceph Admin Web User Interface<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"install-and-setup-ceph-storage-cluster-on-ubuntu-20-04\">Install and Setup Ceph Storage Cluster on Ubuntu 20.04<\/h2>\n\n\n\n<p>Ceph Storage Cluster&nbsp;setup requires at least one Ceph Monitor, Ceph Manager, and Ceph OSD (Object Storage Daemon) and may be Ceph Metadata Server for providing Ceph File System Storage.<\/p>\n\n\n\n<p>All these components perform various roles;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Ceph Admin node (<code>cephadm<\/code>)<\/strong>\n<ul class=\"wp-block-list\">\n<li>It is the node on which Ceph deployment script (<a href=\"https:\/\/docs.ceph.com\/en\/latest\/cephadm\/#cephadm\" target=\"_blank\" rel=\"noreferrer noopener\">cephadm<\/a>) is installed on.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Ceph Object Storage Daemon (OSD,&nbsp;<\/strong><code><strong>ceph-osd<\/strong><\/code><strong>)<\/strong>\n<ul class=\"wp-block-list\">\n<li>It provides ceph object data store.<\/li>\n\n\n\n<li>It also performs data replication , data recovery, rebalancing and provides storage information to Ceph Monitor.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Ceph Monitor (<\/strong><code><strong>ceph-mon<\/strong><\/code><strong>)<\/strong>\n<ul class=\"wp-block-list\">\n<li>It maintains maps of the entire ceph cluster state including monitor map, manager map, the OSD map, and the CRUSH map.<\/li>\n\n\n\n<li>manages authentication between daemons and clients.<\/li>\n\n\n\n<li>A minimum of three monitors is required but it is recommended to deploy five monitors if there are five or more nodes in your cluster.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Ceph Manager<\/strong>&nbsp;(<strong><code>ceph-mgr<\/code><\/strong>)\n<ul class=\"wp-block-list\">\n<li>keeps track of runtime metrics and the current state of the Ceph cluster, including storage utilization, current performance metrics, and system load.<\/li>\n\n\n\n<li>manages and exposes Ceph cluster web dashboard and API.<\/li>\n\n\n\n<li>At least two managers are required for HA.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"architecture-of-our-deployment\">Architecture of our deployment<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"880\" height=\"700\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-architecture-1.png\" alt=\"Install and Setup Ceph Storage Cluster on Ubuntu 20.04\" class=\"wp-image-11663\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-architecture-1.png?v=1645335159 880w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-architecture-1-768x611.png?v=1645335159 768w\" sizes=\"(max-width: 880px) 100vw, 880px\" \/><\/figure><\/div>\n\n\n<p>If your cluster nodes are in the same network subnet, <strong>cephadm<\/strong> will automatically add up to five monitors to the subnet, as new hosts are added to the cluster.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"ceph-pacific-depolyment-requirements\">Ceph Pacific Depolyment Requirements<\/h3>\n\n\n\n<p>Below are the requirements for deploying Ceph Pacific storage cluster;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Python 3<\/li>\n\n\n\n<li>Systemd<\/li>\n\n\n\n<li>Podman or Docker for running containers (<strong>we use docker in this setup<\/strong>)<\/li>\n\n\n\n<li>Time synchronization (such as chrony or NTP)<\/li>\n\n\n\n<li>LVM2 for provisioning storage devices.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"prepare-ceph-nodes-for-ceph-storage-cluster-deployment-on-ubuntu-20-04\">Prepare Ceph Nodes for Ceph Storage Cluster Deployment on Ubuntu 20.04<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"attach-storage-disks-to-ceph-osd-nodes\">Attach Storage Disks to Ceph OSD Nodes<\/h3>\n\n\n\n<p>Each Ceph OSD node in our architecture above has unallocated LVM logical volumes of 4 GB each.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>lvs<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>  LV   VG    Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert\n  lv01 vol01 -wi-a----- &lt;10.00g<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"run-system-update\">Run System Update<\/h3>\n\n\n\n<p>On all the nodes, update your system packages.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>apt update<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"set-hostnames-and-update-hosts-file\">Set hostnames and Update Hosts File<\/h3>\n\n\n\n<p>To begin with, setup up your nodes hostnames;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>hostnamectl set-hostname ceph-admin<\/code><\/pre>\n\n\n\n<p>Do the same on the other nodes.<\/p>\n\n\n\n<p>Next, if you are not using DNS for name resolution, then update the hosts file accordingly.<\/p>\n\n\n\n<p>For example, in our setup, each node hosts file should contain the lines below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>less \/etc\/hosts<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>...\n192.168.59.31 ceph-admin\n192.168.59.30 ceph-mon\n192.168.59.29 ceph-osd1\n192.168.59.28 ceph-osd2<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"set-time-synchronization\">Set Time Synchronization<\/h3>\n\n\n\n<p>Ensure that the time on <strong>all the nodes<\/strong> is synchronized. Thus install Chrony on each and set it up such that all nodes uses the same NTP server.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>apt install chrony -y<\/code><\/pre>\n\n\n\n<p>Edit the Chrony configuration and set your NTP server by replacing the NTP server pools with your NTP server address.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/chrony\/chrony.conf<\/code><\/pre>\n\n\n\n<p>Define your NTP Server. Replace <strong>ntp.kifarunix-demo.com<\/strong> with your respective NTP server address.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>...\n# pool ntp.ubuntu.com        iburst maxsources 4\n# pool 0.ubuntu.pool.ntp.org iburst maxsources 1\n# pool 1.ubuntu.pool.ntp.org iburst maxsources 1\n# pool 2.ubuntu.pool.ntp.org iburst maxsources 2\n<strong>pool ntp.kifarunix-demo.com iburst<\/strong>\n...<\/code><\/pre>\n\n\n\n<p>Restart Chronyd<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl restart chronyd<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-ssh-server-on-each-node\">Install SSH Server on Each Node<\/h3>\n\n\n\n<p>Ceph deployment through cephadm utility requires that an SSH server is installed on all the nodes.<\/p>\n\n\n\n<p>Ubuntu 20.04 comes with SSH server already installed. If not, install and start it as follows;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>apt install openssh-server<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl enable --now sshd<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-python-3\">Install Python3<\/h3>\n\n\n\n<p>Python is required to deploy Ceph on Ubuntu 20.04. Python 3 is installed by default on Ubuntu 20.04<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"create-ceph-deployment-user\">Create Ceph Deployment User<\/h3>\n\n\n\n<p><strong>On the ceph admin node<\/strong>, create a ceph user with sudo right required for installing ceph packages and configurations as shown below. Do not use the username&nbsp;<code>ceph<\/code>&nbsp;as it is reserved.<\/p>\n\n\n\n<p>Replace&nbsp;<code>cephadmin<\/code>&nbsp;username accordingly.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>useradd -m -s \/bin\/bash cephadmin<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>passwd cephadmin<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"cephadmin ALL=(ALL:ALL) NOPASSWD:ALL\" &gt;&gt; \/etc\/sudoers.d\/cephadmin<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>chmod 0440 \/etc\/sudoers.d\/cephadmin<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-docker-on-each-node\">Install Docker on Each Node<\/h3>\n\n\n\n<p id=\"install-docker-on-each-osd-nodes\">The cephadm utility is used to bootstrap a Ceph cluster and to manage ceph daemons deployed with systemd and containers.<\/p>\n\n\n\n<p>Thus, <strong>on each Node<\/strong>, run the command below to install Docker CE;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt install apt-transport-https ca-certificates curl gnupg-agent software-properties-common -y<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -fsSL https:\/\/download.docker.com\/linux\/ubuntu\/gpg | sudo apt-key add -<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"deb &#91;arch=amd64] https:\/\/download.docker.com\/linux\/ubuntu $(lsb_release -sc) stable\" | sudo tee \/etc\/apt\/sources.list.d\/docker-ce.list<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt update<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt install docker-ce docker-ce-cli containerd.io -y<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo systemctl enable --now docker<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"enable-root-login-on-other-nodes\">Enable Root Login on Other Nodes<\/h3>\n\n\n\n<p>In order to add other nodes to the Ceph cluster using Ceph Admin Node, you will have to use the root user account.<\/p>\n\n\n\n<p>Thus, on the Ceph Monitor, Ceph OSD nodes, enable root login from the Ceph Admin node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/ssh\/sshd_config<\/code><\/pre>\n\n\n\n<p>Add the configs below, replacing the IP address for Ceph Admin accordingly.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Match Address 192.168.59.31\n        PermitRootLogin yes<\/code><\/pre>\n\n\n\n<p>Reload ssh;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl reload sshd<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"setup-ceph-storage-cluster-on-ubuntu-20-04\">Setup Ceph Storage Cluster on Ubuntu 20.04<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-cephadm-utility-on-ceph-admin-node\">Install&nbsp;<code>cephadm<\/code>&nbsp;Utility on Ceph Admin Node<\/h3>\n\n\n\n<p><strong>On the Ceph admin node<\/strong>, you need to install the cephadm utility.<\/p>\n\n\n\n<p><em><a href=\"https:\/\/docs.ceph.com\/en\/latest\/cephadm\/#cephadm\" target=\"_blank\" rel=\"noopener\">Cephadm<\/a>&nbsp;installs and manages a Ceph cluster using containers and systemd, with tight integration with the CLI and dashboard GUI.<\/em><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><em>cephadm only supports Octopus and newer releases.<\/em><\/li>\n\n\n\n<li><em>cephadm is fully integrated with the new orchestration API and fully supports the new CLI and dashboard features to manage cluster deployment.<\/em><\/li>\n\n\n\n<li><em>cephadm requires container support (podman or docker) and Python 3<\/em>.<\/li>\n<\/ul>\n\n\n\n<p>You can check other <a href=\"https:\/\/docs.ceph.com\/en\/latest\/install\/#recommended-methods\" target=\"_blank\" rel=\"noreferrer noopener\">recommended methods<\/a> of deploying Ceph.<\/p>\n\n\n\n<p>To install cephadm on Ubuntu 20.04;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo wget -q https:\/\/github.com\/ceph\/ceph\/raw\/pacific\/src\/cephadm\/cephadm -P \/usr\/bin\/<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo chmod +x \/usr\/bin\/cephadm<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"setup-ceph-cluster-monitor\">Setup Ceph Cluster Monitor<\/h3>\n\n\n\n<p>Your nodes are now ready to deploy a Ceph storage cluster. To begin with, switch to <strong><code>cephadmin<\/code><\/strong> user;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>su - cephadmin<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>whoami<\/code><\/pre>\n\n\n\n<p>Output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cephadmin<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"initialize-ceph-cluster-monitor\">Initialize Ceph Cluster monitor<\/h4>\n\n\n\n<p>It is now time to bootstrap the Ceph cluster in order to create the first monitor daemon, the Ceph Admin node.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo cephadm bootstrap --mon-ip 192.168.59.31<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\nVerifying podman|docker is present...\nVerifying lvm2 is present...\nVerifying time synchronization is in place...\nUnit systemd-timesyncd.service is enabled and running\nRepeating the final host check...\ndocker (\/usr\/bin\/docker) is present\nsystemctl is present\nlvcreate is present\nUnit systemd-timesyncd.service is enabled and running\nHost looks OK\nCluster fsid: f959b65e-91c2-11ec-9776-abbffb8a52a1\nVerifying IP 192.168.59.31 port 3300 ...\nVerifying IP 192.168.59.31 port 6789 ...\nMon IP `192.168.59.31` is in CIDR network `192.168.59.0\/24`\n- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network\nPulling container image quay.io\/ceph\/ceph:v16...\nCeph version: ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)\nExtracting ceph user uid\/gid from container image...\nCreating initial keys...\nCreating initial monmap...\nCreating mon...\nWaiting for mon to start...\nWaiting for mon...\nmon is available\nAssimilating anything we can from ceph.conf...\nGenerating new minimal ceph.conf...\nRestarting the monitor...\nSetting mon public_network to 192.168.59.0\/24\nWrote config to \/etc\/ceph\/ceph.conf\nWrote keyring to \/etc\/ceph\/ceph.client.admin.keyring\nCreating mgr...\nVerifying port 9283 ...\nWaiting for mgr to start...\nWaiting for mgr...\nmgr not available, waiting (1\/15)...\nmgr not available, waiting (2\/15)...\nmgr not available, waiting (3\/15)...\nmgr is available\nEnabling cephadm module...\nWaiting for the mgr to restart...\nWaiting for mgr epoch 4...\nmgr epoch 4 is available\nSetting orchestrator backend to cephadm...\nGenerating ssh key...\nWrote public SSH key to \/etc\/ceph\/ceph.pub\nAdding key to root@localhost authorized_keys...\nAdding host ceph-admin...\nDeploying mon service with default placement...\nDeploying mgr service with default placement...\nDeploying crash service with default placement...\nDeploying prometheus service with default placement...\nDeploying grafana service with default placement...\nDeploying node-exporter service with default placement...\nDeploying alertmanager service with default placement...\nEnabling the dashboard module...\nWaiting for the mgr to restart...\nWaiting for mgr epoch 8...\nmgr epoch 8 is available\nGenerating a dashboard self-signed certificate...\nCreating initial admin user...\nFetching dashboard port number...\nCeph Dashboard is now available at:\n\n\t     URL: https:\/\/ceph-admin:8443\/\n\t    User: admin\n\tPassword: 7164vdghsy\n\nEnabling client.admin keyring and conf on hosts with \"admin\" label\nEnabling autotune for osd_memory_target\nYou can access the Ceph CLI with:\n\n\tsudo \/usr\/bin\/cephadm shell --fsid f959b65e-91c2-11ec-9776-abbffb8a52a1 -c \/etc\/ceph\/ceph.conf -k \/etc\/ceph\/ceph.client.admin.keyring\n\nPlease consider enabling telemetry to help improve Ceph:\n\n\tceph telemetry on\n\nFor more information see:\n\n\thttps:\/\/docs.ceph.com\/docs\/pacific\/mgr\/telemetry\/\n\nBootstrap complete.\n<\/code><\/pre>\n\n\n\n<p>According to the <a href=\"https:\/\/docs.ceph.com\/en\/latest\/cephadm\/install\/#running-the-bootstrap-command\" target=\"_blank\" rel=\"noreferrer noopener\">documentation<\/a>, the bootstrap command;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><em>Create a monitor and manager daemon for the new cluster on the local host.<\/em><\/li>\n\n\n\n<li><em>Generate a new SSH key for the Ceph cluster and add it to the root user\u2019s&nbsp;<code>\/root\/.ssh\/authorized_keys<\/code>&nbsp;file.<\/em><\/li>\n\n\n\n<li><em>Write a copy of the public key to&nbsp;<code>\/etc\/ceph\/ceph.pub<\/code>.<\/em><\/li>\n\n\n\n<li><em>Write a minimal configuration file to&nbsp;<code>\/etc\/ceph\/ceph.conf<\/code>. This file is needed to communicate with the new cluster.<\/em><\/li>\n\n\n\n<li><em>Write a copy of the&nbsp;<code>client.admin<\/code>&nbsp;administrative (privileged!) secret key to&nbsp;<code>\/etc\/ceph\/ceph.client.admin.keyring<\/code>.<\/em><\/li>\n\n\n\n<li><em>Add the&nbsp;<code>_admin<\/code>&nbsp;label to the bootstrap host. By default, any host with this label will (also) get a copy of&nbsp;<code>\/etc\/ceph\/ceph.conf<\/code>&nbsp;and&nbsp;<code>\/etc\/ceph\/ceph.client.admin.keyring<\/code>.<\/em><\/li>\n<\/ul>\n\n\n\n<p>Ceph admin containers are installed;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker ps<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\nCONTAINER ID   IMAGE                                      COMMAND                  CREATED          STATUS          PORTS     NAMES\nfa8b37976c6f   quay.io\/prometheus\/alertmanager:v0.20.0    \"\/bin\/alertmanager -\u2026\"   21 minutes ago   Up 21 minutes             ceph-f959b65e-91c2-11ec-9776-abbffb8a52a1-alertmanager-ceph-admin\na3345c3df15e   quay.io\/ceph\/ceph-grafana:6.7.4            \"\/bin\/sh -c 'grafana\u2026\"   21 minutes ago   Up 21 minutes             ceph-f959b65e-91c2-11ec-9776-abbffb8a52a1-grafana-ceph-admin\n562cb5595aa7   quay.io\/prometheus\/prometheus:v2.18.1      \"\/bin\/prometheus --c\u2026\"   21 minutes ago   Up 21 minutes             ceph-f959b65e-91c2-11ec-9776-abbffb8a52a1-prometheus-ceph-admin\n81aa6bff037a   quay.io\/prometheus\/node-exporter:v0.18.1   \"\/bin\/node_exporter \u2026\"   22 minutes ago   Up 22 minutes             ceph-f959b65e-91c2-11ec-9776-abbffb8a52a1-node-exporter-ceph-admin\ncba3508f638b   quay.io\/ceph\/ceph                          \"\/usr\/bin\/ceph-crash\u2026\"   24 minutes ago   Up 24 minutes             ceph-f959b65e-91c2-11ec-9776-abbffb8a52a1-crash-ceph-admin\n6ce79ae859b2   quay.io\/ceph\/ceph:v16                      \"\/usr\/bin\/ceph-mgr -\u2026\"   25 minutes ago   Up 25 minutes             ceph-f959b65e-91c2-11ec-9776-abbffb8a52a1-mgr-ceph-admin-yxxusl\n3e901accdfb6   quay.io\/ceph\/ceph:v16                      \"\/usr\/bin\/ceph-mon -\u2026\"   25 minutes ago   Up 25 minutes             ceph-f959b65e-91c2-11ec-9776-abbffb8a52a1-mon-ceph-admin\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"enable-ceph-cli\">Enable Ceph CLI<\/h3>\n\n\n\n<p>When bootstrap command completes, a command for accessing Ceph CLI is provided. Execute that command to access Ceph CLI;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo \/usr\/bin\/cephadm shell --fsid f959b65e-91c2-11ec-9776-abbffb8a52a1 -c \/etc\/ceph\/ceph.conf -k \/etc\/ceph\/ceph.client.admin.keyring<\/code><\/pre>\n\n\n\n<p>This drops you onto Ceph Docker CLI;<\/p>\n\n\n\n<p>You can run the ceph commands eg to check the Ceph status;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph -s<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\n  cluster:\n    id:     f959b65e-91c2-11ec-9776-abbffb8a52a1\n    health: HEALTH_WARN\n            OSD count 0 < osd_pool_default_size 3\n \n  services:\n    mon: 1 daemons, quorum ceph-admin (age 23m)\n    mgr: ceph-admin.yxxusl(active, since 20m)\n    osd: 0 osds: 0 up, 0 in\n \n  data:\n    pools:   0 pools, 0 pgs\n    objects: 0 objects, 0 B\n    usage:   0 B used, 0 B \/ 0 B avail\n    pgs: \n<\/code><\/pre>\n\n\n\n<p>You can exit the docker CLI by pressing <strong>Ctrl+D<\/strong> or enter <strong>exit<\/strong>.<\/p>\n\n\n\n<p>There are other ways in which you can access the Ceph CLI. For example, you can run Ceph CLI commands using cephadm command.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo cephadm shell -- ceph -s<\/code><\/pre>\n\n\n\n<p>Or Install Ceph CLI tools on the host;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo cephadm add-repo --release pacific\nsudo cephadm install ceph-common<\/code><\/pre>\n\n\n\n<p>With this method, then you can just ran the Ceph commands easily;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph -s<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"add-ceph-monitor-node-to-ceph-cluster\">Add Ceph Monitor Node to Ceph Cluster<\/h3>\n\n\n\n<p>At this point, we have just provisioned Ceph Admin node only;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch host ls<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>HOST        ADDR           LABELS  STATUS  \nceph-admin  192.168.59.31  _admin <\/code><\/pre>\n\n\n\n<p>So next, add the Ceph Monitor node to the cluster;<\/p>\n\n\n\n<p>Copy the SSH key generated by the bootstrap command to Ceph Monitor's root user account. Ensure Root Login is permitted on the Ceph monitor node.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ssh-copy-id -f -i \/etc\/ceph\/ceph.pub root@ceph-mon<\/code><\/pre>\n\n\n\n<p>Once you have copied the Ceph SSH public key, execute the command below to add the Ceph Monitor to the cluster;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch host add ceph-mon<\/code><\/pre>\n\n\n\n<p>Sample command output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Added host 'ceph-mon' with addr '192.168.59.30'<\/code><\/pre>\n\n\n\n<p>Next, label the host with its role (remember our ceph-monitor also doubles up as an OSD);<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch host label add ceph-mon mon\/osd<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"add-ceph-osd-nodes-to-ceph-cluster\">Add Ceph OSD Nodes to Ceph Cluster<\/h3>\n\n\n\n<p>Similarly, copy the SSH keys to the OSD Nodes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>for i in ceph-osd1 ceph-osd2; do sudo ssh-copy-id -f -i \/etc\/ceph\/ceph.pub root@$i; done<\/code><\/pre>\n\n\n\n<p>Add them to the cluster.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch host add ceph-osd1<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch host add ceph-osd2<\/code><\/pre>\n\n\n\n<p>Define their respective labels;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>for i in ceph-osd1 ceph-osd2; do sudo ceph orch host label add $i osd; done<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"list-ceph-cluster-nodes\">List Ceph Cluster Nodes;<\/h3>\n\n\n\n<p>You can list the Ceph cluster nodes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch host ls<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\nHOST        ADDR           LABELS  STATUS  \nceph-admin  192.168.59.31  _admin          \nceph-mon    192.168.59.30  mon             \nceph-osd1   192.168.59.29  osd             \nceph-osd2   192.168.59.28  osd\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"attach-logical-storage-volumes-to-ceph-osd-nodes\">Attach Logical Storage Volumes to Ceph OSD Nodes<\/h3>\n\n\n\n<p>In our setup, we have unallocated logical volume of 4 GB on each OSD node to be used as a backstore for OSD daemons.<\/p>\n\n\n\n<p>To attach the logical volumes to the OSD node, run the command below.&nbsp;<strong>Replace vg01\/lv01<\/strong>&nbsp;with Volume group and logical volume accordingly.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch daemon add osd ceph-mon:vg01\/lv01<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch daemon add osd ceph-mon:vol01\/lv01<\/code><\/pre>\n\n\n\n<p>Repeat the same for the other OSD nodes.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch daemon add osd ceph-osd1:vol01\/lv01<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch daemon add osd ceph-osd2:vol01\/lv01<\/code><\/pre>\n\n\n\n<p>The Ceph Nodes are now ready for OSD use.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"check-ceph-cluster-health\">Check Ceph Cluster Health<\/h3>\n\n\n\n<p>To verify the health status of the ceph cluster, simply execute the command&nbsp;<code>ceph <\/code>s&nbsp;on each OSD node.<\/p>\n\n\n\n<p>To check Ceph cluster health status from the admin node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph -s<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n  cluster:\n    id:     f959b65e-91c2-11ec-9776-abbffb8a52a1\n    health: HEALTH_OK\n \n  services:\n    mon: 4 daemons, quorum ceph-admin,ceph-mon,ceph-osd1,ceph-osd2 (age 12m)\n    mgr: ceph-admin.yxxusl(active, since 93m), standbys: ceph-mon.rivvvs\n    osd: 3 osds: 3 up (since 4m), 3 in (since 4m)\n \n  data:\n    pools:   1 pools, 1 pgs\n    objects: 0 objects, 0 B\n    usage:   15 MiB used, 12 GiB \/ 12 GiB avail\n    pgs:     1 active+clean\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"accessing-ceph-admin-web-user-interface\">Accessing Ceph Admin Web User Interface<\/h3>\n\n\n\n<p>The bootstrap commands give a url and credentials to use to access the Ceph admin web user interface;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Ceph Dashboard is now available at:\n\n\t     URL: https:\/\/ceph-admin:8443\/\n\t    User: admin\n\tPassword: 7164vdghsy<\/code><\/pre>\n\n\n\n<p>Thus, open the browser and navigate to the URL.<\/p>\n\n\n\n<p>Reset your admin password and proceed to login to Ceph Admin UI.<\/p>\n\n\n\n<div><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-login-page.png\" class=\"td-modal-image\"><figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1791\" height=\"892\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-login-page.png\" alt=\"Install and Setup Ceph Storage Cluster on Ubuntu 20.04\" class=\"wp-image-11664\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-login-page.png?v=1645335293 1791w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-login-page-768x382.png?v=1645335293 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-login-page-1536x765.png?v=1645335293 1536w\" sizes=\"(max-width: 1791px) 100vw, 1791px\" \/><\/figure><\/a><\/div>\n\n\n\n<p>If you want, you can activate the <a href=\"https:\/\/docs.ceph.com\/en\/latest\/mgr\/telemetry\/#telemetry-module\" target=\"_blank\" rel=\"noreferrer noopener\">telemetry module<\/a>.<\/p>\n\n\n\n<p>Go through other Ceph menu to see more about Ceph.<\/p>\n\n\n\n<div><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-admin-UI.png\" class=\"td-modal-image\"><figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1911\" height=\"714\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-admin-UI.png\" alt=\"Install and Setup Ceph Storage Cluster on Ubuntu 20.04\" class=\"wp-image-11665\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-admin-UI.png?v=1645335335 1911w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-admin-UI-768x287.png?v=1645335335 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-admin-UI-1536x574.png?v=1645335335 1536w\" sizes=\"(max-width: 1911px) 100vw, 1911px\" \/><\/figure><\/a><\/div>\n\n\n\n<p>Ceph Dashboard;<\/p>\n\n\n\n<div><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-dashboard.png\" class=\"td-modal-image\"><figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1917\" height=\"939\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-dashboard.png\" alt=\"Install and Setup Ceph Storage Cluster on Ubuntu 20.04\" class=\"wp-image-11666\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-dashboard.png?v=1645335364 1917w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-dashboard-768x376.png?v=1645335364 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/02\/ceph-dashboard-1536x752.png?v=1645335364 1536w\" sizes=\"(max-width: 1917px) 100vw, 1917px\" \/><\/figure><\/a><\/div>\n\n\n\n<p>There you go. That marks the end of our tutorial on how to install and setup Ceph storage cluster on Ubuntu.<\/p>\n\n\n\n<p>Reference<\/p>\n\n\n\n<p><a href=\"https:\/\/docs.ceph.com\/en\/latest\/cephadm\/install\/#deploying-a-new-ceph-cluster\" target=\"_blank\" rel=\"noreferrer noopener\">Deploying a new ceph cluster<\/a><\/p>\n\n\n\n<p>Other Tutorials<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-owncloud-server-on-ubuntu-22-04\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install ownCloud Server on Ubuntu 22.04<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/mount-remote-filesystem-over-ssh-using-sshfs\/\" target=\"_blank\" rel=\"noreferrer noopener\">Mount Remote Filesystem Over SSH using SSHFS<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-configure-nfs-server-on-rocky-linux-8\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install and Configure NFS Server on Rocky Linux 8<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Follow through this post to learn how to install and setup Ceph Storage cluster on Ubuntu 20.04. Ceph is a scalable distributed storage system designed<\/p>\n","protected":false},"author":1,"featured_media":11667,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121,1338,39],"tags":[4613,4614,4615,4609,4611,4616,4612,4610],"class_list":["post-11648","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","category-ceph","category-storage","tag-ceph-pacific","tag-cephadm","tag-deploy-ceph-using-cephadm","tag-install-and-setup-ceph-storage-cluster","tag-install-ceph-on-ubuntu-20-04","tag-install-cephadm-ubuntu-20-04","tag-setup-ceph-pacific-storage-cluster","tag-ubuntu-20-04-ceph-pacific","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/11648"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=11648"}],"version-history":[{"count":19,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/11648\/revisions"}],"predecessor-version":[{"id":22036,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/11648\/revisions\/22036"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/11667"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=11648"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=11648"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=11648"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}