{"id":4538,"date":"2020-03-14T10:47:46","date_gmt":"2020-03-14T07:47:46","guid":{"rendered":"https:\/\/kifarunix.com\/?p=4538"},"modified":"2023-11-18T17:00:54","modified_gmt":"2023-11-18T14:00:54","slug":"setup-three-node-ceph-storage-cluster-on-ubuntu-18-04","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/setup-three-node-ceph-storage-cluster-on-ubuntu-18-04\/","title":{"rendered":"Setup Three Node Ceph Storage Cluster on Ubuntu 18.04"},"content":{"rendered":"\n<p>Welcome to our tutorial on how to setup three node ceph storage cluster on Ubuntu 18.04. Ceph is a scalable distributed storage system designed for cloud infrastructure and web-scale <a aria-label=\"object storage (opens in a new tab)\" href=\"https:\/\/docs.ceph.com\/docs\/master\/glossary\/#term-ceph-object-storage\" target=\"_blank\" rel=\"noreferrer noopener\">object storage<\/a>. It can also be used to provide <a aria-label=\"Ceph Block Storage (opens in a new tab)\" href=\"https:\/\/docs.ceph.com\/docs\/master\/glossary\/#term-ceph-block-device\" target=\"_blank\" rel=\"noreferrer noopener\">Ceph Block Storage<\/a> as well as <a aria-label=\"Ceph File System storage (opens in a new tab)\" href=\"https:\/\/docs.ceph.com\/docs\/master\/glossary\/#term-ceph-file-system\" target=\"_blank\" rel=\"noreferrer noopener\">Ceph File System storage<\/a>.<\/p>\n\n\n\n<p>You can check an updated guide on <a href=\"https:\/\/kifarunix.com\/install-and-setup-ceph-storage-cluster-on-ubuntu-2204\/\" target=\"_blank\" rel=\"noreferrer noopener\">how to deploy Ceph cluster on Ubuntu 22.04<\/a>.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#setup-three-node-ceph-storage-cluster-on-ubuntu-18-04\">Setup Three Node Ceph Storage Cluster on Ubuntu 18.04<\/a><ul><li><a href=\"#architecture-of-our-deployment\">Architecture of our deployment<\/a><\/li><\/ul><\/li><li><a href=\"#prepare-ceph-nodes-for-ceph-storage-cluster-deployment-on-ubuntu-18-04\">Prepare Ceph Nodes for Ceph Storage Cluster Deployment on Ubuntu 18.04<\/a><ul><li><a href=\"#attach-storage-disks-to-ceph-osd-nodes\">Attach Storage Disks to Ceph OSD Nodes<\/a><\/li><li><a href=\"#run-system-update\">Run System Update<\/a><\/li><li><a href=\"#update-hosts-file\">Update Hosts File<\/a><\/li><li><a href=\"#set-time-synchronization\">Set Time Synchronization<\/a><\/li><li><a href=\"#install-ssh-server\">Install SSH Server<\/a><\/li><li><a href=\"#install-python-2\">Install Python 2<\/a><\/li><li><a href=\"#create-ceph-deployment-user\">Create Ceph Deployment User<\/a><\/li><li><a href=\"#setup-password-less-ssh-login\">Setup Password-less SSH login<\/a><\/li><\/ul><\/li><li><a href=\"#setup-ceph-storage-cluster-on-ubuntu-18-04\">Setup Ceph Storage Cluster on Ubuntu 18.04<\/a><ul><li><a href=\"#install-ceph-deploy-utility-on-ceph-admin-node\">Install ceph-deploy Utility on Ceph Admin Node<\/a><ul><li><a href=\"#create-ceph-repository-on-ubuntu-18-04\">Create Ceph repository on Ubuntu 18.04<\/a><\/li><\/ul><\/li><li><a href=\"#setup-ceph-cluster-monitor\">Setup Ceph Cluster Monitor<\/a><\/li><li><a href=\"#install-ceph-packages-on-ceph-nodes\">Install Ceph Packages on Ceph Nodes<\/a><\/li><li><a href=\"#deploy-ceph-cluster-monitor\">Deploy Ceph Cluster Monitor<\/a><\/li><li><a href=\"#copy-the-ceph-configuration-files-and-keys\">Copy the Ceph Configuration files and Keys<\/a><\/li><li><a href=\"#deploy-ceph-manager-daemon\">Deploy Ceph Manager Daemon<\/a><\/li><li><a href=\"#attach-logical-storage-volumes-to-ceph-osd-nodes\">Attach Logical Storage Volumes to Ceph OSD Nodes<\/a><\/li><li><a href=\"#check-ceph-cluster-health\">Check Ceph Cluster Health<\/a><\/li><li><a href=\"#uninstalling-the-ceph-packages\">Uninstalling the Ceph Packages<\/a><\/li><\/ul><\/li><li><a href=\"#expanding-ceph-cluster-on-ubuntu-18-04\">Expanding Ceph Cluster on Ubuntu 18.04<\/a><ul><li><a href=\"#deploying-additional-ceph-monitors-on-ceph-cluster\">Deploying Additional Ceph Monitors on Ceph Cluster<\/a><\/li><li><a href=\"#deploying-additional-ceph-managers-on-ceph-cluster\">Deploying Additional Ceph Managers on Ceph Cluster<\/a><\/li><li><a href=\"#related-tutorials\">Related Tutorials<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"setup-three-node-ceph-storage-cluster-on-ubuntu-18-04\">Setup Three Node Ceph Storage Cluster on Ubuntu 18.04<\/h2>\n\n\n\n<p>Ceph Storage Cluster&nbsp;setup requires at least one Ceph Monitor, Ceph Manager, and Ceph OSD (Object Storage Daemon) and may be Ceph Metadata Server for providing Ceph File System Storage.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"architecture-of-our-deployment\">Architecture of our deployment<\/h4>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"658\" height=\"298\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/03\/ceph-deployment-architecture-1.png\" alt=\"Setup Three Node Ceph Storage Cluster on Ubuntu 18.04\" class=\"wp-image-5208\" title=\"\"><\/figure><\/div>\n\n\n<p>Different Ceph components are used in the above;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Ceph Object Storage Daemon (OSD, <\/strong><code><strong>ceph-osd<\/strong><\/code><strong>)<\/strong>\n<ul class=\"wp-block-list\">\n<li>It provides ceph object data store.<\/li>\n\n\n\n<li>It also performs data replication , data recovery, rebalancing and provides storage information to Ceph Monitor.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Ceph Monitor (<\/strong><code><strong>ceph-mon<\/strong><\/code><strong>)<\/strong>\n<ul class=\"wp-block-list\">\n<li>It maintains maps of the entire ceph cluster state including monitor map, manager map, the OSD map, and the CRUSH map.<\/li>\n\n\n\n<li>manages authentication between daemons and clients<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Ceph Deploy\/Admin node (<code>ceph-deploy<\/code>)<\/strong>\n<ul class=\"wp-block-list\">\n<li>It is the node on which Ceph deployment script (ceph-deploy) is installed on.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Ceph Manager<\/strong> (<strong><code>ceph-mgr<\/code><\/strong>)\n<ul class=\"wp-block-list\">\n<li>keeps track of runtime metrics and the current state of the Ceph cluster, including storage utilization, current performance metrics, and system load.<\/li>\n\n\n\n<li>manages and exposes Ceph cluster web dashboard and API.<\/li>\n\n\n\n<li>At least two managers are required for HA.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"prepare-ceph-nodes-for-ceph-storage-cluster-deployment-on-ubuntu-18-04\">Prepare Ceph Nodes for Ceph Storage Cluster Deployment on Ubuntu 18.04<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"attach-storage-disks-to-ceph-osd-nodes\">Attach Storage Disks to Ceph OSD Nodes<\/h3>\n\n\n\n<p>Each Ceph OSD node in our architecture above has unallocated LVM logical volumes of 4 GB each.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>lvs<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>LV   VG   Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert\n<strong>lv01 vg01 -wi-a----- 3.99g<\/strong><\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"run-system-update\">Run System Update<\/h3>\n\n\n\n<p>On all the nodes, update your system packages.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apt update<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"update-hosts-file\">Update Hosts File<\/h3>\n\n\n\n<p>To begin with, setup up your nodes, assign the addresses and ensure that they can communicate by updating the hosts file. For example, in our setup, each node hosts file should contain the lines below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>less \/etc\/hosts<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>...\n192.168.2.114 ceph-admin.kifarunix-demo.com  ceph-admin\n192.168.2.115 ceph-osd01.kifarunix-demo.com  ceph-osd01\n192.168.2.116 ceph-osd02.kifarunix-demo.com  ceph-osd02\n192.168.2.117 ceph-osd03.kifarunix-demo.com  ceph-osd03<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"set-time-synchronization\">Set Time Synchronization<\/h3>\n\n\n\n<p>Ensure that the time on all the nodes is synchronized. Thus install Chrony on each and set it up such that all nodes uses the same NTP server.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apt install chrony<\/code><\/pre>\n\n\n\n<p>Edit the Chrony configuration and set your NTP server by replacing the  NTP server pools with your NTP server address.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/chrony\/chrony.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>...\n# pool ntp.ubuntu.com        iburst maxsources 4\n# pool 0.ubuntu.pool.ntp.org iburst maxsources 1\n# pool 1.ubuntu.pool.ntp.org iburst maxsources 1\n# pool 2.ubuntu.pool.ntp.org iburst maxsources 2\n<strong>pool ntp.kifarunix-demo.com iburst<\/strong>\n...<\/code><\/pre>\n\n\n\n<p>Restart Chronyd<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl restart chronyd<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-ssh-server\">Install SSH Server<\/h3>\n\n\n\n<p>Ceph deployment through ceph-deploy utility requires that an SSH server is installed on all the nodes.<\/p>\n\n\n\n<p>Ubuntu 18.04 comes with SSH server already installed. If not, install and start it as follows;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apt install openssh-server<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl enable --now sshd<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-python-2\">Install Python 2<\/h3>\n\n\n\n<p>Python 2 is required to deploy Ceph on Ubuntu 18.04. You can install Python 2 by executing the command below <strong>on all the Ceph nodes<\/strong>;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apt install python-minimal<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"create-ceph-deployment-user\">Create Ceph Deployment User<\/h3>\n\n\n\n<p><strong>On all the ceph nodes<\/strong>, create the ceph user with passwordless sudo required for installing ceph packages and configurations as shown below. Do not use the username <code>ceph<\/code> as it is reserved.<\/p>\n\n\n\n<p>Replace <code>cephadmin<\/code> username accordingly.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>useradd -m -s \/bin\/bash cephadmin<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>passwd cephadmin<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>echo \"cephadmin ALL=(ALL:ALL) NOPASSWD:ALL\" &gt;&gt; \/etc\/sudoers.d\/cephadmin<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>chmod 0440 \/etc\/sudoers.d\/cephadmin<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"setup-password-less-ssh-login\">Setup Password-less SSH login<\/h3>\n\n\n\n<p>To be able to seamlessly run the ceph configurations on the ceph nodes using the ceph-deploy utility, you need to setup password-less ssh login <strong>from the ceph admin node.<\/strong><\/p>\n\n\n\n<p>On <strong>ceph-admin node,<\/strong> switch to <code>cephadmin<\/code> user created above;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>su - cephadmin<\/code><\/pre>\n\n\n\n<p>Next, generate password-less SSH keys by leaving the passphrase empty.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ssh-keygen<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Generating public\/private rsa key pair.\nEnter file in which to save the key (\/home\/cephadmin\/.ssh\/id_rsa): \nCreated directory '\/home\/cephadmin\/.ssh'.\nEnter passphrase (empty for no passphrase): \nEnter same passphrase again: \nYour identification has been saved in \/home\/cephadmin\/.ssh\/id_rsa.\nYour public key has been saved in \/home\/cephadmin\/.ssh\/id_rsa.pub.\nThe key fingerprint is:\nSHA256:hCX07NgcOmFx7gJrS8QCphNBqdjToPe73AtRNaC+f9k cephadmin@ceph-admin.kifarunix-demo.com\nThe key's randomart image is:\n+---&#91;RSA 2048]----+\n|=+.  .=.=        |\n|o+.. . @ .       |\n|=o.o= = *        |\n|+.++.= X .       |\n| . o* = S        |\n|   o.+ o         |\n|    +.   o       |\n|   ..+  o E      |\n|    o.+o         |\n+----&#91;SHA256]-----+<\/code><\/pre>\n\n\n\n<p>Next, copy the keys to all other nodes. To make the copying easy, update your ~\/.ssh\/config file as shown below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim ~\/.ssh\/config<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>Host ceph-osd01\n   Hostname ceph-osd01\n   User cephadmin\nHost ceph-osd02\n   Hostname ceph-osd02\n   User cephadmin\nHost ceph-osd03\n   Hostname ceph-osd03\n   User cephadmin<\/code><\/pre>\n\n\n\n<p>After that, copy SSH keys.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>for i in ceph-osd01 ceph-osd02 ceph-osd03; do ssh-copy-id $i; done<\/code><\/pre>\n\n\n\n<p>Enter password at each prompt to copy generated SSH keys for password-less authentication from Ceph admin to the Ceph nodes.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"setup-ceph-storage-cluster-on-ubuntu-18-04\">Setup Ceph Storage Cluster on Ubuntu 18.04<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-ceph-deploy-utility-on-ceph-admin-node\">Install <code>ceph-deploy<\/code> Utility on Ceph Admin Node<\/h3>\n\n\n\n<p><strong>On the Ceph admin node<\/strong>, you need to install the ceph-deploy utility. To install ceph-deploy utility and other Ceph packages, you need to create the Ceph repositories.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-ceph-repository-on-ubuntu-18-04\">Create Ceph repository on Ubuntu 18.04<\/h4>\n\n\n\n<p>Install Ceph repository signing key. Execute these commands as root user.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>wget -q -O- 'https:\/\/download.ceph.com\/keys\/release.asc' | apt-key add -<\/code><\/pre>\n\n\n\n<p>Check the <a rel=\"noreferrer noopener\" aria-label=\"latest stable release version of ceph (opens in a new tab)\" href=\"https:\/\/docs.ceph.com\/docs\/master\/releases\/\" target=\"_blank\">latest stable release version of ceph<\/a> and replace the <strong>{ceph-stable-release}<\/strong> with the version name.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>echo deb https:\/\/download.ceph.com\/debian-<strong>{ceph-stable-release}<\/strong>\/ $(lsb_release -sc) main | tee \/etc\/apt\/sources.list.d\/ceph.list <\/code><\/pre>\n\n\n\n<p>For example, to install the repos for Ceph Mimic on Ubuntu 18.04;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>echo deb https:\/\/download.ceph.com\/debian-mimic\/ $(lsb_release -sc<strong>)<\/strong> main | tee \/etc\/apt\/sources.list.d\/ceph.list<\/code><\/pre>\n\n\n\n<p>Resynchronize the repos to their latest versions.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apt update<\/code><\/pre>\n\n\n\n<p>Install Ceph deploy utility.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apt install ceph-deploy<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"setup-ceph-cluster-monitor\">Setup Ceph Cluster Monitor<\/h3>\n\n\n\n<p>Your nodes are now ready to deploy a Ceph storage cluster. To begin with, switch to <code>cephadmin<\/code> user created above.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>su - cephadmin<\/code><\/pre>\n\n\n\n<p><strong>NOTE: Do not run&nbsp;<code>ceph-deploy<\/code>&nbsp;with&nbsp;<code>sudo<\/code>&nbsp;nor run it as&nbsp;<code>root<\/code><\/strong>.<\/p>\n\n\n\n<p>Create a directory on Ceph admin home directory for storing configuration files and keys generated by the ceph-deploy command;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>mkdir kifarunix-cluster<\/code><\/pre>\n\n\n\n<p>Navigate to the directory created above.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>cd kifarunix-cluster<\/code><\/pre>\n\n\n\n<p>Initialize Ceph cluster monitor. Our Ceph cluster monitor is running on Ceph OSD 01.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ceph-deploy new ceph-osd01<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>...\n&#91;ceph_deploy.new]&#91;DEBUG ] Creating new cluster named ceph\n&#91;ceph_deploy.new]&#91;INFO  ] making sure passwordless SSH succeeds\n&#91;ceph-osd01]&#91;DEBUG ] connected to host: ceph-admin.kifarunix-demo.com \n&#91;ceph-osd01]&#91;INFO  ] Running command: ssh -CT -o BatchMode=yes ceph-osd01\n&#91;ceph-osd01]&#91;DEBUG ] connection detected need for sudo\n&#91;ceph-osd01]&#91;DEBUG ] connected to host: ceph-osd01 \n&#91;ceph-osd01]&#91;DEBUG ] detect platform information from remote host\n&#91;ceph-osd01]&#91;DEBUG ] detect machine type\n&#91;ceph-osd01]&#91;DEBUG ] find the location of an executable\n&#91;ceph-osd01]&#91;INFO  ] Running command: sudo \/bin\/ip link show\n&#91;ceph-osd01]&#91;INFO  ] Running command: sudo \/bin\/ip addr show\n&#91;ceph-osd01]&#91;DEBUG ] IP addresses found: &#91;u'192.168.2.114', u'10.0.2.15']\n&#91;ceph_deploy.new]&#91;DEBUG ] Resolving host ceph-osd01\n&#91;ceph_deploy.new]&#91;DEBUG ] Monitor ceph-osd01 at 192.168.2.114\n&#91;ceph_deploy.new]&#91;DEBUG ] Monitor initial members are &#91;'ceph-osd01']\n&#91;ceph_deploy.new]&#91;DEBUG ] Monitor addrs are &#91;'192.168.2.114']\n&#91;ceph_deploy.new]&#91;DEBUG ] Creating a random mon key...\n&#91;ceph_deploy.new]&#91;DEBUG ] Writing monitor keyring to ceph.mon.keyring...\n&#91;ceph_deploy.new]&#91;DEBUG ] Writing initial config to ceph.conf<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-ceph-packages-on-ceph-nodes\">Install Ceph Packages on Ceph Nodes<\/h3>\n\n\n\n<p>Next, install Ceph packages on all the nodes by executing the <code><strong>ceph-deploy install<\/strong><\/code> command on the Ceph Admin node as shown below.<\/p>\n\n\n\n<p>By default, the ceph-deploy too installs the latest stable release version of Ceph, which is <strong>Ceph Mimic<\/strong> as of this writing.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ceph-deploy install ceph-osd01 ceph-osd02 ceph-osd03<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploy-ceph-cluster-monitor\">Deploy Ceph Cluster Monitor<\/h3>\n\n\n\n<p>Ceph cluster monitor has been initialized above. To deploy it, execute the command below on Ceph Admin node.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ceph-deploy mon create-initial<\/code><\/pre>\n\n\n\n<p>This command will generate a number of keys on the current working directory.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>...\n&#91;ceph_deploy.gatherkeys]&#91;INFO  ] Storing ceph.client.admin.keyring\n&#91;ceph_deploy.gatherkeys]&#91;INFO  ] Storing ceph.bootstrap-mds.keyring\n&#91;ceph_deploy.gatherkeys]&#91;INFO  ] Storing ceph.bootstrap-mgr.keyring\n&#91;ceph_deploy.gatherkeys]&#91;INFO  ] keyring 'ceph.mon.keyring' already exists\n&#91;ceph_deploy.gatherkeys]&#91;INFO  ] Storing ceph.bootstrap-osd.keyring\n&#91;ceph_deploy.gatherkeys]&#91;INFO  ] Storing ceph.bootstrap-rgw.keyring\n&#91;ceph_deploy.gatherkeys]&#91;INFO  ] Destroy temp directory \/tmp\/tmpIzZYbV<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"copy-the-ceph-configuration-files-and-keys\">Copy the Ceph Configuration files and Keys<\/h3>\n\n\n\n<p>Next, copy the configuration file and admin keys gathered above <strong>to all your Ceph nodes<\/strong> to enable you use the <code>ceph<\/code>&nbsp;CLI without having to specify the monitor address and&nbsp;<code>ceph.client.admin.keyring<\/code>&nbsp;each time you execute a command.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ceph-deploy admin ceph-osd01 ceph-osd02 ceph-osd03<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>...\n&#91;ceph_deploy.admin]&#91;DEBUG ] Pushing admin keys and conf to ceph-osd01\n&#91;ceph-osd01]&#91;DEBUG ] connection detected need for sudo\n&#91;ceph-osd01]&#91;DEBUG ] connected to host: ceph-osd01 \n&#91;ceph-osd01]&#91;DEBUG ] detect platform information from remote host\n&#91;ceph-osd01]&#91;DEBUG ] detect machine type\n&#91;ceph-osd01]&#91;DEBUG ] write cluster configuration to \/etc\/ceph\/{cluster}.conf\n&#91;ceph_deploy.admin]&#91;DEBUG ] Pushing admin keys and conf to ceph-osd02\n&#91;ceph-osd02]&#91;DEBUG ] connection detected need for sudo\n&#91;ceph-osd02]&#91;DEBUG ] connected to host: ceph-osd02 \n&#91;ceph-osd02]&#91;DEBUG ] detect platform information from remote host\n&#91;ceph-osd02]&#91;DEBUG ] detect machine type\n&#91;ceph-osd02]&#91;DEBUG ] write cluster configuration to \/etc\/ceph\/{cluster}.conf\n&#91;ceph_deploy.admin]&#91;DEBUG ] Pushing admin keys and conf to ceph-osd03\n&#91;ceph-osd03]&#91;DEBUG ] connection detected need for sudo\n&#91;ceph-osd03]&#91;DEBUG ] connected to host: ceph-osd03 \n&#91;ceph-osd03]&#91;DEBUG ] detect platform information from remote host\n&#91;ceph-osd03]&#91;DEBUG ] detect machine type\n&#91;ceph-osd03]&#91;DEBUG ] write cluster configuration to \/etc\/ceph\/{cluster}.conf<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploy-ceph-manager-daemon\">Deploy Ceph Manager Daemon<\/h3>\n\n\n\n<p>Once you have copied the Ceph configuration files and keys to all the nodes, deploy a Ceph Cluster manager daemon by executing the command below.<\/p>\n\n\n\n<p><strong>Note that our Ceph Manager resides on Ceph node 01, osd01<\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ceph-deploy mgr create ceph-osd01<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"attach-logical-storage-volumes-to-ceph-osd-nodes\">Attach Logical Storage Volumes to Ceph OSD Nodes<\/h3>\n\n\n\n<p>In our setup, we have unallocated logical volume of 4 GB on each OSD node to be used as a backstore for OSD daemons.<\/p>\n\n\n\n<p>To attach the logical volumes to the OSD node, run the command below. <strong>Replace vg01\/lv01<\/strong> with Volume group and logical volume accordingly.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ceph-deploy osd create --data vg01\/lv01 ceph-osd01<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>...\n&#91;ceph-osd01]&#91;WARNIN] --&gt; ceph-volume lvm activate successful for osd ID: 0\n&#91;ceph-osd01]&#91;WARNIN] --&gt; ceph-volume lvm create successful for: vg01\/lv01\n&#91;ceph-osd01]&#91;INFO  ] checking OSD status...\n&#91;ceph-osd01]&#91;DEBUG ] find the location of an executable\n&#91;ceph-osd01]&#91;INFO  ] Running command: sudo \/usr\/bin\/ceph --cluster=ceph osd stat --format=json\n&#91;ceph_deploy.osd]&#91;DEBUG ] Host ceph-osd01 is now ready for osd use.<\/code><\/pre>\n\n\n\n<p>Repeat the same for the other OSD nodes.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ceph-deploy osd create --data vg01\/lv01 ceph-osd02<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ceph-deploy osd create --data vg01\/lv01 ceph-osd03<\/code><\/pre>\n\n\n\n<p>The Ceph Nodes are now ready for OSD use.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"check-ceph-cluster-health\">Check Ceph Cluster Health<\/h3>\n\n\n\n<p>To verify the health status of the ceph cluster, simply execute the command <code>ceph health<\/code> on each OSD node.<\/p>\n\n\n\n<p>To check Ceph cluster health status from the admin node;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>cephadmin@ceph-admin:~$ ssh ceph-osd01 sudo ceph health<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code><strong>HEALTH_OK<\/strong><\/code><\/pre>\n\n\n\n<p>Repeat the same for other nodes.<\/p>\n\n\n\n<p>You can as well check the Cluster health status from the ceph nodes.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>cephadmin@ceph-osd01:~$ sudo ceph health<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>HEALTH_OK<\/code><\/pre>\n\n\n\n<p>To check complete Cluster health status, run <code>ceph -s<\/code>, <code>ceph --status<\/code> or <code>ceph status<\/code>.<\/p>\n\n\n\n<p>From the Ceph Admin node;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>cephadmin@ceph-admin:~$ <strong>ssh ceph-osd01 sudo ceph -s<\/strong><\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>  cluster:\n    id:     ecc4e749-830a-4ec5-8af9-22fcb5cadbca\n    health: HEALTH_OK\n \n  services:\n    mon: 1 daemons, quorum ceph-osd01\n    mgr: ceph-osd01(active)\n    osd: 3 osds: 3 up, 3 in\n \n  data:\n    pools:   0 pools, 0 pgs\n    objects: 0  objects, 0 B\n    usage:   3.0 GiB used, 9.0 GiB \/ 12 GiB avail\n    pgs:<\/code><\/pre>\n\n\n\n<p>To check complete health Cluster from the Ceph nodes;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>cephadmin@ceph-osd01:~$ <strong>sudo ceph -s<\/strong><\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>  cluster:\n    id:     ecc4e749-830a-4ec5-8af9-22fcb5cadbca\n    health: HEALTH_OK\n \n  services:\n    mon: 1 daemons, quorum ceph-osd01\n    mgr: ceph-osd01(active)\n    osd: 3 osds: 3 up, 3 in\n \n  data:\n    pools:   0 pools, 0 pgs\n    objects: 0  objects, 0 B\n    usage:   3.0 GiB used, 9.0 GiB \/ 12 GiB avail\n    pgs:<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"uninstalling-the-ceph-packages\">Uninstalling the Ceph Packages<\/h3>\n\n\n\n<p>If for some reasons, you want to restart Ceph deployment, remove the Ceph packages, delete the Ceph configuration data on all the nodes by executing the commands below on the Ceph Admin node.<\/p>\n\n\n\n<p>Replace the Ceph nodes accordingly.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ceph-deploy purge ceph-osd01 ceph-osd02 ceph-osd03\nceph-deploy purgedata ceph-osd01 ceph-osd02 ceph-osd03\nceph-deploy forgetkeys\nrm ceph.*<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"expanding-ceph-cluster-on-ubuntu-18-04\">Expanding Ceph Cluster on Ubuntu 18.04<\/h2>\n\n\n\n<p>Now that your basic Ceph cluster is up and running, you can expand it to ensure reliability and high availability of the Ceph cluster.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploying-additional-ceph-monitors-on-ceph-cluster\">Deploying Additional Ceph Monitors on Ceph Cluster<\/h3>\n\n\n\n<p>You can be able to add more Ceph monitor daemons (ceph-mon) to your Ceph cluster nodes.<\/p>\n\n\n\n<p>For example, to add a Ceph Monitor to Ceph Node 02,ceph-osd02, you can run the command below from the Ceph Admin node.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>su - cephadmin<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>cd kifarunix-cluster\/<\/code><\/pre>\n\n\n\n<p>Add your Public network to the <code>ceph.conf<\/code> configuration file even if your nodes have a single network interface only.<\/p>\n\n\n\n<p>Note the line; <strong><code>public network = 192.168.2.0\/24<\/code><\/strong><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim ceph.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>[global]\nfsid = ecc4e749-830a-4ec5-8af9-22fcb5cadbca\nmon_initial_members = ceph-osd01\nmon_host = 192.168.2.114\nauth_cluster_required = cephx\nauth_service_required = cephx\nauth_client_required = cephx\n<strong>public network = 192.168.2.0\/24<\/strong><\/code><\/pre>\n\n\n\n<p>Next, add the Ceph monitor to your Ceph node.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ceph-deploy mon add ceph-osd02<\/code><\/pre>\n\n\n\n<p>You can check the Ceph monitor quorum status by running the command below from Ceph OSD nodes.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ceph quorum_status --format json-pretty<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploying-additional-ceph-managers-on-ceph-cluster\">Deploying Additional Ceph Managers on Ceph Cluster<\/h3>\n\n\n\n<p>Ceph manager&nbsp;daemon (<code>ceph-mgr<\/code>) runs alongside monitor daemons, to provide additional monitoring and interfaces to external monitoring and management systems. They operate in an active\/standby such that if one daemon or host fails, another one can take over without interrupting service.<\/p>\n\n\n\n<p>You can add more Ceph managers to the Cluster by running the command;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ceph-deploy mgr create {ceph-node1} {ceph-node2}<\/code><\/pre>\n\n\n\n<p>For example, to deploy an extra Ceph Manger on our Ceph node 2, ceph-osd02 from Ceph Admin node;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>su - cephadmin<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>cd kifarunix-cluster\/<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ceph-deploy mgr create ceph-osd02<\/code><\/pre>\n\n\n\n<p>If you check the Ceph cluster status, you should see that one of the managers is active while the other is on Standby.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>cephadmin@ceph-admin:~\/kifarunix-cluster$ <strong>ssh ceph-osd01 sudo ceph -s<\/strong><\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>  cluster:\n    id:     ecc4e749-830a-4ec5-8af9-22fcb5cadbca\n    health: HEALTH_OK\n \n  services:\n    mon: 2 daemons, quorum ceph-osd01,ceph-osd02\n    <strong>mgr: ceph-osd01(active), standbys: ceph-osd02<\/strong>\n    osd: 3 osds: 3 up, 3 in\n \n  data:\n    pools:   0 pools, 0 pgs\n    objects: 0  objects, 0 B\n    usage:   3.0 GiB used, 9.0 GiB \/ 12 GiB avail\n    pgs: <\/code><\/pre>\n\n\n\n<p>There you go. Your Ceph storage cluster is now ready to provide, <a rel=\"noreferrer noopener\" aria-label=\"Object Storage (opens in a new tab)\" href=\"https:\/\/docs.ceph.com\/docs\/dumpling\/glossary\/#term-ceph-object-storage\" target=\"_blank\">Object Storage<\/a>, <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.ceph.com\/docs\/master\/glossary\/#term-ceph-block-device\" target=\"_blank\">Block Storage<\/a> or <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.ceph.com\/docs\/master\/glossary\/#term-ceph-file-system\" target=\"_blank\">File System storage<\/a>.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-configure-ceph-block-device-on-ubuntu-18-04\/\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">Install and Configure Ceph Block Device on Ubuntu 18.04<\/a><\/p>\n\n\n\n<p>That marks the end of your guide on how to install and setup three node Ceph Storage Cluster on Ubuntu 18.04.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"related-tutorials\">Related Tutorials<\/h3>\n\n\n\n<p><a rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/install-and-setup-glusterfs-on-ubuntu-18-04\/\" target=\"_blank\">Install and Setup GlusterFS on Ubuntu 18.04<\/a><\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/how-to-install-and-configure-iscsi-storage-server-on-ubuntu-18-04\/\" target=\"_blank\">How to install and Configure iSCSI Storage Server on Ubuntu 18.04<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/how-to-install-and-configure-nfs-server-on-rhel-centos-7\/\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">How to Install and Configure NFS Server on RHEL\/CentOS 7<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Welcome to our tutorial on how to setup three node ceph storage cluster on Ubuntu 18.04. Ceph is a scalable distributed storage system designed for<\/p>\n","protected":false},"author":1,"featured_media":11667,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[39,1338,121],"tags":[1339,1340,1343,1341,1342,1344,67],"class_list":["post-4538","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-storage","category-ceph","category-howtos","tag-ceph-storage-cluster-ubuntu-18-04","tag-ceph-ubuntu-18-04","tag-ceph-mgr","tag-ceph-mon","tag-ceph-osd","tag-install-ceph-storage-ubuntu-18-04","tag-ubuntu-18-04","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/4538"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=4538"}],"version-history":[{"count":19,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/4538\/revisions"}],"predecessor-version":[{"id":19325,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/4538\/revisions\/19325"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/11667"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=4538"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=4538"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=4538"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}