{"id":22053,"date":"2024-04-11T22:26:13","date_gmt":"2024-04-11T19:26:13","guid":{"rendered":"https:\/\/kifarunix.com\/?p=22053"},"modified":"2024-04-11T22:26:17","modified_gmt":"2024-04-11T19:26:17","slug":"how-to-setup-ceph-admin-node-in-high-availability","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/how-to-setup-ceph-admin-node-in-high-availability\/","title":{"rendered":"How to Setup Ceph Admin Node in High Availability"},"content":{"rendered":"\n<p>In this tutorial, you will learn how to setup Ceph Admin node in high availability. In most setups, <a href=\"https:\/\/docs.ceph.com\/en\/latest\/install\/\" target=\"_blank\" rel=\"noreferrer noopener\">Ceph cluster<\/a> usually runs with a single admin. So, in case the admin node goes down for some reason, how can you be able to administer the cluster? Hence, this is the reason we are trying to explore the option of setting up Ceph admin node in HA.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#how-to-setup-ceph-admin-node-in-high-availability\">How to Setup Ceph Admin Node in High Availability<\/a><ul><li><a href=\"#deployment-architecture\">Deployment Architecture<\/a><\/li><li><a href=\"#deploy-fresh-ceph-cluster-with-admin-node-in-ha\">Deploy Fresh Ceph Cluster with Admin node in HA<\/a><ul><li><a href=\"#setup-high-availability-between-admin-nodes\">Setup High Availability between Admin Nodes<\/a><\/li><li><a href=\"#install-ceph-cluster-with-cephadm\">Install Ceph Cluster with Cephadm<\/a><\/li><li><a href=\"#add-nodes-into-the-ceph-cluster\">Add Nodes into the Ceph Cluster<\/a><\/li><li><a href=\"#setup-second-ceph-admin-node\">Setup Second Ceph Admin Node<\/a><\/li><li><a href=\"#deploy-monitoring-stack-on-ceph-cluster\">Deploy Monitoring Stack on Ceph Cluster<\/a><\/li><\/ul><\/li><li><a href=\"#updating-existing-ceph-cluster-with-admin-node-ha-setup\">Updating existing Ceph cluster with Admin Node HA setup<\/a><ul><li><a href=\"#designate-second-node-as-admin-node\">Designate Second Node as Admin Node<\/a><\/li><li><a href=\"#update-monitoring-stack\">Update Monitoring Stack<\/a><\/li><li><a href=\"#verify-access-to-ceph-cluster-dashboard-and-monitoring-dashboard-via-vip\">Verify Access to Ceph Cluster Dashboard and Monitoring Dashboard via VIP<\/a><\/li><\/ul><\/li><li><a href=\"#conclusion\">Conclusion<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"how-to-setup-ceph-admin-node-in-high-availability\">How to Setup Ceph Admin Node in High Availability<\/h2>\n\n\n\n<p>There are two approaches we will take to configure Ceph cluster with highly available admin node.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Deploying fresh Ceph Cluster with Admin node in HA<\/li>\n\n\n\n<li>Updating existing Ceph cluster with Admin HA setup<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deployment-architecture\">Deployment Architecture<\/h3>\n\n\n\n<p>Below is how our architecture of setting up Ceph storage cluster with Admin nodes in HA.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"824\" height=\"544\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-cluster-admin-node-HA.png\" alt=\"\" class=\"wp-image-22077\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-cluster-admin-node-HA.png?v=1712488811 824w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-cluster-admin-node-HA-768x507.png?v=1712488811 768w\" sizes=\"(max-width: 824px) 100vw, 824px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploy-fresh-ceph-cluster-with-admin-node-in-ha\">Deploy Fresh Ceph Cluster with Admin node in HA<\/h3>\n\n\n\n<p>Assuming you are setting up your Ceph storage cluster from scratch and want to ensure that your admin node is deployed in high availability, then follow through.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"setup-high-availability-between-admin-nodes\">Setup High Availability between Admin Nodes<\/h4>\n\n\n\n<p>We have two nodes that we would like to use as Ceph admin nodes. These are, <strong>node01<\/strong> and <strong>node02<\/strong>.<\/p>\n\n\n\n<p>To setup high availability between these two nodes, we will use <strong><a href=\"https:\/\/keepalived.readthedocs.io\/en\/latest\/introduction.html\" target=\"_blank\" rel=\"noopener\">keepalived<\/a><\/strong>. To install and setup Keepalived, follow the links below.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/kifarunix.com\/configure-highly-available-haproxy-with-keepalived-on-ubuntu\/#install-keepalived-on-ha-proxy-servers\">Install Keepalived on two Admin nodes<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kifarunix.com\/configure-highly-available-haproxy-with-keepalived-on-ubuntu\/#configure-ip-forwarding-and-non-local-binding\">Configure IP forwarding and non-local binding<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kifarunix.com\/configure-highly-available-haproxy-with-keepalived-on-ubuntu\/#configure-keepalived\">Configure Keepalived<\/a><\/li>\n<\/ul>\n\n\n\n<p>This is our sample Keepalived configuration on the two nodes.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>grep -vE \"^$|^#\" \/etc\/keepalived\/keepalived.conf<\/code><\/pre>\n\n\n\n<p>Node01:<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>vrrp_instance HA_VIP {\n    interface enp1s0\n    state MASTER\n    priority 101\n    virtual_router_id 51\n  \n    authentication {\n        auth_type AH\n        auth_pass myP@ssword\n    }\n    unicast_src_ip 192.168.122.78\n    unicast_peer {\n        192.168.122.79\n   }\n  \n    virtual_ipaddress {\n        192.168.122.200\/24\n    }\n     \n}\n<\/code><\/pre>\n\n\n\n<p>Node02;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>vrrp_instance HA_VIP {\n    interface enp1s0\n    state BACKUP\n    priority 100\n    virtual_router_id 51\n  \n    authentication {\n        auth_type AH\n        auth_pass myP@ssword\n    }\n    unicast_src_ip 192.168.122.79\n    unicast_peer {\n        192.168.122.78\n   }\n  \n    virtual_ipaddress {\n        192.168.122.200\/24\n    }\n     \n}\n<\/code><\/pre>\n\n\n\n<p>And Keepalived is running.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl status keepalived<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\u25cf keepalived.service - Keepalive Daemon (LVS and VRRP)\n     Loaded: loaded (\/lib\/systemd\/system\/keepalived.service; enabled; vendor preset: enabled)\n     Active: active (running) since Tue 2024-04-09 18:57:16 UTC; 4min 55s ago\n   Main PID: 348222 (keepalived)\n      Tasks: 2 (limit: 4558)\n     Memory: 2.6M\n        CPU: 76ms\n     CGroup: \/system.slice\/keepalived.service\n             \u251c\u2500348222 \/usr\/sbin\/keepalived --dont-fork\n             \u2514\u2500348223 \/usr\/sbin\/keepalived --dont-fork\n\nApr 09 18:57:16 node02 Keepalived[348222]: NOTICE: setting config option max_auto_priority should result in better keepalived performance\nApr 09 18:57:16 node02 Keepalived[348222]: Starting VRRP child process, pid=348223\nApr 09 18:57:16 node02 systemd[1]: keepalived.service: Got notification message from PID 348223, but reception only permitted for main PID 348222\nApr 09 18:57:16 node02 Keepalived_vrrp[348223]: (\/etc\/keepalived\/keepalived.conf: Line 9) Truncating auth_pass to 8 characters\nApr 09 18:57:16 node02 Keepalived[348222]: Startup complete\nApr 09 18:57:16 node02 systemd[1]: Started Keepalive Daemon (LVS and VRRP).\nApr 09 18:57:16 node02 Keepalived_vrrp[348223]: (HA_VIP) Entering BACKUP STATE (init)\nApr 09 18:59:51 node02 Keepalived_vrrp[348223]: (HA_VIP) Entering MASTER STATE\nApr 09 18:59:55 node02 Keepalived_vrrp[348223]: (HA_VIP) Master received advert from 192.168.122.78 with higher priority 101, ours 100\nApr 09 18:59:55 node02 Keepalived_vrrp[348223]: (HA_VIP) Entering BACKUP STATE\n<\/code><\/pre>\n\n\n\n<p>And Admin node IP addresses;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cephadmin@node01:~$ ip -br a<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>lo               UNKNOWN        127.0.0.1\/8 ::1\/128 \nenp1s0           UP             192.168.122.78\/24 192.168.122.200\/24 fe80::5054:ff:fe8b:8029\/64 \ndocker0          DOWN           172.17.0.1\/16 \n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-ceph-cluster-with-cephadm\">Install Ceph Cluster with Cephadm<\/h4>\n\n\n\n<p>In our previous guides, we have comprehensively provided the steps required to install Ceph cluster using cephadm.<\/p>\n\n\n\n<p>However, at the step where you need to boostrap the Ceph cluster with first monitor, skip the deployment of the Ceph monitoring stack.<\/p>\n\n\n\n<p>See, we are initializing the cluster with the normal admin node IP and NOT the VIP.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo cephadm bootstrap --mon-ip <strong>192.168.122.78<\/strong> --skip-monitoring-stack<\/code><\/pre>\n\n\n\n<p>Update your admin node IP address accordingly.<\/p>\n\n\n\n<p>Follow the link below learn how to deploy Ceph storage cluster.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/?s=deploy+ceph\" target=\"_blank\" rel=\"noreferrer noopener\">How to deploy Ceph storage cluster in Linux<\/a><\/p>\n\n\n\n<p>We are skipping the automatic deployment of the monitoring stack so that we can do it manually later when the cluster is up and running. This will enable us to control on which Ceph cluster nodes the main monitoring components, Grafana and Prometheus will run. Ideally, we will want to be able to run these components on the HA admins nodes to ensure that they are also highly available.<\/p>\n\n\n\n<p>Current cluster status;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph -s<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>  cluster:\n    id:     17ef548c-f68b-11ee-9a19-4d1575fdfd98\n    health: HEALTH_WARN\n            OSD count 0 < osd_pool_default_size 3\n \n  services:\n    mon: 1 daemons, quorum node01 (age 4m)\n    mgr: node01.upxxuf(active, since 4m)\n    osd: 0 osds: 0 up, 0 in\n \n  data:\n    pools:   0 pools, 0 pgs\n    objects: 0 objects, 0 B\n    usage:   0 B used, 0 B \/ 0 B avail\n    pgs:\n<\/code><\/pre>\n\n\n\n<p>Similarly, these are the services\/deamons currently running on the first admin node.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch ps<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME               HOST    PORTS        STATUS        REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  \ncrash.node01       node01               running (4m)     4m ago   4m    7092k        -  18.2.2   1c40e0e88d74  17e2319f10dc  \nmgr.node01.upxxuf  node01  *:9283,8765  running (5m)     4m ago   5m     447M        -  18.2.2   1c40e0e88d74  d1a0585d07b1  \nmon.node01         node01               running (5m)     4m ago   5m    25.2M    2048M  18.2.2   1c40e0e88d74  85890ac8c45f\n<\/code><\/pre>\n\n\n\n<p>As you can see, we ONLY have Crash module, Manager and Monitor services running.<\/p>\n\n\n\n<p>Even when you add other nodes, these are the only services that will be ran on them!<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"add-nodes-into-the-ceph-cluster\">Add Nodes into the Ceph Cluster<\/h4>\n\n\n\n<p>Next, proceed to add your nodes into the cluster, These steps have been provided in our previous guides.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/kifarunix.com\/how-to-deploy-ceph-storage-cluster-on-debian\/#enable-ceph-cli\" target=\"_blank\" rel=\"noreferrer noopener\">Enable Ceph CLI<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kifarunix.com\/how-to-deploy-ceph-storage-cluster-on-debian\/#copy-ssh-keys-to-other-ceph-nodes\" target=\"_blank\" rel=\"noreferrer noopener\">Copy SSH Keys to Other Ceph Nodes<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kifarunix.com\/how-to-deploy-ceph-storage-cluster-on-debian\/#drop-into-ceph-cli\" target=\"_blank\" rel=\"noreferrer noopener\">Drop into Ceph CLI<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kifarunix.com\/how-to-deploy-ceph-storage-cluster-on-debian\/#add-ceph-monitor-node-to-ceph-cluster\" target=\"_blank\" rel=\"noreferrer noopener\">Add Ceph Monitor Node to Ceph Cluster<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kifarunix.com\/how-to-deploy-ceph-storage-cluster-on-debian\/#add-ceph-osd-nodes-to-ceph-cluster\" target=\"_blank\" rel=\"noreferrer noopener\">Add Ceph OSD Nodes to Ceph Cluster<\/a><\/li>\n<\/ul>\n\n\n\n<p>Here are all my nodes, with custom labels, added to cluster already:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch host ls<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>HOST    ADDR            LABELS          STATUS  \nnode01  192.168.122.78  _admin,admin01          \nnode02  192.168.122.79  admin02                 \nnode03  192.168.122.80  osd1                    \nnode04  192.168.122.90  osd2                    \nnode05  192.168.122.91  osd3                    \n5 hosts in cluster\n<\/code><\/pre>\n\n\n\n<p>Also, OSDs have been created.<\/p>\n\n\n\n<p>Services currently running on these nodes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch ps<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME               HOST    PORTS        STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  \ncrash.node01       node01               running (80m)     7m ago  80m    7088k        -  18.2.2   1c40e0e88d74  17e2319f10dc  \ncrash.node02       node02               running (19m)     7m ago  19m    7107k        -  18.2.2   1c40e0e88d74  aa78609c1bf7  \ncrash.node03       node03               running (18m)     7m ago  18m    21.5M        -  18.2.2   1c40e0e88d74  b957dc6b2dec  \ncrash.node04       node04               running (19m)     7m ago  19m    7151k        -  18.2.2   1c40e0e88d74  549db30e0271  \ncrash.node05       node05               running (18m)     7m ago  18m    22.5M        -  18.2.2   1c40e0e88d74  3b7f1d77461a  \nmgr.node01.upxxuf  node01  *:9283,8765  running (81m)     7m ago  81m     479M        -  18.2.2   1c40e0e88d74  d1a0585d07b1  \nmgr.node02.bqzghl  node02  *:8443,8765  running (19m)     7m ago  19m     425M        -  18.2.2   1c40e0e88d74  a473f17a57e8  \nmon.node01         node01               running (81m)     7m ago  81m    51.9M    2048M  18.2.2   1c40e0e88d74  85890ac8c45f  \nmon.node02         node02               running (19m)     7m ago  19m    35.0M    2048M  18.2.2   1c40e0e88d74  7dd8886151ec  \nmon.node03         node03               running (18m)     7m ago  18m    32.0M    2048M  18.2.2   1c40e0e88d74  203db5ded05e  \nmon.node04         node04               running (18m)     7m ago  18m    29.3M    2048M  18.2.2   1c40e0e88d74  17bfaae81be9  \nmon.node05         node05               running (18m)     7m ago  18m    31.6M    2048M  18.2.2   1c40e0e88d74  cf38ec8aa5fc\nosd.0              node05               running (15m)     7m ago  15m    48.8M    4096M  18.2.2   1c40e0e88d74  309e5582f1d9  \nosd.1              node03               running (15m)     7m ago  15m    54.4M    4096M  18.2.2   1c40e0e88d74  82da63b2386e  \nosd.2              node04               running (15m)     7m ago  15m    53.8M    4096M  18.2.2   1c40e0e88d74  bb605719cb20\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"setup-second-ceph-admin-node\">Setup Second Ceph Admin Node<\/h4>\n\n\n\n<p>We will use node02 as our second admin node.<\/p>\n\n\n\n<p>Therefore, install cephadm as well as other ceph tools;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>wget -q -O- 'https:\/\/download.ceph.com\/keys\/release.asc' | \\\nsudo gpg --dearmor -o \/etc\/apt\/trusted.gpg.d\/cephadm.gpg<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>echo 'deb https:\/\/download.ceph.com\/debian-reef\/ $(lsb_release -sc) main' \\\nsudo tee \/etc\/apt\/sources.list.d\/ceph.list<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt update<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt install cephadm ceph-common<\/code><\/pre>\n\n\n\n<p>Designate second node as the second admin node by adding the admin label, <strong>_admin<\/strong>. Run this command, of course, on the first admin node.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch host label add node02 _admin<\/code><\/pre>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p><strong>Note:<\/strong>&nbsp;By default, adding the&nbsp;<code>_admin<\/code>&nbsp;label to additional nodes in the cluster,&nbsp;<code>cephadm<\/code>&nbsp;copies the&nbsp;<code>ceph.conf<\/code>&nbsp;and&nbsp;<code>client.admin<\/code>&nbsp;keyring files to that node.<\/p>\n<\/blockquote>\n\n\n\n<p>You can confirm that the Ceph config and keyrings have been copied;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ls -1 \/etc\/ceph\/*<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\/etc\/ceph\/ceph.client.admin.keyring\n\/etc\/ceph\/ceph.conf\n\/etc\/ceph\/rbdmap\n<\/code><\/pre>\n\n\n\n<p>You should now be able to administer cluster from the second node.<\/p>\n\n\n\n<p>See example command below run on node02;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch ps<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>cephadmin@node02:~$ sudo ceph orch ps\nNAME               HOST    PORTS        STATUS        REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  \ncrash.node01       node01               running (3h)     6m ago   3h    7088k        -  18.2.2   1c40e0e88d74  17e2319f10dc  \ncrash.node02       node02               running (2h)     6m ago   2h    8643k        -  18.2.2   1c40e0e88d74  aa78609c1bf7  \ncrash.node03       node03               running (2h)     6m ago   2h    21.4M        -  18.2.2   1c40e0e88d74  b957dc6b2dec  \ncrash.node04       node04               running (2h)     6m ago   2h    7151k        -  18.2.2   1c40e0e88d74  549db30e0271  \ncrash.node05       node05               running (2h)     6m ago   2h    22.5M        -  18.2.2   1c40e0e88d74  3b7f1d77461a  \nmgr.node01.upxxuf  node01  *:9283,8765  running (3h)     6m ago   3h     483M        -  18.2.2   1c40e0e88d74  d1a0585d07b1  \nmgr.node02.bqzghl  node02  *:8443,8765  running (2h)     6m ago   2h     432M        -  18.2.2   1c40e0e88d74  a473f17a57e8  \nmon.node01         node01               running (3h)     6m ago   3h    83.5M    2048M  18.2.2   1c40e0e88d74  85890ac8c45f  \nmon.node02         node02               running (2h)     6m ago   2h    69.0M    2048M  18.2.2   1c40e0e88d74  7dd8886151ec  \nmon.node03         node03               running (2h)     6m ago   2h    67.4M    2048M  18.2.2   1c40e0e88d74  203db5ded05e  \nmon.node04         node04               running (2h)     6m ago   2h    66.0M    2048M  18.2.2   1c40e0e88d74  17bfaae81be9  \nmon.node05         node05               running (2h)     6m ago   2h    66.0M    2048M  18.2.2   1c40e0e88d74  cf38ec8aa5fc\nosd.0              node05               running (2h)     6m ago   2h    48.8M    4096M  18.2.2   1c40e0e88d74  309e5582f1d9  \nosd.1              node03               running (2h)     6m ago   2h    54.4M    4096M  18.2.2   1c40e0e88d74  82da63b2386e  \nosd.2              node04               running (2h)     6m ago   2h    53.8M    4096M  18.2.2   1c40e0e88d74  bb605719cb20 \n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"deploy-monitoring-stack-on-ceph-cluster\">Deploy Monitoring Stack on Ceph Cluster<\/h4>\n\n\n\n<p>Remember we skipped the default monitoring stack when we boostrapped the cluster. If you need monitoring stack to be highly available as well, then proceed as follows.<\/p>\n\n\n\n<p>So, we will set the Grafana and Prometheus to run on both admin nodes, and let other components such as node exporter, ceph exporter and alert manager to run on all nodes.<\/p>\n\n\n\n<p>You can deploy services using service specification file or simply do it from command line. We will use command line here.<\/p>\n\n\n\n<p>Deploy Grafana and Prometheus on the two admin nodes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch apply grafana --placement=\"node01 node02\"<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch apply prometheus --placement=\"node01 node02\"<\/code><\/pre>\n\n\n\n<p>Confirm the same;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch ps --service_name grafana<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME            HOST    PORTS   STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  \ngrafana.node01  node01  *:3000  running (12s)     5s ago  45s    58.6M        -  9.4.7    954c08fa6188  aea3b826052e  \ngrafana.node02  node02  *:3000  running (11s)     5s ago  43s    58.7M        -  9.4.7    954c08fa6188  5ad7ee7d8e61\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch ps --service_name prometheus<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME            HOST    PORTS   STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  \ngrafana.node01  node01  *:3000  running (43s)    36s ago  76s    58.6M        -  9.4.7    954c08fa6188  aea3b826052e  \ngrafana.node02  node02  *:3000  running (41s)    36s ago  74s    58.7M        -  9.4.7    954c08fa6188  5ad7ee7d8e61\n<\/code><\/pre>\n\n\n\n<p>Deploy other services on all the nodes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>for i in ceph-exporter node-exporter alertmanager; do sudo ceph orch apply $i; done<\/code><\/pre>\n\n\n\n<p>If you check the services again;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch ps<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                  HOST    PORTS        STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  \nalertmanager.node05   node05  *:9093,9094  running (3m)      2m ago   4m    14.8M        -  0.25.0   c8568f914cd2  6b1d5898c410  \nceph-exporter.node01  node01               running (4m)      3m ago   4m    7847k        -  18.2.2   1c40e0e88d74  3e8f1d4ccbbb  \nceph-exporter.node02  node02               running (4m)      3m ago   4m    6836k        -  18.2.2   1c40e0e88d74  6303034fea00  \nceph-exporter.node03  node03               running (4m)      3m ago   4m    6536k        -  18.2.2   1c40e0e88d74  a77a95ccdc46  \nceph-exporter.node04  node04               running (4m)      3m ago   4m    6431k        -  18.2.2   1c40e0e88d74  d258e24b65c0  \nceph-exporter.node05  node05               running (4m)      2m ago   4m    6471k        -  18.2.2   1c40e0e88d74  6c9d33d111ed  \ncrash.node01          node01               running (4h)      3m ago   4h    7088k        -  18.2.2   1c40e0e88d74  17e2319f10dc  \ncrash.node02          node02               running (3h)      3m ago   3h    8939k        -  18.2.2   1c40e0e88d74  aa78609c1bf7  \ncrash.node03          node03               running (3h)      3m ago   3h    21.4M        -  18.2.2   1c40e0e88d74  b957dc6b2dec  \ncrash.node04          node04               running (3h)      3m ago   3h    7151k        -  18.2.2   1c40e0e88d74  549db30e0271  \ncrash.node05          node05               running (3h)      2m ago   3h    22.5M        -  18.2.2   1c40e0e88d74  3b7f1d77461a  \ngrafana.node01        node01  *:3000       running (11m)     3m ago  11m    75.3M        -  9.4.7    954c08fa6188  aea3b826052e  \ngrafana.node02        node02  *:3000       running (11m)     3m ago  11m    74.2M        -  9.4.7    954c08fa6188  5ad7ee7d8e61  \nmgr.node01.upxxuf     node01  *:9283,8765  running (4h)      3m ago   4h     472M        -  18.2.2   1c40e0e88d74  d1a0585d07b1  \nmgr.node02.bqzghl     node02  *:8443,8765  running (3h)      3m ago   3h     429M        -  18.2.2   1c40e0e88d74  a473f17a57e8  \nmon.node01            node01               running (4h)      3m ago   4h    93.0M    2048M  18.2.2   1c40e0e88d74  85890ac8c45f  \nmon.node02            node02               running (3h)      3m ago   3h    77.7M    2048M  18.2.2   1c40e0e88d74  7dd8886151ec  \nmon.node03            node03               running (3h)      3m ago   3h    73.0M    2048M  18.2.2   1c40e0e88d74  203db5ded05e  \nmon.node04            node04               running (3h)      3m ago   3h    71.6M    2048M  18.2.2   1c40e0e88d74  17bfaae81be9  \nmon.node05            node05               running (3h)      2m ago   3h    72.8M    2048M  18.2.2   1c40e0e88d74  cf38ec8aa5fc  \nnode-exporter.node01  node01  *:9100       running (4m)      3m ago   4m    2836k        -  1.5.0    0da6a335fe13  e48cacd54ca1  \nnode-exporter.node02  node02  *:9100       running (4m)      3m ago   4m    3235k        -  1.5.0    0da6a335fe13  fd0597727467  \nnode-exporter.node03  node03  *:9100       running (4m)      3m ago   4m    4263k        -  1.5.0    0da6a335fe13  ce5710d50c54  \nnode-exporter.node04  node04  *:9100       running (4m)      3m ago   4m    3447k        -  1.5.0    0da6a335fe13  e74a6bde46a3  \nnode-exporter.node05  node05  *:9100       running (4m)      2m ago   4m    8640k        -  1.5.0    0da6a335fe13  d2223d92dcca  \nosd.0                 node05               running (3h)      3m ago   3h    48.8M    4096M  18.2.2   1c40e0e88d74  309e5582f1d9  \nosd.1                 node03               running (3h)      3m ago   3h    54.4M    4096M  18.2.2   1c40e0e88d74  82da63b2386e  \nosd.2                 node04               running (3h)      3m ago   3h    53.8M    4096M  18.2.2   1c40e0e88d74  bb605719cb20\nprometheus.node01     node01  *:9095       running (3m)      3m ago  11m    20.7M        -  2.43.0   a07b618ecd1d  ea4cb99f01da  \nprometheus.node02     node02  *:9095       running (3m)      3m ago  11m    22.3M        -  2.43.0   a07b618ecd1d  fa3968b4803b\n<\/code><\/pre>\n\n\n\n<p>You should now be able to administer your cluster using either of the admin nodes with VIP.<\/p>\n\n\n\n<p>See access to dashboard via second admin node.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1623\" height=\"721\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-cluster-via-admin-node-02-IP.png?v=1712699885\" alt=\"\" class=\"wp-image-22107\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-cluster-via-admin-node-02-IP.png?v=1712699885 1623w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-cluster-via-admin-node-02-IP-768x341.png?v=1712699885 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-cluster-via-admin-node-02-IP-1536x682.png?v=1712699885 1536w\" sizes=\"(max-width: 1623px) 100vw, 1623px\" \/><\/figure>\n\n\n\n<p>Similarly, you should access cluster also with the VIP address.<\/p>\n\n\n\n<p>Update the Grafana Prometheus data source address on each admin node to use the VIP to connect to Prometheus service. This is so just in case either of the node is down, it can still pull metrics using a VIP on one of available admin node that has taken cluster managerial roles<\/p>\n\n\n\n<p>On Node01<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo sed -i 's\/\\bnode&#91;0-9]\\+\\b\/192.168.122.200\/g' \/var\/lib\/ceph\/17ef548c-f68b-11ee-9a19-4d1575fdfd98\/grafana.node01\/etc\/grafana\/provisioning\/datasources\/ceph-dashboard.yml<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo systemctl restart ceph-17ef548c-f68b-11ee-9a19-4d1575fdfd98@grafana.node01.service<\/code><\/pre>\n\n\n\n<p>On Node02<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo sed -i 's\/\\bnode&#91;0-9]\\+\\b\/192.168.122.200\/g' \/var\/lib\/ceph\/17ef548c-f68b-11ee-9a19-4d1575fdfd98\/grafana.node02\/etc\/grafana\/provisioning\/datasources\/ceph-dashboard.yml<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo systemctl restart ceph-17ef548c-f68b-11ee-9a19-4d1575fdfd98@grafana.node02.service<\/code><\/pre>\n\n\n\n<p>See sample Grafana dashboards via the VIP;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1608\" height=\"715\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-cluster-via-admin-ha-IP-1.png\" alt=\"\" class=\"wp-image-22106\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-cluster-via-admin-ha-IP-1.png?v=1712699847 1608w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-cluster-via-admin-ha-IP-1-768x341.png?v=1712699847 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-cluster-via-admin-ha-IP-1-1536x683.png?v=1712699847 1536w\" sizes=\"(max-width: 1608px) 100vw, 1608px\" \/><\/figure>\n\n\n\n<p>And that is it!<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"updating-existing-ceph-cluster-with-admin-node-ha-setup\">Updating existing Ceph cluster with Admin Node HA setup<\/h3>\n\n\n\n<p>If you have already deployed Ceph cluster with a single admin node and you want to update the setup such that you have two admin nodes for high availability, then proceed as follows.<\/p>\n\n\n\n<p>Let's check current Ceph cluster status;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph -s<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>  cluster:\n    id:     17ef548c-f68b-11ee-9a19-4d1575fdfd98\n    health: HEALTH_OK\n \n  services:\n    mon: 5 daemons, quorum node01,node02,node05,node03,node04 (age 19m)\n    mgr: node02.bqzghl(active, since 44h), standbys: node01.upxxuf\n    osd: 3 osds: 3 up (since 44h), 3 in (since 44h)\n \n  data:\n    pools:   4 pools, 97 pgs\n    objects: 296 objects, 389 MiB\n    usage:   576 MiB used, 299 GiB \/ 300 GiB avail\n    pgs:     97 active+clean\n \n  io:\n    client:   170 B\/s rd, 170 B\/s wr, 0 op\/s rd, 0 op\/s wr\n<\/code><\/pre>\n\n\n\n<p>Also see current services\/daemon distribution across the cluster nodes.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch ps<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                  HOST    PORTS        STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  \nalertmanager.node05   node05  *:9093,9094  running (45h)     4m ago  45h    14.3M        -  0.25.0   c8568f914cd2  6b1d5898c410  \nceph-exporter.node01  node01               running (45h)   100s ago  45h    16.7M        -  18.2.2   1c40e0e88d74  3e8f1d4ccbbb  \nceph-exporter.node02  node02               running (45h)    99s ago  45h    16.3M        -  18.2.2   1c40e0e88d74  6303034fea00  \nceph-exporter.node03  node03               running (45h)     4m ago  45h    17.7M        -  18.2.2   1c40e0e88d74  a77a95ccdc46  \nceph-exporter.node04  node04               running (45h)     4m ago  45h    17.6M        -  18.2.2   1c40e0e88d74  d258e24b65c0  \nceph-exporter.node05  node05               running (45h)     4m ago  45h    17.6M        -  18.2.2   1c40e0e88d74  6c9d33d111ed  \ncrash.node01          node01               running (2d)    100s ago   2d    6891k        -  18.2.2   1c40e0e88d74  17e2319f10dc  \ncrash.node02          node02               running (2d)     99s ago   2d    7659k        -  18.2.2   1c40e0e88d74  aa78609c1bf7  \ncrash.node03          node03               running (2d)      4m ago   2d    21.1M        -  18.2.2   1c40e0e88d74  b957dc6b2dec  \ncrash.node04          node04               running (2d)      4m ago   2d    7959k        -  18.2.2   1c40e0e88d74  549db30e0271  \ncrash.node05          node05               running (2d)      4m ago   2d    22.8M        -  18.2.2   1c40e0e88d74  3b7f1d77461a  \ngrafana.node02        node02  *:3000       running (11m)    99s ago  45h    77.7M        -  9.4.7    954c08fa6188  fbba2415751e  \nmgr.node01.upxxuf     node01  *:9283,8765  running (2d)    100s ago   2d     448M        -  18.2.2   1c40e0e88d74  d1a0585d07b1  \nmgr.node02.bqzghl     node02  *:8443,8765  running (2d)     99s ago   2d     581M        -  18.2.2   1c40e0e88d74  a473f17a57e8  \nmon.node01            node01               running (2d)    100s ago   2d     423M    2048M  18.2.2   1c40e0e88d74  85890ac8c45f  \nmon.node02            node02               running (2d)     99s ago   2d     420M    2048M  18.2.2   1c40e0e88d74  7dd8886151ec  \nmon.node03            node03               running (2d)      4m ago   2d     422M    2048M  18.2.2   1c40e0e88d74  203db5ded05e  \nmon.node04            node04               running (2d)      4m ago   2d     428M    2048M  18.2.2   1c40e0e88d74  17bfaae81be9  \nmon.node05            node05               running (2d)      4m ago   2d     423M    2048M  18.2.2   1c40e0e88d74  cf38ec8aa5fc  \nnode-exporter.node01  node01  *:9100       running (45h)   100s ago  45h    9484k        -  1.5.0    0da6a335fe13  e48cacd54ca1  \nnode-exporter.node02  node02  *:9100       running (45h)    99s ago  45h    9883k        -  1.5.0    0da6a335fe13  fd0597727467  \nnode-exporter.node03  node03  *:9100       running (45h)     4m ago  45h    12.6M        -  1.5.0    0da6a335fe13  ce5710d50c54  \nnode-exporter.node04  node04  *:9100       running (45h)     4m ago  45h    10.3M        -  1.5.0    0da6a335fe13  e74a6bde46a3  \nnode-exporter.node05  node05  *:9100       running (45h)     4m ago  45h    10.4M        -  1.5.0    0da6a335fe13  d2223d92dcca  \nosd.0                 node05               running (44h)     4m ago  44h     146M    4096M  18.2.2   1c40e0e88d74  309e5582f1d9  \nosd.1                 node03               running (44h)     4m ago  44h     158M    4096M  18.2.2   1c40e0e88d74  82da63b2386e  \nosd.2                 node04               running (44h)     4m ago  44h     151M    4096M  18.2.2   1c40e0e88d74  bb605719cb20  \nprometheus.node01     node01  *:9095       running (43h)   100s ago  45h    83.6M        -  2.43.0   a07b618ecd1d  3f489e6e48ed\n<\/code><\/pre>\n\n\n\n<p>Random services\/daemons running on random nodes! In this setup, if the Ceph admin goes down, you wont be able to administer your ceph cluster.<\/p>\n\n\n\n<p>Therefore:<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"designate-second-node-as-admin-node\">Designate Second Node as Admin Node<\/h4>\n\n\n\n<p>To setup another node as admin on a Ceph cluster, you have to add the <strong>_admin<\/strong> label to that node.<\/p>\n\n\n\n<p>Refer to how to <a href=\"https:\/\/kifarunix.com\/how-to-setup-ceph-admin-node-in-high-availability#setup-second-ceph-admin-node\" target=\"_blank\" rel=\"noreferrer noopener\">Setup Second Ceph Admin Node section<\/a> above.<\/p>\n\n\n\n<p>Now, we have designated node02 as admin node as well;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch host ls<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>HOST    ADDR            LABELS          STATUS  \nnode01  192.168.122.78  _admin,admin01          \nnode02  192.168.122.79  admin02,_admin          \nnode03  192.168.122.80  osd1                    \nnode04  192.168.122.90  osd2                    \nnode05  192.168.122.91  osd3                    \n5 hosts in cluster\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"update-monitoring-stack\">Update Monitoring Stack<\/h4>\n\n\n\n<p>If you enabled monitoring stack in your Ceph cluster, you need to update them just to ensure that, at least Grafana and Prometheus are reachable via the admin nodes VIP.<\/p>\n\n\n\n<p>Grafana and Prometheus are scheduled to run on random nodes.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch ps --service_name grafana<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch ps --service_name prometheus<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME            HOST    PORTS   STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  \ngrafana.node02  node02  *:3000  running (58m)     6m ago  46h    78.3M        -  9.4.7    954c08fa6188  fbba2415751e\nprometheus.node01  node01  *:9095  running (44h)     6m ago  46h    85.1M        -  2.43.0   a07b618ecd1d  3f489e6e48e\n<\/code><\/pre>\n\n\n\n<p>See, we only have one instance of each running. So, if one of the nodes go down, one of the services may not be accessible and hence, no visibility into the cluster.<\/p>\n\n\n\n<p>Therefore, set them to run on both admin nodes.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch apply grafana --placement=\"node01 node02\"<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch apply prometheus --placement=\"node01 node02\"<\/code><\/pre>\n\n\n\n<p>Once the updated is complete, verify the same;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch ps --service_name prometheus<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME               HOST    PORTS   STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  \nprometheus.node01  node01  *:9095  running (44h)     3m ago  46h    86.5M        -  2.43.0   a07b618ecd1d  3f489e6e48ed  \nprometheus.node02  node02  *:9095  running (4m)      3m ago   4m    27.6M        -  2.43.0   a07b618ecd1d  cd686cf8041c\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ceph orch ps --service_name grafana<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME            HOST    PORTS   STATUS        REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  \ngrafana.node01  node01  *:3000  running (4m)     4m ago   4m    59.1M        -  9.4.7    954c08fa6188  673eab1114dc  \ngrafana.node02  node02  *:3000  running (4m)     4m ago  46h    57.1M        -  9.4.7    954c08fa6188  017761188c91\n<\/code><\/pre>\n\n\n\n<p>Next, on both admin nodes, update the Grafana dashboard Prometheus datasource address to use the VIP address.<\/p>\n\n\n\n<p>On Node01<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo sed -i 's\/\\bnode&#91;0-9]\\+\\b\/192.168.122.200\/g' \/var\/lib\/ceph\/17ef548c-f68b-11ee-9a19-4d1575fdfd98\/grafana.node01\/etc\/grafana\/provisioning\/datasources\/ceph-dashboard.yml<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo systemctl restart ceph-17ef548c-f68b-11ee-9a19-4d1575fdfd98@grafana.node01.service<\/code><\/pre>\n\n\n\n<p>On Node02<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo sed -i 's\/\\bnode&#91;0-9]\\+\\b\/192.168.122.200\/g' \/var\/lib\/ceph\/17ef548c-f68b-11ee-9a19-4d1575fdfd98\/grafana.node02\/etc\/grafana\/provisioning\/datasources\/ceph-dashboard.yml<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo systemctl restart ceph-17ef548c-f68b-11ee-9a19-4d1575fdfd98@grafana.node02.service<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"verify-access-to-ceph-cluster-dashboard-and-monitoring-dashboard-via-vip\">Verify Access to Ceph Cluster Dashboard and Monitoring Dashboard via VIP<\/h4>\n\n\n\n<p>At this point, you should now be able to access the Ceph cluster dashboard as well as the Grafana\/Prometheus via the VIP. Note that if both admin nodes (manager nodes) are available, if you try to use VIP, you will be redirected to one of the admin nodes.<\/p>\n\n\n\n<p>Grafana Dashboard via our VIP, 192.168.122.200;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1612\" height=\"803\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-grafana-dashboard-via-VIP.png?v=1712861141\" alt=\"\" class=\"wp-image-22120\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-grafana-dashboard-via-VIP.png?v=1712861141 1612w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-grafana-dashboard-via-VIP-768x383.png?v=1712861141 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-grafana-dashboard-via-VIP-1536x765.png?v=1712861141 1536w\" sizes=\"(max-width: 1612px) 100vw, 1612px\" \/><\/figure>\n\n\n\n<p>Prometheus Dashboard via our VIP, 192.168.122.200;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1616\" height=\"822\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-prometheus-dashboard-via-VIP.png?v=1712861180\" alt=\"\" class=\"wp-image-22121\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-prometheus-dashboard-via-VIP.png?v=1712861180 1616w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-prometheus-dashboard-via-VIP-768x391.png?v=1712861180 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/ceph-prometheus-dashboard-via-VIP-1536x781.png?v=1712861180 1536w\" sizes=\"(max-width: 1616px) 100vw, 1616px\" \/><\/figure>\n\n\n\n<p>To test the Ceph functionality with one of the admin nodes down, Let me take down admin node01.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo ip link set enp1s0 down<\/code><\/pre>\n\n\n\n<p>Then node02 becomes our VIP;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>root@node02:~# ip -br a\nlo               UNKNOWN        127.0.0.1\/8 ::1\/128 \nenp1s0           UP             192.168.122.79\/24 192.168.122.200\/24 fe80::5054:ff:fe35:3981\/64 \ndocker0          DOWN           172.17.0.1\/16 \n<\/code><\/pre>\n\n\n\n<p>Ceph Status from node02;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>root@node02:~# ceph -s\n  cluster:\n    id:     17ef548c-f68b-11ee-9a19-4d1575fdfd98\n    health: HEALTH_WARN\n            1 hosts fail cephadm check\n            1\/5 mons down, quorum node02,node05,node03,node04\n \n  services:\n    mon: 5 daemons, quorum node02,node05,node03,node04 (age 34s), out of quorum: node01\n    mgr: node02.bqzghl(active, since 46h), standbys: node01.upxxuf\n    osd: 3 osds: 3 up (since 45h), 3 in (since 45h)\n \n  data:\n    pools:   4 pools, 97 pgs\n    objects: 296 objects, 389 MiB\n    usage:   576 MiB used, 299 GiB \/ 300 GiB avail\n    pgs:     97 active+clean\n \n  io:\n    client:   170 B\/s rd, 170 B\/s wr, 0 op\/s rd, 0 op\/s wr\n<\/code><\/pre>\n\n\n\n<p>Access to Dashboard;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1621\" height=\"616\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/access-dashboard-via-VIP-ceph.png?v=1712861662\" alt=\"\" class=\"wp-image-22124\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/access-dashboard-via-VIP-ceph.png?v=1712861662 1621w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/access-dashboard-via-VIP-ceph-768x292.png?v=1712861662 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/access-dashboard-via-VIP-ceph-1536x584.png?v=1712861662 1536w\" sizes=\"(max-width: 1621px) 100vw, 1621px\" \/><\/figure>\n\n\n\n<p>I have also <a href=\"https:\/\/kifarunix.com\/part-1-integrate-openstack-with-ceph-storage-cluster\/\" target=\"_blank\" rel=\"noreferrer noopener\">integrated my Openstack with Ceph cluster<\/a>. So I tested creating an instance with volume and an image on OpenStack with one admin node down and it worked as usual!<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"conclusion\">Conclusion<\/h3>\n\n\n\n<p>That concludes our guide on configuring a highly available Ceph storage cluster admin node.<\/p>\n\n\n\n<p>Read more on <a href=\"https:\/\/docs.ceph.com\/en\/latest\/cephadm\/host-management\/\" target=\"_blank\" rel=\"noreferrer noopener\">Ceph host management<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this tutorial, you will learn how to setup Ceph Admin node in high availability. In most setups, Ceph cluster usually runs with a single<\/p>\n","protected":false},"author":10,"featured_media":22128,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[39,1338,121],"tags":[7438,7439,7436,7437],"class_list":["post-22053","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-storage","category-ceph","category-howtos","tag-ceph-cluster","tag-ceph-cluster-ha","tag-configure-ceph-admin-node-in-high-availability","tag-higly-available-ceph-admin-node","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/22053"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=22053"}],"version-history":[{"count":26,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/22053\/revisions"}],"predecessor-version":[{"id":22129,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/22053\/revisions\/22129"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/22128"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=22053"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=22053"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=22053"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}