{"id":13326,"date":"2022-06-30T00:58:23","date_gmt":"2022-06-29T21:58:23","guid":{"rendered":"https:\/\/kifarunix.com\/?p=13326"},"modified":"2024-03-09T14:54:18","modified_gmt":"2024-03-09T11:54:18","slug":"setup-multinode-elasticsearch-8-x-cluster","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/setup-multinode-elasticsearch-8-x-cluster\/","title":{"rendered":"Setup Multinode Elasticsearch 8.x Cluster"},"content":{"rendered":"<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"617\" height=\"317\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/07\/elasticsearch-cluster.png\" alt=\"Setup Multinode Elasticsearch 8.x Cluster\" class=\"wp-image-8931\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/07\/elasticsearch-cluster.png 617w, https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/07\/elasticsearch-cluster-150x77.png 150w, https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/07\/elasticsearch-cluster-300x154.png 300w\" sizes=\"(max-width: 617px) 100vw, 617px\" \/><\/figure><\/div>\n\n\n<p>Follow through this tutorial to learn how to setup multinode Elasticsearch 8.x cluster. As of this writing, Elastic Stack 8.3 is the current release. This means that <a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/release-highlights.html\" target=\"_blank\" rel=\"noreferrer noopener\">Elasticsearch 8.3<\/a>, one of the major components of the Elastics Stack is also the current release version as of this writing.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#setup-multinode-elasticsearch-8-x-cluster\">Setup Multinode Elasticsearch 8.x Cluster<\/a><ul><li><a href=\"#install-elasticsearch-8-x-on-all-cluster-nodes\">Install Elasticsearch 8.x on All Cluster Nodes<\/a><\/li><li><a href=\"#configure-elasticsearch-8-x\">Configure Elasticsearch 8.x<\/a><ul><li><a href=\"#set-the-name-of-the-cluster-on-all-cluster-nodes\">Set the Name of the Cluster on All Cluster Nodes<\/a><\/li><li><a href=\"#set-the-node-name-on-each-of-the-cluster-nodes\">Set the Node Name on Each of the cluster Nodes;<\/a><\/li><li><a href=\"#define-the-roles-of-elasticsearch-node\">Define the Roles of Elasticsearch Node<\/a><\/li><li><a href=\"#enable-memory-lock\">Enable Memory Lock<\/a><\/li><li><a href=\"#bind-elasticsearch-to-non-loopback-address\">Bind Elasticsearch to Non-Loopback Address<\/a><\/li><li><a href=\"#discovery-and-cluster-formation-settings\">Discovery and Cluster Formation settings<\/a><\/li><li><a href=\"#configure-elasticsearch-cluster-https-connection\">Configure Elasticsearch Cluster HTTPS Connection<\/a><\/li><\/ul><\/li><li><a href=\"#configure-other-important-elasticsearch-systems-settings\">Configure Other Important Elasticsearch Systems Settings<\/a><ul><li><a href=\"#disable-memory-swapping-on-all-cluster-nodes\">Disable Memory Swapping on All Cluster Nodes<\/a><\/li><li><a href=\"#set-jvm-heap-size-on-all-cluster-nodes\">Set JVM Heap Size on All Cluster Nodes<\/a><\/li><li><a href=\"#set-maximum-open-file-descriptor-on-all-cluster-nodes\">Set maximum Open File Descriptor on All Cluster Nodes<\/a><\/li><li><a href=\"#update-virtual-memory-settings-on-all-cluster-nodes\">Update Virtual Memory Settings on All Cluster Nodes<\/a><\/li><li><a href=\"#start-and-enable-elasticsearch-service-on-node-01\">Start and Enable Elasticsearch Service on Node 01<\/a><\/li><\/ul><\/li><li><a href=\"#enroll-other-nodes-into-elasticsearch-cluster\">Enroll Other Nodes into Elasticsearch Cluster<\/a><ul><li><a href=\"#generate-elasticsearch-cluster-enrollment-token\">Generate Elasticsearch Cluster Enrollment Token<\/a><\/li><li><a href=\"#start-elasticsearch-on-other-nodes\">Start Elasticsearch on Other Nodes<\/a><\/li><li><a href=\"#discovery-and-cluster-formation-settings-1\">Discovery and Cluster Formation settings<\/a><\/li><li><a href=\"#check-the-cluster-nodes\">Check the Cluster Nodes<\/a><\/li><\/ul><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"setup-multinode-elasticsearch-8-x-cluster\">Setup Multinode Elasticsearch 8.x Cluster<\/h2>\n\n\n\n<p>In our previous tutorial, we learnt how to <a href=\"https:\/\/kifarunix.com\/setup-multi-node-elasticsearch-cluster\/\" target=\"_blank\" rel=\"noreferrer noopener\">setup a three node Elasticsearch 7.x cluster<\/a>.<\/p>\n\n\n\n<p>We will as well be configuring a three node Elasticsearch 8.x cluster in this tutorial.<\/p>\n\n\n\n<p>My Environment:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Node 1: es-node01.kifarunix-demo.com<\/li>\n\n\n\n<li>Node 2: es-node02.kifarunix-demo.com<\/li>\n\n\n\n<li>Node 3: es-node03.kifarunix-demo.com<\/li>\n<\/ul>\n\n\n\n<p>Ensure that the hostnames are resolvable on each node. If you do not have a DNS server, then you can use your hosts file.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>192.168.122.12 es-node01.kifarunix-demo.com es-node01\n192.168.122.73 es-node02.kifarunix-demo.com es-node02\n192.168.122.50 es-node03.kifarunix-demo.com es-node03<\/code><\/pre>\n\n\n\n<p>With the release of the Elastic 8.x, there are some few changes in Elasticsearch setup procedure.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-elasticsearch-8-x-on-all-cluster-nodes\">Install Elasticsearch 8.x on All Cluster Nodes<\/h3>\n\n\n\n<p>You need to install the same version of Elasticsearch 8.x on all the cluster nodes.<\/p>\n\n\n\n<p>In this tutorial, we will be using Ubuntu 22.04 system.<\/p>\n\n\n\n<p>Thus, to install Elasticsearch 8.x on Ubuntu, you need to install the Elastic APT repositories as follows;<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>apt install apt-transport-https \\\n\tca-certificates \\\n\tcurl \\\n\tgnupg2 \\\n\tsoftware-properties-common\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>wget -qO - https:\/\/artifacts.elastic.co\/GPG-KEY-elasticsearch | \\\n\tgpg --dearmor -o \/etc\/apt\/trusted.gpg.d\/elastic.gpg<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"deb https:\/\/artifacts.elastic.co\/packages\/8.x\/apt stable main\" &gt; \\\n\/etc\/apt\/sources.list.d\/elastic-8.x.list<\/code><\/pre>\n\n\n\n<p>Run system update;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>apt update<\/code><\/pre>\n\n\n\n<p>Once the repos are in place, install Elasticsearch 8.x on all the cluster nodes using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>apt install elasticsearch<\/code><\/pre>\n\n\n\n<p>During the installation, the Security features will be enabled by default;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Authentication and authorization are enabled.<\/li>\n\n\n\n<li>TLS for the transport and HTTP layers is enabled and configured. Self-signed SSL certs are generated and used.<\/li>\n\n\n\n<li>Elastic super user account (elastic) and its password is created.<\/li>\n\n\n\n<li>Elasticsearch is configured as a single node cluster.<\/li>\n<\/ul>\n\n\n\n<p>Sample installation output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>...\nSetting up elasticsearch (8.11.1) ...\n--------------------------- Security autoconfiguration information ------------------------------\n\nAuthentication and authorization are enabled.\nTLS for the transport and HTTP layers is enabled and configured.\n\nThe generated password for the elastic built-in superuser is : 2BEkVDLjbN77ODLsTGmA\n\nIf this node should join an existing cluster, you can reconfigure this with\n'\/usr\/share\/elasticsearch\/bin\/elasticsearch-reconfigure-node --enrollment-token <token-here>'\nafter creating an enrollment token on your existing cluster.\n\nYou can complete the following actions at any time:\n\nReset the password of the elastic built-in superuser with \n'\/usr\/share\/elasticsearch\/bin\/elasticsearch-reset-password -u elastic'.\n\nGenerate an enrollment token for Kibana instances with \n '\/usr\/share\/elasticsearch\/bin\/elasticsearch-create-enrollment-token -s kibana'.\n\nGenerate an enrollment token for Elasticsearch nodes with \n'\/usr\/share\/elasticsearch\/bin\/elasticsearch-create-enrollment-token -s node'.\n\n-------------------------------------------------------------------------------------------------\n### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd\n sudo systemctl daemon-reload\n sudo systemctl enable elasticsearch.service\n### You can start elasticsearch service by executing\n sudo systemctl start elasticsearch.service\n<\/code><\/pre>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p><strong>DO NOT start Elasticsearch service yet<\/strong>!<\/p>\n<\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configure-elasticsearch-8-x\">Configure Elasticsearch 8.x<\/h3>\n\n\n\n<p>Open the Elasticsearch configuration file for editing;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/elasticsearch\/elasticsearch.yml<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-the-name-of-the-cluster-on-all-cluster-nodes\">Set the Name of the Cluster on All Cluster Nodes<\/h4>\n\n\n\n<p>Optionally set the name of the cluster on each Node;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n# ---------------------------------- Cluster -----------------------------------\n#\n# Use a descriptive name for your cluster:\n#\n<strong>cluster.name: kifarunix-demo\n<\/strong>...\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-the-node-name-on-each-of-the-cluster-nodes\">Set the Node Name on Each of the cluster Nodes;<\/h4>\n\n\n\n<pre class=\"scroll-box\"><code>\n# ------------------------------------ Node ------------------------------------\n#\n# Use a descriptive name for the node:\n#\n<strong>node.name: es-node01<\/strong>\n...\n<\/code><\/pre>\n\n\n\n<p>You can add other custom attributes if you want.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"define-the-roles-of-elasticsearch-node\">Define the Roles of Elasticsearch Node<\/h4>\n\n\n\n<p>As stated above, you can assign each node a respective role as&nbsp;<strong>master<\/strong>, data node, ingest node, coordinating node.. In this setup, we will configure all the three nodes to act as both&nbsp;<strong><code>master<\/code><\/strong>&nbsp;and&nbsp;<strong><code>data<\/code><\/strong>&nbsp;node to make the cluster resilient to the loss of any single node.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>...\n# ------------------------------------ Node ------------------------------------\n#\n# Use a descriptive name for the node:\n#\nnode.name: es-node01\n<strong>node.roles: [ master, data ]\n<\/strong>...\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"enable-memory-lock\">Enable Memory Lock<\/h4>\n\n\n\n<p>To ensure good performance of Elasticsearch, you need to&nbsp;<a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/master\/setup-configuration-memory.html\" target=\"_blank\" rel=\"noreferrer noopener\">disable memory swapping<\/a>&nbsp;is by enabling memory lock. Hence, uncomment the line&nbsp;<strong>bootstrap.memory_lock: true<\/strong>. This is one of the many ways of disabling swappiness.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n...\n# ----------------------------------- Memory -----------------------------------\n#\n# Lock the memory on startup:\n#\n<strong>bootstrap.memory_lock: true\n<\/strong>...\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"bind-elasticsearch-to-non-loopback-address\">Bind Elasticsearch to Non-Loopback Address<\/h4>\n\n\n\n<p>Elasticsearch binds to loopback addresses&nbsp;by default. For a node to form a cluster, you need to bind it to non-loopback address. This can be done by setting the IP address of the node as the value of&nbsp;<strong>network.host<\/strong>.<\/p>\n\n\n\n<p>Node 01<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>network.host: 192.168.56.125<\/code><\/pre>\n\n\n\n<p>Update the rest of the nodes, accordingly.<\/p>\n\n\n\n<p>Elasticsearch by default uses TCP port 9200 to expose REST APIs. TCP port 9300-9400 is used for node communication;<\/p>\n\n\n\n<p>You can comment out the port definition;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>http.port: 9200<\/code><\/pre>\n\n\n\n<p>The <strong><code>http.host<\/code><\/strong> is set to 0.0.0.0 by default. Since we have bound ES to an interface IP, which MUST be static in this case, comment out this line;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Allow HTTP API connections from anywhere\n# Connections are encrypted and require user authentication\n<strong>#http.host: 0.0.0.0<\/strong><\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"discovery-and-cluster-formation-settings\">Discovery and Cluster Formation settings<\/h4>\n\n\n\n<p>There are two important discovery and cluster formation settings that should be configured before going to production so that nodes in the cluster can discover each other and elect a master node;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>discovery.seed_hosts<\/strong><\/li>\n<\/ul>\n\n\n\n<p><strong>discovery.seed_hosts<\/strong>&nbsp;setting provides a list of master-eligible nodes in the cluster. Each value has the format&nbsp;<code>host:port<\/code>&nbsp;or&nbsp;<code>host<\/code>, where&nbsp;<code>port<\/code>&nbsp;defaults to the setting&nbsp;<code>transport.profiles.default.port<\/code>. This setting was previously known as&nbsp;<strong>discovery.zen.ping.unicast.hosts<\/strong>.<\/p>\n\n\n\n<p>Configure this setting on all Nodes.<\/p>\n\n\n\n<p><strong>HOWEVER<\/strong>, due to auto-configuration of Elasticsearch during the installation, we will need to be able to start Elasticsearch service on each node before we can join them to the cluster.<\/p>\n\n\n\n<p>As a result, we will skip the configuration of this setting for now.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>cluster.initial_master_nodes<\/strong><\/li>\n<\/ul>\n\n\n\n<p><strong>cluster.initial_master_nodes<\/strong>&nbsp;setting defines the initial set of master-eligible nodes. This is important when starting an Elasticsearch cluster for the very first time. After the cluster has formed, remove this setting from each node\u2019s configuration. The value of this setting <strong>MUST<\/strong> match the value of <strong><code>node.name<\/code><\/strong>.<\/p>\n\n\n\n<p>Note that during the installation, Elasticsearch is auto-configured as a single node cluster. For example on Node01.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n...\n# Create a new cluster with the current node only\n# Additional nodes can still join the cluster later\n<strong>cluster.initial_master_nodes: [\"es-node01.kifarunix-demo.com\"]<\/strong>\n...\n<\/code><\/pre>\n\n\n\n<p>Update this line such that the name of the node matches the value of <strong>node.name<\/strong>.<\/p>\n\n\n\n<p>For example, in setup, this line should be;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>cluster.initial_master_nodes: &#91;\"es-node01\"]<\/strong><\/code><\/pre>\n\n\n\n<p><a href=\"#cluster-formation\">See Discovery and Cluster Formation settings below.<\/a><\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-elasticsearch-cluster-https-connection\">Configure Elasticsearch Cluster HTTPS Connection<\/h4>\n\n\n\n<p>By default, Elasticsearch 8.x is auto-configured with self-signed SSL certitificates for both the Transport (connection between the nodes) and HTTP (HTTP API client connections, such as Kibana, Logstash, and Agents).<\/p>\n\n\n\n<p>Later when you add other nodes to the cluster, all the security auto-configurations will be removed on those nodes being added to a cluster and <em>certificates from first node copied over to the node being enrolled<\/em>.<\/p>\n\n\n\n<p>For communication between the nodes, you should see such configurations on <strong><code>elasticsearch.yml<\/code><\/strong>.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n# Enable encryption and mutual authentication between cluster nodes\nxpack.security.transport.ssl:\n  enabled: true\n  verification_mode: certificate\n  keystore.path: certs\/transport.p12\n  truststore.path: certs\/transport.p12\n<\/code><\/pre>\n\n\n\n<p>For HTTP API client connections, such as Kibana, Logstash, and Agents;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>xpack.security.http.ssl:\n  enabled: true\n  keystore.path: certs\/http.p12<\/code><\/pre>\n\n\n\n<p>Save changes made and exit the file.<\/p>\n\n\n\n<p>Kindly note that the CA certificate is generated and stored as <strong><code>\/etc\/elasticsearch\/certs\/http_ca.crt<\/code><\/strong>.<\/p>\n\n\n\n<p>The CA key for the CA certificate, is stored in the <strong><code>\/etc\/elasticsearch\/certs\/http.p12 file<\/code><\/strong>. This file is password protected and the password used to protect it is found in the Elasticsearch Keystore.<\/p>\n\n\n\n<p>Thus, to get the CA key, you first need to retrieve the password that was used to protect it by executing the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/usr\/share\/elasticsearch\/bin\/elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password<\/code><\/pre>\n\n\n\n<p>The command will print the Keystore password to standard output.<\/p>\n\n\n\n<p>Once you have the Keystore password, extract the CA key;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>openssl pkcs12 -in \/etc\/elasticsearch\/certs\/http.p12 -nocerts\t-nodes<\/code><\/pre>\n\n\n\n<p>You will be prompted to enter the password. Use the one retrieved above.<\/p>\n\n\n\n<p>The command will print two keys each with a friendlyName, e.g, <strong><code>friendlyName: http_ca<\/code><\/strong> and     <strong><code>friendlyName: http<\/code><\/strong>. <\/p>\n\n\n\n<pre class=\"scroll-box\"><code>Bag Attributes\n    <strong>friendlyName: http_ca<\/strong>\n    localKeyID: 54 69 6D 65 20 31 37 30 30 37 35 38 35 31 39 32 33 39 \nKey Attributes: &lt;No Attributes>\n-----BEGIN PRIVATE KEY-----\nMIIJQQIBADANBgkqhkiG9w0BAQEFAASCCSswggknAgEAAoICAQC4y7ivLZ2UJJqp\n9xKj2q5yWO6RFSXoJo92fNtaVdfu4QULNLSn540Z4nGE+pjkP1u15\/H5mFzQLQQ0\nrNvoZUxGgzmOLmo8eXsY3iyh5Q3suVNfYY3v+zTmFbD6A7f87MKLLSU6ee2ojdFl\nOKT6j00DgmqaPYuqd6CYY7FP2\/3asAiSZC9o48RplvkSRBx4Hda7SpOAzlPDGgcm\nvnWQqf0Nd+TH7Y8M2Cn+\/\/95YVuNOL03kaXeoZixjPK3zWpYCc6nVRQkR5e9ihbf\nWWCBPtJq6nKRkZXWCpE8VsEwI0Pd\/0z5aqqXgM9r7\/YaOABZksYpX0HR49y8SkU2\nC28BMN8X3229Zy2tkdYbGUOaGMuzhb\/ce+SsUZDN43gYNiEcgIxNe11Xp+R0dbwc\nLkolkoYNddSqi75Vem2PsdSLTR4z1xn7BWf7BxP+PY2\/jcvVD4NDNAMDWwKAIW+P\nqBrsJizYFLWBz0p280XI6Vm+4KMfvF9wI6wCA4UdLJBvK6Syh3CaJ\/27ufiFLgQ3\nxlxjY8dmWxAp8rW3pPF00wwcYF6MBjC8uGijga1fgvqjlz9qJqk0\/CK2OGXRQdyK\nNBh+YmYJccrShxh3wPE42S1NTsHNhiaKub6TbBn6ANKAV8luq425buH\/qZhyNBy+\nAP5ewwCF41H0U8eKOycbmmodYhX2QwIDAQABAoICABbv\/TCL8kSdITAk9toCdCdw\nBSIFBKYtgx6AnkmMNjf0aPKHIZVlwbc2IEO9Rz+tjZilUv0EKHZpNV3EcaywRijK\nc65uRx7ShVuvgs1lkmDmcAQle79kkvWsOOy6NsWHgxj7YDpDQCNCgdHql1pyccC\/\nGLRPHLLqvl2r93+jctMI3pzXyqVeVYGLDRzIpNqpTtSt4Q6FOGhoe\/1seUic5kgc\nWvgQnfJI3ThFq6meqAUoxx51Tv1fWYX6\/3WeYwDNLCrLqKsIJ+bi6QvixX5wFHYo\n5lJN8SdBkIgeIe9JBqIw5Ou14w7Ycjvs68M9XZhrW3rEfssABDoVje8IuZ065Qs\/\nXFWFhPKQwkxyhkgcYnthbydI+BRWXSNPm6cfzag08brao9Ozzm6R9+Zj9J4SoH01\nkWaECWbEFk+SZfjxDlUozpSIBmFcRA9crHqGKXqnUrBHNWgd+aqlgt\/kxf0Nd2Qi\nNiLyXO9YMZCS7h7mkXTadHPt+rD+PZubXGYEGQP5LgwQqJ7knxhP\/S0wpdqblOXl\nI79MT6vdKCWEKUyh79sHfDgbTa8VnzALLUme\/PzL8\/zR+TUMTuY7fxYWSGNSoXj5\nbrinxHpyMogeSXD9h\/49Fk8ENjPhIVneQj16q\/JR8+3yB1lcZsTaB5sqcRQCUkvg\nnyqUtSMCsukLfqL0grghAoIBAQDsHobEPD0O5OaiD\/q+0wrDZLdMR7iFygMsjPVE\nzvFcwtytnmDDFuZGoTLK6n3XuamTIDDLAiGQDqA7OgADNmFGS\/Sxq3Br1wlEJ6uK\nEklCGq6+6CVZCEwf10JfVYa8MBTVIBZoR5N6E1hymvtXAV\/XoT09ubXauHxZDaXB\nVbjyUdZ3LsQVPB0AaKN3MTd86nIWl14CbaIPg+0QeZ4M4p1GJ3P+vHaRil4jFQJI\ncPINm1JDYuSvj876UwxvUycX9uR01Ybp9mmMAgVgZWqTOY1DeM+iu0xWjyTjOqI7\nTByTm0FRtwHdVo9j0sn6PDCw0LXPJh+5C0kBDIGO2apX+XpVAoIBAQDIWu8+37rF\nAKZ86ALMJdDgPGhcUiNNWHemy9+9Q\/xQaec+\/xcMArpwhwg7y1hyzDo89qT3\/+Fq\nptL960vqzee9oGnmocTj06Xwl5Mc\/ByKUEwkXLllU0PvYNP87nW2Nx8B1J4IZPrk\ny\/KxcmFe56x3+42\/7o2sDaaLVsTeMQNVBm4h3rdycmbHw925cToKCpo9iecVyNrl\np9rfsCJqQNJymeRpxASNVSM0QiCmFfnvfSPdx0a5WAGIXcLAE8xZWNWB+2XYOC7P\nRpLSZqT1EiPRvwPlSxxjt7g3mvBglpdvqElZt8mbaVTLcvPrhMG\/lAdf5dLjVewi\n9xyCEfzdM\/Y3AoIBACQ9rzuxb0G4+nlHcJoXdGB3NfccTwMh9YjB\/edYyA0Pz6VK\nWNm4yxFuLoico0IYOiHd3\/9YzWP6CrfseBIGJ7oNcHpQsQmsULzIRfkEQ4BE7itZ\nIFpg\/qYoeQR+8RYzXw8zRJ0u10D2dS3qAcfh6x8CnY660WHHzTYx9Q4OpTt0OjHC\n2M\/VkTH93ZEu2bBvn6E+DGzc\/PHasulDcXHS2obpCVHBBYXjMf26fMY3p79U5rR3\nGlij\/wXG93Ki1a5E893G5FVGv\/6AVOjtyATe4YLIqT3194qVn\/jMiBgH3578RnNP\nXpblcU7GDMA+us54yl8IH\/hb9hGWJNCTMiUcop0CggEAFl9BE847tr\/J12y40z7t\nwfw80wJA+uQECFX\/l8y9oeURc1Fcq3SqiOIrIzIjoD35ytDnvuGNtDEIol3hzkhL\ntjrxWFV7\/MynXOQwAy6LqU4qEm1gLlZL4bD4OugNsNka10N8mgdclKvJX9Mb2FMa\nSROH0oS6wRmNHUYFGaJzcs0TAykxZtArdecDHS\/tgS2J55E8gow5FSfXIt8yGJdA\naJeSj9TZhUaadb1kk+ckT3+zv5H+7bdMBIQjuPr8+IUY4jjGOIydfXut7VyU0uDB\nqywfLT1j4CalpV\/hs4ddRZSPDDufgYETfNxLLGNYlyHaaPTqxYPUqFrbjhKgH7sr\npQKCAQAxW3yYgmPjwpyglWtiSwVLs9mTDh4ahRl+FSUwk8ZSwRD1k9APEpoPmEbP\nqyalnv61aQTQN7cbEzgGm0D0t93sRWtEMU7Efq\/kU\/7acvc\/dNJ8p4hRZXxUNNaN\nreZ42\/IssirFJTpTFauxSPtYb\/7R6gIDAxf3J4+lGInIjWwZtvTb406rOKztvMqj\nD8B5Cf9vGrO\/CAjnY87BJNRFBrehhnLNFeh1pbdEMAORibfMxtn7k9EGYRdXSdrN\nXv+Zfn4rcd4zFjtMZ8fjOZXXhansJrmBAwAX9SmFtliD96OaNhEV1+3HLScDoR\/K\n0FZM\/3K5DrR8Ed3vWKqAtEOgi5AK\n-----END PRIVATE KEY-----\nBag Attributes\n    <strong>friendlyName: http<\/strong>\n    localKeyID: 54 69 6D 65 20 31 37 30 30 37 35 38 35 31 39 32 36 32 \nKey Attributes: &lt;No Attributes>\n-----BEGIN PRIVATE KEY-----\nMIIJQwIBADANBgkqhkiG9w0BAQEFAASCCS0wggkpAgEAAoICAQC8kJYcWvgzcjRd\nqzMagpo3Op94hNDJ2AX2gKP3V5B1kX4tlbjZxWwGLknfBA\/Sz5fTkle8z\/P0dVCf\nSXuRM9e92RyQoP1gwrPXJ+McFmmgH\/NwETN5aOaTThGmNN756xdyZt9qnxI3pdMs\n\/KUJcBM428YqlY6xFxkFWNQvkNSgC+fujwwmSGUEJ2+WdLO1UUkW2PcanRGrFj5y\nVBu7KaQG9PZ6DFDcdclJQGFnxkJC6eEvWgUxA2vJQUxsfw3+NUdZeXoh3vZ+Nsvw\n0Qw2DP6lZQN3IULw4kgk\/Sf1tPwVVN93QfTtZK899QYwAywKjJQmkCj9oTXQuZMb\nSZgf0ZMgLNpWlluRhUXxXf9GQcVbtL6kFWxxa22wGrWEmmDsd\/\/kn4b47rPKpmu4\niSGqS035xLXV+mutjWzDtZFRvzvI4a4\/va9Q4XCr3oACUDcgpHdOMpEreuaVt6o7\nJdF+W4Sn80FHpsLenfh36meQ4G5LjASF1pdzpMgPJRmz2PtO30afVAQivT3jOPuJ\nv\/szAdIwekTLCuFEPJ3nbCQ7osxIcKUB3z3Zbin7Rr8njeev6WlvWiR3uKso29xG\n80sOLzmYhLGiv6xVR2ESR\/BJ+1TJXUiLn9GNBt1x8xZ3bzZGzFxIHub6nmx+7K9S\nFxJ1i8Y2sjN5bE1+2LBjKRLqhQuOjwIDAQABAoICAEzTJ6WD1q23MOic0KRTY5tu\n0fwbOKmTztIglS03ys5rqZnO71IiYiGeiFBJalz8YSI89Gwj2YPWrDNJkAOXuKVL\nqJywSqs6iGT3hksyQnSPr\/IPwAYOHCsV2pD554KxKotcqFfbWWO8tu4UPPU8aUh4\nGiPNbSZvf60zBLggbNCMKUGDnHKYejeYDZmiEHmmFX4uhXadWc53sJeJ9wZpL7\/5\n29Xb4LIBdzHV\/hl2qBZ1DV7qcUUH3MlRRl7J4RyZ\/mjQ2lakbY78qliCA1SeNL2U\nCzZlIc3ck4ImdjpgM3hJbQY8pBp4NAGtK4kyMaZWoiGYCGtVXASfJvUuJcGI9ESq\nlxKlC+ndLfPmSFT9Yyi+34qIhCBlqpcfwT3aw8tVlVvHeI2rpPjs01s+8YnGJACY\nBapcjNk0GWQZj+HrkLcOs3MUbqdpQbFIggI6US1zAzq3MrfvY03t8Qdzxsiln2nW\n34nKMACAwKRcXeSSnxHE8J50VoWVZ1jaV81RqnvknGxhR4DoSmL9RWB3S9fTD1ff\nqaiGzriqrl88LXqnsqiUDkgdB4brchDs6c3OF06\/WS3oNRGzh\/oAPFN69lTMScMg\nxeFwv+auJW+c4V1Wz4ynwBZiqCy5yep2GXzJqNbl8Ad\/b3hfo\/K4E7mIifS\/4p0H\nrjLpkFkX+9DQS9GIg61xAoIBAQDFlESZdeVRkWXzMOGoYrf4a3PuLiPgq+odbYmI\nXj+kR+0hNYLWviDOxQySyxLScjT7LRuqTd9We8lItHPa\/jfXEsrGkDUIOQ+WBzyd\nW0SbnW0u5c6oJpj6GaQz6rcGpEAlqYgE00Ng5scXRoN\/ntLiHvta0vA9P7v0iA62\nnXbfGeCAaHpkoLRk\/XcDBmpJueUSSJ5cZDUgSfNkLoxOM7\/ZJa5oLWjcylJP\/35d\nwo3yM+A+VaZUW1pSLljzWnDfrEvK3ju7Kz0V9sVBoPAewtciowdp3Mi2MdlLiwLL\nh0YhexDJ20QmEf4QAOOJQKaxqnoI2b3hDhsK9BaXRo+4y3WXAoIBAQD0UfnTamU7\ntnOoIGweWuatJRseD8jP\/8TpiWQ31k1vkaQ1BHOvaN9y66KStJUKOBTJUhf5ArQe\nFZxaA3sxIxHlQitN8yuI5oNOKf42+4BCfE5ZDYn1\/euksd\/LNojHHOijpxoEnTaK\nsHr6+RT3dvbFHzKEYxzxHbUwKPN65qAfd0LBblNLyGuqQP9fyjueV0HgQdFEtxHD\nOEsteCfCYY4nxhTQ\/YOg3hHX6titPmNbVXRnPmjXoCVwM64iNT7HtWuJFQI54pPi\nB0DMH2kfm6njKWI+gMCnKfp3s+0bXDySxHpOn1EKj+wecFiOr\/X4nN+4hBhCQ43f\nAPfEALR1Wv3JAoIBAQCkcNd7Zc07Sykv86IMdEUro7r0JEDGEC8k2PLbpZ2QTDDH\nL\/M6aTmC\/iuNzShYMKd52bwpzPAx2YrYUSAPxv+QqFpOUt1gf6VCN7myObDrV2X0\n311VQ\/KUTV0FvLch9vhQoD3NzktIziorbAur3vMjMaf\/mopKX84+IjMkt\/+khbP9\nC3e6YqphVzeDNAaoX+iQhBIRUXGJ8yJ4YelyeM2WnC8BE7Iv+M\/zNvDkdjWPvK9J\nEh4CHZoZxetQdxh0gMEAEmBiUgVgT2czTRAseft2H3vfFtuSEAELR6JG5MpwuS1q\n42xfxx\/OD\/9QT5etTPaOgTLwqb3GKwrWUurxYZz3AoIBAQCVzCU2wMcs42LqNGbV\n\/bntcxC7T8Y58YHUBk8SBS0ZONzLPN2JMO2\/0kWtWVcAGv5zqQfVvxicUXe5oOTJ\nbc0tGXLfqjaJC3x6UjjxkSZEnV6ULz7lOjhelEi4mckm+8yPeobzSUkFo8rjUhDO\n4XvPxJ7+mJ5IH4TV8GXIdtq6MC8X1gLwNgP9MTjijGYdYTacvv4F+ZDEdyFw0Xjf\nL10veb2NncI0wbODBRSws83LVAu4uYUin6gUsTsU8jx5yfwS4nSo9Qjizrul66RS\nouh4xQjddaepVo2hEwpPejARdLHgvsG7jh6hUxSY974CHnz07tjI2A6GKlu\/Kwd5\n5L4xAoIBABIIJOsPkaMS2Nc3iRz9GweT00QFKoZekUiwrafJa4U8bc9B5gMXja77\nozTZShXDFKxVrvTLsraCUCK4RsFMTQE+pG5Q\/SEt8Lnc6D5VvY1oQu4zdbMsNfaO\nvI3vVB0+IQazDIRLhV6JNCaUZAr8jxnShyz2y4N5ZJYToOp20QT5BB5z2uRdtb9Q\nLrXnUNXS07bOVrBhYoi7pNbrvfiGrbrZ5aInn+NVSKy7Mkav7VaiwfhxMBwhD0kj\nesbwv62ZEoAziXeW95iQxvprroZgEAgUsyZJ\/cHilJ4c5YIkv2en21pGcGEtoWpv\nLc00BYUVRYhNU3H1h6CRQkbnHsNB5X4=\n-----END PRIVATE KEY-----\n<\/code><\/pre>\n\n\n\n<p>The CA key will the one under the <strong><code>friendlyName: http_ca<\/code><\/strong>.<\/p>\n\n\n\n<p>You can copy the key, anything between <code><strong>-----BEGIN PRIVATE KEY-----<\/strong><\/code> and <strong><code>-----END PRIVATE KEY-----<\/code><\/strong> and store them in a file of your choice, e.g <strong><code>\/etc\/elasticsearch\/certs\/http_ca_key.crt<\/code><\/strong>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configure-other-important-elasticsearch-systems-settings\">Configure Other Important Elasticsearch Systems Settings<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"disable-memory-swapping-on-all-cluster-nodes\">Disable Memory Swapping on All Cluster Nodes<\/h4>\n\n\n\n<p>Enabling memory lock as done above is on the ways of disabling swappiness. You therefore need to ensure that memory locking&nbsp;is enabled on the Elasticsearch service level. This can be done as follows;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;&#91; -d \/etc\/systemd\/system\/elasticsearch.service.d ]] || mkdir \/etc\/systemd\/system\/elasticsearch.service.d<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>echo -e '&#91;Service]\\nLimitMEMLOCK=infinity' &gt; \\\n\/etc\/systemd\/system\/elasticsearch.service.d\/override.conf<\/code><\/pre>\n\n\n\n<p>Whenever a systemd service is modified, you need to reload the systemd configurations.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">systemctl daemon-reload<\/pre>\n\n\n\n<p>One of the recommended ways to disable swapping is to completely disable swap if Elasticsearch is the only service running on the server.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">swapoff -a<\/pre>\n\n\n\n<p>Edit the&nbsp;<code>\/etc\/fstab<\/code>&nbsp;file and comment out any lines that contain the word&nbsp;<code>swap<\/code>;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sed -i.bak '\/swap\/s\/^\/#\/' \/etc\/fstab<\/code><\/pre>\n\n\n\n<p>Otherwise, disable swappiness in the kernel configuration;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo 'vm.swappiness=1' &gt;&gt; \/etc\/sysctl.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sysctl -p<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-jvm-heap-size-on-all-cluster-nodes\">Set JVM Heap Size on All Cluster Nodes<\/h4>\n\n\n\n<p>Elasticsearch usually sets the heap size automatically based on the role of the node. However, if you want to go with manual configuration, as a rule of thump, set&nbsp;<code>Xmx<\/code>&nbsp;to no more than 50% of your physical RAM. Any custom JVM settings should be placed under <code><strong>\/etc\/elasticsearch\/jvm.options.d<\/strong><\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">echo -e \"-Xms1g\\n-Xmx1g\" &gt; \/etc\/elasticsearch\/jvm.options.d\/jvm.options<\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-maximum-open-file-descriptor-on-all-cluster-nodes\">Set maximum Open File Descriptor on All Cluster Nodes<\/h4>\n\n\n\n<p>Set the maximum number of open files for the&nbsp;<code>elasticsearch<\/code>&nbsp;user to 65,536. This is already set by default in the,&nbsp;<strong><code>\/usr\/lib\/systemd\/system\/elasticsearch.service<\/code><\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>less \/usr\/lib\/systemd\/system\/elasticsearch.service<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>...\n# Specifies the maximum file descriptor number that can be opened by this process\nLimitNOFILE=65535\n...<\/code><\/pre>\n\n\n\n<p>You also should set the maximum number of processes.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>...\n# Specifies the maximum number of processes\nLimitNPROC=4096<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"update-virtual-memory-settings-on-all-cluster-nodes\">Update Virtual Memory Settings on All Cluster Nodes<\/h4>\n\n\n\n<p>Elasticsearch uses a&nbsp;<a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/index-modules-store.html#mmapfs\" target=\"_blank\" rel=\"noreferrer noopener\"><code>mmapfs<\/code><\/a>&nbsp;directory by default to store its indices. To ensure that you do not run out of virtual memory, edit the&nbsp;<strong>\/etc\/sysctl.conf<\/strong>&nbsp;and update the value of&nbsp;<strong>vm.max_map_count<\/strong>&nbsp;as shown below.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">vm.max_map_count=262144<\/pre>\n\n\n\n<p>You can simply run the command below to configure virtual memory settings.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">echo \"vm.max_map_count=262144\" &gt;&gt; \/etc\/sysctl.conf<\/pre>\n\n\n\n<p>To apply the changes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sysctl -p<\/code><\/pre>\n\n\n\n<p>To that far, below is the configuration file on each node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>grep -Ev '^#|^$' \/etc\/elasticsearch\/elasticsearch.yml<\/code><\/pre>\n\n\n\n<p>Node 01;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>cluster.name: kifarunix-demo\nnode.name: es-node01\nnode.roles: [ master, data ]\npath.data: \/var\/lib\/elasticsearch\npath.logs: \/var\/log\/elasticsearch\nbootstrap.memory_lock: true\nnetwork.host: 192.168.122.12\nhttp.port: 9200\nxpack.security.enabled: true\nxpack.security.enrollment.enabled: true\nxpack.security.http.ssl:\n  enabled: true\n  keystore.path: certs\/http.p12\nxpack.security.transport.ssl:\n  enabled: true\n  verification_mode: certificate\n  keystore.path: certs\/transport.p12\n  truststore.path: certs\/transport.p12\ncluster.initial_master_nodes: [\"es-node01\"]\n<\/code><\/pre>\n\n\n\n<p>Node 02<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>cluster.name: kifarunix-demo\nnode.name: es-node02\nnode.roles: [ master, data ]\npath.data: \/var\/lib\/elasticsearch\npath.logs: \/var\/log\/elasticsearch\nbootstrap.memory_lock: true\nnetwork.host: 192.168.122.73\nhttp.port: 9200\nxpack.security.enabled: true\nxpack.security.enrollment.enabled: true\nxpack.security.http.ssl:\n  enabled: true\n  keystore.path: certs\/http.p12\nxpack.security.transport.ssl:\n  enabled: true\n  verification_mode: certificate\n  keystore.path: certs\/transport.p12\n  truststore.path: certs\/transport.p12\ncluster.initial_master_nodes: [\"es-node02\"]\n<\/code><\/pre>\n\n\n\n<p>Node 03<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>cluster.name: kifarunix-demo\nnode.name: es-node03\nnode.roles: [ master, data ]\npath.data: \/var\/lib\/elasticsearch\npath.logs: \/var\/log\/elasticsearch\nbootstrap.memory_lock: true\nnetwork.host: 192.168.122.50\nhttp.port: 9200\nxpack.security.enabled: true\nxpack.security.enrollment.enabled: true\nxpack.security.http.ssl:\n  enabled: true\n  keystore.path: certs\/http.p12\nxpack.security.transport.ssl:\n  enabled: true\n  verification_mode: certificate\n  keystore.path: certs\/transport.p12\n  truststore.path: certs\/transport.p12\ncluster.initial_master_nodes: [\"en-node03\"]\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"start-and-enable-elasticsearch-service-on-node-01\">Start and Enable Elasticsearch Service on Node 01<\/h4>\n\n\n\n<p>For now, start and enable Elasticsearch service to run on system boot on Node 01 ONLY.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl enable --now elasticsearch<\/code><\/pre>\n\n\n\n<p>Confirm that Elasticsearch is running;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl status elasticsearch<\/code><\/pre>\n\n\n\n<p>The password was in the installation output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\u25cf elasticsearch.service - Elasticsearch\n     Loaded: loaded (\/lib\/systemd\/system\/elasticsearch.service; enabled; preset: enabled)\n    Drop-In: \/etc\/systemd\/system\/elasticsearch.service.d\n             \u2514\u2500override.conf\n     Active: active (running) since Thu 2023-11-23 12:25:26 EST; 4s ago\n       Docs: https:\/\/www.elastic.co\n   Main PID: 580 (java)\n      Tasks: 80 (limit: 4645)\n     Memory: 1.7G\n        CPU: 27.822s\n     CGroup: \/system.slice\/elasticsearch.service\n             \u251c\u2500580 \/usr\/share\/elasticsearch\/jdk\/bin\/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=\/usr\/share\/elasticsearch\/bin\/elasticsearch -Dcl>\n             \u251c\u2500638 \/usr\/share\/elasticsearch\/jdk\/bin\/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+>\n             \u2514\u2500658 \/usr\/share\/elasticsearch\/modules\/x-pack-ml\/platform\/linux-x86_64\/bin\/controller\n\nNov 23 12:25:12 es-node01.kifarunix-demo.com systemd[1]: Starting elasticsearch.service - Elasticsearch...\nNov 23 12:25:14 es-node01.kifarunix-demo.com systemd-entrypoint[580]: Nov 23, 2023 12:25:14 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>\nNov 23 12:25:14 es-node01.kifarunix-demo.com systemd-entrypoint[580]: WARNING: COMPAT locale provider will be removed in a future release\nNov 23 12:25:26 es-node01.kifarunix-demo.com systemd[1]: Started elasticsearch.service - Elasticsearch.\n<\/code><\/pre>\n\n\n\n<p>Confirm the ports are opened.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ss -altnp | grep -iE '92|93'<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>LISTEN 0      4096   [::ffff:192.168.122.12]:9200            *:*    users:((\"java\",pid=638,fd=438))\nLISTEN 0      4096   [::ffff:192.168.122.12]:9300            *:*    users:((\"java\",pid=638,fd=433))\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -k -u elastic https:\/\/es-node01:9200<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>{\n  \"name\" : \"es-node01\",\n  \"cluster_name\" : \"kifarunix-demo\",\n  \"cluster_uuid\" : \"KuX8vWFOTry9GDJCtXvo_g\",\n  \"version\" : {\n    \"number\" : \"8.11.1\",\n    \"build_flavor\" : \"default\",\n    \"build_type\" : \"deb\",\n    \"build_hash\" : \"6f9ff581fbcde658e6f69d6ce03050f060d1fd0c\",\n    \"build_date\" : \"2023-11-11T10:05:59.421038163Z\",\n    \"build_snapshot\" : false,\n    \"lucene_version\" : \"9.8.0\",\n    \"minimum_wire_compatibility_version\" : \"7.17.0\",\n    \"minimum_index_compatibility_version\" : \"7.0.0\"\n  },\n  \"tagline\" : \"You Know, for Search\"\n}\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"enroll-other-nodes-into-elasticsearch-cluster\">Enroll Other Nodes into Elasticsearch Cluster<\/h3>\n\n\n\n<p>At this point, Elasticsearch is running on Node 01 ONLY.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"generate-elasticsearch-cluster-enrollment-token\">Generate Elasticsearch Cluster Enrollment Token<\/h4>\n\n\n\n<p>Next, you need to generate Elasticsearch cluster enrollment token. <strong>Do this only on a single node<\/strong> where ES is already started.<\/p>\n\n\n\n<p>In this setup, we will generate Elasticsearch cluster enrollment token on ES Node01 ONLY since we have started Elasticsearch service on this node.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/usr\/share\/elasticsearch\/bin\/elasticsearch-create-enrollment-token -s node<\/code><\/pre>\n\n\n\n<p>Sample token;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>eyJ2ZXIiOiI4LjExLjEiLCJhZHIiOlsiMTkyLjE2OC4xMjIuMTI6OTIwMCJdLCJmZ3IiOiI0MDAxNjNiYmEzNDA3NjM4ZGEyNmNmYWI4OTVmNTFjNDU1NzU2NWNmYzljNGZhMzY4OGNhOTEwNDE1MWQxM2ViIiwia2V5IjoiMnRRNl9Zc0JsYjlwbjlBcGk2Qk06emZqU1ZPMXNRbmFvTDZ5N0FORVdHZyJ9<\/code><\/pre>\n\n\n\n<p>Once you have the token, enroll other nodes.<\/p>\n\n\n\n<p>Enroll Elasticsearch Node 02;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>\/usr\/share\/elasticsearch\/bin\/elasticsearch-reconfigure-node --enrollment-token &lt;PASTE TOKEN ABOVE&gt; <\/strong><\/code><\/pre>\n\n\n\n<p>For example;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/usr\/share\/elasticsearch\/bin\/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjExLjEiLCJhZHIiOlsiMTkyLjE2OC4xMjIuMTI6OTIwMCJdLCJmZ3IiOiI0MDAxNjNiYmEzNDA3NjM4ZGEyNmNmYWI4OTVmNTFjNDU1NzU2NWNmYzljNGZhMzY4OGNhOTEwNDE1MWQxM2ViIiwia2V5IjoiMnRRNl9Zc0JsYjlwbjlBcGk2Qk06emZqU1ZPMXNRbmFvTDZ5N0FORVdHZyJ9<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\nThis node will be reconfigured to join an existing cluster, using the enrollment token that you provided.\nThis operation will overwrite the existing configuration. Specifically: \n  - Security auto configuration will be removed from elasticsearch.yml\n  - The [certs] config directory will be removed\n  - Security auto configuration related secure settings will be removed from the elasticsearch.keystore\nDo you want to continue with the reconfiguration process [y\/N]<strong>y<\/strong>\n<\/code><\/pre>\n\n\n\n<p>Similarly, run the same enrollment command on other nodes.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"start-elasticsearch-on-other-nodes\">Start Elasticsearch on Other Nodes<\/h4>\n\n\n\n<p>Once the enrollment is done, start the Elasticsearch service on the node.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl enable --now elasticsearch<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"discovery-and-cluster-formation-settings-1\">Discovery and Cluster Formation settings<\/h4>\n\n\n\n<p>If you noticed, the enrollment command reconfigures the cluster initial nodes setting, <strong><code>cluster.initial_master_nodes<\/code><\/strong>, on other nodes that are enrolled into the cluster.<\/p>\n\n\n\n<p>Also <strong><code>discovery.seed_hosts:<\/code><\/strong> is configured with the address of the first node.<\/p>\n\n\n\n<p>Now, we need to configure the new nodes, Node02 and Node03 in this setup,  as we did with Node01.<\/p>\n\n\n\n<p>The only change you need to do on one of the nodes, for example, we do this on Node 01 is to define the list of the cluster nodes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/elasticsearch\/elasticsearch.yml<\/code><\/pre>\n\n\n\n<p>Basically, you need to define:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>cluster members (<code>discovery.seed_hosts: [\"es-node01\", \"es-node02\", \"es-node03\"]<\/code>)<\/li>\n\n\n\n<li>list of <strong>initial<\/strong> set of master-eligible nodes (<code>cluster.initial_master_nodes: [\"es-node01\", \"es-node02\", \"es-node03\"]<\/code>).<\/li>\n<\/ul>\n\n\n\n<pre class=\"scroll-box\"><code>\n# --------------------------------- Discovery ----------------------------------\n#\n# Pass an initial list of hosts to perform discovery when this node is started:\n# The default list of hosts is [\"127.0.0.1\", \"[::1]\"]\n#\n<strong>discovery.seed_hosts: [\"es-node01\", \"es-node02\", \"es-node03\"]\n<\/strong>#\n# Bootstrap the cluster using an initial set of master-eligible nodes:\n#\n<strong>cluster.initial_master_nodes: [\"es-node01\", \"es-node02\", \"es-node03\"]<\/strong>\n<\/code><\/pre>\n\n\n\n<p>Save and exit the configuration file.<\/p>\n\n\n\n<p>Sample config on Node 01;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat \/etc\/elasticsearch\/elasticsearch.yml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>cluster.name: kifarunix-demo\nnode.name: es-node01\nnode.roles: [ master, data ]\npath.data: \/var\/lib\/elasticsearch\npath.logs: \/var\/log\/elasticsearch\nbootstrap.memory_lock: true\nnetwork.host: 192.168.122.12\nhttp.port: 9200\nxpack.security.enabled: true\nxpack.security.enrollment.enabled: true\nxpack.security.http.ssl:\n  enabled: true\n  keystore.path: certs\/http.p12\nxpack.security.transport.ssl:\n  enabled: true\n  verification_mode: certificate\n  keystore.path: certs\/transport.p12\n  truststore.path: certs\/transport.p12\ncluster.initial_master_nodes: [\"es-node01\", \"es-node02\", \"es-node03\"]\ndiscovery.seed_hosts: [\"es-node01\", \"es-node02\", \"es-node03\"]\n<\/code><\/pre>\n\n\n\n<p>Restart Elasticsearch Service.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl restart elasticsearch<\/code><\/pre>\n\n\n\n<p>Configure Node 02 just like how Node 01 has been configured. Below is our sample Node 02 configuration;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat \/etc\/elasticsearch\/elasticsearch.yml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>cluster.name: kifarunix-demo\nnode.name: es-node02\nnode.roles: [ master, data ]\npath.data: \/var\/lib\/elasticsearch\npath.logs: \/var\/log\/elasticsearch\nbootstrap.memory_lock: true\nnetwork.host: 192.168.122.73\nhttp.port: 9200\nxpack.security.enabled: true\nxpack.security.enrollment.enabled: true\nxpack.security.http.ssl:\n  enabled: true\n  keystore.path: certs\/http.p12\nxpack.security.transport.ssl:\n  enabled: true\n  verification_mode: certificate\n  keystore.path: certs\/transport.p12\n  truststore.path: certs\/transport.p12\ndiscovery.seed_hosts: [\"es-node01\", \"es-node02\", \"es-node03\"]\ncluster.initial_master_nodes: [\"es-node01\", \"es-node02\", \"es-node03\"]\n<\/code><\/pre>\n\n\n\n<p>Restart Elasticsearch Service.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl restart elasticsearch<\/code><\/pre>\n\n\n\n<p>Configure Node 03 as well. Below is our sample Node 03 configuration;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat \/etc\/elasticsearch\/elasticsearch.yml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>cluster.name: kifarunix-demo\nnode.name: es-node03\nnode.roles: [ master, data ]\npath.data: \/var\/lib\/elasticsearch\npath.logs: \/var\/log\/elasticsearch\nbootstrap.memory_lock: true\nnetwork.host: 192.168.122.50\nhttp.port: 9200\nxpack.security.enabled: true\nxpack.security.enrollment.enabled: true\nxpack.security.http.ssl:\n  enabled: true\n  keystore.path: certs\/http.p12\nxpack.security.transport.ssl:\n  enabled: true\n  verification_mode: certificate\n  keystore.path: certs\/transport.p12\n  truststore.path: certs\/transport.p12\ndiscovery.seed_hosts: [\"es-node01\", \"es-node02\", \"es-node03\"]\ncluster.initial_master_nodes: [\"es-node01\", \"es-node02\", \"es-node03\"]\n<\/code><\/pre>\n\n\n\n<p>Restart Elasticsearch Service.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl restart elasticsearch<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"check-the-cluster-nodes\">Check the Cluster Nodes<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -k -XGET \"https:\/\/es-node01:9200\/_cat\/nodes?v\" -u elastic<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Enter host password for user 'elastic':\nip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name\n192.168.122.12           57          50   3    0.08    0.10     0.08 dm        -      es-node01\n192.168.122.50           33          50   8    0.20    0.13     0.05 dm        -      es-node03\n192.168.122.73           49          50   3    0.02    0.13     0.08 dm        *      es-node02\n<\/code><\/pre>\n\n\n\n<p>And there you go. You now have Elasticsearch 8.x cluster.<\/p>\n\n\n\n<p>Check cluster health status;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -k -XGET \"https:\/\/es-node01:9200\/_cat\/health?v\" -u elastic<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Enter host password for user 'elastic':\nepoch      timestamp cluster        status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent\n1700761464 17:44:24  kifarunix-demo green           3         3      2   1    0    0        0             0                  -                100.0%\n<\/code><\/pre>\n\n\n\n<p>As you can see, our cluster status is <strong>GREEN<\/strong>!!<\/p>\n\n\n\n<p><strong>ENSURE<\/strong> that the cluster transport ports (9300\/tcp) are opened on the firewall on each node to allow cluster formation.<\/p>\n\n\n\n<p>Once the cluster is formed as above, remove or comment the following line on <strong>ALL<\/strong> the nodes.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">cluster.initial_master_nodes: [\"es-node01\", \"es-node02\", \"es-node03\"]<\/pre>\n\n\n\n<p>You can remove as follows;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sed -i.bak '\/cluster.initial_master_nodes\/s\/^\/#\/' \/etc\/elasticsearch\/elasticsearch.yml<\/code><\/pre>\n\n\n\n<p>Multinode cluster is up and running!<\/p>\n\n\n\n<p>Other Tutorials<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/deploy-a-single-node-elastic-stack-cluster-on-docker-containers\/\" target=\"_blank\" rel=\"noreferrer noopener\">Deploy a Single Node Elastic Stack Cluster on Docker Containers<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/setup-multi-node-elasticsearch-cluster\/\" target=\"_blank\" rel=\"noreferrer noopener\">Setup Multi-node Elasticsearch Cluster<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Follow through this tutorial to learn how to setup multinode Elasticsearch 8.x cluster. As of this writing, Elastic Stack 8.3 is the current release. This<\/p>\n","protected":false},"author":3,"featured_media":8931,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121,72,34],"tags":[5464,5460,5462,5461,5465,5463,5459],"class_list":["post-13326","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","category-monitoring","category-security","tag-elasticsearch-8","tag-elasticsearch-8-cluster","tag-elasticsearch-8-enrollment-token","tag-elk-3-node-cluster","tag-elk-cluster","tag-enroll-elasticsearch-8-node-into-cluster","tag-setup-multinode-elasticsearch-8-x-cluster","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/13326"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=13326"}],"version-history":[{"count":24,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/13326\/revisions"}],"predecessor-version":[{"id":20555,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/13326\/revisions\/20555"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/8931"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=13326"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=13326"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=13326"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}