{"id":19854,"date":"2024-01-21T01:59:04","date_gmt":"2024-01-20T22:59:04","guid":{"rendered":"https:\/\/kifarunix.com\/?p=19854"},"modified":"2024-03-10T15:33:49","modified_gmt":"2024-03-10T12:33:49","slug":"deploy-elk-stack-8-cluster-on-docker-containers","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/deploy-elk-stack-8-cluster-on-docker-containers\/","title":{"rendered":"Deploy ELK Stack 8 Cluster on Docker Containers"},"content":{"rendered":"\n<p>Follow through this guide to learn how to deploy ELK Stack 8 cluster on Docker containers. Deploying a multinode ELK Stack 8 cluster on Docker <a href=\"https:\/\/www.docker.com\/resources\/what-container\/\" target=\"_blank\" rel=\"noreferrer noopener\">containers<\/a> on separate nodes offers a flexible and scalable solution for managing distributed search and analytics workloads. Docker provides a lightweight and portable containerization environment, allowing Elastic components to be easily deployed and orchestrated across various environments. As of this writing, Elastic 8.12.0 is the <a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/es-release-notes.html\" target=\"_blank\" rel=\"noreferrer noopener\">current stable release<\/a>.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#deploying-elk-stack-8-cluster-on-docker-containers\">Deploying ELK Stack 8 Cluster on Docker Containers<\/a><ul><li><a href=\"#install-docker-engine\">Install Docker Engine<\/a><\/li><li><a href=\"#install-docker-compose\">Install Docker Compose<\/a><\/li><li><a href=\"#important-elasticsearch-settings\">Important Elasticsearch Settings<\/a><ul><li><a href=\"#disable-memory-swapping-on-a-container\">Disable Memory Swapping on a Container<\/a><\/li><li><a href=\"#set-jvm-heap-size-on-all-cluster-nodes\">Set JVM Heap Size on All Cluster Nodes<\/a><\/li><li><a href=\"#set-maximum-open-file-descriptor-and-processes-on-elasticsearch-containers\">Set Maximum Open File Descriptor and Processes on Elasticsearch Containers<\/a><\/li><li><a href=\"#update-virtual-memory-settings-on-all-cluster-nodes\">Update Virtual Memory Settings on All Cluster Nodes<\/a><\/li><\/ul><\/li><li><a href=\"#deploy-elk-stack-8-on-first-cluster-docker-host\">Deploy ELK Stack 8 on First Cluster Docker Host<\/a><ul><li><a href=\"#create-a-network-share-for-docker-volumes\">Create a Network Share for Docker Volumes<\/a><\/li><li><a href=\"#create-docker-compose-file-for-your-container-services\">Create Docker-Compose file for your Container Services<\/a><\/li><li><a href=\"#validate-docker-compose-files\">Validate Docker Compose Files<\/a><\/li><li><a href=\"#start-elk-stack-8-docker-containers\">Start ELK Stack 8 Docker Containers<\/a><\/li><\/ul><\/li><li><a href=\"#verify-the-elk-stack-cluster\">Verify the ELK Stack Cluster<\/a><\/li><li><a href=\"#accessing-kibana\">Accessing Kibana<\/a><\/li><li><a href=\"#sending-data-logs-to-elastic-stack\">Sending Data Logs to Elastic Stack<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"deploying-elk-stack-8-cluster-on-docker-containers\">Deploying ELK Stack 8 Cluster on Docker Containers<\/h2>\n\n\n\n<p>We will be deploying a three node ELK Stack 8 cluster on Docker containers.<\/p>\n\n\n\n<p>Note that a cluster typically involves multiple nodes working together to store and process data. In the context of Elastic, a cluster is formed when multiple Elasticsearch nodes join together to share the workload and provide high availability. If you have three Elasticsearch containers running on a single node, it&#8217;s still a single-node setup, not a cluster. A true Elasticsearch cluster would involve running multiple instances of Elasticsearch on separate nodes, allowing them to communicate and work together as a unified system.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-docker-engine\">Install Docker Engine<\/h3>\n\n\n\n<p>Depending on the your host system distribution, you need to install the Docker engine. You can link below to various guides to install Docker engine.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/?s=install+Docker\" target=\"_blank\" rel=\"noreferrer noopener\">Install Docker CE on Linux<\/a><\/p>\n\n\n\n<p>Checking Installed Docker version;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker version<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Docker version 24.0.7, build afdd53b<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-docker-compose\">Install Docker Compose<\/h3>\n\n\n\n<p>If you are running Elasticsearch cluster in Docker containers for test\/development purposes, you can just create the containers using the docker run commands. Otherwise, use Docker composer to deploy production Elasticsearch containers!<\/p>\n\n\n\n<p>Download and install Docker Compose on a Linux system. Be sure to replace the&nbsp;<strong>VER<\/strong>&nbsp;variable below with the value of the current stable release version of Docker compose.<\/p>\n\n\n\n<p>Check the current stable release version of Docker Compose on their&nbsp;<a href=\"https:\/\/github.com\/docker\/compose\/releases\" target=\"_blank\" rel=\"noreferrer noopener\">Github release page<\/a>. As of this writing, the Docker Compose version 2.24.1 is the current stable release.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>VER=2.24.1<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo curl -L \"https:\/\/github.com\/docker\/compose\/releases\/download\/v$VER\/docker-compose-$(uname -s)-$(uname -m)\" -o \/usr\/local\/bin\/docker-compose<\/code><\/pre>\n\n\n\n<p>This downloads docker compose tool to&nbsp;<code>\/usr\/local\/bin<\/code>&nbsp;directory.<\/p>\n\n\n\n<p>Make the Docker compose binary executable;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>chmod +x \/usr\/local\/bin\/docker-compose<\/code><\/pre>\n\n\n\n<p>You should now be able to use Docker compose (<code><strong>docker-compose<\/strong><\/code>) on the CLI.<\/p>\n\n\n\n<p>Check the version of installed Docker compose to confirm that it is working as expected.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker-compose version<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Docker Compose version v2.24.1<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"important-elasticsearch-settings\">Important Elasticsearch Settings<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"disable-memory-swapping-on-a-container\">Disable Memory Swapping on a Container<\/h4>\n\n\n\n<p>There are different ways to disable swapping. In an Elasticsearch Docker container, you can simply set the value of the <strong><code>bootstrap.memory_lock<\/code><\/strong> option to <strong><code>true<\/code><\/strong>. With this option, you also need to ensure that there is no maximum limit imposed on the locked-in-memory address space (<strong><code>LimitMEMLOCK=infinity<\/code><\/strong>).<\/p>\n\n\n\n<p>For a respective service, this is memory_lock limits are defined in a Docker compose file.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>    ulimits:\n      memlock:\n        soft: -1\n        hard: -1\n<\/code><\/pre>\n\n\n\n<p>You can confirm the default values set on a Docker image using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker run --rm docker.elastic.co\/elasticsearch\/elasticsearch:8.11.4 \/bin\/bash -c 'ulimit -Hm &amp;&amp; ulimit -Sm'<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>unlimited\nunlimited\n<\/code><\/pre>\n\n\n\n<p>If the value other than the required one is printed, then you have to set this via the Docker compose file to ensure that it set to appropriately for both Soft and Hard limits.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-jvm-heap-size-on-all-cluster-nodes\">Set JVM Heap Size on All Cluster Nodes<\/h4>\n\n\n\n<p>Elasticsearch usually sets the heap size automatically based on the roles of the nodes\u2019s&nbsp;roles&nbsp;and the total memory available to the node\u2019s container. This is the recommended approach! For the Elasticsearch Docker containers, any custom JVM settings can be defined using the <strong><code>ES_JAVA_OPTS<\/code><\/strong> environment variable, either in compose file or on command line.<\/p>\n\n\n\n<p>For example, you can set JVM heap size, Min and Max to 512M using the <strong><code>ES_JAVA_OPTS=-Xms512m -Xmx512m<\/code><\/strong>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-maximum-open-file-descriptor-and-processes-on-elasticsearch-containers\">Set Maximum Open File Descriptor and Processes on Elasticsearch Containers<\/h4>\n\n\n\n<p>Set the maximum number of open files (<code>nofile<\/code>) for the&nbsp;containers to 65,536 and max number of processes to 4096 (both soft and hard limits).<\/p>\n\n\n\n<p>You can use the command below to check the maximum number of user processes (-u) and max number of opens files (-n);<\/p>\n\n\n\n<p>Replace the value of VER variable below with the current value of Elastic release!<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>VER=8.12.0<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>docker run --rm docker.elastic.co\/elasticsearch\/elasticsearch:${VER} bash -c 'ulimit -Hn &amp;&amp; ulimit -Sn &amp;&amp; ulimit -Hu &amp;&amp; ulimit -Su'<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>1048576\n1048576\nunlimited\nunlimited\n<\/code><\/pre>\n\n\n\n<p>These values may look like this on Docker compose file. Adjust them accordingly!<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>    ulimits:\n      nofile:\n        soft: 65536\n        hard: 65536\n      nproc:\n        soft: 2048\n        hard: 2048\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"update-virtual-memory-settings-on-all-cluster-nodes\">Update Virtual Memory Settings on All Cluster Nodes<\/h4>\n\n\n\n<p>Elasticsearch uses a&nbsp;<a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/index-modules-store.html#mmapfs\" target=\"_blank\" rel=\"noreferrer noopener\"><code>mmapfs<\/code><\/a>&nbsp;directory by default to store its indices. To ensure that you do not run out of virtual memory, edit the&nbsp;<strong>\/etc\/sysctl.conf<\/strong>&nbsp;on the <strong><code>Docker host<\/code><\/strong> and update the value of&nbsp;<strong>vm.max_map_count<\/strong>&nbsp;as shown below.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vm.max_map_count=262144<\/code><\/pre>\n\n\n\n<p>On each of the cluster node, simply run the command below to configure virtual memory settings.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"vm.max_map_count=262144\" &gt;&gt; \/etc\/sysctl.conf<\/code><\/pre>\n\n\n\n<p>To apply the changes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sysctl -p<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploy-elk-stack-8-on-first-cluster-docker-host\">Deploy ELK Stack 8 on First Cluster Docker Host<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-a-network-share-for-docker-volumes\">Create a Network Share for Docker Volumes<\/h4>\n\n\n\n<p>In this setup, we are running ELK stack 8 containers on three separate nodes. As a result, there will be a number of files that we will need to share among the three containers on three different nodes. As much as there could other ways to share file on containers running on separate nodes, one of the option that became easy and achievable for me in this setup is the use of NFS shares  to share volumes\/files across the containers.<\/p>\n\n\n\n<p>Hence, we have <a href=\"https:\/\/kifarunix.com\/?s=nfs+server\" target=\"_blank\" rel=\"noreferrer noopener\">deployed an NFS server<\/a> and mounted a directory with the files that needs to be shared across the ELK stack 8 cluster.<\/p>\n\n\n\n<p>This is our NFS share export configuration;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat \/etc\/exports<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\/mnt\/elkstack\/\t192.168.122.152(rw,sync,no_root_squash,no_subtree_check) 192.168.122.123(rw,sync,no_root_squash,no_subtree_check) 192.168.122.60(rw,sync,no_root_squash,no_subtree_check)\n<\/code><\/pre>\n\n\n\n<p>As you can see, we create an NFS share under <strong><code>\/mnt\/elkstack<\/code><\/strong> and allow access from our ELK stack cluster nodes.<\/p>\n\n\n\n<p>In this share, we have created the following directories:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><code>\/mnt\/elkstack\/certs<\/code><\/strong> for ELK stack SSL\/TLS certificates;<\/li>\n\n\n\n<li><strong><code>\/mnt\/elkstack\/configs\/{logstash}<\/code><\/strong> for shared configuration files<\/li>\n\n\n\n<li><strong><code>\/mnt\/elkstack\/data\/<\/code><\/strong> for Elasticsearch components data.\n<ul class=\"wp-block-list\">\n<li><strong><code>elasticsearch\/{01,02,03}<\/code><\/strong><\/li>\n\n\n\n<li><strong><code>kibana\/{01,02,03}<\/code><\/strong><\/li>\n\n\n\n<li><strong><code>logstash\/{01,02,03}<\/code><\/strong><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>In the Logstash configuration directory, we have use the <a href=\"https:\/\/kifarunix.com\/deploy-elk-stack-8-on-docker-containers\/#define-logstash-data-processing-pipeline\" target=\"_blank\" rel=\"noreferrer noopener\">same pipeline configuration used in our previous guide<\/a>. However, we adjusted the output to connect to multiple Elasticsearch outputs;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>output {\n   elasticsearch {\n     hosts => [\"https:\/\/${NODE1_NAME}:9200\",\"https:\/\/${NODE2_NAME}:9200\", \"https:\/\/${NODE3_NAME}:9200\"]\n     user => \"${ELASTICSEARCH_USERNAME}\"\n     password => \"${ELASTICSEARCH_PASSWORD}\"\n     ssl => true\n     cacert => \"config\/certs\/ca\/ca.crt\"\n \n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-docker-compose-file-for-your-container-services\">Create Docker-Compose file for your Container Services<\/h4>\n\n\n\n<p>We will be using three nodes in this setup. On each node, we will run a single Docker container of ELK stack components, Elasticsearch 8, Logstash 8, Kibana 8. We will later on connect the Elasticsearch 8 containers together to make a cluster.<\/p>\n\n\n\n<p>Thus, on the first node, let&#8217;s create a Docker compose file for our ELK stack.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>mkdir elkstack-docker<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>cd elkstack-docker<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>vim docker-compose.yml<\/code><\/pre>\n\n\n\n<p>Below is a sample Docker compose file configs for first node of our ELK Stack 8 Cluster Docker container.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>version: '3.8'\n\nservices:\n  es_setup:\n    image: docker.elastic.co\/elasticsearch\/elasticsearch:${STACK_VERSION}\n    volumes:\n      - certs:\/usr\/share\/elasticsearch\/config\/certs\n    user: \"0\"\n    command: >\n      bash -c '\n        echo \"Creating ES certs directory...\"\n        [[ -d config\/certs ]] || mkdir config\/certs\n        # Check if CA certificate exists\n        if [ ! -f config\/certs\/ca\/ca.crt ]; then\n          echo \"Generating Wildcard SSL certs for ES (in PEM format)...\"\n          bin\/elasticsearch-certutil ca --pem --days 3650 --out config\/certs\/elkstack-ca.zip\n          unzip -d config\/certs config\/certs\/elkstack-ca.zip\n          bin\/elasticsearch-certutil cert \\\n            --name elkstack-certs \\\n            --ca-cert config\/certs\/ca\/ca.crt \\\n            --ca-key config\/certs\/ca\/ca.key \\\n            --pem \\\n            --dns \"*.${DOMAIN_SUFFIX},localhost,${NODE1_NAME},${NODE2_NAME},${NODE3_NAME}\" \\\n            --ip 192.168.122.60 \\\n            --ip 192.168.122.123 \\\n            --ip 192.168.122.152 \\\n            --days ${DAYS} \\\n            --out config\/certs\/elkstack-certs.zip\n          unzip -d config\/certs config\/certs\/elkstack-certs.zip\n        else\n          echo \"CA certificate already exists. Skipping Certificates generation.\"\n        fi\n        # Check if Elasticsearch is ready\n        until curl -s --cacert config\/certs\/ca\/ca.crt -u \"elastic:${ELASTIC_PASSWORD}\" https:\/\/${NODE1_NAME}:9200 | grep -q \"${CLUSTER_NAME}\"; do sleep 10; done\n        # Set kibana_system password\n        if curl -sk -XGET --cacert config\/certs\/ca\/ca.crt \"https:\/\/${NODE1_NAME}:9200\" -u \"kibana_system:${KIBANA_PASSWORD}\" | grep -q \"${CLUSTER_NAME}\"; then\n          echo \"Password for kibana_system is working. Proceeding with Elasticsearch setup for kibana_system.\"\n        else\n          echo \"Failed to authenticate with kibana_system password. Trying to set the password for kibana_system.\"\n          until curl -s -XPOST --cacert config\/certs\/ca\/ca.crt -u \"elastic:${ELASTIC_PASSWORD}\" -H \"Content-Type: application\/json\" https:\/\/${NODE1_NAME}:9200\/_security\/user\/kibana_system\/_password -d \"{\\\"password\\\":\\\"${KIBANA_PASSWORD}\\\"}\" | grep -q \"^{}\"; do sleep 10; done\n        fi\n        echo \"Setup is done!\"\n      '\n    networks:\n      - elastic\n    healthcheck:\n      test: [\"CMD-SHELL\", \"[ -f config\/certs\/elkstack-certs\/elkstack-certs.crt ]\"]\n      interval: 1s\n      timeout: 5s\n      retries: 120\n\n  elasticsearch:\n    depends_on:\n      es_setup:\n        condition: service_healthy\n    container_name: ${NODE1_NAME}\n    image: docker.elastic.co\/elasticsearch\/elasticsearch:${STACK_VERSION}\n    environment:\n      - node.name=${NODE1_NAME}\n      - network.publish_host=${NODE1} \n      - cluster.name=${CLUSTER_NAME}\n      - bootstrap.memory_lock=true\n      - \"ES_JAVA_OPTS=-Xms1g -Xmx1g\"\n      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}\n      - xpack.security.enabled=true\n      - xpack.security.http.ssl.enabled=true\n      - xpack.security.transport.ssl.enabled=true\n      - xpack.security.enrollment.enabled=false\n      - xpack.security.autoconfiguration.enabled=false\n      - xpack.security.http.ssl.key=certs\/elkstack-certs\/elkstack-certs.key\n      - xpack.security.http.ssl.certificate=certs\/elkstack-certs\/elkstack-certs.crt\n      - xpack.security.http.ssl.certificate_authorities=certs\/ca\/ca.crt\n      - xpack.security.transport.ssl.key=certs\/elkstack-certs\/elkstack-certs.key\n      - xpack.security.transport.ssl.certificate=certs\/elkstack-certs\/elkstack-certs.crt\n      - xpack.security.transport.ssl.certificate_authorities=certs\/ca\/ca.crt\n      - cluster.initial_master_nodes=${NODE1_NAME},${NODE2_NAME},${NODE3_NAME}\n      - discovery.seed_hosts=192.168.122.60,192.168.122.123,192.168.122.152\n      - KIBANA_USERNAME=${KIBANA_USERNAME}\n      - KIBANA_PASSWORD=${KIBANA_PASSWORD}\n    ulimits:\n      memlock:\n        soft: -1\n        hard: -1\n    volumes:\n      - elasticsearch_data:\/usr\/share\/elasticsearch\/data\n      - certs:\/usr\/share\/elasticsearch\/config\/certs\n      - \/etc\/hosts:\/etc\/hosts\n    ports:\n      - 9200:9200\n      - 9300:9300\n    networks:\n      - elastic\n    healthcheck:\n      test: [\"CMD-SHELL\", \"curl --fail -k -s -u elastic:${ELASTIC_PASSWORD} --cacert config\/certs\/ca\/ca.crt https:\/\/${NODE1_NAME}:9200\"]\n      interval: 30s\n      timeout: 10s\n      retries: 5\n    restart: unless-stopped\n\n  kibana:\n    image: docker.elastic.co\/kibana\/kibana:${STACK_VERSION}\n    container_name: kibana\n    environment:\n      - SERVER_NAME=${KIBANA_SERVER_HOST}\n      - ELASTICSEARCH_HOSTS=https:\/\/${NODE1_NAME}:9200\n      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config\/certs\/ca\/ca.crt\n      - ELASTICSEARCH_USERNAME=${KIBANA_USERNAME}\n      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}\n      - XPACK_REPORTING_ROLES_ENABLED=false\n      - XPACK_REPORTING_KIBANASERVER_HOSTNAME=localhost\n      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${SAVEDOBJECTS_ENCRYPTIONKEY}\n      - XPACK_SECURITY_ENCRYPTIONKEY=${REPORTING_ENCRYPTIONKEY}\n      - XPACK_REPORTING_ENCRYPTIONKEY=${SECURITY_ENCRYPTIONKEY}\n    volumes:\n      - kibana_data:\/usr\/share\/kibana\/data\n      - certs:\/usr\/share\/kibana\/config\/certs\n      - \/etc\/hosts:\/etc\/hosts\n    ports:\n      - ${KIBANA_PORT}:5601\n    networks:\n      - elastic\n    depends_on:\n      elasticsearch:\n        condition: service_healthy\n    restart: unless-stopped\n  \n  logstash:\n    image: docker.elastic.co\/logstash\/logstash:${STACK_VERSION}\n    container_name: logstash\n    environment:\n      - XPACK_MONITORING_ENABLED=false\n      - ELASTICSEARCH_USERNAME=${ES_USER}\n      - ELASTICSEARCH_PASSWORD=${ELASTIC_PASSWORD}\n      - NODE_NAME=${NODE1_NAME}\n    ports:\n      - 5044:5044\n    volumes:\n      - logstash_filters:\/usr\/share\/logstash\/pipeline\/:ro\n      - certs:\/usr\/share\/logstash\/config\/certs\n      - logstash_data:\/usr\/share\/logstash\/data\n      - \/etc\/hosts:\/etc\/hosts\n    networks:\n      - elastic\n    depends_on:\n      elasticsearch:\n        condition: service_healthy\n    restart: unless-stopped\n\nvolumes:\n  certs:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,rw\"\n      device: \":\/mnt\/elkstack\/certs\"\n  elasticsearch_data:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,rw\"\n      device: \":\/mnt\/elkstack\/data\/elasticsearch\/01\"\n  kibana_data:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,rw\"\n      device: \":\/mnt\/elkstack\/data\/kibana\/01\"\n  logstash_filters:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,rw\"\n      device: \":\/mnt\/elkstack\/configs\/logstash\"\n  logstash_data:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,rw\"\n      device: \":\/mnt\/elkstack\/data\/logstash\/01\"\n\nnetworks:\n  elastic:\n<\/code><\/pre>\n\n\n\n<p>Where:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>es_setup<\/code> Service:\n<ul class=\"wp-block-list\">\n<li><strong>Image:<\/strong> Uses the official Elasticsearch Docker image with a specified version (<code>${STACK_VERSION}<\/code>).<\/li>\n\n\n\n<li><strong>Volumes:<\/strong> Mounts a volume named <code>certs<\/code> to the Elasticsearch configuration directory.<\/li>\n\n\n\n<li><strong>User:<\/strong> Sets the user to root (user ID 0).<\/li>\n\n\n\n<li><strong>Command:<\/strong> Shell script for setting up Elasticsearch certificates, checking cluster readiness, and setting the <code>kibana_system<\/code> password.<\/li>\n\n\n\n<li><strong>Networks:<\/strong> Connects to the <code>elastic<\/code> network.<\/li>\n\n\n\n<li><strong>Healthcheck:<\/strong> Uses a custom health check to test the existence of a specific file (<code>config\/certs\/elkstack-certs\/elkstack-certs.crt<\/code>).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>elasticsearch<\/code> Service:\n<ul class=\"wp-block-list\">\n<li><strong>Depends On:<\/strong> Depends on the <code>es_setup<\/code> service with the condition that <code>es_setup<\/code> is healthy.<\/li>\n\n\n\n<li><strong>Container Name:<\/strong> Specifies a custom name for the Elasticsearch container (<code>${NODE1_NAME}<\/code>).<\/li>\n\n\n\n<li><strong>Image:<\/strong> Uses the official Elasticsearch Docker image with the specified version (<code>${STACK_VERSION}<\/code>).<\/li>\n\n\n\n<li><strong>Environment:<\/strong> Configures various environment variables for Elasticsearch, such as node name, network settings, security, etc.<\/li>\n\n\n\n<li><strong>Ulimits:<\/strong> Sets memory lock limits for Elasticsearch.<\/li>\n\n\n\n<li><strong>Volumes:<\/strong> Mounts volumes for Elasticsearch data, certificates, and the hosts file.<\/li>\n\n\n\n<li><strong>Ports:<\/strong> Exposes ports 9200 and 9300.<\/li>\n\n\n\n<li><strong>Networks:<\/strong> Connects to the <code>elastic<\/code> network.<\/li>\n\n\n\n<li><strong>Healthcheck:<\/strong> Uses a custom health check to check Elasticsearch availability.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>kibana<\/code> Service:<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Image:<\/strong> Uses the official Kibana Docker image with the specified version (<code>${STACK_VERSION}<\/code>).<\/li>\n\n\n\n<li><strong>Container Name:<\/strong> Specifies a custom name for the Kibana container (<code>kibana<\/code>).<\/li>\n\n\n\n<li><strong>Environment:<\/strong> Configures environment variables for Kibana, including Elasticsearch connection details and security settings.<\/li>\n\n\n\n<li><strong>Volumes:<\/strong> Mounts volumes for Kibana data, certificates, and the hosts file.<\/li>\n\n\n\n<li><strong>Ports:<\/strong> Exposes the specified port for Kibana.<\/li>\n\n\n\n<li><strong>Networks:<\/strong> Connects to the <code>elastic<\/code> network.<\/li>\n\n\n\n<li><strong>Depends On:<\/strong> Depends on the <code>elasticsearch<\/code> service with the condition that <code>elasticsearch<\/code> is healthy.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>Logstash<\/code> Service:\n<ul class=\"wp-block-list\">\n<li><strong>Image:<\/strong> Uses the official Logstash Docker image with the specified version (<code>${STACK_VERSION}<\/code>).<\/li>\n\n\n\n<li><strong>Container Name:<\/strong> Specifies a custom name for the Logstash container (<code>logstash<\/code>).<\/li>\n\n\n\n<li><strong>Environment:<\/strong> Configures environment variables for Logstash, including Elasticsearch connection details and security settings.<\/li>\n\n\n\n<li><strong>Ports:<\/strong> Exposes port 5044 for Logstash.<\/li>\n\n\n\n<li><strong>Volumes:<\/strong> Mounts volumes for Logstash filters, certificates, and Logstash data.<\/li>\n\n\n\n<li><strong>Networks:<\/strong> Connects to the <code>elastic<\/code> network.<\/li>\n\n\n\n<li><strong>Depends On:<\/strong> Depends on the <code>elasticsearch<\/code> service with the condition that <code>elasticsearch<\/code> is healthy.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Volumes:\n<ul class=\"wp-block-list\">\n<li>Defines multiple volumes for storing certificates, Elasticsearch data, Kibana data, Logstash filters, and Logstash data. These volumes are configured to use NFS.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Networks:\n<ul class=\"wp-block-list\">\n<li>Defines a custom network named <code>elastic<\/code> for connecting the services.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>All the environment variables used are defined in the <code><strong>.env<\/strong><\/code> file.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat .env<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code># Version of Elastic products\nSTACK_VERSION=8.12.0\n\n# Set the cluster name\nCLUSTER_NAME=elk-docker-cluster\n\n# Set Elasticsearch Node Name\nNODE1_NAME=es01\nNODE2_NAME=es02\nNODE3_NAME=es03\n\n# Docker Host IP to advertise to cluster nodes\nNODE1=192.168.122.60\nNODE2=192.168.122.123\nNODE3=192.168.122.152\n\n# Elasticsearch super user\nES_USER=elastic\n\n# Password for the 'elastic' user (at least 6 characters). No special characters, ! or @ or $.\nELASTIC_PASSWORD=ChangeME\n\n# Elasticsearch container name\nES_NAME=elasticsearch\n\n# Port to expose Elasticsearch HTTP API to the host\nES_PORT=9200\n#ES_PORT=127.0.0.1:9200\n\n# Port to expose Kibana to the host\nKIBANA_PORT=5601\nKIBANA_SERVER_HOST=localhost\n\n# Kibana Encryption. Requires atleast 32 characters. Can be generated using `openssl rand -hex 16\nSAVEDOBJECTS_ENCRYPTIONKEY=ca11560aec8410ff002d011c2a172608\nREPORTING_ENCRYPTIONKEY=288f06b3a14a7f36dd21563d50ec76d4\nSECURITY_ENCRYPTIONKEY=62c781d3a2b2eaee1d4cebcc6bf42b48\n\n# Kibana - Elasticsearch Authentication Credentials for user kibana_system\n# Password for the 'kibana_system' user (at least 6 characters). No special characters, ! or @ or $.\nKIBANA_USERNAME=kibana_system\nKIBANA_PASSWORD=ChangeME\n\n# Domain Suffix for ES Wildcard SSL certs\nDOMAIN_SUFFIX=kifarunix-demo.com\n\n# Generated Certs Validity Period\nDAYS=3650\n\n# Logstash Input Port\nLS_PORT=5044\n<\/code><\/pre>\n\n\n\n<p>Similarly, create on the other nodes, create Docker compose file and respective environment variables file. We will use same environment variables across the nodes!<\/p>\n\n\n\n<p>One the second node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat docker-compose.yml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>services:\n  elasticsearch:\n    container_name: ${NODE2_NAME}\n    image: docker.elastic.co\/elasticsearch\/elasticsearch:${STACK_VERSION}\n    command: >\n      bash -c '\n      until [ -f \"config\/certs\/elkstack-certs\/elkstack-certs.crt\" ]; do\n          sleep 10;\n        done;\n        exec \/usr\/local\/bin\/docker-entrypoint.sh\n      '\n    environment:\n      - node.name=${NODE2_NAME}\n      - network.publish_host=${NODE2}\n      - cluster.name=${CLUSTER_NAME}\n      - bootstrap.memory_lock=true\n      - cluster.initial_master_nodes=${NODE1_NAME},${NODE2_NAME},${NODE3_NAME}\n      - discovery.seed_hosts=${NODE1},${NODE2},${NODE3}\n      - \"ES_JAVA_OPTS=-Xms1g -Xmx1g\"\n      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}\n      - xpack.security.enabled=true\n      - xpack.security.http.ssl.enabled=true\n      - xpack.security.enrollment.enabled=false\n      - xpack.security.autoconfiguration.enabled=false        \n      - xpack.security.transport.ssl.enabled=true\n      - xpack.security.http.ssl.key=certs\/elkstack-certs\/elkstack-certs.key\n      - xpack.security.http.ssl.certificate=certs\/elkstack-certs\/elkstack-certs.crt\n      - xpack.security.http.ssl.certificate_authorities=certs\/ca\/ca.crt\n      - xpack.security.transport.ssl.key=certs\/elkstack-certs\/elkstack-certs.key\n      - xpack.security.transport.ssl.certificate=certs\/elkstack-certs\/elkstack-certs.crt\n      - xpack.security.transport.ssl.certificate_authorities=certs\/ca\/ca.crt\n      - KIBANA_USERNAME=${KIBANA_USERNAME}\n      - KIBANA_PASSWORD=${KIBANA_PASSWORD}\n    ulimits:\n      memlock:\n        soft: -1\n        hard: -1\n    volumes:\n      - elasticsearch_data:\/usr\/share\/elasticsearch\/data\n      - certs:\/usr\/share\/elasticsearch\/config\/certs\n      - \/etc\/hosts:\/etc\/hosts\n    ports:\n      - 9200:9200\n      - 9300:9300\n    networks:\n      - elastic\n    healthcheck:\n      test: [\"CMD-SHELL\", \"curl --fail -k -s -u elastic:${ELASTIC_PASSWORD} --cacert config\/certs\/ca\/ca.crt https:\/\/${NODE2_NAME}:9200\"]\n      interval: 30s\n      timeout: 10s\n      retries: 5\n    restart: unless-stopped\n\n  kibana:\n    image: docker.elastic.co\/kibana\/kibana:${STACK_VERSION}\n    container_name: kibana02\n    environment:\n      - SERVER_NAME=${KIBANA_SERVER_HOST}\n      - ELASTICSEARCH_HOSTS=https:\/\/${NODE2_NAME}:9200\n      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config\/certs\/ca\/ca.crt\n      - ELASTICSEARCH_USERNAME=${KIBANA_USERNAME}\n      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}\n      - XPACK_REPORTING_ROLES_ENABLED=false\n      - XPACK_REPORTING_KIBANASERVER_HOSTNAME=localhost\n      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${SAVEDOBJECTS_ENCRYPTIONKEY}\n      - XPACK_SECURITY_ENCRYPTIONKEY=${REPORTING_ENCRYPTIONKEY}\n      - XPACK_REPORTING_ENCRYPTIONKEY=${SECURITY_ENCRYPTIONKEY}\n    volumes:\n      - kibana_data:\/usr\/share\/kibana\/data\n      - certs:\/usr\/share\/kibana\/config\/certs\n      - \/etc\/hosts:\/etc\/hosts\n    ports:\n      - ${KIBANA_PORT}:5601\n    networks:\n      - elastic\n    depends_on:\n      elasticsearch:\n        condition: service_healthy\n    restart: unless-stopped\n  \n  logstash:\n    image: docker.elastic.co\/logstash\/logstash:${STACK_VERSION}\n    container_name: logstash02\n    environment:\n      - XPACK_MONITORING_ENABLED=false\n      - ELASTICSEARCH_USERNAME=${ES_USER}\n      - ELASTICSEARCH_PASSWORD=${ELASTIC_PASSWORD}\n      - NODE_NAME=${NODE2_NAME}\n    ports:\n      - 5044:5044\n    volumes:\n      - certs:\/usr\/share\/logstash\/config\/certs\n      - logstash_filters:\/usr\/share\/logstash\/pipeline\/:ro\n      - logstash_data:\/usr\/share\/logstash\/data      \n      - \/etc\/hosts:\/etc\/hosts\n    networks:\n      - elastic\n    depends_on:\n      elasticsearch:\n        condition: service_healthy\n    restart: unless-stopped\n\nvolumes:\n  certs:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,ro\"\n      device: \":\/mnt\/elkstack\/certs\"\n  elasticsearch_data:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,rw\"\n      device: \":\/mnt\/elkstack\/data\/elasticsearch\/02\"\n  kibana_data:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,rw\"\n      device: \":\/mnt\/elkstack\/data\/kibana\/02\"\n  logstash_filters:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,ro\"\n      device: \":\/mnt\/elkstack\/configs\/logstash\"\n  logstash_data:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,rw\"\n      device: \":\/mnt\/elkstack\/data\/logstash\/02\"\n\nnetworks:\n  elastic:\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>cat .env<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code># Version of Elastic products\nSTACK_VERSION=8.12.0\n\n# Set the cluster name\nCLUSTER_NAME=elk-docker-cluster\n\n# Set Elasticsearch Node Name\nNODE1_NAME=es01\nNODE2_NAME=es02\nNODE3_NAME=es03\n\n# Docker Host IP to advertise to cluster nodes\nNODE1=192.168.122.60\nNODE2=192.168.122.123\nNODE3=192.168.122.152\n\n# Elasticsearch super user\nES_USER=elastic\n\n# Password for the 'elastic' user (at least 6 characters). No special characters, ! or @ or $.\nELASTIC_PASSWORD=ChangeME\n\n# Elasticsearch container name\nES_NAME=elasticsearch\n\n# Port to expose Elasticsearch HTTP API to the host\nES_PORT=9200\n#ES_PORT=127.0.0.1:9200\n\n# Port to expose Kibana to the host\nKIBANA_PORT=5601\nKIBANA_SERVER_HOST=localhost\n\n# Kibana Encryption. Requires atleast 32 characters. Can be generated using `openssl rand -hex 16\nSAVEDOBJECTS_ENCRYPTIONKEY=ca11560aec8410ff002d011c2a172608\nREPORTING_ENCRYPTIONKEY=288f06b3a14a7f36dd21563d50ec76d4\nSECURITY_ENCRYPTIONKEY=62c781d3a2b2eaee1d4cebcc6bf42b48\n\n# Kibana - Elasticsearch Authentication Credentials for user kibana_system\n# Password for the 'kibana_system' user (at least 6 characters). No special characters, ! or @ or $.\nKIBANA_USERNAME=kibana_system\nKIBANA_PASSWORD=ChangeME\n\n# Domain Suffix for ES Wildcard SSL certs\nDOMAIN_SUFFIX=kifarunix-demo.com\n\n# Generated Certs Validity Period\nDAYS=3650\n\n# Logstash Input Port\nLS_PORT=5044\n<\/code><\/pre>\n\n\n\n<p>On the third node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat docker-compose.yml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>services:\n  elasticsearch:\n    container_name: ${NODE3_NAME}\n    image: docker.elastic.co\/elasticsearch\/elasticsearch:${STACK_VERSION}\n    command: >\n      bash -c '\n      until [ -f \"config\/certs\/elkstack-certs\/elkstack-certs.crt\" ]; do\n          sleep 10;\n        done;\n        exec \/usr\/local\/bin\/docker-entrypoint.sh\n      '\n    environment:\n      - node.name=${NODE3_NAME}\n      - network.publish_host=${NODE3}\n      - cluster.name=${CLUSTER_NAME}\n      - bootstrap.memory_lock=true\n      - cluster.initial_master_nodes=${NODE1_NAME},${NODE2_NAME},${NODE3_NAME}\n      - discovery.seed_hosts=${NODE1},${NODE2},${NODE3}\n      - \"ES_JAVA_OPTS=-Xms1g -Xmx1g\"\n      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}\n      - xpack.security.enabled=true\n      - xpack.security.http.ssl.enabled=true\n      - xpack.security.transport.ssl.enabled=true\n      - xpack.security.enrollment.enabled=false\n      - xpack.security.autoconfiguration.enabled=false        \n      - xpack.security.http.ssl.key=certs\/elkstack-certs\/elkstack-certs.key\n      - xpack.security.http.ssl.certificate=certs\/elkstack-certs\/elkstack-certs.crt\n      - xpack.security.http.ssl.certificate_authorities=certs\/ca\/ca.crt\n      - xpack.security.transport.ssl.key=certs\/elkstack-certs\/elkstack-certs.key\n      - xpack.security.transport.ssl.certificate=certs\/elkstack-certs\/elkstack-certs.crt\n      - xpack.security.transport.ssl.certificate_authorities=certs\/ca\/ca.crt\n      - KIBANA_USERNAME=${KIBANA_USERNAME}\n      - KIBANA_PASSWORD=${KIBANA_PASSWORD}\n    ulimits:\n      memlock:\n        soft: -1\n        hard: -1\n    volumes:\n      - elasticsearch_data:\/usr\/share\/elasticsearch\/data\n      - certs:\/usr\/share\/elasticsearch\/config\/certs\n      - \/etc\/hosts:\/etc\/hosts\n    ports:\n      - 9200:9200\n      - 9300:9300\n    networks:\n      - elastic\n    healthcheck:\n      test: [\"CMD-SHELL\", \"curl --fail -k -s -u elastic:${ELASTIC_PASSWORD} --cacert config\/certs\/ca\/ca.crt https:\/\/${NODE3_NAME}:9200\"]\n      interval: 30s\n      timeout: 10s\n      retries: 5\n    restart: unless-stopped\n\n  kibana:\n    image: docker.elastic.co\/kibana\/kibana:${STACK_VERSION}\n    container_name: kibana\n    environment:\n      - SERVER_NAME=${KIBANA_SERVER_HOST}\n      - ELASTICSEARCH_HOSTS=https:\/\/${NODE3_NAME}:9200\n      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config\/certs\/ca\/ca.crt\n      - ELASTICSEARCH_USERNAME=${KIBANA_USERNAME}\n      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}\n      - XPACK_REPORTING_ROLES_ENABLED=false\n      - XPACK_REPORTING_KIBANASERVER_HOSTNAME=localhost\n      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${SAVEDOBJECTS_ENCRYPTIONKEY}\n      - XPACK_SECURITY_ENCRYPTIONKEY=${REPORTING_ENCRYPTIONKEY}\n      - XPACK_REPORTING_ENCRYPTIONKEY=${SECURITY_ENCRYPTIONKEY}\n    volumes:\n      - kibana_data:\/usr\/share\/kibana\/data\n      - certs:\/usr\/share\/kibana\/config\/certs\n      - \/etc\/hosts:\/etc\/hosts\n    ports:\n      - ${KIBANA_PORT}:5601\n    networks:\n      - elastic\n    depends_on:\n      elasticsearch:\n        condition: service_healthy\n    restart: unless-stopped\n  \n  logstash:\n    image: docker.elastic.co\/logstash\/logstash:${STACK_VERSION}\n    container_name: logstash\n    environment:\n      - XPACK_MONITORING_ENABLED=false\n      - ELASTICSEARCH_USERNAME=${ES_USER}\n      - ELASTICSEARCH_PASSWORD=${ELASTIC_PASSWORD}\n      - NODE_NAME=${NODE3_NAME}\n    ports:\n      - 5044:5044\n    volumes:\n      - certs:\/usr\/share\/logstash\/config\/certs\n      - logstash_filters:\/usr\/share\/logstash\/pipeline\/:ro\n      - logstash_data:\/usr\/share\/logstash\/data      \n      - \/etc\/hosts:\/etc\/hosts\n    networks:\n      - elastic\n    depends_on:\n      elasticsearch:\n        condition: service_healthy\n    restart: unless-stopped\n\nvolumes:\n  certs:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,ro\"\n      device: \":\/mnt\/elkstack\/certs\"\n  elasticsearch_data:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,rw\"\n      device: \":\/mnt\/elkstack\/data\/elasticsearch\/03\"\n  kibana_data:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,rw\"\n      device: \":\/mnt\/elkstack\/data\/kibana\/03\"\n  logstash_filters:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,ro\"\n      device: \":\/mnt\/elkstack\/configs\/logstash\"\n  logstash_data:\n    driver: local\n    driver_opts:\n      type: nfs\n      o: \"addr=192.168.122.60,nfsvers=4,rw\"\n      device: \":\/mnt\/elkstack\/data\/logstash\/03\"\n\nnetworks:\n  elastic:\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>cat .env<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code># Version of Elastic products\nSTACK_VERSION=8.12.0\n\n# Set the cluster name\nCLUSTER_NAME=elk-docker-cluster\n\n# Set Elasticsearch Node Name\nNODE1_NAME=es01\nNODE2_NAME=es02\nNODE3_NAME=es03\n\n# Docker Host IP to advertise to cluster nodes\nNODE1=192.168.122.60\nNODE2=192.168.122.123\nNODE3=192.168.122.152\n\n# Elasticsearch super user\nES_USER=elastic\n\n# Password for the 'elastic' user (at least 6 characters). No special characters, ! or @ or $.\nELASTIC_PASSWORD=ChangeME\n\n# Elasticsearch container name\nES_NAME=elasticsearch\n\n# Port to expose Elasticsearch HTTP API to the host\nES_PORT=9200\n#ES_PORT=127.0.0.1:9200\n\n# Port to expose Kibana to the host\nKIBANA_PORT=5601\nKIBANA_SERVER_HOST=localhost\n\n# Kibana Encryption. Requires atleast 32 characters. Can be generated using `openssl rand -hex 16\nSAVEDOBJECTS_ENCRYPTIONKEY=ca11560aec8410ff002d011c2a172608\nREPORTING_ENCRYPTIONKEY=288f06b3a14a7f36dd21563d50ec76d4\nSECURITY_ENCRYPTIONKEY=62c781d3a2b2eaee1d4cebcc6bf42b48\n\n# Kibana - Elasticsearch Authentication Credentials for user kibana_system\n# Password for the 'kibana_system' user (at least 6 characters). No special characters, ! or @ or $.\nKIBANA_USERNAME=kibana_system\nKIBANA_PASSWORD=ChangeME\n\n# Domain Suffix for ES Wildcard SSL certs\nDOMAIN_SUFFIX=kifarunix-demo.com\n\n# Generated Certs Validity Period\nDAYS=3650\n\n# Logstash Input Port\nLS_PORT=5044\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"validate-docker-compose-files\">Validate Docker Compose Files<\/h4>\n\n\n\n<p>It is important to check the syntax, environment variables, and other settings in your Docker compose file. This can be done using the command;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker-compose config<\/code><\/pre>\n\n\n\n<p>Or;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker compose config<\/code><\/pre>\n\n\n\n<p>The command will:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Validate the syntax of the <code>Docker compose<\/code> file.<\/li>\n\n\n\n<li>Parse the file and merge the configuration from all relevant sources (e.g., environment variables, variable interpolation).<\/li>\n\n\n\n<li>Output the final, resolved configuration.<\/li>\n<\/ol>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"start-elk-stack-8-docker-containers\">Start ELK Stack 8 Docker Containers<\/h4>\n\n\n\n<p>Once you have everything in place, then it is time to start the containers.<\/p>\n\n\n\n<p>Note, in our setup, we have to start the containers on the first on the first node. This is because, it is from this node that we are generating the shared SSL\/TLS certs required for the cluster.<\/p>\n\n\n\n<p>Thus, having confirmed all is set to go, fire up!<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker-compose up -d<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>[+] Running 37\/26\n \u2714 kibana 12 layers [\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff]      0B\/0B      Pulled                                                                                                                                      31.9s \n \u2714 es_setup Pulled                                                                                                                                                                             25.4s \n \u2714 elasticsearch 10 layers [\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff]      0B\/0B      Pulled                                                                                                                                 25.4s \n \u2714 logstash 11 layers [\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff\u28ff]      0B\/0B      Pulled                                                                                                                                     24.8s                                                                                                                                                                                                   \n...                                                                                                                                                                                                \n[+] Running 10\/10\n \u2714 Network elkstack-docker_elastic              Created                                                                                                                                         0.1s \n \u2714 Volume \"elkstack-docker_elasticsearch_data\"  Created                                                                                                                                         0.0s \n \u2714 Volume \"elkstack-docker_kibana_data\"         Created                                                                                                                                         0.0s \n \u2714 Volume \"elkstack-docker_certs\"               Created                                                                                                                                         0.0s \n \u2714 Volume \"elkstack-docker_logstash_filters\"    Created                                                                                                                                         0.0s \n \u2714 Volume \"elkstack-docker_logstash_data\"       Created                                                                                                                                         0.0s \n \u2714 Container elkstack-docker-es_setup-1         Healthy                                                                                                                                         0.6s \n \u2714 Container es01                               Healthy                                                                                                                                         0.2s \n \u2714 Container kibana                             Started                                                                                                                                         0.2s \n \u2714 Container logstash                           Started \n<\/code><\/pre>\n\n\n\n<p>Immediately the containers on the first node run, login to the two other nodes and laucnh the containers;<\/p>\n\n\n\n<p>Node2\/Node3;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker-compose up -d<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>[+] Running 9\/9\n \u2714 Network elkstack-docker_elastic              Created                                                                                                                                         0.1s \n \u2714 Volume \"elkstack-docker_certs\"               Created                                                                                                                                         0.0s \n \u2714 Volume \"elkstack-docker_kibana_data\"         Created                                                                                                                                         0.0s \n \u2714 Volume \"elkstack-docker_logstash_filters\"    Created                                                                                                                                         0.0s \n \u2714 Volume \"elkstack-docker_logstash_data\"       Created                                                                                                                                         0.0s \n \u2714 Volume \"elkstack-docker_elasticsearch_data\"  Created                                                                                                                                         0.0s \n \u2714 Container es02                               Healthy                                                                                                                                         0.3s \n \u2714 Container kibana02                           Started                                                                                                                                         0.1s \n \u2714 Container logstash02                         Started\n<\/code><\/pre>\n\n\n\n<p>If everything goes well, then your three node ELK Stack 8 cluster on Docker containers should be up now!<\/p>\n\n\n\n<p>Check containers running on first node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker ps<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>CONTAINER ID   IMAGE                                                  COMMAND                  CREATED         STATUS                   PORTS                                                                                  NAMES\n22fc737061c6   docker.elastic.co\/logstash\/logstash:8.12.0             \"\/usr\/local\/bin\/dock\u2026\"   4 minutes ago   Up 3 minutes             0.0.0.0:5044->5044\/tcp, :::5044->5044\/tcp, 9600\/tcp                                    logstash\ne71f04eefbb4   docker.elastic.co\/kibana\/kibana:8.12.0                 \"\/bin\/tini -- \/usr\/l\u2026\"   4 minutes ago   Up 3 minutes             0.0.0.0:5601->5601\/tcp, :::5601->5601\/tcp                                              kibana\n8e623c5e277e   docker.elastic.co\/elasticsearch\/elasticsearch:8.12.0   \"\/bin\/tini -- \/usr\/l\u2026\"   4 minutes ago   Up 4 minutes (healthy)   0.0.0.0:9200->9200\/tcp, :::9200->9200\/tcp, 0.0.0.0:9300->9300\/tcp, :::9300->9300\/tcp   es01\n<\/code><\/pre>\n\n\n\n<p>You do the same on other nodes.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"verify-the-elk-stack-cluster\">Verify the ELK Stack Cluster<\/h3>\n\n\n\n<p>Now, check if the cluster is up and running.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker exec -it es01 bash -c 'curl -sk -XGET https:\/\/es01:9200\/_cat\/nodes?pretty -u elastic'<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>192.168.122.152 15 98  6 0.13 0.54 0.34 dm - es03\n192.168.122.123 51 98  8 0.58 0.74 0.43 dm * es02\n192.168.122.60  28 98 16 1.72 1.88 1.10 dm - es01\n<\/code><\/pre>\n\n\n\n<p>Hurray! We have the cluster up and running! Check, container on node 2 is the master now!<\/p>\n\n\n\n<p>You can also check individual container logs under <strong><code>\/var\/lib\/docker\/containers\/<\/code><\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo tail -f \/var\/lib\/docker\/containers\/*\/*.log<\/code><\/pre>\n\n\n\n<p>You can replace asterisk with individual containers id to check each&#8217;s container logs.<\/p>\n\n\n\n<p>You can also grep keywords like added|master node|elected-as in the logs;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>grep -irE --color 'added|master node|elected-as' \/var\/lib\/docker\/containers\/*\/*.log<\/code><\/pre>\n\n\n\n<p>Check the ports exposed by the services on the host;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ss -atlnp<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess                                                     \nLISTEN 0      4096         0.0.0.0:5601       0.0.0.0:*    users:((\"docker-proxy\",pid=2945394,fd=4))                  \nLISTEN 0      4096         0.0.0.0:9300       0.0.0.0:*    users:((\"docker-proxy\",pid=2944085,fd=4))                  \nLISTEN 0      128          0.0.0.0:22         0.0.0.0:*    users:((\"sshd\",pid=710,fd=3))                              \nLISTEN 0      4096         0.0.0.0:5044       0.0.0.0:*    users:((\"docker-proxy\",pid=2945415,fd=4))                  \nLISTEN 0      4096         0.0.0.0:9200       0.0.0.0:*    users:((\"docker-proxy\",pid=2944108,fd=4))                  \nLISTEN 0      4096            [::]:5601          [::]:*    users:((\"docker-proxy\",pid=2945401,fd=4))                  \nLISTEN 0      4096            [::]:9300          [::]:*    users:((\"docker-proxy\",pid=2944092,fd=4))                  \nLISTEN 0      128             [::]:22            [::]:*    users:((\"sshd\",pid=710,fd=4))                              \nLISTEN 0      4096            [::]:5044          [::]:*    users:((\"docker-proxy\",pid=2945421,fd=4))                  \nLISTEN 0      4096            [::]:9200          [::]:*    users:((\"docker-proxy\",pid=2944115,fd=4)) \n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"accessing-kibana\">Accessing Kibana<\/h3>\n\n\n\n<p>Note that in our setup, we are running complete ELK on the three nodes. However, the Elasticsearch is what configured to be in cluster. Hence, you can access the Kibana on any node IP address and still access the same data.<\/p>\n\n\n\n<p>Let&#8217;s check Kibana on the first node, <strong><code>http:\/\/192.168.122.60:5601<\/code><\/strong>.<\/p>\n\n\n\n<p>We are using the superuser credentials to login!<\/p>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2024\\\/01\\\/elk-stack-8-cluster-kibana.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-19927&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1444,&quot;targetHeight&quot;:690,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image: Deploy ELK Stack 8 Cluster on Docker Containers&quot;,&quot;alt&quot;:&quot;Deploy ELK Stack 8 Cluster on Docker Containers&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1444\" height=\"690\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/elk-stack-8-cluster-kibana.png?v=1705788579\" alt=\"Deploy ELK Stack 8 Cluster on Docker Containers\" class=\"wp-image-19927\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/elk-stack-8-cluster-kibana.png?v=1705788579 1444w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/elk-stack-8-cluster-kibana-768x367.png?v=1705788579 768w\" sizes=\"(max-width: 1444px) 100vw, 1444px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image: Deploy ELK Stack 8 Cluster on Docker Containers\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<p>Explore ELK stack 8 cluster dashboards!<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"sending-data-logs-to-elastic-stack\">Sending Data Logs to Elastic Stack<\/h3>\n\n\n\n<p>Since we configured our Logstash receive event data from the Beats on port 5044\/tcp, we will configure Filebeat to forward events to any of the three <\/p>\n\n\n\n<p>We already covered how to <a href=\"https:\/\/kifarunix.com\/?s=install+filebeat\" target=\"_blank\" rel=\"noreferrer noopener\">install and configure Filebeat<\/a> to forward event data in our previous guides;<\/p>\n\n\n\n<p>Once you forward data to your Logstash container, the next thing you need to do is create Kibana index.<\/p>\n\n\n\n<p>Open the menu, then go to&nbsp;<strong>Stack Management<\/strong>&nbsp;&gt;&nbsp;<strong>Kibana<\/strong>&nbsp;&gt;&nbsp;<strong>Index<\/strong>&nbsp;Patterns.<\/p>\n\n\n\n<p>Once done, heading to Discover menu to view your data. You should now be able to see your Logstash custom fields populated.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1619\" height=\"842\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/logstash-8-event-data.png\" alt=\"\" class=\"wp-image-19930\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/logstash-8-event-data.png?v=1705790742 1619w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/logstash-8-event-data-768x399.png?v=1705790742 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/logstash-8-event-data-1536x799.png?v=1705790742 1536w\" sizes=\"(max-width: 1619px) 100vw, 1619px\" \/><\/figure>\n\n\n\n<p>That marks the end of our tutorial on how to deploy a single node Elastic Stack cluster on Docker Containers.<\/p>\n\n\n\n<p>More tutorials;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/deploy-elk-stack-8-on-docker-containers\/\" target=\"_blank\" rel=\"noreferrer noopener\">Deploy ELK Stack 8 on Docker Containers<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/configure-kibana-dashboards-visualizations-to-use-custom-index\/\" target=\"_blank\" rel=\"noreferrer noopener\">Configure Kibana Dashboards\/Visualizations to use Custom Index<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Follow through this guide to learn how to deploy ELK Stack 8 cluster on Docker containers. Deploying a multinode ELK Stack 8 cluster on Docker<\/p>\n","protected":false},"author":10,"featured_media":19906,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[72,1076,1077,910,121],"tags":[7363,7364,7365],"class_list":["post-19854","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-monitoring","category-containers","category-docker","category-elastic-stack","category-howtos","tag-elk-stack-8-cluster-docker-containers","tag-elk-stack-8-multinode-containers","tag-separate-nodes-for-elk-stack-8-containers","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19854"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=19854"}],"version-history":[{"count":38,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19854\/revisions"}],"predecessor-version":[{"id":20908,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19854\/revisions\/20908"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/19906"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=19854"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=19854"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=19854"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}