{"id":19888,"date":"2024-01-17T00:10:16","date_gmt":"2024-01-16T21:10:16","guid":{"rendered":"https:\/\/kifarunix.com\/?p=19888"},"modified":"2024-03-10T15:35:01","modified_gmt":"2024-03-10T12:35:01","slug":"deploy-elk-stack-8-on-docker-containers","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/deploy-elk-stack-8-on-docker-containers\/","title":{"rendered":"Deploy ELK Stack 8 on Docker Containers"},"content":{"rendered":"\n<p>How to run single node ELK stack 8 on Docker? In this tutorial, you will learn how to deploy <a href=\"https:\/\/www.elastic.co\/elastic-stack\" target=\"_blank\" rel=\"noreferrer noopener\">ELK stack<\/a> 8 on Docker containers. ELK stack is a group of open source software projects: Elasticsearch, Logstash, and Kibana and Beats, where:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Elasticsearch is a search and analytics engine<\/li>\n\n\n\n<li>Logstash is a server\u2011side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a &#8220;stash&#8221; like Elasticsearch<\/li>\n\n\n\n<li>Kibana lets users visualize data with charts and graphs in Elasticsearch and <\/li>\n\n\n\n<li><a href=\"https:\/\/www.elastic.co\/beats\/\" target=\"_blank\" rel=\"noreferrer noopener\">Beats<\/a> are the data shippers. They ship system logs, network, infrastructure data, etc to either Logstash for further processing or Elasticsearch for indexing.<\/li>\n<\/ul>\n\n\n\n<p>We will be running Elasticsearch 8.11.4, Logstash 8.11.4, Kibana 8.11.4, as Docker containers in this tutorial.<\/p>\n\n\n\n<p>Kindly note that Elasticsearch 8.x enables authentication and TLS\/SSL connection by default. <\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#deploying-elk-stack-8-on-docker-containers\">Deploying ELK Stack 8 on Docker Containers<\/a><ul><li><a href=\"#install-docker-engine\">Install Docker Engine<\/a><\/li><li><a href=\"#install-docker-compose\">Install Docker Compose<\/a><\/li><li><a href=\"#running-docker-as-non-root-user\">Running Docker as Non-Root User<\/a><\/li><li><a href=\"#important-elasticsearch-settings\">Important Elasticsearch Settings<\/a><ul><li><a href=\"#disable-memory-swapping-on-a-container\">Disable Memory Swapping on a Container<\/a><\/li><li><a href=\"#set-jvm-heap-size-on-all-cluster-nodes\">Set JVM Heap Size on All Cluster Nodes<\/a><\/li><li><a href=\"#set-maximum-open-file-descriptor-and-processes-on-elasticsearch-containers\">Set Maximum Open File Descriptor and Processes on Elasticsearch Containers<\/a><\/li><li><a href=\"#update-virtual-memory-settings-on-all-cluster-nodes\">Update Virtual Memory Settings on All Cluster Nodes<\/a><\/li><\/ul><\/li><li><a href=\"#deploying-elk-stack-8-on-docker-using-docker-compose\">Deploying ELK Stack 8 on Docker Using Docker Compose<\/a><ul><li><a href=\"#create-docker-compose-file-for-elk-stack-8-services\">Create Docker Compose file for ELK Stack 8 Services<\/a><\/li><li><a href=\"#create-elk-stack-8-docker-compose-environment-variables\">Create ELK Stack 8 Docker Compose Environment Variables<\/a><\/li><li><a href=\"#define-logstash-data-processing-pipeline\">Define Logstash Data Processing Pipeline<\/a><\/li><li><a href=\"#verify-docker-compose-file-syntax\">Verify Docker Compose File Syntax<\/a><\/li><li><a href=\"#deploy-elk-stack-8-using-docker-compose-file\">Deploy ELK Stack 8 Using Docker Compose file<\/a><\/li><\/ul><\/li><\/ul><\/li><li><a href=\"#accessing-kibana-container-from-browser\">Accessing Kibana Container from Browser<\/a><\/li><li><a href=\"#sending-data-logs-to-elastic-stack\">Sending Data Logs to Elastic Stack<\/a><\/li><li><a href=\"#other-tutorials\">Other Tutorials<\/a><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"deploying-elk-stack-8-on-docker-containers\">Deploying ELK Stack 8 on Docker Containers<\/h2>\n\n\n\n<p>In this tutorial, therefore, we will learn how to deploy a single node ELK Stack 8 on Docker containers using Docker and Docker compose. We will run all required ELK stack 8 Docker containers on a single node.<\/p>\n\n\n\n<p><a href=\"https:\/\/docs.docker.com\/get-started\/\" target=\"_blank\" rel=\"noreferrer noopener\">Docker<\/a>&nbsp;is a platform that enables developers and system administrators to&nbsp;<strong>build, run, and share<\/strong>&nbsp;applications with containers. It provides command line interface tools such as <code>docker<\/code> and <code>docker-compose<\/code> that are used for managing Docker containers. While docker is a Docker cli for managing single Docker containers, docker-compose on the other hand is used for running and managing multiple Docker containers.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-docker-engine\">Install Docker Engine<\/h3>\n\n\n\n<p>Depending on the your host system distribution, you need to install the Docker engine. You can follow the links below to install Docker Engine on Ubuntu\/Debian\/CentOS 8.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/?s=install+docker+ce\" target=\"_blank\" rel=\"noreferrer noopener\">Install and Use Docker on Linux<\/a><\/p>\n\n\n\n<p>Checking Installed Docker version;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker version<\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Docker version 24.0.7, build afdd53b<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-docker-compose\">Install Docker Compose<\/h3>\n\n\n\n<p>For Docker compose to work, ensure that you have Docker Engine installed. You can follow the links above to install Docker Engine.<\/p>\n\n\n\n<p>Once you have the Docker engine installed, proceed to install Docker compose.<\/p>\n\n\n\n<p>Check the current stable release version of Docker Compose on their&nbsp;<a href=\"https:\/\/github.com\/docker\/compose\/releases\" target=\"_blank\" rel=\"noreferrer noopener\">Github release page<\/a>. As of this writing, the Docker Compose version 2.24.0 is the current stable release.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>VER=2.24.0<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo curl -L \"https:\/\/github.com\/docker\/compose\/releases\/download\/v$VER\/docker-compose-$(uname -s)-$(uname -m)\" -o \/usr\/local\/bin\/docker-compose<\/code><\/pre>\n\n\n\n<p>This downloads docker compose tool to&nbsp;<code>\/usr\/local\/bin<\/code>&nbsp;directory.<\/p>\n\n\n\n<p>Make the Docker compose binary executable;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>chmod +x \/usr\/local\/bin\/docker-compose<\/code><\/pre>\n\n\n\n<p>You should now be able to use Docker compose (<code><strong>docker-compose<\/strong><\/code>) on the CLI.<\/p>\n\n\n\n<p>Check the version of installed Docker compose to confirm that it is working as expected.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker-compose version<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Docker Compose version v2.24.0<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"running-docker-as-non-root-user\">Running Docker as Non-Root User<\/h3>\n\n\n\n<p>We are running both Docker and Docker compose as non root user. To be able to do this, ensure that your add your standard user to the <code><strong>docker<\/strong><\/code> group.<\/p>\n\n\n\n<p>For example, am running this setup as user <code>kifarunix<\/code>. So, add the user to Docker group.  Replace the username accordingly.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo usermod -aG docker kifarunix<\/pre>\n\n\n\n<p>Log out and log in again as the user that is added to the docker group and you should be able to run the docker and docker-compose CLI tools.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"important-elasticsearch-settings\">Important Elasticsearch Settings<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"disable-memory-swapping-on-a-container\">Disable Memory Swapping on a Container<\/h4>\n\n\n\n<p>There are different ways to disable swapping on Elasticsearch. For an Elasticsearch Docker container, you can simply set the value of the <strong><code>bootstrap.memory_lock<\/code><\/strong> option to <strong><code>true<\/code><\/strong>. With this option, you also need to ensure that there is no maximum limit imposed on the locked-in-memory address space (<strong><code>LimitMEMLOCK=infinity<\/code><\/strong>). We will see how this can be achieved in a Docker compose file.<\/p>\n\n\n\n<p>You can confirm the default value set on the Docker image using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker run --rm docker.elastic.co\/elasticsearch\/elasticsearch:8.11.4 \/bin\/bash -c 'ulimit -Hm &amp;&amp; ulimit -Sm'<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>unlimited\nunlimited\n<\/code><\/pre>\n\n\n\n<p>If the value, other than unlimited is printed, then you have to set this via the Docker compose file to ensure that it set to unlimited for both Soft and Hard limits.<\/p>\n\n\n\n<p>For a respective service, this is how this config looks like in Docker compose file.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>    ulimits:\n      memlock:\n        soft: -1\n        hard: -1\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-jvm-heap-size-on-all-cluster-nodes\">Set JVM Heap Size on All Cluster Nodes<\/h4>\n\n\n\n<p>Elasticsearch usually sets the heap size automatically based on the roles of the nodes\u2019s&nbsp;roles&nbsp;and the total memory available to the node\u2019s container. This is the recommended approach! For the Elasticsearch Docker containers, any custom JVM settings can be defined using the <strong><code>ES_JAVA_OPTS<\/code><\/strong> environment variable, either in compose file or on command line. For example, you can set JVM heap size, Min and Max to 512M using the <strong><code>ES_JAVA_OPTS=-Xms512m -Xmx512m<\/code><\/strong>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-maximum-open-file-descriptor-and-processes-on-elasticsearch-containers\">Set Maximum Open File Descriptor and Processes on Elasticsearch Containers<\/h4>\n\n\n\n<p>Set the maximum number of open files (<code>nofile<\/code>) for the&nbsp;containers to 65,536 and max number of processes to 4096 (both soft and hard limits).<\/p>\n\n\n\n<p>We will also define these limits in the compose file. However, the Elasticsearch images usually comes with these limits already defined. You can use the command below to check the maximum number of user processes (-u) and max number of opens files (-n);<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker run --rm docker.elastic.co\/elasticsearch\/elasticsearch:8.11.4 \/bin\/bash -c 'ulimit -Hn &amp;&amp; ulimit -Sn &amp;&amp; ulimit -Hu &amp;&amp; ulimit -Su'<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>1048576\n1048576\nunlimited\nunlimited\n<\/code><\/pre>\n\n\n\n<p>These values may look like this on Docker compose file. Adjust them accordingly!<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>    ulimits:\n      nofile:\n        soft: 65536\n        hard: 65536\n      nproc:\n        soft: 2048\n        hard: 2048\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"update-virtual-memory-settings-on-all-cluster-nodes\">Update Virtual Memory Settings on All Cluster Nodes<\/h4>\n\n\n\n<p>Elasticsearch uses a&nbsp;<a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/index-modules-store.html#mmapfs\" target=\"_blank\" rel=\"noreferrer noopener\"><code>mmapfs<\/code><\/a>&nbsp;directory by default to store its indices. To ensure that you do not run out of virtual memory, edit the&nbsp;<strong>\/etc\/sysctl.conf<\/strong>&nbsp;on the <strong><code>Docker host<\/code><\/strong> and update the value of&nbsp;<strong>vm.max_map_count<\/strong>&nbsp;as shown below.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vm.max_map_count=262144<\/code><\/pre>\n\n\n\n<p>On the Docker host, simply run the command below to configure virtual memory settings.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"vm.max_map_count=262144\" &gt;&gt; \/etc\/sysctl.conf<\/code><\/pre>\n\n\n\n<p>To apply the changes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sysctl -p<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploying-elk-stack-8-on-docker-using-docker-compose\">Deploying ELK Stack 8 on Docker Using Docker Compose<\/h3>\n\n\n\n<p>In this setup, we will deploy a single node ELK stack 8 cluster with all the three components, Elasticsearch, Logstash and Kibana containers running on the same host as Docker containers.<\/p>\n\n\n\n<p>If you are running Elastic stack on Docker containers for development or testing purposes, then you can run each component Docker container using <strong><code>docker run<\/code><\/strong> commands. Use of Docker compose is recommended for production containers!<\/p>\n\n\n\n<p>To begin, create a parent directory from where you will build your stack from.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">mkdir $HOME\/elastic-docker<\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-docker-compose-file-for-elk-stack-8-services\">Create Docker Compose file for ELK Stack 8 Services<\/h4>\n\n\n\n<p><em>Docker compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application\u2019s services. Then, with a single command, you create and start all the services from your configuratio<\/em>n.<\/p>\n\n\n\n<p><em>Using Compose is basically a three-step process:<\/em><\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><em>Define your app\u2019s environment with a&nbsp;<code>Dockerfile<\/code>&nbsp;so it can be reproduced anywhere.<\/em><\/li>\n\n\n\n<li><em>Define the services that make up your app in&nbsp;<code>docker-compose.yml<\/code>&nbsp;so they can be run together in an isolated environment.<\/em><\/li>\n\n\n\n<li><em>Run&nbsp;<code>docker-compose up<\/code>&nbsp;and Compose starts and runs your entire app<\/em><\/li>\n<\/ol>\n\n\n\n<p>In this setup, we will build everything using a Docker Compose file.<\/p>\n\n\n\n<p>Now it is time to create the Docker Compose file for our deployment. Note, we will be using Docker images of the current stable release versions of the Elastic components, v8.11.4.<\/p>\n\n\n\n<figure class=\"wp-block-pullquote has-small-font-size\"><blockquote><p>The default path for a Compose file is&nbsp;<code>compose.yaml<\/code>&nbsp;(preferred) or&nbsp;<code>compose.yml<\/code>&nbsp;that is placed in the working directory. Compose also supports&nbsp;<code>docker-compose.yaml<\/code>&nbsp;and&nbsp;<code>docker-compose.yml<\/code>&nbsp;for backwards compatibility of earlier versions. If both files exist, Compose prefers the canonical&nbsp;<code>compose.yaml<\/code>.<\/p><\/blockquote><\/figure>\n\n\n\n<pre class=\"wp-block-code\"><code>cd $HOME\/elastic-docker<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">vim docker-compose.yml<\/pre>\n\n\n\n<p>Usually, when deploying ELK stack, the specific order to follow in the setup is Elasticsearch &gt; Kibana &gt; Logstash &gt; and finally Beat agents.<\/p>\n\n\n\n<p>Similarly, while defining the ELK stack 8 services in the Docker compose file, Elasticsearch should be the first container to be deployed followed by the rest of the components.<\/p>\n\n\n\n<p>Below is our Docker compose outlining the ELK stack services to be deployed.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>version: '3.8'\n\nservices:\n  es_setup:\n    image: docker.elastic.co\/elasticsearch\/elasticsearch:${STACK_VERSION}\n    volumes:\n      - certs:\/usr\/share\/elasticsearch\/config\/certs\n    user: \"0\"\n    command: >\n      bash -c '\n        echo \"Creating ES certs directory...\"\n        [[ -d config\/certs ]] || mkdir config\/certs\n        # Check if CA certificate exists\n        if [ ! -f config\/certs\/ca\/ca.crt ]; then\n          echo \"Generating Wildcard SSL certs for ES (in PEM format)...\"\n          bin\/elasticsearch-certutil ca --pem --days 3650 --out config\/certs\/elkstack-ca.zip\n          unzip -d config\/certs config\/certs\/elkstack-ca.zip\n          bin\/elasticsearch-certutil cert \\\n            --name elkstack-certs \\\n            --ca-cert config\/certs\/ca\/ca.crt \\\n            --ca-key config\/certs\/ca\/ca.key \\\n            --pem \\\n            --dns \"*.${DOMAIN_SUFFIX},${ES_NAME}\" \\\n            --days ${DAYS} \\\n            --out config\/certs\/elkstack-certs.zip\n          unzip -d config\/certs config\/certs\/elkstack-certs.zip\n        else\n          echo \"CA certificate already exists. Skipping Certificates generation.\"\n        fi\n    \n        until curl -s --cacert config\/certs\/ca\/ca.crt -u \"elastic:${ELASTIC_PASSWORD}\" https:\/\/${ES_NAME}:9200 | grep -q \"${CLUSTER_NAME}\"; do sleep 10; done\n        until curl -s -XPOST --cacert config\/certs\/ca\/ca.crt -u \"elastic:${ELASTIC_PASSWORD}\" -H \"Content-Type: application\/json\" https:\/\/${ES_NAME}:9200\/_security\/user\/kibana_system\/_password -d \"{\\\"password\\\":\\\"${KIBANA_PASSWORD}\\\"}\" | grep -q \"^{}\"; do sleep 10; done\n        echo \"Setup is done!\"\n      '\n    networks:\n      - elastic\n    healthcheck:\n      test: [\"CMD-SHELL\", \"[ -f config\/certs\/elkstack-certs\/elkstack-certs.crt ]\"]\n      interval: 1s\n      timeout: 5s\n      retries: 120\n\n  elasticsearch:\n    depends_on:\n      es_setup:\n        condition: service_healthy\n    container_name: ${ES_NAME}\n    image: docker.elastic.co\/elasticsearch\/elasticsearch:${STACK_VERSION}\n    environment:\n      - node.name=${NODE_NAME}\n      - cluster.name=${CLUSTER_NAME}\n      - bootstrap.memory_lock=true\n      - \"ES_JAVA_OPTS=-Xms1g -Xmx1g\"\n      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}\n      - xpack.security.enabled=true\n      - xpack.security.http.ssl.enabled=true\n      - xpack.security.transport.ssl.enabled=true\n      - xpack.security.http.ssl.key=certs\/elkstack-certs\/elkstack-certs.key\n      - xpack.security.http.ssl.certificate=certs\/elkstack-certs\/elkstack-certs.crt\n      - xpack.security.http.ssl.certificate_authorities=certs\/ca\/ca.crt\n      - xpack.security.transport.ssl.key=certs\/elkstack-certs\/elkstack-certs.key\n      - xpack.security.transport.ssl.certificate=certs\/elkstack-certs\/elkstack-certs.crt\n      - xpack.security.transport.ssl.certificate_authorities=certs\/ca\/ca.crt\n      - cluster.initial_master_nodes=${NODE_NAME}\n      - KIBANA_USERNAME=${KIBANA_USERNAME}\n      - KIBANA_PASSWORD=${KIBANA_PASSWORD}\n    ulimits:\n      memlock:\n        soft: -1\n        hard: -1\n    volumes:\n      - elasticsearch-data:\/usr\/share\/elasticsearch\/data\n      - certs:\/usr\/share\/elasticsearch\/config\/certs\n    ports:\n      - 9200:9200\n      - 9300:9300\n    networks:\n      - elastic\n    healthcheck:\n      test: [\"CMD-SHELL\", \"curl --fail -k -s -u elastic:${ELASTIC_PASSWORD} --cacert config\/certs\/ca\/ca.crt https:\/\/${ES_NAME}:9200\"]\n      interval: 30s\n      timeout: 10s\n      retries: 5\n    restart: unless-stopped\n\n  kibana:\n    image: docker.elastic.co\/kibana\/kibana:${STACK_VERSION}\n    container_name: kibana\n    environment:\n      - SERVER_NAME=${KIBANA_SERVER_HOST}\n      - ELASTICSEARCH_HOSTS=https:\/\/${ES_NAME}:9200\n      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config\/certs\/ca\/ca.crt\n      - ELASTICSEARCH_USERNAME=${KIBANA_USERNAME}\n      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}\n      - XPACK_REPORTING_ROLES_ENABLED=false\n      - XPACK_REPORTING_KIBANASERVER_HOSTNAME=localhost\n      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${SAVEDOBJECTS_ENCRYPTIONKEY}\n      - XPACK_SECURITY_ENCRYPTIONKEY=${REPORTING_ENCRYPTIONKEY}\n      - XPACK_REPORTING_ENCRYPTIONKEY=${SECURITY_ENCRYPTIONKEY}\n    volumes:\n      - kibana-data:\/usr\/share\/kibana\/data\n      - certs:\/usr\/share\/kibana\/config\/certs\n    ports:\n      - ${KIBANA_PORT}:5601\n    networks:\n      - elastic\n    depends_on:\n      elasticsearch:\n        condition: service_healthy\n    restart: unless-stopped\n  \n  logstash:\n    image: docker.elastic.co\/logstash\/logstash:${STACK_VERSION}\n    container_name: logstash\n    environment:\n      - XPACK_MONITORING_ENABLED=false\n      - ELASTICSEARCH_USERNAME=${ES_USER}\n      - ELASTICSEARCH_PASSWORD=${ELASTIC_PASSWORD}\n      - ES_NAME=${ES_NAME}\n    ports:\n      - 5044:5044\n    volumes:\n      - .\/logstash\/conf.d\/:\/usr\/share\/logstash\/pipeline\/:ro\n      - certs:\/usr\/share\/logstash\/config\/certs\n      - logstash-data:\/usr\/share\/logstash\/data\n    networks:\n      - elastic\n    depends_on:\n      elasticsearch:\n        condition: service_healthy\n    restart: unless-stopped\n\nvolumes:\n  elasticsearch-data:\n    driver: local\n  kibana-data:\n    driver: local\n  logstash-data:\n    driver:\n      local\n  certs:\n    driver: local\n\nnetworks:\n  elastic:\n<\/code><\/pre>\n\n\n\n<p>In summary:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>This Docker Compose file defines a multi-container setup for the Elasticsearch (ES) stack, including Elasticsearch, Kibana, and Logstash, along with a setup service (<code>es_setup<\/code>).<\/li>\n\n\n\n<li>The <code>es_setup<\/code> service initializes the Elasticsearch environment, generating SSL certificates, setting up security configurations, and ensuring the cluster is healthy before starting the other services.<\/li>\n\n\n\n<li>The <code>elasticsearch<\/code> service runs the Elasticsearch node with specified configurations, including security settings, network configurations, and health checks.<\/li>\n\n\n\n<li>The <code>kibana<\/code> service sets up Kibana with connections to Elasticsearch and SSL configurations.<\/li>\n\n\n\n<li>Lastly, the <code>logstash<\/code> service runs Logstash with configurations for Elasticsearch connections and SSL settings.<\/li>\n\n\n\n<li>The <code>volumes<\/code> section defines data volumes for Elasticsearch, Kibana, Logstash, and SSL certificates.<\/li>\n\n\n\n<li>The entire setup is connected through a Docker network named <code>elastic<\/code>.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-elk-stack-8-docker-compose-environment-variables\">Create ELK Stack 8 Docker Compose Environment Variables<\/h4>\n\n\n\n<p>You can define all the values that are dynamic using environment variables in a <a href=\"https:\/\/docs.docker.com\/compose\/environment-variables\/set-environment-variables\/#substitute-with-an-env-file\" target=\"_blank\" rel=\"noreferrer noopener\"><strong><code>.env<\/code><\/strong> file<\/a>. This environment variables file should reside in the same directory are compose file.<\/p>\n\n\n\n<p>See below how we have assigned actual values to the environment variables referenced in the compose file.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat .env<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code># Version of Elastic products\nSTACK_VERSION=8.11.4\n\n# Set the cluster name\nCLUSTER_NAME=elk-docker-cluster\n\n# Set Elasticsearch Node Name\nNODE_NAME=es01\n\n# Elasticsearch super user\nES_USER=elastic\n\n# Password for the 'elastic' user (at least 6 characters). No special characters, ! or @ or $.\nELASTIC_PASSWORD=ChangeME\n\n# Elasticsearch container name\nES_NAME=elasticsearch\n\n# Port to expose Elasticsearch HTTP API to the host\nES_PORT=9200\n#ES_PORT=127.0.0.1:9200\n\n# Port to expose Kibana to the host\nKIBANA_PORT=5601\nKIBANA_SERVER_HOST=localhost\n\n# Kibana Encryption. Requires atleast 32 characters. Can be generated using `openssl rand -hex 16`\nSAVEDOBJECTS_ENCRYPTIONKEY=ca11560aec8410ff002d011c2a172608\nREPORTING_ENCRYPTIONKEY=288f06b3a14a7f36dd21563d50ec76d4\nSECURITY_ENCRYPTIONKEY=62c781d3a2b2eaee1d4cebcc6bf42b48\n\n# Kibana - Elasticsearch Authentication Credentials for user kibana_system\n# Password for the 'kibana_system' user (at least 6 characters). No special characters, ! or @ or $.\nKIBANA_USERNAME=kibana_system\nKIBANA_PASSWORD=ChangeME\n\n# Domain Suffix for ES Wildcard SSL certs\nDOMAIN_SUFFIX=kifarunix-demo.com\n\n# Elasticsearch Certificate Validity Period\nDAYS=3650\n\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"define-logstash-data-processing-pipeline\">Define Logstash Data Processing Pipeline<\/h4>\n\n\n\n<p>In this setup, we will configure Logstash to receive event data from Beats (Filebeat to be specific) for further processing and stashing onto the search analytics engine, Elasticsearch.<\/p>\n\n\n\n<p>Note that Logstash is only necessary if you need to apply further processing to your event data. For example, extracting custom fields from the event data, mutating the event data etc. Otherwise, you can push the data directly to Elasticsearch from Beats.<\/p>\n\n\n\n<p>So we use a sample Logstash processing pipeline for ModSecurity audit logs;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">mkdir -p logstash\/conf.d<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">vim logstash\/conf.d\/modsec.conf<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\ninput {\n  beats {\n    port =&gt; 5044\n  }\n}\nfilter {\n    # Extract event time, log severity level, source of attack (client), and the alert message.\n    grok {\n      match =&gt; { \"message\" =&gt; \"(?&lt;event_time&gt;%{MONTH}\\s%{MONTHDAY}\\s%{TIME}\\s%{YEAR})\\] \\[\\:%{LOGLEVEL:log_level}.*client\\s%{IPORHOST:src_ip}:\\d+]\\s(?&lt;alert_message&gt;.*)\" }\n    }\n    # Extract Rules File from Alert Message\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(?&lt;rulesfile&gt;\\[file \\\"(\/.+.conf)\\\"\\])\" }\n    }\t\n    grok {\n      match =&gt; { \"rulesfile\" =&gt; \"(?&lt;rules_file&gt;\/.+.conf)\" }\n    }\t\n    # Extract Attack Type from Rules File\n    grok {\n      match =&gt; { \"rulesfile\" =&gt; \"(?&lt;attack_type&gt;[A-Z]+-[A-Z][^.]+)\" }\n    }\t\n    # Extract Rule ID from Alert Message\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(?&lt;ruleid&gt;\\[id \\\"(\\d+)\\\"\\])\" }\n    }\t\n    grok {\n      match =&gt; { \"ruleid\" =&gt; \"(?&lt;rule_id&gt;\\d+)\" }\n    }\n    # Extract Attack Message (msg) from Alert Message \t\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(?&lt;msg&gt;\\[msg \\S(.*?)\\\"\\])\" }\n    }\t\n    grok {\n      match =&gt; { \"msg\" =&gt; \"(?&lt;alert_msg&gt;\\\"(.*?)\\\")\" }\n    }\n    # Extract the User\/Scanner Agent from Alert Message\t\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(?&lt;scanner&gt;User-Agent' \\SValue: `(.*?)')\" }\n    }\t\n    grok {\n      match =&gt; { \"scanner\" =&gt; \"(?&lt;user_agent&gt;:(.*?)\\')\" }\n    }\t\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(?&lt;agent&gt;User-Agent: (.*?)\\')\" }\n    }\t\n    grok {\n      match =&gt; { \"agent\" =&gt; \"(?&lt;user_agent&gt;: (.*?)\\')\" }\n    }\t\n    # Extract the Target Host\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(hostname \\\"%{IPORHOST:dst_host})\" }\n    }\t\n    # Extract the Request URI\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(uri \\\"%{URIPATH:request_uri})\" }\n    }\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(?&lt;ref&gt;referer: (.*))\" }\n    }\t\n    grok {\n      match =&gt; { \"ref\" =&gt; \"(?&lt;referer&gt; (.*))\" }\n    }\n    mutate {\n      # Remove unnecessary characters from the fields.\n      gsub =&gt; [\n        \"alert_msg\", \"[\\\"]\", \"\",\n        \"user_agent\", \"[:\\\"'`]\", \"\",\n        \"user_agent\", \"^\\s*\", \"\",\n        \"referer\", \"^\\s*\", \"\"\n      ]\n      # Remove the Unnecessary fields so we can only remain with\n      # General message, rules_file, attack_type, rule_id, alert_msg, user_agent, hostname (being attacked), Request URI and Referer. \n      remove_field =&gt; [ \"alert_message\", \"rulesfile\", \"ruleid\", \"msg\", \"scanner\", \"agent\", \"ref\" ]\n    }\t\n}\noutput {\n   elasticsearch {\n     hosts =&gt [\"https:\/\/${ES_NAME}:9200\"]\n     user =&gt \"${ELASTICSEARCH_USERNAME}\"\n     password =&gt \"${ELASTICSEARCH_PASSWORD}\"\n     ssl =&gt true\n     cacert =&gt \"config\/certs\/ca\/ca.crt\"\n   }\n}\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"verify-docker-compose-file-syntax\">Verify Docker Compose File Syntax<\/h4>\n\n\n\n<p>Check Docker Compose file Syntax;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker-compose -f docker-compose.yml config<\/pre>\n\n\n\n<p>If there is any error, it will be printed. Otherwise, the Docker compose file contents are printed to standard output.<\/p>\n\n\n\n<p>If you are in the same directory where <code>docker-compose.yml<\/code> file is located, simply run;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker-compose config<\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"deploy-elk-stack-8-using-docker-compose-file\">Deploy ELK Stack 8 Using Docker Compose file<\/h4>\n\n\n\n<p>Everything is now setup and we are ready to build and start our Elastic Stack instances using the <code>docker-compose up<\/code> command.<\/p>\n\n\n\n<p>Navigate to the main directory where the Docker compose file is located. In my setup the directory is <code><strong>$HOME\/elastic-docker<\/strong><\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">cd <strong>$HOME\/elastic-docker<\/strong><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">docker-compose up<\/pre>\n\n\n\n<blockquote class=\"wp-block-quote has-small-font-size is-layout-flow wp-block-quote-is-layout-flow\" style=\"font-style:italic;font-weight:600\">\n<p>The command creates and starts the containers in foreground.<\/p>\n<\/blockquote>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>...\nelasticsearch  | {\"@timestamp\":\"2024-01-16T19:02:40.690Z\", \"log.level\": \"INFO\",  \"current.health\":\"GREEN\",\"message\":\"Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.security-7][0]]]).\",\"previous.health\":\"YELLOW\",\"reason\":\"shards started [[.security-7][0]]\" , \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es01][masterService#updateTask][T#1]\",\"log.logger\":\"org.elasticsearch.cluster.routing.allocation.AllocationService\",\"elasticsearch.cluster.uuid\":\"lhhCqqyfRrOuJ5lIHuOCww\",\"elasticsearch.node.id\":\"Q9-_fRo7Q6awepd8PYGvnQ\",\"elasticsearch.node.name\":\"es01\",\"elasticsearch.cluster.name\":\"elk-docker-cluster\"}\n...\n...\nlogstash       | [2024-01-16T19:03:03,172][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>\"main\", \"pipeline.workers\"=>4, \"pipeline.batch.size\"=>125, \"pipeline.batch.delay\"=>50, \"pipeline.max_inflight\"=>500, \"pipeline.sources\"=>[\"\/usr\/share\/logstash\/pipeline\/modsec.conf\"], :thread=>\"#<Thread:0x7d23f067 \/usr\/share\/logstash\/logstash-core\/lib\/logstash\/java_pipeline.rb:134 run>\"}\nlogstash       | [2024-01-16T19:03:03,927][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {\"seconds\"=>0.75}\nlogstash       | [2024-01-16T19:03:03,939][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>\"0.0.0.0:5044\"}\nlogstash       | [2024-01-16T19:03:03,944][INFO ][logstash.javapipeline    ][main] Pipeline started {\"pipeline.id\"=>\"main\"}\nlogstash       | [2024-01-16T19:03:03,962][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}\nlogstash       | [2024-01-16T19:03:04,023][INFO ][org.logstash.beats.Server][main][5c2281c3c7dc3fa0cecb74e0eb418d31f5ca88d19e9d33bf9ac5902cf7ffec49] Starting server on port: 5044\n...\n...\n{\"log\":\"[2024-01-16T19:03:08.805+00:00][INFO ][plugins.alerting] Installing ILM policy .alerts-ilm-policy\\n\",\"stream\":\"stdout\",\"time\":\"2024-01-16T19:03:08.80584097Z\"}\n{\"log\":\"[2024-01-16T19:03:08.807+00:00][INFO ][plugins.alerting] Installing component template .alerts-framework-mappings\\n\",\"stream\":\"stdout\",\"time\":\"2024-01-16T19:03:08.807954546Z\"}\n{\"log\":\"[2024-01-16T19:03:08.809+00:00][INFO ][plugins.alerting] Installing component template .alerts-legacy-alert-mappings\\n\",\"stream\":\"stdout\",\"time\":\"2024-01-16T19:03:08.809793742Z\"}\n{\"log\":\"[2024-01-16T19:03:08.826+00:00][INFO ][plugins.alerting] Installing component template .alerts-ecs-mappings\\n\",\"stream\":\"stdout\",\"time\":\"2024-01-16T19:03:08.827201234Z\"}\n{\"log\":\"[2024-01-16T19:03:08.839+00:00][INFO ][plugins.ruleRegistry] Installing component template .alerts-technical-mappings\\n\",\"stream\":\"stdout\",\"time\":\"2024-01-16T19:03:08.840094086Z\"}\n{\"log\":\"[2024-01-16T19:03:10.330+00:00][INFO ][http.server.Kibana] http server running at http:\/\/0.0.0.0:5601\\n\",\"stream\":\"stdout\",\"time\":\"2024-01-16T19:03:10.330814866Z\"}\n...\n<\/code><\/pre>\n\n\n\n<p>When you stop the <code>docker-compose up<\/code> command, all containers are stopped.<\/p>\n\n\n\n<p>From another console, you can check running containers. Note that you can use <code>docker-compose<\/code> command as you would <code>docker<\/code> command. However, to use it, you need to be in the same directory as compose file or specify path using the <code>-f<\/code> option..<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker-compose ps<\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code>NAME            IMAGE                                                  COMMAND                  SERVICE         CREATED          STATUS                    PORTS\nelasticsearch   docker.elastic.co\/elasticsearch\/elasticsearch:8.11.4   \"\/bin\/tini -- \/usr\/l\u2026\"   elasticsearch   21 minutes ago   Up 17 minutes (healthy)   0.0.0.0:9200->9200\/tcp, :::9200->9200\/tcp, 0.0.0.0:9300->9300\/tcp, :::9300->9300\/tcp\nkibana          docker.elastic.co\/kibana\/kibana:8.11.4                 \"\/bin\/tini -- \/usr\/l\u2026\"   kibana          21 minutes ago   Up 16 minutes             0.0.0.0:5601->5601\/tcp, :::5601->5601\/tcp\nlogstash        docker.elastic.co\/logstash\/logstash:8.11.4             \"\/usr\/local\/bin\/dock\u2026\"   logstash        21 minutes ago   Up 16 minutes             0.0.0.0:5044->5044\/tcp, :::5044->5044\/tcp, 9600\/tcp\n<\/code><\/pre>\n\n\n\n<p>From the output, you can see that the containers are running and their ports exposed on the host (any IP address) to allow external access.<\/p>\n\n\n\n<p>You can run the stack containers in background using the <code><strong>-d<\/strong><\/code> option. You can press <code>ctrl+c<\/code> to cancel the command and stop the containers.<\/p>\n\n\n\n<p>To relaunch containers in background<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker-compose up -d<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>[+] Running 4\/4\n[+] Running 4\/4kstack-docker-es_setup-1  Healthy                                                                                                                                                0.0s \n \u2714 Container elkstack-docker-es_setup-1  Healthy                                                                                                                                                0.0s \n \u2714 Container elasticsearch               Healthy                                                                                                                                                0.0s \n \u2714 Container logstash                    Started                                                                                                                                                0.0s \n \u2714 Container kibana                      Started\n<\/code><\/pre>\n\n\n\n<p>You can as well list the running containers using docker command;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker ps<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>CONTAINER ID   IMAGE                                                  COMMAND                  CREATED          STATUS                    PORTS                                                                                  NAMES\n40516edb2827   docker.elastic.co\/logstash\/logstash:8.11.4             \"\/usr\/local\/bin\/dock\u2026\"   24 minutes ago   Up 22 seconds             0.0.0.0:5044->5044\/tcp, :::5044->5044\/tcp, 9600\/tcp                                    logstash\n2efaeccc67a3   docker.elastic.co\/kibana\/kibana:8.11.4                 \"\/bin\/tini -- \/usr\/l\u2026\"   24 minutes ago   Up 22 seconds             0.0.0.0:5601->5601\/tcp, :::5601->5601\/tcp                                              kibana\na3f453974592   docker.elastic.co\/elasticsearch\/elasticsearch:8.11.4   \"\/bin\/tini -- \/usr\/l\u2026\"   24 minutes ago   Up 53 seconds (healthy)   0.0.0.0:9200->9200\/tcp, :::9200->9200\/tcp, 0.0.0.0:9300->9300\/tcp, :::9300->9300\/tcp   elasticsearch\n<\/code><\/pre>\n\n\n\n<p>To find the details of each container, use <code><em><strong>docker inspect &lt;container-name&gt;<\/strong><\/em><\/code> command. For example<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker inspect elasticsearch<\/pre>\n\n\n\n<p>To get the logs of a container, use the command <em><strong><code>docker logs [OPTIONS] CONTAINER<\/code><\/strong><\/em>. For example, to get Elasticsearch container logs;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker logs elasticsearch<\/pre>\n\n\n\n<p>If you need to check specific number of logs, you can use the <code><strong>tail<\/strong><\/code> option. E.g to get the last 50 log lines;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker logs --tail 50 -f elasticsearch<\/pre>\n\n\n\n<p>Or check the <strong><code>\/var\/lib\/docker\/containers\/&lt;long-docker-id&gt;\/&lt;long-docker-id&gt;.log<\/code><\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"accessing-kibana-container-from-browser\">Accessing Kibana Container from Browser<\/h2>\n\n\n\n<p>Once the stack is up and running, you can access Kibana externally using the host IP address and the port on which it is exposed on. In our setup, Kibana container port 5601 is exposed on the same port on the host;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker port kibana<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">5601\/tcp -&gt; 0.0.0.0:5601\n5601\/tcp -&gt; [::]:5601<\/pre>\n\n\n\n<p>This means that you can access Kibana container port on via any interface on the host, port 5601. Similarly, you can check container port exposure using the command above.<\/p>\n\n\n\n<p>Therefore, you can access Kibana using your Container host address, <strong>http:\/\/&lt;IP-Address&gt;:5601<\/strong>.<\/p>\n\n\n\n<p>Login using the Elastic user. You can add other accounts thereafter.<\/p>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2024\\\/01\\\/kibana-8-docker-container-login-page.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-19904&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1234,&quot;targetHeight&quot;:644,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image: Deploy ELK Stack 8 on Docker Containers&quot;,&quot;alt&quot;:&quot;Deploy ELK Stack 8 on Docker Containers&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1234\" height=\"644\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/kibana-8-docker-container-login-page.png?v=1705438148\" alt=\"Deploy ELK Stack 8 on Docker Containers\" class=\"wp-image-19904\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/kibana-8-docker-container-login-page.png?v=1705438148 1234w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/kibana-8-docker-container-login-page-768x401.png?v=1705438148 768w\" sizes=\"(max-width: 1234px) 100vw, 1234px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image: Deploy ELK Stack 8 on Docker Containers\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<p>And there you go!<\/p>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2024\\\/01\\\/kibana-8-dashboard.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-19905&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1619,&quot;targetHeight&quot;:790,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image&quot;,&quot;alt&quot;:&quot;&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1619\" height=\"790\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/kibana-8-dashboard.png?v=1705438164\" alt=\"\" class=\"wp-image-19905\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/kibana-8-dashboard.png?v=1705438164 1619w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/kibana-8-dashboard-768x375.png?v=1705438164 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/kibana-8-dashboard-1536x749.png?v=1705438164 1536w\" sizes=\"(max-width: 1619px) 100vw, 1619px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"sending-data-logs-to-elastic-stack\">Sending Data Logs to Elastic Stack<\/h2>\n\n\n\n<p>Since we configured our Logstash receive event data from the Beats, we will configure Filebeat to forward events.<\/p>\n\n\n\n<p>We already covered how to install and configure Filebeat to forward event data in our previous guides;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-configure-filebeat-on-centos-8\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install and Configure Filebeat on CentOS 8<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-filebeat-on-fedora-30-fedora-29-centos-7\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Filebeat on Fedora 30\/Fedora 29\/CentOS 7<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-configure-filebeat-7-on-ubuntu-18-04-debian-9-8\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install and Configure Filebeat 7 on Ubuntu 18.04\/Debian 9.8<\/a><\/p>\n\n\n\n<p>Once you forward data to your Logstash container, the next thing you need to do is create Kibana index.<\/p>\n\n\n\n<p>Open the menu, then go to <strong>Stack Management<\/strong> &gt;&nbsp;<strong>Kibana<\/strong>&nbsp;&gt;&nbsp;<strong>Index<\/strong>&nbsp;Patterns.<\/p>\n\n\n\n<p>Once done, heading to Discover menu to view your data. You should now be able to see your Logstash custom fields populated.<\/p>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2024\\\/01\\\/sample-logs-logstash-8-filebeat-8.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-19903&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1620,&quot;targetHeight&quot;:800,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image&quot;,&quot;alt&quot;:&quot;&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1620\" height=\"800\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/sample-logs-logstash-8-filebeat-8.png?v=1705438115\" alt=\"\" class=\"wp-image-19903\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/sample-logs-logstash-8-filebeat-8.png?v=1705438115 1620w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/sample-logs-logstash-8-filebeat-8-768x379.png?v=1705438115 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/01\/sample-logs-logstash-8-filebeat-8-1536x759.png?v=1705438115 1536w\" sizes=\"(max-width: 1620px) 100vw, 1620px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<p>That marks the end of our tutorial on how to deploy a single node Elastic Stack cluster on Docker Containers.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"other-tutorials\">Other Tutorials<\/h2>\n\n\n\n<p><a href=\"https:\/\/www.elastic.co\/guide\/en\/elastic-stack-get-started\/current\/get-started-docker.html\" target=\"_blank\" rel=\"noreferrer noopener\"><\/a><a href=\"https:\/\/kifarunix.com\/configure-kibana-dashboards-visualizations-to-use-custom-index\/\">Configure Kibana Dashboards\/Visualizations to use Custom Index<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>How to run single node ELK stack 8 on Docker? In this tutorial, you will learn how to deploy ELK stack 8 on Docker containers.<\/p>\n","protected":false},"author":10,"featured_media":19906,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[72,910,121],"tags":[7362,7360,7359,7361],"class_list":["post-19888","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-monitoring","category-elastic-stack","category-howtos","tag-dockerize-elk-8","tag-elasticsearch-8-docker","tag-elk-8-docker","tag-elk-stack-8-in-docker","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19888"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=19888"}],"version-history":[{"count":15,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19888\/revisions"}],"predecessor-version":[{"id":20909,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19888\/revisions\/20909"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/19906"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=19888"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=19888"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=19888"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}