{"id":6658,"date":"2020-08-11T00:01:29","date_gmt":"2020-08-10T21:01:29","guid":{"rendered":"https:\/\/kifarunix.com\/?p=6658"},"modified":"2024-03-14T22:39:07","modified_gmt":"2024-03-14T19:39:07","slug":"deploy-a-single-node-elk-stack-cluster-on-docker-containers","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/deploy-a-single-node-elk-stack-cluster-on-docker-containers\/","title":{"rendered":"Deploy a Single Node ELK Stack Cluster on Docker Containers"},"content":{"rendered":"\n<p>How to run ELK stack on Docker? In this tutorial, we are going to learn how to deploy a single node <a href=\"https:\/\/www.elastic.co\/elastic-stack\" target=\"_blank\" rel=\"noreferrer noopener\">ELK stack<\/a> cluster on Docker containers. Elastic 7.17.16 is the latest release on Elastic 7.x releases. ELK stack is a group of open source software projects: Elasticsearch, Logstash, and Kibana and Beats, where:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Elasticsearch is a search and analytics engine<\/li>\n\n\n\n<li>Logstash is a server\u2011side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a &#8220;stash&#8221; like Elasticsearch<\/li>\n\n\n\n<li>Kibana lets users visualize data with charts and graphs in Elasticsearch and <\/li>\n\n\n\n<li><a href=\"https:\/\/www.elastic.co\/beats\/\" target=\"_blank\" rel=\"noreferrer noopener\">Beats<\/a> are the data shippers. They ship system logs, network, infrastructure data, etc to either Logstash for further processing or Elasticsearch for indexing.<\/li>\n<\/ul>\n\n\n\n<p>We will be running Elasticsearch 7.17.16, Logstash 7.17.16, Kibana 7.17.16, as Docker containers in this tutorial.<\/p>\n\n\n\n<p>Kindly note that Elastic 7.x doesn&#8217;t support authentication nor TLS\/SSL connection by default. If you want, you can use ELK stack 8.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/deploy-elk-stack-8-on-docker-containers\/\" target=\"_blank\" rel=\"noreferrer noopener\">Deploy ELK Stack 8 on Docker Containers<\/a><\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#deploying-single-node-elk-stack-cluster-on-docker-containers\">Deploying Single Node ELK Stack Cluster on Docker Containers<\/a><ul><li><a href=\"#install-docker-engine\">Install Docker Engine<\/a><\/li><li><a href=\"#install-docker-compose\">Install Docker Compose<\/a><\/li><li><a href=\"#running-docker-as-non-root-user\">Running Docker as Non-Root User<\/a><\/li><li><a href=\"#important-elasticsearch-settings\">Important Elasticsearch Settings<\/a><ul><li><a href=\"#disable-memory-swapping-on-a-container\">Disable Memory Swapping on a Container<\/a><\/li><li><a href=\"#set-jvm-heap-size-on-all-cluster-nodes\">Set JVM Heap Size on All Cluster Nodes<\/a><\/li><li><a href=\"#set-maximum-open-file-descriptor-and-processes-on-elasticsearch-containers\">Set Maximum Open File Descriptor and Processes on Elasticsearch Containers<\/a><\/li><li><a href=\"#update-virtual-memory-settings-on-all-cluster-nodes\">Update Virtual Memory Settings on All Cluster Nodes<\/a><\/li><\/ul><\/li><li><a href=\"#deploying-elk-stack-cluster-on-docker-using-docker-compose\">Deploying ELK Stack Cluster on Docker Using Docker Compose<\/a><ul><li><a href=\"#create-docker-compose-file-for-deploying-elastic-stack\">Create Docker Compose file for Deploying Elastic Stack<\/a><\/li><\/ul><\/li><li><a href=\"#setup-docker-compose-file-for-elastic-stack\">Setup Docker Compose file for Elastic Stack<\/a><ul><li><a href=\"#define-logstash-data-processing-pipeline\">Define Logstash Data Processing Pipeline<\/a><\/li><\/ul><\/li><li><a href=\"#deploy-elastic-stack-using-docker-compose-file\">Deploy Elastic Stack Using Docker Compose file<\/a><\/li><\/ul><\/li><li><a href=\"#accessing-kibana-container-from-browser\">Accessing Kibana Container from Browser<\/a><ul><li><a href=\"#sending-data-to-elastic-stack\">Sending data to Elastic Stack<\/a><\/li><li><a href=\"#reference\">Reference<\/a><\/li><li><a href=\"#other-related-tutorials\">Other Related Tutorials<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"deploying-single-node-elk-stack-cluster-on-docker-containers\">Deploying Single Node ELK Stack Cluster on Docker Containers<\/h2>\n\n\n\n<p>In this tutorial, therefore, we will learn how to deploy ELK Stack cluster on Docker using Docker and Docker compose.<\/p>\n\n\n\n<p><a href=\"https:\/\/docs.docker.com\/get-started\/\" target=\"_blank\" rel=\"noreferrer noopener\">Docker<\/a>&nbsp;is a platform that enables developers and system administrators to&nbsp;<strong>build, run, and share<\/strong>&nbsp;applications with containers. It provides command line interface tools such as <code>docker<\/code> and <code>docker-compose<\/code> that are used for managing Docker containers. While docker is a Docker cli for managing single Docker containers, docker-compose on the other hand is used for running and managing multiple Docker containers.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-docker-engine\">Install Docker Engine<\/h3>\n\n\n\n<p>Depending on the your host system distribution, you need to install the Docker engine. You can follow the links below to install Docker Engine on Ubuntu\/Debian\/CentOS 8.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/?s=install+docker+ce\" target=\"_blank\" rel=\"noreferrer noopener\">Install and Use Docker on Linux<\/a><\/p>\n\n\n\n<p>Checking Installed Docker version;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker version<\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Docker version 24.0.7, build afdd53b<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-docker-compose\"><a href=\"#install-docker-compose\">Install Docker Compose<\/a><\/h3>\n\n\n\n<p>For Docker compose to work, ensure that you have Docker Engine installed. You can follow the links above to install Docker Engine.<\/p>\n\n\n\n<p>Once you have the Docker engine installed, proceed to install Docker compose.<\/p>\n\n\n\n<p>Check the current stable release version of Docker Compose on their&nbsp;<a href=\"https:\/\/github.com\/docker\/compose\/releases\" target=\"_blank\" rel=\"noreferrer noopener\">Github release page<\/a>. As of this writing, the Docker Compose version 2.24.0 is the current stable release.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>VER=2.24.0<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo curl -L \"https:\/\/github.com\/docker\/compose\/releases\/download\/v$VER\/docker-compose-$(uname -s)-$(uname -m)\" -o \/usr\/local\/bin\/docker-compose<\/code><\/pre>\n\n\n\n<p>This downloads docker compose tool to&nbsp;<code>\/usr\/local\/bin<\/code>&nbsp;directory.<\/p>\n\n\n\n<p>Make the Docker compose binary executable;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>chmod +x \/usr\/local\/bin\/docker-compose<\/code><\/pre>\n\n\n\n<p>You should now be able to use Docker compose (<code><strong>docker-compose<\/strong><\/code>) on the CLI.<\/p>\n\n\n\n<p>Check the version of installed Docker compose to confirm that it is working as expected.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker-compose version<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Docker Compose version v2.24.0<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"running-docker-as-non-root-user\">Running Docker as Non-Root User<\/h3>\n\n\n\n<p>We are running both Docker and Docker compose as non root user. To be able to do this, ensure that your add your standard user to the <code><strong>docker<\/strong><\/code> group.<\/p>\n\n\n\n<p>For example, am running this setup as user <code>kifarunix<\/code>. So, add the user to Docker group.  Replace the username accordingly.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo usermod -aG docker kifarunix<\/pre>\n\n\n\n<p>Log out and log in again as the user that is added to the docker group and you should be able to run the docker and docker-compose CLI tools.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"important-elasticsearch-settings\">Important Elasticsearch Settings<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"disable-memory-swapping-on-a-container\">Disable Memory Swapping on a Container<\/h4>\n\n\n\n<p>There are different ways to disable swapping on Elasticsearch. For an Elasticsearch Docker container, you can simply set the value of the <strong><code>bootstrap.memory_lock<\/code><\/strong> option to <strong><code>true<\/code><\/strong>. With this option, you also need to ensure that there is no maximum limit imposed on the locked-in-memory address space (<strong><code>LimitMEMLOCK=infinity<\/code><\/strong>). We will see how this can be achieved in a Docker compose file.<\/p>\n\n\n\n<p>You can confirm the default value set on the Docker image using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker run --rm docker.elastic.co\/elasticsearch\/elasticsearch:7.17.16 \/bin\/bash -c 'ulimit -Hm &amp;&amp; ulimit -Sm'<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>unlimited\nunlimited\n<\/code><\/pre>\n\n\n\n<p>If the value, other than unlimited is printed, then you have to set this via the Docker compose file to ensure that it set to unlimited for both Soft and Hard limits.<\/p>\n\n\n\n<p>For a respective service, this is how this config looks like in Docker compose file.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>    ulimits:\n      memlock:\n        soft: -1\n        hard: -1\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-jvm-heap-size-on-all-cluster-nodes\">Set JVM Heap Size on All Cluster Nodes<\/h4>\n\n\n\n<p>Elasticsearch usually sets the heap size automatically based on the roles of the nodes\u2019s&nbsp;roles&nbsp;and the total memory available to the node\u2019s container. This is the recommended approach! For the Elasticsearch Docker containers, any custom JVM settings can be defined using the <strong><code>ES_JAVA_OPTS<\/code><\/strong> environment variable, either in compose file or on command line. For example, you can set JVM heap size, Min and Max to 512M using the <strong><code>ES_JAVA_OPTS=-Xms512m -Xmx512m<\/code><\/strong>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-maximum-open-file-descriptor-and-processes-on-elasticsearch-containers\">Set Maximum Open File Descriptor and Processes on Elasticsearch Containers<\/h4>\n\n\n\n<p>Set the maximum number of open files (<code>nofile<\/code>) for the&nbsp;containers to 65,536 and max number of processes to 4096 (both soft and hard limits).<\/p>\n\n\n\n<p>We will also define these limits in the compose file. However, the Elasticsearch images usually comes with these limits already defined. You can use the command below to check the maximum number of user processes (-u) and max number of opens files (-n);<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker run --rm docker.elastic.co\/elasticsearch\/elasticsearch:7.17.16 \/bin\/bash -c 'ulimit -Hn &amp;&amp; ulimit -Sn &amp;&amp; ulimit -Hu &amp;&amp; ulimit -Su'<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>1048576\n1048576\nunlimited\nunlimited\n<\/code><\/pre>\n\n\n\n<p>These values may look like this on Docker compose file. Adjust them accordingly!<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>    ulimits:\n      nofile:\n        soft: 65536\n        hard: 65536\n      nproc:\n        soft: 2048\n        hard: 2048\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"update-virtual-memory-settings-on-all-cluster-nodes\">Update Virtual Memory Settings on All Cluster Nodes<\/h4>\n\n\n\n<p>Elasticsearch uses a&nbsp;<a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/index-modules-store.html#mmapfs\" target=\"_blank\" rel=\"noreferrer noopener\"><code>mmapfs<\/code><\/a>&nbsp;directory by default to store its indices. To ensure that you do not run out of virtual memory, edit the&nbsp;<strong>\/etc\/sysctl.conf<\/strong>&nbsp;on the <strong><code>Docker host<\/code><\/strong> and update the value of&nbsp;<strong>vm.max_map_count<\/strong>&nbsp;as shown below.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vm.max_map_count=262144<\/code><\/pre>\n\n\n\n<p>On the Docker host, simply run the command below to configure virtual memory settings.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"vm.max_map_count=262144\" &gt;&gt; \/etc\/sysctl.conf<\/code><\/pre>\n\n\n\n<p>To apply the changes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sysctl -p<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploying-elk-stack-cluster-on-docker-using-docker-compose\"><a href=\"#single-node-elk-docker-compose\">Deploying ELK Stack Cluster on Docker Using Docker Compose<\/a><\/h3>\n\n\n\n<p>In this setup, we will deploy a single node Elastic Stack cluster with all the three components, Elasticsearch, Logstash and Kibana containers running on the same host as Docker containers.<\/p>\n\n\n\n<p>If you are running Elastic stack on Docker containers for development or testing purposes, then you can run each component Docker container using <strong><code>docker run<\/code><\/strong> commands.<\/p>\n\n\n\n<p>To begin, create a parent directory from where you will build your stack from.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">mkdir $HOME\/elastic-docker<\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-docker-compose-file-for-deploying-elastic-stack\">Create Docker Compose file for Deploying Elastic Stack<\/h4>\n\n\n\n<p><em>Docker compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application\u2019s services. Then, with a single command, you create and start all the services from your configuratio<\/em>n.<\/p>\n\n\n\n<p><em>Using Compose is basically a three-step process:<\/em><\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><em>Define your app\u2019s environment with a&nbsp;<code>Dockerfile<\/code>&nbsp;so it can be reproduced anywhere.<\/em><\/li>\n\n\n\n<li><em>Define the services that make up your app in&nbsp;<code>docker-compose.yml<\/code>&nbsp;so they can be run together in an isolated environment.<\/em><\/li>\n\n\n\n<li><em>Run&nbsp;<code>docker-compose up<\/code>&nbsp;and Compose starts and runs your entire app<\/em><\/li>\n<\/ol>\n\n\n\n<p>In this setup, we will build everything using a Docker Compose file.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"setup-docker-compose-file-for-elastic-stack\">Setup Docker Compose file for Elastic Stack<\/h3>\n\n\n\n<p>Now it is time to create the Docker Compose file for our deployment. Note, we will be using Docker images of the current stable release versions of the Elastic components, v7.17.16.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">vim $HOME\/elastic-docker\/docker-compose.yml<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>version: '3.8'\nservices:\n  elasticsearch:\n    container_name: kifarunix-demo-es\n    image: docker.elastic.co\/elasticsearch\/elasticsearch:7.17.16\n    environment:\n      - node.name=kifarunix-demo-es\n      - cluster.name=es-docker-cluster\n      - discovery.type=single-node\n      - bootstrap.memory_lock=true\n      - \"ES_JAVA_OPTS=-Xms1g -Xmx1g\"\n    mem_limit: 2147483648\n    ulimits:\n      memlock:\n        soft: -1\n        hard: -1\n    volumes:\n      - es-data:\/usr\/share\/elasticsearch\/data\n    ports:\n      - 9200:9200\n    networks:\n      - elastic\n\n  kibana:\n    depends_on:\n      - elasticsearch\n    image: docker.elastic.co\/kibana\/kibana:7.17.16\n    container_name: kifarunix-demo-kibana\n    environment:\n      ELASTICSEARCH_URL: http:\/\/kifarunix-demo-es:9200\n      ELASTICSEARCH_HOSTS: http:\/\/kifarunix-demo-es:9200\n    mem_limit: 1073741824\n    ports:\n      - 5601:5601\n    networks:\n      - elastic\n  \n  logstash:\n    depends_on:\n      - elasticsearch\n    image: docker.elastic.co\/logstash\/logstash:7.17.16\n    container_name: kifarunix-demo-ls\n    ports:\n      - \"5044:5044\"\n    mem_limit: 2147483648\n    volumes:\n      - .\/logstash\/conf.d\/:\/usr\/share\/logstash\/pipeline\/:ro\n    networks:\n      - elastic\n\nvolumes:\n  es-data:\n    driver: local\n\nnetworks:\n  elastic:\n    driver: bridge\n<\/code><\/pre>\n\n\n\n<p>For a complete description of all the Docker compose configuration options, refer to <a href=\"https:\/\/docs.docker.com\/compose\/compose-file\/\" target=\"_blank\" rel=\"noreferrer noopener\">Docker compose reference page<\/a>.<\/p>\n\n\n\n<p>You can define all the values that are dynamic using environment variables in a <a href=\"https:\/\/docs.docker.com\/compose\/environment-variables\/set-environment-variables\/#substitute-with-an-env-file\" target=\"_blank\" rel=\"noreferrer noopener\"><strong><code>.env<\/code><\/strong> file<\/a>. This environment variables file  should reside in the same directory are compose file.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"define-logstash-data-processing-pipeline\">Define Logstash Data Processing Pipeline<\/h4>\n\n\n\n<p>In this setup, we will configure Logstash to receive event data from Beats (Filebeat to be specific) for further processing and stashing onto the search analytics engine, Elasticsearch.<\/p>\n\n\n\n<p>Note that Logstash is only necessary if you need to apply further processing to your event data. For example, extracting custom fields from the event data, mutating the event data etc. Otherwise, you can push the data directly to Elasticsearch from Beats.<\/p>\n\n\n\n<p>In this setup, we will use a sample Logstash processing pipeline for ModSecurity audit logs;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">mkdir -p $HOME\/elastic-docker\/logstash\/conf.d<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">vim $HOME\/elastic-docker\/logstash\/conf.d\/modsec.conf<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\ninput {\n  beats {\n    port =&gt; 5044\n  }\n}\nfilter {\n    # Extract event time, log severity level, source of attack (client), and the alert message.\n    grok {\n      match =&gt; { \"message\" =&gt; \"(?&lt;event_time&gt;%{MONTH}\\s%{MONTHDAY}\\s%{TIME}\\s%{YEAR})\\] \\[\\:%{LOGLEVEL:log_level}.*client\\s%{IPORHOST:src_ip}:\\d+]\\s(?&lt;alert_message&gt;.*)\" }\n    }\n    # Extract Rules File from Alert Message\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(?&lt;rulesfile&gt;\\[file \\\"(\/.+.conf)\\\"\\])\" }\n    }\t\n    grok {\n      match =&gt; { \"rulesfile\" =&gt; \"(?&lt;rules_file&gt;\/.+.conf)\" }\n    }\t\n    # Extract Attack Type from Rules File\n    grok {\n      match =&gt; { \"rulesfile\" =&gt; \"(?&lt;attack_type&gt;[A-Z]+-[A-Z][^.]+)\" }\n    }\t\n    # Extract Rule ID from Alert Message\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(?&lt;ruleid&gt;\\[id \\\"(\\d+)\\\"\\])\" }\n    }\t\n    grok {\n      match =&gt; { \"ruleid\" =&gt; \"(?&lt;rule_id&gt;\\d+)\" }\n    }\n    # Extract Attack Message (msg) from Alert Message \t\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(?&lt;msg&gt;\\[msg \\S(.*?)\\\"\\])\" }\n    }\t\n    grok {\n      match =&gt; { \"msg\" =&gt; \"(?&lt;alert_msg&gt;\\\"(.*?)\\\")\" }\n    }\n    # Extract the User\/Scanner Agent from Alert Message\t\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(?&lt;scanner&gt;User-Agent' \\SValue: `(.*?)')\" }\n    }\t\n    grok {\n      match =&gt; { \"scanner\" =&gt; \"(?&lt;user_agent&gt;:(.*?)\\')\" }\n    }\t\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(?&lt;agent&gt;User-Agent: (.*?)\\')\" }\n    }\t\n    grok {\n      match =&gt; { \"agent\" =&gt; \"(?&lt;user_agent&gt;: (.*?)\\')\" }\n    }\t\n    # Extract the Target Host\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(hostname \\\"%{IPORHOST:dst_host})\" }\n    }\t\n    # Extract the Request URI\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(uri \\\"%{URIPATH:request_uri})\" }\n    }\n    grok {\n      match =&gt; { \"alert_message\" =&gt; \"(?&lt;ref&gt;referer: (.*))\" }\n    }\t\n    grok {\n      match =&gt; { \"ref\" =&gt; \"(?&lt;referer&gt; (.*))\" }\n    }\n    mutate {\n      # Remove unnecessary characters from the fields.\n      gsub =&gt; [\n        \"alert_msg\", \"[\\\"]\", \"\",\n        \"user_agent\", \"[:\\\"'`]\", \"\",\n        \"user_agent\", \"^\\s*\", \"\",\n        \"referer\", \"^\\s*\", \"\"\n      ]\n      # Remove the Unnecessary fields so we can only remain with\n      # General message, rules_file, attack_type, rule_id, alert_msg, user_agent, hostname (being attacked), Request URI and Referer. \n      remove_field =&gt; [ \"alert_message\", \"rulesfile\", \"ruleid\", \"msg\", \"scanner\", \"agent\", \"ref\" ]\n    }\t\n}\noutput {\n   elasticsearch {\n     hosts =&gt; [\"kifarunix-demo-es:9200\"]\n   }\n}\n<\/code><\/pre>\n\n\n\n<p>Check Docker Compose file Syntax;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker-compose -f docker-compose.yml config<\/pre>\n\n\n\n<p>If there is any error, it will be printed. Otherwise, the Docker compose file contents are printed to standard output.<\/p>\n\n\n\n<p>If you are in the same directory where <code>docker-compose.yml<\/code> file is located, simply run;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker-compose config<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploy-elastic-stack-using-docker-compose-file\">Deploy Elastic Stack Using Docker Compose file<\/h3>\n\n\n\n<p>Everything is now setup and we are ready to build and start our Elastic Stack instances using the <code>docker-compose up<\/code> command.<\/p>\n\n\n\n<p>Navigate to the main directory where the Docker compose file is located. In my setup the directory is <code><strong>$HOME\/elastic-docker<\/strong><\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">cd <strong>$HOME\/elastic-docker<\/strong><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">docker-compose up<\/pre>\n\n\n\n<p>The command creates and starts the containers in foreground.<\/p>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>...\nkifarunix-demo-es | {\"type\": \"server\", \"timestamp\": \"2024-01-15T19:35:00,516Z\", \"level\": \"INFO\", \"component\": \"o.e.c.m.MetadataIndexTemplateService\", \"cluster.name\": \"es-docker-cluster\", \"node.name\": \"kifarunix-demo-es\", \"message\": \"adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]\", \"cluster.uuid\": \"DBy4Mwk-TB2Jum_AWDY0jw\", \"node.id\": \"4Fb4-CZ0QhG2KcZL79-8cw\"  }\nkifarunix-demo-ls | [2024-01-15T19:35:00,590][INFO ][logstash.inputs.beats    ][main] Beats inputs: Starting input listener {:address=&gt;\"0.0.0.0:5044\"}\nkifarunix-demo-ls | [2024-01-15T19:35:00,606][INFO ][logstash.javapipeline    ][main] Pipeline started {\"pipeline.id\"=&gt;\"main\"}\nkifarunix-demo-ls | [2024-01-15T19:35:00,672][INFO ][logstash.agent           ] Pipelines running {:count=&gt;1, :running_pipelines=&gt;[:main], :non_running_pipelines=&gt;[]}\nkifarunix-demo-ls | [2024-01-15T19:35:00,730][INFO ][org.logstash.beats.Server][main][ee92a68a4dc1b148e25ac3c899680db31f95563138e922a364e18e3dc052d084] Starting server on port: 5044\nkifarunix-demo-ls | [2024-01-15T19:35:00,947][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=&gt;9600}\n...\n...\nkifarunix-demo-kibana | {\"type\":\"log\",\"@timestamp\":\"2024-01-15T19:36:49Z\",\"tags\":[\"status\",\"plugin:ui_metric@7.17.16\",\"info\"],\"pid\":8,\"state\":\"green\",\"message\":\"Status changed from uninitialized to green - Ready\",\"prevState\":\"uninitialized\",\"prevMsg\":\"uninitialized\"}\nkifarunix-demo-kibana | {\"type\":\"log\",\"@timestamp\":\"2024-01-15T19:36:49Z\",\"tags\":[\"listening\",\"info\"],\"pid\":8,\"message\":\"Server running at http:\/\/0:5601\"}\nkifarunix-demo-es | {\"type\": \"server\", \"timestamp\": \"2024-01-15T19:36:49,957Z\", \"level\": \"INFO\", \"component\": \"o.e.c.m.MetadataMappingService\", \"cluster.name\": \"es-docker-cluster\", \"node.name\": \"kifarunix-demo-es\", \"message\": \"[.kibana_task_manager_1\/a3B8lwzxQjiNHtFEAExOaQ] update_mapping [_doc]\", \"cluster.uuid\": \"DBy4Mwk-TB2Jum_AWDY0jw\", \"node.id\": \"4Fb4-CZ0QhG2KcZL79-8cw\"  }\nkifarunix-demo-kibana | {\"type\":\"log\",\"@timestamp\":\"2024-01-15T19:36:50Z\",\"tags\":[\"info\",\"http\",\"server\",\"Kibana\"],\"pid\":8,\"message\":\"http server running at http:\/\/0:5601\"}\n...\n<\/code><\/pre>\n\n\n\n<p>When you stop the <code>docker-compose up<\/code> command, all containers are stopped.<\/p>\n\n\n\n<p>From another console, you can check running containers. Note that you can user <code>docker-compose<\/code> command as you would <code>docker<\/code> command.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker-compose ps<\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code>        Name                       Command               State                Ports              \n-------------------------------------------------------------------------------------------------\nkifarunix-demo-es       \/tini -- \/usr\/local\/bin\/do ...   Up      0.0.0.0:9200-&gt;9200\/tcp, 9300\/tcp\nkifarunix-demo-kibana   \/usr\/local\/bin\/dumb-init - ...   Up      0.0.0.0:5601-&gt;5601\/tcp          \nkifarunix-demo-ls       \/usr\/local\/bin\/docker-entr ...   Up      0.0.0.0:5044-&gt;5044\/tcp, 9600\/tcp\n<\/code><\/pre>\n\n\n\n<p>From the output, you can see that the containers are running and their ports exposed on the host (any IP address) to allow external access.<\/p>\n\n\n\n<p>You can run the stack containers in background using the <code><strong>-d<\/strong><\/code> option. You can press <code>ctrl+c<\/code> to cancel the command and stop the containers.<\/p>\n\n\n\n<p>To relaunch containers in background<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker-compose up -d<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Starting kifarunix-demo-ls     ... done\nStarting kifarunix-demo-es     ... done\nStarting kifarunix-demo-kibana ... done\n<\/code><\/pre>\n\n\n\n<p>You can as well list the running containers using docker command;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker ps<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                              NAMES\n130eec8df661        docker.elastic.co\/kibana\/kibana:7.17.16                 \"\/usr\/local\/bin\/dumb\u2026\"   38 minutes ago      Up About a minute   0.0.0.0:5601-&gt;5601\/tcp             kifarunix-demo-kibana\n6648df61c44b        docker.elastic.co\/logstash\/logstash:7.17.16             \"\/usr\/local\/bin\/dock\u2026\"   41 minutes ago      Up About a minute   0.0.0.0:5044-&gt;5044\/tcp, 9600\/tcp   kifarunix-demo-ls\ndb9936abbee2        docker.elastic.co\/elasticsearch\/elasticsearch:7.17.16   \"\/tini -- \/usr\/local\u2026\"   41 minutes ago      Up About a minute   0.0.0.0:9200-&gt;9200\/tcp, 9300\/tcp   kifarunix-demo-es\n<\/code><\/pre>\n\n\n\n<p>To find the details of each container, use <code><em><strong>docker inspect &lt;container-name&gt;<\/strong><\/em><\/code> command. For example<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker inspect kifarunix-demo-es<\/pre>\n\n\n\n<p>To get the logs of a container, use the command <em><strong><code>docker logs [OPTIONS] CONTAINER<\/code><\/strong><\/em>. For example, to get Elasticsearch container logs;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker logs kifarunix-demo-es<\/pre>\n\n\n\n<p>If you need to check specific number of logs, you can use the <code><strong>tail<\/strong><\/code> option. E.g to get the last 50 log lines;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker logs --tail 50 -f kifarunix-demo-es<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"accessing-kibana-container-from-browser\">Accessing Kibana Container from Browser<\/h2>\n\n\n\n<p>Once the stack is up and running, you can access Kibana externally using the host IP address and the port on which it is exposed on. In our setup, Kibana container port 5601 is exposed on the same port on the host;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker port kifarunix-demo-kibana<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">5601\/tcp -&gt; 0.0.0.0:5601<\/pre>\n\n\n\n<p>This means that you can access Kibana container port on via any interface on the host, port 5601. Similarly, you can check container port exposure using the command above.<\/p>\n\n\n\n<p>Therefore, you can access Kibana using your Container host address, <strong>http:\/\/&lt;IP-Address&gt;:5601<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1699\" height=\"722\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/08\/kibana-docker.png\" alt=\"Deploy a Single Node Elastic Stack Cluster on Docker Containers\" class=\"wp-image-6667\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/08\/kibana-docker.png?v=1597092857 1699w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/08\/kibana-docker-768x326.png?v=1597092857 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/08\/kibana-docker-1536x653.png?v=1597092857 1536w\" sizes=\"(max-width: 1699px) 100vw, 1699px\" \/><\/figure>\n\n\n\n<p>As you can see, we do not have any data yet in our stack.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"sending-data-to-elastic-stack\">Sending data to Elastic Stack<\/h3>\n\n\n\n<p>Since we configured our Logstash receive event data from the Beats, we will configure Filebeat to forward events.<\/p>\n\n\n\n<p>We already covered how to install and configure Filebeat to forward event data in our previous guides;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-configure-filebeat-on-centos-8\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install and Configure Filebeat on CentOS 8<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-filebeat-on-fedora-30-fedora-29-centos-7\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Filebeat on Fedora 30\/Fedora 29\/CentOS 7<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-configure-filebeat-7-on-ubuntu-18-04-debian-9-8\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install and Configure Filebeat 7 on Ubuntu 18.04\/Debian 9.8<\/a><\/p>\n\n\n\n<p>Once you forward data to your Logstash container, the next thing you need to do is create Kibana index.<\/p>\n\n\n\n<p>Open the menu, then go to <strong>Stack Management<\/strong> &gt;&nbsp;<strong>Kibana<\/strong>&nbsp;&gt;&nbsp;<strong>Index<\/strong>&nbsp;Patterns.<\/p>\n\n\n\n<p>Once done, heading to Discover menu to view your data. You should now be able to see your Logstash custom fields populated.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1893\" height=\"852\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/08\/kibana-docker-data.png\" alt=\"\" class=\"wp-image-6666\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/08\/kibana-docker-data.png?v=1597092809 1893w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/08\/kibana-docker-data-768x346.png?v=1597092809 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/08\/kibana-docker-data-1536x691.png?v=1597092809 1536w\" sizes=\"(max-width: 1893px) 100vw, 1893px\" \/><\/figure>\n\n\n\n<p>That marks the end of our tutorial on how to deploy a single node Elastic Stack cluster on Docker Containers.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"reference\">Reference<\/h3>\n\n\n\n<p><a href=\"https:\/\/www.elastic.co\/guide\/en\/elastic-stack-get-started\/current\/get-started-docker.html\" target=\"_blank\" rel=\"noreferrer noopener\">Running Elastic Stack on Docker<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"other-related-tutorials\">Other Related Tutorials<\/h3>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/process-and-visualize-modsecurity-logs-on-elk-stack\/\" target=\"_blank\" rel=\"noreferrer noopener\">Process and Visualize ModSecurity Logs on ELK Stack<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/create-kibana-visualization-dashboards-for-modsecurity-logs\/\" target=\"_blank\" rel=\"noreferrer noopener\">Create Kibana Visualization Dashboards for ModSecurity Logs<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/deploy-all-in-one-openstack-with-kolla-ansible-on-ubuntu-18-04\/\" target=\"_blank\" rel=\"noreferrer noopener\">Deploy All-In-One OpenStack with Kolla-Ansible on Ubuntu 18.04<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-run-mariadb-as-a-docker-container\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install and Run MariaDB as a Docker Container<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>How to run ELK stack on Docker? In this tutorial, we are going to learn how to deploy a single node ELK stack cluster on<\/p>\n","protected":false},"author":3,"featured_media":16074,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121,1076,1077,72],"tags":[1673,1896,913,1895,1894,1893],"class_list":["post-6658","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","category-containers","category-docker","category-monitoring","tag-docker-containers","tag-elastic-stack-docker-containers","tag-elk","tag-elk-stack-docker","tag-elk-stack-on-docker-containers","tag-single-node-elastic-stack-cluster-on-docker-containers","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/6658"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=6658"}],"version-history":[{"count":18,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/6658\/revisions"}],"predecessor-version":[{"id":21490,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/6658\/revisions\/21490"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/16074"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=6658"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=6658"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=6658"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}