{"id":5754,"date":"2020-05-04T20:40:06","date_gmt":"2020-05-04T17:40:06","guid":{"rendered":"https:\/\/kifarunix.com\/?p=5754"},"modified":"2024-03-14T20:40:33","modified_gmt":"2024-03-14T17:40:33","slug":"install-elastic-elk-stack-on-ubuntu-20-04","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/install-elastic-elk-stack-on-ubuntu-20-04\/","title":{"rendered":"Install ELK Stack on Ubuntu 20.04"},"content":{"rendered":"<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"521\" height=\"485\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/05\/elk-stack.png\" alt=\"Install Elastic (ELK) Stack on Ubuntu 20.04\" class=\"wp-image-5764\" title=\"\"><\/figure><\/div>\n\n\n<p>Welcome to our guide on how to install ELK Stack on Ubuntu 20.04. <a rel=\"noreferrer noopener\" href=\"https:\/\/www.elastic.co\/what-is\/elk-stack\" target=\"_blank\">ELK<\/a>, currently known as Elastic Stack, is the acronym for open source projects comprising;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Elasticsearch<\/strong>: a search and analytics engine<\/li>\n\n\n\n<li><strong>Kibana<\/strong>: a data visualization and dash-boarding tool that enables you to analyze data stored on Elasticsearch.<\/li>\n\n\n\n<li><strong>Logstash<\/strong>: a server\u2011side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then stashes it on search analytics engine like Elasticsearch<\/li>\n\n\n\n<li><strong>Beats<\/strong>&nbsp;on the other hand are the log shippers that collects logs from different endpoints and sends them to either Logstash or directly to Elasticsearch.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Installing ELK Stack on Ubuntu 20.04<\/h2>\n\n\n\n<p>Installation of Elastic Stack follows a specific order. Below is the order of installing Elastic Stack components;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"#install-elasticsearch-ubuntu\">Install Elasticsearch<\/a><\/li>\n\n\n\n<li><a href=\"#install-kibana-ubuntu\">Install Kibana<\/a><\/li>\n\n\n\n<li><a href=\"#install-logstash\">Install Logstash<\/a><\/li>\n\n\n\n<li><a href=\"#install-filebeat-ubuntu-20.04\">Install Beats<\/a><\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Run system update<\/h3>\n\n\n\n<p>Before you can start the installation, ensure that the system packages are up-to-date.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-elasticsearch-ubuntu\">Install Elasticsearch on Ubuntu 20.04<\/h3>\n\n\n\n<p>You can install Elasticsearch automatically from Elastic repos or you can download Elasticsearch DEB binary package and install it. However, to simplify the installation of all Elastic Stack components, we will create Elastic Stack repos;<\/p>\n\n\n\n<p>Import the Elastic stack PGP repository signing Key<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>wget -qO - https:\/\/artifacts.elastic.co\/GPG-KEY-elasticsearch --no-check-certificate | sudo apt-key add -<\/code><\/pre>\n\n\n\n<p>Install Elasticsearch;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>echo \"deb https:\/\/artifacts.elastic.co\/packages\/7.x\/apt stable main\" | sudo tee -a \/etc\/apt\/sources.list.d\/elastic-7.x.list<\/code><\/pre>\n\n\n\n<p> Update package cache and install Elasticsearch;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apt update<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apt install elasticsearch<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Configure Elasticsearch on Ubuntu 20.04<\/h4>\n\n\n\n<p>There are only a few configuration changes we are going to make on this tutorial. First off, we configure ES to listen on a specific Interface IP to allow external access. Elasticsearch is listening on&nbsp;<strong>localhost<\/strong>&nbsp;by default.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/elasticsearch\/elasticsearch.yml<\/code><\/pre>\n\n\n\n<p>You can choose to change the default cluster name;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>...\n# ---------------------------------- Cluster -----------------------------------\n#\n# Use a descriptive name for your cluster:\n#\n#cluster.name: my-application\n<strong>cluster.name: kifarunix-demo<\/strong>\n...<\/code><\/pre>\n\n\n\n<p>Uncomment and change the value of&nbsp;<strong><code>network.host<\/code><\/strong>&nbsp;as well the&nbsp;<strong><code>http.port<\/code><\/strong>&nbsp;line under the Network settings section.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code># ---------------------------------- Network -----------------------------------\n#\n# Set the bind address to a specific IP (IPv4 or IPv6):\n#\n#network.host: 192.168.0.1\n<strong>network.host: 10.10.9.9<\/strong>\n#\n# Set a custom port for HTTP:\n#\n<strong>http.port: 9200<\/strong>\n#\n# For more information, consult the network module documentation<\/code><\/pre>\n\n\n\n<p>Since we are running a single node Elasticsearch, specify the same in the configuration by adding the line, <strong><code>discovery.type: single-node<\/code><\/strong> under the Discovery settings section.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>...\n# --------------------------------- Discovery ----------------------------------\n#\n# Pass an initial list of hosts to perform discovery when this node is started:\n# The default list of hosts is [\"127.0.0.1\", \"[::1]\"]\n...\n...\n#cluster.initial_master_nodes: [\"node-1\", \"node-2\"]\n# This is for Single Node Elastic stack\n<strong>discovery.type: single-node<\/strong>\n...<\/code><\/pre>\n\n\n\n<p>Save and exit the config.<\/p>\n\n\n\n<p>Next, configure JVM heap size to no more than half the size of your memory. In this case, our test server has 2G RAM and the heap size is set to 512M for both maximum and minimum sizes.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/elasticsearch\/jvm.options<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>...\n################################################################\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n<strong>-Xms512m\n-Xmx512m<\/strong>\n...<\/code><\/pre>\n\n\n\n<p>Those are just about the few changes we would make on ES.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Running Elasticsearch<\/h4>\n\n\n\n<p>Start and enable Elasticsearch to run on system boot;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl enable --now elasticsearch<\/code><\/pre>\n\n\n\n<p>To check the status;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl status elasticsearch<\/code><\/pre>\n\n\n\n<p>You can as well verify ES status using curl command. Replace the IP accordingly.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>curl http:\/\/10.10.9.9:9200<\/code><\/pre>\n\n\n\n<p>If you get such an output, then all is well.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"name\" : \"ubuntu20\",\n  \"cluster_name\" : \"kifarunix-demo\",\n  \"cluster_uuid\" : \"3RY1LTvyTD2Bie74xGw6Vg\",\n  \"version\" : {\n    \"number\" : \"7.6.2\",\n    \"build_flavor\" : \"default\",\n    \"build_type\" : \"deb\",\n    \"build_hash\" : \"ef48eb35cf30adf4db14086e8aabd07ef6fb113f\",\n    \"build_date\" : \"2020-03-26T06:34:37.794943Z\",\n    \"build_snapshot\" : false,\n    \"lucene_version\" : \"8.4.0\",\n    \"minimum_wire_compatibility_version\" : \"6.8.0\",\n    \"minimum_index_compatibility_version\" : \"6.0.0-beta1\"\n  },\n  \"tagline\" : \"You Know, for Search\"\n}<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-kibana-ubuntu\">Install Kibana on Ubuntu 20.04<\/h3>\n\n\n\n<p>Since we already setup Elastic repos, simply install Kibana by running the command;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apt install kibana<\/code><\/pre>\n\n\n\n<p>Kibana is set to run on&nbsp;<strong>localhost:5601<\/strong>&nbsp;by default. To allow external access, edit the configuration file and replace the value of <code>server.host<\/code> with an interface IP.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/kibana\/kibana.yml<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code># Kibana is served by a back end server. This setting specifies the port to use.\n<strong>server.port: 5601<\/strong>\n\n...\n# To allow connections from remote users, set this parameter to a non-loopback address.\n#server.host: \"localhost\"\n<strong>server.host: \"10.10.9.9\"<\/strong><\/code><\/pre>\n\n\n\n<p>Set the Elasticsearch URL<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>...\n# The URLs of the Elasticsearch instances to use for all your queries.\n#elasticsearch.hosts: [\"http:\/\/localhost:9200\"]\n<strong>elasticsearch.hosts: [\"http:\/\/10.10.9.9:9200\"]<\/strong>\n<\/code><\/pre>\n\n\n\n<p>If you need to secure Kibana by proxying it with Nginx, you can check how to on our previous by following the link below;<\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/install-elastic-stack-7-on-ubuntu-18-04-debian-9-8\/#proxykibanawithnginx\" target=\"_blank\">Configure Nginx with SSL to Proxy Kibana<\/a><\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Running Kibana<\/h4>\n\n\n\n<p>Once the installation is done, start and enable Kibana to run on system boot.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl enable --now kibana<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Access Kibana Dashboard<\/h4>\n\n\n\n<p>You can  now access Kibana from your browser using the url, <code>http:\/\/&lt;server-IP&gt;:5601<\/code>.<\/p>\n\n\n\n<p>If UFW is running, Open Kibana port;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ufw allow 5601\/tcp<\/code><\/pre>\n\n\n\n<p>Upon accessing Kibana interface, on the welcome page, you are prompted on whether to get started with Kibana sample data since we do not have any data in our cluster yet.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"696\" height=\"678\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/05\/kibanaui-2.png\" alt=\"Install Elastic (ELK) Stack on Ubuntu 20.04\" class=\"wp-image-5784\" title=\"\"><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\" id=\"install-logstash\">Install Logstash on Ubuntu 20.04<\/h3>\n\n\n\n<p>Logstash requires Java 8 or Java 11. You can use the&nbsp;official Oracle distribution&nbsp;or an open-source distribution such as&nbsp;OpenJDK. To install OpenJDK 11;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apt install openjdk-11-jdk -y<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>java --version<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>openjdk 11.0.7 2020-04-14\nOpenJDK Runtime Environment (build 11.0.7+10-post-Ubuntu-3ubuntu1)\nOpenJDK 64-Bit Server VM (build 11.0.7+10-post-Ubuntu-3ubuntu1, mixed mode, sharing)<\/code><\/pre>\n\n\n\n<p>Since we already have Elastic repos in place, install Logstash by running the command below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apt install logstash<\/code><\/pre>\n\n\n\n<p>Once the installation is done, configure Logstash to process any data to be collected from the remote hosts. Follow the link below to learn how to  configure Logstash.<\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/install-and-configure-logstash-7-on-ubuntu-18-debian-9-8\/#configurelogstash\" target=\"_blank\">How to Configure Logstash data processing pipeline<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-filebeat-ubuntu-20.04\"><a href=\"#install-filebeat-ubuntu-20.04\" class=\"rank-math-link\">Install Filebeat on Ubuntu 20.04<\/a><\/h3>\n\n\n\n<p>Filebeat is a lightweight shipper for collecting, forwarding and centralizing event log data. It is installed as an agent on the servers you are collecting logs from. It can forward the logs it is collecting to either Elasticsearch or Logstash for indexing.<\/p>\n\n\n\n<p>To install Filebeat from Elastic repos;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apt install filebeat<\/code><\/pre>\n\n\n\n<p>Once the installation, follow the link below to configure Filebeat for data collection.<\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/install-and-configure-filebeat-7-on-ubuntu-18-04-debian-9-8\/#configurefilebeatonubuntu\" target=\"_blank\">Configure Filebeat log shipper on Ubuntu<\/a><\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Verify Elasticsearch Index Data Reception<\/h4>\n\n\n\n<p>Once you have configured Filebeat to ship authentication logs to Logstash for processing, you can verify is any data has been written to the index defined. For example, in our example setup provided in the link above, we are sending SSH authentication events to <code>ssh_auth-YYYY.MM<\/code> index. This can be verified by querying status of ES indices.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>curl -XGET http:\/\/10.10.9.9:9200\/_cat\/indices?v<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>health status index                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size\ngreen  open   .kibana_task_manager_1           BkfUXSstQdGhcOAD9EyhvQ   1   0          2            0     31.9kb         31.9kb\ngreen  open   .apm-agent-configuration         duVPmKSWQj6k_hGzezyheg   1   0          0            0       283b           283b\ngreen  open   ilm-history-1-000001             pvdSCTQDQXWwlzdlEY1ykg   1   0         18            0     25.3kb         25.3kb\ngreen  open   .kibana_1                        aU6EZ-c4RTGQOsH7coOUqg   1   0          8            0     22.7kb         22.7kb\n<strong>yellow open   ssh_auth-2020.05                 xVQyHuz2SWCEFMQFKYWVDA   1   1        186            0    248.5kb        248.5kb<\/strong><\/code><\/pre>\n\n\n\n<p>From the output, you can see that our SSH index has data. For health color status, read more on <a rel=\"noreferrer noopener\" href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/cluster-health.html\" target=\"_blank\">Cluster Health API<\/a>.<\/p>\n\n\n\n<p>To confirm data reception on Kibana, navigate to Kibana dashboard on the web browser and create your index. Click on <strong>Management tab (on the left side panel) &gt; Kibana&gt; Index Patterns &gt; Create Index Pattern<\/strong>. Enter the wildcard for your index name.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1365\" height=\"654\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/05\/kibana-index.png\" alt=\"\" class=\"wp-image-5797\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/05\/kibana-index.png?v=1588613436 1365w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/05\/kibana-index-768x368.png?v=1588613436 768w\" sizes=\"(max-width: 1365px) 100vw, 1365px\" \/><\/figure>\n\n\n\n<p>Click Next and select timestamp as the time filter.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1318\" height=\"500\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/05\/timefilter.png\" alt=\"\" class=\"wp-image-5798\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/05\/timefilter.png?v=1588613454 1318w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/05\/timefilter-768x291.png?v=1588613454 768w\" sizes=\"(max-width: 1318px) 100vw, 1318px\" \/><\/figure>\n\n\n\n<p>Then click <strong>Create Index pattern<\/strong> to create your index pattern.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">View Data on Kibana<\/h4>\n\n\n\n<p>Once that is done, you can now view your event data on Kibana by clicking on the discover tab on the left pane. Expand your time range accordingly.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1340\" height=\"652\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/05\/kibana-data.png\" alt=\"\" class=\"wp-image-5795\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/05\/kibana-data.png?v=1588613381 1340w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/05\/kibana-data-768x374.png?v=1588613381 768w\" sizes=\"(max-width: 1340px) 100vw, 1340px\" \/><\/figure>\n\n\n\n<p>To filter the SSH events just processed with Logstash, add the fields that were defined on the Logstash grok filter pattern;<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1341\" height=\"467\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/05\/selected-fields.png\" alt=\"\" class=\"wp-image-5796\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/05\/selected-fields.png?v=1588613411 1341w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/05\/selected-fields-768x267.png?v=1588613411 768w\" sizes=\"(max-width: 1341px) 100vw, 1341px\" \/><\/figure>\n\n\n\n<p>And there you go. You can now ingest more data and create Logstash filters if you need further processing before the events gets to ES.<\/p>\n\n\n\n<p>Reference<\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/www.elastic.co\/guide\/en\/elastic-stack\/current\/installing-elastic-stack.html\" target=\"_blank\">Installing Elastic Stack<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Related Tutorials<\/h3>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/installing-elk-stack-on-centos-8\/\" target=\"_blank\" rel=\"noreferrer noopener\">Installing ELK Stack on CentOS 8<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-elastic-stack-7-on-fedora-30-fedora-29-centos-7\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Elastic Stack 7 on Fedora 30\/Fedora 29\/CentOS 7<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-elastic-stack-7-on-ubuntu-18-04-debian-9-8\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Elastic Stack 7 on Ubuntu 18.04\/Debian 9.8<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-icinga-2-and-icinga-web-2-on-ubuntu-20-04\/\">Install Icinga 2 and Icinga Web 2 on Ubuntu 20.04<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Welcome to our guide on how to install ELK Stack on Ubuntu 20.04. ELK, currently known as Elastic Stack, is the acronym for open source<\/p>\n","protected":false},"author":3,"featured_media":9442,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[72,910,121],"tags":[912,1571,1569,1570,1200],"class_list":["post-5754","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-monitoring","category-elastic-stack","category-howtos","tag-elastic-stack","tag-elk-stack-on-ubuntu-20-04","tag-install-elastic-stack-on-ubuntu-20-04","tag-install-elk-on-ubuntu-20-04","tag-ubuntu-20-04","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/5754"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=5754"}],"version-history":[{"count":16,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/5754\/revisions"}],"predecessor-version":[{"id":21400,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/5754\/revisions\/21400"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/9442"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=5754"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=5754"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=5754"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}