{"id":13474,"date":"2022-07-16T13:41:13","date_gmt":"2022-07-16T10:41:13","guid":{"rendered":"https:\/\/kifarunix.com\/?p=13474"},"modified":"2024-03-09T20:16:53","modified_gmt":"2024-03-09T17:16:53","slug":"install-logstash-8-on-ubuntu-debian","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/install-logstash-8-on-ubuntu-debian\/","title":{"rendered":"Install Logstash 8 on Ubuntu\/Debian"},"content":{"rendered":"<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"991\" height=\"556\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/07\/install-logstash-8.x.png\" alt=\"Install Logstash 8 on Ubuntu\/Debian\" class=\"wp-image-13483\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/07\/install-logstash-8.x.png?v=1657967781 991w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/07\/install-logstash-8.x-768x431.png?v=1657967781 768w\" sizes=\"(max-width: 991px) 100vw, 991px\" \/><\/figure><\/div>\n\n\n<p>This guide is about how to install and configure Logstash 8 on Ubuntu\/Debian as a continuation of our guide on how to setup Elastic Stack 8. We have already covered the installation of Elasticsearch and Kibana.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-elk-stack-8-x-on-ubuntu\/\">Install Elastic\/ELK Stack on Ubuntu\/Debian<\/a><\/p>\n\n\n\n<p>According to the installation order, Logstash is the third component in the line. Note that for Elastic Stack to function well, the versions of all the components <strong>must<\/strong> match.<\/p>\n\n\n\n<p>Note that you can install Logstash 8 on the same node where Elasticsearch is running or on a different node.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#install-logstash-8-on-ubuntu-debian\">Install Logstash 8 on Ubuntu\/Debian<\/a><ul><li><a href=\"#configuring-logstash-8\">Configuring Logstash 8<\/a><ul><li><a href=\"#configure-logstash-input-plugin\">Configure Logstash Input plugin<\/a><\/li><li><a href=\"#configure-logstash-filters\">Configure Logstash Filters<\/a><\/li><li><a href=\"#configure-logstash-output\">Configure Logstash Output<\/a><\/li><li><a href=\"#test-logstash-configuration\">Test Logstash Configuration<\/a><\/li><\/ul><\/li><li><a href=\"#running-logstash\">Running Logstash<\/a><\/li><li><a href=\"#other-tutorials\">Other Tutorials<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"install-logstash-8-on-ubuntu-debian\">Install Logstash 8 on Ubuntu\/Debian<\/h2>\n\n\n\n<p>You can install Logstash 8 by either using DEB binary or right from APT repositories.<\/p>\n\n\n\n<p>We prefer the installation from APT repositories as this ensure a seamless upgrade incase of new package releases.<\/p>\n\n\n\n<p>Thus, to install Logstash 8 via the APT repos;<\/p>\n\n\n\n<p>Install Repository public signing key<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>wget -qO - https:\/\/artifacts.elastic.co\/GPG-KEY-elasticsearch | gpg --dearmor &gt; \/etc\/apt\/trusted.gpg.d\/elasticsearch.gpg<\/code><\/pre>\n\n\n\n<p>Install Elastic Stack APT repositories<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>apt install apt-transport-https<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"deb https:\/\/artifacts.elastic.co\/packages\/8.x\/apt stable main\" | sudo tee \/etc\/apt\/sources.list.d\/elastic-8.x.list<\/code><\/pre>\n\n\n\n<p>Install Logstash 8;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>apt update<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>apt install logstash<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configuring-logstash-8\">Configuring Logstash 8<\/h3>\n\n\n\n<p>Once the installation is done, proceed to configure Logstash.<\/p>\n\n\n\n<p>Logstash data processing pipeline has three sections;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>INPUT<\/strong>: input section is used to ingest data from different endpoints into Logstash.<\/li>\n\n\n\n<li><strong>FILTERS<\/strong>: which processes and transform the data received.<\/li>\n\n\n\n<li><strong>OUTPUT<\/strong>: which stashes processed data into a specified destination, which can be Elasticsearch.<\/li>\n<\/ul>\n\n\n\n<p>You can read more about Logstash Pipeline <a rel=\"noreferrer noopener\" aria-label=\"here (opens in a new tab)\" href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/7.0\/pipeline.html\" target=\"_blank\">here<\/a>.<\/p>\n\n\n\n<p>While configuring Logstash, you can have separate configuration files each for INPUT, FILTERS and OUTPUT. You can as well have single configuration file for all the sections. This guides uses a single configuration files.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-logstash-input-plugin\">Configure Logstash Input plugin<\/h4>\n\n\n\n<p>To kick off with, you need to define how data will be ingested into Logstash. For example, to configure Logstash to receive data from <a aria-label=\" (opens in a new tab)\" href=\"https:\/\/www.elastic.co\/downloads\/beats\" target=\"_blank\" rel=\"noreferrer noopener\">Beats<\/a> on TCP port 5044, the input configuration may look like;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>input {\n  beats {\n    port =&gt; 5044\n  }\n}\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-logstash-filters\">Configure Logstash Filters<\/h4>\n\n\n\n<p>Now that we have defined an Logstash Input plugin as Beats. Hence, proceed to configure a filter plugin to process events received from the beats. This guide uses <strong><a rel=\"noreferrer noopener\" aria-label=\"grok (opens in a new tab)\" href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/plugins-filters-grok.html\" target=\"_blank\">grok<\/a><\/strong> filter plugin. You can read about other plugins <a rel=\"noreferrer noopener\" aria-label=\"here (opens in a new tab)\" href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/filter-plugins.html\" target=\"_blank\">here<\/a>.<\/p>\n\n\n\n<p>For demonstration purposes, we are going to configure beats to collect SSH authentication events from Ubuntu\/CentOS systems. Hence, we are going to create a filter to process such kind of events as shown below.<\/p>\n\n\n\n<p>The grok pattern used in this example matches the ssh authetication log lines below;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>May  1 13:15:23 elk sshd[1387]: <strong>Failed password<\/strong> for testuser from 192.168.0.102 port 60004 ssh2<br>May  1 13:08:30 elk sshd[1338]: <strong>Accepted password<\/strong> for testuser from 192.168.0.102 port 59958 ssh2\n<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>filter {\n  grok {\n    match =&gt; { \"message\" =&gt; \"%{SYSLOGTIMESTAMP:timestamp}\\s+%{IPORHOST:dst_host}\\s+%{WORD:syslog_program}\\[\\d+\\]:\\s+(?&lt;status&gt;\\w+\\s+password)\\s+for\\s+%{USER:auth_user}\\s+from\\s+%{SYSLOGHOST:src_host}.*\" }\n    add_field =&gt; { \"activity\" =&gt; \"SSH Logins\" }\n    add_tag =&gt; \"linux_auth\"\n    }\n}\n<\/code><\/pre>\n\n\n\n<p>Kibana comes bundled with Grok Debugger which is similar to <a aria-label=\"herokuapp grokdebugger (opens in a new tab)\" href=\"http:\/\/grokdebug.herokuapp.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">herokuapp grokdebugger<\/a>.  You can access Kibana Grok debugger under <strong>Dev Tools &gt; Grok Debugger<\/strong>. You can utilize this to generate the correct grok patterns.<\/p>\n\n\n\n<p>You can as well check common Logstash grok patterns <a aria-label=\"here (opens in a new tab)\" href=\"https:\/\/github.com\/logstash-plugins\/logstash-patterns-core\/blob\/master\/patterns\/grok-patterns\" target=\"_blank\" rel=\"noreferrer noopener\">here<\/a>.<\/p>\n\n\n\n<p>Also for the purposes of making demo a simple, we will add a filter to drop all the events that do not match our grok filter for SSH authentication events used above;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">if \"_grokparsefailure\" in [tags] { drop {} }<\/pre>\n\n\n\n<p>Such that our filter looks like;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>filter {\n  grok {\n    match =&gt; { \"message\" =&gt; \"%{SYSLOGTIMESTAMP:timestamp}\\s+%{IPORHOST:dst_host}\\s+%{WORD:syslog_program}\\[\\d+\\]:\\s+(?&lt;status&gt;\\w+\\s+password)\\s+for\\s+%{USER:auth_user}\\s+from\\s+%{SYSLOGHOST:src_host}.*\" }\n    add_field =&gt; { \"activity\" =&gt; \"SSH Logins\" }\n    add_tag =&gt; \"linux_auth\"\n    }\n  if \"_grokparsefailure\" in [tags] { drop {} }\n}\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-logstash-output\">Configure Logstash Output<\/h4>\n\n\n\n<p>There are different <a aria-label=\"output plugins (opens in a new tab)\" href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/output-plugins.html\" target=\"_blank\" rel=\"noreferrer noopener\">output plugins<\/a> that enables Logstash to sent event data to particular destinations. This guide uses <strong>elasticsearch<\/strong> output plugin that enables Logstash to sent data directly to Elasticsearch.<\/p>\n\n\n\n<p>Note the with Elastic Stack 8, Elasticsearch 8 is configured with SSL\/TLS as well as authentication is enabled by default.<\/p>\n\n\n\n<p>This means that if you are using Elasticsearch output;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>you need to have the Elastic Stack SSL\/TLS CA certificates to be able to connect to Elasticsearch.<\/li>\n\n\n\n<li>You also need to have the right credentials to be able to write to an index on Elasticsearch.<\/li>\n<\/ul>\n\n\n\n<p>Thus, download Elasticsearch CA certificate<\/p>\n\n\n\n<p>Logstash output configuration may look like;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\noutput {\n  elasticsearch {\n    hosts => [\"https:\/\/es-node01.kifarunix-demo.com:9200\"]\n    cacert => '\/etc\/logstash\/elasticsearch-ca.crt'\n    user => 'elastic'\n    password => '&lt;elastic_user_password>'\n  }\n}\n<\/code><\/pre>\n\n\n\n<p>This configuration;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>sends data to Elasticsearch running on host <strong><code>es-node01.kifarunix-demo.com<\/code><\/strong>. The name should be resolvable.<\/li>\n\n\n\n<li>Ensure port 9200\/TCP is opened on firewall.<\/li>\n\n\n\n<li>uses the superuser Elasticsearch user (<strong><code>elastic<\/code><\/strong>) which can write to any index. <strong>Consider creating a different user and give the specific permissions to write to specific index only<\/strong>. We use the elastic user password from our guide on <a href=\"https:\/\/kifarunix.com\/install-elk-stack-8-x-on-ubuntu\/\" target=\"_blank\" rel=\"noreferrer noopener\">installing elastic stack 8<\/a>.<\/li>\n\n\n\n<li>will create and write data to the default Logstash Index, <strong><code>logstash-*<\/code><\/strong> index on Elasticsearch (since we used the superuser)<\/li>\n<\/ul>\n\n\n\n<p>Check <a href=\"https:\/\/kifarunix.com\/configure-logstash-elasticsearch-basic-authentication\/#logstash-elasticearch-authentication\" target=\"_blank\" rel=\"noreferrer noopener\">how to create publishing roles for specific user on a specific index<\/a>.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Download Elasticsearch CA certificate and save it to a file specified by <code><strong>cacert<\/strong><\/code> parameter above, <strong><code>\/etc\/logstash\/elasticsearch-ca.crt<\/code><\/strong>.<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>openssl s_client -showcerts -connect es-node01.kifarunix-demo.com:9200 &lt;\/dev\/null 2&gt;\/dev\/null \\\n| openssl x509 &gt; \/etc\/logstash\/elasticsearch-ca.crt<\/code><\/pre>\n\n\n\n<p>Putting togerther the NPUT, FILTER and OUTPUT configs in one file, then create a configuration file as shown below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">vim \/etc\/logstash\/conf.d\/ssh-authentication.conf<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>input {\n  beats {\n    port =&gt; 5044\n  }\n}\nfilter {\n  grok {\n    match =&gt; { \"message\" =&gt; \"%{SYSLOGTIMESTAMP:timestamp}\\s+%{IPORHOST:dst_host}\\s+%{WORD:syslog_program}\\[\\d+\\]:\\s+(?&lt;status&gt;\\w+\\s+password)\\s+for\\s+%{USER:auth_user}\\s+from\\s+%{SYSLOGHOST:src_host}.*\" }\n    add_field =&gt; { \"activity\" =&gt; \"SSH Logins\" }\n    add_tag =&gt; \"linux_auth\"\n    }\n  if \"_grokparsefailure\" in [tags] { drop {} }\n}\noutput {\n  elasticsearch {\n    hosts => [\"https:\/\/es-node01.kifarunix-demo.com:9200\"]\n    cacert => '\/etc\/logstash\/elasticsearch-ca.crt'\n    user => 'elastic'\n    password => '&lt;elastic_user_password>'\n  }\n}\n<\/code><\/pre>\n\n\n\n<p>You can also <a href=\"https:\/\/kifarunix.com\/configure-logstash-elasticsearch-basic-authentication\/#logstash-secure-keystore\" target=\"_blank\" rel=\"noreferrer noopener\">store the password in a keystore instead of placing in the configuration in plaintext<\/a>.<\/p>\n\n\n\n<p>If you need to sent the event data to standard output for the purposes of debugging plugin configurations, then you would add the line, <strong>stdout { codec =&gt; rubydebug }<\/strong> to the output configuration section.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\noutput {\n  elasticsearch {\n    hosts => [\"https:\/\/es-node01.kifarunix-demo.com:9200\"]\n    cacert => '\/etc\/logstash\/elasticsearch-ca.crt'\n    user => 'elastic'\n    password => '&lt;elastic_user_password>'\n  }\n}\n  stdout { codec =&gt; rubydebug }\n}\n<\/code><\/pre>\n\n\n\n<p>You can also check sample Logstash pipelines <a rel=\"noreferrer noopener\" aria-label=\"here (opens in a new tab)\" href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/7.0\/logstash-config-for-filebeat-modules.html#parsing-system\" target=\"_blank\">here<\/a>.<\/p>\n\n\n\n<p>If you need to debug Logstash Grok Filters to confirm that they can actually parse your logs into the required fields, see the link below on how to debug Logstash Grok filters.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/how-to-debug-logstash-grok-filters\/\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">How to Debug Logstash Grok Filters<\/a><\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"test-logstash-configuration\">Test Logstash Configuration<\/h4>\n\n\n\n<p>Once you are done with configurations, run the command below to verify the Logstash configuration before you can start it.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo -u logstash \/usr\/share\/logstash\/bin\/logstash --path.settings \/etc\/logstash -t<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>Configuration OK\n<\/strong>&#91;2022-07-16T07:42:10,315]&#91;INFO ]&#91;logstash.runner          ] Using config.test_and_exit mode. <strong>Config Validation Result: OK<\/strong>. Exiting Logstash<\/code><\/pre>\n\n\n\n<p>Well, if you get <strong>Configuration OK<\/strong> then you are good to go.<\/p>\n\n\n\n<p>To run Logstash and load a specific configuration file for debugging, you can execute the command below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>sudo -u logstash \/usr\/share\/logstash\/bin\/logstash -f \/etc\/logstash\/conf.d\/<strong>config-file.conf<\/strong><\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"running-logstash\">Running Logstash<\/h3>\n\n\n\n<p>You can now start and enable Logstash to run on system boot.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">systemctl enable --now logstash<\/pre>\n\n\n\n<p>You can also check the Logstash configuration file for any errors, <strong>\/var\/log\/logstash\/logstash-plain.log<\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>tail -f \/var\/log\/logstash\/logstash-plain.log<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\n...\n[2022-07-16T10:38:30,215][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}\n[2022-07-16T10:38:30,450][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>\"main\", \"pipeline.workers\"=>2, \"pipeline.batch.size\"=>125, \"pipeline.batch.delay\"=>50, \"pipeline.max_inflight\"=>250, \"pipeline.sources\"=>[\"\/etc\/logstash\/conf.d\/beats-input.conf\"], :thread=>\"#<Thread:0x2cc0fa11 run>\"}\n[2022-07-16T10:38:31,130][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {\"seconds\"=>0.68}\n[2022-07-16T10:38:31,157][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>\"0.0.0.0:5044\"}\n[2022-07-16T10:38:31,197][INFO ][logstash.javapipeline    ][main] Pipeline started {\"pipeline.id\"=>\"main\"}\n[2022-07-16T10:38:31,302][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}\n[2022-07-16T10:38:31,495][INFO ][org.logstash.beats.Server][main][a8ac06f266dffa737ca74e142bc94412d96789db1667ea3f32d274fc6578859b] Starting server on port: 5044\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ss -altnp<\/code><\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code>\nLISTEN         0              50                     [::ffff:127.0.0.1]:9600                              *:*             users:((\"java\",pid=46795,fd=58))                  \nLISTEN         0              4096                                    *:5044                              *:*             users:((\"java\",pid=46795,fd=102))\n<\/code><\/pre>\n\n\n\n<p>And that is how easy it is to install and run Logstash 8 on Ubuntu\/Debian.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"other-tutorials\">Other Tutorials<\/h3>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/setup-multinode-elasticsearch-8-x-cluster\/\" target=\"_blank\" rel=\"noreferrer noopener\">Setup Multinode Elasticsearch 8.x Cluster<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/configure-logstash-elasticsearch-basic-authentication\/\" target=\"_blank\" rel=\"noreferrer noopener\">Configure Logstash Elasticsearch Basic Authentication<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/logstash-write-specific-events-to-specific-index\/\" target=\"_blank\" rel=\"noreferrer noopener\">Logstash: Write Specific Events to Specific Index<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This guide is about how to install and configure Logstash 8 on Ubuntu\/Debian as a continuation of our guide on how to setup Elastic Stack<\/p>\n","protected":false},"author":1,"featured_media":8883,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121,72],"tags":[5552,5555,5551,5554,5553],"class_list":["post-13474","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","category-monitoring","tag-elastic-stack-8-logstash","tag-elk-8","tag-install-logstash-8-on-ubuntu-debian","tag-logstash-8-install","tag-logstash-8-x-install","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/13474"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=13474"}],"version-history":[{"count":6,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/13474\/revisions"}],"predecessor-version":[{"id":20590,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/13474\/revisions\/20590"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/8883"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=13474"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=13474"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=13474"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}