{"id":3448,"date":"2019-06-29T16:24:08","date_gmt":"2019-06-29T13:24:08","guid":{"rendered":"https:\/\/kifarunix.com\/?p=3448"},"modified":"2024-03-11T22:54:15","modified_gmt":"2024-03-11T19:54:15","slug":"install-logstash-7-on-fedora-30-fedora-29-centos-7","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/install-logstash-7-on-fedora-30-fedora-29-centos-7\/","title":{"rendered":"Install Logstash 7 on Fedora 30\/Fedora 29\/CentOS 7"},"content":{"rendered":"\n<p>This guide will focus on how to install Logstash 7 on Fedora 30\/Fedora 29\/CentOS 7 as a continuation of our guide on how to <a href=\"https:\/\/kifarunix.com\/install-elastic-stack-7-on-fedora-30-fedora-29-centos-7\/\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">setup Elastic Stack 7 on Fedora 30\/Fedora 29\/CentOS 7<\/a>.<\/p>\n\n\n\n<p>The installation of the first two components of ELastic Stack, Elasticsearch and Kibana have been discussed in our previous guides;<\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/install-elasticsearch-7-on-fedora-30\/\" target=\"_blank\">Install Elasticsearch 7 on Fedora 30<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-elasticsearch-7-x-on-centos-7-fedora-29\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Elasticsearch 7.x on CentOS 7\/Fedora 29<\/a><\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/install-elastic-stack-7-on-fedora-30-fedora-29-centos-7\/\" target=\"_blank\">Install Kibana 7 on Fedora 30\/Fedora 29\/CentOS 7<\/a><\/p>\n\n\n\n<p>Once you have Elasticsearch and Kibana installed, proceed to install Logstash.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Installing Logstash 7 on Fedora 30\/Fedora 29\/CentOS 7<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Prerequisites<\/h3>\n\n\n\n<p>As a prerequisite, Logstash requires Java 8 or Java 11. You can install Java 8 on Fedora 30\/Fedora 29\/CentOS 7 by running the command below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>yum install java-1.8.0-openjdk.x86_64<\/code><\/pre>\n\n\n\n<p>Once the installation is done, you can verify the version as in below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>java -version\nopenjdk version \"1.8.0_212\"\nOpenJDK Runtime Environment (build 1.8.0_212-b04)\nOpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode)<\/code><\/pre>\n\n\n\n<p>If you need to use Java 11, install it as shown below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>dnf install java-11-openjdk.x86_64<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Installing Logstash 7 on Fedora 30\/Fedora 29\/CentOS 7<\/h3>\n\n\n\n<p>As stated before, this is a continuation of our guide on how to setup Elastic Stack on Fedora 30\/Fedora 29\/CentOS 7. Therefore, we have already created the Elastic Stack repos in our servers.You can however create Elastic 7.x repos by executing the command below;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Import Elastic Repo GPG signing key<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>rpm --import https:\/\/artifacts.elastic.co\/GPG-KEY-elasticsearch<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Create Elastic 7.x Repo<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>cat &gt; \/etc\/yum.repos.d\/elastic-7.x.repo &lt;&lt; EOF\n&#91;elasticsearch-7.x]\nname=Elasticsearch repository for 7.x packages\nbaseurl=https:\/\/artifacts.elastic.co\/packages\/7.x\/yum\ngpgcheck=1\ngpgkey=https:\/\/artifacts.elastic.co\/GPG-KEY-elasticsearch\nenabled=1\nautorefresh=1\ntype=rpm-md\nEOF<\/code><\/pre>\n\n\n\n<p>And now you can just install Logstash using the YUM\/DNF package manager.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>yum install logstash<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Testing Logstash<\/h3>\n\n\n\n<p>To test your Logstash installation, run the most basic Logstash pipeline.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>cd \/usr\/share\/logstash\/bin\/<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>.\/logstash -e 'input { stdin { } } output { stdout {} }'<\/code><\/pre>\n\n\n\n<p>Once you see the, Pipeline main started, type any string and press ENTER.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>...\n&#91;INFO ] 2019-06-29 15:12:32.023 &#91;Ruby-0-Thread-1: \/usr\/share\/logstash\/lib\/bootstrap\/environment.rb:6] agent - Pipelines running {:count=&gt;1, :running_pipelines=&gt;&#91;:main], :non_running_pipelines=&gt;&#91;]}\n&#91;INFO ] 2019-06-29 15:12:32.821 &#91;Api Webserver] agent - Successfully started Logstash API endpoint {:port=&gt;9600}\nHello world\n...<\/code><\/pre>\n\n\n\n<p>Logstash adds timestamp and host address information to the message.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>...\n{\n          \"host\" =&gt; \"elastic.example.com\",\n      \"@version\" =&gt; \"1\",\n       \"message\" =&gt; \"Hello world\",\n    \"@timestamp\" =&gt; 2019-06-29T12:13:06.994Z\n}\n...<\/code><\/pre>\n\n\n\n<p>Stop Logstash by pressing Ctrl+D.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Configuring Logstash 7 on Fedora 30\/Fedora 29\/CentOS 7<\/h3>\n\n\n\n<p>Once the installation is done, proceed to configure Logstash. Logstash data processing pipeline has three sections;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>INPUT<\/strong>: input section is used to ingest data from different endpoints into Logstash.<\/li>\n\n\n\n<li><strong>FILTERS<\/strong>: which processes and transform the data received.<\/li>\n\n\n\n<li><strong>OUTPUT<\/strong>: which stashes processed data into a specified destination, which can be Elasticsearch.<\/li>\n<\/ul>\n\n\n\n<p>You can read more about Logstash Pipeline <a rel=\"noreferrer noopener\" aria-label=\"here (opens in a new tab)\" href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/7.0\/pipeline.html\" target=\"_blank\">here<\/a>.<\/p>\n\n\n\n<p>While configuring Logstash, you can have separate configuration files each for INPUT, FILTERS and OUTPUT. You can as well have single configuration file for all the sections. This guides uses separate configuration files.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Configure Logstash Input plugin<\/h4>\n\n\n\n<p>Create Logstash input configuration file. In this guide, Beats are used as the data shippers. Hence, to configure Logstash to receive data from <a rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\" href=\"https:\/\/www.elastic.co\/downloads\/beats\" target=\"_blank\">Beats<\/a> on TCP port 5044, create an input configuration file say, <strong>\/etc\/logstash\/conf.d\/beats-input.conf<\/strong>, with the content below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/logstash\/conf.d\/beats-input.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>input {\n  beats {\n    port =&gt; 5044\n  }\n}<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"mce_21\">Configure Logstash Filters<\/h4>\n\n\n\n<p>Configure a filter plugin to process events received from the beats. This guide uses <strong><a rel=\"noreferrer noopener\" aria-label=\"grok (opens in a new tab)\" href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/plugins-filters-grok.html\" target=\"_blank\">grok<\/a><\/strong> filter plugin. You can read about other plugins <a rel=\"noreferrer noopener\" aria-label=\"here (opens in a new tab)\" href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/filter-plugins.html\" target=\"_blank\">here<\/a>.<\/p>\n\n\n\n<p>For demonstration purposes, we are going to configure beats to collect SSH authentication events from Ubuntu\/CentOS systems. Hence, we are going to create a filter to process such kind of events as shown below.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/logstash\/conf.d\/ssh-auth-filter.conf<\/code><\/pre>\n\n\n\n<p>The grok pattern used in this example matches the ssh authetication log lines below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>Jun 29 13:19:13 fedora29 sshd[2764]: <strong>Failed password<\/strong> for root from 192.168.43.17 port 40284 ssh2\nJun 29 13:13:31 fedora29 sshd[2598]: <strong>Accepted password<\/strong> for root from 192.168.43.17 port 40182 ssh2<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>filter {\n  grok {\n    match =&gt; { \"message\" =&gt; \"%{SYSLOGTIMESTAMP:timestamp}\\s+%{IPORHOST:dst_host}\\s+%{WORD:syslog_program}\\&#91;\\d+\\]:\\s+(?&lt;status&gt;\\w+\\s+password)\\s+for\\s+%{USER:auth_user}\\s+from\\s+%{SYSLOGHOST:src_host}.*\" }\n    add_field =&gt; { \"activity\" =&gt; \"SSH Logins\" }\n    add_tag =&gt; \"linux_auth\"\n      }\n    }\n<\/code><\/pre>\n\n\n\n<p>The lines, <strong>if [fileset][module] == &#8220;system&#8221;<\/strong>, <strong>if [fileset][name] == &#8220;auth&#8221;<\/strong> would be used to specify to ask Logstash to apply Grok filters on the events sent by this module. However, I used these and my Grok Pattern failed to extract data fields. In case you have an idea around this, drop it in comments.<\/p>\n\n\n\n<p>Kibana 7 comes bundled with Grok Debugger which is similar to <a rel=\"noreferrer noopener\" aria-label=\"herokuapp grokdebugger (opens in a new tab)\" href=\"http:\/\/grokdebug.herokuapp.com\/\" target=\"_blank\">herokuapp grokdebugger<\/a>.  You can access Kibana Grok debugger under <strong>Dev Tools &gt; Grok Debugger<\/strong>. You can utilize this to generate the correct grok patterns. You can as well check common logstash grok patterns <a rel=\"noreferrer noopener\" aria-label=\"here (opens in a new tab)\" href=\"https:\/\/github.com\/logstash-plugins\/logstash-patterns-core\/blob\/master\/patterns\/grok-patterns\" target=\"_blank\">here<\/a>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"mce_28\">Configure Logstash Output<\/h4>\n\n\n\n<p>There are different <a rel=\"noreferrer noopener\" aria-label=\"output plugins (opens in a new tab)\" href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/output-plugins.html\" target=\"_blank\">output plugins<\/a> that enables Logstash to sent event data to particular destinations. This guide uses <strong>elasticsearch<\/strong> that enables Logstash to sent data to Elasticsearch.<\/p>\n\n\n\n<p>Create Logstash output configuration file with the content below. This confguration sents data to Elasticsearch running on the same host.<\/p>\n\n\n\n<p>The index defines the index to write events to, <strong>logstash-%{+YYYY.MM.dd}<\/strong> is the default index.<\/p>\n\n\n\n<p>If Elasticsearch is listening on non-loopback interface, replace localhost, <em>hosts =&gt; [&#8220;<strong>localhost<\/strong>:9200&#8243;]<\/em>  with an interface IP, for example; <em>hosts =&gt; [&#8220;<strong>192.168.0.101<\/strong>:9200&#8243;]<\/em><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/logstash\/conf.d\/elasticsearch-output.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>output {<br>   elasticsearch {<br>     hosts =&gt; [\"<em><strong>192.168.0.101<\/strong><\/em>:9200\"]<br>     manage_template =&gt; false<br>     index =&gt; \"ssh_auth-%{+YYYY.MM}\"<br> }<br>}<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"logstash-configuration\"><a href=\"#logstash-configuration\" class=\"rank-math-link\">All in one Logstash configuration file<\/a><\/h4>\n\n\n\n<p>So far we have used different configuration file for each Logstash section. <\/p>\n\n\n\n<p>If you need to put them in one file, then create a configuration file as shown below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/logstash\/conf.d\/ssh-authentication.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>input {\n  beats {\n    port =&gt; 5044\n  }\n}\nfilter {\n  grok {\n    match =&gt; { \"message\" =&gt; \"%{SYSLOGTIMESTAMP:timestamp}\\s+%{IPORHOST:dst_host}\\s+%{WORD:syslog_program}\\&#91;\\d+\\]:\\s+(?&lt;status&gt;\\w+\\s+password)\\s+for\\s+%{USER:auth_user}\\s+from\\s+%{SYSLOGHOST:src_host}.*\" }\n    add_field =&gt; { \"activity\" =&gt; \"SSH Logins\" }\n    add_tag =&gt; \"linux_auth\"\n    }\n}\noutput {\n   elasticsearch {\n     hosts =&gt; &#91;\"localhost:9200\"]\n     manage_template =&gt; false\n     index =&gt; \"ssh_auth-%{+YYYY.MM}\"\n }\n}<\/code><\/pre>\n\n\n\n<p>If you need to sent the event data to standard output as well for the purposes of debugging plugin configurations, then you would add the line, <strong>stdout { codec =&gt; rubydebug }<\/strong> to the output configuration section.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>output {\n  elasticsearch {\n    hosts =&gt; &#91;\"localhost:9200\"]\n    index =&gt; \"ssh_auth-%{+YYYY.MM}\"\n}\n  stdout { codec =&gt; rubydebug }\n}<\/code><\/pre>\n\n\n\n<p>You can also check sample Logstash pipelines <a rel=\"noreferrer noopener\" aria-label=\"here (opens in a new tab)\" href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/7.0\/logstash-config-for-filebeat-modules.html#parsing-system\" target=\"_blank\">here<\/a>.<\/p>\n\n\n\n<p>Learn how to debug Logstash Grok Filters by following the link below;<\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/how-to-debug-logstash-grok-filters\/\" target=\"_blank\">How to Debug Logstash Grok Filters<\/a><\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Test Logstash Configuration<\/h4>\n\n\n\n<p>Once you are done with configurations, run the command below to verify the Logstash configuration before you can start it.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>sudo -u logstash \/usr\/share\/logstash\/bin\/logstash --path.settings \/etc\/logstash -t<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>Sending Logstash logs to \/var\/log\/logstash which is now configured via log4j2.properties\n<strong>Configuration OK<\/strong>\n...<\/code><\/pre>\n\n\n\n<p>Well, if you get <strong>Configuration OK<\/strong> then you are good to go.<\/p>\n\n\n\n<p>To run Logstash and load a specific configuration file for debugging, you can execute the command below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>sudo -u logstash \/usr\/share\/logstash\/bin\/logstash -f \/etc\/logstash\/conf.d\/<strong>configuration-file.conf<\/strong> --path.settings \/etc\/logstash\/<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Running Logstash<\/h3>\n\n\n\n<p>You can now start and enable Logstash to run on system boot.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl start logstash<br>systemctl enable logstash<\/code><\/pre>\n\n\n\n<p>On CentOS 7, if you try to start Logstash and get the error, <strong>Unit logstash.service could not be found<\/strong>, run the command below to generate systemd unit file.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>\/usr\/share\/logstash\/bin\/system-install \/etc\/logstash\/startup.options systemd<\/code><\/pre>\n\n\n\n<p>To check the <strong>\/var\/log\/logstash\/logstash-plain.log<\/strong> log file for any logstash configuration errors.<\/p>\n\n\n\n<p>If Firewalld is running and you want to receive event data from remote systems, ensure that TCP 5044.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>firewall-cmd --add-port=5044\/tcp --permanent\nfirewall-cmd --reload<\/code><\/pre>\n\n\n\n<p>Once you are done with configuration, proceed to install and configure Filebeat data shippers. See our next guide on how to install Filebeats on Fedora 30\/Fedora 29\/CentOS 7.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-filebeat-on-fedora-30-fedora-29-centos-7\/\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">Install Filebeat on Fedora 30\/Fedora 29\/CentOS 7<\/a><\/p>\n\n\n\n<p>That is all on how to install and configure Logstash 7 on on Fedora 30\/Fedora 29\/CentOS 7.<\/p>\n\n\n\n<p>Reference:<\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" aria-label=\"Getting Started with Logstash (opens in a new tab)\" href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/getting-started-with-logstash.html\" target=\"_blank\">Getting Started with Logstash<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Related Tutorials;<\/h3>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/install-and-configure-logstash-7-on-ubuntu-18-debian-9-8\/\" target=\"_blank\">Install and Configure Logstash 7 on Ubuntu 18\/Debian 9.8<\/a><\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/install-and-configure-filebeat-7-on-ubuntu-18-04-debian-9-8\/\" target=\"_blank\">Install and Configure Filebeat 7 on Ubuntu 18.04\/Debian 9.8<\/a><\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/install-elastic-stack-7-on-ubuntu-18-04-debian-9-8\/\" target=\"_blank\">Install Elastic Stack 7 on Ubuntu 18.04\/Debian 9.8<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-elasticsearch-7-x-on-ubuntu-18-04-debian-9-8\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Elasticsearch 7.x on Ubuntu 18.04\/Debian 9.8<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This guide will focus on how to install Logstash 7 on Fedora 30\/Fedora 29\/CentOS 7 as a continuation of our guide on how to setup<\/p>\n","protected":false},"author":1,"featured_media":8883,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[72,910],"tags":[88,912,913,289,924,3563,3562,921,1019],"class_list":["post-3448","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-monitoring","category-elastic-stack","tag-centos-7","tag-elastic-stack","tag-elk","tag-fedora-29","tag-fedora-30","tag-install-logstash-centos","tag-install-logstash-fedora","tag-logstash","tag-logstash-7","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/3448"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=3448"}],"version-history":[{"count":12,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/3448\/revisions"}],"predecessor-version":[{"id":21153,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/3448\/revisions\/21153"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/8883"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=3448"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=3448"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=3448"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}