{"id":6957,"date":"2020-08-19T18:58:15","date_gmt":"2020-08-19T15:58:15","guid":{"rendered":"https:\/\/kifarunix.com\/?p=6957"},"modified":"2024-03-14T22:26:07","modified_gmt":"2024-03-14T19:26:07","slug":"setup-kibana-elasticsearch-and-fluentd-on-centos-8","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/setup-kibana-elasticsearch-and-fluentd-on-centos-8\/","title":{"rendered":"Setup Kibana Elasticsearch and Fluentd on CentOS 8"},"content":{"rendered":"\n<p>Hello there. In this tutorial, you will learn how to setup Kibana Elasticsearch and Fluentd on CentOS 8. Normally, you would setup Elasticsearch with Logstash, Kibana and beats. But in this setup, we will see how Fluentd can be used instead of Logstash and Beats to collect and ship logs to Elasticsearch, a search and analytics engine. So, what is Fluentd? <a href=\"https:\/\/www.fluentd.org\/architecture\" target=\"_blank\" rel=\"noreferrer noopener\">Fluentd<\/a> <em>&#8220;is an open source data collector for unified logging layer&#8221;<\/em>. It can act as a log <code>aggregator<\/code> (sits on the same server as Elasticsearch for example) and as a log <code>forwarder<\/code> (collecting logs from the nodes being monitored).<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1208\" height=\"451\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/09\/efk.png\" alt=\"\" class=\"wp-image-6975\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/09\/efk.png?v=1600594078 1208w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/09\/efk-768x287.png?v=1600594078 768w\" sizes=\"(max-width: 1208px) 100vw, 1208px\" \/><\/figure>\n\n\n\n<p>Below are the the key features of Fluentd.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Provides unified logging with JSON<\/strong>: Fluentd attempts to structure collected data as JSON thus allowing it&nbsp;to&nbsp;<strong>unify<\/strong>&nbsp;all facets of processing log data: collecting, filtering, buffering, and outputting logs across&nbsp;<strong>multiple sources and destinations<\/strong>. This makes it easy for the data processors to process the data,<\/li>\n\n\n\n<li><strong>Supports a pluggable architecture<\/strong>: This makes it easy for the community to extend the functionality of the Fluentd as they can develop any custom plugins to collect their logs.<\/li>\n\n\n\n<li><strong>Consumes minimum system resources<\/strong>: Fluentd requires very little system resources with the vanilla version requiring between 30-40 MB of memory and can process 13,000 events\/second\/core. There also exists a Fluentd lightweight forwarder called <a href=\"http:\/\/fluentbit.io\/\" target=\"_blank\" rel=\"noreferrer noopener\">Fluent Bit<\/a>.<\/li>\n\n\n\n<li><strong>Built-in Reliability<\/strong>: Fluentd supports memory and file-based buffering to prevent inter-data node loss. Fluentd also supports robust failover and can be set up for high availability.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Install and Configure Kibana, Elasticsearch and Fluentd<\/h2>\n\n\n\n<p>In order to setup Kibana, Elasticsearch and Fluentd, we will install and configure each component separately as follows.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"elk-stack-repo-centos-8\"><a href=\"#elk-stack-repo-centos-8\">Creating Elastic Stack Repository on CentOS 8<\/a><\/h3>\n\n\n\n<p>Run the command below to create Elastic Stack version 7.x repo on CentOS 8.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat &gt; \/etc\/yum.repos.d\/elasticstack.repo &lt;&lt; EOL\n&#91;elasticstack]\nname=Elastic repository for 7.x packages\nbaseurl=https:\/\/artifacts.elastic.co\/packages\/7.x\/yum\ngpgcheck=1\ngpgkey=https:\/\/artifacts.elastic.co\/GPG-KEY-elasticsearch\nenabled=1\nautorefresh=1\ntype=rpm-md\nEOL<\/code><\/pre>\n\n\n\n<p>Run system package update.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>dnf update<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-elasticsearch-on-centos-8\"><a href=\"#install-elasticsearch-on-centos-8\">Install Elasticsearch on CentOS 8<\/a><\/h3>\n\n\n\n<p>Install Elasticsearch on CentOS 8 from the Elastic repos;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>dnf install elasticsearch<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Configuring Elasticsearch<\/h4>\n\n\n\n<p>Out of the box, Elasticsearch works well with the default configuration options. In this setup, we will make a few changes as per&nbsp;<a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/7.5\/important-settings.html\" target=\"_blank\" rel=\"noreferrer noopener\">Important Elasticsearch Configurations<\/a>.<\/p>\n\n\n\n<p>Set the Elasticsearch bind address to a specific system IP if you need to enable remote access either from Kibana.&nbsp;<strong>Replace the IP, 192.168.56.154, with your appropriate server IP address<\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>sed -i 's\/#network.host: 192.168.0.1\/network.host: 192.168.56.154\/' \/etc\/elasticsearch\/elasticsearch.yml<\/code><\/pre>\n\n\n\n<p>You can as well leave the default settings to only allow local access to Elasticsearch.<\/p>\n\n\n\n<p>When configured to listen on a non-loopback interface, Elasticsearch expects to join a&nbsp;<a rel=\"noreferrer noopener\" href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/7.5\/cluster.name.html\" target=\"_blank\">cluster<\/a>. But since we are setting up a single node Elastic Stack, you need to specify in the ES configuration that this is a single node setup, by entering the line,&nbsp;<strong><code>discovery.type: single-node<\/code><\/strong>, under discovery configuration options. However, you can skip this if your ES is listening on a loopback interface.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/elasticsearch\/elasticsearch.yml<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code># --------------------------------- Discovery ----------------------------------\n#\n# Pass an initial list of hosts to perform discovery when this node is started:\n# The default list of hosts is [\"127.0.0.1\", \"[::1]\"]\n#\n#discovery.seed_hosts: [\"host1\", \"host2\"]\n#\n# Bootstrap the cluster using an initial set of master-eligible nodes:\n#\n#cluster.initial_master_nodes: [\"node-1\", \"node-2\"]\n# Single Node Discovery\n<strong>discovery.type: single-node<\/strong><\/code><\/pre>\n\n\n\n<p>Next, configure JVM heap size to no more than half the size of your memory. In this case, our test server has 2G RAM and the heap size is set to 512M for both maximum and minimum sizes.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/elasticsearch\/jvm.options<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>...\n################################################################\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n<strong>-Xms512m\n-Xmx512m<\/strong>\n...<\/code><\/pre>\n\n\n\n<p>Start and enable ES to run on system boot.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl daemon-reload\nsystemctl enable --now elasticsearch<\/code><\/pre>\n\n\n\n<p>Verify that Elasticsearch is running as expected.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>curl -XGET 192.168.56.154:9200<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"name\" : \"centos8.kifarunix-demo.com\",\n  \"cluster_name\" : \"elasticsearch\",\n  \"cluster_uuid\" : \"rVPJG0k9TKK9-I-mVmoV_Q\",\n  \"version\" : {\n    \"number\" : \"7.9.1\",\n    \"build_flavor\" : \"default\",\n    \"build_type\" : \"rpm\",\n    \"build_hash\" : \"083627f112ba94dffc1232e8b42b73492789ef91\",\n    \"build_date\" : \"2020-09-01T21:22:21.964974Z\",\n    \"build_snapshot\" : false,\n    \"lucene_version\" : \"8.6.2\",\n    \"minimum_wire_compatibility_version\" : \"6.8.0\",\n    \"minimum_index_compatibility_version\" : \"6.0.0-beta1\"\n  },\n  \"tagline\" : \"You Know, for Search\"\n}<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-kibana-centos-8\"><a href=\"#install-kibana-centos-8\">Install Kibana on CentOS 8<\/a><\/h3>\n\n\n\n<p>The next Elastic Stack component to install is Kabana. Since we already created the Elastic Stack repos, you can simply run the command below to install it.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>yum install kibana<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Configuring Kibana<\/h4>\n\n\n\n<p>To begin with, you need to configure Kibana to allow remote access. By default, it allows local access on port 5601\/tcp. Hence, open the Kibana configuration file for editing and uncomment and change the following lines;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/kibana\/kibana.yml<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>...\n<strong>#server.port: 5601<\/strong>\n...\n# To allow connections from remote users, set this parameter to a non-loopback address.\n<strong>#server.host: \"localhost\"<\/strong>\n...\n# The URLs of the Elasticsearch instances to use for all your queries.\n<strong>#elasticsearch.hosts: [\"http:\/\/localhost:9200\"]<\/strong><\/code><\/pre>\n\n\n\n<p>Such that it look like as shown below:<\/p>\n\n\n\n<p><strong>Replace the IP addresses of Kibana and Elasticsearch accordingly. Note that in this demo, All Elastic Stack components are running on the same host.<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>...\n<strong>server.port: 5601<\/strong>\n...\n# To allow connections from remote users, set this parameter to a non-loopback address.\n<strong>server.host: \"192.168.56.154\"<\/strong>\n...\n# The URLs of the Elasticsearch instances to use for all your queries.\n<strong>elasticsearch.hosts: [\"http:\/\/192.168.56.154:9200\"]<\/strong><\/code><\/pre>\n\n\n\n<p>Start and enable Kibana to run on system boot.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl enable --now kibana<\/code><\/pre>\n\n\n\n<p>Open Kibana Port on FirewallD, if it is running;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>firewall-cmd --add-port=5601\/tcp --permanent<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>firewall-cmd --reload<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Accessing Kibana Interface<\/h4>\n\n\n\n<p>You can now access Kibana from your browser by using the URL,&nbsp;<code>http:\/\/kibana-server-hostname-OR-IP:5601<\/code>.<\/p>\n\n\n\n<p>On Kibana web interface, you can choose to try sample data since we do not have any data being sent to Elasticsearch yet. You can as well choose to explore your own data, of course after sending data to ES.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Install and Configure Fluentd on CentOS 8<\/h3>\n\n\n\n<p>Next, install and configure Fluentd to collect logs into Elasticsearch. On the same server running Elasticsearch, we will install Fluentd aggregator so it can receive logs from the end point nodes using the Fluentd forwarder.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Prereqs of Installing Fluentd<\/h4>\n\n\n\n<p>There are a number of requirements that you need to consider while setting up Fluentd.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ensure that your system time is synchronized with the up-to-date time server (NTP) so that the logs can have the correct event timestamp entries.<\/li>\n\n\n\n<li>Increase the maximum number of open file descriptors. By default, the max number of open file descriptors is set to 1024;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ulimit -n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>1024<\/code><\/pre>\n\n\n\n<p>You can set the max number to 65536 by editing the limits.conf file and adding the lines below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/security\/limits.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>root            soft    nofile  65536\nroot            hard    nofile  65536\n*               soft    nofile  65536\n*               hard    nofile  65536<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Next, if you gonna have several Fluentd nodes with the high load being expected, you need to adjust some Network kernel parameter.<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>cat &gt;&gt; \/etc\/sysctl.conf &lt;&lt; 'EOL'\nnet.core.somaxconn = 1024\nnet.core.netdev_max_backlog = 5000\nnet.core.rmem_max = 16777216\nnet.core.wmem_max = 16777216\nnet.ipv4.tcp_wmem = 4096 12582912 16777216\nnet.ipv4.tcp_rmem = 4096 12582912 16777216\nnet.ipv4.tcp_max_syn_backlog = 8096\nnet.ipv4.tcp_slow_start_after_idle = 0\nnet.ipv4.tcp_tw_reuse = 1\nnet.ipv4.ip_local_port_range = 10240 65535\nEOL<\/code><\/pre>\n\n\n\n<p>Update the changes by rebooting your system or by just running the command below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>sysctl -p<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><a href=\"#install-fluentd-centos-8\">Install Fluentd Aggregator on CentOS 8<\/a><\/h3>\n\n\n\n<p>Fluentd installation has been made easier through the use of the <code><strong>td-agent<\/strong><\/code> (Treasure Agent), an RPM package that provides a stable distribution of Fluentd based data collector and is managed and maintained by <a href=\"http:\/\/www.treasuredata.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">Treasure Data, Inc<\/a>.<\/p>\n\n\n\n<p>To install <code><strong>td-agent<\/strong><\/code> package, run the command below to download and execute a script that will create the td-agent RPM repository and installs td-agent on CentOS 8.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>dnf install curl<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>curl -L https:\/\/toolbelt.treasuredata.com\/sh\/install-redhat-td-agent4.sh | sh<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Running Fluentd td-agent on CentOS 8<\/h4>\n\n\n\n<p>When installed, td-agent installs a systemd service unit for managing it. You can therefore start and enable it to run on system boot by executing the command below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl enable --now td-agent<\/code><\/pre>\n\n\n\n<p>To check the status;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl status td-agent<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>\u25cf td-agent.service - td-agent: Fluentd based data collector for Treasure Data\n   Loaded: loaded (\/usr\/lib\/systemd\/system\/td-agent.service; enabled; vendor preset: disabled)\n   Active: active (running) since Fri 2020-09-18 22:09:40 EAT; 29s ago\n     Docs: https:\/\/docs.treasuredata.com\/articles\/td-agent\n  Process: 2543 ExecStart=\/opt\/td-agent\/bin\/fluentd --log $TD_AGENT_LOG_FILE --daemon \/var\/run\/td-agent\/td-agent.pid $TD_AGENT_OPTIONS (code=exited, status=0\/SUCCESS)\n Main PID: 2549 (fluentd)\n    Tasks: 9 (limit: 5027)\n   Memory: 89.4M\n   CGroup: \/system.slice\/td-agent.service\n           \u251c\u25002549 \/opt\/td-agent\/bin\/ruby \/opt\/td-agent\/bin\/fluentd --log \/var\/log\/td-agent\/td-agent.log --daemon \/var\/run\/td-agent\/td-agent.pid\n           \u2514\u25002552 \/opt\/td-agent\/bin\/ruby -Eascii-8bit:ascii-8bit \/opt\/td-agent\/bin\/fluentd --log \/var\/log\/td-agent\/td-agent.log --daemon \/var\/run\/td-agent\/td-agent.pid --u&gt;\n\nSep 18 22:09:38 centos8.kifarunix-demo.com systemd&#91;1]: Starting td-agent: Fluentd based data collector for Treasure Data...\nSep 18 22:09:40 centos8.kifarunix-demo.com systemd&#91;1]: Started td-agent: Fluentd based data collector for Treasure Data.<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Installing Fluentd Elasticsearch Plugin<\/h4>\n\n\n\n<p>In this setup, we will use Elasticsearch as our search and analytics engine and hence, all the data collected by the Fluentd. As such, install Fluentd Elasticsearch plugin.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>td-agent-gem install fluent-plugin-elasticsearch<\/code><\/pre>\n\n\n\n<p>Also, if you are gonna sent the logs to Fluentd via Internet, you need to install <code><strong>secure_forward<\/strong><\/code> Fluentd output plugin that sends data securely.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>td-agent-gem install fluent-plugin-secure-forward<\/code><\/pre>\n\n\n\n<p>You can see a whole list of Fluentd plugins on <a href=\"https:\/\/www.fluentd.org\/plugins\" target=\"_blank\" rel=\"noreferrer noopener\">list of plugins by category page<\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configure-fluentd-aggregator-centos-8\"><a href=\"#configure-fluentd-aggregator-centos-8\">Configuring Fluentd Aggregator on CentOS 8<\/a><\/h3>\n\n\n\n<p>The default configuration file for Fluentd installed via the td-agent RPM, is <code><strong>\/etc\/td-agent\/td-agent.conf<\/strong><\/code>. The configuration file consists of the following directives:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong><code>source<\/code><\/strong> directives determine the input sources<\/li>\n\n\n\n<li><strong><code>match<\/code><\/strong> directives determine the output destinations<\/li>\n\n\n\n<li><strong><code>filter<\/code><\/strong> directives determine the event processing pipelines<\/li>\n\n\n\n<li><strong><code>system<\/code><\/strong> directives set system wide configuration<\/li>\n\n\n\n<li><strong><code>label<\/code><\/strong> directives group the output and filter for internal routing<\/li>\n\n\n\n<li><strong><code>@include<\/code><\/strong> directives include other files<\/li>\n<\/ol>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-fluentd-input-plugin\"><a href=\"#configure-fluentd-input-plugin\">Configure Fluentd Aggregator Input Plugins<\/a><\/h4>\n\n\n\n<p>First off, there are quite a number of <a href=\"https:\/\/docs.fluentd.org\/input\" target=\"_blank\" rel=\"noreferrer noopener\">input plugins<\/a> which Fluentd aggregator can use to accept\/receive data from the Fluentd forwarders.<\/p>\n\n\n\n<p>In this setup, we are receiving logs via the Fluentd the <a href=\"https:\/\/docs.fluentd.org\/input\/forward\" target=\"_blank\" rel=\"noreferrer noopener\"><code>forward<\/code> input plugin<\/a>. <code>forward<\/code> input plugin listens to a TCP socket to receive the event stream. It also listens to a UDP socket to receive heartbeat messages. The default port for Fluentd forward plugin is 24224.<\/p>\n\n\n\n<p>Create a configuration backup;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>cp \/etc\/td-agent\/td-agent.conf{,.old}<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/td-agent\/td-agent.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>...\n&lt;source&gt;\n  @type forward\n  port 24224\n  bind 192.168.60.6\n&lt;\/source&gt;\n...<\/code><\/pre>\n\n\n\n<p>Be sure to open this port on firewall.<\/p>\n\n\n\n<pre id=\"block-7e3eb87c-8ae3-4d93-b21c-ae400f8dc30b\" class=\"wp-block-preformatted\">firewall-cmd --add-port=24224\/{tcp,udp} --permanent<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>firewall-cmd --reload<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-fluentd-output-plugin\"><a href=\"#configure-fluentd-output-plugin\">Configure Fluentd Aggregator Output Plugins<\/a><\/h4>\n\n\n\n<p>Configure Fluentd to sent data to Elasticsearch via the <a href=\"https:\/\/docs.fluentd.org\/output\/elasticsearch\" target=\"_blank\" rel=\"noreferrer noopener\">elasticsearch Fluentd output<\/a> plugin.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/td-agent\/td-agent.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>####\n## Output descriptions:\n##\n&lt;match *.**&gt;\n  @type elasticsearch\n  host 192.168.60.6\n  port 9200\n  logstash_format true\n  logstash_prefix fluentd\n  enable_ilm true\n  index_date_pattern \"now\/m{yyyy.mm}\"\n  flush_interval 10s\n&lt;\/match&gt;\n\n####\n## Source descriptions:\n##\n&lt;source&gt;\n  @type forward\n  port 24224\n  bind 192.168.60.6\n&lt;\/source&gt;<\/code><\/pre>\n\n\n\n<p>The match directive wildcard is explained on the <a href=\"https:\/\/docs.fluentd.org\/configuration\/config-file#wildcards-expansions-and-other-tips\" target=\"_blank\" rel=\"noreferrer noopener\">File syntax page<\/a>.<\/p>\n\n\n\n<p>That is our modified Fluentd aggregator configuration file. You can adjust it to meet your requirements.<\/p>\n\n\n\n<p>Restart Fluentd td-agent;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl restart td-agent<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"install-fluentd-agents-on-remote-nodes\"><a href=\"#install-fluentd-agents-on-remote-nodes\">Install Fluentd Forwarder on Remote Nodes<\/a><\/h2>\n\n\n\n<p>Now that the Kibana, Elasticsearch and Fluentd Aggregator is setup and ready to receive collected data from the remote end points, proceed to install the Fluentd forwarders to push the logs to the the Fluentd aggregator.<\/p>\n\n\n\n<p>In this setup, we are using a remote CentOS 8 as the remote end point to collect logs from.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>curl -L https:\/\/toolbelt.treasuredata.com\/sh\/install-redhat-td-agent4.sh | sh<\/code><\/pre>\n\n\n\n<p>Ubuntu 20.04;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>curl -L https:\/\/toolbelt.treasuredata.com\/sh\/install-ubuntu-focal-td-agent4.sh | sh<\/code><\/pre>\n\n\n\n<p>Ubuntu 18.04<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>curl -L https:\/\/toolbelt.treasuredata.com\/sh\/install-ubuntu-bionic-td-agent4.sh | sh<\/code><\/pre>\n\n\n\n<p>For more systems installation, refer to <a href=\"https:\/\/docs.fluentd.org\/installation\" target=\"_blank\" rel=\"noreferrer noopener\">Fluentd installation<\/a> page.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Configure Fluentd Forwarder to Ship Logs to Fluentd Aggregator<\/h3>\n\n\n\n<p>Similarly, make a copy of the configuration file.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>cp \/etc\/td-agent\/td-agent.conf{,.old}<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Configure Fluentd Forwarder Input and Output<\/h4>\n\n\n\n<p>In this setup, just as an example, we will collect the system authentication logs, <code><strong>\/var\/log\/secure<\/strong><\/code>, from a remove CentOS 8 system.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/td-agent\/td-agent.conf<\/code><\/pre>\n\n\n\n<p>We will use the <a href=\"https:\/\/docs.fluentd.org\/input\/tail\" target=\"_blank\" rel=\"noreferrer noopener\">tail input plugin<\/a> to read the log files by tailing them. Therefore, our input configuration looks like;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&lt;source&gt;\n  @type tail\n  path \/var\/log\/secure\n  pos_file \/var\/log\/td-agent\/secure.pos\n  tag ssh.auth\n  &lt;parse&gt;\n    @type syslog\n  &lt;\/parse&gt;\n&lt;\/source&gt;<\/code><\/pre>\n\n\n\n<p>Next, configure how logs are shipped to Fluentd aggregator. In this setup, we utilize the <a href=\"https:\/\/docs.fluentd.org\/output\/forward\" target=\"_blank\" rel=\"noreferrer noopener\">forward output plugin<\/a> to sent the data to our log manager server running Elasticsearch, Kibana and Fluentd aggregator, listening on port 24224 TCP\/UDP.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&lt;match pattern&gt;\n  @type forward\n  send_timeout 60s\n  recover_wait 10s\n  hard_timeout 60s\n\n  &lt;server&gt;\n    name log_mgr\n    host 192.168.60.6\n    port 24224\n    weight 60\n  &lt;\/server&gt;\n&lt;\/match&gt;<\/code><\/pre>\n\n\n\n<p>In general, our Fluentd forwarder configuration looks like;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>####\n## Output descriptions:\n##\n&lt;match *.**&gt;\n  @type forward\n  send_timeout 60s\n  recover_wait 10s\n  hard_timeout 60s\n\n  &lt;server&gt;\n    name log_mgr\n    host 192.168.60.6\n    port 24224\n    weight 60\n  &lt;\/server&gt;\n&lt;\/match&gt;\n####\n## Source descriptions:\n##\n&lt;source&gt;\n  @type tail\n  path \/var\/log\/secure\n  pos_file \/var\/log\/td-agent\/secure.pos\n  tag ssh.auth\n  &lt;parse&gt;\n    @type syslog\n  &lt;\/parse&gt;\n&lt;\/source&gt;<\/code><\/pre>\n\n\n\n<p>Save and exit the configuration file.<\/p>\n\n\n\n<p>Next, give Fluentd read access to the authentication logs file or any log file being collected. By default, only root can read the logs;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ls -alh \/var\/log\/secure<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>-rw-------. 1 root root 14K Sep 19 00:33 \/var\/log\/secure<\/code><\/pre>\n\n\n\n<p>To ensure that Fluentd can read this log file, give the group and world read permissions;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>chmod og+r \/var\/log\/secure<\/code><\/pre>\n\n\n\n<p>The permissions should now look like;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ll \/var\/log\/secure<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>-rw-r--r--. 1 root root 13708 Sep 19 00:33 \/var\/log\/secure<\/code><\/pre>\n\n\n\n<p>Next, start and enable Fluentd Forwarder to run on system boot;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl enable --now td-agent<\/code><\/pre>\n\n\n\n<p>Check the status;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl status td-agent<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>\u25cf td-agent.service - td-agent: Fluentd based data collector for Treasure Data\n   Loaded: loaded (\/usr\/lib\/systemd\/system\/td-agent.service; enabled; vendor preset: disabled)\n   Active: active (running) since Sat 2020-09-19 01:23:40 EAT; 29s ago\n     Docs: https:\/\/docs.treasuredata.com\/articles\/td-agent\n  Process: 3163 ExecStart=\/opt\/td-agent\/bin\/fluentd --log $TD_AGENT_LOG_FILE --daemon \/var\/run\/td-agent\/td-agent.pid $TD_AGENT_OPTIONS (code=exited, status=0\/SUCCESS)\n Main PID: 3169 (fluentd)\n    Tasks: 8 (limit: 11476)\n   Memory: 71.0M\n   CGroup: \/system.slice\/td-agent.service\n           \u251c\u25003169 \/opt\/td-agent\/bin\/ruby \/opt\/td-agent\/bin\/fluentd --log \/var\/log\/td-agent\/td-agent.log --daemon \/var\/run\/td-agent\/td-agent.pid\n           \u2514\u25003172 \/opt\/td-agent\/bin\/ruby -Eascii-8bit:ascii-8bit \/opt\/td-agent\/bin\/fluentd --log \/var\/log\/td-agent\/td-agent.log --daemon \/var\/run\/td-agent\/td-agent.pid --u&gt;\n\nSep 19 01:23:39 localrepo.kifarunix-demo.com systemd&#91;1]: Starting td-agent: Fluentd based data collector for Treasure Data...\nSep 19 01:23:40 localrepo.kifarunix-demo.com systemd&#91;1]: Started td-agent: Fluentd based data collector for Treasure Data.<\/code><\/pre>\n\n\n\n<p>If you tail the Fluentd forwarder logs, you should see that it starts to read the log file;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>tail -f \/var\/log\/td-agent\/td-agent.log<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>  &lt;\/source&gt;\n&lt;\/ROOT&gt;\n2020-09-19 01:23:40 +0300 &#91;info]: starting fluentd-1.11.2 pid=3163 ruby=\"2.7.1\"\n2020-09-19 01:23:40 +0300 &#91;info]: spawn command to main:  cmdline=&#91;\"\/opt\/td-agent\/bin\/ruby\", \"-Eascii-8bit:ascii-8bit\", \"\/opt\/td-agent\/bin\/fluentd\", \"--log\", \"\/var\/log\/td-agent\/td-agent.log\", \"--daemon\", \"\/var\/run\/td-agent\/td-agent.pid\", \"--under-supervisor\"]\n2020-09-19 01:23:41 +0300 &#91;info]: adding match pattern=\"pattern\" type=\"forward\"\n2020-09-19 01:23:41 +0300 &#91;info]: #0 adding forwarding server 'log_mgr' host=\"192.168.60.6\" port=24224 weight=60 plugin_id=\"object:71c\"\n2020-09-19 01:23:41 +0300 &#91;info]: adding source type=\"tail\"\n2020-09-19 01:23:41 +0300 &#91;info]: #0 starting fluentd worker pid=3172 ppid=3169 worker=0\n2020-09-19 01:23:41 +0300 &#91;info]: #0 following tail of \/var\/log\/secure\n2020-09-19 01:23:41 +0300 &#91;info]: #0 fluentd worker is now running worker=0\n...<\/code><\/pre>\n\n\n\n<p>On the server running Elasticsearch, Kibana and Fluentd aggregator, you can check if any data is being received on the port 24224;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>tcpdump -i enp0s8 -nn dst port 24224<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\nlistening on enp0s8, link-type EN10MB (Ethernet), capture size 262144 bytes\n01:28:37.183634 IP 192.168.60.5.39452 &gt; 192.168.60.6.24224: Flags &#91;S], seq 2062965426, win 29200, options &#91;mss 1460,sackOK,TS val 3228636873 ecr 0,nop,wscale 7], length 0\n01:28:37.184740 IP 192.168.60.5.39452 &gt; 192.168.60.6.24224: Flags &#91;.], ack 2675674893, win 229, options &#91;nop,nop,TS val 3228636875 ecr 354613533], length 0\n01:28:37.185145 IP 192.168.60.5.39452 &gt; 192.168.60.6.24224: Flags &#91;F.], seq 0, ack 1, win 229, options &#91;nop,nop,TS val 3228636875 ecr 354613533], length 0\n01:28:38.181546 IP 192.168.60.5.39454 &gt; 192.168.60.6.24224: Flags &#91;S], seq 1970844825, win 29200, options &#91;mss 1460,sackOK,TS val 3228637794 ecr 0,nop,wscale 7], length 0\n01:28:38.182649 IP 192.168.60.5.39454 &gt; 192.168.60.6.24224: Flags &#91;.], ack 2454001874, win 229, options &#91;nop,nop,TS val 3228637796 ecr 354614454], length 0\n...<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Check Available Indices on Elasticsearch<\/h4>\n\n\n\n<p>Perform failed and successful SSH authentication to your host running Fluentd forwarder. After that, check if your Elasticsearch index has been created. In this setup, we set our index prefix to fluentd, <code><strong>logstash_prefix fluentd<\/strong><\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>curl -XGET http:\/\/192.168.60.6:9200\/_cat\/indices?v<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>health status index                          uuid                   pri rep docs.count docs.deleted store.size pri.store.size\ngreen  open   .apm-custom-link               kuDD9tq0RAapIEtF4k79zw   1   0          0            0       208b           208b\ngreen  open   .kibana-event-log-7.9.1-000001 gJ6tr6p5TCWmu1GhUNaD4A   1   0          9            0     48.4kb         48.4kb\ngreen  open   .kibana_task_manager_1         T-dC9DFNTsy2uoYAJmvDtg   1   0          6           20    167.1kb        167.1kb\ngreen  open   .apm-agent-configuration       lNCadKowT3eIg_heAruB-w   1   0          0            0       208b           208b\n<strong>yellow open   fluentd-2020.09.19             nWU0KLe2Rv-T5eMD53kcoA   1   1         30            0     36.2kb         36.2kb\n<\/strong>green  open   .async-search                  C1gXukCuQIe5grCFpLwxaQ   1   0          0            0       231b           231b\ngreen  open   .kibana_1                      Mw6PD83xT1KksRqAvO1BKg   1   0         22            5     10.4mb         10.4mb<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Create Fluentd Kibana Index<\/h4>\n\n\n\n<p>Once you confirm that the data has been received on Elasticsearch and written to your index, navigate to Kibana web interface, <code><strong>http:\/\/server-IP-or-hostname:5601<\/strong><\/code>, and create the index.<\/p>\n\n\n\n<p>Click on&nbsp;<strong>Management tab (on the left side panel) &gt; Kibana&gt; Index Patterns &gt; Create Index Pattern<\/strong>. Enter the wildcard for your index name.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1867\" height=\"768\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/09\/fluentd-index.png\" alt=\"Setup Kibana Elasticsearch and Fluentd on CentOS 8\" class=\"wp-image-6969\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/09\/fluentd-index.png?v=1600529992 1867w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/09\/fluentd-index-768x316.png?v=1600529992 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/09\/fluentd-index-1536x632.png?v=1600529992 1536w\" sizes=\"(max-width: 1867px) 100vw, 1867px\" \/><\/figure>\n\n\n\n<p>In the next step, select <strong>timestamp<\/strong> as the time filter then click&nbsp;<strong>Create Index pattern<\/strong>&nbsp;to create your index pattern.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"view-fluentd-data-kibana\"><a href=\"#view-fluentd-data-kibana\">Viewing Fluentd Data on Kibana<\/a><\/h4>\n\n\n\n<p>Once you have created Fluentd Kibana index, you can now view your event data on Kibana by clicking on the <strong>Discover<\/strong> tab on the left pane. Expand your time range accordingly.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1895\" height=\"944\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/09\/kibana-fluentd-data.png\" alt=\"\" class=\"wp-image-6970\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/09\/kibana-fluentd-data.png?v=1600530002 1895w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/09\/kibana-fluentd-data-768x383.png?v=1600530002 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/09\/kibana-fluentd-data-1536x765.png?v=1600530002 1536w\" sizes=\"(max-width: 1895px) 100vw, 1895px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Further Reading<\/h3>\n\n\n\n<p><a href=\"https:\/\/docs.fluentd.org\/installation\" target=\"_blank\" rel=\"noopener\">Fluentd Installation<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/docs.fluentd.org\/configuration\" target=\"_blank\" rel=\"noreferrer noopener\">Fluentd Installation<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/github.com\/uken\/fluent-plugin-elasticsearch\" target=\"_blank\" rel=\"noreferrer noopener\">Fluentd-Elasticsearch Plugin Reference<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Related Tutorials<\/h3>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/installing-elk-stack-on-centos-8\/\" target=\"_blank\" rel=\"noreferrer noopener\">Installing ELK Stack on CentOS 8<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-elastic-stack-7-on-fedora-30-fedora-29-centos-7\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Elastic Stack 7 on Fedora 30\/Fedora 29\/CentOS 7<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-elastic-stack-7-on-ubuntu-18-04-debian-9-8\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Elastic Stack 7 on Ubuntu 18.04\/Debian 9.8<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-icinga-2-and-icinga-web-2-on-ubuntu-20-04\/\">Install Icinga 2 and Icinga Web 2 on Ubuntu 20.04<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hello there. In this tutorial, you will learn how to setup Kibana Elasticsearch and Fluentd on CentOS 8. Normally, you would setup Elasticsearch with Logstash,<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121,910,72],"tags":[2168,2165,2166,2164,2167],"class_list":["post-6957","post","type-post","status-publish","format-standard","hentry","category-howtos","category-elastic-stack","category-monitoring","tag-fluentd-with-kibana-and-elasticsearch","tag-install-elasticsearch-centos-8","tag-install-fluentd-centos-8","tag-install-kibana-centos-8","tag-install-kibana-elasticsearch-and-fluentd-on-centos-8","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/6957"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=6957"}],"version-history":[{"count":10,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/6957\/revisions"}],"predecessor-version":[{"id":21472,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/6957\/revisions\/21472"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=6957"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=6957"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=6957"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}