{"id":2775,"date":"2019-05-02T13:36:08","date_gmt":"2019-05-02T10:36:08","guid":{"rendered":"https:\/\/kifarunix.com\/?p=2775"},"modified":"2021-08-28T22:34:07","modified_gmt":"2021-08-28T19:34:07","slug":"install-and-configure-filebeat-7-on-ubuntu-18-04-debian-9-8","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/install-and-configure-filebeat-7-on-ubuntu-18-04-debian-9-8\/","title":{"rendered":"Install and Configure Filebeat 7 on Ubuntu 18.04\/Debian 9.8"},"content":{"rendered":"\n

This guide discusses how to install and configure Filebeat 7 on Ubuntu 18.04\/Debian 9.8. This comes as the last part of our guide on how to setup Elastic Stack on Ubuntu 18.04\/Debian 9.8. We have covered the installation of Elasticsearch, Kibana and Logstash in our previous guides. See the Links below;<\/p>\n\n\n\n

Install Elasticsearch 7.x on Ubuntu 18.04\/Debian 9.8<\/a><\/p>\n\n\n\n

Install Kibana 7 on Ubuntu 18.04\/Debian 9.8<\/a><\/p>\n\n\n\n

Install and Configure Logstash 7 on Ubuntu 18\/Debian 9.8<\/a><\/p>\n\n\n\n

So what is Filebeat? Filebeat is a lightweight shipper for collecting, forwarding and centralizing event log data. It is installed as an agent on the servers you are collecting logs from. It can forward the logs it is collecting to either Elasticsearch or Logstash for indexing.<\/p>\n\n\n\n

There are other types of Beats as described here<\/a>.<\/p>\n\n\n\n

In this guide, Filebeat is configured to forward event logs, SSH authentication events to Logstash.<\/p>\n\n\n\n

Install and Configure Filebeat 7 on Ubuntu 18.04\/Debian 9.8<\/h2>\n\n\n\n

Add Elastic Stack 7 APT Repository<\/h3>\n\n\n\n

Filebeat can installed using APT package manager by creating the Elastic Stack repos on the server you want to collect logs from.<\/p>\n\n\n\n

Run the command below to download and install the Elastic Stack Repo public signing key<\/p>\n\n\n\n

wget -qO - https:\/\/artifacts.elastic.co\/GPG-KEY-elasticsearch | sudo apt-key add -<\/code><\/pre>\n\n\n\n
Create the Elastic Stack 7 Apt repository.<\/p>\n\n\n\n
echo \"deb https:\/\/artifacts.elastic.co\/packages\/7.x\/apt stable main\" | sudo tee \/etc\/apt\/sources.list.d\/elastic-7.x.list<\/code><\/pre>\n\n\n\n
Install Filebeat 7.x<\/h3>\n\n\n\n
apt install apt-transport-https<\/code><\/pre>\n\n\n\n
apt update<\/code><\/pre>\n\n\n\n
apt install filebeat<\/code><\/pre>\n\n\n\n
Once the installation is done, you can verify the version by running;<\/p>\n\n\n\n
apt-cache policy filebeat<\/code><\/pre>\n\n\n\n
 filebeat:\n   Installed: 7.0.0\n   Candidate: 7.0.0\n   Version table:\n  *** 7.0.0 500\n         500 https:\/\/artifacts.elastic.co\/packages\/7.x\/apt stable\/main amd64 Packages\n         100 \/var\/lib\/dpkg\/status<\/code><\/pre>\n\n\n\n
Configure Filebeat 7<\/a><\/h2>\n\n\n\n
By default, Filebeat is set to sent data to Elasticsearc. In this guide, Logstash is configured to receive event data from Filebeat.<\/p>\n\n\n\n
Hence, open Filebeat configuration file, \/etc\/filebeat\/filebeat.yml<\/strong>, and under the Output sections, comment out Elasticsearch output and enable Logstash Output as shown below; <\/p>\n\n\n\n
vim \/etc\/filebeat\/filebeat.yml<\/code><\/pre>\n\n\n\n
\n...\n#================================ Outputs =====================================\n\n# Configure what output to use when sending the data collected by the beat.\n\n#-------------------------- Elasticsearch output ------------------------------\n# output.elasticsearch:\n  # Array of hosts to connect to.\n  # hosts: [\"localhost:9200\"]<\/strong>\n\n  # Optional protocol and basic auth credentials.\n  #protocol: \"https\"\n  #username: \"elastic\"\n  #password: \"changeme\"\n\n#----------------------------- Logstash output --------------------------------\noutput.logstash:\n  # The Logstash hosts\n  hosts: [\"192.168.0.101:5044\"]<\/strong>\n...\n<\/code><\/pre>\n\n\n\n
Testing Filebeat Output connection;<\/p>\n\n\n\n
systemctl stop filebeat<\/code><\/pre>\n\n\n\n
filebeat -e test output<\/code><\/pre>\n\n\n\n
\nlogstash: 192.168.0.101:5044...\n  connection...\n    parse host... OK\n    dns lookup... OK\n    addresses: 192.168.0.101\n    dial up... OK\n  TLS... WARN secure connection disabled\n  talk to server... OK\n<\/code><\/pre>\n\n\n\n
Testing the Config for any errors<\/p>\n\n\n\n
filebeat -e test config<\/code><\/pre>\n\n\n\n
If all is well, you should get, Config OK<\/code><\/strong> from the output.<\/p>\n\n\n\n
Enable Filebeat System Module<\/h3>\n\n\n\n
If you remember, our Logstash Filter was configured to parse system auth events. System module collects and parses logs created by the system logging service of common Unix\/Linux based distributions. This module is disabled by default. You can list the modules as shown below. From the output, no module is enabled.<\/p>\n\n\n\n
filebeat modules list<\/code><\/pre>\n\n\n\n
\nEnabled:\n\nDisabled:\napache\nauditd\nelasticsearch\nhaproxy\nicinga\niis\niptables\nkafka\nkibana\nlogstash\nmongodb\nmysql\nnginx\nosquery\npostgresql\nredis\nsanta\nsuricata\nsystem\ntraefik\nzeek\n<\/code><\/pre>\n\n\n\n
You can also check from the modules configuration file;<\/p>\n\n\n\n
ls \/etc\/filebeat\/modules.d\/<\/code><\/pre>\n\n\n\n
\napache.yml.disabled         kafka.yml.disabled     postgresql.yml.disabled\nauditd.yml.disabled         kibana.yml.disabled    redis.yml.disabled\nelasticsearch.yml.disabled  logstash.yml.disabled  santa.yml.disabled\nhaproxy.yml.disabled        mongodb.yml.disabled   suricata.yml.disabled\nicinga.yml.disabled         mysql.yml.disabled     system.yml.disabled\niis.yml.disabled            nginx.yml.disabled     traefik.yml.disabled\niptables.yml.disabled       osquery.yml.disabled   zeek.yml.disabled\n<\/code><\/pre>\n\n\n\n
To enable system module, run the command below;<\/p>\n\n\n\n
filebeat modules enable system<\/code><\/pre>\n\n\n\n
To verify that system module has been enabled;<\/p>\n\n\n\n
filebeat modules list<\/code><\/pre>\n\n\n\n
\nEnabled:\nsystem\n\nDisabled:\napache\nauditd\n...\n<\/code><\/pre>\n\n\n\n
This will remove the disabled<\/strong> suffix from the system module.<\/p>\n\n\n\n
You can further modify the system module to read only authentication logs. See var.paths<\/strong><\/p>\n\n\n\n
vim \/etc\/filebeat\/modules.d\/system.yml<\/code><\/pre>\n\n\n\n
\n- module: system\n  # Syslog\n  syslog:\n    enabled: false\n...\n  # Authorization logs\n  auth:\n    enabled: true\n\n    # Set custom paths for the log files. If left empty,\n    # Filebeat will choose the paths depending on your OS.\n    var.paths: [\"\/var\/log\/auth.log\"]\n...\n<\/code><\/pre>\n\n\n\n
Load the index template in Elasticsearch<\/h3>\n\n\n\n
in order to load the index template into Elasticsearch, there should be  connection to Elasticsearch. Since our guide is using Logstash as the output, we will load the template into Elasticsearch manually as shown below;<\/p>\n\n\n\n
Verify connection to Elasticsearch<\/h4>\n\n\n\n
telnet 192.168.0.101 9200<\/code><\/pre>\n\n\n\n
Trying 192.168.0.101...\nConnected to 192.168.0.101.\nEscape character is '^]'.<\/code><\/pre>\n\n\n\n
Load the Index Template<\/h4>\n\n\n\n
filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=[\"192.168.0.101:9200\"]'<\/code><\/pre>\n\n\n\n
You see the output.<\/p>\n\n\n\n
Index setup complete.<\/code><\/pre>\n\n\n\n
If the host doesn’t have direct connectivity to Elasticsearch, you can generate the index template, copy it to Elastic Stack Server and install it locally.<\/p>\n\n\n\n
To generate the template;<\/p>\n\n\n\n
filebeat export template > filebeat.template.json<\/code><\/pre>\n\n\n\n
To install the template on Elastic Stack server, copy it and run locally on Elastic Stack server.<\/p>\n\n\n\n
curl -XPUT -H 'Content-Type: application\/json' http:\/\/localhost:9200\/_template\/filebeat-7.0.1 -d@filebeat.template.json<\/code><\/pre>\n\n\n\n
Once you are done with that, start and enable Filebeat to run on system boot.<\/p>\n\n\n\n
systemctl start filebeat\nsystemctl enable filebeat<\/code><\/pre>\n\n\n\n
Verify Elasticsearch Data Reception<\/a><\/h3>\n\n\n\n
Next, perform successful and failed login<\/strong> on the system with Filebeat installed.<\/p>\n\n\n\n
After that login to Elastic Stack server and verify if Elasticsearch is receiving data.<\/p>\n\n\n\n
curl -X GET 192.168.0.101:9200\/_cat\/indices?v<\/code><\/pre>\n\n\n\n
\nhealth status index                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size\nyellow open   filebeat-7.0.0-2019.05.01-000001 3vhNqUvLS6CTgJQlSkp1Lg   1   1          0            0       283b           283b\ngreen  open   .kibana_1                        w-vOaD46QGa7LldMAvJVtw   1   0          3            1       20kb           20kb\nyellow open   ssh_auth-2019.05                 TyZHEQx_SR2q05a8yCEU-A   1   1       5940            0      2.2mb          2.2mb\ngreen  open   .kibana_task_manager             XhZ5kmOMRzWvQ3VDEvoNRA   1   0          2            0     45.4kb         45.4kb\n<\/code><\/pre>\n\n\n\n
As you can see, our index health status is yellow<\/strong>. This means that all data is available but some replicas are not yet allocated, for our case, this is a single node deployment.<\/p>\n\n\n\n
yellow open   ssh_auth-2019.05                 TyZHEQx_SR2q05a8yCEU-A   1   1       5940            0      2.2mb          2.2mb<\/code><\/pre>\n\n\n\n
Check ssh_auth-2019.05 index;<\/p>\n\n\n\n
curl -X GET 192.168.0.101:9200\/ssh_auth-*\/_search?pretty<\/code><\/pre>\n\n\n\n
\n{\n  \"took\" : 948,\n  \"timed_out\" : false,\n  \"_shards\" : {\n    \"total\" : 1,\n    \"successful\" : 2,\n    \"skipped\" : 0,\n    \"failed\" : 0\n  },\n  \"hits\" : {\n    \"total\" : {\n      \"value\" : 6405,\n      \"relation\" : \"eq\"\n    },\n    \"max_score\" : 1.0,\n    \"hits\" : [\n      {\n        \"_index\" : \"ssh_auth-2019.05\",\n        \"_type\" : \"_doc\",\n...\n<\/code><\/pre>\n\n\n\n
Next, Login to Kibana dashboard and add your index.<\/p>\n\n\n\n
\"Install<\/figure><\/a><\/div>\n\n\n\n
Click Next step to set the timestamp. After click Create index pattern<\/strong> to create your index.<\/p>\n\n\n\n
\"Install<\/figure><\/a><\/div>\n\n\n\n
Once the index is created, click on the Discover tab on the left pane. You should be able to see your data.<\/p>\n\n\n\n
\"Install<\/figure><\/a><\/div>\n\n\n\n
To test this, perform a failed ssh login to the host machine with Filebeat installed and search for keyword failed<\/strong>, on Kibana the dashboard.<\/p>\n\n\n\n
\"Install<\/figure><\/a><\/div>\n\n\n\n
You can then proceed to create Kibana dashboards once you get all the data you need. Well, that is all about how to install and configure Filebeat 7 on Ubuntu 18.04\/Debian 9.8. We will cover Kibana dashboard configuration in our next guides.<\/p>\n\n\n\n
Reference:<\/p>\n\n\n\n
Getting Started with Filebeat<\/a><\/p>\n\n\n\n
Other Related Guides:<\/p>\n\n\n\n
How to Debug Logstash Grok Filters<\/a><\/p>\n\n\n\n
Install Logstash 7 on Fedora 30\/Fedora 29\/CentOS 7<\/a><\/p>\n\n\n\n
Install Elastic Stack 7 on Fedora 30\/Fedora 29\/CentOS 7<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
This guide discusses how to install and configure Filebeat 7 on Ubuntu 18.04\/Debian 9.8. This comes as the last part of our guide on how<\/p>\n","protected":false},"author":1,"featured_media":5014,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[72,910],"tags":[923,4029,912,913,922,4028,4027,4030],"class_list":["post-2775","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-monitoring","category-elastic-stack","tag-beats","tag-debian-filebeat-install","tag-elastic-stack","tag-elk","tag-filebeat","tag-install-filebeat-on-debian-9","tag-install-filebeat-on-ubuntu-18-04","tag-ubuntu-filebeat-install","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/2775"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=2775"}],"version-history":[{"count":20,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/2775\/revisions"}],"predecessor-version":[{"id":10247,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/2775\/revisions\/10247"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/5014"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=2775"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=2775"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=2775"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}