{"id":11330,"date":"2022-01-22T00:21:52","date_gmt":"2022-01-21T21:21:52","guid":{"rendered":"https:\/\/kifarunix.com\/?p=11330"},"modified":"2024-03-14T19:09:30","modified_gmt":"2024-03-14T16:09:30","slug":"configure-filebeat-elasticsearch-authentication","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/configure-filebeat-elasticsearch-authentication\/","title":{"rendered":"Configure Filebeat-Elasticsearch Authentication"},"content":{"rendered":"\n<p>This tutorial will take you through how you can configure filebeat-elasticsearch authentication. You realize that when you enable <a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/security-minimal-setup.html\" target=\"_blank\" rel=\"noreferrer noopener\">Elastic basic authentication<\/a>, you need to valid user credentials to authenticate and validate access to restricted Elastic resources.<\/p>\n\n\n\n<p>Our previous guide showed how to enable Elastic stack basic authenticaion.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/how-to-enable-basic-authentication-on-elk-stack\/\" target=\"_blank\" rel=\"noreferrer noopener\">How to Enable Basic Authentication on ELK Stack<\/a><\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#configureing-filebeat-elasticsearch-authentication\">Configureing Filebeat Elasticsearch Authentication<\/a><ul><li><a href=\"#create-required-publishing-roles\">Create Required Publishing Roles<\/a><\/li><li><a href=\"#create-indexing-user-and-assign-respective-roles\">Create Indexing User and Assign Respective Roles<\/a><\/li><li><a href=\"#install-filebeat\">Install Filebeat<\/a><\/li><li><a href=\"#configure-authentication-in-elasticsearch-output\">Configure Authentication in Elasticsearch Output<\/a><\/li><li><a href=\"#verify-filebeat-elasticsearch-authentication\">Verify Filebeat-Elasticsearch Authentication<\/a><\/li><li><a href=\"#other-tutorials\">Other Tutorials<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"configureing-filebeat-elasticsearch-authentication\">Configureing Filebeat Elasticsearch Authentication<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"create-required-publishing-roles\">Create Required Publishing Roles<\/h3>\n\n\n\n<p>Before you can proceed, first create Filebeat users and assign the user specific roles to be able to write\/publish data to specific indices.<\/p>\n\n\n\n<p>To begin with, login to Kibana and navigate <strong>Management &gt; Stack Management &gt; Security &gt; Roles<\/strong> to create a publishing role.<\/p>\n\n\n\n<p>On the roles page, click <strong>Create role<\/strong> and;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Set the name of the role, e.g <code><strong>filebeat_publisher<\/strong><\/code>.<\/li>\n\n\n\n<li><strong><code>Cluster privileges<\/code><\/strong>: If you are running ELK cluster, you need to define the cluster privileges such as;\n<ul class=\"wp-block-list\">\n<li><strong><code>monitor<\/code><\/strong>: provides all cluster read-only operations, like cluster health and state, hot threads, node info, node and cluster stats, and pending cluster tasks.<\/li>\n\n\n\n<li><strong>manage<\/strong>: Builds on&nbsp;<code>monitor<\/code>&nbsp;and adds cluster operations that change values in the cluster.<\/li>\n\n\n\n<li>See example cluster privileges on the <a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/security-privileges.html#privileges-list-cluster\" target=\"_blank\" rel=\"noreferrer noopener\">Security privileges page<\/a>.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><code>Run As privileges<\/code><\/strong>: this defines a user that is allowed to submit requests on behalf of other users. We wont use this in our setup.<\/li>\n\n\n\n<li><strong><code>Index Privileges<\/code><\/strong>:\n<ul class=\"wp-block-list\">\n<li><strong>Indices<\/strong>: Select specific index from the list or simply enter the wildcard name of your index and press ENTER. We used <strong>*<\/strong> (<em>asterisk to specify any index<\/em>)<\/li>\n\n\n\n<li><strong>Privileges<\/strong>: Define the privileges that allows a user to publish events on the specific index. Such privileges can include;\n<ul class=\"wp-block-list\">\n<li><strong><code>monitor<\/code><\/strong>: enables the user to retrieve cluster details<\/li>\n\n\n\n<li><strong><code>create_index<\/code><\/strong>: enables a user create an index or data stream.<\/li>\n\n\n\n<li><strong><code>create_doc<\/code><\/strong>: enables a user to write events into an index<\/li>\n\n\n\n<li><strong><code>view_index_metadata<\/code><\/strong>: enables a user to check for alias when connecting to clusters that support ILM.&nbsp;<\/li>\n\n\n\n<li><strong><code>manage_ilm<\/code><\/strong>: gives a user all index lifecycle management operations relating to managing the execution of policies of an index or data stream.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>Read more on <a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/security-privileges.html#privileges-list-indices\" target=\"_blank\" rel=\"noreferrer noopener\">Index Privileges<\/a> page.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1800\" height=\"904\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/01\/elk-index-role-privileges.png\" alt=\"Configure Filebeat-Elasticsearch Authentication\" class=\"wp-image-11347\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/01\/elk-index-role-privileges.png?v=1642799700 1800w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/01\/elk-index-role-privileges-768x386.png?v=1642799700 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/01\/elk-index-role-privileges-1536x771.png?v=1642799700 1536w\" sizes=\"(max-width: 1800px) 100vw, 1800px\" \/><\/figure><\/div>\n\n\n<p>Once done defining the roles, scroll down the page and click <strong>Create role<\/strong>.<\/p>\n\n\n\n<p>The role should now appear under the list of roles.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"create-indexing-user-and-assign-respective-roles\">Create Indexing User and Assign Respective Roles<\/h3>\n\n\n\n<p>Under <strong>Security<\/strong> &gt; <strong>Users<\/strong>, click <strong>Create user<\/strong>.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Set the username<\/li>\n\n\n\n<li>Set the password<\/li>\n\n\n\n<li>Assign the respective roles<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1365\" height=\"936\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/01\/elk-index-users.png\" alt=\"kibana create user and assign roles\" class=\"wp-image-11348\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/01\/elk-index-users.png?v=1642799763 1365w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/01\/elk-index-users-768x527.png?v=1642799763 768w\" sizes=\"(max-width: 1365px) 100vw, 1365px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Create the user<\/li>\n\n\n\n<li>The user is now listed under users page.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-filebeat\">Install Filebeat<\/h3>\n\n\n\n<p>You can follow the links below to install Filebeat on your favorite Unix distro;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-filebeat-on-freebsd\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Filebeat on FreeBSD<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-configure-filebeat-on-centos-8\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install and Configure Filebeat on CentOS 8<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-filebeat-on-fedora-30-fedora-29-centos-7\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Filebeat on Fedora 30\/Fedora 29\/CentOS 7<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-configure-filebeat-7-on-ubuntu-18-04-debian-9-8\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install and Configure Filebeat 7 on Ubuntu 18.04\/Debian 9.8<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configure-authentication-in-elasticsearch-output\">Configure Authentication in Elasticsearch Output<\/h3>\n\n\n\n<p>Once you have installed Filebeat, it is now time to configure it so that it can be able to authenticate to Elastic stack and be able to write events to the specific index defined on the roles assigned to the user being used.<\/p>\n\n\n\n<p>Thus, open the <code>filebeat.yml<\/code>;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/filebeat\/filebeat.yml<\/code><\/pre>\n\n\n\n<p>Under Elasticsearch Output;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>define the Elasticsearch host<\/li>\n\n\n\n<li>define the authentication credentials. In this case it is the username\/password.<\/li>\n<\/ul>\n\n\n\n<pre class=\"scroll-box\"><code>\n# ================================== Outputs ===================================\n\n# Configure what output to use when sending the data collected by the beat.\n\n# ---------------------------- Elasticsearch Output ----------------------------\noutput.elasticsearch:\n  # Array of hosts to connect to.\n  hosts: [\"192.168.:9200\"]\n\n  # Protocol - either `http` (default) or `https`.\n  #protocol: \"https\"\n\n  # Authentication credentials - either API key or username\/password.\n  #api_key: \"id:api_key\"\n  #username: \"elastic\"\n  #password: \"changeme\"\n<\/code><\/pre>\n\n\n\n<p>While defining the credentials, you can specify them in plain text or store them more securely using Filebeat keystore.<\/p>\n\n\n\n<p>To define the credentials in plain text, define the values for the <strong>username<\/strong> and <strong>password<\/strong> in the configuration file;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n# ================================== Outputs ===================================\n\n# Configure what output to use when sending the data collected by the beat.\n\n# ---------------------------- Elasticsearch Output ----------------------------\noutput.elasticsearch:\n  # Array of hosts to connect to.\n  hosts: [\"192.168.58.22:9200\"]\n\n  # Protocol - either `http` (default) or `https`.\n  #protocol: \"https\"\n\n  # Authentication credentials - either API key or username\/password.\n  #api_key: \"id:api_key\"\n  username: \"fbpublisher\"\n  password: \"p@ssw0rd-fb\"\n<\/code><\/pre>\n\n\n\n<p>To store the password in Filebeat Keystore;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Create Filebeat keystore.<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat keystore create<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Add the username and password into the keystore using variables e.g USER for username and PASS for password.\n<ul class=\"wp-block-list\">\n<li>Add the username by running the command below. When prompted, enter the publishing username, <code><strong>fbpublisher<\/strong><\/code>, for example.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat keystore add USER<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Add the password. When prompted, enter the password for the publishing user.<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat keystore add PASS<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Specify the variables for the username and password in the configuration file, <strong><code>username: \"${ES_USER}\"<\/code><\/strong> and <code><strong>password: \"${ES_PWD}\"<\/strong><\/code>;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/filebeat\/filebeat.yml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code># ================================== Outputs ===================================\n\n# Configure what output to use when sending the data collected by the beat.\n\n# ---------------------------- Elasticsearch Output ----------------------------\noutput.elasticsearch:\n  # Array of hosts to connect to.\n  hosts: [\"192.168.58.22:9200\"]\n\n  # Protocol - either `http` (default) or `https`.\n  #protocol: \"https\"\n\n  # Authentication credentials - either API key or username\/password.\n  #api_key: \"id:api_key\"\n  username: \"${ES_USER}\"\n  password: \"${ES_PWD}\"\n<\/code><\/pre>\n\n\n\n<p>Save and exit the configuration file.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"verify-filebeat-elasticsearch-authentication\">Verify Filebeat-Elasticsearch Authentication<\/h3>\n\n\n\n<p>To check whether the authenticated connection works and whether Filebeat can be able to create an index, just run the test output command;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat test output<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\nelasticsearch: http:\/\/192.168.58.22:9200...\n  parse url... OK\n  connection...\n    parse host... OK\n    dns lookup... OK\n    addresses: 192.168.58.22\n    dial up... OK\n  TLS... WARN secure connection disabled\n  talk to server... OK\n  version: 7.16.3\n<\/code><\/pre>\n\n\n\n<p>If there is any authentication issue, you should be able to see from the command output.<\/p>\n\n\n\n<p>Now when you run your Filebeat, it shoud be able to connect to Elasticsearch and publish event data without any issue.<\/p>\n\n\n\n<p>You can run Filebeat in debug mode to test this as well. This will show to you some errors to standard output, including the permissions\/roles issues.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat -e<\/code><\/pre>\n\n\n\n<p>Sample index privilege error;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>2022-01-21T23:56:32.328+0300\tERROR\t&#91;index-management.ilm]\tilm\/std.go:133\tIndex Alias filebeat-7.16.3 setup failed: failed to create alias: {\"error\":{\"root_cause\":&#91;{\"type\":\"security_exception\",\"reason\":\"action &#91;indices:admin\/aliases] is unauthorized for user &#91;fbpublisher] with roles &#91;filebeat_publisher], this action is granted by the index privileges &#91;manage,all]\"}],\"type\":\"security_exception\",\"reason\":\"action &#91;indices:admin\/aliases] is unauthorized for user &#91;fbpublisher] with roles &#91;filebeat_publisher], this action is granted by the index privileges &#91;manage,all]\"},\"status\":403}: 403 Forbidden: {\"error\":{\"root_cause\":&#91;{\"type\":\"security_exception\",\"reason\":\"action &#91;indices:admin\/aliases] is unauthorized for user &#91;fbpublisher] with roles &#91;filebeat_publisher], this action is granted by the index privileges &#91;manage,all]\"}],\"type\":\"security_exception\",\"reason\":\"action &#91;indices:admin\/aliases] is unauthorized for user &#91;fbpublisher] with roles &#91;filebeat_publisher], this action is granted by the index privileges &#91;manage,all]\"},\"status\":403}.<\/code><\/pre>\n\n\n\n<p>Such an error is due to missing <strong><code>manage<\/code><\/strong> privilege on the specified role. Thus you can edit the role and add <strong>manage<\/strong> index privilege as stated on the log.<\/p>\n\n\n\n<p>Filebeat should now be able to authenticate to Elasticsearch and write data to various indices that you can define. Remember while specifying indices to give access to, we used * to mean the user can publish the events to any defined index.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"other-tutorials\">Other Tutorials<\/h3>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/easy-way-to-configure-filebeat-logstash-ssl-tls-connection\/\" target=\"_blank\" rel=\"noreferrer noopener\">Easy way to configure Filebeat-Logstash SSL\/TLS Connection<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/ship-system-logs-to-elk-stack-using-elastic-agents\/\" target=\"_blank\" rel=\"noreferrer noopener\">Ship System Logs to ELK Stack using Elastic Agents<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This tutorial will take you through how you can configure filebeat-elasticsearch authentication. You realize that when you enable Elastic basic authentication, you need to valid<\/p>\n","protected":false},"author":3,"featured_media":11333,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[34,910,121],"tags":[4467,4459,4460,4469,4468,4470],"class_list":["post-11330","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-security","category-elastic-stack","category-howtos","tag-configure-filebeat-elasticsearch-authentication","tag-elasticsearch-authentication","tag-elk-authentication","tag-filebeat-authentication","tag-filebeat-elasticsearch-authentication","tag-filebeat-keystore","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/11330"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=11330"}],"version-history":[{"count":7,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/11330\/revisions"}],"predecessor-version":[{"id":21308,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/11330\/revisions\/21308"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/11333"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=11330"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=11330"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=11330"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}