{"id":17544,"date":"2023-06-28T10:54:44","date_gmt":"2023-06-28T07:54:44","guid":{"rendered":"https:\/\/kifarunix.com\/?p=17544"},"modified":"2024-03-10T09:28:51","modified_gmt":"2024-03-10T06:28:51","slug":"configure-filebeat-8-to-write-logs-to-specific-data-stream","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/configure-filebeat-8-to-write-logs-to-specific-data-stream\/","title":{"rendered":"Configure Filebeat 8 to Write Logs to Specific Data Stream"},"content":{"rendered":"\n<p>This guide will take you through how to configure Filebeat 8 to write logs to specific data stream. Are you collecting logs using Filebeat 8 and want to write them to specific data stream on Elasticsearch 8? Well, look no further as this guide is for you!<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#configuring-filebeat-8-to-write-logs-to-specific-data-stream\">Configuring Filebeat 8 to Write Logs to Specific Data Stream<\/a><ul><li><a href=\"#default-filebeat-data-streams\">Default Filebeat Data Streams<\/a><\/li><li><a href=\"#configuring-filebeat-8-to-write-logs-to-specific-data-stream-1\">Configuring Filebeat 8 to Write Logs to Specific Data Stream<\/a><ul><li><a href=\"#optional-create-index-lifecycle-management-policy\">[Optional] Create Index Lifecycle Management Policy<\/a><\/li><li><a href=\"#create-component-index-template\">Create Component Index Template<\/a><\/li><li><a href=\"#create-index-template\">Create Index Template<\/a><\/li><li><a href=\"#create-the-data-stream\">Create the Data Stream<\/a><\/li><\/ul><\/li><li><a href=\"#configuring-filebeat-8-to-write-logs-to-specific-data-stream-2\">Configuring Filebeat 8 to Write Logs to Specific Data Stream<\/a><ul><li><a href=\"#verify-data-reception-on-custom-data-stream\">Verify Data Reception on Custom Data Stream<\/a><\/li><\/ul><\/li><li><a href=\"#create-kibana-data-view\">Create Kibana Data View<\/a><\/li><li><a href=\"#visualize-data-on-kibana\">Visualize Data on Kibana<\/a><\/li><li><a href=\"#other-tutorials\">Other Tutorials<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"configuring-filebeat-8-to-write-logs-to-specific-data-stream\">Configuring Filebeat 8 to Write Logs to Specific Data Stream<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"default-filebeat-data-streams\">Default Filebeat Data Streams<\/h3>\n\n\n\n<p>By default, Filebeat 8 uses a new feature on Elasticsearch 8 called data streams. <a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/8.8\/data-streams.html\" target=\"_blank\" rel=\"noreferrer noopener\">Data stream<\/a> is a logical groupings of indices, that are created using index templates. They are used to store append-only time series data across multiple backing indices. Data stream backing indices are usually hidden by default.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p>Data streams are designed for use cases where existing data is rarely, if ever, updated. You cannot send update or deletion requests for existing documents directly to a data stream. Instead, use the&nbsp;<a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/8.8\/use-a-data-stream.html#update-docs-in-a-data-stream-by-query\" target=\"_blank\" rel=\"noopener\">update by query<\/a>&nbsp;and&nbsp;<a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/8.8\/use-a-data-stream.html#delete-docs-in-a-data-stream-by-query\" target=\"_blank\" rel=\"noopener\">delete by query<\/a>&nbsp;APIs.<\/p>\n\n\n\n<p>If needed, you can&nbsp;<a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/8.8\/use-a-data-stream.html#update-delete-docs-in-a-backing-index\" target=\"_blank\" rel=\"noopener\">update or delete documents<\/a>&nbsp;by submitting requests directly to the document\u2019s backing index.<\/p>\n\n\n\n<p>If you frequently update or delete existing time series data, use an index alias with a write index instead of a data stream. See&nbsp;<a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/8.8\/getting-started-index-lifecycle-management.html#manage-time-series-data-without-data-streams\" target=\"_blank\" rel=\"noopener\">Manage time series data without data streams<\/a>.<\/p>\n<cite>Elasticsearch Data Streams<\/cite><\/blockquote>\n\n\n\n<p>Consider the Filebeat we installed on Debian 12 in our previous guide;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-filebeat-8-on-debian\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Filebeat 8 on Debian 12<\/a><\/p>\n\n\n\n<p>In the guide above, no any custom changes were made in relation to what data streams Filebeat will write to. Thus, it writes any event data collected to the default data stream, <strong>filebeat-8.8.1<\/strong>.<\/p>\n\n\n\n<p>To confirm, see under <strong>Stack Management &gt; Data &gt; Index Management &gt; Data Streams<\/strong>;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1889\" height=\"623\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/data-streams.png\" alt=\"Configure Filebeat 8 to Write Logs to Specific Data Stream\" class=\"wp-image-17572\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/data-streams.png?v=1687897152 1889w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/data-streams-768x253.png?v=1687897152 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/data-streams-1536x507.png?v=1687897152 1536w\" sizes=\"(max-width: 1889px) 100vw, 1889px\" \/><\/figure>\n\n\n\n<p>If you want to see Data stream indices, click <strong>Indices<\/strong> under <strong>Index Management<\/strong> and toggle the <strong>include hidden indices<\/strong> option.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1897\" height=\"546\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/data-stream-indices.png\" alt=\"Configure Filebeat 8 to Write Logs to Specific Data Stream\" class=\"wp-image-17574\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/data-stream-indices.png?v=1687900388 1897w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/data-stream-indices-768x221.png?v=1687900388 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/data-stream-indices-1536x442.png?v=1687900388 1536w\" sizes=\"(max-width: 1897px) 100vw, 1897px\" \/><\/figure>\n\n\n\n<p>As already mentioned, data streams are created using index templates. Index templates define how Elasticsearch has to configure an index when it is created. For example, <strong>filebeat-8.8.1<\/strong> index is created by the index template, <strong>Filebeat-8.8.1<\/strong>. You can find index templates under <strong>Index Templates<\/strong> section.<\/p>\n\n\n\n<p>You can get the details about the index template using the command below. Update it to match your ELK setup;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -k -XGET https:\/\/elk.kifarunix-demo.com:9200\/_index_template\/<strong>filebeat-8.8.1<\/strong>?pretty \\\n-u elastic --cacert \/etc\/elasticsearch\/certs\/http_ca.crt<\/code><\/pre>\n\n\n\n<p>Or login to Kibana, <strong>Management<\/strong> &gt; <strong>DevTools<\/strong> &gt; <strong>Console<\/strong> and execute the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>GET _index_template\/<strong>filebeat-8.8.1<\/strong><\/code><\/pre>\n\n\n\n<p>If you want to use indices instead and write to custom\/specific indices;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/configure-filebeat-8-to-write-logs-to-specific-index\/\" target=\"_blank\" rel=\"noreferrer noopener\">Configure Filebeat 8 to Write Logs to Specific Index<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configuring-filebeat-8-to-write-logs-to-specific-data-stream-1\">Configuring Filebeat 8 to Write Logs to Specific Data Stream<\/h3>\n\n\n\n<p>Now, what if you want to control how Filebeat writes logs to data streams?<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"optional-create-index-lifecycle-management-policy\">[Optional] Create Index Lifecycle Management Policy<\/h4>\n\n\n\n<p>This step is <strong>optional<\/strong>, but if you want to control the lifecycle tasks of your indices such as creation, deletion, rollover to new phases etc, ILM policies come in very handy. You can manage the ILM policies on <strong>Kibana<\/strong> under <strong>Stack Management<\/strong> &gt; <strong>Data<\/strong> &gt; <strong>Index Lifecycle Policies<\/strong>.<\/p>\n\n\n\n<p>So, for the purposes of demonstration, let&#8217;s create a custom ILM policy to apply to our custom data stream indices. Thus;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Navigate to <strong>Kibana &gt; Stack Management &gt; Data &gt; Index Lifecycle Policies &gt; Create Policy<\/strong>.<\/li>\n\n\n\n<li>Enter the name of the policy, for example, <strong>kifarunix<\/strong> in our example.<\/li>\n\n\n\n<li>Configure the policy phases;\n<ul class=\"wp-block-list\">\n<li><strong>Hot Phase<\/strong>: Can be used to store <strong>Most<\/strong> recent and most frequently searched data. This phase is <strong>Required<\/strong>.<\/li>\n\n\n\n<li><strong>Warm Phase<\/strong>: Stores the data that you are still likely to search it, but infrequently need to update it.<\/li>\n\n\n\n<li><strong>Cold Phase<\/strong>: Stores the data that you less often search and don\u2019t need to update it.<\/li>\n\n\n\n<li><strong>Delete Phase<\/strong>: At this phase, you can delete data you no longer need.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Note that you can jump straight into delete phase after each phase by clicking the <strong>trash<\/strong> icon.<\/li>\n<\/ul>\n\n\n\n<p>Here is a screenshot of our ILM policy configuration.<\/p>\n\n\n\n<p>Hot Phase<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1598\" height=\"2213\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/ilm-policy-hot-phase.png\" alt=\"\" class=\"wp-image-17579\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/ilm-policy-hot-phase.png?v=1687926716 1598w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/ilm-policy-hot-phase-768x1064.png?v=1687926716 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/ilm-policy-hot-phase-1109x1536.png?v=1687926716 1109w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/ilm-policy-hot-phase-1479x2048.png?v=1687926716 1479w\" sizes=\"(max-width: 1598px) 100vw, 1598px\" \/><\/figure>\n\n\n\n<p>Warm, cold, phases;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1577\" height=\"1542\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/warm-cold-delete-phases.png\" alt=\"\" class=\"wp-image-17580\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/warm-cold-delete-phases.png?v=1687927401 1577w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/warm-cold-delete-phases-768x751.png?v=1687927401 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/warm-cold-delete-phases-1536x1502.png?v=1687927401 1536w\" sizes=\"(max-width: 1577px) 100vw, 1577px\" \/><\/figure>\n\n\n\n<p>Create a policy that suits your needs!<\/p>\n\n\n\n<p>You can always verify your policy with API command. Replace the index pattern accordingly.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>GET .ds-kifarunix-*\/_ilm\/explain<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-component-index-template\">Create Component Index Template<\/h4>\n\n\n\n<p>Component index template defines mappings, settings, and aliases that can be used while creating index templates.<\/p>\n\n\n\n<p>We will use the default component index templates in this guide.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-index-template\">Create Index Template<\/h4>\n\n\n\n<p>An index template on the other hand is a template that is used to define specific settings for a specific index. Index templates can contain settings and mappings that are defined in component templates, as well as settings and mappings that are specific to the index.<\/p>\n\n\n\n<p>So, let&#8217;s create our own custom data stream index template.<\/p>\n\n\n\n<p>Navigate to <strong>Kibana &gt; Stack Management &gt; Data &gt; Index Management &gt; Index Templates<\/strong>.<\/p>\n\n\n\n<p>To make our life easier, let&#8217;s clone an existing Filebeat index template and modify it to suit our needs.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1597\" height=\"662\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/clone-an-index-template.png\" alt=\"Configure Filebeat 8 to Write Logs to Specific Data Stream\" class=\"wp-image-17584\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/clone-an-index-template.png?v=1687932050 1597w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/clone-an-index-template-768x318.png?v=1687932050 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/clone-an-index-template-1536x637.png?v=1687932050 1536w\" sizes=\"(max-width: 1597px) 100vw, 1597px\" \/><\/figure>\n\n\n\n<p>Under logistics, set the name of the index template, the index pattern, enable <strong>Create Data Stream<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1541\" height=\"1220\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/index-template-logistics-name-pattern-data-stream.png\" alt=\"\" class=\"wp-image-17589\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/index-template-logistics-name-pattern-data-stream.png?v=1687932542 1541w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/index-template-logistics-name-pattern-data-stream-768x608.png?v=1687932542 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/index-template-logistics-name-pattern-data-stream-1536x1216.png?v=1687932542 1536w\" sizes=\"(max-width: 1541px) 100vw, 1541px\" \/><\/figure>\n\n\n\n<p>Ensure the priority number, if defined, does not match with the value of the index pattern being cloned.<\/p>\n\n\n\n<p>Under components template, we will use default settings and just proceed to next page.<\/p>\n\n\n\n<p>Under <strong>Index Settings<\/strong>, we will only change the ILM policy. When you clone Filebeat index template, it will be configured to use the Filebeat ILM policy by default, So, the only thing we change here is the name of the ILM policy.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1554\" height=\"780\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/index-settings-ilm-policy.png\" alt=\"\" class=\"wp-image-17586\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/index-settings-ilm-policy.png?v=1687932289 1554w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/index-settings-ilm-policy-768x385.png?v=1687932289 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/index-settings-ilm-policy-1536x771.png?v=1687932289 1536w\" sizes=\"(max-width: 1554px) 100vw, 1554px\" \/><\/figure>\n\n\n\n<p>Mappings and Aliases, we will use default settings and proceed to Index template preview;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1534\" height=\"853\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/review-template-and-create.png\" alt=\"\" class=\"wp-image-17587\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/review-template-and-create.png?v=1687932312 1534w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/review-template-and-create-768x427.png?v=1687932312 768w\" sizes=\"(max-width: 1534px) 100vw, 1534px\" \/><\/figure>\n\n\n\n<p>Create the template.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-the-data-stream\">Create the Data Stream<\/h4>\n\n\n\n<p>To begin using the data stream, you first have to create it by submitting an indexing request that targets the stream&#8217;s name.<\/p>\n\n\n\n<p>To create a data stream, you can execute the API command below from Kibana console, (<strong>Kibana &gt; Management &gt; DevTools &gt; Console<\/strong>)<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>PUT _data_stream\/<strong>name-of-stream<\/strong><\/code><\/pre>\n\n\n\n<p><strong>The stream\u2019s name must still match one of your template\u2019s index patterns.<\/strong><\/p>\n\n\n\n<p>For example, in my setup;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>PUT _data_stream\/kifarunix<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"acknowledged\": true\n}<\/code><\/pre>\n\n\n\n<p>You can also do this from command line as long as you have access to Elasticsearch;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -k -X PUT -u elastic \\\n\"https:\/\/elk.kifarunix-demo.com:9200\/_data_stream\/kifarunix?pretty\" \\\n--cacert \/etc\/elasticsearch\/certs\/http_ca.crt<\/code><\/pre>\n\n\n\n<p>You should now be able to see your data stream created and of course first backing index;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1605\" height=\"648\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/custom-data-streams.png\" alt=\"Configure Filebeat 8 to Write Logs to Specific Data Stream\" class=\"wp-image-17591\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/custom-data-streams.png?v=1687936538 1605w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/custom-data-streams-768x310.png?v=1687936538 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/custom-data-streams-1536x620.png?v=1687936538 1536w\" sizes=\"(max-width: 1605px) 100vw, 1605px\" \/><\/figure>\n\n\n\n<p>Click 1 under indices to view the backing index;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1612\" height=\"546\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/data-stream-custom-backing-indices.png\" alt=\"\" class=\"wp-image-17592\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/data-stream-custom-backing-indices.png?v=1687936642 1612w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/data-stream-custom-backing-indices-768x260.png?v=1687936642 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/data-stream-custom-backing-indices-1536x520.png?v=1687936642 1536w\" sizes=\"(max-width: 1612px) 100vw, 1612px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configuring-filebeat-8-to-write-logs-to-specific-data-stream-2\">Configuring Filebeat 8 to Write Logs to Specific Data Stream<\/h3>\n\n\n\n<p>Now that we have data stream setup, how can you configure Filebeat to be able to write data the specific custom data stream?<\/p>\n\n\n\n<p>Open the Filebeat configuration file for editing;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/filebeat\/filebeat.yml<\/code><\/pre>\n\n\n\n<p>Define the <strong>index name<\/strong> and <strong>set the template<\/strong> and <strong>template pattern<\/strong> to match what you created under index templates above.<\/p>\n\n\n\n<p>See my config below;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n# ---------------------------- Elasticsearch Output ----------------------------\noutput.elasticsearch:\n  hosts: [\"elk.kifarunix-demo.com:9200\"]\n  protocol: \"https\"\n  ssl.certificate_authorities: [\"\/etc\/filebeat\/elastic-ca.crt\"]\n<strong>  index: kifarunix\n<\/strong>  username: \"elastic\"\n  password: \"ALL16n6Xv5yJclrWt5Sc\"\n#\n<strong>setup.template.name: \"kifarunix\"\nsetup.template.pattern: \"kifarunix\"<\/strong>\n<\/code><\/pre>\n\n\n\n<p>Save and exit the file.<\/p>\n\n\n\n<p>Check Filebeat for any configuration syntax and ensure output is <strong>Config OK<\/strong>;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat test config<\/code><\/pre>\n\n\n\n<p>(Re)start\/ Filebeat;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl restart filebeat<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"verify-data-reception-on-custom-data-stream\">Verify Data Reception on Custom Data Stream<\/h4>\n\n\n\n<p>Navigate to Index management &gt; Indices and search for your data stream index pattern<\/p>\n\n\n\n<p>Similarly, toggle the <strong>include hidden indices<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1801\" height=\"551\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/verify-custom-data-stream-index.png\" alt=\"Configure Filebeat 8 to Write Logs to Specific Data Stream\" class=\"wp-image-17593\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/verify-custom-data-stream-index.png?v=1687937640 1801w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/verify-custom-data-stream-index-768x235.png?v=1687937640 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/verify-custom-data-stream-index-1536x470.png?v=1687937640 1536w\" sizes=\"(max-width: 1801px) 100vw, 1801px\" \/><\/figure>\n\n\n\n<p>As you can see, the size is now at 16+mb, which means, data is being written;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"create-kibana-data-view\">Create Kibana Data View<\/h3>\n\n\n\n<p>You can now create Kibana data view for your custom data stream  to allow you visualize the data.<\/p>\n\n\n\n<p>Hence, navigate to <strong>Management &gt; Kibana &gt; Data Views &gt; Create Data View<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1779\" height=\"792\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/kibana-data-views.png\" alt=\"\" class=\"wp-image-17595\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/kibana-data-views.png?v=1687937884 1779w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/kibana-data-views-768x342.png?v=1687937884 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/kibana-data-views-1536x684.png?v=1687937884 1536w\" sizes=\"(max-width: 1779px) 100vw, 1779px\" \/><\/figure>\n\n\n\n<p>Save the data view.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"visualize-data-on-kibana\">Visualize Data on Kibana<\/h3>\n\n\n\n<p>You can now visualize the data on Kibana by navigating to <strong>Analytics &gt; Discover<\/strong> and select your data view from the drop down;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1895\" height=\"837\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/kibana-data-visualization.png\" alt=\"\" class=\"wp-image-17596\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/kibana-data-visualization.png?v=1687938068 1895w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/kibana-data-visualization-768x339.png?v=1687938068 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/06\/kibana-data-visualization-1536x678.png?v=1687938068 1536w\" sizes=\"(max-width: 1895px) 100vw, 1895px\" \/><\/figure>\n\n\n\n<p>And there you go! That is all on configuring Filebeat 8 to Write Logs to Specific Data Stream.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"other-tutorials\">Other Tutorials<\/h3>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/easily-configure-elasticsearch-https-connection\/\" target=\"_blank\" rel=\"noreferrer noopener\">Easily Configure Elasticsearch HTTPS Connection<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/how-to-fix-filebeat-glibc-related-errors\/\" target=\"_blank\" rel=\"noreferrer noopener\">How to Fix Filebeat Glibc Related Errors on Ubuntu 22.04<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This guide will take you through how to configure Filebeat 8 to write logs to specific data stream. Are you collecting logs using Filebeat 8<\/p>\n","protected":false},"author":10,"featured_media":17598,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[72,910,121],"tags":[6972,6973,6969,6970,6968,6971],"class_list":["post-17544","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-monitoring","category-elastic-stack","category-howtos","tag-change-data-stream-name-in-filebeat-8","tag-create-filebeat-data-stream","tag-elasticsearch-custom-data-stream","tag-elasticsearch-custom-index-templates","tag-filebeat-custom-data-stream","tag-write-filebeat-data-to-custom-data-stream","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/17544"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=17544"}],"version-history":[{"count":23,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/17544\/revisions"}],"predecessor-version":[{"id":20758,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/17544\/revisions\/20758"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/17598"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=17544"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=17544"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=17544"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}