{"id":14356,"date":"2023-03-25T01:09:56","date_gmt":"2023-03-24T22:09:56","guid":{"rendered":"https:\/\/kifarunix.com\/?p=14356"},"modified":"2024-03-10T08:26:36","modified_gmt":"2024-03-10T05:26:36","slug":"configure-log-retention-period-in-elk-stack","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/configure-log-retention-period-in-elk-stack\/","title":{"rendered":"Configure Log Retention Period in ELK Stack"},"content":{"rendered":"\n<p>How do I change the log retention policy in elk stack? In this tutorial, you will learn how to configure log retention period in ELK stack. Elasticsearch uses ILM (<a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/index-lifecycle-management.html\" target=\"_blank\" rel=\"noreferrer noopener\">Index Lifecycle Management<\/a>) policies to define what actions to be applied to indices according to your performance, resiliency, and retention requirements.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#configuring-log-retention-period-in-elk-stack\">Configuring Log Retention Period in ELK Stack<\/a><ul><li><a href=\"#create-custom-index-lifecycle-management-policy\">Create Custom Index Lifecycle Management Policy<\/a><\/li><li><a href=\"#create-ilm-policy-from-kibana\">Create ILM Policy from Kibana<\/a><ul><li><a href=\"#configure-hot-phase\">Configure Hot Phase;<\/a><\/li><li><a href=\"#enable-and-configure-warm-phase\">Enable and Configure Warm Phase;<\/a><\/li><li><a href=\"#enable-and-configure-cold-phase\">Enable and Configure Cold Phase<\/a><\/li><li><a href=\"#configure-delete-phase\">Configure Delete Phase<\/a><\/li><\/ul><\/li><li><a href=\"#create-custom-index-template\">Create Custom Index Template<\/a><\/li><li><a href=\"#apply-ilm-policy-into-the-index-template\">Apply ILM Policy into the Index Template<\/a><ul><li><a href=\"#apply-ilm-policy-into-the-index-template-via-elasticsearch-api-on-elasticsearch-node-terminal\">Apply ILM Policy into the Index Template via Elasticsearch API on Elasticsearch node terminal<\/a><\/li><li><a href=\"#apply-ilm-policy-into-the-index-template-via-kibana-dev-tools\">Apply ILM Policy into the Index Template via Kibana Dev Tools<\/a><\/li><\/ul><\/li><li><a href=\"#verify-ilm-policy-and-data-retention-on-elk-stack\">Verify ILM Policy and Data Retention on ELK Stack<\/a><\/li><li><a href=\"#other-tutorials\">Other Tutorials<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"configuring-log-retention-period-in-elk-stack\">Configuring Log Retention Period in ELK Stack<\/h2>\n\n\n\n<p>By default, if no custom policy is applied to Elasticsearch indices, it will store data as long as possible without deleting them. Long data retention will make management of data cumbersome. There are also implications of cost of required storage and relevancy of the data as it ages. You should however be always guided by your regulatory and compliance policies on how long you should retain your data.<\/p>\n\n\n\n<p>So, how can you retain logs for a specific period in ELK stack?<\/p>\n\n\n\n<p>As already mentioned, Elastic uses ILM policies to define how long the log data can be stored. Some of the index lifecycle policies can trigger actions to permanently remove an index, including all of its data and metadata.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"create-custom-index-lifecycle-management-policy\">Create Custom Index Lifecycle Management Policy<\/h3>\n\n\n\n<p>To begin with, if you are running a cluster, you need to ensure that all nodes are running same version of Elasticsearch.<\/p>\n\n\n\n<p>Next, proceed to create your custom ILM policy to apply to your indices.<\/p>\n\n\n\n<p>You can create custom ILM policy from Kibana or using Elasticsearch API.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>create ILM Policy from Kibana<\/li>\n\n\n\n<li>create ILM policy from Elasticsearch API<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"create-ilm-policy-from-kibana\">Create ILM Policy from Kibana<\/h3>\n\n\n\n<p>To create a custom ILM policy from Kibana;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Navigate to Kibana menu &gt; <strong>Management &gt; Stack Management &gt; Data &gt; Index Lifecycle Policies<\/strong> &gt;&nbsp;<strong>Create policy<\/strong>.<\/li>\n<\/ul>\n\n\n\n<div><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/create-elasticsearch-policies.png\" class=\"td-modal-image\"><figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1882\" height=\"1442\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/create-elasticsearch-policies.png\" alt=\"Configure Log Retention Period in ELK Stack\" class=\"wp-image-15785\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/create-elasticsearch-policies.png?v=1679333211 1882w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/create-elasticsearch-policies-768x588.png?v=1679333211 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/create-elasticsearch-policies-1536x1177.png?v=1679333211 1536w\" sizes=\"(max-width: 1882px) 100vw, 1882px\" \/><\/figure><\/a><\/div>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Enter the name of the policy,<\/li>\n\n\n\n<li>Define various lifecycle&nbsp;phases. There exists five different phases as described below;\n<ul class=\"wp-block-list\">\n<li><strong>Hot<\/strong>: The index is actively being updated and queried. This phase is <strong>required<\/strong>.<\/li>\n\n\n\n<li><strong>Warm<\/strong>: The index is no longer being updated but is still being queried.<\/li>\n\n\n\n<li><strong>Cold<\/strong>: The index is no longer being updated and is queried infrequently. The information still needs to be searchable, but it\u2019s okay if those queries are slower.<\/li>\n\n\n\n<li><strong>Frozen<\/strong>: The index is no longer being updated and is queried rarely. The information still needs to be searchable, but it\u2019s okay if those queries are extremely slow.<\/li>\n\n\n\n<li><strong>Delete<\/strong>: The index is no longer needed and can safely be removed.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>You can configure your policy to <a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/ilm-index-lifecycle.html#ilm-phase-transitions\" target=\"_blank\" rel=\"noreferrer noopener\">transition<\/a> your data across all the phases above. For example, from hot phase &gt; warm phase &gt; cold phase &gt; delete phase, while defining respective settings at each phase. You can even have a policy that transition your index between just hot and delete phase based on some conditions.<\/li>\n\n\n\n<li>By default, all phases stores data indefinitely. <img loading=\"lazy\" decoding=\"async\" width=\"1882\" height=\"567\" class=\"wp-image-15788\" style=\"width: 1882px;\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/hot-phase-indifinite-retention.png\" alt=\"Configure Log Retention Period in ELK Stack\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/hot-phase-indifinite-retention.png?v=1679334815 1550w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/hot-phase-indifinite-retention-768x231.png?v=1679334815 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/hot-phase-indifinite-retention-1536x463.png?v=1679334815 1536w\" sizes=\"(max-width: 1882px) 100vw, 1882px\" \/><\/li>\n\n\n\n<li>To delete your index after a given period of time, you need to enable delete phase by toggling the <strong>Trash can icon<\/strong> on that phase. For example, if you want to move your index data from <strong>hot phase<\/strong> into a <strong>delete phase<\/strong> after a specific period of time, toggle the Trash can icon on the Hot phase.<\/li>\n\n\n\n<li>You should be able to see <strong>delete phase<\/strong> enabled.<img loading=\"lazy\" decoding=\"async\" width=\"1882\" height=\"2187\" class=\"wp-image-15796\" style=\"width: 1882px;\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/index-delete-phase.png\" alt=\"Configure Log Retention Period in ELK Stack\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/index-delete-phase.png?v=1679335830 1560w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/index-delete-phase-768x893.png?v=1679335830 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/index-delete-phase-1322x1536.png?v=1679335830 1322w\" sizes=\"(max-width: 1882px) 100vw, 1882px\" \/><\/li>\n\n\n\n<li>Rollover from a phase to phase is determined by size of the primary shard or the maximum\/minimum age of an index in a phase:\n<ul class=\"wp-block-list\">\n<li><strong>Maximum Age<\/strong>: This is the maximum amount of time an index can remain in a phase before it moves to the next phase.<\/li>\n\n\n\n<li><strong>Minimum Age<\/strong>: This is the minimum amount of time an index must remain in a phase before it can move to the next phase.<\/li>\n\n\n\n<li><strong>Maximum Size<\/strong>: This is the maximum size an index can be before it moves to the next phase.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>For demonstration purpose, let&#8217;s see how you can be able to configure your data to transition from hot &gt; warm &gt; cold &gt; delete phase.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-hot-phase\">Configure Hot Phase;<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Under Hot phase settings, expand Advanced Settings<\/li>\n\n\n\n<li>Disable &#8220;<strong>Use recommended defaults<\/strong>&#8221; option to be able to define your own thresholds.<\/li>\n\n\n\n<li>Enable rollover to start writing to a new index when the current index reaches a certain size, document count, or age.<\/li>\n\n\n\n<li>Set maximum primary shard size. In this demo, we use 1MB so we don&#8217;t have to wait for ages to demo the setup.<\/li>\n\n\n\n<li>Set the index maximum age. We use 10 minutes here, again for easy demo.<\/li>\n\n\n\n<li>You can leave other options with default settings.<\/li>\n<\/ul>\n\n\n\n<div><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/hot-phase-ilm-policy.png\" class=\"td-modal-image\"><figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1880\" height=\"2033\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/hot-phase-ilm-policy.png\" alt=\"Configure Log Retention Period in ELK Stack\" class=\"wp-image-15889\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/hot-phase-ilm-policy.png?v=1679694521 1880w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/hot-phase-ilm-policy-768x831.png?v=1679694521 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/hot-phase-ilm-policy-1420x1536.png?v=1679694521 1420w\" sizes=\"(max-width: 1880px) 100vw, 1880px\" \/><\/figure><\/a><\/div>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"enable-and-configure-warm-phase\">Enable and Configure Warm Phase;<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Choose when the data from Hot phase is moved into Warm phase. We use 5 minutes here for easy demonstrations.<\/li>\n\n\n\n<li>We also delete replicas when moving into warm phase by setting number of replica to 0.<\/li>\n<\/ul>\n\n\n\n<div><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/warm-phase-settings.png\" class=\"td-modal-image\"><figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1884\" height=\"1087\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/warm-phase-settings.png\" alt=\"Configure Log Retention Period in ELK Stack\" class=\"wp-image-15870\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/warm-phase-settings.png?v=1679598621 1884w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/warm-phase-settings-768x443.png?v=1679598621 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/warm-phase-settings-1536x886.png?v=1679598621 1536w\" sizes=\"(max-width: 1884px) 100vw, 1884px\" \/><\/figure><\/a><\/div>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"enable-and-configure-cold-phase\">Enable and Configure Cold Phase<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Next, enable and configure cold phase;<\/li>\n\n\n\n<li>Define when to move data from warm phase to cold phase. We set it to 5 minutes. The value must be greater than or equal to the warm phase value.<\/li>\n\n\n\n<li>Set the data to be deleted after this phase by toggling<strong> trash can icon<\/strong>.<\/li>\n\n\n\n<li>You can leave other options with default settings.<\/li>\n\n\n\n<li>You will need Enterprise license to be able to create a searchable snapshot in this phase.<\/li>\n<\/ul>\n\n\n\n<div><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/cold-phase-settings-1.png\" class=\"td-modal-image\"><figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1886\" height=\"1218\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/cold-phase-settings-1.png\" alt=\"\" class=\"wp-image-15871\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/cold-phase-settings-1.png?v=1679599111 1886w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/cold-phase-settings-1-768x496.png?v=1679599111 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/cold-phase-settings-1-1536x992.png?v=1679599111 1536w\" sizes=\"(max-width: 1886px) 100vw, 1886px\" \/><\/figure><\/a><\/div>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-delete-phase\">Configure Delete Phase<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Enable delete phase after cold phase.<\/li>\n\n\n\n<li>We set the data to be moved into this phase after 6 minutes. Again, the value must be greater or equal than the cold phase value.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1882\" height=\"922\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/delete-phase-settings.png\" alt=\"\" class=\"wp-image-15872\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/delete-phase-settings.png?v=1679599168 1882w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/delete-phase-settings-768x376.png?v=1679599168 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/delete-phase-settings-1536x752.png?v=1679599168 1536w\" sizes=\"(max-width: 1882px) 100vw, 1882px\" \/><\/figure>\n\n\n\n<p>Save the Policy and it should now be listed in your policies;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1888\" height=\"540\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/custom-ilm-policy-1.png\" alt=\"\" class=\"wp-image-15873\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/custom-ilm-policy-1.png?v=1679599224 1888w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/custom-ilm-policy-1-768x220.png?v=1679599224 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/custom-ilm-policy-1-1536x439.png?v=1679599224 1536w\" sizes=\"(max-width: 1888px) 100vw, 1888px\" \/><\/figure>\n\n\n\n<p>You can get the ILM policy details from command line;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -XGET localhost:9200\/_ilm\/policy\/kifarunix-demo?pretty<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>{\n  \"kifarunix-demo\" : {\n    \"version\" : 4,\n    \"modified_date\" : \"2023-03-23T19:22:16.898Z\",\n    \"policy\" : {\n      \"phases\" : {\n        \"warm\" : {\n          \"min_age\" : \"5m\",\n          \"actions\" : {\n            \"allocate\" : {\n              \"number_of_replicas\" : 0,\n              \"include\" : { },\n              \"exclude\" : { },\n              \"require\" : { }\n            }\n          }\n        },\n        \"cold\" : {\n          \"min_age\" : \"5m\",\n          \"actions\" : { }\n        },\n        \"hot\" : {\n          \"min_age\" : \"0ms\",\n          \"actions\" : {\n            \"rollover\" : {\n              \"max_primary_shard_size\" : \"1mb\",\n              \"max_age\" : \"10m\"\n            }\n          }\n        },\n        \"delete\" : {\n          \"min_age\" : \"5m\",\n          \"actions\" : {\n            \"delete\" : {\n              \"delete_searchable_snapshot\" : true\n            }\n          }\n        }\n      }\n    },\n    \"in_use_by\" : {\n      \"indices\" : [ ],\n      \"data_streams\" : [ ],\n      \"composable_templates\" : [\n        \"kifarunix-demo\"\n      ]\n    }\n  }\n}\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"create-custom-index-template\">Create Custom Index Template<\/h3>\n\n\n\n<p>An index template is a pre-defined set of rules and settings that define how new indices should be created. Later on, we will configure our new index template to use the custom ILM policy while creating new indices.<\/p>\n\n\n\n<p>Thus, navigate to Kibana menu &gt; <strong>Management &gt; Stack Management &gt; Data &gt; Index Management<\/strong> &gt;&nbsp;<strong>Index Templates &gt; Create Template<\/strong>.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Under logistics, set the name and the index pattern of the new index template.<br><img loading=\"lazy\" decoding=\"async\" width=\"1800\" height=\"795\" class=\"wp-image-15822\" style=\"width: 1800px;\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/index-template-logistics.png\" alt=\"\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/index-template-logistics.png?v=1679427564 1874w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/index-template-logistics-768x339.png?v=1679427564 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/index-template-logistics-1536x679.png?v=1679427564 1536w\" sizes=\"(max-width: 1800px) 100vw, 1800px\" \/><\/li>\n\n\n\n<li>For step 2, 3, 4, 5, we will leave the default settings on.<\/li>\n\n\n\n<li>On Step 6, review your index template settings and create it.<\/li>\n<\/ul>\n\n\n\n<div><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/review_and_create-index-template.png\" class=\"td-modal-image\"><figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1802\" height=\"853\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/review_and_create-index-template.png\" alt=\"Configure Log Retention Period in ELK Stack\" class=\"wp-image-15842\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/review_and_create-index-template.png?v=1679482295 1802w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/review_and_create-index-template-768x364.png?v=1679482295 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/review_and_create-index-template-1536x727.png?v=1679482295 1536w\" sizes=\"(max-width: 1802px) 100vw, 1802px\" \/><\/figure><\/a><\/div>\n\n\n\n<p>You index template should now be listed among other existing templates.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"apply-ilm-policy-into-the-index-template\">Apply ILM Policy into the Index Template<\/h3>\n\n\n\n<p>&#8220;<em>To use a policy that triggers the rollover action, you need to configure the policy in the index template used to create each new index. You specify the name of the policy and the alias used to reference the rolling indices<\/em>&#8220;.<\/p>\n\n\n\n<p>To apply  an ILM Policy into the Index Template, you can use either of these methods;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"#apply-ilm-policy-into-the-index-template-via-elasticsearch-api-on-elasticsearch-node-terminal\">Elasticsearch API via Elasticsearch node terminal<\/a><\/li>\n\n\n\n<li><a href=\"#apply-ilm-policy-into-the-index-template-via-kibana-dev-tools\">Elasticsearch API via Kibana Dev tools;<\/a><\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"apply-ilm-policy-into-the-index-template-via-elasticsearch-api-on-elasticsearch-node-terminal\">Apply ILM Policy into the Index Template via Elasticsearch API on Elasticsearch node terminal<\/h4>\n\n\n\n<p>Login to Elasticsearch node or any node from which you can access Elasticsearch node and execute the command below;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Replace kifarunix-demo with your respective index name\/pattern.<\/li>\n\n\n\n<li>Change IP of the Elasticsearch accordingly.<\/li>\n<\/ul>\n\n\n\n<pre class=\"scroll-box\"><code>curl -X PUT \"192.168.58.22:9200\/_index_template\/kifarunix-demo?pretty\" -H 'Content-Type: application\/json' -d'\n{\n  \"index_patterns\": [\"kifarunix-demo-*\"],\n  \"template\": {\n    \"settings\": {\n      \"number_of_shards\": 1,\n      \"number_of_replicas\": 1,\n      \"index.lifecycle.name\": \"kifarunix-demo\",\n      \"index.lifecycle.rollover_alias\": \"kifarunix-demo\"\n    }\n  }\n}\n'\n<\/code><\/pre>\n\n\n\n<p>If all is good, you should get such an output;<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>{\n  \"acknowledged\" : true\n}\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"apply-ilm-policy-into-the-index-template-via-kibana-dev-tools\">Apply ILM Policy into the Index Template via Kibana Dev Tools<\/h4>\n\n\n\n<p>Navigate to Kibana menu &gt; Stack Management &gt; Management &gt; Dev Tools &gt; Console.<\/p>\n\n\n\n<p>Paste and execute the command below;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>PUT _index_template\/kifarunix-demo?pretty\n{\n  \"index_patterns\": [\"kifarunix-demo-*\"],\n  \"template\": {\n    \"settings\": {\n      \"number_of_shards\": 1,\n      \"number_of_replicas\": 1,\n      \"index.lifecycle.name\": \"kifarunix-demo\",\n      \"index.lifecycle.rollover_alias\": \"kifarunix-demo\"\n    }\n  }\n}\n<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1890\" height=\"518\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/update-index-template-from-kibana.png\" alt=\"\" class=\"wp-image-15845\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/update-index-template-from-kibana.png?v=1679483158 1890w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/update-index-template-from-kibana-768x210.png?v=1679483158 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/update-index-template-from-kibana-1536x421.png?v=1679483158 1536w\" sizes=\"(max-width: 1890px) 100vw, 1890px\" \/><\/figure>\n\n\n\n<p>You are now ready use your custom indices with your custom policy.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"verify-ilm-policy-and-data-retention-on-elk-stack\">Verify ILM Policy and Data Retention on ELK Stack<\/h3>\n\n\n\n<p>In order to verify that our policy and rollover works as expected, we will configure a Filebeat agent to sent logs to the specific index, kifarunix-demo.<\/p>\n\n\n\n<p>Before we can proceed, it is good to note that index lifecycle management (ILM) checks the current state of indices every 10 minutes by default. This value is controlled by the setting, <strong>index.lifecycle.poll_interval<\/strong>.<\/p>\n\n\n\n<p>The default value of 10m seem reasonable and thus changing the value of this setting may have an impact on the performance of your Elasticsearch cluster, especially if you set a very short interval.<\/p>\n\n\n\n<p>If you want to change this value temporarily, e.g to 5m, then you can do this by running the following command under <strong>Kibana DevTools<\/strong> console;<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>PUT _cluster\/settings\n{\n  \"transient\": {\n    \"indices.lifecycle.poll_interval\": \"5m\"\n  }\n}\n<\/code><\/pre>\n\n\n\n<p>Or the command below on terminal on Elasticsearch node;<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>curl -XPUT \"http:\/\/localhost:9200\/_cluster\/settings\" -H 'Content-Type: application\/json' -d '{\n  \"transient\": {\n    \"indices.lifecycle.poll_interval\": \"5m\"\n  }\n}'\n<\/code><\/pre>\n\n\n\n<p>When you restart Elasticsearch, defaults are restored. To set permanent value, replace <strong><code>transient<\/code><\/strong> keyword with <strong><code>permanent<\/code><\/strong> in the above commands.<\/p>\n\n\n\n<p> Next, configure Filebeat agent to sent event data to an index alias of kifarunix-demo. This is what we use in our filebeat config;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>setup.ilm.enabled: auto\nsetup.ilm.rollover_alias: \"kifarunix-demo\"<\/code><\/pre>\n\n\n\n<p>Below is a sample filebeat.yml file contents;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>filebeat.inputs:\n- type: filestream\n  enabled: true\n  paths:\n    - \/var\/log\/auth.log\nfilebeat.config.modules:\n  path: ${path.config}\/modules.d\/*.yml\n  reload.enabled: false\nsetup.template.settings:\n  index.number_of_shards: 1\nsetup.kibana:\noutput.elasticsearch:\n  hosts: [\"192.168.58.22:9200\"]\nprocessors:\n  - add_host_metadata:\n      when.not.contains.tags: forwarded\n  - add_cloud_metadata: ~\n  - add_docker_metadata: ~\n  - add_kubernetes_metadata: ~\n<strong>setup.ilm.enabled: auto\nsetup.ilm.rollover_alias: \"kifarunix-demo\"<\/strong>\nseccomp:\n  default_action: allow\n  syscalls:\n  - action: allow\n    names:\n    - rseq\n<\/code><\/pre>\n\n\n\n<p>Start filebeat and check indices on Kibana index management;<\/p>\n\n\n\n<div><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/initial-index-created.png\" class=\"td-modal-image\"><figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1885\" height=\"607\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/initial-index-created.png\" alt=\"\" class=\"wp-image-15883\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/initial-index-created.png?v=1679693528 1885w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/initial-index-created-768x247.png?v=1679693528 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/initial-index-created-1536x495.png?v=1679693528 1536w\" sizes=\"(max-width: 1885px) 100vw, 1885px\" \/><\/figure><\/a><\/div>\n\n\n\n<p>Sample Elasticsearch logs<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>tail -f \/var\/log\/elasticsearch\/elasticsearch.log<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>...\n[2023-03-24T23:09:05,232][INFO ][o.e.c.m.MetadataCreateIndexService] [debian11] [kifarunix-demo-2023.03.24-000001] <strong>creating index<\/strong>, cause [api], templates [kifarunix-demo], shards [1]\/[1]\n[2023-03-24T23:09:05,337][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [null] to [{\"phase\":\"new\",\"action\":\"complete\",\"name\":\"complete\"}] in policy [kifarunix-demo]\n[2023-03-24T23:09:05,534][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"new\",\"action\":\"complete\",\"name\":\"complete\"}] to [{\"phase\":\"hot\",\"action\":\"unfollow\",\"name\":\"branch-check-unfollow-prerequisites\"}] in policy [kifarunix-demo]\n[2023-03-24T23:09:05,616][INFO ][o.e.c.m.MetadataMappingService] [debian11] [kifarunix-demo-2023.03.24-000001\/W8R1iYLeSCiGoQ4Cg72UHg] create_mapping [_doc]\n[2023-03-24T23:09:05,711][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"hot\",\"action\":\"unfollow\",\"name\":\"branch-check-unfollow-prerequisites\"}] to [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"check-rollover-ready\"}] in policy [kifarunix-demo]\n...\n<\/code><\/pre>\n\n\n\n<p>You can use <code><strong>GET &lt;target&gt;\/_ilm\/explain<\/strong><\/code> to show the current lifecycle status for one or more indices.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>GET kifarunix-demo\/_ilm\/explain<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\n{\n  \"indices\" : {\n    \"kifarunix-demo-2023.03.24-000001\" : {\n      \"index\" : \"kifarunix-demo-2023.03.24-000001\",\n      \"managed\" : true,\n      \"policy\" : \"kifarunix-demo\",\n      \"lifecycle_date_millis\" : 1679688545230,\n      \"age\" : \"4.83m\",\n      \"phase\" : \"hot\",\n      \"phase_time_millis\" : 1679688545397,\n      \"action\" : \"rollover\",\n      \"action_time_millis\" : 1679688545597,\n      \"step\" : \"check-rollover-ready\",\n      \"step_time_millis\" : 1679688545597,\n      \"phase_execution\" : {\n        \"policy\" : \"kifarunix-demo\",\n        \"phase_definition\" : {\n          \"min_age\" : \"0ms\",\n          \"actions\" : {\n            \"rollover\" : {\n              \"max_primary_shard_size\" : \"1mb\",\n              \"max_age\" : \"10m\"\n            }\n          }\n        },\n        \"version\" : 1,\n        \"modified_date_in_millis\" : 1679688094268\n      }\n    }\n  }\n}\n<\/code><\/pre>\n\n\n\n<p>As you can see from the above information, the index is just created.<\/p>\n\n\n\n<p>It is good to note that the rollover may not happen at the exact time specified in the policy due to several factors, such as the size of the data being indexed, the load on the Elasticsearch cluster, and any other factors that may delay processing. But eventually, the rollover will happen based on the size of index\/max age of the primary shard.<\/p>\n\n\n\n<p>For example, as per the logs , rollover happened in like 14 minutes after we saw a log about <strong><code>creating index<\/code><\/strong> in the logs above;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n[2023-03-24T23:23:10,329][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"check-rollover-ready\"}] to [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"attempt-rollover\"}] in policy [kifarunix-demo]\n<strong>[2023-03-24T23:23:10,439][INFO ][o.e.c.m.MetadataCreateIndexService] [debian11] [kifarunix-demo-2023.03.24-000002] creating index, cause [rollover_index], templates [kifarunix-demo], shards [1]\/[1]<\/strong>\n[2023-03-24T23:23:10,521][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [null] to [{\"phase\":\"new\",\"action\":\"complete\",\"name\":\"complete\"}] in policy [kifarunix-demo]\n[2023-03-24T23:23:10,522][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"attempt-rollover\"}] to [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"wait-for-active-shards\"}] in policy [kifarunix-demo]\n[2023-03-24T23:23:10,710][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [{\"phase\":\"new\",\"action\":\"complete\",\"name\":\"complete\"}] to [{\"phase\":\"hot\",\"action\":\"unfollow\",\"name\":\"branch-check-unfollow-prerequisites\"}] in policy [kifarunix-demo]\n[2023-03-24T23:23:10,710][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"wait-for-active-shards\"}] to [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"update-rollover-lifecycle-date\"}] in policy [kifarunix-demo]\n[2023-03-24T23:23:10,711][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"update-rollover-lifecycle-date\"}] to [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"set-indexing-complete\"}] in policy [kifarunix-demo]\n[2023-03-24T23:23:10,871][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [{\"phase\":\"hot\",\"action\":\"unfollow\",\"name\":\"branch-check-unfollow-prerequisites\"}] to [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"check-rollover-ready\"}] in policy [kifarunix-demo]\n[2023-03-24T23:23:10,872][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"set-indexing-complete\"}] to [{\"phase\":\"hot\",\"action\":\"complete\",\"name\":\"complete\"}] in policy [kifarunix-demo]\n[2023-03-24T23:23:11,098][INFO ][o.e.c.m.MetadataMappingService] [debian11] [kifarunix-demo-2023.03.24-000002\/dz3BAJJMTcixNJANLQrY8A] create_mapping [_doc]\n<\/code><\/pre>\n\n\n\n<div><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/indices-after-rollover.png\" class=\"td-modal-image\"><figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1878\" height=\"604\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/indices-after-rollover.png\" alt=\"Configure Log Retention Period in ELK Stack\" class=\"wp-image-15885\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/indices-after-rollover.png?v=1679693682 1878w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/indices-after-rollover-768x247.png?v=1679693682 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/indices-after-rollover-1536x494.png?v=1679693682 1536w\" sizes=\"(max-width: 1878px) 100vw, 1878px\" \/><\/figure><\/a><\/div>\n\n\n\n<p>After the rollover, new index <strong><code>kifarunix-demo-2023.03.24-000002<\/code><\/strong> is created. Both indices will stay in the hot phase until the next check conditions are met.<\/p>\n\n\n\n<p>The rolled-over index, <strong><code>kifarunix-demo-2023.03.24-000001<\/code><\/strong>, will stay for 5 minutes in rolled-over state or the minimum age defined in the policy, after which it is moved to <strong><code>warm phase<\/code><\/strong>.<\/p>\n\n\n\n<p>Since our poll interval is set to 10 minutes, by the time the next check happens, the rolled-over index, kifarunix-demo-2023.03.24-000001, will be older than the <strong><code>min_age<\/code><\/strong> (5 minutes) defined in the policy (in our case) and thus, rollover to <strong><code>warm phase<\/code><\/strong> happens.<\/p>\n\n\n\n<p>Similarly, it will have also completed the <code><strong>min_age<\/strong><\/code> (5 mins) for moving to <strong><code>cold phase<\/code><\/strong>. Hence, it will be moved to <strong><code>warm phase<\/code><\/strong> then to <code><strong>cold phase<\/strong><\/code> immediately.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n[2023-03-24T23:33:10,327][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] <strong>from [{\"phase\":\"hot<\/strong>\",\"action\":\"complete\",\"name\":\"complete\"}] <strong>to [{\"phase\":\"warm<\/strong>\",\"action\":\"allocate\",\"name\":\"allocate\"}] in policy [kifarunix-demo]\n[2023-03-24T23:33:10,432][INFO ][o.e.c.m.MetadataUpdateSettingsService] [debian11] updating number_of_replicas to [0] for indices [kifarunix-demo-2023.03.24-000001]\n[2023-03-24T23:33:10,516][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"warm\",\"action\":\"allocate\",\"name\":\"allocate\"}] to [{\"phase\":\"warm\",\"action\":\"allocate\",\"name\":\"check-allocation\"}] in policy [kifarunix-demo]\n[2023-03-24T23:33:10,603][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"warm\",\"action\":\"allocate\",\"name\":\"check-allocation\"}] to [{\"phase\":\"warm\",\"action\":\"migrate\",\"name\":\"branch-check-skip-action\"}] in policy [kifarunix-demo]\n[2023-03-24T23:33:10,605][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"warm\",\"action\":\"migrate\",\"name\":\"branch-check-skip-action\"}] to [{\"phase\":\"warm\",\"action\":\"migrate\",\"name\":\"migrate\"}] in policy [kifarunix-demo]\n[2023-03-24T23:33:10,782][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"warm\",\"action\":\"migrate\",\"name\":\"migrate\"}] to [{\"phase\":\"warm\",\"action\":\"migrate\",\"name\":\"check-migration\"}] in policy [kifarunix-demo]\n<strong>[2023-03-24T23:33:10,872][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"warm\",\"action\":\"migrate\",\"name\":\"check-migration\"}] to [{\"phase\":\"warm\",\"action\":\"complete\",\"name\":\"complete\"}] in policy [kifarunix-demo]<\/strong>\n[2023-03-24T23:33:10,965][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] <strong>from [{\"phase\":\"warm\"<\/strong>,\"action\":\"complete\",\"name\":\"complete\"}] <strong>to [{\"phase\":\"cold\"<\/strong>,\"action\":\"allocate\",\"name\":\"allocate\"}] in policy [kifarunix-demo]\n[2023-03-24T23:33:11,050][INFO ][o.e.c.m.MetadataUpdateSettingsService] [debian11] updating number_of_replicas to [0] for indices [kifarunix-demo-2023.03.24-000001]\n[2023-03-24T23:33:11,050][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"cold\",\"action\":\"allocate\",\"name\":\"allocate\"}] to [{\"phase\":\"cold\",\"action\":\"allocate\",\"name\":\"check-allocation\"}] in policy [kifarunix-demo]\n[2023-03-24T23:33:11,140][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"cold\",\"action\":\"allocate\",\"name\":\"check-allocation\"}] to [{\"phase\":\"cold\",\"action\":\"migrate\",\"name\":\"branch-check-skip-action\"}] in policy [kifarunix-demo]\n[2023-03-24T23:33:11,140][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"cold\",\"action\":\"migrate\",\"name\":\"branch-check-skip-action\"}] to [{\"phase\":\"cold\",\"action\":\"migrate\",\"name\":\"migrate\"}] in policy [kifarunix-demo]\n[2023-03-24T23:33:11,333][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"cold\",\"action\":\"migrate\",\"name\":\"migrate\"}] to [{\"phase\":\"cold\",\"action\":\"migrate\",\"name\":\"check-migration\"}] in policy [kifarunix-demo]\n<strong>[2023-03-24T23:33:11,434][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"cold\",\"action\":\"migrate\",\"name\":\"check-migration\"}] to [{\"phase\":\"cold\",\"action\":\"complete\",\"name\":\"complete\"}] in policy [kifarunix-demo]<\/strong>\n[2023-03-24T23:33:11,523][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"cold\",\"action\":\"complete\",\"name\":\"complete\"}] to [{\"phase\":\"delete\",\"action\":\"delete\",\"name\":\"wait-for-shard-history-leases\"}] in policy [kifarunix-demo]\n<\/code><\/pre>\n\n\n\n<p>So, as per our policy, the index <strong><code>kifarunix-demo-2023.03.24-000001<\/code><\/strong>, stays in <strong>cold phase<\/strong> 5 minutes before being moved to <strong>delete phase<\/strong>. At the same time, the second index is rolled-over and process continues.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n[2023-03-24T23:38:10,326][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"check-rollover-ready\"}] to [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"attempt-rollover\"}] in policy [kifarunix-demo]\n<strong>[2023-03-24T23:38:10,415][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"delete\",\"action\":\"delete\",\"name\":\"wait-for-shard-history-leases\"}] to [{\"phase\":\"delete\",\"action\":\"delete\",\"name\":\"cleanup-snapshot\"}] in policy [kifarunix-demo]<\/strong>\n[2023-03-24T23:38:10,492][INFO ][o.e.c.m.MetadataCreateIndexService] [debian11] [kifarunix-demo-2023.03.24-000003] creating index, cause [rollover_index], templates [kifarunix-demo], shards [1]\/[1]\n<strong>[2023-03-24T23:38:10,580][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{\"phase\":\"delete\",\"action\":\"delete\",\"name\":\"cleanup-snapshot\"}] to [{\"phase\":\"delete\",\"action\":\"delete\",\"name\":\"delete\"}] in policy [kifarunix-demo]<\/strong>\n[2023-03-24T23:38:10,581][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000003] from [null] to [{\"phase\":\"new\",\"action\":\"complete\",\"name\":\"complete\"}] in policy [kifarunix-demo]\n[2023-03-24T23:38:10,583][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"attempt-rollover\"}] to [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"wait-for-active-shards\"}] in policy [kifarunix-demo]\n<strong>[2023-03-24T23:38:10,764][INFO ][o.e.c.m.MetadataDeleteIndexService] [debian11] [kifarunix-demo-2023.03.24-000001\/W8R1iYLeSCiGoQ4Cg72UHg] deleting index<\/strong>\n[2023-03-24T23:38:10,853][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000003] from [{\"phase\":\"new\",\"action\":\"complete\",\"name\":\"complete\"}] to [{\"phase\":\"hot\",\"action\":\"unfollow\",\"name\":\"branch-check-unfollow-prerequisites\"}] in policy [kifarunix-demo]\n[2023-03-24T23:38:10,853][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"wait-for-active-shards\"}] to [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"update-rollover-lifecycle-date\"}] in policy [kifarunix-demo]\n[2023-03-24T23:38:10,854][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"update-rollover-lifecycle-date\"}] to [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"set-indexing-complete\"}] in policy [kifarunix-demo]\n[2023-03-24T23:38:11,033][INFO ][o.e.c.m.MetadataMappingService] [debian11] [kifarunix-demo-2023.03.24-000003\/3y03bVJHQkOXnOLsBWURPw] create_mapping [_doc]\n[2023-03-24T23:38:11,126][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000003] from [{\"phase\":\"hot\",\"action\":\"unfollow\",\"name\":\"branch-check-unfollow-prerequisites\"}] to [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"check-rollover-ready\"}] in policy [kifarunix-demo]\n[2023-03-24T23:38:11,126][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"set-indexing-complete\"}] to [{\"phase\":\"hot\",\"action\":\"complete\",\"name\":\"complete\"}] in policy [kifarunix-demo]\n<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1888\" height=\"644\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/index-after-deletion-phase.png\" alt=\"\" class=\"wp-image-15886\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/index-after-deletion-phase.png?v=1679693773 1888w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/index-after-deletion-phase-768x262.png?v=1679693773 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/03\/index-after-deletion-phase-1536x524.png?v=1679693773 1536w\" sizes=\"(max-width: 1888px) 100vw, 1888px\" \/><\/figure>\n\n\n\n<p>So, the first index has completed the cycle and has been deleted!<\/p>\n\n\n\n<p>And that should suffice to configure log retention period in ELK Stack. Just ensure that you set the right age as per your compliance requirements.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"other-tutorials\">Other Tutorials<\/h3>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/how-to-integrate-elk-stack-with-thehive\/\" target=\"_blank\" rel=\"noreferrer noopener\">How to Integrate ELK Stack with TheHive<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/create-custom-elk-ingest-pipeline-for-custom-log-processing\/\" target=\"_blank\" rel=\"noreferrer noopener\">Create Custom ELK Ingest Pipeline for Custom Log Processing<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>How do I change the log retention policy in elk stack? In this tutorial, you will learn how to configure log retention period in ELK<\/p>\n","protected":false},"author":1,"featured_media":15894,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[72,910,121],"tags":[6465,6464,6463,6467,6466,6462],"class_list":["post-14356","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-monitoring","category-elastic-stack","category-howtos","tag-configure-log-retention-period-in-elk-stack","tag-data-retention","tag-elk-data-retention-period","tag-how-do-i-change-the-log-retention-policy-in-elk-stack","tag-how-do-i-set-log-retention-in-elasticsearch","tag-log-retention-policy","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/14356"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=14356"}],"version-history":[{"count":47,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/14356\/revisions"}],"predecessor-version":[{"id":20702,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/14356\/revisions\/20702"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/15894"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=14356"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=14356"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=14356"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}