{"id":14259,"date":"2022-10-06T15:17:13","date_gmt":"2022-10-06T12:17:13","guid":{"rendered":"https:\/\/kifarunix.com\/?p=14259"},"modified":"2024-03-09T23:01:28","modified_gmt":"2024-03-09T20:01:28","slug":"create-custom-elk-ingest-pipeline-for-custom-log-processing","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/create-custom-elk-ingest-pipeline-for-custom-log-processing\/","title":{"rendered":"Create Custom ELK Ingest Pipeline for Custom Log Processing"},"content":{"rendered":"\n<p>Follow through this tutorial to learn how create custom ELK ingest pipeline for custom log processing. Elastic Stack is so flexible that it can give you ability to create your own custom pipeline processors to parse your custom logs. <a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/ingest.html#ingest\" target=\"_blank\" rel=\"noreferrer noopener\">Elastic ingest pipelines<\/a> &#8220;<em>let you perform common transformations on your data before indexing. For example, you can use pipelines to remove fields, extract values from text, and enrich your data<\/em>&#8220;.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#create-custom-elk-ingest-pipeline-to-process-custom-log\">Create Custom ELK Ingest Pipeline to Process Custom Log<\/a><ul><li><a href=\"#create-custom-elk-ingest-pipeline-using-kibanas-ingest-pipelines-feature\">Create Custom ELK Ingest Pipeline using Kibana\u2019s\u00a0Ingest Pipelines\u00a0feature<\/a><ul><li><a href=\"#create-grok-filters-to-parse-your-custom-logs\">Create Grok Filters to Parse your Custom Logs<\/a><\/li><li><a href=\"#create-new-pipeline-for-custom-log\">Create New Pipeline for Custom Log<\/a><\/li><\/ul><\/li><li><a href=\"#configure-filebeat-to-ship-custom-logs-to-custom-ingest-pipeline\">Configure Filebeat to Ship Custom Logs to Custom Ingest Pipeline<\/a><\/li><li><a href=\"#other-tutorials\">Other Tutorials<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"create-custom-elk-ingest-pipeline-to-process-custom-log\">Create Custom ELK Ingest Pipeline to Process Custom Log<\/h2>\n\n\n\n<p>In this tutorial, we will create custom ELK ingest pipeline to process Modsecurity logs.<\/p>\n\n\n\n<p>Below is a sample Modsecurity Log we will be working with;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;Wed Oct 05 18:37:22.744204 2022] &#91;:error] &#91;pid 12683:tid 139658067420736] &#91;client 192.168.56.124:59696] &#91;client 192.168.56.124] ModSecurity: Access denied with code 403 (phase 2). Matched phrase \"nikto\" at REQUEST_HEADERS:User-Agent. &#91;file \"\/etc\/modsecurity\/crs\/rules\/REQUEST-913-SCANNER-DETECTION.conf\"] &#91;line \"56\"] &#91;id \"913100\"] &#91;msg \"Found User-Agent associated with security scanner\"] &#91;data \"Matched Data: nikto found within REQUEST_HEADERS:User-Agent: mozilla\/5.00 (nikto\/2.1.5) (evasions:none) (test:000562)\"] &#91;severity \"CRITICAL\"] &#91;ver \"OWASP_CRS\/3.2.0\"] &#91;tag \"application-multi\"] &#91;tag \"language-multi\"] &#91;tag \"platform-multi\"] &#91;tag \"attack-reputation-scanner\"] &#91;tag \"paranoia-level\/1\"] &#91;tag \"OWASP_CRS\"] &#91;tag \"OWASP_CRS\/AUTOMATION\/SECURITY_SCANNER\"] &#91;tag \"WASCTC\/WASC-21\"] &#91;tag \"OWASP_TOP_10\/A7\"] &#91;tag \"PCI\/6.5.10\"] &#91;hostname \"sales.kifarunix.com\"] &#91;uri \"\/index.php\"] &#91;unique_id \"Yz3O4pMZhpOcYpdhYgoXwQAAAEs\"]<\/code><\/pre>\n\n\n\n<p>There are different ways in which you can create custom ELK ingest pipeline;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"http:\/\/create-ingest-pipeline-via-kibana\" target=\"_blank\">Using Kibana\u2019s&nbsp;Ingest Pipelines&nbsp;feature<\/a><\/li>\n\n\n\n<li>Using&nbsp;ELK stack ingest APIs<\/li>\n<\/ul>\n\n\n\n<p>Before you can proceed to create custom ELK ingest pipeline, be sure to check the ingest pipeline prerequisites:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>To use ingest pipelines, your cluster must have at least one node with the&nbsp;<code>ingest<\/code>&nbsp;role. For heavy ingest loads, it is recommend to have a dedicated ingest nodes.<\/li>\n\n\n\n<li>If the Elasticsearch security features are enabled, you must have the&nbsp;<strong>manage_pipeline<\/strong>&nbsp;cluster privilege&nbsp;to manage ingest pipelines. <\/li>\n\n\n\n<li>You also need the&nbsp;<code>cluster:monitor\/nodes\/info<\/code>&nbsp;cluster privileges use Kibana\u2019s&nbsp;<strong>Ingest Pipelines<\/strong>&nbsp;feature<\/li>\n\n\n\n<li>Pipelines including the&nbsp;<a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/ingest-enriching-data.html\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>enrich<\/strong><\/a>&nbsp;processor require additional setup<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"create-custom-elk-ingest-pipeline-using-kibanas-ingest-pipelines-feature\">Create Custom ELK Ingest Pipeline using Kibana\u2019s&nbsp;Ingest Pipelines&nbsp;feature<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-grok-filters-to-parse-your-custom-logs\">Create Grok Filters to Parse your Custom Logs<\/h4>\n\n\n\n<p>One of the ways of parsing a custom log is by extracting the fields of your preference. You can use grok patterns or anything that can get the job done for you;<\/p>\n\n\n\n<p>Here is our grok pattern for extracting fields from the ModSecurity logs.<\/p>\n\n\n\n<p><strong>We assume that the Log will always have the same pattern. Thus, develop your grok pattern accordingly.<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\\&#91;(?&lt;event_date&gt;%{DAY}\\s+%{MONTH}\\s+%{MONTHDAY}\\s+%{TIME}\\s+%{YEAR})\\]\\s+\\&#91;\\:(?&lt;log_level&gt;\\w+)\\]\\s+\\S+.+client\\s+(?&lt;source_ip&gt;%{IP})\\]\\s+(?&lt;error_message&gt;ModSecurity\\S+.+code\\s+(?&lt;status_code&gt;%{INT}).+)\\s+\\&#91;file\\s+\\\"(?&lt;rules_file&gt;\\S+.+)\\\"\\]\\s+\\&#91;line\\s+\\\"(?&lt;rule_line_num&gt;%{INT})\\\"\\]\\s+\\&#91;id\\s+\\\"(?&lt;rule_id&gt;%{INT})\\\"\\]\\s+\\&#91;msg\\s+\\\"(?&lt;msg&gt;\\S+.+)\\\"\\]\\s+\\&#91;data\\s+\\\"(?&lt;data&gt;\\S+.+)\\\"\\]\\s+\\&#91;severity\\s+\\\"(?&lt;severity&gt;\\w+)\\\"\\]\\s+\\&#91;ver\\s+\\\"(?&lt;owasp_crs_version&gt;\\S+)\\\"\\]\\s+(?&lt;tags&gt;\\S+.+)\\s+\\&#91;hostname\\s+\\\"(?&lt;hostname&gt;%{IPORHOST})\\\"\\]\\s+\\&#91;uri\\s+\\\"(?&lt;uri&gt;\/|\\S+.+)\\\"\\]\\s+\\&#91;unique_id\\s+\\\"(?&lt;unique_id&gt;\\S+.+)\\\"\\]<\/code><\/pre>\n\n\n\n<p>These are the fields as shown on Kibana Grok debugger;<\/p>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2022\\\/10\\\/modsecurity-log-grok-pattern.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-14331&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1863,&quot;targetHeight&quot;:876,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image&quot;,&quot;alt&quot;:&quot;&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1863\" height=\"876\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/modsecurity-log-grok-pattern.png\" alt=\"\" class=\"wp-image-14331\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/modsecurity-log-grok-pattern.png?v=1665049568 1863w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/modsecurity-log-grok-pattern-768x361.png?v=1665049568 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/modsecurity-log-grok-pattern-1536x722.png?v=1665049568 1536w\" sizes=\"(max-width: 1863px) 100vw, 1863px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<p>For this grok pattern to be accepted by the processor, it needs to be properly escaped by preceding brackets with double backslashes (<code>\\\\[<\/code>), (<code>\\\\]<\/code>), colon by double backslashes (<code>\\\\:<\/code>) and double quotes (<strong>&#8220;<\/strong>) with triple backslashes (<code>\\\\\\\"<\/code>).<\/p>\n\n\n\n<p>You can replace all \\s+ with <code>%{SPACE}+<\/code>, \\S+ (<code>%{NOTSPACE}+<\/code>), \\d (<code>%{INT}<\/code>), \\w (<code>%{WORD}<\/code>) e.tc.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\\\\&#91;(?&lt;event_date&gt;%{DAY}%{SPACE}+%{MONTH}%{SPACE}+%{MONTHDAY}%{SPACE}+%{TIME}%{SPACE}+%{YEAR})\\\\]%{SPACE}+\\\\&#91;\\\\:(?&lt;log_level&gt;%{WORD}+)\\\\]%{SPACE}+%{NOTSPACE}+.+client%{SPACE}+(?&lt;source_ip&gt;%{IP})\\\\]%{SPACE}+(?&lt;error_message&gt;ModSecurity%{NOTSPACE}+.+code%{SPACE}+(?&lt;status_code&gt;%{INT}).+)%{SPACE}+\\\\&#91;file%{SPACE}+\\\\\\\"(?&lt;rules_file&gt;%{NOTSPACE}+.+)\\\\\\\"\\\\]%{SPACE}+\\\\&#91;line%{SPACE}+\\\\\\\"(?&lt;rule_line_num&gt;%{INT})\\\\\\\"\\\\]%{SPACE}+\\\\&#91;id%{SPACE}+\\\\\\\"(?&lt;rule_id&gt;%{INT})\\\\\\\"\\\\]%{SPACE}+\\\\&#91;msg%{SPACE}+\\\\\\\"(?&lt;msg&gt;%{NOTSPACE}+.+)\\\\\\\"\\\\]%{SPACE}+\\\\&#91;data%{SPACE}+\\\\\\\"(?&lt;data&gt;%{NOTSPACE}+.+)\\\\\\\"\\\\]%{SPACE}+\\\\&#91;severity%{SPACE}+\\\\\\\"(?&lt;severity&gt;%{WORD}+)\\\\\\\"\\\\]%{SPACE}+\\\\&#91;ver%{SPACE}+\\\\\\\"(?&lt;owasp_crs_version&gt;%{NOTSPACE}+)\\\\\\\"\\\\]%{SPACE}+(?&lt;tags&gt;%{NOTSPACE}+.+)%{SPACE}+\\\\&#91;hostname%{SPACE}+\\\\\\\"(?&lt;hostname&gt;%{IPORHOST})\\\\\\\"\\\\]%{SPACE}+\\\\&#91;uri%{SPACE}+\\\\\\\"(?&lt;uri&gt;\/|%{NOTSPACE}+.+)\\\\\\\"\\\\]%{SPACE}+\\\\&#91;unique_id%{SPACE}+\\\\\\\"(?&lt;unique_id&gt;%{NOTSPACE}+.+)\\\\\\\"\\\\]<\/code><\/pre>\n\n\n\n<p>If not well escaped, you will get such an error as <strong><code>Invalid JSON String<\/code><\/strong>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-new-pipeline-for-custom-log\">Create New Pipeline for Custom Log<\/h4>\n\n\n\n<p>Once you have grok pattern\/filter for your custom log;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Navigate to <strong>Kibana &gt; main menu &gt; Management &gt; Stack Management &gt; Ingest &gt; Ingest Pipelines<\/strong>.<\/li>\n<\/ul>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2022\\\/10\\\/ingest-pipelines.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-14339&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1905,&quot;targetHeight&quot;:745,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image: Create Custom ELK Ingest Pipeline&quot;,&quot;alt&quot;:&quot;Create Custom ELK Ingest Pipeline&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1905\" height=\"745\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/ingest-pipelines.png\" alt=\"Create Custom ELK Ingest Pipeline\" class=\"wp-image-14339\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/ingest-pipelines.png?v=1665055446 1905w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/ingest-pipelines-768x300.png?v=1665055446 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/ingest-pipelines-1536x601.png?v=1665055446 1536w\" sizes=\"(max-width: 1905px) 100vw, 1905px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image: Create Custom ELK Ingest Pipeline\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Click <strong>Create Pipeline<\/strong>\n<ul class=\"wp-block-list\">\n<li>Enter the name of the pipeline<\/li>\n\n\n\n<li>Optionally add version number and description of the pipeline<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2022\\\/10\\\/ingest-pipeline-name.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-14341&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1896,&quot;targetHeight&quot;:658,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image&quot;,&quot;alt&quot;:&quot;&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1896\" height=\"658\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/ingest-pipeline-name.png\" alt=\"\" class=\"wp-image-14341\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/ingest-pipeline-name.png?v=1665055552 1896w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/ingest-pipeline-name-768x267.png?v=1665055552 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/ingest-pipeline-name-1536x533.png?v=1665055552 1536w\" sizes=\"(max-width: 1896px) 100vw, 1896px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Scroll down under <strong>Processors<\/strong>, and <strong>add a processor<\/strong> to use for transforming the custom log before indexing.<\/li>\n\n\n\n<li>We are using a Grok processor in this example.\n<ul class=\"wp-block-list\">\n<li>Hence, select appropriate <strong>processor<\/strong> for extracting the fields from the custom log, <strong>Grok<\/strong>.<\/li>\n\n\n\n<li><strong>Field<\/strong>: use <strong>message<\/strong>.<\/li>\n\n\n\n<li><strong>Patterns<\/strong>: Paste your well escaped custom log grok pattern. If not escaped well, you will get an error, <strong>Invalid JSON String<\/strong>.<\/li>\n\n\n\n<li>The rest of the settings are optional<\/li>\n\n\n\n<li>Click <strong>Add<\/strong>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2022\\\/10\\\/configure-processor.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-14340&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1892,&quot;targetHeight&quot;:871,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image&quot;,&quot;alt&quot;:&quot;&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1892\" height=\"871\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/configure-processor.png\" alt=\"\" class=\"wp-image-14340\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/configure-processor.png?v=1665055486 1892w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/configure-processor-768x354.png?v=1665055486 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/configure-processor-1536x707.png?v=1665055486 1536w\" sizes=\"(max-width: 1892px) 100vw, 1892px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Next, based on our custom fields, we need to convert the <strong>source_ip<\/strong> to GeoIP field. Hence, click <strong>Add a processor<\/strong>.<\/li>\n<\/ul>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2022\\\/10\\\/grok-pattern-extract-fields.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-14343&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1898,&quot;targetHeight&quot;:884,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image&quot;,&quot;alt&quot;:&quot;&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1898\" height=\"884\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/grok-pattern-extract-fields.png\" alt=\"\" class=\"wp-image-14343\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/grok-pattern-extract-fields.png?v=1665055811 1898w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/grok-pattern-extract-fields-768x358.png?v=1665055811 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/grok-pattern-extract-fields-1536x715.png?v=1665055811 1536w\" sizes=\"(max-width: 1898px) 100vw, 1898px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Processor: select <strong>GeoIP<\/strong><\/li>\n\n\n\n<li>Field: Enter your custom source IP field, in this example is source_ip.<\/li>\n\n\n\n<li>Leave other options default\/<\/li>\n\n\n\n<li>You can disable <strong>First only.<\/strong><\/li>\n\n\n\n<li>You can enable <strong>Ignore missing<\/strong> fields.<\/li>\n<\/ul>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2022\\\/10\\\/source-ip.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-14344&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1607,&quot;targetHeight&quot;:876,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image: configure ingest processor&quot;,&quot;alt&quot;:&quot;configure ingest processor&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1607\" height=\"876\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/source-ip.png\" alt=\"configure ingest processor\" class=\"wp-image-14344\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/source-ip.png?v=1665055876 1607w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/source-ip-768x419.png?v=1665055876 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/source-ip-1536x837.png?v=1665055876 1536w\" sizes=\"(max-width: 1607px) 100vw, 1607px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image: configure ingest processor\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Click <strong>Add<\/strong>.<\/li>\n\n\n\n<li>Your custom log processors now look like;<\/li>\n<\/ul>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2022\\\/10\\\/custom-processors.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-14346&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1864,&quot;targetHeight&quot;:620,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image&quot;,&quot;alt&quot;:&quot;&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1864\" height=\"620\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/custom-processors.png\" alt=\"\" class=\"wp-image-14346\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/custom-processors.png?v=1665056016 1864w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/custom-processors-768x255.png?v=1665056016 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/custom-processors-1536x511.png?v=1665056016 1536w\" sizes=\"(max-width: 1864px) 100vw, 1864px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Next, you need to configure the processor to process only specific logs that matches the pattern and drop the rest. Remember, in this example, the ModSecurity logs are written to Apache, which will contain other web server logs that will not match the pattern created. Hence, you can drop all messages that the pattern fails to parse.<\/li>\n\n\n\n<li>Therefore, under <strong>Failure processors<\/strong>, Add a processor;<\/li>\n<\/ul>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2022\\\/10\\\/drop-unparsed-events.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-14347&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1792,&quot;targetHeight&quot;:878,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image&quot;,&quot;alt&quot;:&quot;&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1792\" height=\"878\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/drop-unparsed-events.png\" alt=\"\" class=\"wp-image-14347\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/drop-unparsed-events.png?v=1665056152 1792w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/drop-unparsed-events-768x376.png?v=1665056152 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/drop-unparsed-events-1536x753.png?v=1665056152 1536w\" sizes=\"(max-width: 1792px) 100vw, 1792px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Select <strong>Drop<\/strong> processor<\/li>\n\n\n\n<li>Leave other options with defaults<\/li>\n\n\n\n<li>Click <strong>Add<\/strong>.<\/li>\n<\/ul>\n\n\n\n<p>Next;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Click <strong>Create Pipeline<\/strong> to create your custom log pipeline processor.<\/li>\n\n\n\n<li>Your pipeline should now appear under Ingest pipelines;<\/li>\n<\/ul>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2022\\\/10\\\/custom-log-pipeline-modsecurity.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-14349&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1901,&quot;targetHeight&quot;:697,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image: Custom ELK Ingest Pipeline&quot;,&quot;alt&quot;:&quot;Custom ELK Ingest Pipeline&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1901\" height=\"697\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/custom-log-pipeline-modsecurity.png\" alt=\"Custom ELK Ingest Pipeline\" class=\"wp-image-14349\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/custom-log-pipeline-modsecurity.png?v=1665056212 1901w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/custom-log-pipeline-modsecurity-768x282.png?v=1665056212 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/custom-log-pipeline-modsecurity-1536x563.png?v=1665056212 1536w\" sizes=\"(max-width: 1901px) 100vw, 1901px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image: Custom ELK Ingest Pipeline\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configure-filebeat-to-ship-custom-logs-to-custom-ingest-pipeline\">Configure Filebeat to Ship Custom Logs to Custom Ingest Pipeline<\/h3>\n\n\n\n<p>Next, you need to configure your data shippers, in our case, Filebeat to sent custom logs to the custom ingest pipeline.<\/p>\n\n\n\n<p>You can refer to our various tutorials on how to install filebeat;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/?s=install+filebeat\" target=\"_blank\" rel=\"noreferrer noopener\">How to install Filebeat Data Shipper<\/a><\/p>\n\n\n\n<p>In our web server (Ubuntu 22.04), we have deployed filebeat v7.17.0.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/filebeat\/filebeat.yml<\/code><\/pre>\n\n\n\n<p>We are reading the Modsecurity logs from the Apache Web server error logs using the filestream input type;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code><strong>filebeat.inputs:\n- type: filestream\n  enabled: true\n  paths:\n    - \/var\/log\/apache2\/error.log<\/strong>\n<\/code><\/pre>\n\n\n\n<p>To configure Filebeat to ship data to specific ingest pipeline, you need to add the pipeline option and the name of the pipeline under Elasticsearch output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>output.elasticsearch:\n  hosts: [\"192.168.58.22:9200\"]\n <strong> pipeline: modsecurity_logs<\/strong>\n<\/code><\/pre>\n\n\n\n<p>In general, this is our sample Filebeat configuration file;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>filebeat.inputs:\n- type: filestream\n  enabled: true\n  paths:\n    - \/var\/log\/apache2\/error.log\nfilebeat.config.modules:\n  path: ${path.config}\/modules.d\/*.yml\n  reload.enabled: false\nsetup.template.settings:\n  index.number_of_shards: 1\nsetup.kibana:\noutput.elasticsearch:\n  hosts: [\"192.168.58.22:9200\"]\n<strong>  pipeline: modsecurity_logs\n<\/strong>processors:\n  - add_host_metadata:\n      when.not.contains.tags: forwarded\nseccomp:\n  default_action: allow\n  syscalls:\n  - action: allow\n    names:\n    - rseq\n<\/code><\/pre>\n\n\n\n<p>With this config, Filebeat will create and use the default filebeat index on Elasticsearch. You can define your own custom index if you want.<\/p>\n\n\n\n<p>Save and file and exit.<\/p>\n\n\n\n<p>Test the configuration if all good.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat test config<\/code><\/pre>\n\n\n\n<p>Ensure the output is <strong>Config OK<\/strong>.<\/p>\n\n\n\n<p>You can also check connection to Elasticsearch;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat test output<\/code><\/pre>\n\n\n\n<p>Start Filebeat;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl enable --now filebeat<\/code><\/pre>\n\n\n\n<p>If the logs are actively being written to the log file, Filebeat should have read, collect them and pushed them to the Elasticsearch index for processing.<\/p>\n\n\n\n<p>In our setup, the logs are written to Filebeat index pattern.<\/p>\n\n\n\n<p>You can view the data on Kibana Discover menu;<\/p>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2022\\\/10\\\/ingest-pipeline-processed-logs.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-14336&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1892,&quot;targetHeight&quot;:846,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image: Create Custom ELK Ingest Pipeline for Custom Log Processing&quot;,&quot;alt&quot;:&quot;Create Custom ELK Ingest Pipeline for Custom Log Processing&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1892\" height=\"846\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/ingest-pipeline-processed-logs.png\" alt=\"Create Custom ELK Ingest Pipeline for Custom Log Processing\" class=\"wp-image-14336\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/ingest-pipeline-processed-logs.png?v=1665055261 1892w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/ingest-pipeline-processed-logs-768x343.png?v=1665055261 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/ingest-pipeline-processed-logs-1536x687.png?v=1665055261 1536w\" sizes=\"(max-width: 1892px) 100vw, 1892px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image: Create Custom ELK Ingest Pipeline for Custom Log Processing\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<p>Expanding one of the events;<\/p>\n\n\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/kifarunix.com\\\/wp-content\\\/uploads\\\/2022\\\/10\\\/modsecurity-logs-processed-with-custom-ingest-pipelines.png&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-full&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-14338&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1916,&quot;targetHeight&quot;:3557,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image: logs processed by custom ingest pipeline&quot;,&quot;alt&quot;:&quot;logs processed by custom ingest pipeline&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"1916\" height=\"3557\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/modsecurity-logs-processed-with-custom-ingest-pipelines.png\" alt=\"logs processed by custom ingest pipeline\" class=\"wp-image-14338\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/modsecurity-logs-processed-with-custom-ingest-pipelines.png?v=1665055327 1916w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/modsecurity-logs-processed-with-custom-ingest-pipelines-768x1426.png?v=1665055327 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/modsecurity-logs-processed-with-custom-ingest-pipelines-827x1536.png?v=1665055327 827w, https:\/\/kifarunix.com\/wp-content\/uploads\/2022\/10\/modsecurity-logs-processed-with-custom-ingest-pipelines-1103x2048.png?v=1665055327 1103w\" sizes=\"(max-width: 1916px) 100vw, 1916px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image: logs processed by custom ingest pipeline\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n<p>And that is how easy it is to use custom ingest pipelines to process custom logs.<\/p>\n\n\n\n<p>That marks the end of our guide on how to create custom ELK ingest pipeline for custom log processing.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"other-tutorials\">Other Tutorials<\/h3>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/process-modsecurity-logs-using-wazuh\/\" target=\"_blank\" rel=\"noreferrer noopener\">Process ModSecurity Logs using Wazuh<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/process-and-visualize-modsecurity-logs-on-elk-stack\/\" target=\"_blank\" rel=\"noreferrer noopener\">Process and Visualize ModSecurity Logs on ELK Stack<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Follow through this tutorial to learn how create custom ELK ingest pipeline for custom log processing. Elastic Stack is so flexible that it can give<\/p>\n","protected":false},"author":3,"featured_media":14351,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[72,121],"tags":[5919,5921,5922,5923,5920],"class_list":["post-14259","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-monitoring","category-howtos","tag-create-custom-elk-ingest-pipeline-for-custom-log-processing","tag-custom-logs-ingest-pipeline","tag-elk-ingest-pipeline","tag-ingest-pipelines","tag-kibana-custom-ingest-pipeline","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/14259"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=14259"}],"version-history":[{"count":31,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/14259\/revisions"}],"predecessor-version":[{"id":20641,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/14259\/revisions\/20641"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/14351"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=14259"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=14259"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=14259"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}