{"id":17899,"date":"2023-07-16T08:40:18","date_gmt":"2023-07-16T05:40:18","guid":{"rendered":"https:\/\/kifarunix.com\/?p=17899"},"modified":"2024-03-10T10:19:32","modified_gmt":"2024-03-10T07:19:32","slug":"configuring-filebeat-to-send-logs-to-kafka","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/configuring-filebeat-to-send-logs-to-kafka\/","title":{"rendered":"Complete Guide: Configuring Filebeat to Send Logs to Kafka"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1050\" height=\"586\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/07\/filebeat-kafka-connection.png\" alt=\"Complete Guide: Configuring Filebeat to Send Logs to Kafka\" class=\"wp-image-17925\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/07\/filebeat-kafka-connection.png?v=1689570711 1050w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/07\/filebeat-kafka-connection-768x429.png?v=1689570711 768w\" sizes=\"(max-width: 1050px) 100vw, 1050px\" \/><\/figure>\n\n\n\n<p>How can I configure Filebeat to send logs to Kafka? This is a complete guide on configuring Filebeat to send logs to Kafka. <a href=\"https:\/\/www.elastic.co\/guide\/en\/beats\/filebeat\/current\/filebeat-overview.html\" target=\"_blank\" rel=\"noreferrer noopener\">Filebeat<\/a> is one of the Elastic stack beats that is used to collect system log data and sent them either to&nbsp;Elasticsearch&nbsp;or&nbsp;Logstash&nbsp;or to distributed event store and handling large volumes of data streams processing platforms such as Kafka.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#configure-filebeat-to-send-logs-to-kafka\">Configure Filebeat to Send Logs to Kafka<\/a><ul><li><a href=\"#install-and-configure-kafka\">Install and Configure Kafka<\/a><\/li><li><a href=\"#install-filebeat\">Install Filebeat<\/a><\/li><li><a href=\"#configure-filebeat-to-send-logs-to-kafka-with-ssl-tls-plaintext\">Configuring Filebeat to Send Logs to Kafka without SSL\/TLS (PLAINTEXT)<\/a><ul><li><a href=\"#configure-filebeat-to-connect-to-plaintext-kafka\">Configure Filebeat to Connect to Plaintext Kafka<\/a><\/li><li><a href=\"#confirm-creation-of-kafka-topics-on-kafka-broker\">Confirm Creation of Kafka Topics on Kafka Broker<\/a><\/li><li><a href=\"#reading-logs-from-kafka-topic\">Reading Logs from Kafka Topic<\/a><\/li><\/ul><\/li><li><a href=\"#configure-filebeat-to-send-logs-to-kafka-with-ssl-tls\">Configure Filebeat to Send Logs to Kafka with SSL\/TLS<\/a><ul><li><a href=\"#configure-filebeat-kafka-output-ssl-tls-connection\">Configure Filebeat-Kafka Output SSL\/TLS Connection<\/a><\/li><li><a href=\"#confirm-creation-of-kafka-topics-on-kafka-broker-1\">Confirm Creation of Kafka Topics on Kafka Broker<\/a><\/li><li><a href=\"#reading-logs-from-kafka-topic-2\">Reading Logs from Kafka Topic<\/a><\/li><\/ul><\/li><li><a href=\"#configure-consumers\">Configure Consumers<\/a><\/li><li><a href=\"#other-tutorials\">Other Tutorials<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"configure-filebeat-to-send-logs-to-kafka\">Configure Filebeat to Send Logs to Kafka<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-and-configure-kafka\">Install and Configure Kafka<\/h3>\n\n\n\n<p>You can check our guides on the links below to learn how to install and configure Kafka.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-apache-kafka-on-debian\/\" target=\"_blank\" rel=\"noreferrer noopener\">Easy Steps: Install Apache Kafka on Debian 12<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/configure-apache-kafka-ssl-tls-encryption\/\" target=\"_blank\" rel=\"noreferrer noopener\">Quick Guide: Configure Apache Kafka SSL\/TLS Encryption for Enhanced Security<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-filebeat\">Install Filebeat<\/h3>\n\n\n\n<p>You need Filebeat to be able to collect system logs from various Linux systems and push them to Kafka.<\/p>\n\n\n\n<p>You can check our guides on how to install Filebeat;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/?s=install+filebeat\" target=\"_blank\" rel=\"noreferrer noopener\">How to install Filebeat on Linux<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configure-filebeat-to-send-logs-to-kafka-with-ssl-tls-plaintext\">Configure Filebeat to Send Logs to Kafka with SSL\/TLS (PLAINTEXT)<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-filebeat-to-connect-to-plaintext-kafka\">Configure Filebeat to Connect to Plaintext Kafka<\/h4>\n\n\n\n<p>By default, Filebeat is set to send event data to Elasticsearch as you can see in the output section of the <strong><code>filebeat.yml<\/code><\/strong> configuration file.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n# ---------------------------- Elasticsearch Output ----------------------------\n<strong>output.elasticsearch:\n  # Array of hosts to connect to.\n    hosts: [\"localhost:9200\"]<\/strong>\n\n  # Protocol - either `http` (default) or `https`.\n  #protocol: \"https\"\n\n  # Authentication credentials - either API key or username\/password.\n  #api_key: \"id:api_key\"\n  #username: \"elastic\"\n  #password: \"changeme\"\n\n<\/code><\/pre>\n\n\n\n<p>In order to configure Filebeat to send logs to Kafka, edit the Filebeat configuration file and update the output section by configuring the Apache Kafka connection and other details.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/filebeat\/filebeat.yml<\/code><\/pre>\n\n\n\n<p>Start by commenting out the Elasticsearch output configs;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n# ---------------------------- Elasticsearch Output ----------------------------\n<strong>#output.elasticsearch:\n  # Array of hosts to connect to.\n  #hosts: [\"localhost:9200\"]<\/strong>\n\n  # Protocol - either `http` (default) or `https`.\n  #protocol: \"https\"\n\n  # Authentication credentials - either API key or username\/password.\n  #api_key: \"id:api_key\"\n  #username: \"elastic\"\n  #password: \"changeme\"\n\n<\/code><\/pre>\n\n\n\n<p>Next, add a section for Apache Kafka output beneath (<em>not necessarily<\/em>) Elasticsearch output. If you are not sending any sensitive information to Kafka or the connection is just local to your network, this configuration below works for Kafka plaintext connection where traffic is send un-encrypted.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n# ------------------------------ Kafka Output -------------------------------\noutput.kafka:\n  # initial brokers for reading cluster metadata\n  hosts: [\"192.168.57.32:9092\"]\n\n  # message topic selection + partitioning\n  topic: \"filebeat\"\n  partition.round_robin:\n    reachable_only: false\n\n  required_acks: 1\n  compression: gzip\n  max_message_bytes: 1000000\n<\/code><\/pre>\n\n\n\n<p>So, what do the options used here mean?<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code><strong>output.kafka<\/strong><\/code>: This specifies that output plugin to be used by Filebeat.<\/li>\n\n\n\n<li><code><strong>hosts<\/strong><\/code>: Defines the addresses and ports of the Kafka broker(s) to connect to for writing event data. In this case, Filebeat is configured to connect to a Kafka broker running on the IP address 192.168.57.32 and port 9092. If you have a Kafka cluster, you can use an HA approach;<br><code><strong>hosts: [\"kafka1:9092\", \"kafka2:9092\", \"kafka3:9092\"]<\/strong><\/code><\/li>\n\n\n\n<li><code><strong>topic<\/strong><\/code>: It specifies the Kafka topic to which Filebeat will send the messages. In this case, the topic is set to <strong><code>filebeat<\/code><\/strong>. It can be anything, including dynamic names.<\/li>\n\n\n\n<li><code><strong>partition.round_robin<\/strong><\/code>: This setting determines how Filebeat selects the partition to write messages to. With <code>reachable_only<\/code> set to <code>false<\/code>, Filebeat will attempt to write to all available partitions, even if some of them are not currently reachable.<\/li>\n\n\n\n<li><code><strong>required_acks<\/strong><\/code>: It indicates the number of acknowledgments required from Kafka before considering a message as successfully written. A value of 1 means that only the leader of the partition needs to acknowledge the message.<\/li>\n\n\n\n<li><code><strong>compression<\/strong><\/code>: This parameter specifies the compression codec to use for compressing messages sent to Kafka. Here, &#8216;gzip&#8217; compression is enabled, which can reduce the message size and network bandwidth usage.<\/li>\n\n\n\n<li><code><strong>max_message_bytes<\/strong><\/code>: It sets the maximum size of a message that Filebeat can send to Kafka. In this case, the maximum message size is set to 1,000,000 bytes (1 megabyte).<\/li>\n<\/ul>\n\n\n\n<p>Read more on <a href=\"https:\/\/www.elastic.co\/guide\/en\/beats\/filebeat\/current\/kafka-output.html#_configuration_options_21\" target=\"_blank\" rel=\"noreferrer noopener\">Filebeat Kafka output configuration options<\/a>.<\/p>\n\n\n\n<p>Also check how to <a href=\"https:\/\/www.elastic.co\/guide\/en\/beats\/filebeat\/current\/configuration-output-codec.html\" target=\"_blank\" rel=\"noreferrer noopener\">change output codec<\/a> BUT output remains as <strong>output.kafka<\/strong>.<\/p>\n\n\n\n<p>Save the changes made to the Filebeat configuration and exit.<\/p>\n\n\n\n<p>Next, test the configuration for any syntax error;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat test config<\/code><\/pre>\n\n\n\n<p>Test the connection to Kafka broker;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat test output<\/code><\/pre>\n\n\n\n<p>If everything is okay, you should see an output similar to;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\nKafka: 192.168.57.32:9092...\n  parse host... OK\n  dns lookup... OK\n  addresses: 192.168.57.32\n  dial up... OK\n<\/code><\/pre>\n\n\n\n<p>Start Filebeat.<\/p>\n\n\n\n<p>You can begin by running it in the foreground;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat -e<\/code><\/pre>\n\n\n\n<p>If everything is fine, you should see a message about the connection to Kafka;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>Connection to kafka(192.168.57.32:9092) established<\/strong><\/code><\/pre>\n\n\n\n<p>You can then start Filebeat as service. Thus press <strong>ctrl+c<\/strong> to stop the foreground process.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl enable --now<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"confirm-creation-of-kafka-topics-on-kafka-broker\">Confirm Creation of Kafka Topics on Kafka Broker<\/h4>\n\n\n\n<p>On Kafka broker\/server terminal, execute this command to check if your topic has been created;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/opt\/kafka\/bin\/kafka-topics.sh --list --bootstrap-server localhost:9092<\/code><\/pre>\n\n\n\n<p>My sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>filebeat<\/strong>\nkafka-topic-test<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"reading-logs-from-kafka-topic\">Reading Logs from Kafka Topic<\/h4>\n\n\n\n<p>Read the logs on the topic, use the command;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/opt\/kafka\/bin\/kafka-console-consumer.sh --bootstrap-server &lt;broker_host&gt;:&lt;broker_port&gt; --topic &lt;topic_name&gt; &#91;--from-beginning]<\/code><\/pre>\n\n\n\n<p>For example;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/opt\/kafka\/bin\/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic filebeat<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n{\"@timestamp\":\"2023-07-16T19:42:06.697Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.8.2\",\"pipeline\":\"filebeat-8.8.2-system-auth-pipeline\"},\"log\":{\"offset\":2384006,\"file\":{\"path\":\"\/var\/log\/auth.log\"}},\"input\":{\"type\":\"log\"},\"fileset\":{\"name\":\"auth\"},\"service\":{\"type\":\"system\"},\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"Jul 16 19:42:01 mx CRON[18368]: pam_unix(cron:session): session opened for user root by (uid=0)\",\"host\":{\"hostname\":\"mx.kifarunix-demo.com\",\"architecture\":\"x86_64\",\"os\":{\"family\":\"debian\",\"name\":\"Ubuntu\",\"kernel\":\"5.4.0-153-generic\",\"codename\":\"focal\",\"type\":\"linux\",\"platform\":\"ubuntu\",\"version\":\"20.04.6 LTS (Focal Fossa)\"},\"id\":\"e46f0b2ecf5841be8df054ae93c48214\",\"name\":\"mx.kifarunix-demo.com\",\"containerized\":false,\"ip\":[\"10.0.2.15\",\"fe80::a00:27ff:fee4:5139\",\"192.168.57.33\",\"fe80::a00:27ff:fe89:7e5f\"],\"mac\":[\"08-00-27-89-7E-5F\",\"08-00-27-E4-51-39\"]},\"agent\":{\"ephemeral_id\":\"27ebb032-f900-4d05-88d8-51de9b9c1c6d\",\"id\":\"08b49b73-3585-41ba-9ef5-c0c731203047\",\"name\":\"mx.kifarunix-demo.com\",\"type\":\"filebeat\",\"version\":\"8.8.2\"},\"event\":{\"module\":\"system\",\"dataset\":\"system.auth\",\"timezone\":\"+00:00\"}}\n{\"@timestamp\":\"2023-07-16T19:42:06.697Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.8.2\",\"pipeline\":\"filebeat-8.8.2-system-auth-pipeline\"},\"input\":{\"type\":\"log\"},\"event\":{\"module\":\"system\",\"dataset\":\"system.auth\",\"timezone\":\"+00:00\"},\"fileset\":{\"name\":\"auth\"},\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"Jul 16 19:42:01 mx CRON[18369]: pam_unix(cron:session): session opened for user sogo by (uid=0)\",\"log\":{\"offset\":2384102,\"file\":{\"path\":\"\/var\/log\/auth.log\"}},\"service\":{\"type\":\"system\"},\"host\":{\"id\":\"e46f0b2ecf5841be8df054ae93c48214\",\"containerized\":false,\"name\":\"mx.kifarunix-demo.com\",\"ip\":[\"10.0.2.15\",\"fe80::a00:27ff:fee4:5139\",\"192.168.57.33\",\"fe80::a00:27ff:fe89:7e5f\"],\"mac\":[\"08-00-27-89-7E-5F\",\"08-00-27-E4-51-39\"],\"hostname\":\"mx.kifarunix-demo.com\",\"architecture\":\"x86_64\",\"os\":{\"name\":\"Ubuntu\",\"kernel\":\"5.4.0-153-generic\",\"codename\":\"focal\",\"type\":\"linux\",\"platform\":\"ubuntu\",\"version\":\"20.04.6 LTS (Focal Fossa)\",\"family\":\"debian\"}},\"agent\":{\"version\":\"8.8.2\",\"ephemeral_id\":\"27ebb032-f900-4d05-88d8-51de9b9c1c6d\",\"id\":\"08b49b73-3585-41ba-9ef5-c0c731203047\",\"name\":\"mx.kifarunix-demo.com\",\"type\":\"filebeat\"}}\n{\"@timestamp\":\"2023-07-16T19:42:06.697Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.8.2\",\"pipeline\":\"filebeat-8.8.2-system-auth-pipeline\"},\"message\":\"Jul 16 19:42:01 mx CRON[18368]: pam_unix(cron:session): session closed for user root\",\"log\":{\"offset\":2384198,\"file\":{\"path\":\"\/var\/log\/auth.log\"}},\"service\":{\"type\":\"system\"},\"ecs\":{\"version\":\"8.0.0\"},\"fileset\":{\"name\":\"auth\"},\"input\":{\"type\":\"log\"},\"event\":{\"module\":\"system\",\"dataset\":\"system.auth\",\"timezone\":\"+00:00\"},\"host\":{\"ip\":[\"10.0.2.15\",\"fe80::a00:27ff:fee4:5139\",\"192.168.57.33\",\"fe80::a00:27ff:fe89:7e5f\"],\"mac\":[\"08-00-27-89-7E-5F\",\"08-00-27-E4-51-39\"],\"name\":\"mx.kifarunix-demo.com\",\"hostname\":\"mx.kifarunix-demo.com\",\"architecture\":\"x86_64\",\"os\":{\"version\":\"20.04.6 LTS (Focal Fossa)\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"kernel\":\"5.4.0-153-generic\",\"codename\":\"focal\",\"type\":\"linux\",\"platform\":\"ubuntu\"},\"id\":\"e46f0b2ecf5841be8df054ae93c48214\",\"containerized\":false},\"agent\":{\"version\":\"8.8.2\",\"ephemeral_id\":\"27ebb032-f900-4d05-88d8-51de9b9c1c6d\",\"id\":\"08b49b73-3585-41ba-9ef5-c0c731203047\",\"name\":\"mx.kifarunix-demo.com\",\"type\":\"filebeat\"}}\n{\"@timestamp\":\"2023-07-16T19:42:09.055Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.8.2\",\"pipeline\":\"filebeat-8.8.2-system-syslog-pipeline\"},\"log\":{\"offset\":549530,\"file\":{\"path\":\"\/var\/log\/syslog\"}},\"message\":\"Jul 16 19:42:01 mx CRON[18370]: (sogo) CMD (\/usr\/sbin\/sogo-tool expire-sessions 30 >\/dev\/null 2>&1; \/usr\/sbin\/sogo-ealarms-notify >\/dev\/null 2>&1)\",\"event\":{\"module\":\"system\",\"timezone\":\"+00:00\",\"dataset\":\"system.syslog\"},\"fileset\":{\"name\":\"syslog\"},\"agent\":{\"name\":\"mx.kifarunix-demo.com\",\"type\":\"filebeat\",\"version\":\"8.8.2\",\"ephemeral_id\":\"27ebb032-f900-4d05-88d8-51de9b9c1c6d\",\"id\":\"08b49b73-3585-41ba-9ef5-c0c731203047\"},\"input\":{\"type\":\"log\"},\"service\":{\"type\":\"system\"},\"ecs\":{\"version\":\"1.12.0\"},\"host\":{\"hostname\":\"mx.kifarunix-demo.com\",\"architecture\":\"x86_64\",\"os\":{\"family\":\"debian\",\"name\":\"Ubuntu\",\"kernel\":\"5.4.0-153-generic\",\"codename\":\"focal\",\"type\":\"linux\",\"platform\":\"ubuntu\",\"version\":\"20.04.6 LTS (Focal Fossa)\"},\"id\":\"e46f0b2ecf5841be8df054ae93c48214\",\"containerized\":false,\"ip\":[\"10.0.2.15\",\"fe80::a00:27ff:fee4:5139\",\"192.168.57.33\",\"fe80::a00:27ff:fe89:7e5f\"],\"mac\":[\"08-00-27-89-7E-5F\",\"08-00-27-E4-51-39\"],\"name\":\"mx.kifarunix-demo.com\"}}\n^CProcessed a total of 4 messages\n\n<\/code><\/pre>\n\n\n\n<p>This will read the logs in realtime only.<\/p>\n\n\n\n<p>You can read the logs from the beginning and in realtime at the same time;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/opt\/kafka\/bin\/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic filebeat --from-beginning<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configure-filebeat-to-send-logs-to-kafka-with-ssl-tls\">Configure Filebeat to Send Logs to Kafka with SSL\/TLS<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-filebeat-kafka-output-ssl-tls-connection\">Configure Filebeat-Kafka Output SSL\/TLS Connection<\/h4>\n\n\n\n<p>What if your Kafka is <a href=\"https:\/\/kifarunix.com\/configure-apache-kafka-ssl-tls-encryption\/\" target=\"_blank\" rel=\"noreferrer noopener\">configured with SSL\/TLS encryption<\/a>, how can you configure Filebeat to send logs to Kafka with SSL\/TLS?<\/p>\n\n\n\n<p>Simply update the SSL connection settings;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n# ------------------------------ Kafka Output -------------------------------\noutput.kafka:\n  # initial brokers for reading cluster metadata\n  hosts: [\"kafka.kifarunix-demo.com:9092\"]\n  <strong>ssl:\n    enabled: true\n    certificate_authorities: [\"\/etc\/filebeat\/kafka.ca.crt\"]\n    certificate: \"\/etc\/filebeat\/kafka\/server.crt\"\n    key: \"\/etc\/filebeat\/kafka\/server.key\"<\/strong>\n\n  # message topic selection + partitioning\n  topic: \"filebeat-ssl\"\n  partition.round_robin:\n    reachable_only: false\n\n  required_acks: 1\n  compression: gzip\n  max_message_bytes: 1000000\n  \n<\/code><\/pre>\n\n\n\n<p>You need to copy the CA, server certificate and key from the Kafka server.<\/p>\n\n\n\n<p>Ensure the Kafka hostname is resolvable.<\/p>\n\n\n\n<p>Save and exit file after making changes.<\/p>\n\n\n\n<p>Test Filebeat configuration;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat test config<\/code><\/pre>\n\n\n\n<p>Unfortunately, in Filebeat, <em>Kafka output doesn&#8217;t support TLS testing<\/em>.<\/p>\n\n\n\n<p>Start Filebeat, in the foreground, just to verify the connection;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat -e<\/code><\/pre>\n\n\n\n<p>If you see such an output, then you are good.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>\"Connection to kafka(kafka.kifarunix-demo.com:9092) established\"<\/strong><\/code><\/pre>\n\n\n\n<p>You can start Filebeat as a service;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl start filebeat<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"confirm-creation-of-kafka-topics-on-kafka-broker-1\">Confirm Creation of Kafka Topics on Kafka Broker<\/h4>\n\n\n\n<p>On Kafka broker\/server terminal, execute this command to check if your topic has been created;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/opt\/kafka\/bin\/kafka-topics.sh --list --bootstrap-server localhost:9094<\/code><\/pre>\n\n\n\n<p>(Note that we have configure port 9094 on the Kafka server for local Kafka connections without SSL)<\/p>\n\n\n\n<p>My sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>__consumer_offsets\nfilebeat\n<strong>filebeat-ssl\n<\/strong>kafka-topic-test<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"reading-logs-from-kafka-topic-2\">Reading Logs from Kafka Topic<\/h4>\n\n\n\n<p>Read the logs on the topic, use the command;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/opt\/kafka\/bin\/kafka-console-consumer.sh --bootstrap-server localhost:9094 --topic filebeat-ssl<\/code><\/pre>\n\n\n\n<p>Sample events;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n\"@timestamp\":\"2023-07-17T04:51:02.761Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.8.2\",\"pipeline\":\"filebeat-8.8.2-system-auth-pipeline\"},\"service\":{\"type\":\"system\"},\"input\":{\"type\":\"log\"},\"ecs\":{\"version\":\"8.0.0\"},\"host\":{\"mac\":[\"08-00-27-89-7E-5F\",\"08-00-27-E4-51-39\"],\"hostname\":\"mx.kifarunix-demo.com\",\"name\":\"mx.kifarunix-demo.com\",\"architecture\":\"x86_64\",\"os\":{\"codename\":\"focal\",\"type\":\"linux\",\"platform\":\"ubuntu\",\"version\":\"20.04.6 LTS (Focal Fossa)\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"kernel\":\"5.4.0-153-generic\"},\"id\":\"e46f0b2ecf5841be8df054ae93c48214\",\"containerized\":false,\"ip\":[\"10.0.2.15\",\"fe80::a00:27ff:fee4:5139\",\"192.168.57.33\",\"fe80::a00:27ff:fe89:7e5f\"]},\"agent\":{\"ephemeral_id\":\"02d93da9-5449-4333-9e6e-26250b28456d\",\"id\":\"08b49b73-3585-41ba-9ef5-c0c731203047\",\"name\":\"mx.kifarunix-demo.com\",\"type\":\"filebeat\",\"version\":\"8.8.2\"},\"log\":{\"offset\":2602826,\"file\":{\"path\":\"\/var\/log\/auth.log\"}},\"message\":\"Jul 17 04:51:01 mx CRON[35424]: pam_unix(cron:session): session opened for user root by (uid=0)\",\"fileset\":{\"name\":\"auth\"},\"event\":{\"dataset\":\"system.auth\",\"timezone\":\"+00:00\",\"module\":\"system\"}}\n{\"@timestamp\":\"2023-07-17T04:51:02.761Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.8.2\",\"pipeline\":\"filebeat-8.8.2-system-auth-pipeline\"},\"fileset\":{\"name\":\"auth\"},\"input\":{\"type\":\"log\"},\"ecs\":{\"version\":\"8.0.0\"},\"service\":{\"type\":\"system\"},\"host\":{\"ip\":[\"10.0.2.15\",\"fe80::a00:27ff:fee4:5139\",\"192.168.57.33\",\"fe80::a00:27ff:fe89:7e5f\"],\"mac\":[\"08-00-27-89-7E-5F\",\"08-00-27-E4-51-39\"],\"hostname\":\"mx.kifarunix-demo.com\",\"architecture\":\"x86_64\",\"os\":{\"platform\":\"ubuntu\",\"version\":\"20.04.6 LTS (Focal Fossa)\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"kernel\":\"5.4.0-153-generic\",\"codename\":\"focal\",\"type\":\"linux\"},\"name\":\"mx.kifarunix-demo.com\",\"id\":\"e46f0b2ecf5841be8df054ae93c48214\",\"containerized\":false},\"agent\":{\"type\":\"filebeat\",\"version\":\"8.8.2\",\"ephemeral_id\":\"02d93da9-5449-4333-9e6e-26250b28456d\",\"id\":\"08b49b73-3585-41ba-9ef5-c0c731203047\",\"name\":\"mx.kifarunix-demo.com\"},\"message\":\"Jul 17 04:51:01 mx CRON[35425]: pam_unix(cron:session): session opened for user sogo by (uid=0)\",\"log\":{\"offset\":2602922,\"file\":{\"path\":\"\/var\/log\/auth.log\"}},\"event\":{\"module\":\"system\",\"dataset\":\"system.auth\",\"timezone\":\"+00:00\"}}\n{\"@timestamp\":\"2023-07-17T04:51:02.761Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.8.2\",\"pipeline\":\"filebeat-8.8.2-system-auth-pipeline\"},\"event\":{\"module\":\"system\",\"dataset\":\"system.auth\",\"timezone\":\"+00:00\"},\"fileset\":{\"name\":\"auth\"},\"ecs\":{\"version\":\"8.0.0\"},\"message\":\"Jul 17 04:51:01 mx CRON[35424]: pam_unix(cron:session): session closed for user root\",\"log\":{\"offset\":2603018,\"file\":{\"path\":\"\/var\/log\/auth.log\"}},\"service\":{\"type\":\"system\"},\"input\":{\"type\":\"log\"},\"host\":{\"id\":\"e46f0b2ecf5841be8df054ae93c48214\",\"containerized\":false,\"ip\":[\"10.0.2.15\",\"fe80::a00:27ff:fee4:5139\",\"192.168.57.33\",\"fe80::a00:27ff:fe89:7e5f\"],\"mac\":[\"08-00-27-89-7E-5F\",\"08-00-27-E4-51-39\"],\"hostname\":\"mx.kifarunix-demo.com\",\"architecture\":\"x86_64\",\"os\":{\"version\":\"20.04.6 LTS (Focal Fossa)\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"kernel\":\"5.4.0-153-generic\",\"codename\":\"focal\",\"type\":\"linux\",\"platform\":\"ubuntu\"},\"name\":\"mx.kifarunix-demo.com\"},\"agent\":{\"name\":\"mx.kifarunix-demo.com\",\"type\":\"filebeat\",\"version\":\"8.8.2\",\"ephemeral_id\":\"02d93da9-5449-4333-9e6e-26250b28456d\",\"id\":\"08b49b73-3585-41ba-9ef5-c0c731203047\"}}\n{\"@timestamp\":\"2023-07-17T04:51:02.761Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.8.2\",\"pipeline\":\"filebeat-8.8.2-system-auth-pipeline\"},\"host\":{\"hostname\":\"mx.kifarunix-demo.com\",\"architecture\":\"x86_64\",\"os\":{\"type\":\"linux\",\"platform\":\"ubuntu\",\"version\":\"20.04.6 LTS (Focal Fossa)\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"kernel\":\"5.4.0-153-generic\",\"codename\":\"focal\"},\"id\":\"e46f0b2ecf5841be8df054ae93c48214\",\"containerized\":false,\"ip\":[\"10.0.2.15\",\"fe80::a00:27ff:fee4:5139\",\"192.168.57.33\",\"fe80::a00:27ff:fe89:7e5f\"],\"mac\":[\"08-00-27-89-7E-5F\",\"08-00-27-E4-51-39\"],\"name\":\"mx.kifarunix-demo.com\"},\"message\":\"Jul 17 04:51:02 mx CRON[35425]: pam_unix(cron:session): session closed for user sogo\",\"log\":{\"offset\":2603103,\"file\":{\"path\":\"\/var\/log\/auth.log\"}},\"event\":{\"dataset\":\"system.auth\",\"module\":\"system\",\"timezone\":\"+00:00\"},\"fileset\":{\"name\":\"auth\"},\"ecs\":{\"version\":\"8.0.0\"},\"input\":{\"type\":\"log\"},\"service\":{\"type\":\"system\"},\"agent\":{\"type\":\"filebeat\",\"version\":\"8.8.2\",\"ephemeral_id\":\"02d93da9-5449-4333-9e6e-26250b28456d\",\"id\":\"08b49b73-3585-41ba-9ef5-c0c731203047\",\"name\":\"mx.kifarunix-demo.com\"}}\n{\"@timestamp\":\"2023-07-17T04:51:09.813Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.8.2\",\"pipeline\":\"filebeat-8.8.2-system-syslog-pipeline\"},\"fileset\":{\"name\":\"syslog\"},\"ecs\":{\"version\":\"1.12.0\"},\"host\":{\"architecture\":\"x86_64\",\"os\":{\"kernel\":\"5.4.0-153-generic\",\"codename\":\"focal\",\"type\":\"linux\",\"platform\":\"ubuntu\",\"version\":\"20.04.6 LTS (Focal Fossa)\",\"family\":\"debian\",\"name\":\"Ubuntu\"},\"id\":\"e46f0b2ecf5841be8df054ae93c48214\",\"containerized\":false,\"name\":\"mx.kifarunix-demo.com\",\"ip\":[\"10.0.2.15\",\"fe80::a00:27ff:fee4:5139\",\"192.168.57.33\",\"fe80::a00:27ff:fe89:7e5f\"],\"mac\":[\"08-00-27-89-7E-5F\",\"08-00-27-E4-51-39\"],\"hostname\":\"mx.kifarunix-demo.com\"},\"log\":{\"file\":{\"path\":\"\/var\/log\/syslog\"},\"offset\":87749},\"event\":{\"module\":\"system\",\"dataset\":\"system.syslog\",\"timezone\":\"+00:00\"},\"input\":{\"type\":\"log\"},\"agent\":{\"type\":\"filebeat\",\"version\":\"8.8.2\",\"ephemeral_id\":\"02d93da9-5449-4333-9e6e-26250b28456d\",\"id\":\"08b49b73-3585-41ba-9ef5-c0c731203047\",\"name\":\"mx.kifarunix-demo.com\"},\"message\":\"Jul 17 04:51:01 mx CRON[35426]: (sogo) CMD (\/usr\/sbin\/sogo-tool expire-sessions 30 >\/dev\/null 2>&1; \/usr\/sbin\/sogo-ealarms-notify >\/dev\/null 2>&1)\",\"service\":{\"type\":\"system\"}}\n{\"@timestamp\":\"2023-07-17T04:51:09.813Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.8.2\",\"pipeline\":\"filebeat-8.8.2-system-syslog-pipeline\"},\"fileset\":{\"name\":\"syslog\"},\"ecs\":{\"version\":\"1.12.0\"},\"agent\":{\"id\":\"08b49b73-3585-41ba-9ef5-c0c731203047\",\"name\":\"mx.kifarunix-demo.com\",\"type\":\"filebeat\",\"version\":\"8.8.2\",\"ephemeral_id\":\"02d93da9-5449-4333-9e6e-26250b28456d\"},\"host\":{\"name\":\"mx.kifarunix-demo.com\",\"architecture\":\"x86_64\",\"os\":{\"version\":\"20.04.6 LTS (Focal Fossa)\",\"family\":\"debian\",\"name\":\"Ubuntu\",\"kernel\":\"5.4.0-153-generic\",\"codename\":\"focal\",\"type\":\"linux\",\"platform\":\"ubuntu\"},\"id\":\"e46f0b2ecf5841be8df054ae93c48214\",\"containerized\":false,\"ip\":[\"10.0.2.15\",\"fe80::a00:27ff:fee4:5139\",\"192.168.57.33\",\"fe80::a00:27ff:fe89:7e5f\"],\"mac\":[\"08-00-27-89-7E-5F\",\"08-00-27-E4-51-39\"],\"hostname\":\"mx.kifarunix-demo.com\"},\"input\":{\"type\":\"log\"},\"event\":{\"module\":\"system\",\"dataset\":\"system.syslog\",\"timezone\":\"+00:00\"},\"service\":{\"type\":\"system\"},\"log\":{\"offset\":87896,\"file\":{\"path\":\"\/var\/log\/syslog\"}},\"message\":\"Jul 17 04:51:01 mx CRON[35427]: (root) CMD (\/bin\/bash \/usr\/local\/bin\/fail2ban_banned_db unban_db)\"}\n^CProcessed a total of 6 messages\n<\/code><\/pre>\n\n\n\n<p>And that is it! You should now have encrypted connection between your Filebeat and Kafka broker, for enhanced security.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configure-consumers\">Configure Consumers<\/h3>\n\n\n\n<p>The logs are now hitting your Kafka topics. Similarly, configure your consumers to use SSL\/TLS connection while reading log data from the Kafka topics.<\/p>\n\n\n\n<p>That closes our guide on configure Filebeat to send logs to Kafka via un-encrypted and encrypted channels.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"other-tutorials\">Other Tutorials<\/h3>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/check-ssl-certificate-expiry-date-from-certificate-file\/\" target=\"_blank\" rel=\"noreferrer noopener\">Check SSL Certificate Expiry Date from Certificate File<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/monitor-ssl-tls-certificate-expiry-with-prometheus-and-grafana\/\" target=\"_blank\" rel=\"noreferrer noopener\">Monitor SSL\/TLS Certificate Expiry with Prometheus and Grafana<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>How can I configure Filebeat to send logs to Kafka? This is a complete guide on configuring Filebeat to send logs to Kafka. Filebeat is<\/p>\n","protected":false},"author":10,"featured_media":17925,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[72,910,121,34,1187],"tags":[7062,7061,7063,7064],"class_list":["post-17899","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-monitoring","category-elastic-stack","category-howtos","category-security","category-ssl-tls","tag-filbeat-kafka-output","tag-filebeat-kafka-log-collection","tag-filebeat-kafka-ssl-connection","tag-ssl-tls-kafka-and-filebeat","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/17899"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=17899"}],"version-history":[{"count":15,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/17899\/revisions"}],"predecessor-version":[{"id":20807,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/17899\/revisions\/20807"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/17925"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=17899"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=17899"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=17899"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}