{"id":17933,"date":"2023-07-19T00:44:43","date_gmt":"2023-07-18T21:44:43","guid":{"rendered":"https:\/\/kifarunix.com\/?p=17933"},"modified":"2024-03-10T10:15:46","modified_gmt":"2024-03-10T07:15:46","slug":"setup-a-three-node-kafka-kraft-cluster","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/setup-a-three-node-kafka-kraft-cluster\/","title":{"rendered":"Setup a Three-Node Kafka KRaft Cluster for Scalable Data Streaming"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1061\" height=\"591\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/07\/kafka-kraft-cluster.png\" alt=\"Setup a Three-Node Kafka KRaft Cluster\" class=\"wp-image-17953\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/07\/kafka-kraft-cluster.png?v=1689715915 1061w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/07\/kafka-kraft-cluster-768x428.png?v=1689715915 768w\" sizes=\"(max-width: 1061px) 100vw, 1061px\" \/><\/figure>\n\n\n\n<p>In this step-by-step guide, we will walk you through the process of how to setup a three-node Kafka KRaft cluster for scalable data streaming. Building a robust and fault-tolerant Kafka cluster is crucial for reliable and scalable data streaming. <a href=\"https:\/\/developer.confluent.io\/learn\/kraft\/\" target=\"_blank\" rel=\"noreferrer noopener\">KRaft<\/a>, is a new consensus algorithm introduced in Kafka v2.8.0 and is production ready for new clusters as of&nbsp;<a href=\"https:\/\/www.confluent.io\/blog\/apache-kafka-3-3-0-new-features-and-updates\/\" target=\"_blank\" rel=\"noreferrer noopener\">Apache Kafka 3.3<\/a>. It removed the need to run Kafka with Zookeeper.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#setting-up-three-node-kafka-k-raft-cluster\">Setting up Three-Node Kafka KRaft Cluster<\/a><ul><li><a href=\"#high-level-what-is-k-raft-and-how-does-it-work\">High Level: What is KRaft and How does it work?<\/a><\/li><li><a href=\"#prerequisites-for-setting-up-kafka-cluster\">Prerequisites for Setting up Kafka Cluster<\/a><\/li><li><a href=\"#install-and-setup-kafka-with-k-raft-on-three-nodes\">Install and Setup Kafka with KRaft on three Nodes<\/a><\/li><li><a href=\"#setting-up-three-node-kafka-k-raft-cluster-1\">Setting up Three-Node Kafka KRaft Cluster<\/a><ul><li><a href=\"#define-the-role-of-each-node-in-the-cluster\">Define the role of Each Node in the cluster<\/a><\/li><li><a href=\"#set-the-node-id-for-each-node-in-the-cluster\">Set the Node ID for Each Node in the cluster<\/a><\/li><li><a href=\"#specify-a-list-of-controller-nodes-in-the-cluster\">Specify a list of Controller Nodes in the Cluster<\/a><\/li><li><a href=\"#set-the-name-of-the-brokers-and-controllers-listener\">Set the Name of the Brokers and Controllers Listener<\/a><\/li><li><a href=\"#set-the-socket-server-address\">Set the Socket Server Address<\/a><\/li><li><a href=\"#update-the-broker-advertised-listener-address\">Update the Broker Advertised Listener Address<\/a><\/li><li><a href=\"#define-the-number-of-log-partitions-per-topic\">Define the Number of Log Partitions per Topic<\/a><\/li><li><a href=\"#open-controller-broker-ports-on-firewall\">Open Controller\/Broker Ports on Firewall<\/a><\/li><li><a href=\"#are-you-upgrading-your-already-existing-kafka-cluster\">Are you Upgrading your Already Existing Kafka Cluster<\/a><ul><li><a href=\"#1-manually-update-the-current-voters-id-in-the-quorum-state-file\">1. Manually update the current voters ID in the quorum-state file;<\/a><\/li><li><a href=\"#2-delete-the-quorum-state-file\">2. Delete the Quorum-state file<\/a><\/li><\/ul><\/li><li><a href=\"#update-the-cluster-id\">Update the Cluster ID<\/a><\/li><\/ul><\/li><li><a href=\"#restart-kafka-service\">Restart Kafka Service<\/a><\/li><li><a href=\"#configure-producers-to-ship-logs-to-kafka-cluster\">Configure Producers to Ship Logs to Kafka Cluster<\/a><\/li><li><a href=\"#kafka-cluster-management-scripts\">Kafka Cluster Management Scripts<\/a><\/li><li><a href=\"#manage-kafka-cluster-from-ui\">Manage Kafka Cluster from UI<\/a><\/li><li><a href=\"#setup-kafka-kraft-cluster-with-ssl-tls\">Setup Kafka Kraft Cluster with SSL\/TLS<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"setting-up-three-node-kafka-k-raft-cluster\">Setting up Three-Node Kafka KRaft Cluster<\/h2>\n\n\n\n<p>So, how can one setup a three-node Kafka KRaft cluster? Follow through!<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"high-level-what-is-k-raft-and-how-does-it-work\">High Level: What is KRaft and How does it work?<\/h3>\n\n\n\n<p>As already mentioned, Kafka Raft (KRaft) is a consensus algorithm that has been adopted by Kafka, over Zookeeper. It is a distributed algorithm that allows a group of Kafka broker controllers to agree on the state of a replicated event data.<\/p>\n\n\n\n<p>Read more on the <a href=\"https:\/\/docs.confluent.io\/platform\/current\/kafka-metadata\/kraft.html#kraft-overview\" target=\"_blank\" rel=\"noreferrer noopener\">KRaft overview page<\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"prerequisites-for-setting-up-kafka-cluster\">Prerequisites for Setting up Kafka Cluster<\/h3>\n\n\n\n<p>There are a few things that you need to consider while setting up Kafka cluster;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ensure you assign each node enough disk space depending on the volume of the data you are expecting to store based on your data retention policies<\/li>\n\n\n\n<li>Ensure you have a reliable network connection between your cluster nodes<\/li>\n\n\n\n<li>Ensure that the CPU and RAM assigned to your cluster brokers can handle the load related to the data streaming.<\/li>\n\n\n\n<li>Ensure you have an odd number of nodes in the cluster to avoid the split-brain scenario.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-and-setup-kafka-with-k-raft-on-three-nodes\">Install and Setup Kafka with KRaft on three Nodes<\/h3>\n\n\n\n<p>To install and setup Kafka, follow our guide below;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-apache-kafka-on-debian\/\" target=\"_blank\" rel=\"noreferrer noopener\">Easy Steps: Install Apache Kafka on Debian<\/a><\/p>\n\n\n\n<p>You need to install and setup Kafka on three Nodes.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"setting-up-three-node-kafka-k-raft-cluster-1\">Setting up Three-Node Kafka KRaft Cluster<\/h3>\n\n\n\n<p>Assuming you have installed and Kafka with KRaft is running on three nodes, proceed to configure three-node KRaft Kafka cluster.<\/p>\n\n\n\n<p>In our setup, we have three nodes;<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td><strong>#<\/strong><\/td><td><strong>Node Hostname<\/strong><\/td><td><strong>Node IP address<\/strong><\/td><td><strong>Node Role<\/strong><\/td><\/tr><tr><td>1<\/td><td>knode1.kifarunix-demo.com<\/td><td>192.168.57.32<\/td><td>controller\/broker<\/td><\/tr><tr><td>2<\/td><td>knode2.kifarunix-demo.com<\/td><td>192.168.57.33<\/td><td>controller\/broker<\/td><\/tr><tr><td>3<\/td><td>knode3.kifarunix-demo.com<\/td><td>192.168.57.35<\/td><td>controller\/broker<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">Kafka KRaft Cluster Nodes<\/figcaption><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"define-the-role-of-each-node-in-the-cluster\">Define the role of Each Node in the cluster<\/h4>\n\n\n\n<p>In Kafka KRaft cluster, a node can either be a controller, a broker or can perform both roles.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A <strong><code>controller node<\/code><\/strong> coordinates the Kafka cluster and manages tracking of the event metadata. It also monitors the health and status of brokers, partitions, and replicas, leader election, partition reassignment, and handling broker failures.<\/li>\n\n\n\n<li>A <strong><code>broker node<\/code><\/strong> acts as a data plane. It hosts and manages Kafka topics and partitions. It is responsible for storing and serving the messages published to Kafka topics. Brokers handle the actual data replication, storage, and retrieval in a distributed manner. Each broker in the cluster may have multiple partitions of different topics assigned to it.<\/li>\n<\/ul>\n\n\n\n<p>In our setup, we will configure our cluster nodes to function both as controller and broker. You may want to separate them!<\/p>\n\n\n\n<p>Thus, open the KRaft <strong>server.properties<\/strong> configuration file and navigate to <strong>Server Basics<\/strong> section <strong>on each node<\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/opt\/kafka\/config\/kraft\/server.properties<\/code><\/pre>\n\n\n\n<p>By default, a node is assigned both roles;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># The role of this server. Setting this puts us in KRaft mode\n<strong>process.roles=broker,controller<\/strong><\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-the-node-id-for-each-node-in-the-cluster\">Set the Node ID for Each Node in the cluster<\/h4>\n\n\n\n<p>To uniquely identify each other, each Node in the cluster must have a unique ID.<\/p>\n\n\n\n<p>Node 1;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># The node id associated with this instance's roles\n<strong>node.id=1<\/strong><\/code><\/pre>\n\n\n\n<p>Node 2;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># The node id associated with this instance's roles\n<strong>node.id=2<\/strong><\/code><\/pre>\n\n\n\n<p>Node 3;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># The node id associated with this instance's roles\n<strong>node.id=3<\/strong><\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"specify-a-list-of-controller-nodes-in-the-cluster\">Specify a list of Controller Nodes in the Cluster<\/h4>\n\n\n\n<p>Next, you need to tell Kafka which nodes to use as controllers. This can be done by updating the value of the <strong><code>controller.quorum.voters<\/code><\/strong> parameter.<\/p>\n\n\n\n<p>The controller is defined as <strong>ID@ADDRESS:PORT<\/strong>. If you have multiple controllers, define them in comma separate. The address could be resolvable hostname or IP address.<\/p>\n\n\n\n<p>By default, Kafka expects to run as a single node cluster hence, the setting, <strong><code>controller.quorum.voters=1@localhost:9093.<\/code><\/strong><\/p>\n\n\n\n<p>Update this setting with the list of your nodes (Do this on all the nodes);<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>controller.quorum.voters=1@192.168.57.32:9093,2@192.168.57.33:9093,3@192.168.57.35:9093<\/code><\/pre>\n\n\n\n<p>Ensure the port used is not used by any other application\/service already.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-the-name-of-the-brokers-and-controllers-listener\">Set the Name of the Brokers and Controllers Listener<\/h4>\n\n\n\n<p>Under the Socket Server Settings, you need to define the name of listener used for communication between brokers and used by the controllers. This is set to <strong>PLAINTEXT<\/strong> and <strong>CONTROLLER<\/strong> (respectively) by default;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>inter.broker.listener.name=PLAINTEXT\n...\ncontroller.listener.names=CONTROLLER<\/code><\/pre>\n\n\n\n<p>We will leave it with the default names! If you want, you can update it. These names will be used in other config settings.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"set-the-socket-server-address\">Set the Socket Server Address<\/h4>\n\n\n\n<p>Next, you need to define address the socket server listens on.<\/p>\n\n\n\n<p>By default, it is set to listen on all interfaces on port 9092\/tcp (Broker listener) and port 9093\/tcp (controller listener), <strong><code>listeners=PLAINTEXT:\/\/:9092,CONTROLLER:\/\/:9093<\/code><\/strong><\/p>\n\n\n\n<p>We will update this to set specific interface;<\/p>\n\n\n\n<p>Node 1;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#listeners=PLAINTEXT:\/\/:9092,CONTROLLER:\/\/:9093\n<strong>listeners=PLAINTEXT:\/\/192.168.57.32:9092,CONTROLLER:\/\/192.168.57.32:9093<\/strong><\/code><\/pre>\n\n\n\n<p>Node 2;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#listeners=PLAINTEXT:\/\/:9092,CONTROLLER:\/\/:9093\n<strong>listeners=PLAINTEXT:\/\/192.168.57.33:9092,CONTROLLER:\/\/192.168.57.33:9093<\/strong><\/code><\/pre>\n\n\n\n<p>Node 3;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#listeners=PLAINTEXT:\/\/:9092,CONTROLLER:\/\/:9093\n<strong>listeners=PLAINTEXT:\/\/192.168.57.35:9092,CONTROLLER:\/\/192.168.57.35:9093<\/strong><\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"update-the-broker-advertised-listener-address\">Update the Broker Advertised Listener Address<\/h4>\n\n\n\n<p>If you didn&#8217;t do this already, you need to update the broker listener address that is advised to the clients.<\/p>\n\n\n\n<p>By default set to localhost.<\/p>\n\n\n\n<p>Node 1<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#advertised.listeners=PLAINTEXT:\/\/localhost:9092\nadvertised.listeners=PLAINTEXT:\/\/192.168.57.32:9092<\/code><\/pre>\n\n\n\n<p>Node 2;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#advertised.listeners=PLAINTEXT:\/\/localhost:9092\nadvertised.listeners=PLAINTEXT:\/\/192.168.57.33:9092<\/code><\/pre>\n\n\n\n<p>Node 3<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#advertised.listeners=PLAINTEXT:\/\/localhost:9092\nadvertised.listeners=PLAINTEXT:\/\/192.168.57.35:9092<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"define-the-number-of-log-partitions-per-topic\">Define the Number of Log Partitions per Topic<\/h4>\n\n\n\n<p>More partitions allow greater parallelism for consumption, but this will also result in more files across the brokers.<\/p>\n\n\n\n<p>The default is set to 1. Ensure that you use a number that is at least divisible by the number of nodes in the cluster. Let&#8217;s use 6 in our case.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#num.partitions=1\n<strong>num.partitions=6<\/strong><\/code><\/pre>\n\n\n\n<p>In the very basic setup, those are just the only configs we can make. Save and exit the file.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"open-controller-broker-ports-on-firewall\">Open Controller\/Broker Ports on Firewall<\/h4>\n\n\n\n<p>Ensure that these ports, 9093\/tcp ( between controller nodes) and 9092\/tcp (between brokers and clients) are opened on firewall.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"are-you-upgrading-your-already-existing-kafka-cluster\">Are you Upgrading your Already Existing Kafka Cluster<\/h4>\n\n\n\n<p>If you are upgrading an already running Kafka cluster, the current state of the quorum of brokers is stored in the file called <strong>quorum-state<\/strong>. The quorum is a group of brokers that are responsible for managing the metadata for a Kafka cluster. The quorum-state file is used to ensure that all of the brokers in the quorum have the same view of the metadata.<\/p>\n\n\n\n<p>This file is usually located in the logs directory;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>find \/opt\/kafka -iname \"*quorum-state*\"<\/code><\/pre>\n\n\n\n<p>Sample outpu;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/opt\/kafka\/logs\/__cluster_metadata-0\/quorum-state<\/code><\/pre>\n\n\n\n<p>As you can see the sample contents of this file;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\"clusterId\":\"\",\"leaderId\":1,\"leaderEpoch\":12,\"votedId\":-1,\"appliedOffset\":0,\"currentVoters\":&#91;{\"voterId\":1}],\"data_version\":0}<\/code><\/pre>\n\n\n\n<p>The contents of the quorum-state tile you provided are as follows:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>clusterId:<\/strong>&nbsp;The ID of the Kafka cluster.<\/li>\n\n\n\n<li><strong>leaderId:<\/strong>&nbsp;The ID of the broker that is currently the leader of the cluster.<\/li>\n\n\n\n<li><strong>leaderEpoch:<\/strong>&nbsp;The epoch of the leader. This is a number that is incremented whenever the leader changes.<\/li>\n\n\n\n<li><strong>votedId:<\/strong>&nbsp;The ID of the broker that cast the most recent vote for leader.<\/li>\n\n\n\n<li><strong>appliedOffset:<\/strong>&nbsp;The offset of the last message that has been applied to the metadata log.<\/li>\n\n\n\n<li><strong>currentVoters:<\/strong>&nbsp;A list of the brokers that have voted for the current leader.<\/li>\n\n\n\n<li><strong>data_version:<\/strong>&nbsp;The version of the quorum-state file.<\/li>\n<\/ul>\n\n\n\n<p>If you restart a Kafka service for a cluster that you just upgraded, you may get the error;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ERROR &#91;SharedServer id=1] Got exception while starting SharedServer (kafka.server.SharedServer)\nkafka-server-start.sh&#91;12628]: java.lang.IllegalStateException: Configured voter set: &#91;1, 2, 3] is different from the voter set read from the state file: &#91;1]. Check if the quorum configuration is up to date, or wipe out the local state file if necessary<\/code><\/pre>\n\n\n\n<p>So, there are two ways to fix this;<\/p>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"1-manually-update-the-current-voters-id-in-the-quorum-state-file\">1. Manually update the current voters ID in the quorum-state file;<\/h5>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/opt\/kafka\/logs\/__cluster_metadata-0\/quorum-state<\/code><\/pre>\n\n\n\n<p>Change this (in my case);<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\"clusterId\":\"\",\"leaderId\":1,\"leaderEpoch\":12,\"votedId\":-1,\"appliedOffset\":0,\"currentVoters\":&#91;{\"voterId\":1}],\"data_version\":0}<\/code><\/pre>\n\n\n\n<p>to;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\"clusterId\":\"\",\"leaderId\":1,\"leaderEpoch\":12,\"votedId\":-1,\"appliedOffset\":0,\"currentVoters\":&#91;{\"voterId\":1},<strong>{\"voterId\":2},{\"voterId\":3}<\/strong>],\"data_version\":0}<\/code><\/pre>\n\n\n\n<p>(If you cloned the other nodes to make cluster, update the same on them)<\/p>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"2-delete-the-quorum-state-file\">2. Delete the Quorum-state file<\/h5>\n\n\n\n<p>You can remove the file, entirely;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>rm -rf \/opt\/kafka\/logs\/__cluster_metadata-0\/quorum-state<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"update-the-cluster-id\">Update the Cluster ID<\/h4>\n\n\n\n<p>When you setup KRaft Kafka, there is a step that you had to format Kafka logs directory to KRaft format. In the process, a random cluster ID (<strong>cluster.id<\/strong>) is generated. This information, is stored in the <strong><code>meta.properties<\/code><\/strong> in the logs directory. In order to avoid unexpected error due to INCONSISTENT_CLUSTER_ID in VOTE response, you need to edit the meta.properties file and change the ID to be same across all nodes.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat \/opt\/kafka\/logs\/meta.properties<\/code><\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code>\n#\n#Sun Jul 16 15:19:36 EDT 2023\nnode.id=1\nversion=1\ncluster.id=13OOWyQeQ_etLS811azqQQ\n<\/code><\/pre>\n\n\n\n<p>Thus, ensure this cluster.id value is same across the all the nodes.<\/p>\n\n\n\n<p>You can alway print the cluster ID using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/opt\/kafka\/bin\/kafka-cluster.sh cluster-id --bootstrap-server BROKER-IP:9092<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"restart-kafka-service\">Restart Kafka Service<\/h3>\n\n\n\n<p>Restart Kafka service to apply the changes.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl restart kafka<\/code><\/pre>\n\n\n\n<p>Check the service;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl status kafka<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\n\u25cf kafka.service - Apache Kafka\n     Loaded: loaded (\/etc\/systemd\/system\/kafka.service; enabled; preset: enabled)\n     Active: active (running) since Tue 2023-07-18 16:50:37 EDT; 50s ago\n   Main PID: 16581 (java)\n      Tasks: 90 (limit: 4642)\n     Memory: 763.7M\n        CPU: 11.661s\n     CGroup: \/system.slice\/kafka.service\n             \u2514\u250016581 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true \"-Xlog:>\n\nJul 18 16:50:42 knode2.kifarunix-demo.com kafka-server-start.sh[16581]: [2023-07-18 16:50:42,674] INFO Awaiting socket connections on 192.168.57.33:9092. (kafka.network.DataPlaneAcceptor)\nJul 18 16:50:42 knode2.kifarunix-demo.com kafka-server-start.sh[16581]: [2023-07-18 16:50:42,675] INFO [BrokerServer id=2] Waiting for all of the authorizer futures to be completed (kafka.server.BrokerServer)\nJul 18 16:50:42 knode2.kifarunix-demo.com kafka-server-start.sh[16581]: [2023-07-18 16:50:42,675] INFO [BrokerServer id=2] Finished waiting for all of the authorizer futures to be completed (kafka.server.Broker>\nJul 18 16:50:42 knode2.kifarunix-demo.com kafka-server-start.sh[16581]: [2023-07-18 16:50:42,675] INFO [BrokerServer id=2] Waiting for all of the SocketServer Acceptors to be started (kafka.server.BrokerServer)\nJul 18 16:50:42 knode2.kifarunix-demo.com kafka-server-start.sh[16581]: [2023-07-18 16:50:42,675] INFO [BrokerServer id=2] Finished waiting for all of the SocketServer Acceptors to be started (kafka.server.Brok>\nJul 18 16:50:42 knode2.kifarunix-demo.com kafka-server-start.sh[16581]: [2023-07-18 16:50:42,675] INFO [BrokerServer id=2] Transition from STARTING to STARTED (kafka.server.BrokerServer)\nJul 18 16:50:42 knode2.kifarunix-demo.com kafka-server-start.sh[16581]: [2023-07-18 16:50:42,675] INFO Kafka version: 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)\nJul 18 16:50:42 knode2.kifarunix-demo.com kafka-server-start.sh[16581]: [2023-07-18 16:50:42,675] INFO Kafka commitId: c97b88d5db4de28d (org.apache.kafka.common.utils.AppInfoParser)\nJul 18 16:50:42 knode2.kifarunix-demo.com kafka-server-start.sh[16581]: [2023-07-18 16:50:42,675] INFO Kafka startTimeMs: 1689713442675 (org.apache.kafka.common.utils.AppInfoParser)\nJul 18 16:50:42 knode2.kifarunix-demo.com kafka-server-start.sh[16581]: [2023-07-18 16:50:42,676] INFO [KafkaRaftServer nodeId=2] Kafka Server started (kafka.server.KafkaRaftServer)\n<\/code><\/pre>\n\n\n\n<p>Check the Kafka KRaft cluster ports;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ss -altnp | grep :90<\/code><\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code>\nLISTEN 0      50     [::ffff:192.168.57.35]:9092             *:*    users:((\"java\",pid=41651,fd=151))                                                                                                                            \nLISTEN 0      50     [::ffff:192.168.57.35]:9093             *:*    users:((\"java\",pid=41651,fd=132))\n<\/code><\/pre>\n\n\n\n<p>If any issue, check the logs for tips.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configure-producers-to-ship-logs-to-kafka-cluster\">Configure Producers to Ship Logs to Kafka Cluster<\/h3>\n\n\n\n<p>It is now time to test your Kafka cluster.<\/p>\n\n\n\n<p>In our previous tutorial, we configure Filebeat to collect system logs and ship them to Kafka.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/configuring-filebeat-to-send-logs-to-kafka\/\">Complete Guide: Configuring Filebeat to Send Logs to Kafka<\/a><\/p>\n\n\n\n<p>You might have a different type of producer setup but it doesn&#8217;t matter, your Kafka cluster should be able to receive logs. We will use the above guide to confirm the same in this guide.<\/p>\n\n\n\n<p>So, as you can see in the guide above, we have configure Filebeat to ship logs to <strong>filebeat<\/strong> topic on node1 in the Kafka cluster.<\/p>\n\n\n\n<p>Thus, login to the node1 and check the topics;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/opt\/kafka\/bin\/kafka-topics.sh --bootstrap-server 192.168.57.32:9092 --list<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>__consumer_offsets\n<strong>filebeat<\/strong>\nfilebeat-ssl\nkafka-topic-test<\/code><\/pre>\n\n\n\n<p>You can consume this topic to confirm if it is receiving logs;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/opt\/kafka\/bin\/kafka-console-consumer.sh --bootstrap-server 192.168.57.32:9092 --topic filebeat<\/code><\/pre>\n\n\n\n<p>Run the same consume script on other nodes;<\/p>\n\n\n\n<p>Node2;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/opt\/kafka\/bin\/kafka-console-consumer.sh --bootstrap-server 192.168.57.33:9092 --topic filebeat<\/code><\/pre>\n\n\n\n<p>Node 3;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/opt\/kafka\/bin\/kafka-console-consumer.sh --bootstrap-server 192.168.57.35:9092 --topic filebeat<\/code><\/pre>\n\n\n\n<p>You should be able to receive similar events.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"kafka-cluster-management-scripts\">Kafka Cluster Management Scripts<\/h3>\n\n\n\n<p>You can find a whole lot of Kafka scripts under <strong><code>\/opt\/kafka\/bin\/<\/code><\/strong> (as per our guide).<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ls -1 \/opt\/kafka\/bin\/ | grep <\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\nconnect-distributed.sh\nconnect-mirror-maker.sh\nconnect-standalone.sh\nkafka-acls.sh\nkafka-broker-api-versions.sh\nkafka-cluster.sh\nkafka-configs.sh\nkafka-console-consumer.sh\nkafka-console-producer.sh\nkafka-consumer-groups.sh\nkafka-consumer-perf-test.sh\nkafka-delegation-tokens.sh\nkafka-delete-records.sh\nkafka-dump-log.sh\nkafka-e2e-latency.sh\nkafka-features.sh\nkafka-get-offsets.sh\nkafka-jmx.sh\nkafka-leader-election.sh\nkafka-log-dirs.sh\nkafka-metadata-quorum.sh\nkafka-metadata-shell.sh\nkafka-mirror-maker.sh\nkafka-producer-perf-test.sh\nkafka-reassign-partitions.sh\nkafka-replica-verification.sh\nkafka-run-class.sh\nkafka-server-start.sh\nkafka-server-stop.sh\nkafka-storage.sh\nkafka-streams-application-reset.sh\nkafka-topics.sh\nkafka-transactions.sh\nkafka-verifiable-consumer.sh\nkafka-verifiable-producer.sh\ntrogdor.sh\nwindows\nzookeeper-security-migration.sh\nzookeeper-server-start.sh\nzookeeper-server-stop.sh\nzookeeper-shell.sh\n<\/code><\/pre>\n\n\n\n<p>To check how to use each script, consult the respective script help page.<\/p>\n\n\n\n<p>For example;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/opt\/kafka\/bin\/kafka-metadata-quorum.sh --help<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\nusage: kafka-metadata-quorum [-h] --bootstrap-server BOOTSTRAP_SERVER [--command-config COMMAND_CONFIG] {describe} ...\n\nThis tool describes kraft metadata quorum status.\n\npositional arguments:\n  {describe}\n    describe             Describe the metadata quorum info\n\noptional arguments:\n  -h, --help             show this help message and exit\n  --bootstrap-server BOOTSTRAP_SERVER\n                         A comma-separated list of host:port pairs to use for establishing the connection to the Kafka cluster.\n  --command-config COMMAND_CONFIG\n                         Property file containing configs to be passed to Admin Client.\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"manage-kafka-cluster-from-ui\">Manage Kafka Cluster from UI<\/h3>\n\n\n\n<p>You can manager your cluster using Kadeck;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-kadeck-apache-kafka-ui-tool-on-debian-ubuntu\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Kadeck Apache Kafka UI Tool on Debian\/Ubuntu<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"setup-kafka-kraft-cluster-with-ssl-tls\">Setup Kafka Kraft Cluster with SSL\/TLS<\/h3>\n\n\n\n<p>The guide below should be helpful in setting up Kafka Kraft cluster with SSL\/TLS;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/configure-apache-kafka-ssl-tls-encryption\/\" target=\"_blank\" rel=\"noreferrer noopener\">Quick Guide: Configure Apache Kafka SSL\/TLS Encryption for Enhanced Security<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this step-by-step guide, we will walk you through the process of how to setup a three-node Kafka KRaft cluster for scalable data streaming. Building<\/p>\n","protected":false},"author":10,"featured_media":17953,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121],"tags":[7073,7330,7071,7074,7329,7072],"class_list":["post-17933","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","tag-kafka-cluster","tag-kafka-cluster-setup","tag-kafka-kraft-cluster","tag-kraft","tag-three-node-kafka-cluster","tag-three-nodes-on-kafka-kraft","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/17933"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=17933"}],"version-history":[{"count":22,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/17933\/revisions"}],"predecessor-version":[{"id":20802,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/17933\/revisions\/20802"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/17953"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=17933"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=17933"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=17933"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}