{"id":22653,"date":"2024-06-08T10:09:56","date_gmt":"2024-06-08T07:09:56","guid":{"rendered":"https:\/\/kifarunix.com\/?p=22653"},"modified":"2024-06-18T22:41:12","modified_gmt":"2024-06-18T19:41:12","slug":"setup-highly-available-kubernetes-cluster-with-haproxy-and-keepalived","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/setup-highly-available-kubernetes-cluster-with-haproxy-and-keepalived\/","title":{"rendered":"Setup Highly Available Kubernetes Cluster with Haproxy and Keepalived"},"content":{"rendered":"\n<p>This tutorial provides a step-by-step guide on how to setup highly available Kubernetes cluster with Haproxy and Keepalived. For anyone managing critical containerized applications with Kubernetes, ensuring reliability isn&#8217;t just an option\u2014it&#8217;s a necessity. Downtime is more than an inconvenience; it&#8217;s a direct threat to revenue, brand reputation, and operational stability. This is where a Highly Available (HA) Kubernetes cluster setup becomes indispensable. An HA cluster guarantees that your Kubernetes deployment can endure node failures without affecting your applications&#8217; performance. But how do you achieve this robustness?<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#how-to-set-up-a-highly-available-kubernetes-cluster\">How to Set Up a Highly Available Kubernetes Cluster<\/a><ul><li><a href=\"#understanding-ha-in-kubernetes\">Understanding HA in Kubernetes<\/a><\/li><li><a href=\"#kubernetes-ha-setup-options\">Kubernetes HA Setup Options<\/a><\/li><li><a href=\"#our-deployment-architecture-stacked-control-plane\">Our Deployment Architecture: Stacked Control Plane<\/a><\/li><li><a href=\"#deploy-kubernetes-in-ha-with-haproxy-and-keepalived\">Deploy Kubernetes in HA with Haproxy and Keepalived<\/a><ul><li><a href=\"#why-choose-ha-proxy-and-keepalived\">Why Choose HAProxy and Keepalived?<\/a><\/li><li><a href=\"#install-haproxy-and-keepalived-on-load-balancer-nodes\">Install Haproxy and Keepalived on Load Balancer Nodes<\/a><\/li><li><a href=\"#configure-keepalived-to-provide-virtual-ip\">Configure Keepalived to Provide Virtual IP<\/a><\/li><li><a href=\"#configure-ha-proxy-load-balancer\">Configure HAProxy Load Balancer<\/a><\/li><li><a href=\"#prepare-nodes-for-kubernetes-deployment\">Prepare Nodes for Kubernetes Deployment<\/a><\/li><li><a href=\"#open-kubernetes-cluster-ports-on-firewall\">Open Kubernetes Cluster Ports on Firewall<\/a><\/li><li><a href=\"#initialize-first-control-plane\">Initialize First Control Plane<\/a><\/li><li><a href=\"#deploy-pod-network-addon-on-the-control-plane\">Deploy Pod Network Addon on the Control Plane<\/a><\/li><li><a href=\"#initialize-other-control-plane-nodes\">Initialize Other Control Plane Nodes<\/a><\/li><li><a href=\"#add-worker-nodes-to-kubernetes-cluster\">Add Worker Nodes to Kubernetes Cluster<\/a><\/li><li><a href=\"#get-kubernetes-cluster-information\">Get Kubernetes Cluster Information<\/a><\/li><li><a href=\"#list-kubernetes-cluster-api-resources\">List Kubernetes Cluster API Resources<\/a><\/li><li><a href=\"#testing-the-ha-setup\">Testing the HA Setup<\/a><\/li><\/ul><\/li><li><a href=\"#app-armor-bug-blocks-runc-signals-pods-stuck-terminating\">AppArmor Bug Blocks runc Signals, Pods Stuck Terminating<\/a><\/li><li><a href=\"#conclusion\">Conclusion<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"how-to-set-up-a-highly-available-kubernetes-cluster\">How to Set Up a Highly Available Kubernetes Cluster<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"understanding-ha-in-kubernetes\">Understanding HA in Kubernetes<\/h3>\n\n\n\n<p>A Kubernetes cluster is usually made up a <strong>control plane<\/strong> and <strong>worker<\/strong> nodes. While <strong>control plane<\/strong>, also known as <strong>master node<\/strong>, controls the Kubernetes cluster by managing the scheduling and orchestration of applications, maintaining the cluster&#8217;s desired state and responding to cluster events, <strong>worker nodes<\/strong> handle the real workload of running and managing containers (<strong>Pods<\/strong>).<\/p>\n\n\n\n<p>Most Kubernetes deployment usually run a multi-worker nodes and a single control plane architectures. While there is a distributed workload among the worker nodes in this setup, it poses a risk of inability to manage and control the cluster&#8217;s resources, such as scheduling new pods, updating configurations, and handling API requests in the event that the control plane goes down.<\/p>\n\n\n\n<p>These are the issues that redundancy and high availability aims to solve as it ensures:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Fault tolerance<\/strong>: HA ensures that the Kubernetes cluster can continue to operate even if individual components or nodes fail.<\/li>\n\n\n\n<li><strong>Business Continuity<\/strong>: HA setup minimizes the risk of downtime, ensuring business continuity and customer satisfaction.<\/li>\n\n\n\n<li><strong>Resilience to Maintenance<\/strong>: HA setup facilitate rolling updates and maintenance activities without disrupting application availability.<\/li>\n\n\n\n<li><strong>Improved Performance<\/strong>: HA configurations can enhance performance by distributing workloads across multiple nodes and leveraging resources more efficiently.<\/li>\n\n\n\n<li><strong>Disaster Recovery<\/strong>: In the event of a catastrophic failure or disaster, HA setup can help minimize data loss and facilitate recovery.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"kubernetes-ha-setup-options\">Kubernetes HA Setup Options<\/h3>\n\n\n\n<p>There are two major options of deploying Kubernetes cluster in a high availability setup: <strong>stacked control plane nodes<\/strong> and <strong>external etcd cluster<\/strong>. Each of these setups have their own pros and cons.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Stacked Control Plane Nodes<\/strong>:\n<ul class=\"wp-block-list\">\n<li>In this setup, multiple master nodes are deployed to provide high availability.<\/li>\n\n\n\n<li>As usual, all control plane components (such as the API server, scheduler, controller manager, and etcd) are installed on each master node.<\/li>\n\n\n\n<li>This setup simplifies deployment and management as all components are colocated on the same nodes. Hence, suitable for smaller deployments.<\/li>\n\n\n\n<li>However, it introduces a risk of compromised redundancy. If one control node goes down, all the components including the key-value data store, <strong>etcd<\/strong>, in the same node, becomes inaccessible.<\/li>\n\n\n\n<li>To mitigate this risk, you need multiple control planes and at least three of them are recommended.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>External etcd Cluster<\/strong>:\n<ul class=\"wp-block-list\">\n<li>In this configuration, the etcd key-value store, which stores cluster state and configuration data, is deployed separately from the master nodes.<\/li>\n\n\n\n<li>The etcd cluster can be deployed in a highly available manner across multiple nodes or even multiple data centers, ensuring resilience against failures.<\/li>\n\n\n\n<li>This setup adds complexity to the deployment and management of the Kubernetes cluster but provides greater fault tolerance and reliability.<\/li>\n\n\n\n<li>It&#8217;s recommended for production environments or large-scale deployments where high availability and data integrity are critical requirements.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"our-deployment-architecture-stacked-control-plane\">Our Deployment Architecture: Stacked Control Plane<\/h3>\n\n\n\n<p>In this guide, we will deploy a highly available Kubernetes cluster using stacked control plane architecture. The diagram below depicts our architecture.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1256\" height=\"702\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/06\/kubernetes-highly-available-setup-with-haproxy-keepalived.png?v=1717693989\" alt=\"\" class=\"wp-image-22701\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/06\/kubernetes-highly-available-setup-with-haproxy-keepalived.png?v=1717693989 1256w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/06\/kubernetes-highly-available-setup-with-haproxy-keepalived-768x429.png?v=1717693989 768w\" sizes=\"(max-width: 1256px) 100vw, 1256px\" \/><\/figure>\n\n\n\n<p>From the architecture above, we have;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>2 load balancers running Haproxy and Keepalived<\/li>\n\n\n\n<li>3 Kubernetes control planes in stacked topology.<\/li>\n\n\n\n<li>3 Kubernetes worker nodes<\/li>\n<\/ul>\n\n\n\n<p>Kubernetes cluster nodes;<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td><strong>Node<\/strong><\/td><td><strong>Hostname<\/strong><\/td><td><strong>IP Address<\/strong><\/td><td><strong>vCPUs<\/strong><\/td><td><strong>RAM (GB)<\/strong><\/td><td><strong>OS<\/strong><\/td><\/tr><tr><td>LB 01<\/td><td>lb-01<\/td><td>192.168.122.56<\/td><td>2<\/td><td>4<\/td><td>Ubuntu 24.04<\/td><\/tr><tr><td>LB 02<\/td><td>lb-02<\/td><td>192.168.122.57<\/td><td>2<\/td><td>4<\/td><td>Ubuntu 24.04<\/td><\/tr><tr><td>Master 1<\/td><td>master-01<\/td><td>192.168.122.58<\/td><td>2<\/td><td>4<\/td><td>Ubuntu 24.04<\/td><\/tr><tr><td>Master 2<\/td><td>master-02<\/td><td>192.168.122.59<\/td><td>2<\/td><td>4<\/td><td>Ubuntu 24.04<\/td><\/tr><tr><td>Master 3<\/td><td>master-03<\/td><td>192.168.122.60<\/td><td>2<\/td><td>4<\/td><td>Ubuntu 24.04<\/td><\/tr><tr><td>Worker 1<\/td><td>worker-01<\/td><td>192.168.122.61<\/td><td>2<\/td><td>4<\/td><td>Ubuntu 24.04<\/td><\/tr><tr><td>Worker 2<\/td><td>worker-02<\/td><td>192.168.122.62<\/td><td>2<\/td><td>4<\/td><td>Ubuntu 24.04<\/td><\/tr><tr><td>Worker 3<\/td><td>worker-03<\/td><td>192.168.122.63<\/td><td>2<\/td><td>4<\/td><td>Ubuntu 24.04<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>All the nodes used in this guide are running Ubuntu 24.04 LTS server.<\/p>\n\n\n\n<p>Therefore, to proceed with this setup;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploy-kubernetes-in-ha-with-haproxy-and-keepalived\">Deploy Kubernetes in HA with Haproxy and Keepalived<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"why-choose-ha-proxy-and-keepalived\">Why Choose HAProxy and Keepalived?<\/h4>\n\n\n\n<p><strong>HAProxy<\/strong> is a powerful, open-source load balancer and proxy server known for its high performance and reliability. It distributes traffic across your Kubernetes control plane nodes, ensuring that the load is balanced and no single node is overwhelmed.<\/p>\n\n\n\n<p><strong>Keepalived<\/strong> is a routing software that provides high availability and load balancing. It complements HAProxy by managing Virtual IP addresses (VIPs), ensuring that if one load balancer goes down, another can take over seamlessly.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-haproxy-and-keepalived-on-load-balancer-nodes\">Install Haproxy and Keepalived on Load Balancer Nodes<\/h4>\n\n\n\n<p>On Load balance node lb-01 and lb-02, install both Haproxy and Keepalived packages.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/how-to-install-keepalived-on-ubuntu-24-04\/\" target=\"_blank\" rel=\"noreferrer noopener\">How to Install Keepalived on Ubuntu 24.04<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-configure-haproxy-on-ubuntu-24-04\/#install-ha-proxy-on-ubuntu-24-04\" target=\"_blank\" rel=\"noreferrer noopener\">Install HAProxy on Ubuntu 24.04<\/a><\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-keepalived-to-provide-virtual-ip\">Configure Keepalived to Provide Virtual IP<\/h4>\n\n\n\n<p>Once you have installed Keepalived package, you can configure it to provide a VIP.<\/p>\n\n\n\n<p>Here is our sample configurations;<\/p>\n\n\n\n<p>LB-01;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat \/etc\/keepalived\/keepalived.conf<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>global_defs {\n    enable_script_security\n    script_user root\n    vrrp_version 3\n    vrrp_min_garp true\n}\n\nvrrp_script chk_haproxy {\n    script \"\/usr\/bin\/systemctl is-active --quiet haproxy\"\n    fall 2\n    rise 2\n    interval 2\n    weight 50\n}\n\nvrrp_instance LB_VIP {\n    state MASTER\n    interface enp1s0\n\n    virtual_router_id 51\n    priority 150\n\n    advert_int 1\n\n    track_interface {\n        enp1s0 weight 50\n    }\n\n    track_script {\n        chk_haproxy\n    }\n\n    virtual_ipaddress {\n        192.168.122.254\/24\n    }\n}\n<\/code><\/pre>\n\n\n\n<p>LB-02;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat \/etc\/keepalived\/keepalived.conf<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>global_defs {\n    enable_script_security\n    script_user root\n    vrrp_version 3\n    vrrp_min_garp true\n}\n\nvrrp_script chk_haproxy {\n    script \"\/usr\/bin\/systemctl is-active --quiet haproxy\"\n    fall 2\n    rise 2\n    interval 2\n    weight 50\n}\n\nvrrp_instance LB_VIP {\n    state BACKUP\n    interface enp1s0\n\n    virtual_router_id 51\n    priority 100\n\n    advert_int 1\n\n    track_interface {\n        enp1s0 weight 50\n    }\n\n    track_script {\n        chk_haproxy\n    }\n\n    virtual_ipaddress {\n        192.168.122.254\/24\n    }\n}\n<\/code><\/pre>\n\n\n\n<p>Start Keepalived on both nodes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo systemctl enable --now keepalived<\/code><\/pre>\n\n\n\n<p>Confirm IP address on the node defined as master lb-01 (with higher priority value);<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ip -br a<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>lo               UNKNOWN        127.0.0.1\/8 ::1\/128 \nenp1s0           UP             192.168.122.56\/24 192.168.122.254\/32 fe80::5054:ff:fedb:1d46\/64\n<\/code><\/pre>\n\n\n\n<p>As you can see, we have VIP (192.168.122.254) assigned.<\/p>\n\n\n\n<p>Sample logs can be checked using;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo journalctl -f -u keepalived<\/code><\/pre>\n\n\n\n<p>You can also check traffic related to VRRP protocol (protocol 112)<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo tcpdump proto 112<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-ha-proxy-load-balancer\">Configure HAProxy Load Balancer<\/h4>\n\n\n\n<p>On both load balancers, the HAProxy configuration files have the same settings.<\/p>\n\n\n\n<p>Sample confugration used;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat \/etc\/haproxy\/haproxy.cfg<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>global\n        log \/var\/log\/haproxy.log local0 info\n        log-send-hostname\n\tchroot \/var\/lib\/haproxy\n\tstats socket \/run\/haproxy\/admin.sock mode 660 level admin\n\tstats timeout 30s\n\tuser haproxy\n\tgroup haproxy\n\tdaemon\n\tmaxconn 524272\n\ndefaults\n\tlog\tglobal\n\tmode\thttp\n\toption\thttplog\n\toption\tdontlognull\n        timeout connect 5000\n        timeout client  50000\n        timeout server  50000\n\terrorfile 400 \/etc\/haproxy\/errors\/400.http\n\terrorfile 403 \/etc\/haproxy\/errors\/403.http\n\terrorfile 408 \/etc\/haproxy\/errors\/408.http\n\terrorfile 500 \/etc\/haproxy\/errors\/500.http\n\terrorfile 502 \/etc\/haproxy\/errors\/502.http\n\terrorfile 503 \/etc\/haproxy\/errors\/503.http\n\terrorfile 504 \/etc\/haproxy\/errors\/504.http\n\nfrontend kube-apiserver\n        bind *:6443     # BIND to any address so it is accessible via VIP\n\tmode tcp\n        option tcplog\n        default_backend kube-apiserver\n\nbackend kube-apiserver\n        balance roundrobin\n\tmode tcp\n\toption tcp-check\n        server  master-01   192.168.122.58:6443 check\n        server  master-02   192.168.122.59:6443 check\n        server  master-03   192.168.122.60:6443 check\n\nlisten stats\n        bind 192.168.122.254:8443\n        stats enable                    # enable statistics reports  \n        stats hide-version              # Hide the version of HAProxy\n        stats refresh 30s               # HAProxy refresh time\n        stats show-node                 # Shows the hostname of the node\n        stats auth haadmin:P@ssword     # Enforce Basic authentication for Stats page\n        stats uri \/stats                # Statistics URL\n\n<\/code><\/pre>\n\n\n\n<p>Be sure to confirm validity of the HAProxy configuration file;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo haproxy -f \/etc\/haproxy\/haproxy.cfg -c -V<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Configuration file is valid<\/code><\/pre>\n\n\n\n<p>Update HAProxy file descriptor (FD)\/open files (NOFILE) limit (<em>done system wide<\/em>);<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"fs.nr_open = 1048599\" | sudo tee -a \/etc\/sysctl.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo sysctl -p<\/code><\/pre>\n\n\n\n<p>Start and enable HAProxy to run on system bootl<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo systemctl enable --now haproxy<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"prepare-nodes-for-kubernetes-deployment\">Prepare Nodes for Kubernetes Deployment<\/h4>\n\n\n\n<p>In this setup, we will use <strong>kubeadm<\/strong> to deploy our Kubernetes cluster, the usual way. Therefore, follow the steps below setup the nodes for Kubernetes deployment in HA.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><a href=\"https:\/\/kifarunix.com\/install-and-setup-kubernetes-cluster-on-ubuntu-24-04\/#disable-swap-on-cluster-nodes\" target=\"_blank\" rel=\"noreferrer noopener\">Disable Swap on Cluster Nodes<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kifarunix.com\/install-and-setup-kubernetes-cluster-on-ubuntu-24-04\/#enable-kernel-ip-forwarding-on-cluster-nodes\" target=\"_blank\" rel=\"noreferrer noopener\">Enable Kernel IP forwarding on Cluster Nodes<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kifarunix.com\/install-and-setup-kubernetes-cluster-on-ubuntu-24-04\/#load-some-required-kernel-modules-on-cluster-nodes\" target=\"_blank\" rel=\"noreferrer noopener\">Load Some Required Kernel Modules on Cluster Nodes<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kifarunix.com\/install-and-setup-kubernetes-cluster-on-ubuntu-24-04\/#install-container-runtime-on-ubuntu-24-04\" target=\"_blank\" rel=\"noreferrer noopener\">Install Container Runtime on Cluster Nodes<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kifarunix.com\/install-and-setup-kubernetes-cluster-on-ubuntu-24-04\/#install-kubernetes-on-ubuntu-24-04\" target=\"_blank\" rel=\"noreferrer noopener\">Install Kubernetes on\u00a0Cluster Nodes<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kifarunix.com\/install-and-setup-kubernetes-cluster-on-ubuntu-24-04\/#mark-hold-kubernetes-packages\" target=\"_blank\" rel=\"noreferrer noopener\">Mark Hold Kubernetes Packages<\/a><\/li>\n<\/ol>\n\n\n\n<p>It is recommended that the versions Kubernetes components, <strong>kubeadm<\/strong>, <strong>kubelet<\/strong>, <strong>kubectl<\/strong>, match.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl version -o yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>clientVersion:\n  buildDate: \"2024-05-14T10:50:53Z\"\n  compiler: gc\n  gitCommit: 6911225c3f747e1cd9d109c305436d08b668f086\n  gitTreeState: clean\n  <strong>gitVersion: v1.30.1\n  goVersion: go1.22.2<\/strong>\n  major: \"1\"\n  minor: \"30\"\n  platform: linux\/amd64\nkustomizeVersion: v5.0.4-0.20230601165947-6ce0bf390ce3\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>kubelet --version<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Kubernetes <strong>v1.30.1<\/strong><\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>kubeadm version -o yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>clientVersion:\n  buildDate: \"2024-05-14T10:49:05Z\"\n  compiler: gc\n  gitCommit: 6911225c3f747e1cd9d109c305436d08b668f086\n  gitTreeState: clean\n<strong>  gitVersion: v1.30.1\n  goVersion: go1.22.2<\/strong>\n  major: \"1\"\n  minor: \"30\"\n  platform: linux\/amd64\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"open-kubernetes-cluster-ports-on-firewall\">Open Kubernetes Cluster Ports on Firewall<\/h4>\n\n\n\n<p>Ensure that the required cluster ports are opened.<\/p>\n\n\n\n<p>On the Load balancer, ensure the API server port, 6443\/tcp is opened;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo iptables -A INPUT -p tcp -m multiport --dports 22,6443 -j ACCEPT<\/code><\/pre>\n\n\n\n<p>On Control plane nodes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo iptables -A INPUT -p tcp -m multiport --dports 6443,2379:2380,10250:10252 -j ACCEPT<\/code><\/pre>\n\n\n\n<p>On Worker Nodes<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo iptables -A INPUT -p tcp -m multiport --dports 10250,10256,30000:32767 -j ACCEPT<\/code><\/pre>\n\n\n\n<p>You can also use UFW or Firewalld&#8230;<\/p>\n\n\n\n<p>Save the rules;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo iptables-save | sudo tee \/etc\/iptables\/rules.v4<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo systemctl restart netfilter-persistent<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"initialize-first-control-plane\">Initialize First Control Plane<\/h4>\n\n\n\n<p>Once the nodes are ready as per above steps, login to one of the control plane nodes, for example, <strong>master-01<\/strong> in our setup, and initialize it using the command below.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>sudo kubeadm init \\\n\t--control-plane-endpoint \"LOAD_BALANCER_ADDRESS:LOAD_BALANCER_PORT\" \\\n\t--upload-certs \\\n\t--pod-network-cidr=POD_NETWORK\n<\/code><\/pre>\n\n\n\n<p>Where<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>&#8211;control-plane-endpoint <\/strong>&#8220;<strong>LOAD_BALANCER_ADDRESS:LOAD_BALANCER_PORT<\/strong>&#8221; defines the Load balancer VIP and port. The control plane components will register themselves with this endpoint. Replace LOAD_BALANCER_ADDRESS with the IP\/DNS name of the load balancer and LOAD_BALANCER_PORT with the port number.<\/li>\n\n\n\n<li><strong>&#8211;upload-certs<\/strong> option tells <code>kubeadm<\/code> to upload the TLS certificates to the Kubernetes control plane configuration. This is necessary for other control plane nodes to join the cluster securely. It is also possible to manually upload the certs<\/li>\n\n\n\n<li><strong>&#8211;pod-network-cidr=POD_NETWORK<\/strong>: Depending on the whether the CNI you will use require the Pod network defined, you need to specify the same. We will be using Calico CNI, which requires Pod network defined, hence. This option sets the CIDR (Classless Inter-Domain Routing) block for the pod network. The pod network CIDR must not overlap with any existing networks in your environment and must be large enough to accommodate the maximum number of pods you anticipate deploying in your cluster<\/li>\n<\/ul>\n\n\n\n<p>So, my first control plane initialization command will be like;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>sudo kubeadm init \\\n\t--control-plane-endpoint \"192.168.122.254:6443\" \\\n\t--upload-certs \\\n\t--pod-network-cidr=10.100.0.0\/16\n<\/code><\/pre>\n\n\n\n<p>Sample control plane initialization command output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>[init] Using Kubernetes version: v1.30.1\n[preflight] Running pre-flight checks\n[preflight] Pulling images required for setting up a Kubernetes cluster\n[preflight] This might take a minute or two, depending on the speed of your internet connection\n[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'\n[certs] Using certificateDir folder \"\/etc\/kubernetes\/pki\"\n[certs] Generating \"ca\" certificate and key\n[certs] Generating \"apiserver\" certificate and key\n[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master-01] and IPs [10.96.0.1 192.168.122.58 192.168.122.254]\n[certs] Generating \"apiserver-kubelet-client\" certificate and key\n[certs] Generating \"front-proxy-ca\" certificate and key\n[certs] Generating \"front-proxy-client\" certificate and key\n[certs] Generating \"etcd\/ca\" certificate and key\n[certs] Generating \"etcd\/server\" certificate and key\n[certs] etcd\/server serving cert is signed for DNS names [localhost master-01] and IPs [192.168.122.58 127.0.0.1 ::1]\n[certs] Generating \"etcd\/peer\" certificate and key\n[certs] etcd\/peer serving cert is signed for DNS names [localhost master-01] and IPs [192.168.122.58 127.0.0.1 ::1]\n[certs] Generating \"etcd\/healthcheck-client\" certificate and key\n[certs] Generating \"apiserver-etcd-client\" certificate and key\n[certs] Generating \"sa\" key and public key\n[kubeconfig] Using kubeconfig folder \"\/etc\/kubernetes\"\n[kubeconfig] Writing \"admin.conf\" kubeconfig file\n[kubeconfig] Writing \"super-admin.conf\" kubeconfig file\n[kubeconfig] Writing \"kubelet.conf\" kubeconfig file\n[kubeconfig] Writing \"controller-manager.conf\" kubeconfig file\n[kubeconfig] Writing \"scheduler.conf\" kubeconfig file\n[etcd] Creating static Pod manifest for local etcd in \"\/etc\/kubernetes\/manifests\"\n[control-plane] Using manifest folder \"\/etc\/kubernetes\/manifests\"\n[control-plane] Creating static Pod manifest for \"kube-apiserver\"\n[control-plane] Creating static Pod manifest for \"kube-controller-manager\"\n[control-plane] Creating static Pod manifest for \"kube-scheduler\"\n[kubelet-start] Writing kubelet environment file with flags to file \"\/var\/lib\/kubelet\/kubeadm-flags.env\"\n[kubelet-start] Writing kubelet configuration to file \"\/var\/lib\/kubelet\/config.yaml\"\n[kubelet-start] Starting the kubelet\n[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory \"\/etc\/kubernetes\/manifests\"\n[kubelet-check] Waiting for a healthy kubelet. This can take up to 4m0s\n[kubelet-check] The kubelet is healthy after 502.855988ms\n[api-check] Waiting for a healthy API server. This can take up to 4m0s\n[api-check] The API server is healthy after 3.505516376s\n[upload-config] Storing the configuration used in ConfigMap \"kubeadm-config\" in the \"kube-system\" Namespace\n[kubelet] Creating a ConfigMap \"kubelet-config\" in namespace kube-system with the configuration for the kubelets in the cluster\n[upload-certs] Storing the certificates in Secret \"kubeadm-certs\" in the \"kube-system\" Namespace\n[upload-certs] Using certificate key:\n8089be8fb63febd32a17e9d623d6c514088235de2637c75add56c9905078575f\n[mark-control-plane] Marking the node master-01 as control-plane by adding the labels: [node-role.kubernetes.io\/control-plane node.kubernetes.io\/exclude-from-external-load-balancers]\n[mark-control-plane] Marking the node master-01 as control-plane by adding the taints [node-role.kubernetes.io\/control-plane:NoSchedule]\n[bootstrap-token] Using token: x1yxt2.3bbj98bg05ynqx6x\n[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles\n[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes\n[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials\n[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token\n[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster\n[bootstrap-token] Creating the \"cluster-info\" ConfigMap in the \"kube-public\" namespace\n[kubelet-finalize] Updating \"\/etc\/kubernetes\/kubelet.conf\" to point to a rotatable kubelet client certificate and key\n[addons] Applied essential addon: CoreDNS\n[addons] Applied essential addon: kube-proxy\n\nYour Kubernetes control-plane has initialized successfully!\n\nTo start using your cluster, you need to run the following as a regular user:\n\n  mkdir -p $HOME\/.kube\n  sudo cp -i \/etc\/kubernetes\/admin.conf $HOME\/.kube\/config\n  sudo chown $(id -u):$(id -g) $HOME\/.kube\/config\n\nAlternatively, if you are the root user, you can run:\n\n  export KUBECONFIG=\/etc\/kubernetes\/admin.conf\n\nYou should now deploy a pod network to the cluster.\nRun \"kubectl apply -f [podnetwork].yaml\" with one of the options listed at:\n  https:\/\/kubernetes.io\/docs\/concepts\/cluster-administration\/addons\/\n\nYou can now join any number of the control-plane node running the following command on each as root:\n\n  kubeadm join 192.168.122.254:6443 --token x1yxt2.3bbj98bg05ynqx6x \\\n\t--discovery-token-ca-cert-hash sha256:9e8aa4b38599a819f5b80de36871d95947295135b30a07915f7cf152760bbf4f \\\n\t--control-plane --certificate-key 8089be8fb63febd32a17e9d623d6c514088235de2637c75add56c9905078575f\n\nPlease note that the certificate-key gives access to cluster sensitive data, keep it secret!\nAs a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use\n\"kubeadm init phase upload-certs --upload-certs\" to reload certs afterward.\n\nThen you can join any number of worker nodes by running the following on each as root:\n\nkubeadm join 192.168.122.254:6443 --token x1yxt2.3bbj98bg05ynqx6x \\\n\t--discovery-token-ca-cert-hash sha256:9e8aa4b38599a819f5b80de36871d95947295135b30a07915f7cf152760bbf4f\n<\/code><\/pre>\n\n\n\n<p>Next, to be able to interact with the cluster from the <strong>first<\/strong> control plane, run the following commands, <strong>as non-root user<\/strong>.<\/p>\n\n\n\n<p>Next, create a Kubernetes cluster directory.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">mkdir -p $HOME\/.kube<\/pre>\n\n\n\n<p>Copy Kubernetes admin configuration file to the cluster directory created above.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo cp -i \/etc\/kubernetes\/admin.conf $HOME\/.kube\/config<\/pre>\n\n\n\n<p>Set the proper ownership for the cluster configuration file.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo chown $(id -u):$(id -g) $HOME\/.kube\/config<\/pre>\n\n\n\n<p>Verify that you can interact with the Kubernetes cluster from the first control plane by running random <strong>kubectl<\/strong> commands;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl cluster-info<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Kubernetes control plane is running at https:\/\/192.168.122.254:6443\nCoreDNS is running at https:\/\/192.168.122.254:6443\/api\/v1\/namespaces\/kube-system\/services\/kube-dns:dns\/proxy\n\nTo further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"deploy-pod-network-addon-on-the-control-plane\">Deploy Pod Network Addon on the Control Plane<\/h4>\n\n\n\n<p>For Pods to communicate with one another, you must deploy a&nbsp;<a href=\"https:\/\/kubernetes.io\/docs\/concepts\/extend-kubernetes\/compute-storage-net\/network-plugins\/#cni\" target=\"_blank\" rel=\"noreferrer noopener\">Container Network Interface<\/a>&nbsp;(CNI) based Pod network add-on.<\/p>\n\n\n\n<p>Check the link below on how to install a CNI.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-setup-kubernetes-cluster-on-ubuntu-24-04\/#install-pod-network-addon-on-master-node\" target=\"_blank\" rel=\"noreferrer noopener\">How to Install Kubernetes Pod Network Addon<\/a><\/p>\n\n\n\n<p>After a short while. you can verify that Calico CNI pods are running;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pods --all-namespaces<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>kubectl get pods --all-namespaces\nNAMESPACE         NAME                                       READY   STATUS    RESTARTS   AGE\ncalico-system     calico-kube-controllers-68cdb9587c-9xffh   1\/1     Running   0          39s\ncalico-system     calico-node-lbhjx                          1\/1     Running   0          39s\ncalico-system     calico-typha-5845c66fc9-7vv7x              1\/1     Running   0          39s\ncalico-system     csi-node-driver-6mpls                      2\/2     Running   0          39s\nkube-system       coredns-7db6d8ff4d-jnswv                   1\/1     Running   0          3m5s\nkube-system       coredns-7db6d8ff4d-n9sz6                   1\/1     Running   0          3m5s\nkube-system       etcd-master-01                             1\/1     Running   0          3m20s\nkube-system       kube-apiserver-master-01                   1\/1     Running   0          3m20s\nkube-system       kube-controller-manager-master-01          1\/1     Running   0          3m20s\nkube-system       kube-proxy-txjrh                           1\/1     Running   0          3m5s\nkube-system       kube-scheduler-master-01                   1\/1     Running   0          3m20s\ntigera-operator   tigera-operator-7d5cd7fcc8-l8bl5           1\/1     Running   0          71s\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"initialize-other-control-plane-nodes\">Initialize Other Control Plane Nodes<\/h4>\n\n\n\n<p>When you initialized the first control plane, a command to initialize and join other control planes into the cluster is printed to the standard output.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>You can now join any number of the control-plane node running the following command on each as root:\n\n  kubeadm join 192.168.122.254:6443 --token x1yxt2.3bbj98bg05ynqx6x \\\n\t--discovery-token-ca-cert-hash sha256:9e8aa4b38599a819f5b80de36871d95947295135b30a07915f7cf152760bbf4f \\\n\t--control-plane --certificate-key 8089be8fb63febd32a17e9d623d6c514088235de2637c75add56c9905078575f\n<\/code><\/pre>\n\n\n\n<p>Thus, copy the command and execute it on the other control plane nodes to join them to the cluster;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>sudo kubeadm join 192.168.122.254:6443 --token x1yxt2.3bbj98bg05ynqx6x \\\n\t--discovery-token-ca-cert-hash sha256:9e8aa4b38599a819f5b80de36871d95947295135b30a07915f7cf152760bbf4f \\\n\t--control-plane \\\n\t--certificate-key 8089be8fb63febd32a17e9d623d6c514088235de2637c75add56c9905078575f\n<\/code><\/pre>\n\n\n\n<blockquote class=\"wp-block-quote has-small-font-size is-layout-flow wp-block-quote-is-layout-flow\">\n<p>Please note that the certificate-key gives access to cluster sensitive data. As a safeguard, uploaded-certs will be deleted in two hours. If you want to add another control plane into the cluster after two hours since you initialized the first control plane, you can use the command below to re-upload the certificates and generate a new decryption key.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo kubeadm init phase upload-certs --upload-certs<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;upload-certs] Storing the certificates in Secret \"kubeadm-certs\" in the \"kube-system\" Namespace\n&#91;upload-certs] Using certificate key:\n<strong>458b0e87a28080c4792333e2d1fdbe7c28ea216e72016998c0b04326a75579c8<\/strong><\/code><\/pre>\n\n\n\n<p>Print the join command<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubeadm token create --print-join-command<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubeadm join 192.168.122.254:6443 --token q7sc7n.snwhru3n8e3o9lsq --discovery-token-ca-cert-hash sha256:ac08ef4c66538dfbf86a9cd554399c3d979ff370dfc9ca9119ac4ec45fdd0691<\/code><\/pre>\n\n\n\n<p>Then the command to join other control plane into cluster becomes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo kubeadm join 192.168.122.254:6443 --token q7sc7n.snwhru3n8e3o9lsq --discovery-token-ca-cert-hash sha256:ac08ef4c66538dfbf86a9cd554399c3d979ff370dfc9ca9119ac4ec45fdd0691 --control-plane --certificate-key XXX<\/code><\/pre>\n\n\n\n<p>Where XXX is the certificate key printed by the <strong>sudo kubeadm init phase upload-certs &#8211;upload-certs<\/strong> command.<\/p>\n<\/blockquote>\n\n\n\n<p>Sample cluster join command output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>[preflight] Running pre-flight checks\n[preflight] Reading configuration from the cluster...\n[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'\n[preflight] Running pre-flight checks before initializing the new control plane instance\n[preflight] Pulling images required for setting up a Kubernetes cluster\n[preflight] This might take a minute or two, depending on the speed of your internet connection\n[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'\n[download-certs] Downloading the certificates in Secret \"kubeadm-certs\" in the \"kube-system\" Namespace\n[download-certs] Saving the certificates to the folder: \"\/etc\/kubernetes\/pki\"\n[certs] Using certificateDir folder \"\/etc\/kubernetes\/pki\"\n[certs] Generating \"apiserver\" certificate and key\n[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master-02] and IPs [10.96.0.1 192.168.122.59 192.168.122.254]\n[certs] Generating \"apiserver-kubelet-client\" certificate and key\n[certs] Generating \"front-proxy-client\" certificate and key\n[certs] Generating \"apiserver-etcd-client\" certificate and key\n[certs] Generating \"etcd\/server\" certificate and key\n[certs] etcd\/server serving cert is signed for DNS names [localhost master-02] and IPs [192.168.122.59 127.0.0.1 ::1]\n[certs] Generating \"etcd\/peer\" certificate and key\n[certs] etcd\/peer serving cert is signed for DNS names [localhost master-02] and IPs [192.168.122.59 127.0.0.1 ::1]\n[certs] Generating \"etcd\/healthcheck-client\" certificate and key\n[certs] Valid certificates and keys now exist in \"\/etc\/kubernetes\/pki\"\n[certs] Using the existing \"sa\" key\n[kubeconfig] Generating kubeconfig files\n[kubeconfig] Using kubeconfig folder \"\/etc\/kubernetes\"\n[kubeconfig] Writing \"admin.conf\" kubeconfig file\n[kubeconfig] Writing \"controller-manager.conf\" kubeconfig file\n[kubeconfig] Writing \"scheduler.conf\" kubeconfig file\n[control-plane] Using manifest folder \"\/etc\/kubernetes\/manifests\"\n[control-plane] Creating static Pod manifest for \"kube-apiserver\"\n[control-plane] Creating static Pod manifest for \"kube-controller-manager\"\n[control-plane] Creating static Pod manifest for \"kube-scheduler\"\n[check-etcd] Checking that the etcd cluster is healthy\n[kubelet-start] Writing kubelet configuration to file \"\/var\/lib\/kubelet\/config.yaml\"\n[kubelet-start] Writing kubelet environment file with flags to file \"\/var\/lib\/kubelet\/kubeadm-flags.env\"\n[kubelet-start] Starting the kubelet\n[kubelet-check] Waiting for a healthy kubelet. This can take up to 4m0s\n[kubelet-check] The kubelet is healthy after 501.537305ms\n[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap\n[etcd] Announced new etcd member joining to the existing etcd cluster\n[etcd] Creating static Pod manifest for \"etcd\"\n{\"level\":\"warn\",\"ts\":\"2024-06-08T05:37:40.997002Z\",\"logger\":\"etcd-client\",\"caller\":\"v3@v3.5.10\/retry_interceptor.go:62\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"etcd-endpoints:\/\/0xc0006b9180\/192.168.122.58:2379\",\"attempt\":0,\"error\":\"rpc error: code = FailedPrecondition desc = etcdserver: can only promote a learner member which is in sync with leader\"}\n{\"level\":\"warn\",\"ts\":\"2024-06-08T05:37:41.494954Z\",\"logger\":\"etcd-client\",\"caller\":\"v3@v3.5.10\/retry_interceptor.go:62\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"etcd-endpoints:\/\/0xc0006b9180\/192.168.122.58:2379\",\"attempt\":0,\"error\":\"rpc error: code = FailedPrecondition desc = etcdserver: can only promote a learner member which is in sync with leader\"}\n[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s\nThe 'update-status' phase is deprecated and will be removed in a future release. Currently it performs no operation\n[mark-control-plane] Marking the node master-02 as control-plane by adding the labels: [node-role.kubernetes.io\/control-plane node.kubernetes.io\/exclude-from-external-load-balancers]\n[mark-control-plane] Marking the node master-02 as control-plane by adding the taints [node-role.kubernetes.io\/control-plane:NoSchedule]\n\nThis node has joined the cluster and a new control plane instance was created:\n\n* Certificate signing request was sent to apiserver and approval was received.\n* The Kubelet was informed of the new secure connection details.\n* Control plane label and taint were applied to the new node.\n* The Kubernetes control plane instances scaled up.\n* A new etcd member was added to the local\/stacked etcd cluster.\n\nTo start administering your cluster from this node, you need to run the following as a regular user:\n\n\tmkdir -p $HOME\/.kube\n\tsudo cp -i \/etc\/kubernetes\/admin.conf $HOME\/.kube\/config\n\tsudo chown $(id -u):$(id -g) $HOME\/.kube\/config\n\nRun 'kubectl get nodes' to see this node join the cluster.\n<\/code><\/pre>\n\n\n\n<p>You can see that the type of cluster is auto-detected, <strong>A new etcd member was added to the local\/stacked etcd cluster.<\/strong><\/p>\n\n\n\n<p>To start administering your cluster from other control plane nodes, you need to run the following as a regular user:<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>mkdir -p $HOME\/.kube\nsudo cp -i \/etc\/kubernetes\/admin.conf $HOME\/.kube\/config\nsudo chown $(id -u):$(id -g) $HOME\/.kube\/config\n<\/code><\/pre>\n\n\n\n<p>Run the same join command on other control plane node and install Kubeconfig to allow you administer cluster as regular user.<\/p>\n\n\n\n<p>Then run the command below to confirm if the node is added to the cluster.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get nodes<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>NAME        STATUS   ROLES           AGE     VERSION\nmaster-01   Ready    control-plane   13m     v1.30.1\nmaster-02   Ready    control-plane   5m39s   v1.30.1\nmaster-03   Ready    control-plane   18s     v1.30.1\n<\/code><\/pre>\n\n\n\n<p>As you can see, we now have three control plane nodes in the cluster.<\/p>\n\n\n\n<p>Similarly, check the Pods related to the control plane on the <strong>kube-system<\/strong> namespace.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pods -n kube-system<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                                READY   STATUS    RESTARTS   AGE\ncoredns-7db6d8ff4d-jnswv            1\/1     Running   0          21m\ncoredns-7db6d8ff4d-n9sz6            1\/1     Running   0          21m\netcd-master-01                      1\/1     Running   0          21m\netcd-master-02                      1\/1     Running   0          14m\netcd-master-03                      1\/1     Running   0          8m55s\nkube-apiserver-master-01            1\/1     Running   0          21m\nkube-apiserver-master-02            1\/1     Running   0          14m\nkube-apiserver-master-03            1\/1     Running   0          9m1s\nkube-controller-manager-master-01   1\/1     Running   0          21m\nkube-controller-manager-master-02   1\/1     Running   0          14m\nkube-controller-manager-master-03   1\/1     Running   0          8m54s\nkube-proxy-hfb98                    1\/1     Running   0          9m3s\nkube-proxy-mfwvj                    1\/1     Running   0          14m\nkube-proxy-txjrh                    1\/1     Running   0          21m\nkube-scheduler-master-01            1\/1     Running   0          21m\nkube-scheduler-master-02            1\/1     Running   0          14m\nkube-scheduler-master-03            1\/1     Running   0          8m59s\n<\/code><\/pre>\n\n\n\n<p>Looks good!<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"add-worker-nodes-to-kubernetes-cluster\">Add Worker Nodes to Kubernetes Cluster<\/h4>\n\n\n\n<p>You can now add Worker nodes to the Kubernetes cluster using the&nbsp;<strong>kubeadm join<\/strong>&nbsp;command.<\/p>\n\n\n\n<p>Ensure that container runtime is installed, configured and running. We are using&nbsp;<strong>containerd<\/strong>&nbsp;CRI;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl status containerd<\/code><\/pre>\n\n\n\n<p>Sample output from worker01 node;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\u25cf containerd.service - containerd container runtime\n     Loaded: loaded (\/usr\/lib\/systemd\/system\/containerd.service; enabled; preset: enabled)\n     Active: active (running) since Fri 2024-06-07 06:20:19 UTC; 23h ago\n       Docs: https:\/\/containerd.io\n   Main PID: 38484 (containerd)\n      Tasks: 9\n     Memory: 12.9M (peak: 13.4M)\n        CPU: 2min 32.555s\n     CGroup: \/system.slice\/containerd.service\n             \u2514\u250038484 \/usr\/bin\/containerd\n\nJun 07 06:20:19 worker-01 containerd[38484]: time=\"2024-06-07T06:20:19.751551807Z\" level=info msg=serving... address=\/run\/containerd\/containerd.sock.ttrpc\nJun 07 06:20:19 worker-01 containerd[38484]: time=\"2024-06-07T06:20:19.751584533Z\" level=info msg=serving... address=\/run\/containerd\/containerd.sock\nJun 07 06:20:19 worker-01 containerd[38484]: time=\"2024-06-07T06:20:19.751793867Z\" level=info msg=\"Start subscribing containerd event\"\nJun 07 06:20:19 worker-01 containerd[38484]: time=\"2024-06-07T06:20:19.751859875Z\" level=info msg=\"Start recovering state\"\nJun 07 06:20:19 worker-01 containerd[38484]: time=\"2024-06-07T06:20:19.751939832Z\" level=info msg=\"Start event monitor\"\nJun 07 06:20:19 worker-01 containerd[38484]: time=\"2024-06-07T06:20:19.751986850Z\" level=info msg=\"Start snapshots syncer\"\nJun 07 06:20:19 worker-01 containerd[38484]: time=\"2024-06-07T06:20:19.751999301Z\" level=info msg=\"Start cni network conf syncer for default\"\nJun 07 06:20:19 worker-01 containerd[38484]: time=\"2024-06-07T06:20:19.752004404Z\" level=info msg=\"Start streaming server\"\nJun 07 06:20:19 worker-01 containerd[38484]: time=\"2024-06-07T06:20:19.752041268Z\" level=info msg=\"containerd successfully booted in 0.068873s\"\nJun 07 06:20:19 worker-01 systemd[1]: Started containerd.service - containerd container runtime.\n<\/code><\/pre>\n\n\n\n<p>Next, get the cluster join command that was output during cluster boot strapping and execute on <strong>each<\/strong> worker node.<\/p>\n\n\n\n<p>Note that this command is displayed after initializing the first control plane above;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>...\nThen you can join any number of worker nodes by running the following on each as root:\n\nkubeadm join 192.168.122.254:6443 --token x1yxt2.3bbj98bg05ynqx6x \\\n\t--discovery-token-ca-cert-hash sha256:9e8aa4b38599a819f5b80de36871d95947295135b30a07915f7cf152760bbf4f\n<\/code><\/pre>\n\n\n\n<p>Get the command and execute it as root user.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>sudo kubeadm join 192.168.122.254:6443 \\\n\t--token x1yxt2.3bbj98bg05ynqx6x \\\n\t--discovery-token-ca-cert-hash sha256:9e8aa4b38599a819f5b80de36871d95947295135b30a07915f7cf152760bbf4f\n<\/code><\/pre>\n\n\n\n<p>If you didn\u2019t save the Kubernetes Cluster joining command, you can at any given time print using the command below on any of the control plane nodes;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">kubeadm token create --print-join-command<\/pre>\n\n\n\n<p>Worker node cluster join command sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>[preflight] Running pre-flight checks\n[preflight] Reading configuration from the cluster...\n[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'\n[kubelet-start] Writing kubelet configuration to file \"\/var\/lib\/kubelet\/config.yaml\"\n[kubelet-start] Writing kubelet environment file with flags to file \"\/var\/lib\/kubelet\/kubeadm-flags.env\"\n[kubelet-start] Starting the kubelet\n[kubelet-check] Waiting for a healthy kubelet. This can take up to 4m0s\n[kubelet-check] The kubelet is healthy after 501.841547ms\n[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap\n\nThis node has joined the cluster:\n* Certificate signing request was sent to apiserver and a response was received.\n* The Kubelet was informed of the new secure connection details.\n\nRun 'kubectl get nodes' on the control-plane to see this node join the cluster.\n<\/code><\/pre>\n\n\n\n<p>On the Kubernetes control plane (<em>as the regular user with which you created the cluster as<\/em>), run the command below to verify that the nodes have joined the cluster.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">kubectl get nodes<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME        STATUS   ROLES           AGE     VERSION\nmaster-01   Ready    control-plane   37m     v1.30.1\nmaster-02   Ready    control-plane   29m     v1.30.1\nmaster-03   Ready    control-plane   24m     v1.30.1\nworker-01   Ready    <none>          2m53s   v1.30.1\nworker-02   Ready    <none>          22s     v1.30.1\nworker-03   Ready    <none>          17s     v1.30.1\n<\/code><\/pre>\n\n\n\n<p>All worker nodes are joined to the cluster and are <strong>Ready<\/strong> to handle workloads.<\/p>\n\n\n\n<p>Role of the Worker nodes may show up as&nbsp;<strong><code>&lt;none&gt;<\/code><\/strong>. This is okay. No role is assigned to the node by default. It is only until the control plane assign a workload on the node then it shows up the correct role.<\/p>\n\n\n\n<p>You can however update this ROLE using the command;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl label node &lt;worker-node-name&gt; node-role.kubernetes.io\/worker=true<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"get-kubernetes-cluster-information\">Get Kubernetes Cluster Information<\/h4>\n\n\n\n<p>As you can see, we now have a cluster. Run the command below to get cluster information.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">kubectl cluster-info<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Kubernetes control plane is running at https:\/\/192.168.122.254:6443\nCoreDNS is running at https:\/\/192.168.122.254:6443\/api\/v1\/namespaces\/kube-system\/services\/kube-dns:dns\/proxy\n\nTo further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"list-kubernetes-cluster-api-resources\">List Kubernetes Cluster API Resources<\/h4>\n\n\n\n<p>You can list all Kubernetes cluster resources using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl api-resources<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                                SHORTNAMES                                      APIVERSION                        NAMESPACED   KIND\nbindings                                                                            v1                                true         Binding\ncomponentstatuses                   cs                                              v1                                false        ComponentStatus\nconfigmaps                          cm                                              v1                                true         ConfigMap\nendpoints                           ep                                              v1                                true         Endpoints\nevents                              ev                                              v1                                true         Event\nlimitranges                         limits                                          v1                                true         LimitRange\nnamespaces                          ns                                              v1                                false        Namespace\nnodes                               no                                              v1                                false        Node\npersistentvolumeclaims              pvc                                             v1                                true         PersistentVolumeClaim\npersistentvolumes                   pv                                              v1                                false        PersistentVolume\npods                                po                                              v1                                true         Pod\npodtemplates                                                                        v1                                true         PodTemplate\nreplicationcontrollers              rc                                              v1                                true         ReplicationController\nresourcequotas                      quota                                           v1                                true         ResourceQuota\nsecrets                                                                             v1                                true         Secret\nserviceaccounts                     sa                                              v1                                true         ServiceAccount\nservices                            svc                                             v1                                true         Service\nmutatingwebhookconfigurations                                                       admissionregistration.k8s.io\/v1   false        MutatingWebhookConfiguration\nvalidatingadmissionpolicies                                                         admissionregistration.k8s.io\/v1   false        ValidatingAdmissionPolicy\nvalidatingadmissionpolicybindings                                                   admissionregistration.k8s.io\/v1   false        ValidatingAdmissionPolicyBinding\nvalidatingwebhookconfigurations                                                     admissionregistration.k8s.io\/v1   false        ValidatingWebhookConfiguration\ncustomresourcedefinitions           crd,crds                                        apiextensions.k8s.io\/v1           false        CustomResourceDefinition\napiservices                                                                         apiregistration.k8s.io\/v1         false        APIService\ncontrollerrevisions                                                                 apps\/v1                           true         ControllerRevision\ndaemonsets                          ds                                              apps\/v1                           true         DaemonSet\ndeployments                         deploy                                          apps\/v1                           true         Deployment\nreplicasets                         rs                                              apps\/v1                           true         ReplicaSet\nstatefulsets                        sts                                             apps\/v1                           true         StatefulSet\nselfsubjectreviews                                                                  authentication.k8s.io\/v1          false        SelfSubjectReview\ntokenreviews                                                                        authentication.k8s.io\/v1          false        TokenReview\nlocalsubjectaccessreviews                                                           authorization.k8s.io\/v1           true         LocalSubjectAccessReview\nselfsubjectaccessreviews                                                            authorization.k8s.io\/v1           false        SelfSubjectAccessReview\nselfsubjectrulesreviews                                                             authorization.k8s.io\/v1           false        SelfSubjectRulesReview\nsubjectaccessreviews                                                                authorization.k8s.io\/v1           false        SubjectAccessReview\nhorizontalpodautoscalers            hpa                                             autoscaling\/v2                    true         HorizontalPodAutoscaler\ncronjobs                            cj                                              batch\/v1                          true         CronJob\njobs                                                                                batch\/v1                          true         Job\ncertificatesigningrequests          csr                                             certificates.k8s.io\/v1            false        CertificateSigningRequest\nleases                                                                              coordination.k8s.io\/v1            true         Lease\nbgpconfigurations                                                                   crd.projectcalico.org\/v1          false        BGPConfiguration\nbgpfilters                                                                          crd.projectcalico.org\/v1          false        BGPFilter\nbgppeers                                                                            crd.projectcalico.org\/v1          false        BGPPeer\nblockaffinities                                                                     crd.projectcalico.org\/v1          false        BlockAffinity\ncaliconodestatuses                                                                  crd.projectcalico.org\/v1          false        CalicoNodeStatus\nclusterinformations                                                                 crd.projectcalico.org\/v1          false        ClusterInformation\nfelixconfigurations                                                                 crd.projectcalico.org\/v1          false        FelixConfiguration\nglobalnetworkpolicies                                                               crd.projectcalico.org\/v1          false        GlobalNetworkPolicy\nglobalnetworksets                                                                   crd.projectcalico.org\/v1          false        GlobalNetworkSet\nhostendpoints                                                                       crd.projectcalico.org\/v1          false        HostEndpoint\nipamblocks                                                                          crd.projectcalico.org\/v1          false        IPAMBlock\nipamconfigs                                                                         crd.projectcalico.org\/v1          false        IPAMConfig\nipamhandles                                                                         crd.projectcalico.org\/v1          false        IPAMHandle\nippools                                                                             crd.projectcalico.org\/v1          false        IPPool\nipreservations                                                                      crd.projectcalico.org\/v1          false        IPReservation\nkubecontrollersconfigurations                                                       crd.projectcalico.org\/v1          false        KubeControllersConfiguration\nnetworkpolicies                                                                     crd.projectcalico.org\/v1          true         NetworkPolicy\nnetworksets                                                                         crd.projectcalico.org\/v1          true         NetworkSet\nendpointslices                                                                      discovery.k8s.io\/v1               true         EndpointSlice\nevents                              ev                                              events.k8s.io\/v1                  true         Event\nflowschemas                                                                         flowcontrol.apiserver.k8s.io\/v1   false        FlowSchema\nprioritylevelconfigurations                                                         flowcontrol.apiserver.k8s.io\/v1   false        PriorityLevelConfiguration\ningressclasses                                                                      networking.k8s.io\/v1              false        IngressClass\ningresses                           ing                                             networking.k8s.io\/v1              true         Ingress\nnetworkpolicies                     netpol                                          networking.k8s.io\/v1              true         NetworkPolicy\nruntimeclasses                                                                      node.k8s.io\/v1                    false        RuntimeClass\napiservers                                                                          operator.tigera.io\/v1             false        APIServer\nimagesets                                                                           operator.tigera.io\/v1             false        ImageSet\ninstallations                                                                       operator.tigera.io\/v1             false        Installation\ntigerastatuses                                                                      operator.tigera.io\/v1             false        TigeraStatus\npoddisruptionbudgets                pdb                                             policy\/v1                         true         PodDisruptionBudget\nbgpconfigurations                   bgpconfig,bgpconfigs                            projectcalico.org\/v3              false        BGPConfiguration\nbgpfilters                                                                          projectcalico.org\/v3              false        BGPFilter\nbgppeers                                                                            projectcalico.org\/v3              false        BGPPeer\nblockaffinities                     blockaffinity,affinity,affinities               projectcalico.org\/v3              false        BlockAffinity\ncaliconodestatuses                  caliconodestatus                                projectcalico.org\/v3              false        CalicoNodeStatus\nclusterinformations                 clusterinfo                                     projectcalico.org\/v3              false        ClusterInformation\nfelixconfigurations                 felixconfig,felixconfigs                        projectcalico.org\/v3              false        FelixConfiguration\nglobalnetworkpolicies               gnp,cgnp,calicoglobalnetworkpolicies            projectcalico.org\/v3              false        GlobalNetworkPolicy\nglobalnetworksets                                                                   projectcalico.org\/v3              false        GlobalNetworkSet\nhostendpoints                       hep,heps                                        projectcalico.org\/v3              false        HostEndpoint\nipamconfigurations                  ipamconfig                                      projectcalico.org\/v3              false        IPAMConfiguration\nippools                                                                             projectcalico.org\/v3              false        IPPool\nipreservations                                                                      projectcalico.org\/v3              false        IPReservation\nkubecontrollersconfigurations                                                       projectcalico.org\/v3              false        KubeControllersConfiguration\nnetworkpolicies                     cnp,caliconetworkpolicy,caliconetworkpolicies   projectcalico.org\/v3              true         NetworkPolicy\nnetworksets                         netsets                                         projectcalico.org\/v3              true         NetworkSet\nprofiles                                                                            projectcalico.org\/v3              false        Profile\nclusterrolebindings                                                                 rbac.authorization.k8s.io\/v1      false        ClusterRoleBinding\nclusterroles                                                                        rbac.authorization.k8s.io\/v1      false        ClusterRole\nrolebindings                                                                        rbac.authorization.k8s.io\/v1      true         RoleBinding\nroles                                                                               rbac.authorization.k8s.io\/v1      true         Role\npriorityclasses                     pc                                              scheduling.k8s.io\/v1              false        PriorityClass\ncsidrivers                                                                          storage.k8s.io\/v1                 false        CSIDriver\ncsinodes                                                                            storage.k8s.io\/v1                 false        CSINode\ncsistoragecapacities                                                                storage.k8s.io\/v1                 true         CSIStorageCapacity\nstorageclasses                      sc                                              storage.k8s.io\/v1                 false        StorageClass\nvolumeattachments                                                                   storage.k8s.io\/v1                 false        VolumeAttachment\n<\/code><\/pre>\n\n\n\n<p>You are now ready to deploy an application on Kubernetes cluster.<\/p>\n\n\n\n<p>You can also check the Load balancer statistics;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1621\" height=\"573\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/06\/kubernetes-api-server-loadbalancer-stats.png?v=1717827398\" alt=\"kubernetes ha cluster stats\" class=\"wp-image-22715\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/06\/kubernetes-api-server-loadbalancer-stats.png?v=1717827398 1621w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/06\/kubernetes-api-server-loadbalancer-stats-768x271.png?v=1717827398 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/06\/kubernetes-api-server-loadbalancer-stats-1536x543.png?v=1717827398 1536w\" sizes=\"(max-width: 1621px) 100vw, 1621px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"testing-the-ha-setup\">Testing the HA Setup<\/h4>\n\n\n\n<p>As a basic HA setup test, simulate a failure by shutting down one control plane node or load balancer. Verify that the cluster remains functional and the VIP is still accessible.<\/p>\n\n\n\n<p>For example, you can shut down one of the Load balancers to begin with;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>root@lb-01:~# systemctl poweroff \n\nBroadcast message from root@lb-01 on pts\/1 (Sat 2024-06-08 06:57:48 UTC):\n\nThe system will power off now!\n\nroot@lb-01:~# Connection to 192.168.122.56 closed by remote host.\nConnection to 192.168.122.56 closed.\n<\/code><\/pre>\n\n\n\n<p>Check that you can still be able to access the API server via the VIP address. For example, let&#8217;s try to get the nodes details from control plane<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get node --selector='node-role.kubernetes.io\/control-plane'<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME        STATUS   ROLES           AGE   VERSION\nmaster-01   Ready    control-plane   88m   v1.30.1\nmaster-02   Ready    control-plane   80m   v1.30.1\nmaster-03   Ready    control-plane   75m   v1.30.1\n<\/code><\/pre>\n\n\n\n<p>The fact that I am able to use the <strong>kubectl<\/strong> to access the cluster basically shows that the LB high availability is working.<\/p>\n\n\n\n<p>You can bring up the load balancer and shut down one of the control plane nodes, e.g master-01;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>kifarunix@master-01:~$ sudo systemctl poweroff \n\nBroadcast message from root@master-01 on pts\/1 (Sat 2024-06-08 07:01:07 UTC):\n\nThe system will power off now!\n<\/code><\/pre>\n\n\n\n<p>From the other control plane nodes, check if you can administer cluster as usual;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get nodes --selector=\"node-role.kubernetes.io\/control-plane\"<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME        STATUS     ROLES           AGE   VERSION\nmaster-01   NotReady   control-plane   93m   v1.30.1\nmaster-02   Ready      control-plane   85m   v1.30.1\nmaster-03   Ready      control-plane   80m   v1.30.1\n<\/code><\/pre>\n\n\n\n<p>Master-01 is not ready but still am able to run the cluster!<\/p>\n\n\n\n<p>You can do thorough tests to be completely sure that you have a working cluster.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"app-armor-bug-blocks-runc-signals-pods-stuck-terminating\">AppArmor Bug Blocks runc Signals, Pods Stuck Terminating<\/h3>\n\n\n\n<p>You might have realized that in the recent version of Ubuntu, there is an issue whereby draining the nodes or deleting the pods get stuck with such errors in apparmor logs as;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>2024-06-14T19:04:43.331091+00:00 worker-01 kernel: audit: type=1400 audit(1718391883.329:221): apparmor=\"<strong>DENIED<\/strong>\" operation=\"signal\" class=\"signal\" profile=\"cri-containerd.apparmor.d\" pid=7445 comm=\"runc\" requested_mask=\"receive\" denied_mask=\"receive\" signal=kill peer=\"runc<\/code><\/pre>\n\n\n\n<p>This is a bug on AppArmor profile that denies signals from runc. This results in many pods being stuck in a terminating state. The bug was reported by Sebastian Podjasek on 2024-05-10. It affects Ubuntu containerd-app package.<\/p>\n\n\n\n<p>Read how to fix on <a href=\"https:\/\/kifarunix.com\/kubernetes-nodes-maintenance-drain-vs-cordon-demystified\/#kubectl-drain-node-gets-stuck-forever-apparmor-bug\" target=\"_blank\" rel=\"noreferrer noopener\">kubectl drain node gets stuck forever [Apparmor Bug]<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"conclusion\">Conclusion<\/h3>\n\n\n\n<p>In this blog post, you have successfully deployed Kubernetes in High availability with HAProxy and Keepalived. To ensure you have a visibility on what is happening on the cluster, you can introduce monitoring and alerting.<\/p>\n\n\n\n<p>Read more on <a href=\"https:\/\/kubernetes.io\/docs\/setup\/production-environment\/tools\/kubeadm\/high-availability\/\" target=\"_blank\" rel=\"noreferrer noopener\">Creating Highly Available Clusters with kubeadm<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This tutorial provides a step-by-step guide on how to setup highly available Kubernetes cluster with Haproxy and Keepalived. For anyone managing critical containerized applications with<\/p>\n","protected":false},"author":10,"featured_media":22701,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[1076,121,1668],"tags":[7514,7516,7515],"class_list":["post-22653","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-containers","category-howtos","category-kubernetes","tag-install-kubernetes-in-high-availability","tag-k8s-3-control-plane-nodes","tag-k8s-ha","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/22653"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=22653"}],"version-history":[{"count":27,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/22653\/revisions"}],"predecessor-version":[{"id":22942,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/22653\/revisions\/22942"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/22701"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=22653"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=22653"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=22653"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}