{"id":22539,"date":"2024-05-14T20:34:32","date_gmt":"2024-05-14T17:34:32","guid":{"rendered":"https:\/\/kifarunix.com\/?p=22539"},"modified":"2024-07-06T00:02:53","modified_gmt":"2024-07-05T21:02:53","slug":"install-and-setup-kubernetes-cluster-on-ubuntu-24-04","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/install-and-setup-kubernetes-cluster-on-ubuntu-24-04\/","title":{"rendered":"Install and Setup Kubernetes Cluster on Ubuntu 24.04"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1062\" height=\"597\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/06\/kubernetes-cluster-setup.png\" alt=\"Setup Kubernetes Cluster on Ubuntu\" class=\"wp-image-16327\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/06\/kubernetes-cluster-setup.png?v=1682803985 1062w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/06\/kubernetes-cluster-setup-768x432.png?v=1682803985 768w\" sizes=\"(max-width: 1062px) 100vw, 1062px\" \/><\/figure>\n\n\n\n<p>This tutorial provides a step by step guide on how to install and setup Kubernetes Cluster on Ubuntu 24.04. Kubernetes, according to <a href=\"https:\/\/kubernetes.io\/\" target=\"_blank\" rel=\"noreferrer noopener\">kubernetes.io<\/a>, is an open-source production-grade container orchestration platform. It facilitates automated deployment, scaling and management of containerized applications.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"install-and-setup-kubernetes-cluster-on-ubuntu-24-04\">Install and Setup Kubernetes Cluster on Ubuntu 24.04<\/h2>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#install-and-setup-kubernetes-cluster-on-ubuntu-24-04\">Install and Setup Kubernetes Cluster on Ubuntu 24.04<\/a><ul><li><a href=\"#kubernetes-cluster-architecture\">Kubernetes Cluster Architecture<\/a><\/li><li><a href=\"#run-system-update-on-cluster-nodes\">Run System Update on Cluster Nodes<\/a><\/li><li><a href=\"#disable-swap-on-cluster-nodes\">Disable Swap on Cluster Nodes<\/a><\/li><li><a href=\"#enable-kernel-ip-forwarding-on-cluster-nodes\">Enable Kernel IP forwarding on Cluster Nodes<\/a><\/li><li><a href=\"#load-some-required-kernel-modules-on-cluster-nodes\">Load Some Required Kernel Modules on Cluster Nodes<\/a><\/li><li><a href=\"#install-container-runtime-on-ubuntu-24-04\">Install Container Runtime on Ubuntu 24.04<\/a><ul><li><a href=\"#install-containerd-runtime-on-all-cluster-nodes\">Install Containerd Runtime on all Cluster Nodes<\/a><\/li><li><a href=\"#configure-cgroup-driver-for-container-d\">Configure Cgroup Driver for ContainerD<\/a><\/li><\/ul><\/li><li><a href=\"#install-kubernetes-on-ubuntu-24-04\">Install Kubernetes on Ubuntu 24.04<\/a><ul><li><a href=\"#install-kubernetes-repository-gpg-signing-key\">Install Kubernetes Repository GPG Signing Key<\/a><\/li><li><a href=\"#install-kubernetes-repository-on-ubuntu-24-04\">Install Kubernetes Repository on Ubuntu 24.04<\/a><\/li><li><a href=\"#install-kubernetes-components-on-all-the-nodes\">Install Kubernetes components on all the nodes<\/a><\/li><li><a href=\"#mark-hold-kubernetes-packages\">Mark Hold Kubernetes Packages<\/a><\/li><\/ul><\/li><li><a href=\"#initialize-kubernetes-cluster-on-control-plane-using-kubeadm\">Initialize Kubernetes Cluster on Control Plane using Kubeadm<\/a><\/li><li><a href=\"#install-pod-network-addon-on-master-node\">Install Pod Network Addon on Master Node<\/a><\/li><li><a href=\"#get-running-pods-in-the-kubernetes-cluster\">Get Running Pods in the Kubernetes cluster<\/a><\/li><li><a href=\"#open-kubernetes-cluster-ports-on-firewall\">Open Kubernetes Cluster Ports on Firewall<\/a><\/li><li><a href=\"#add-worker-nodes-to-kubernetes-cluster\">Add Worker Nodes to Kubernetes Cluster<\/a><\/li><li><a href=\"#get-kubernetes-cluster-information\">Get Kubernetes Cluster Information<\/a><\/li><li><a href=\"#list-kubernetes-cluster-api-resources\">List Kubernetes Cluster API Resources<\/a><\/li><li><a href=\"#remove-worker-nodes-from-cluster\">Remove Worker Nodes from Cluster<\/a><\/li><li><a href=\"#app-armor-blocks-runc-signals-pods-stuck-terminating\">AppArmor Blocks runc Signals, Pods Stuck Terminating<\/a><\/li><li><a href=\"#install-kubernetes-dashboard\">Install Kubernetes Dashboard<\/a><\/li><li><a href=\"#further-reading\">Further Reading<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"kubernetes-cluster-architecture\">Kubernetes Cluster Architecture<\/h3>\n\n\n\n<p>In this tutorial, we are going install and setup a four node (one control plane and three worker nodes) Kubernetes cluster.<\/p>\n\n\n\n<p>A Kubernetes cluster is composed a Master node which hosts the control plane and a Worker node which hosts Pods.<\/p>\n\n\n\n<p>Check our guide on a high-level overview of Kubernetes cluster to understand more on this.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/kubernetes-architecture-a-high-level-overview-of-kubernetes-cluster-components\/\" target=\"_blank\" rel=\"noreferrer noopener\">Kubernetes Architecture: A High-level Overview of Kubernetes Cluster Components<\/a><\/p>\n\n\n\n<p>Below are our node details.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td><strong>Node<\/strong><\/td><td class=\"has-text-align-left\" data-align=\"left\"><strong>Hostname<\/strong><\/td><td><strong>IP Address<\/strong><\/td><td class=\"has-text-align-center\" data-align=\"center\"><strong>vCPUs<\/strong><\/td><td class=\"has-text-align-center\" data-align=\"center\"><strong>RAM (GB)<\/strong><\/td><td><strong>OS<\/strong><\/td><\/tr><tr><td>Master<\/td><td class=\"has-text-align-left\" data-align=\"left\">master.kifarunix.com<\/td><td>192.168.122.60<\/td><td class=\"has-text-align-center\" data-align=\"center\">2<\/td><td class=\"has-text-align-center\" data-align=\"center\">2<\/td><td>Ubuntu 24.04<\/td><\/tr><tr><td>Worker 1<\/td><td class=\"has-text-align-left\" data-align=\"left\">wk01.kifarunix.com<\/td><td>192.168.122.61<\/td><td class=\"has-text-align-center\" data-align=\"center\">2<\/td><td class=\"has-text-align-center\" data-align=\"center\">2<\/td><td>Ubuntu 24.04<\/td><\/tr><tr><td>Worker 2<\/td><td class=\"has-text-align-left\" data-align=\"left\">wk02.kifarunix.com<\/td><td>192.168.122.62<\/td><td class=\"has-text-align-center\" data-align=\"center\">2<\/td><td class=\"has-text-align-center\" data-align=\"center\">2<\/td><td>Ubuntu 24.04<\/td><\/tr><tr><td>Worker 3<\/td><td class=\"has-text-align-left\" data-align=\"left\">wk03.kifarunix.com<\/td><td>192.168.122.63<\/td><td class=\"has-text-align-center\" data-align=\"center\">2<\/td><td class=\"has-text-align-center\" data-align=\"center\">2<\/td><td>Ubuntu 24.04<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">Kubernetes cluster nodes<\/figcaption><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"run-system-update-on-cluster-nodes\">Run System Update on Cluster Nodes<\/h3>\n\n\n\n<p>To begin with, update system package cache on all the nodes;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo apt update<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"disable-swap-on-cluster-nodes\">Disable Swap on Cluster Nodes<\/h3>\n\n\n\n<p>Running Kubernetes requires that you disable swap. <\/p>\n\n\n\n<p>Check if swap is enabled.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">swapon --show<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME      TYPE SIZE USED PRIO\n\/swap.img file   2G   0B   -2\n<\/code><\/pre>\n\n\n\n<p>If there is no output, then swap is not enabled. If it is enabled as shown in the output above, run the command below to disable it.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo swapoff -v \/swap.img<\/pre>\n\n\n\n<p>Or simply<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo swapoff -a<\/pre>\n\n\n\n<p>To permanently disable swap, comment out or remove the swap line on \/etc\/fstab file.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo sed -i '\/swap\/s\/^\/#\/' \/etc\/fstab<\/pre>\n\n\n\n<p>or Simply remove it;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sed \"-i.bak\" '\/swap\/d' \/etc\/fstab<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"enable-kernel-ip-forwarding-on-cluster-nodes\">Enable Kernel IP forwarding on Cluster Nodes<\/h3>\n\n\n\n<p>In order to permit the communication between Pods across different networks, the system should able to route traffic between them. This can be achieved by enabling IP forwarding. Without IP forwarding, containers won&#8217;t be able to communicate with resources outside of their network namespace, which would limit their functionality and utility.<\/p>\n\n\n\n<p>To enable IP forwarding, set the value of <strong><code>net.ipv4.ip_forward<\/code><\/strong> to <code>1<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"net.ipv4.ip_forward=1\" | sudo tee -a  \/etc\/sysctl.conf<\/code><\/pre>\n\n\n\n<p>Apply the changes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo sysctl -p<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"load-some-required-kernel-modules-on-cluster-nodes\">Load Some Required Kernel Modules on Cluster Nodes<\/h3>\n\n\n\n<p><code>overlay<\/code> module provides support for the overlay filesystem. OverlayFS is type of union filesystem used by container runtimes to layer the container&#8217;s root filesystem over the host filesystem.<\/p>\n\n\n\n<p><code>br_netfilter<\/code> module provides support for packet filtering in Linux bridge networks based on various criteria, such as source and destination IP address, port numbers, and protocol type.<\/p>\n\n\n\n<p>Check if these modules are enabled\/loaded;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo lsmod | grep -E \"overlay|br_netfilter\"<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>br_netfilter           32768  0\nbridge                307200  1 br_netfilter\noverlay               151552  9<\/code><\/pre>\n\n\n\n<p>If not loaded, just load them as follows;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo 'overlay\nbr_netfilter' | sudo tee \/etc\/modules-load.d\/kubernetes.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo modprobe overlay<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo modprobe br_netfilter<\/code><\/pre>\n\n\n\n<p>Similarly, enable Linux kernel&#8217;s bridge netfilter to pass bridge traffic to iptables for filtering. This means that the packets that are bridged between network interfaces can be filtered using iptables\/ip6tables, just as if they were routed packets.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo tee -a \/etc\/sysctl.conf &lt;&lt; 'EOL'\nnet.bridge.bridge-nf-call-iptables  = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nEOL<\/code><\/pre>\n\n\n\n<p>Apply the changes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo sysctl -p<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-container-runtime-on-ubuntu-24-04\">Install Container Runtime on Ubuntu 24.04<\/h3>\n\n\n\n<p>Kubernetes uses container runtime to run containers in Pods. It supports multiple container runtimes including Docker Engine,&nbsp;containerd,&nbsp;CRI-O, Mirantis Container Runtime.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-containerd-runtime-on-all-cluster-nodes\">Install Containerd Runtime on all Cluster Nodes<\/h4>\n\n\n\n<p>In this demo, we will use <a href=\"https:\/\/containerd.io\/docs\/\" target=\"_blank\" rel=\"noreferrer noopener\">containerd<\/a> runtime. Therefore, on all nodes, master and workers, you need to install containerd runtime.<\/p>\n\n\n\n<p>You can install containerd using official binaries or from the Docker Engine APT repos. We will use the later in this guide, thus;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>sudo apt install apt-transport-https \\\n\tca-certificates curl \\\n\tgnupg-agent \\\n\tsoftware-properties-common\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -fsSL https:\/\/download.docker.com\/linux\/ubuntu\/gpg | \\\nsudo gpg --dearmor -o \/etc\/apt\/trusted.gpg.d\/docker.gpg<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"deb &#91;arch=amd64] https:\/\/download.docker.com\/linux\/ubuntu $(lsb_release -sc) stable\" | sudo tee \/etc\/apt\/sources.list.d\/docker-ce.list<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt update<\/code><\/pre>\n\n\n\n<p>Install containerd;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo apt install -y containerd.io<\/pre>\n\n\n\n<p>The kubelet automatically detects the container runtime present on the node and uses it to run the containers.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-cgroup-driver-for-container-d\">Configure Cgroup Driver for ContainerD<\/h4>\n\n\n\n<p>Cgroup (control groups) is a Linux kernel feature that allows for the isolation, prioritization, and monitoring of system resources like CPU, memory, and disk I\/O for a group of processes. Kubernetes (kubelet and container runtime such as <em>containerd<\/em>) uses cgroup drivers to interface with control groups in order to manage and set limit for the resources allocated to the containers.<\/p>\n\n\n\n<p>Kubernetes support three types of Cgroup drivers;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><code>cgroupfs<\/code><\/strong> (control groups filesystem): This is the default cgroup driver used by Kubernetes kubelet to manage resources for containers.<\/li>\n\n\n\n<li><strong><code>systemd<\/code><\/strong>: This is the default initialization system and service manager in some Linux systems. it offers functions such as starting of daemons, keeping track of processes using Linux cgroups etc.<\/li>\n<\/ul>\n\n\n\n<p>For systems that use Systemd as their default Init system, it is recommended to use systemd cgroup driver for Kubernetes instead of cgroupfs.<\/p>\n\n\n\n<p>The default configuration file for containerd is <strong><code>\/etc\/containerd\/config.toml<\/code><\/strong>. When containerd is installed from Docker APT repos, this file is created with little configs. If installed from the official binaries, the containerd confguration file is not created.<\/p>\n\n\n\n<p>Either way, update the containerd configuration file by executing the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91; -d \/etc\/containerd ] || sudo mkdir \/etc\/containerd<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>containerd config default | sudo tee \/etc\/containerd\/config.toml<\/code><\/pre>\n\n\n\n<p>Sample configuration.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\ndisabled_plugins = []\nimports = []\noom_score = 0\nplugin_dir = \"\"\nrequired_plugins = []\nroot = \"\/var\/lib\/containerd\"\nstate = \"\/run\/containerd\"\ntemp = \"\"\nversion = 2\n\n[cgroup]\n  path = \"\"\n\n[debug]\n  address = \"\"\n  format = \"\"\n  gid = 0\n  level = \"\"\n  uid = 0\n\n[grpc]\n  address = \"\/run\/containerd\/containerd.sock\"\n  gid = 0\n  max_recv_message_size = 16777216\n  max_send_message_size = 16777216\n  tcp_address = \"\"\n  tcp_tls_ca = \"\"\n  tcp_tls_cert = \"\"\n  tcp_tls_key = \"\"\n  uid = 0\n\n[metrics]\n  address = \"\"\n  grpc_histogram = false\n\n[plugins]\n\n  [plugins.\"io.containerd.gc.v1.scheduler\"]\n    deletion_threshold = 0\n    mutation_threshold = 100\n    pause_threshold = 0.02\n    schedule_delay = \"0s\"\n    startup_delay = \"100ms\"\n\n  [plugins.\"io.containerd.grpc.v1.cri\"]\n    device_ownership_from_security_context = false\n    disable_apparmor = false\n    disable_cgroup = false\n    disable_hugetlb_controller = true\n    disable_proc_mount = false\n    disable_tcp_service = true\n    enable_selinux = false\n    enable_tls_streaming = false\n    enable_unprivileged_icmp = false\n    enable_unprivileged_ports = false\n    ignore_image_defined_volumes = false\n    max_concurrent_downloads = 3\n    max_container_log_line_size = 16384\n    netns_mounts_under_state_dir = false\n    restrict_oom_score_adj = false\n    sandbox_image = \"registry.k8s.io\/pause:3.6\"\n    selinux_category_range = 1024\n    stats_collect_period = 10\n    stream_idle_timeout = \"4h0m0s\"\n    stream_server_address = \"127.0.0.1\"\n    stream_server_port = \"0\"\n    systemd_cgroup = false\n    tolerate_missing_hugetlb_controller = true\n    unset_seccomp_profile = \"\"\n\n    [plugins.\"io.containerd.grpc.v1.cri\".cni]\n      bin_dir = \"\/opt\/cni\/bin\"\n      conf_dir = \"\/etc\/cni\/net.d\"\n      conf_template = \"\"\n      ip_pref = \"\"\n      max_conf_num = 1\n\n    [plugins.\"io.containerd.grpc.v1.cri\".containerd]\n      default_runtime_name = \"runc\"\n      disable_snapshot_annotations = true\n      discard_unpacked_layers = false\n      ignore_rdt_not_enabled_errors = false\n      no_pivot = false\n      snapshotter = \"overlayfs\"\n\n      [plugins.\"io.containerd.grpc.v1.cri\".containerd.default_runtime]\n        base_runtime_spec = \"\"\n        cni_conf_dir = \"\"\n        cni_max_conf_num = 0\n        container_annotations = []\n        pod_annotations = []\n        privileged_without_host_devices = false\n        runtime_engine = \"\"\n        runtime_path = \"\"\n        runtime_root = \"\"\n        runtime_type = \"\"\n\n        [plugins.\"io.containerd.grpc.v1.cri\".containerd.default_runtime.options]\n\n      [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes]\n\n        [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc]\n          base_runtime_spec = \"\"\n          cni_conf_dir = \"\"\n          cni_max_conf_num = 0\n          container_annotations = []\n          pod_annotations = []\n          privileged_without_host_devices = false\n          runtime_engine = \"\"\n          runtime_path = \"\"\n          runtime_root = \"\"\n          runtime_type = \"io.containerd.runc.v2\"\n\n          [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc.options]\n            BinaryName = \"\"\n            CriuImagePath = \"\"\n            CriuPath = \"\"\n            CriuWorkPath = \"\"\n            IoGid = 0\n            IoUid = 0\n            NoNewKeyring = false\n            NoPivotRoot = false\n            Root = \"\"\n            ShimCgroup = \"\"\n            SystemdCgroup = false\n\n      [plugins.\"io.containerd.grpc.v1.cri\".containerd.untrusted_workload_runtime]\n        base_runtime_spec = \"\"\n        cni_conf_dir = \"\"\n        cni_max_conf_num = 0\n        container_annotations = []\n        pod_annotations = []\n        privileged_without_host_devices = false\n        runtime_engine = \"\"\n        runtime_path = \"\"\n        runtime_root = \"\"\n        runtime_type = \"\"\n\n        [plugins.\"io.containerd.grpc.v1.cri\".containerd.untrusted_workload_runtime.options]\n\n    [plugins.\"io.containerd.grpc.v1.cri\".image_decryption]\n      key_model = \"node\"\n\n    [plugins.\"io.containerd.grpc.v1.cri\".registry]\n      config_path = \"\"\n\n      [plugins.\"io.containerd.grpc.v1.cri\".registry.auths]\n\n      [plugins.\"io.containerd.grpc.v1.cri\".registry.configs]\n\n      [plugins.\"io.containerd.grpc.v1.cri\".registry.headers]\n\n      [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors]\n\n    [plugins.\"io.containerd.grpc.v1.cri\".x509_key_pair_streaming]\n      tls_cert_file = \"\"\n      tls_key_file = \"\"\n\n  [plugins.\"io.containerd.internal.v1.opt\"]\n    path = \"\/opt\/containerd\"\n\n  [plugins.\"io.containerd.internal.v1.restart\"]\n    interval = \"10s\"\n\n  [plugins.\"io.containerd.internal.v1.tracing\"]\n    sampling_ratio = 1.0\n    service_name = \"containerd\"\n\n  [plugins.\"io.containerd.metadata.v1.bolt\"]\n    content_sharing_policy = \"shared\"\n\n  [plugins.\"io.containerd.monitor.v1.cgroups\"]\n    no_prometheus = false\n\n  [plugins.\"io.containerd.runtime.v1.linux\"]\n    no_shim = false\n    runtime = \"runc\"\n    runtime_root = \"\"\n    shim = \"containerd-shim\"\n    shim_debug = false\n\n  [plugins.\"io.containerd.runtime.v2.task\"]\n    platforms = [\"linux\/amd64\"]\n    sched_core = false\n\n  [plugins.\"io.containerd.service.v1.diff-service\"]\n    default = [\"walking\"]\n\n  [plugins.\"io.containerd.service.v1.tasks-service\"]\n    rdt_config_file = \"\"\n\n  [plugins.\"io.containerd.snapshotter.v1.aufs\"]\n    root_path = \"\"\n\n  [plugins.\"io.containerd.snapshotter.v1.btrfs\"]\n    root_path = \"\"\n\n  [plugins.\"io.containerd.snapshotter.v1.devmapper\"]\n    async_remove = false\n    base_image_size = \"\"\n    discard_blocks = false\n    fs_options = \"\"\n    fs_type = \"\"\n    pool_name = \"\"\n    root_path = \"\"\n\n  [plugins.\"io.containerd.snapshotter.v1.native\"]\n    root_path = \"\"\n\n  [plugins.\"io.containerd.snapshotter.v1.overlayfs\"]\n    root_path = \"\"\n    upperdir_label = false\n\n  [plugins.\"io.containerd.snapshotter.v1.zfs\"]\n    root_path = \"\"\n\n  [plugins.\"io.containerd.tracing.processor.v1.otlp\"]\n    endpoint = \"\"\n    insecure = false\n    protocol = \"\"\n\n[proxy_plugins]\n\n[stream_processors]\n\n  [stream_processors.\"io.containerd.ocicrypt.decoder.v1.tar\"]\n    accepts = [\"application\/vnd.oci.image.layer.v1.tar+encrypted\"]\n    args = [\"--decryption-keys-path\", \"\/etc\/containerd\/ocicrypt\/keys\"]\n    env = [\"OCICRYPT_KEYPROVIDER_CONFIG=\/etc\/containerd\/ocicrypt\/ocicrypt_keyprovider.conf\"]\n    path = \"ctd-decoder\"\n    returns = \"application\/vnd.oci.image.layer.v1.tar\"\n\n  [stream_processors.\"io.containerd.ocicrypt.decoder.v1.tar.gzip\"]\n    accepts = [\"application\/vnd.oci.image.layer.v1.tar+gzip+encrypted\"]\n    args = [\"--decryption-keys-path\", \"\/etc\/containerd\/ocicrypt\/keys\"]\n    env = [\"OCICRYPT_KEYPROVIDER_CONFIG=\/etc\/containerd\/ocicrypt\/ocicrypt_keyprovider.conf\"]\n    path = \"ctd-decoder\"\n    returns = \"application\/vnd.oci.image.layer.v1.tar+gzip\"\n\n[timeouts]\n  \"io.containerd.timeout.bolt.open\" = \"0s\"\n  \"io.containerd.timeout.shim.cleanup\" = \"5s\"\n  \"io.containerd.timeout.shim.load\" = \"5s\"\n  \"io.containerd.timeout.shim.shutdown\" = \"3s\"\n  \"io.containerd.timeout.task.state\" = \"2s\"\n\n[ttrpc]\n  address = \"\"\n  gid = 0\n  uid = 0\n<\/code><\/pre>\n\n\n\n<p>Once you generate the default config, you need to enable systemd cgroup for the containerd low-level container runtime, <strong><code>runc<\/code><\/strong> by changing the value of <strong><code>SystemdCgroup<\/code><\/strong> from <strong><code>false<\/code><\/strong> to <strong><code>true<\/code><\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo sed -i '\/SystemdCgroup\/s\/false\/true\/' \/etc\/containerd\/config.toml<\/code><\/pre>\n\n\n\n<p>Also, as of this writing, it is recommended to use &#8220;<strong>registry.k8s.io\/pause:3.9<\/strong>&#8221; as the CRI sandbox image. <strong>pause<\/strong> container image is a minimalistic container image that enables containerd to provide network isolation for pods in Kubernetes. Containerd uses <strong>pause:3.8<\/strong>. <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>grep sandbox_image \/etc\/containerd\/config.toml<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>    sandbox_image = \"registry.k8s.io\/pause:3.8\"<\/code><\/pre>\n\n\n\n<p>To change this to <strong>pause:3.9;<\/strong> <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo sed -i '\/pause:3.8\/s\/3.8\/3.9\/' \/etc\/containerd\/config.toml<\/code><\/pre>\n\n\n\n<p>If the default version is other than 3.8, then adjust the number accordingly.<\/p>\n\n\n\n<p>Verify the changes again;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>grep sandbox_image \/etc\/containerd\/config.toml<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>    sandbox_image = \"registry.k8s.io\/pause:3.9\"<\/code><\/pre>\n\n\n\n<p>Start and enable containerd to run on system boot;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo systemctl enable --now containerd<\/code><\/pre>\n\n\n\n<p>Confirm the status;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl status containerd<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\u25cf containerd.service - containerd container runtime\n     Loaded: loaded (\/usr\/lib\/systemd\/system\/containerd.service; enabled; preset: enabled)\n     Active: active (running) since Mon 2024-05-13 17:50:46 UTC; 3min 45s ago\n       Docs: https:\/\/containerd.io\n   Main PID: 2826 (containerd)\n      Tasks: 8\n     Memory: 13.5M (peak: 14.0M)\n        CPU: 1.303s\n     CGroup: \/system.slice\/containerd.service\n             \u2514\u25002826 \/usr\/bin\/containerd\n\nMay 13 17:50:46 master.kifarunix.com containerd[2826]: time=\"2024-05-13T17:50:46.057491119Z\" level=info msg=\"Start subscribing containerd event\"\nMay 13 17:50:46 master.kifarunix.com containerd[2826]: time=\"2024-05-13T17:50:46.057526439Z\" level=info msg=\"Start recovering state\"\nMay 13 17:50:46 master.kifarunix.com containerd[2826]: time=\"2024-05-13T17:50:46.057810777Z\" level=info msg=\"Start event monitor\"\nMay 13 17:50:46 master.kifarunix.com containerd[2826]: time=\"2024-05-13T17:50:46.057825824Z\" level=info msg=\"Start snapshots syncer\"\nMay 13 17:50:46 master.kifarunix.com containerd[2826]: time=\"2024-05-13T17:50:46.057831474Z\" level=info msg=\"Start cni network conf syncer for default\"\nMay 13 17:50:46 master.kifarunix.com containerd[2826]: time=\"2024-05-13T17:50:46.057833164Z\" level=info msg=serving... address=\/run\/containerd\/containerd.sock.ttrpc\nMay 13 17:50:46 master.kifarunix.com containerd[2826]: time=\"2024-05-13T17:50:46.057840998Z\" level=info msg=\"Start streaming server\"\nMay 13 17:50:46 master.kifarunix.com containerd[2826]: time=\"2024-05-13T17:50:46.057873084Z\" level=info msg=serving... address=\/run\/containerd\/containerd.sock\nMay 13 17:50:46 master.kifarunix.com systemd[1]: Started containerd.service - containerd container runtime.\nMay 13 17:50:46 master.kifarunix.com containerd[2826]: time=\"2024-05-13T17:50:46.059380011Z\" level=info msg=\"containerd successfully booted in 0.019204s\"\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-kubernetes-on-ubuntu-24-04\">Install Kubernetes on Ubuntu 24.04<\/h3>\n\n\n\n<p>There are a number of node components required to provide Kubernetes runtime environment that needs to be installed on each node. These include:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>kubelet<\/code>: runs as an agent on each worker node and ensures that containers are running in a Pod.<\/li>\n\n\n\n<li><code>kubeadm<\/code>: Bootstraps Kubernetes cluster<\/li>\n\n\n\n<li><code>kubectl<\/code>: Used to run commands against Kubernetes clusters.&nbsp;<\/li>\n<\/ul>\n\n\n\n<p>These components are not available on the default Ubuntu repos. Thus, you need to install Kubernetes repos to install them.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-kubernetes-repository-gpg-signing-key\">Install Kubernetes Repository GPG Signing Key<\/h4>\n\n\n\n<p>Run the command below to install Kubernetes repo GPG key.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt install gnupg2 -y<\/code><\/pre>\n\n\n\n<p>Replace the value of the VER variable below with the release number of Kubernetes you need to run! In this guide, I will be using the current latest minor release version, v1.30.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>VER=1.30<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -fsSL https:\/\/pkgs.k8s.io\/core:\/stable:\/v${VER}\/deb\/Release.key | \\\nsudo gpg --dearmor -o \/etc\/apt\/trusted.gpg.d\/k8s.gpg<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-kubernetes-repository-on-ubuntu-24-04\">Install Kubernetes Repository on Ubuntu 24.04<\/h4>\n\n\n\n<p>Next install the Kubernetes repository of the version matching the GPG key installed above;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">echo \"deb https:\/\/pkgs.k8s.io\/core:\/stable:\/v${VER}\/deb\/ \/\" | sudo tee \/etc\/apt\/sources.list.d\/kurbenetes.list<\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-kubernetes-components-on-all-the-nodes\">Install Kubernetes components on all the nodes<\/h4>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo apt update<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo apt install kubelet kubeadm kubectl -y<\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"mark-hold-kubernetes-packages\">Mark Hold Kubernetes Packages<\/h4>\n\n\n\n<p>In order to maintain the stability of the cluster, it is important to maintain specific versions of critical packages like <code>kubeadm<\/code>, <code>kubelet<\/code>, and <code>kubectl<\/code>. This can be done by instructing the package management system (APT) to prevent those packages from being upgraded using the <strong>apt-mark hold<\/strong> command.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt-mark hold kubeadm kubelet kubectl<\/code><\/pre>\n\n\n\n<p>To check whether packages are on hold or not, you can use <code>apt-mark showhold<\/code>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt-mark showhold<\/code><\/pre>\n\n\n\n<p>If you want to allow <code>apt<\/code> to upgrade these packages again, you can remove the hold using <code>apt-mark unhold<\/code>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt-mark unhold kubeadm kubelet kubectl<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"initialize-kubernetes-cluster-on-control-plane-using-kubeadm\">Initialize Kubernetes Cluster on Control Plane using Kubeadm<\/h3>\n\n\n\n<p>Once the above steps are completed, initialize the Kubernetes cluster on the master node. The Kubernetes master is responsible for maintaining the desired state for your cluster.<\/p>\n\n\n\n<p>We will be using <strong>kubeadm<\/strong> tool to deploy our K8S cluster.<\/p>\n\n\n\n<p>The cluster can be initiated using the <strong>kubeadm<\/strong> tool by passing the <strong><code>init<\/code><\/strong> command argument;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubeadm init &lt;args&gt;<\/code><\/pre>\n\n\n\n<p>Some of the common arguments\/options include;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>&#8211;apiserver-advertise-address<\/strong>: Defines the IP address the API Server will listen on. If not set the default network interface will be used. An example usage is <code>--apiserver-advertise-address=192.168.122.60<\/code>.<\/li>\n\n\n\n<li><strong>&#8211;pod-network-cidr<\/strong>: Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node. You use this to define your preferred network range if there is a chance for collision between your network plugin\u2019s preferred Pod network addon and some of your host networks to happen e.g<code>--pod-network-cidr=10.100.0.0\/16<\/code>.<\/li>\n\n\n\n<li><strong>&#8211;control-plane-endpoint<\/strong>: Specifies the hostname and port that the API server will listen on. This is recommended over the use of <code>--apiserver-advertise-address<\/code> because it enables you to define a shared endpoint such as load balance DNS name or an IP address that can be used when you upgrade single master node to highly available node. For example, <code>--control-plane-endpoint=cluster.kifarunix-demo.com:6443<\/code>.<\/li>\n<\/ul>\n\n\n\n<p>Since we are just running a single master node Kubernetes cluster in this guide (for demo purposes), with no plans to upgrade to highly available cluster, then we will specify just the IP address of the control plane while bootstrapping our cluster.<\/p>\n\n\n\n<p>Thus, run the command below on the master node to bootstrap the Kubernetes control-plane node.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo kubeadm init --apiserver-advertise-address=192.168.122.60 --pod-network-cidr=10.100.0.0\/16<\/pre>\n\n\n\n<p>The command will start by pre-pulling (<strong><code>kubeadm config images pull<\/code><\/strong>) the required container images for a Kubernetes cluster before initializing the cluster.<\/p>\n\n\n\n<p>Once the initialization is done, you should be able to see an output similar to the one below;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>[init] Using Kubernetes version: v1.30.0\n[preflight] Running pre-flight checks\n[preflight] Pulling images required for setting up a Kubernetes cluster\n[preflight] This might take a minute or two, depending on the speed of your internet connection\n[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'\n[certs] Using certificateDir folder \"\/etc\/kubernetes\/pki\"\n[certs] Generating \"ca\" certificate and key\n[certs] Generating \"apiserver\" certificate and key\n[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master.kifarunix.com] and IPs [10.96.0.1 192.168.122.60]\n[certs] Generating \"apiserver-kubelet-client\" certificate and key\n[certs] Generating \"front-proxy-ca\" certificate and key\n[certs] Generating \"front-proxy-client\" certificate and key\n[certs] Generating \"etcd\/ca\" certificate and key\n[certs] Generating \"etcd\/server\" certificate and key\n[certs] etcd\/server serving cert is signed for DNS names [localhost master.kifarunix.com] and IPs [192.168.122.60 127.0.0.1 ::1]\n[certs] Generating \"etcd\/peer\" certificate and key\n[certs] etcd\/peer serving cert is signed for DNS names [localhost master.kifarunix.com] and IPs [192.168.122.60 127.0.0.1 ::1]\n[certs] Generating \"etcd\/healthcheck-client\" certificate and key\n[certs] Generating \"apiserver-etcd-client\" certificate and key\n[certs] Generating \"sa\" key and public key\n[kubeconfig] Using kubeconfig folder \"\/etc\/kubernetes\"\n[kubeconfig] Writing \"admin.conf\" kubeconfig file\n[kubeconfig] Writing \"super-admin.conf\" kubeconfig file\n[kubeconfig] Writing \"kubelet.conf\" kubeconfig file\n[kubeconfig] Writing \"controller-manager.conf\" kubeconfig file\n[kubeconfig] Writing \"scheduler.conf\" kubeconfig file\n[etcd] Creating static Pod manifest for local etcd in \"\/etc\/kubernetes\/manifests\"\n[control-plane] Using manifest folder \"\/etc\/kubernetes\/manifests\"\n[control-plane] Creating static Pod manifest for \"kube-apiserver\"\n[control-plane] Creating static Pod manifest for \"kube-controller-manager\"\n[control-plane] Creating static Pod manifest for \"kube-scheduler\"\n[kubelet-start] Writing kubelet environment file with flags to file \"\/var\/lib\/kubelet\/kubeadm-flags.env\"\n[kubelet-start] Writing kubelet configuration to file \"\/var\/lib\/kubelet\/config.yaml\"\n[kubelet-start] Starting the kubelet\n[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory \"\/etc\/kubernetes\/manifests\"\n[kubelet-check] Waiting for a healthy kubelet. This can take up to 4m0s\n[kubelet-check] The kubelet is healthy after 501.313365ms\n[api-check] Waiting for a healthy API server. This can take up to 4m0s\n[api-check] The API server is healthy after 4.003829486s\n[upload-config] Storing the configuration used in ConfigMap \"kubeadm-config\" in the \"kube-system\" Namespace\n[kubelet] Creating a ConfigMap \"kubelet-config\" in namespace kube-system with the configuration for the kubelets in the cluster\n[upload-certs] Skipping phase. Please see --upload-certs\n[mark-control-plane] Marking the node master.kifarunix.com as control-plane by adding the labels: [node-role.kubernetes.io\/control-plane node.kubernetes.io\/exclude-from-external-load-balancers]\n[mark-control-plane] Marking the node master.kifarunix.com as control-plane by adding the taints [node-role.kubernetes.io\/control-plane:NoSchedule]\n[bootstrap-token] Using token: 2ntlip.lsw3yriy62bs16pp\n[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles\n[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes\n[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials\n[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token\n[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster\n[bootstrap-token] Creating the \"cluster-info\" ConfigMap in the \"kube-public\" namespace\n[kubelet-finalize] Updating \"\/etc\/kubernetes\/kubelet.conf\" to point to a rotatable kubelet client certificate and key\n[addons] Applied essential addon: CoreDNS\n[addons] Applied essential addon: kube-proxy\n<strong>\nYour Kubernetes control-plane has initialized successfully!\n\nTo start using your cluster, you need to run the following as a regular user:\n\n  mkdir -p $HOME\/.kube\n  sudo cp -i \/etc\/kubernetes\/admin.conf $HOME\/.kube\/config\n  sudo chown $(id -u):$(id -g) $HOME\/.kube\/config\n\nAlternatively, if you are the root user, you can run:\n\n  export KUBECONFIG=\/etc\/kubernetes\/admin.conf\n\nYou should now deploy a pod network to the cluster.\nRun \"kubectl apply -f [podnetwork].yaml\" with one of the options listed at:\n  https:\/\/kubernetes.io\/docs\/concepts\/cluster-administration\/addons\/\n\nThen you can join any number of worker nodes by running the following on each as root:\n\nkubeadm join 192.168.122.60:6443 --token 2ntlip.lsw3yriy62bs16pp \\\n\t--discovery-token-ca-cert-hash sha256:a22e5b78b50c54af7de5390ec804b311d28ea40048d9c6b66ee21660bbe4d212<\/strong>\n<\/code><\/pre>\n\n\n\n<p>As suggested on the output above, you need to run the commands provided on the <strong>master node<\/strong> to start using your cluster.<\/p>\n\n\n\n<p>Be sure to run the commands as regular user (<strong>recommended<\/strong>), with sudo rights.<\/p>\n\n\n\n<p>Thus, if you are root, then switch to regular user with sudo rights (kifarunix is our regular, it could be a different user for you)<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">su - kifarunix<\/pre>\n\n\n\n<p>Next, create a Kubernetes cluster directory.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">mkdir -p $HOME\/.kube<\/pre>\n\n\n\n<p>Copy Kubernetes admin configuration file to the cluster directory created above.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo cp -i \/etc\/kubernetes\/admin.conf $HOME\/.kube\/config<\/pre>\n\n\n\n<p>Set the proper ownership for the cluster configuration file.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo chown $(id -u):$(id -g) $HOME\/.kube\/config<\/pre>\n\n\n\n<p>Verify the status of the Kubernetes cluster;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">kubectl get nodes<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                   STATUS     ROLES           AGE     VERSION\nmaster.kifarunix.com   NotReady   control-plane   8m14s   v1.30.0\n<\/code><\/pre>\n\n\n\n<p>As you can see, the cluster is not ready yet.<\/p>\n\n\n\n<p>You can also get the address of the control plane and cluster services;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl cluster-info<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Kubernetes control plane is running at https:\/\/192.168.122.60:6443\nCoreDNS is running at https:\/\/192.168.122.60:6443\/api\/v1\/namespaces\/kube-system\/services\/kube-dns:dns\/proxy\n\nTo further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\n\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-pod-network-addon-on-master-node\">Install Pod Network Addon on Master Node<\/h3>\n\n\n\n<p>A Pod is a group of one or more related containers in a Kubernetes cluster. They share the same lifecycle, storage\/network. For Pods to communicate with one another, you must deploy a <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/extend-kubernetes\/compute-storage-net\/network-plugins\/#cni\" target=\"_blank\" rel=\"noreferrer noopener\">Container Network Interface<\/a>&nbsp;(CNI) based Pod network add-on.<\/p>\n\n\n\n<p>There are multiple Pod network addons that you can choose from. Refer to <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/cluster-administration\/addons\/\" target=\"_blank\" rel=\"noreferrer noopener\">Addons page<\/a> for more information.<\/p>\n\n\n\n<p>To deploy a CNI Pod network, run the command below on the master node;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">kubectl apply -f [podnetwork].yaml<\/pre>\n\n\n\n<p>Where <strong><code>[podnetwork].yaml<\/code><\/strong> is the path to your preferred CNI YAML file. In this demo, we will use <a href=\"https:\/\/docs.tigera.io\/calico\/latest\/getting-started\/kubernetes\/requirements\" target=\"_blank\" rel=\"noreferrer noopener\">Calico network plugin<\/a>.<\/p>\n\n\n\n<p>Install Calico Pod network addon Operator by running the command below. <strong>Execute the command as the user with which you created the Kubernetes cluster<\/strong>.<\/p>\n\n\n\n<p>Current release version is v3.28.0.<\/p>\n\n\n\n<p>Get the current release version from <a href=\"https:\/\/github.com\/projectcalico\/calico\/releases\" target=\"_blank\" rel=\"noreferrer noopener\">releases page<\/a> and replace the value of CNI_VER below.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>CNI_VER=3.28.0<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl create -f https:\/\/raw.githubusercontent.com\/projectcalico\/calico\/v${CNI_VER}\/manifests\/tigera-operator.yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>namespace\/tigera-operator created\ncustomresourcedefinition.apiextensions.k8s.io\/bgpconfigurations.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/bgpfilters.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/bgppeers.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/blockaffinities.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/caliconodestatuses.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/clusterinformations.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/felixconfigurations.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/globalnetworkpolicies.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/globalnetworksets.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/hostendpoints.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/ipamblocks.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/ipamconfigs.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/ipamhandles.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/ippools.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/ipreservations.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/kubecontrollersconfigurations.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/networkpolicies.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/networksets.crd.projectcalico.org created\ncustomresourcedefinition.apiextensions.k8s.io\/apiservers.operator.tigera.io created\ncustomresourcedefinition.apiextensions.k8s.io\/imagesets.operator.tigera.io created\ncustomresourcedefinition.apiextensions.k8s.io\/installations.operator.tigera.io created\ncustomresourcedefinition.apiextensions.k8s.io\/tigerastatuses.operator.tigera.io created\nserviceaccount\/tigera-operator created\nclusterrole.rbac.authorization.k8s.io\/tigera-operator created\nclusterrolebinding.rbac.authorization.k8s.io\/tigera-operator created\ndeployment.apps\/tigera-operator created\n<\/code><\/pre>\n\n\n\n<p>Next, download the custom resources necessary to configure Calico. The default network for Calico plugin is 192.168.0.0\/16. <strong>If you used custom&nbsp;<code>pod CIDR<\/code>&nbsp;as defined above (<em>&#8211;pod-network-cidr=10.100.0.0\/16<\/em>), download the custom resource file and modify the network to match your custom one.<\/strong><\/p>\n\n\n\n<p>We will the manifest of the same version of CNI above.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>wget https:\/\/raw.githubusercontent.com\/projectcalico\/calico\/v${CNI_VER}\/manifests\/custom-resources.yaml<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>cat custom-resources.yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code># This section includes base Calico installation configuration.\n# For more information, see: https:\/\/docs.tigera.io\/calico\/latest\/reference\/installation\/api#operator.tigera.io\/v1.Installation\napiVersion: operator.tigera.io\/v1\nkind: Installation\nmetadata:\n  name: default\nspec:\n  # Configures Calico networking.\n  calicoNetwork:\n    ipPools:\n    - name: default-ipv4-ippool\n      blockSize: 26\n      cidr: 192.168.0.0\/16\n      encapsulation: VXLANCrossSubnet\n      natOutgoing: Enabled\n      nodeSelector: all()\n\n---\n\n# This section configures the Calico API server.\n# For more information, see: https:\/\/docs.tigera.io\/calico\/latest\/reference\/installation\/api#operator.tigera.io\/v1.APIServer\napiVersion: operator.tigera.io\/v1\nkind: APIServer\nmetadata:\n  name: default\nspec: {}\n<\/code><\/pre>\n\n\n\n<p>The network section of the custom resource file  will now look like below by default;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>    - blockSize: 26\n      cidr: 192.168.0.0\/16<\/code><\/pre>\n\n\n\n<p>Update the network subnet to match your subnet.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sed -i 's\/192.168\/10.100\/' custom-resources.yaml<\/code><\/pre>\n\n\n\n<p>Apply the changes<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl create -f custom-resources.yaml<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>installation.operator.tigera.io\/default created\napiserver.operator.tigera.io\/default created<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"get-running-pods-in-the-kubernetes-cluster\">Get Running Pods in the Kubernetes cluster<\/h3>\n\n\n\n<p>Once the command completes, you can list the Pods in the namespaces by running the command below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">kubectl get pods --all-namespaces<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAMESPACE         NAME                                           READY   STATUS              RESTARTS   AGE\ncalico-system     calico-kube-controllers-795f68c999-xg8l6       0\/1     ContainerCreating   0          31s\ncalico-system     calico-node-phgxz                              0\/1     Running             0          24s\ncalico-system     calico-typha-5c68d5959d-kkxwc                  1\/1     Terminating         0          32s\ncalico-system     calico-typha-5ff5999dc9-vr2q9                  0\/1     Pending             0          31s\ncalico-system     csi-node-driver-fjxfd                          0\/2     ContainerCreating   0          32s\nkube-system       coredns-7db6d8ff4d-762fg                       0\/1     Running             0          24m\nkube-system       coredns-7db6d8ff4d-flnrq                       0\/1     Running             0          24m\nkube-system       etcd-master.kifarunix.com                      1\/1     Running             2          24m\nkube-system       kube-apiserver-master.kifarunix.com            1\/1     Running             2          24m\nkube-system       kube-controller-manager-master.kifarunix.com   1\/1     Running             2          24m\nkube-system       kube-proxy-jv9x9                               1\/1     Running             0          24m\nkube-system       kube-scheduler-master.kifarunix.com            1\/1     Running             2          24m\ntigera-operator   tigera-operator-7d5cd7fcc8-7bw5j               1\/1     Running             0          12m\n<\/code><\/pre>\n\n\n\n<p>You can list Pods on specific namespaces;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pods -n calico-system<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAMESPACE         NAME                                           READY   STATUS              RESTARTS   AGE\ncalico-system     calico-kube-controllers-795f68c999-xg8l6       0\/1     ContainerCreating   0          31s\ncalico-system     calico-node-phgxz                              0\/1     Running             0          24s\ncalico-system     calico-typha-5c68d5959d-kkxwc                  1\/1     Terminating         0          32s\ncalico-system     calico-typha-5ff5999dc9-vr2q9                  0\/1     Pending             0          31s\ncalico-system     csi-node-driver-fjxfd                          0\/2     ContainerCreating   0          32s\n<\/code><\/pre>\n\n\n\n<p>As can be seen, all Pods on calico-system namespace are <code>running<\/code>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"open-kubernetes-cluster-ports-on-firewall\">Open Kubernetes Cluster Ports on Firewall<\/h3>\n\n\n\n<p>If firewall is running on the nodes, then there are some ports that needs to be opened on the firewall;<\/p>\n\n\n\n<p>Control Plane ports;<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Protocol<\/th><th>Direction<\/th><th>Port Range<\/th><th>Purpose<\/th><th>Used By<\/th><\/tr><\/thead><tbody><tr><td>TCP<\/td><td>Inbound<\/td><td>6443<\/td><td>Kubernetes API server<\/td><td>All<\/td><\/tr><tr><td>TCP<\/td><td>Inbound<\/td><td>2379-2380<\/td><td>etcd server client API<\/td><td>kube-apiserver, etcd<\/td><\/tr><tr><td>TCP<\/td><td>Inbound<\/td><td>10250<\/td><td>Kubelet API<\/td><td>Self, Control plane<\/td><\/tr><tr><td>TCP<\/td><td>Inbound<\/td><td>10259<\/td><td>kube-scheduler<\/td><td>Self<\/td><\/tr><tr><td>TCP<\/td><td>Inbound<\/td><td>10257<\/td><td>kube-controller-manager<\/td><td>Self<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>So the ports that should be open and accessible from outside the node are:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>6443<\/code> &#8211; Kubernetes API Server (secure port)<\/li>\n\n\n\n<li><code>2379-2380<\/code> &#8211; etcd server client API<\/li>\n\n\n\n<li><code>10250<\/code> &#8211; Kubelet API<\/li>\n\n\n\n<li><code>10251<\/code> &#8211; kube-scheduler<\/li>\n\n\n\n<li><code>10252<\/code> &#8211; kube-controller-manager<\/li>\n<\/ul>\n\n\n\n<p>In my setup, I am using UFW. Hence, you only need to open the ports below on Master\/Control Plane;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>for i in 6443 2379:2380 10250:10252; do sudo ufw allow from any to any port $i proto tcp; done<\/code><\/pre>\n\n\n\n<p>You can restrict access to the API from specific networks\/IPS.<\/p>\n\n\n\n<p>Worker Nodes;<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Protocol<\/th><th>Direction<\/th><th>Port Range<\/th><th>Purpose<\/th><th>Used By<\/th><\/tr><\/thead><tbody><tr><td>TCP<\/td><td>Inbound<\/td><td>10250<\/td><td>Kubelet API<\/td><td>Self, Control plane<\/td><\/tr><tr><td>TCP<\/td><td>Inbound<\/td><td>30000-32767<\/td><td>NodePort Services<\/td><td>All<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>On each Woker node, open the Kubelete API port;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ufw allow from any to any port 10250 proto tcp comment \"Open Kubelet API port\"<\/code><\/pre>\n\n\n\n<p>You can restrict access to the API from specific networks\/IPs.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"add-worker-nodes-to-kubernetes-cluster\">Add Worker Nodes to Kubernetes Cluster<\/h3>\n\n\n\n<p>You can now add Worker nodes to the Kubernetes cluster using the <strong>kubeadm join<\/strong> command as follows.<\/p>\n\n\n\n<p>Before that, ensure that container runtime is installed, configured and running. We are using <strong>containerd<\/strong> CRI;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl status containerd<\/code><\/pre>\n\n\n\n<p>Sample output from worker01 node;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\u25cf containerd.service - containerd container runtime\n     Loaded: loaded (\/usr\/lib\/systemd\/system\/containerd.service; enabled; preset: enabled)\n     Active: active (running) since Mon 2024-05-13 20:38:59 UTC; 20h ago\n       Docs: https:\/\/containerd.io\n    Process: 4430 ExecStartPre=\/sbin\/modprobe overlay (code=exited, status=0\/SUCCESS)\n   Main PID: 4432 (containerd)\n      Tasks: 8\n     Memory: 13.4M (peak: 13.9M)\n        CPU: 1min 45.227s\n     CGroup: \/system.slice\/containerd.service\n             \u2514\u25004432 \/usr\/bin\/containerd\n\nMay 13 20:38:59 wk01.kifarunix.com containerd[4432]: time=\"2024-05-13T20:38:59.786586300Z\" level=info msg=\"Start subscribing containerd event\"\nMay 13 20:38:59 wk01.kifarunix.com containerd[4432]: time=\"2024-05-13T20:38:59.786651551Z\" level=info msg=\"Start recovering state\"\nMay 13 20:38:59 wk01.kifarunix.com containerd[4432]: time=\"2024-05-13T20:38:59.786806494Z\" level=info msg=serving... address=\/run\/containerd\/containerd.sock.ttrpc\nMay 13 20:38:59 wk01.kifarunix.com containerd[4432]: time=\"2024-05-13T20:38:59.786874950Z\" level=info msg=\"Start event monitor\"\nMay 13 20:38:59 wk01.kifarunix.com containerd[4432]: time=\"2024-05-13T20:38:59.786931542Z\" level=info msg=\"Start snapshots syncer\"\nMay 13 20:38:59 wk01.kifarunix.com containerd[4432]: time=\"2024-05-13T20:38:59.786886363Z\" level=info msg=serving... address=\/run\/containerd\/containerd.sock\nMay 13 20:38:59 wk01.kifarunix.com containerd[4432]: time=\"2024-05-13T20:38:59.786971950Z\" level=info msg=\"Start cni network conf syncer for default\"\nMay 13 20:38:59 wk01.kifarunix.com containerd[4432]: time=\"2024-05-13T20:38:59.787003012Z\" level=info msg=\"Start streaming server\"\nMay 13 20:38:59 wk01.kifarunix.com containerd[4432]: time=\"2024-05-13T20:38:59.787040271Z\" level=info msg=\"containerd successfully booted in 0.021146s\"\nMay 13 20:38:59 wk01.kifarunix.com systemd[1]: Started containerd.service - containerd container runtime.\n<\/code><\/pre>\n\n\n\n<p>Once you have confirmed that, get the cluster join command that was output during cluster boot strapping and execute on each node.<\/p>\n\n\n\n<p>Note that this command is displayed after initializing the control plane above and it should be executed as root user.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>sudo kubeadm join 192.168.122.60:6443 \\\n\t--token 2ntlip.lsw3yriy62bs16pp \\\n\t--discovery-token-ca-cert-hash sha256:a22e5b78b50c54af7de5390ec804b311d28ea40048d9c6b66ee21660bbe4d212\n<\/code><\/pre>\n\n\n\n<p>If you didn&#8217;t save the Kubernetes Cluster joining command, you can at any given time print using the command below on the Master or control plane;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">kubeadm token create --print-join-command<\/pre>\n\n\n\n<p>Once the command runs, you will get an output similar to below;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>[preflight] Running pre-flight checks\n[preflight] Reading configuration from the cluster...\n[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'\n[kubelet-start] Writing kubelet configuration to file \"\/var\/lib\/kubelet\/config.yaml\"\n[kubelet-start] Writing kubelet environment file with flags to file \"\/var\/lib\/kubelet\/kubeadm-flags.env\"\n[kubelet-start] Starting the kubelet\n[kubelet-check] Waiting for a healthy kubelet. This can take up to 4m0s\n[kubelet-check] The kubelet is healthy after 501.461586ms\n[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap\n\nThis node has joined the cluster:\n* Certificate signing request was sent to apiserver and a response was received.\n* The Kubelet was informed of the new secure connection details.\n\nRun 'kubectl get nodes' on the control-plane to see this node join the cluster.\n<\/code><\/pre>\n\n\n\n<p>On the Kubernetes control plane (master, <em>as the regular user with which you created the cluster as<\/em>), run the command below to verify that the nodes have joined the cluster.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">kubectl get nodes<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                   STATUS     ROLES           AGE    VERSION\nmaster.kifarunix.com   Ready      control-plane   20h    v1.30.0\nwk01.kifarunix.com     Ready      <none>          107s   v1.30.0\nwk02.kifarunix.com     NotReady   <none>          11s    v1.30.0\nwk03.kifarunix.com     NotReady   <none>          6s     v1.30.0\n<\/code><\/pre>\n\n\n\n<p>There are different node stati;<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>NotReady<\/strong>: The node has been added to the cluster but is not yet ready to accept workloads.<\/li>\n\n\n\n<li><strong>SchedulingDisabled<\/strong>: The node is not able to receive new workloads because it is marked as unschedulable.<\/li>\n\n\n\n<li><strong>Ready<\/strong>: The node is ready to accept workloads.<\/li>\n\n\n\n<li><strong>OutOfDisk<\/strong>: Indicates that the node is running out of disk space.<\/li>\n\n\n\n<li><strong>MemoryPressure<\/strong>: Indicates that the node is running out of memory.<\/li>\n\n\n\n<li><strong>PIDPressure<\/strong>: indicates that there are too many processes on the node<\/li>\n\n\n\n<li><strong>DiskPressure<\/strong>: Indicates that the node is running out of disk space.<\/li>\n\n\n\n<li><strong>NetworkUnavailable<\/strong>: Indicates that the node is not reachable via the network.<\/li>\n\n\n\n<li><strong>Unschedulable<\/strong>: Indicates that the node is not schedulable for new workloads.<\/li>\n\n\n\n<li><strong>ConditionUnknown<\/strong>: Indicates that the node status is unknown due to an error.<\/li>\n<\/ol>\n\n\n\n<p>Role of the Worker nodes may show up as <strong><code>&lt;none&gt;<\/code><\/strong>.  This is okay. No role is assigned to the node by default. It is only until the control plane assign a workload on the node then it shows up the correct role.<\/p>\n\n\n\n<p>You can however update this ROLE using the command;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl label node &lt;worker-node-name&gt; node-role.kubernetes.io\/worker=true<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"get-kubernetes-cluster-information\">Get Kubernetes Cluster Information<\/h3>\n\n\n\n<p>As you can see, we now have a cluster. Run the command below to get cluster information.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">kubectl cluster-info<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Kubernetes control plane is running at https:\/\/192.168.122.60:6443\nCoreDNS is running at https:\/\/192.168.122.60:6443\/api\/v1\/namespaces\/kube-system\/services\/kube-dns:dns\/proxy\n\nTo further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"list-kubernetes-cluster-api-resources\">List Kubernetes Cluster API Resources<\/h3>\n\n\n\n<p>You can list all Kubernetes cluster resources using the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl api-resources<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>NAME                                SHORTNAMES   APIVERSION                        NAMESPACED   KIND\nbindings                                         v1                                true         Binding\ncomponentstatuses                   cs           v1                                false        ComponentStatus\nconfigmaps                          cm           v1                                true         ConfigMap\nendpoints                           ep           v1                                true         Endpoints\nevents                              ev           v1                                true         Event\nlimitranges                         limits       v1                                true         LimitRange\nnamespaces                          ns           v1                                false        Namespace\nnodes                               no           v1                                false        Node\npersistentvolumeclaims              pvc          v1                                true         PersistentVolumeClaim\npersistentvolumes                   pv           v1                                false        PersistentVolume\npods                                po           v1                                true         Pod\npodtemplates                                     v1                                true         PodTemplate\nreplicationcontrollers              rc           v1                                true         ReplicationController\nresourcequotas                      quota        v1                                true         ResourceQuota\nsecrets                                          v1                                true         Secret\nserviceaccounts                     sa           v1                                true         ServiceAccount\nservices                            svc          v1                                true         Service\nmutatingwebhookconfigurations                    admissionregistration.k8s.io\/v1   false        MutatingWebhookConfiguration\nvalidatingadmissionpolicies                      admissionregistration.k8s.io\/v1   false        ValidatingAdmissionPolicy\nvalidatingadmissionpolicybindings                admissionregistration.k8s.io\/v1   false        ValidatingAdmissionPolicyBinding\nvalidatingwebhookconfigurations                  admissionregistration.k8s.io\/v1   false        ValidatingWebhookConfiguration\ncustomresourcedefinitions           crd,crds     apiextensions.k8s.io\/v1           false        CustomResourceDefinition\napiservices                                      apiregistration.k8s.io\/v1         false        APIService\ncontrollerrevisions                              apps\/v1                           true         ControllerRevision\ndaemonsets                          ds           apps\/v1                           true         DaemonSet\ndeployments                         deploy       apps\/v1                           true         Deployment\nreplicasets                         rs           apps\/v1                           true         ReplicaSet\nstatefulsets                        sts          apps\/v1                           true         StatefulSet\nselfsubjectreviews                               authentication.k8s.io\/v1          false        SelfSubjectReview\ntokenreviews                                     authentication.k8s.io\/v1          false        TokenReview\nlocalsubjectaccessreviews                        authorization.k8s.io\/v1           true         LocalSubjectAccessReview\nselfsubjectaccessreviews                         authorization.k8s.io\/v1           false        SelfSubjectAccessReview\nselfsubjectrulesreviews                          authorization.k8s.io\/v1           false        SelfSubjectRulesReview\nsubjectaccessreviews                             authorization.k8s.io\/v1           false        SubjectAccessReview\nhorizontalpodautoscalers            hpa          autoscaling\/v2                    true         HorizontalPodAutoscaler\ncronjobs                            cj           batch\/v1                          true         CronJob\njobs                                             batch\/v1                          true         Job\ncertificatesigningrequests          csr          certificates.k8s.io\/v1            false        CertificateSigningRequest\nleases                                           coordination.k8s.io\/v1            true         Lease\nbgpconfigurations                                crd.projectcalico.org\/v1          false        BGPConfiguration\nbgpfilters                                       crd.projectcalico.org\/v1          false        BGPFilter\nbgppeers                                         crd.projectcalico.org\/v1          false        BGPPeer\nblockaffinities                                  crd.projectcalico.org\/v1          false        BlockAffinity\ncaliconodestatuses                               crd.projectcalico.org\/v1          false        CalicoNodeStatus\nclusterinformations                              crd.projectcalico.org\/v1          false        ClusterInformation\nfelixconfigurations                              crd.projectcalico.org\/v1          false        FelixConfiguration\nglobalnetworkpolicies                            crd.projectcalico.org\/v1          false        GlobalNetworkPolicy\nglobalnetworksets                                crd.projectcalico.org\/v1          false        GlobalNetworkSet\nhostendpoints                                    crd.projectcalico.org\/v1          false        HostEndpoint\nipamblocks                                       crd.projectcalico.org\/v1          false        IPAMBlock\nipamconfigs                                      crd.projectcalico.org\/v1          false        IPAMConfig\nipamhandles                                      crd.projectcalico.org\/v1          false        IPAMHandle\nippools                                          crd.projectcalico.org\/v1          false        IPPool\nipreservations                                   crd.projectcalico.org\/v1          false        IPReservation\nkubecontrollersconfigurations                    crd.projectcalico.org\/v1          false        KubeControllersConfiguration\nnetworkpolicies                                  crd.projectcalico.org\/v1          true         NetworkPolicy\nnetworksets                                      crd.projectcalico.org\/v1          true         NetworkSet\nendpointslices                                   discovery.k8s.io\/v1               true         EndpointSlice\nevents                              ev           events.k8s.io\/v1                  true         Event\nflowschemas                                      flowcontrol.apiserver.k8s.io\/v1   false        FlowSchema\nprioritylevelconfigurations                      flowcontrol.apiserver.k8s.io\/v1   false        PriorityLevelConfiguration\ningressclasses                                   networking.k8s.io\/v1              false        IngressClass\ningresses                           ing          networking.k8s.io\/v1              true         Ingress\nnetworkpolicies                     netpol       networking.k8s.io\/v1              true         NetworkPolicy\nruntimeclasses                                   node.k8s.io\/v1                    false        RuntimeClass\napiservers                                       operator.tigera.io\/v1             false        APIServer\nimagesets                                        operator.tigera.io\/v1             false        ImageSet\ninstallations                                    operator.tigera.io\/v1             false        Installation\ntigerastatuses                                   operator.tigera.io\/v1             false        TigeraStatus\npoddisruptionbudgets                pdb          policy\/v1                         true         PodDisruptionBudget\nclusterrolebindings                              rbac.authorization.k8s.io\/v1      false        ClusterRoleBinding\nclusterroles                                     rbac.authorization.k8s.io\/v1      false        ClusterRole\nrolebindings                                     rbac.authorization.k8s.io\/v1      true         RoleBinding\nroles                                            rbac.authorization.k8s.io\/v1      true         Role\npriorityclasses                     pc           scheduling.k8s.io\/v1              false        PriorityClass\ncsidrivers                                       storage.k8s.io\/v1                 false        CSIDriver\ncsinodes                                         storage.k8s.io\/v1                 false        CSINode\ncsistoragecapacities                             storage.k8s.io\/v1                 true         CSIStorageCapacity\nstorageclasses                      sc           storage.k8s.io\/v1                 false        StorageClass\nvolumeattachments                                storage.k8s.io\/v1                 false        VolumeAttachment\n<\/code><\/pre>\n\n\n\n<p>You are now ready to deploy an application on Kubernetes cluster.<\/p>\n\n\n\n<p>Want to dive deeper in getting Kubernetes up and running? Check out this book, Kubernetes: Up and Running: Dive into the Future of Infrastructure&nbsp;3rd Edition by Brendan Burns.<\/p>\n\n\n\n<figure class=\"wp-block-embed aligncenter is-type-rich is-provider-amazon wp-block-embed-amazon\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Kubernetes: Up and Running: Dive into the Future of Infrastructure\" type=\"text\/html\" width=\"1200\" height=\"550\" frameborder=\"0\" allowfullscreen style=\"max-width:100%\" src=\"https:\/\/read.amazon.com\/kp\/card?preview=inline&#038;linkCode=ll1&#038;ref_=k4w_oembed_RczSwfjUoLjwgx&#038;asin=109811020X&#038;tag=dc42a8f60962-20\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"remove-worker-nodes-from-cluster\">Remove Worker Nodes from Cluster<\/h3>\n\n\n\n<p>You can gracefully remove a node from Kubernetes cluster as described in the guide below;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/remove-worker-node-from-kubernetes-cluster\/\" target=\"_blank\" rel=\"noreferrer noopener\">Gracefully Remove Worker Node from Kubernetes Cluster<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"app-armor-blocks-runc-signals-pods-stuck-terminating\">AppArmor Blocks runc Signals, Pods Stuck Terminating<\/h3>\n\n\n\n<p>You might have realized that in the recent version of Ubuntu, there is an issue whereby draining the nodes or deleting the pods get stuck with such errors in apparmor logs as;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>2024-06-14T19:04:43.331091+00:00 worker-01 kernel: audit: type=1400 audit(1718391883.329:221): apparmor=\"<strong>DENIED<\/strong>\" operation=\"signal\" class=\"signal\" profile=\"cri-containerd.apparmor.d\" pid=7445 comm=\"runc\" requested_mask=\"receive\" denied_mask=\"receive\" signal=kill peer=\"runc<\/code><\/pre>\n\n\n\n<p>This is a bug on AppArmor profile that denies signals from runc. This results in many pods being stuck in a terminating state. The bug was reported by Sebastian Podjasek on 2024-05-10. It affects Ubuntu containerd-app package.<\/p>\n\n\n\n<p>Read how to fix on <a href=\"https:\/\/kifarunix.com\/kubernetes-nodes-maintenance-drain-vs-cordon-demystified\/#kubectl-drain-node-gets-stuck-forever-apparmor-bug\" target=\"_blank\" rel=\"noreferrer noopener\">kubectl drain node gets stuck forever [Apparmor Bug]<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-kubernetes-dashboard\">Install Kubernetes Dashboard<\/h3>\n\n\n\n<p>You can manage your cluster from the dashboard using Kubernetes. See the guide below;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-kubernetes-dashboard-on-ubuntu\/\" target=\"_blank\" rel=\"noreferrer noopener\">How to Install Kubernetes Dashboard<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"further-reading\">Further Reading<\/h3>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kubernetes.io\/docs\/setup\/\" target=\"_blank\">Getting Started with Kubernetes<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This tutorial provides a step by step guide on how to install and setup Kubernetes Cluster on Ubuntu 24.04. Kubernetes, according to kubernetes.io, is an<\/p>\n","protected":false},"author":10,"featured_media":16327,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[1076,121,1668],"tags":[7492,7490,7491],"class_list":["post-22539","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-containers","category-howtos","category-kubernetes","tag-kubeadm","tag-kubernetes-cluster-ubuntu-24-04","tag-ubuntu-24-04-k8s-cluster","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/22539"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=22539"}],"version-history":[{"count":14,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/22539\/revisions"}],"predecessor-version":[{"id":23087,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/22539\/revisions\/23087"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/16327"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=22539"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=22539"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=22539"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}