{"id":16415,"date":"2023-05-06T22:01:24","date_gmt":"2023-05-06T19:01:24","guid":{"rendered":"https:\/\/kifarunix.com\/?p=16415"},"modified":"2024-03-10T09:17:23","modified_gmt":"2024-03-10T06:17:23","slug":"remove-worker-node-from-kubernetes-cluster","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/remove-worker-node-from-kubernetes-cluster\/","title":{"rendered":"Gracefully Remove Worker Node from Kubernetes Cluster"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1178\" height=\"676\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/remove-worker-node-from-cluster.png\" alt=\"Gracefully Remove Worker Node from Kubernetes Cluster\" class=\"wp-image-16430\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/remove-worker-node-from-cluster.png?v=1683399327 1178w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/remove-worker-node-from-cluster-768x441.png?v=1683399327 768w\" sizes=\"(max-width: 1178px) 100vw, 1178px\" \/><\/figure>\n\n\n\n<p>In this tutorial, you will learn how to gracefully remove worker node from Kubernetes cluster. If you are managing a Kubernetes cluster, there might arise a need to remove a worker node from it. The process of removing a worker node from a Kubernetes cluster may slightly vary with the tool or platform that is used to deploy Kubernetes cluster. For example, the process of removing a worker node from a kubeadm Kubernetes cluster might be different from removing a worker node from a Kubernetes deployed using GKE, Rancher, OpenShift, Kubermatic, or kops.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#removing-worker-node-from-kubernetes-cluster-gracefully\">Removing Worker Node from Kubernetes Cluster Gracefully<\/a><ul><li><a href=\"#prepare-the-node-for-removal\">Prepare the Node  for Removal<\/a><\/li><li><a href=\"#drain-the-worker-node\">Drain the Worker Node<\/a><\/li><li><a href=\"#delete-a-worker-node\">Delete a Worker Node<\/a><\/li><li><a href=\"#confirm-removal-of-worker-node\">Confirm Removal of Worker Node<\/a><\/li><li><a href=\"#optionally-reset-the-node\">Optionally Reset the Node<\/a><\/li><\/ul><\/li><li><a href=\"#conclusion\">Conclusion<\/a><\/li><li><a href=\"#other-tutorials\">Other Tutorials<\/a><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"removing-worker-node-from-kubernetes-cluster-gracefully\">Removing Worker Node from Kubernetes Cluster Gracefully<\/h2>\n\n\n\n<p>In this guide, we will learn how to safely or gracefully remove a worker node from a Kubernetes cluster created with kubeadm.<\/p>\n\n\n\n<p>So, what are the steps to removing worker node from Kubernetes cluster gracefully?<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"prepare-the-node-for-removal\">Prepare the Node  for Removal<\/h3>\n\n\n\n<p>In a production environment, you can&#8217;t just wake up and decide to remove a node that is running production workloads. You need to prepare for the removal in order to minimize the risk of downtime or performance issues in the Kubernetes.<\/p>\n\n\n\n<p>Preparing the node for removal from Kubernetes cluster involves various tasks such as;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ensure that there is sufficient resources in the cluster to support the cluster workloads<\/li>\n\n\n\n<li>Ensure that services hosted by specific node being removed are distributed across Kubernetes cluster nodes.<\/li>\n\n\n\n<li>Backup any data and configurations on the node being removed from the cluster.<\/li>\n\n\n\n<li>Ensure that removal of the node doesn&#8217;t affect any networking settings on the cluster.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"drain-the-worker-node\">Drain the Worker Node<\/h3>\n\n\n\n<p>Before you can remove a worker node from Kubernetes cluster, you need to migrate all the Pods scheduled on that node to other nodes in the cluster. This is what is called draining a node in Kubernetes and is one of the Kubernetes workloads voluntary <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/workloads\/pods\/disruptions\/\" target=\"_blank\" rel=\"noreferrer noopener\">disruption<\/a> action. This will ensure that the workloads runs uninterrupted during the decommissioning of the node.<\/p>\n\n\n\n<p><strong><code>kubectl drain<\/code><\/strong> is the command that can be used to migrate the nodes running on the node marked for decommissioning into other cluster nodes.<\/p>\n\n\n\n<p>The command needs to be executed on the Kubernetes control plane\/master node.<\/p>\n\n\n\n<p>The syntax of the kubectl drain command is;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl drain NODE &#91;options]<\/code><\/pre>\n\n\n\n<p>List the nodes on the cluster;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get nodes<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\nNAME                        STATUS   ROLES           AGE     VERSION\nmaster.kifarunix-demo.com   Ready    control-plane   6d16h   v1.27.1\nwk01.kifarunix-demo.com     Ready    worker          6d16h   v1.27.1\nwk02.kifarunix-demo.com     Ready    &lt;none>          6d15h   v1.27.1\nwk03.kifarunix-demo.com     Ready    &lt;none>          6d15h   v1.27.1\n<\/code><\/pre>\n\n\n\n<p>You can then run the command below to drain the worker node marked for decommissioning.<\/p>\n\n\n\n<p>For example, let&#8217;s drain wk01 in example output above;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl drain wk01.kifarunix-demo.com<\/code><\/pre>\n\n\n\n<p>The &#8216;drain&#8217; command waits for graceful termination. You should not operate on the machine until the command completes.<\/p>\n\n\n\n<p>The drain command will also cordone the node. This means that the node is marked as <strong><code>unschedulable<\/code><\/strong> and prevents the Kubernetes scheduler from placing any new pods onto the node.<\/p>\n\n\n\n<p>If the node has daemon set-managed pods, the drain command wont drain it. Similarly, if there are any pods that are neither mirror pods nor managed by a replication controller, replica set, daemon set, stateful set, or job, then drain will not delete any pods unless you use <strong><code>--force<\/code><\/strong>. The <code><strong>--force<\/strong><\/code> will also allow deletion to proceed if the managing resource of one or more pods is missing.<\/p>\n\n\n\n<p>Sample output in my drain command above;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\nnode\/wk01.kifarunix-demo.com cordoned\nerror: unable to drain node \"wk01.kifarunix-demo.com\" due to error:cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): calico-system\/calico-node-g4vlv, calico-system\/csi-node-driver-9b9xx, kube-system\/kube-proxy-sn66b, continuing command...\nThere are pending nodes to be drained:\n wk01.kifarunix-demo.com\ncannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): calico-system\/calico-node-g4vlv, calico-system\/csi-node-driver-9b9xx, kube-system\/kube-proxy-sn66b\n<\/code><\/pre>\n\n\n\n<p>To forcefully drain the node, i just add the <strong><code>--force<\/code><\/strong>, <strong><code>--ignore-daemonsets<\/code><\/strong> options to the command.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl drain wk01.kifarunix-demo.com --force --ignore-daemonsets<\/code><\/pre>\n\n\n\n<p>Use these flags with caution as it can lead to disruption of running pods<\/p>\n\n\n\n<p>As already mentioned, let the command terminates gracefully.<\/p>\n\n\n\n<p>Note that this process may take sometime to complete.<\/p>\n\n\n\n<p>Check more options using the <code>--help<\/code> option.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl drain --help<\/code><\/pre>\n\n\n\n<p>Once the command runs, you can check nodes status to ensure that it is now not read to accept any workloads.<\/p>\n\n\n\n<p>There are also a number of constraints that may affect the drain process of a node;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><code>Pod Disruption Budgets (PDBs)<\/code><\/strong>: This is a Kubernetes feature that sets a minimum number of replicas a given pod must have during voluntary disruption actions such during node maintenance, updates, etc. Draining a node a cannot violate the PDBs, as this could result in a service disruption. And thus, your drain command may just hang and does nothing. You can check for any PDBs on all namespaces;<br><code>kubectl get poddisruptionbudget --all-namespaces<\/code>.<\/li>\n\n\n\n<li><strong><code>Resource Constraints<\/code><\/strong>: The cluster must have enough resources to accommodate the Pods being evicted from the node being drained.<\/li>\n\n\n\n<li><strong><code>Node Affinity\/Anti-affinity<\/code><\/strong>. These are some of the Kubernetes Pod scheduler decision making concepts. Affinity defines conditions under which a Pod is scheduled on a node while anti-affinity define conditions under which a Pod should not be scheduled on a node. Drain process respects whatever conditions defined by the scheduler<\/li>\n\n\n\n<li><strong><code>Daemonsets<\/code><\/strong>: In Kubernetes, daemonsets ensure that a pod is running on every node in the cluster. Before a node is successfully drained, the pods running on it must be rescheduled elsewhere. Drain constraints must account for daemonsets to avoid service disruption.<\/li>\n\n\n\n<li><strong><code>Taints\/Tolerations<\/code><\/strong>: Taints is the opposite of Node affinity. They are defined to prevent certain pods from being scheduled on specific nodes. Tolerations on the other hand are applied to Pods to allow them to be scheduled on nodes with matching taints. The drain command will only evict Pods from tainted nodes if they have tolerations that match.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"delete-a-worker-node\">Delete a Worker Node<\/h3>\n\n\n\n<p>Once the node has been successfully evicted, you can now remove it from the cluster.<\/p>\n\n\n\n<p>You can remove an evicted node from the cluster using the <strong><code>kubectl delete<\/code><\/strong> command.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl delete node wk01.kifarunix-demo.com<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"confirm-removal-of-worker-node\">Confirm Removal of Worker Node<\/h3>\n\n\n\n<p>You can verify that the node has been removed using the <strong><code>kubectl get nodes<\/code><\/strong> command.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get nodes<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\nNAME                        STATUS   ROLES           AGE     VERSION\nmaster.kifarunix-demo.com   Ready    control-plane   6d21h   v1.27.1\nwk02.kifarunix-demo.com     Ready    &lt;none>          6d21h   v1.27.1\nwk03.kifarunix-demo.com     Ready    &lt;none>          6d21h   v1.27.1\n<\/code><\/pre>\n\n\n\n<p>wk01 is gone!<\/p>\n\n\n\n<p>You can also check Pods;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pod --all-namespaces<\/code><\/pre>\n\n\n\n<p>To list Pods running on a specific node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pods --field-selector spec.nodeName=&lt;node-name&gt;<\/code><\/pre>\n\n\n\n<p>E.g;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pods --field-selector spec.nodeName=wk01.kifarunix-demo.com<\/code><\/pre>\n\n\n\n<p>You should have now successfully and gracefully removed a node from the kubeadm Kubernetes cluster.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"optionally-reset-the-node\">Optionally Reset the Node<\/h3>\n\n\n\n<p>You can optionally remove all the configurations related to K8s on the node.<\/p>\n\n\n\n<p>Thus, login to the node that was drained and run the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubeadm reset<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"conclusion\">Conclusion<\/h2>\n\n\n\n<p>In summary, the steps you need to take gracefully remove worker node from cluster;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>ensure cluster is ready to support workload after removal of a specific node<\/li>\n\n\n\n<li>drain the node to from the cluster (kubectl drain &lt;node-name&gt;)<\/li>\n\n\n\n<li>fix any constraints that might prevent the removal of a node<\/li>\n\n\n\n<li>delete the node from the cluster<\/li>\n<\/ul>\n\n\n\n<p>That marks the end of our guide on removing worker node from Kubernetes cluster gracefully.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"other-tutorials\">Other Tutorials<\/h2>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/what-are-the-core-concepts-in-kubernetes\/\" target=\"_blank\" rel=\"noreferrer noopener\">What are the core concepts in Kubernetes?<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/kubernetes-architecture-a-high-level-overview-of-kubernetes-cluster-components\/\" target=\"_blank\" rel=\"noreferrer noopener\">Kubernetes Architecture: A High-level Overview of Kubernetes Cluster Components<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this tutorial, you will learn how to gracefully remove worker node from Kubernetes cluster. If you are managing a Kubernetes cluster, there might arise<\/p>\n","protected":false},"author":10,"featured_media":16430,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[1076,121,1668],"tags":[6609,6608,6607,6610,6612,6611],"class_list":["post-16415","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-containers","category-howtos","category-kubernetes","tag-delete-worker-node-from-cluster","tag-drain-worker-node-from-kubernetes-cluster","tag-gracefully-remove-worker-node-from-kubernetes-cluster","tag-kubectl","tag-kubectl-delete-node","tag-kubectl-drain-node","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/16415"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=16415"}],"version-history":[{"count":22,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/16415\/revisions"}],"predecessor-version":[{"id":20752,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/16415\/revisions\/20752"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/16430"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=16415"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=16415"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=16415"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}