Today I spend some time to investigate how to remove nodes from the k8s cluster that built by kubeadm.
For example, I have a 3 nodes cluster called k8stest, I deploy the application in namespacetest-1, each worker node (k8stest2 and k8stest3) holds some pods:
You can use kubectl drain to safely evict all of your pods from a node before you perform maintenance on the node (e.g. kernel upgrade, hardware maintenance, etc.). Safe evictions allow the pod’s containers to gracefully terminate and will respect the PodDisruptionBudgets you have specified.
The drain evicts or deletes all pods except mirror pods (which cannot be deleted through the API server). If there are DaemonSet-managed pods, drain will not proceed without --ignore-daemonsets, and regardless it will not delete any DaemonSet-managed pods, because those pods would be immediately replaced by the DaemonSet controller, which ignores unschedulable markings. If there are any pods that are neither mirror pods nor managed by ReplicationController, ReplicaSet, DaemonSet, StatefulSet or Job, then drain will not delete any pods unless you use --force. --force will also allow deletion to proceed if the managing resource of one or more pods is missing.
When kubectl drain returns successfully, that indicates that all of the pods (except the ones excluded as described in the previous paragraph) have been safely evicted (respecting the desired graceful termination period, and without violating any application-level disruption SLOs). It is then safe to bring down the node by powering down its physical machine or, if running on a cloud platform, deleting its virtual machine.
Let’s ssh to k8stest2 node and see what happens here, the payloads were gone:
1 2 3 4 5 6 7 8 9 10
ssh k8stest2.fyre.ibm.com docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0fbbb64d93d0 fa6f35a1c14d "/install-cni.sh" 6 hours ago Up 6 hours k8s_install-cni_calico-node-txjpn_kube-system_4b916269-3d49-11e9-b6b3-00163e01eecc_0 b78013d4f454 427a0694c75c "start_runit" 6 hours ago Up 6 hours k8s_calico-node_calico-node-txjpn_kube-system_4b916269-3d49-11e9-b6b3-00163e01eecc_0 c6aaf7cbf713 01cfa56edcfc "/usr/local/bin/kube..." 6 hours ago Up 6 hours k8s_kube-proxy_kube-proxy-52njn_kube-system_4b944a11-3d49-11e9-b6b3-00163e01eecc_0 542bc4662ee4 k8s.gcr.io/pause:3.1 "/pause" 6 hours ago Up 6 hours k8s_POD_calico-node-txjpn_kube-system_4b916269-3d49-11e9-b6b3-00163e01eecc_0 86ee508f0aa1 k8s.gcr.io/pause:3.1 "/pause" 6 hours ago Up 6 hours k8s_POD_kube-proxy-52njn_kube-system_4b944a11-3d49-11e9-b6b3-00163e01eecc_0
The given node will be marked unschedulable to prevent new pods from arriving.
1 2 3 4 5 6
kubectl get nodes
NAME STATUS ROLES AGE VERSION k8stest1.fyre.ibm.com Ready master 6h11m v1.13.2 k8stest2.fyre.ibm.com Ready,SchedulingDisabled <none> 5h57m v1.13.2 k8stest3.fyre.ibm.com Ready <none> 5h57m v1.13.2
Because the dedicated node k8stest2 was drained, so is-servicesdocker and is-xmetadocker keep pending: