K8s Cluster Setup by Kubeadm

Posted on Edited on In

How to be a contribution beginner on k8s? https://youtu.be/o68ff5NokR8

  1. find issues, search label good first issue
  2. communication
  3. build relationships

Kubernetes version 1.13.2 This article mainly talks about setting up k8s cluster by kubeadm manually. As far as I know there are no coming changes that will significantly impact the validity of these steps.

Cluster Info

I have a 3 nodes bare-metal cluster called myk8s with CentOS version 7.5, the /etc/hosts file in each node:

1
2
3
172.16.158.44    myk8s1.fyre.ibm.com myk8s1
172.16.171.110   myk8s2.fyre.ibm.com myk8s2
172.16.171.227   myk8s3.fyre.ibm.com myk8s3

Let’ see the network interface on master node:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# ifconfig -a

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.158.44  netmask 255.255.0.0  broadcast 172.16.255.255
        ether 00:16:3e:01:9e:2c  txqueuelen 1000  (Ethernet)
        RX packets 1617615790  bytes 203050237209 (189.1 GiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 436  bytes 50037 (48.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 9.30.97.218  netmask 255.255.254.0  broadcast 9.30.97.255
        ether 00:20:09:1e:61:da  txqueuelen 1000  (Ethernet)
        RX packets 13350021  bytes 1424223654 (1.3 GiB)
        RX errors 0  dropped 5  overruns 0  frame 0
        TX packets 246436  bytes 45433438 (43.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
...

Configure

For every node in cluster, following instruction below

Install utilities

1
2
3
yum update -y
yum install -y vim
yum install -y git

Disable firewall

Check firewall status and disable it if active

1
systemctl status firewalld
1
2
systemctl disable firewalld
systemctl stop firewalld

Install kubeadm kubectl and kubelet

Install kubeadm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF

# Set SELinux in permissive mode (effectively disabling it)
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

systemctl enable --now kubelet

Setting SELinux in permissive mode by running setenforce 0 and sed ... effectively disables it. This is required to allow containers to access the host filesystem, which is needed by pod networks for example. You have to do this until SELinux support is improved in the kubelet.

Currently installed:

1
2
Installed:
  kubeadm.x86_64 0:1.13.3-0               kubectl.x86_64 0:1.13.3-0               kubelet.x86_64 0:1.13.3-0

check /etc/sysctl.conf file, for example:

1
2
3
4
5
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv4.ip_forward = 0
...

ensure that these 3 options exist and set to 1, because some users on RHEL/CentOS 7 have reported issues with traffic being routed incorrectly due to iptables being bypassed

1
2
3
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

if these 3 items not set, edit net.ipv4.ip_forward = 1 and append net.bridge.bridge-nf-call-ip6tables = 1 and net.bridge.bridge-nf-call-iptables = 1 in sysctl.conf file

then make sure that the net.bridge.bridge-nf-call is enabled, check if br_netfilter module is loaded. This can be done by running

1
lsmod | grep br_netfilter

if not, to load it explicitly call

1
modprobe br_netfilter

next run this command to reload setting

1
sysctl --system

then you can check the final setting:

1
sysctl -a | grep -E "net.bridge|net.ipv4.ip_forward"

Install docker

CRI installation in Kubernetes

Uninstall old versions

Older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies.

1
2
3
4
5
6
7
8
yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

Official Docker installation guides

Install Docker CE

currently Docker version 18.06.2 is recommended, but 1.11, 1.12, 1.13 and 17.03 are known to work as well. Keep track of the latest verified Docker version in the Kubernetes release notes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# Set up the repository
yum install yum-utils device-mapper-persistent-data lvm2

# Add docker repository.
yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

# Install docker ce.
yum update && yum install docker-ce-18.06.2.ce

# Create /etc/docker directory.
mkdir /etc/docker

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker

check result

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@myk8s1 ~] docker version
Client:
 Version:           18.06.2-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        6d37f41
 Built:             Sun Feb 10 03:46:03 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.2-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       6d37f41
  Built:            Sun Feb 10 03:48:29 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Disable swap

why we need to disable swap? Swap brings disk IO overhead, as well as breaking cgroups for pod memory control.

in /etc/fstab file, comment out swap setting

1
/dev/mapper/centos-swap swap                    swap    defaults        0 0

activate new configuration and check

1
swapoff -a
1
2
3
4
5
[root@myk8s3 ~] free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        189M        5.7G        136M        1.8G        7.0G
Swap:            0B          0B          0B

for worker nodes in cluster, stop here. Continue steps in master node:

Initialize kubernetes cluster

I will use Calico as the container network solution, in master node, run

1
kubeadm init --pod-network-cidr=192.168.0.0/16

you can specify the version by using --kubernetes-version v1.13.3, otherwise it will pull latest version from Internet.

you can see the output like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
...
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"

...
Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 9.30.97.218:6443 --token jjkiw2.n478eree0wrr3bmc --discovery-token-ca-cert-hash sha256:79659fb0b3fb0044f382ab5a5e317d4f775e821a61d0df4a401a4cbd8d8c5a7f


keep the last command for joining worker node later

1
2
kubeadm join 9.30.97.218:6443 --token jjkiw2.n478eree0wrr3bmc --discovery-token-ca-cert-hash sha256:79659fb0b3fb0044f382ab5a5e317d4f775e821a61d0df4a401a4cbd8d8c5a7f

then run following command in master node:

1
2
3
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

now if you run kubectl version, you will get something like below:

1
2
3
[root@myk8s1 ~] kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

let’s check what kind of docker images pulled from network to create the cluster in master

1
2
3
4
5
6
7
8
9
[root@myk8s1 ~] docker images
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-apiserver            v1.13.3             fe242e556a99        2 weeks ago         181MB
k8s.gcr.io/kube-controller-manager   v1.13.3             0482f6400933        2 weeks ago         146MB
k8s.gcr.io/kube-proxy                v1.13.3             98db19758ad4        2 weeks ago         80.3MB
k8s.gcr.io/kube-scheduler            v1.13.3             3a6f709e97a0        2 weeks ago         79.6MB
k8s.gcr.io/coredns                   1.2.6               f59dcacceff4        3 months ago        40MB
k8s.gcr.io/etcd                      3.2.24              3cab8e1b9802        4 months ago        220MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        14 months ago       742kB

Launch cluster network

1
2
3
4
5
6
7
8
9
# kubectl get pods --all-namespaces
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   coredns-86c58d9df4-5dfh9                      0/1     Pending   0          9m30s
kube-system   coredns-86c58d9df4-d9bfm                      0/1     Pending   0          9m30s
kube-system   etcd-myk8s1.fyre.ibm.com                      1/1     Running   0          8m52s
kube-system   kube-apiserver-myk8s1.fyre.ibm.com            1/1     Running   0          8m37s
kube-system   kube-controller-manager-myk8s1.fyre.ibm.com   1/1     Running   0          8m34s
kube-system   kube-proxy-wxjx8                              1/1     Running   0          9m31s
kube-system   kube-scheduler-myk8s1.fyre.ibm.com            1/1     Running   0          8m46s

you can find some pods are not ready, for example coredns-86c58d9df4-5dfh9 and coredns-86c58d9df4-d9bfm, also the master node

1
2
3
[root@myk8s1 ~] kubectl get nodes
NAME                  STATUS     ROLES    AGE   VERSION
myk8s1.fyre.ibm.com   NotReady   master   11m   v1.13.3

it’s time to set up network, you should first figure out which Calico version you need, check kubernetes release note, we see currently it support Calico version 3.3.1:

you can also refer this link to install, it’s the same setting as below: 

1
2
3
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

after applying rbac-kdd.yaml and calico.yaml, now you can see

1
2
3
4
5
6
7
8
9
10
# kubectl get pods --all-namespaces
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   calico-node-4vm2c                             2/2     Running   0          45s
kube-system   coredns-86c58d9df4-5dfh9                      1/1     Running   0          37m
kube-system   coredns-86c58d9df4-d9bfm                      1/1     Running   0          37m
kube-system   etcd-myk8s1.fyre.ibm.com                      1/1     Running   0          36m
kube-system   kube-apiserver-myk8s1.fyre.ibm.com            1/1     Running   0          36m
kube-system   kube-controller-manager-myk8s1.fyre.ibm.com   1/1     Running   0          36m
kube-system   kube-proxy-wxjx8                              1/1     Running   0          37m
kube-system   kube-scheduler-myk8s1.fyre.ibm.com            1/1     Running   0          36m
1
2
3
# kubectl get nodes
NAME                  STATUS   ROLES    AGE   VERSION
myk8s1.fyre.ibm.com   Ready    master   38m   v1.13.3

Note that I encountered the problem that when join the worker nodes, the calico-node becomes not ready

1
2
3
4
5
6
# kubectl get pods --all-namespaces
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   calico-node-4vm2c                             1/2     Running   0          11m
kube-system   calico-node-zsbjj                             1/2     Running   0          96s
kube-system   coredns-86c58d9df4-5dfh9                      1/1     Running   0          48m
...

The reason is my master node has multiple eth, I need to specify which one to use in order to be consistent among all nodes.

1
wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

delete the previous Calico deployment and then edit and apply yaml file again:

1
2
3
4
5
6
[root@myk8s1 ~] kubectl get pods --all-namespaces
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   calico-node-dpcsp                             2/2     Running   0          6m15s
kube-system   calico-node-gc5hs                             2/2     Running   0          6m15s
kube-system   coredns-86c58d9df4-5dfh9                      1/1     Running   0          81m
...

Join worker nodes

Join worker nodes is pretty easy, run this command on all worker nodes:

1
kubeadm join 9.30.97.218:6443 --token jjkiw2.n478eree0wrr3bmc --discovery-token-ca-cert-hash sha256:79659fb0b3fb0044f382ab5a5e317d4f775e821a61d0df4a401a4cbd8d8c5a7f

Check node status

1
2
3
4
5
# kubectl get nodes
NAME                  STATUS   ROLES    AGE   VERSION
myk8s1.fyre.ibm.com   Ready    master   83m   v1.13.3
myk8s2.fyre.ibm.com   Ready    <none>   36m   v1.13.3
myk8s3.fyre.ibm.com   Ready    <none>   33s   v1.13.3

By default, tokens expire after 24 hours. If you are joining a node to the cluster after the current token has expired, you can create a new token by running the following command on the master node

1
kubeadm token create

If you don’t have the value of --discovery-token-ca-cert-hash, you can get it by running the following command chain on the master node:

1
2
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
   openssl dgst -sha256 -hex | sed 's/^.* //'

Now a fresh kubernetes cluster with 1 master and 2 worker nodes is created.