How to be a contribution beginner on k8s?
https://youtu.be/o68ff5NokR8
find issues, search label good first issue
communication
build relationships
Kubernetes version 1.13.2
This article mainly talks about setting up k8s cluster by kubeadm
manually. As far as I know there are no coming changes that will significantly impact the validity of these steps.
Cluster Info
I have a 3 nodes bare-metal cluster called myk8s
with CentOS version 7.5, the /etc/hosts
file in each node:
1 2 3 172.16.158.44 myk8s1.fyre.ibm.com myk8s1 172.16.171.110 myk8s2.fyre.ibm.com myk8s2 172.16.171.227 myk8s3.fyre.ibm.com myk8s3
Let’ see the network interface on master node:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.16.158.44 netmask 255.255.0.0 broadcast 172.16.255.255 ether 00:16:3e:01:9e:2c txqueuelen 1000 (Ethernet) RX packets 1617615790 bytes 203050237209 (189.1 GiB) RX errors 0 dropped 1 overruns 0 frame 0 TX packets 436 bytes 50037 (48.8 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 9.30.97.218 netmask 255.255.254.0 broadcast 9.30.97.255 ether 00:20:09:1e:61:da txqueuelen 1000 (Ethernet) RX packets 13350021 bytes 1424223654 (1.3 GiB) RX errors 0 dropped 5 overruns 0 frame 0 TX packets 246436 bytes 45433438 (43.3 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ...
For every node in cluster, following instruction below
Install utilities
1 2 3 yum update -y yum install -y vim yum install -y git
Disable firewall
Check firewall status and disable it if active
1 systemctl status firewalld
1 2 systemctl disable firewalld systemctl stop firewalld
Install kubeadm kubectl and kubelet
Install kubeadm
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg exclude=kube* EOF setenforce 0 sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes systemctl enable --now kubelet
Setting SELinux in permissive mode by running setenforce 0
and sed ...
effectively disables it. This is required to allow containers to access the host filesystem, which is needed by pod networks for example. You have to do this until SELinux support is improved in the kubelet.
Currently installed:
1 2 Installed: kubeadm.x86_64 0:1.13.3-0 kubectl.x86_64 0:1.13.3-0 kubelet.x86_64 0:1.13.3-0
check /etc/sysctl.conf
file, for example:
1 2 3 4 5 net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 net.ipv4.ip_forward = 0 ...
ensure that these 3 options exist and set to 1, because some users on RHEL/CentOS 7 have reported issues with traffic being routed incorrectly due to iptables being bypassed
1 2 3 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1
if these 3 items not set, edit net.ipv4.ip_forward = 1
and append net.bridge.bridge-nf-call-ip6tables = 1
and net.bridge.bridge-nf-call-iptables = 1
in sysctl.conf
file
then make sure that the net.bridge.bridge-nf-call
is enabled, check if br_netfilter
module is loaded. This can be done by running
1 lsmod | grep br_netfilter
if not, to load it explicitly call
next run this command to reload setting
then you can check the final setting:
1 sysctl -a | grep -E "net.bridge|net.ipv4.ip_forward"
Install docker
CRI installation in Kubernetes
Uninstall old versions
Older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies.
1 2 3 4 5 6 7 8 yum remove docker \ docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-engine
Official Docker installation guides
Install Docker CE
currently Docker version 18.06.2
is recommended, but 1.11, 1.12, 1.13 and 17.03 are known to work as well. Keep track of the latest verified Docker version in the Kubernetes release notes
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 yum install yum-utils device-mapper-persistent-data lvm2 yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo yum update && yum install docker-ce-18.06.2.ce mkdir /etc/dockercat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": [ "overlay2.override_kernel_check=true" ] } EOF mkdir -p /etc/systemd/system/docker.service.dsystemctl daemon-reload systemctl restart docker
check result
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 [root@myk8s1 ~] docker version Client: Version: 18.06.2-ce API version: 1.38 Go version: go1.10.3 Git commit: 6d37f41 Built: Sun Feb 10 03:46:03 2019 OS/Arch: linux/amd64 Experimental: false Server: Engine: Version: 18.06.2-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: 6d37f41 Built: Sun Feb 10 03:48:29 2019 OS/Arch: linux/amd64 Experimental: false
Disable swap
why we need to disable swap?
Swap brings disk IO overhead, as well as breaking cgroups for pod memory control.
in /etc/fstab
file, comment out swap setting
1 /dev/mapper/centos-swap swap swap defaults 0 0
activate new configuration and check
1 2 3 4 5 [root@myk8s3 ~] free -h total used free shared buff/cache available Mem: 7.6G 189M 5.7G 136M 1.8G 7.0G Swap: 0B 0B 0B
for worker
nodes in cluster, stop here. Continue steps in master
node:
Initialize kubernetes cluster
I will use Calico
as the container network solution, in master node, run
1 kubeadm init --pod-network-cidr=192.168.0.0/16
you can specify the version by using --kubernetes-version v1.13.3
, otherwise it will pull latest version from Internet.
you can see the output like this
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ... [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" ... Your Kubernetes master has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME /.kube sudo cp -i /etc/kubernetes/admin.conf $HOME /.kube/config sudo chown $(id -u):$(id -g) $HOME /.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of machines by running the following on each node as root: kubeadm join 9.30.97.218:6443 --token jjkiw2.n478eree0wrr3bmc --discovery-token-ca-cert-hash sha256:79659fb0b3fb0044f382ab5a5e317d4f775e821a61d0df4a401a4cbd8d8c5a7f
keep the last command for joining worker node later
1 2 kubeadm join 9.30.97.218:6443 --token jjkiw2.n478eree0wrr3bmc --discovery-token-ca-cert-hash sha256:79659fb0b3fb0044f382ab5a5e317d4f775e821a61d0df4a401a4cbd8d8c5a7f
then run following command in master node:
1 2 3 mkdir -p $HOME /.kubecp -i /etc/kubernetes/admin.conf $HOME /.kube/configchown $(id -u):$(id -g) $HOME /.kube/config
now if you run kubectl version
, you will get something like below:
1 2 3 [root@myk8s1 ~] kubectl version Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
let’s check what kind of docker images pulled from network to create the cluster in master
1 2 3 4 5 6 7 8 9 [root@myk8s1 ~] docker images REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-apiserver v1.13.3 fe242e556a99 2 weeks ago 181MB k8s.gcr.io/kube-controller-manager v1.13.3 0482f6400933 2 weeks ago 146MB k8s.gcr.io/kube-proxy v1.13.3 98db19758ad4 2 weeks ago 80.3MB k8s.gcr.io/kube-scheduler v1.13.3 3a6f709e97a0 2 weeks ago 79.6MB k8s.gcr.io/coredns 1.2.6 f59dcacceff4 3 months ago 40MB k8s.gcr.io/etcd 3.2.24 3cab8e1b9802 4 months ago 220MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 14 months ago 742kB
Launch cluster network
1 2 3 4 5 6 7 8 9 NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-86c58d9df4-5dfh9 0/1 Pending 0 9m30s kube-system coredns-86c58d9df4-d9bfm 0/1 Pending 0 9m30s kube-system etcd-myk8s1.fyre.ibm.com 1/1 Running 0 8m52s kube-system kube-apiserver-myk8s1.fyre.ibm.com 1/1 Running 0 8m37s kube-system kube-controller-manager-myk8s1.fyre.ibm.com 1/1 Running 0 8m34s kube-system kube-proxy-wxjx8 1/1 Running 0 9m31s kube-system kube-scheduler-myk8s1.fyre.ibm.com 1/1 Running 0 8m46s
you can find some pods are not ready, for example coredns-86c58d9df4-5dfh9
and coredns-86c58d9df4-d9bfm
, also the master node
1 2 3 [root@myk8s1 ~] kubectl get nodes NAME STATUS ROLES AGE VERSION myk8s1.fyre.ibm.com NotReady master 11m v1.13.3
it’s time to set up network, you should first figure out which Calico
version you need, check kubernetes release note , we see currently it support Calico
version 3.3.1:
you can also refer this link to install, it’s the same setting as below:
1 2 3 kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
after applying rbac-kdd.yaml
and calico.yaml
, now you can see
1 2 3 4 5 6 7 8 9 10 NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-node-4vm2c 2/2 Running 0 45s kube-system coredns-86c58d9df4-5dfh9 1/1 Running 0 37m kube-system coredns-86c58d9df4-d9bfm 1/1 Running 0 37m kube-system etcd-myk8s1.fyre.ibm.com 1/1 Running 0 36m kube-system kube-apiserver-myk8s1.fyre.ibm.com 1/1 Running 0 36m kube-system kube-controller-manager-myk8s1.fyre.ibm.com 1/1 Running 0 36m kube-system kube-proxy-wxjx8 1/1 Running 0 37m kube-system kube-scheduler-myk8s1.fyre.ibm.com 1/1 Running 0 36m
1 2 3 NAME STATUS ROLES AGE VERSION myk8s1.fyre.ibm.com Ready master 38m v1.13.3
Note that I encountered the problem that when join the worker nodes, the calico-node
becomes not ready
1 2 3 4 5 6 NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-node-4vm2c 1/2 Running 0 11m kube-system calico-node-zsbjj 1/2 Running 0 96s kube-system coredns-86c58d9df4-5dfh9 1/1 Running 0 48m ...
The reason is my master node has multiple eth
, I need to specify which one to use in order to be consistent among all nodes.
1 wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
delete the previous Calico
deployment and then edit and apply yaml file again:
1 2 3 4 5 6 [root@myk8s1 ~] kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-node-dpcsp 2/2 Running 0 6m15s kube-system calico-node-gc5hs 2/2 Running 0 6m15s kube-system coredns-86c58d9df4-5dfh9 1/1 Running 0 81m ...
Join worker nodes
Join worker nodes is pretty easy, run this command on all worker nodes:
1 kubeadm join 9.30.97.218:6443 --token jjkiw2.n478eree0wrr3bmc --discovery-token-ca-cert-hash sha256:79659fb0b3fb0044f382ab5a5e317d4f775e821a61d0df4a401a4cbd8d8c5a7f
Check node status
1 2 3 4 5 NAME STATUS ROLES AGE VERSION myk8s1.fyre.ibm.com Ready master 83m v1.13.3 myk8s2.fyre.ibm.com Ready <none> 36m v1.13.3 myk8s3.fyre.ibm.com Ready <none> 33s v1.13.3
By default, tokens expire after 24 hours. If you are joining a node to the cluster after the current token has expired, you can create a new token by running the following command on the master node
If you don’t have the value of --discovery-token-ca-cert-hash
, you can get it by running the following command chain on the master node:
1 2 openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ openssl dgst -sha256 -hex | sed 's/^.* //'
Now a fresh kubernetes cluster with 1 master and 2 worker nodes is created.