This blog used to walk through some ssh(secure shell), scp and sftp use cases.

Aside: difference between OpenSSH vs OpenSSL The file format is different but they both encode the same kind of keys. Moreover, they are both generated with the same code.

Notice that restart sshd daemon will not disconnect current ssh connection, even if you stop the sshd daemon for a short time or restart the network daemon(don’t stop it!), the current ssh session is still working, see this issue: The reason is, sshd fork a child process on each connection, the child process will not die if either sshd or the whole network is restarted. sshd listens on port 22 for incoming connections. When someone connects it spawns a new process for that connection and goes back to listening.

1
2
3
4
5
6
7
8
9

# check pid and ppid of current ssh child process
ps -ef | grep -v grep | grep ssh

# restart some daemon
systemctl restart sshd
systemctl restart network

# the old child ssh session ppid was changed to 1
ps -ef | grep -v grep | grep ssh

Install SSH (SCP SFTP)

Notice that ssh, scp and sftp are all installed from openssh-clients package. 操作的目标机器的用户以及密码就是目标机器上在/etc/passwd中对应的用户及其密码。

First, understand that there are openssh client and openssh server: Installing and Enabling OpenSSH on CentOS 7, this article briefly introduces openssh configuration and firewall setting for it.

1

yum –y install openssh-server openssh-clients

If only install openssh-clients, you can ssh to others but others cannot ssh to you, since you don’t have ssh server listening at port 22.

After installing openssh-server, enable and start the sshd daemon

1
2
3
4

systemctl enable sshd
systemctl start sshd
# check status
systemctl status sshd

The system OpenSSH server configuration file is /etc/ssh/sshd_config, the custom configuration file is ~/.ssh/config. The /etc/ssh/ssh_config is for system-wide client behavior.

Restricted configuration you may need on server side:

1
2
3
4
5

Port 22
PermitRootLogin prohibit-password
PubkeyAuthentication yes
# after copy the public key in
PasswordAuthentication no

After making changes, restart sshd daemon.

Firewall setting for ssh is file /etc/sysconfig/iptables.

SSHFS

This is a remote mount implemented by SSH, handy if NFS is not workable, search my blog <<Linux Storage System>>.

SSH Tunnel

Forward a local port to remote port, one on one mapping. 用在比如database or web servers 没有对外开放端口，我们可以通过SSH穿过防火墙(SSH的端口是开 放的)去远程映射它们的端口到本地，然后通过localhost访问。

1
2
3
4
5
6
7

# -L: port forward
# 10003: local port
# 8000: remote port of a web server, for example a python simple http server.
# -N: Do not execute a remote command. This is useful for just forwarding ports.
# If the 8000 port is blocked by firewall remotely, but after mapping,
# we can access it locally with the local port 1234.
ssh -L [127.0.0.1:]10003:remotehost:8000 user@remotehost -N

Then go to localhost:10003 on browser to see the web page.

The port forwarding approach is limited on that single port mapping, for unlimited access, you need SOCKS proxy tunneling, see next section.

SSH SOCKS Proxy Tunnel

Introduction to SOCKS proxy

Although by default SOCKS proxy does not provide encryption, but we run it over SSH, so the traffic is encrypted.

How To Route Web Traffic Securely Without a VPN Using a SOCKS Tunnel: A SOCKS proxy is basically an SSH tunnel in which specific applications forward their traffic down the tunnel to the server, and then on the server end, the proxy forwards the traffic out to the general Internet. Unlike a VPN, a SOCKS proxy has to be configured on an app by app basis on the client machine, but can be set up without any specialty client agents.

The remote host must has ssh server running.

1
2
3
4
5
6
7
8
9
10

# -D: dynamic application-level port forwarding, see curl man for more
# explanation about SOCKS support.
# [127.0.0.1:]11000: local mapping port.
# -N: Do not execute a remote command. This is useful for just forwarding ports.
# -C: Compresses the data before sending it
# -q: quiet

# -f: Forks the process in the background
# don't like tunnel, on the remote host it is dynamic forwarding
ssh -D [127.0.0.1:]11000 -f -C -N -q user@remotehost 

This is actaully a SOCKS5 proxy created by SSH, after it is established, you can check by:

1
2

# Now you can access the web that original can only access by remote host.
curl -ILk -x socks5://localhost:11000 "https://web_can_only_access_by_remotehost"

Or configuring the web browser to use this SOCKS5 proxy: localhost:11000. On Firefox FoxyProxy plugin, set and use it. Now we can access whatever the remotehost can access.

Manually kill the tunnel process if you use -f.

SSH X11 Forwarding

Similar to VNC, but VNC transmits whole desktop which is more expensive. Linux has good support to X11, on Mac, need to install XQuartz(still not work on Mac).

1
2
3
4
5

# -X: X11 forwarding
ssh -X user@remotehost

# gedit is running on remotehost but reflect GUI locally
> gedit

SSH Agent

很久之前看书的时候没明白这个概念. 一个常见的用处就是保护originating host的private key. A handy program called ssh-agent simplifies working with SSH private keys.

In Mac, ssh-agent is running by default, but in Linux, start by yourself (ensure only one instance of ssh-agent is running).

1
2
3
4
5
6
7

# Don't need do this if you are Mac, or your company laptop has agent running by
# default you can check by:
ssh-add -l

# First check if only one instance is running
ps aux | grep ssh-agent
# if it is there but cannot work, kill it.

If you run ssh-agent, it will output environment vars you need to set, for example, you can also manually export these instead of using eval:

1
2
3
4
5
6
7
8
9

ssh-agent

# export these manually is OK
SSH_AUTH_SOCK=/tmp/ssh-YI7PBGlkOteo/agent.2547; export SSH_AUTH_SOCK;
SSH_AGENT_PID=2548; export SSH_AGENT_PID;
echo Agent pid 2548;

# Start it.
eval $(ssh-agent)

Add your private key to ssh-agent, sometimes git ssh clone failed, you may need to add private key to agent:

1
2
3
4
5
6
7
8
9
10
11

# default path ~/.ssh/id-rsa
ssh-add
ssh-add <other private key path>

# list all identities 
ssh-add -l

# delete all identities
ssh-add -D
# delete specified identity
ssh-add -d <private key path>

How to start ssh-agent on login: https://stackoverflow.com/questions/18880024/start-ssh-agent-on-login Add below to your .bash_profile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

SSH_ENV="$HOME/.ssh/env"

function start_agent {
    echo "Initialising new SSH agent..."
    /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
    echo 'succeeded'
    chmod 600 "${SSH_ENV}"
    . "${SSH_ENV}" > /dev/null
    # add private key ~/.ssh/id_rsa.pub
    /usr/bin/ssh-add;
}

# Source SSH settings, if applicable
if [ -f "${SSH_ENV}" ]; then
    # need resource, but ssh-agent is there
    . "${SSH_ENV}" > /dev/null
    # ps ${SSH_AGENT_PID} doesn't work under cywgin
    ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
        # statement block
        start_agent;
    }
else
    start_agent;
fi

When you try to make a connection to a remote host, and you have ssh-agent running, the SSH client will automatically use the keys stored in ssh-agent to authenticate with the host.

Advantages:

1. For encrypted SSH private keys，只有第一次加入ssh-agent的时候要求输入password 如果 不使用ssh-agent，每次SSH都会要求输入password
2. If you are using Ansible to manage hosts that use different SSH keys, using an SSH agent simplifies your Ansible configuration files.
3. ssh-agent forwarding, see below

ssh-agent 还解决了一个问题，比如你有personal and work git account各一个，但各自是不同的 ssh key pairs，在git clone的时候如何指定用哪个private key呢? see this link.

SSH Agent Forwarding

If you are cloning a Git repository on remote host via SSH, you’ll need to use an SSH private key recognized by your Git server. I like to avoid copying private SSH keys to my remote host (for example, a EC2 instance), in order to limit the damage in case a host ever gets compromised.

1
2
3
4
5
6
7
8

# The example.xx.com does not have the private key to access git repo but the
# local host has.
# -A: agent forwarding
ssh -A root@example.xxx.com

# git clone via ssh mechanism on remote host with the private key provided by
# agent from local host.
git clone git@github.com:lorin/mezzanine-example.git

Here -A limit the agent forwarding in this session only, you can have ssh config to set agent forwarding on a broad view.

ProxyJump

现在想想，当时登录openshift or softlayer 的master时，也需要经过堡垒机，所以应该可以配置 proxyjump. 这个和ssh-agent没有必然联系，如果没用ssh-agent, 则应该可以在配置config file中 指定key的位置.

Using OpenSSH ProxyJump It uses vagrant VMs to demonstrate. 但是我觉得不需要再指定port, user, identifykey对 target server了，这些应该在bastion上已经配置好了。

SSH agent and ProxyJump explained Talking risk of SSH agent forwarding, access internal hosts through a bastion, ProxyJump is much safer. 也谈到了SSH是怎么handshake, 传输中用的是新的对称钥匙.

JumpBox or Bastion Host: Notice that you need to generate key and copy the public key to bastion host first.

• Using the SSH Config File
• SSH to remote hosts though a proxy or bastion with ProxyJump

Baston hosts are usually public-facing, hardened systems that serve as an entrypoint to systems behind a firewall or other restricted location, and they are especially popular with the rise of cloud computing.

The ssh command has an easy way to make use of bastion hosts to connect to a remote host with a single command. Instead of first SSHing to the bastion host and then using ssh on the bastion to connect to the remote host, ssh can create the initial and second connections itself by using ProxyJump.

1
2
3
4
5

# -J specify the jumphost
ssh -J <bastion-host> <remote-host> [-l <remote login user>] [-i <pem file>]
ssh -J user@<bastion:port> <user@remote:port>
# Jump through a series of hosts
ssh -J <bastion1>,<bastion2> <remote>

最主要的还是配置~/.ssh/config 文件, basic settings: 注意，之前遇到过奇怪的问题，用同样的config file，别人一切正常，但我就连不上，简化了一下config file后就好了，当时的解决办法是把Match host模块移到匹配的Host 下面，其实Match host不要也行 很多可以合并的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

# May have more options, for example, User, Port, AgentForward, etc.
# refer `man ssh_config`

# The `Host` sections are read in order and the options matched will get
# accumulated

# The Bastion Host
Host <jump-host-nickname>
  User <user name>
  # default is no
  ProxyUseFdpass no
  # jumpbox port
  Port 22
  # jumpbox IP
  HostName <hostname or IP>

# The Remote Host
Host <remote-host-nickname>
  Hostname <remote-hostname or ip address for example: 172.12.234.12>
  User <user name>
  AddKeysToAgent yes
  IdentitiesOnly yes
  # may need pem file, the private key
  IdentityFile ~/.ssh/file.pem
  StrictHostKeyChecking no
  ServerAliveInterval 60

# The remote host match this IP will use jumpbox
# 这个可以不要，就是用来match Host的
Match host 172.??.*
  ProxyJump <jump-host-nickname>

# Or can specify jumpbox directly
Host <remote-host-nickname>
  HostName < remote-hostname or ip address>
  ProxyJump bastion-host-nickname

Then you can ssh directly: ssh remote-host-nickname

ProxyCommand is an alternative of ProxyJump, but it is old.

1

ssh -o ProxyCommand="ssh -W %h:%p bastion-host" remote-host

Force SSH Password Login

Usually SSH password authentication is disabled, that is, you can only log in over SSH using public key authentication, to enable password login:

1
2
3
4
5
6
7

# /etc/ssh/sshd_config
# set to yes
PasswordAuthentication yes
# you may also need to allow root login
PermitRootLogin yes
# restart sshd
systemctl restart sshd

Create new user to test:

1
2

useradd alice
passwd alice

Logout and try:

1
2
3
4
5
6
7

# PubkeyAuthentication may be needed
# then input the password
ssh -p 2222 \
    -o StrictHostKeyChecking=no \
    -o UserKnownHostsFile=/dev/null \
    -o PubkeyAuthentication=no \
    alice@127.0.0.1

Debug

1. Use -v, -vv, -vvv flag in ssh command (比如之前pem file 权限和格式没设置对都 有提示)
2. Wireshark capture ssh traffic on that interface, you should see SSHv2 protocol and more details
3. Check system log, journalctl | grep sshd.
4. Launch sshd on another port in debug mode: sudo /usr/bin/sshd -d -p 2020, then ssh to this port 2020 from client ssh -p 2020 user@remote_server.
5. Possibly restriced by firewall

Usage Summary

1
2
3
4
5
6
7
8
9
10
11
12
13

10/01/2018 ssh send command
11/14/2018 ssh run shell script
12/19/2018 ssh-copy-id
01/06/2019 ssh-kenscan
01/08/2019 ECDSA host key changed
01/22/2019 no prompt first time
01/23/2019 sshpass
02/21/2019 scp folder or files
03/11/2019 ssh -i option
03/12/2019 recover public key
09/05/2020 sftp
01/20/2021 ssh config
03/17/2022 ssh config permission

10/01/2018

use ssh send commands to execute on remote machine: -t flag allow you to interact with remote machine:

11/14/2018

use ssh run shell script in remote machine

12/19/2018

use ssh-copy-id to copy local machine public key to remote machine’s ~/.ssh/authorized_keys file, so next time when you ssh, scp or sftp again, no prompt to require password:

Sometimes I see people use ~/.ssh/id_rsa with ssh-copy-id, that confused me because that is private key, OK, man tells me why:

1
2
3
4

-i identity_file
        ...If the filename does not end in .pub this is added.  If the filename
        is omitted, the default_ID_file is used.

01/06/2019

use ssh-keyscan to get remote machine ecdsa identity, you can put this item into local known_hosts file, so when first time ssh login, there is no prompt to input yes:

Actually better to use -o StrictHostKeyChecking=no flag.

01/08/2019

I create a new cluster with the same master hostname as the deleted one, so when I try to ssh to it, interesting thing happens:

go to ~/.ssh/known_hosts file and delete the corresponding ECDSA line

01/22/2019

when you first time ssh or scp, sftp to remote machine, it will prompt to add remote machine to ~/.ssh/known_hosts file, this may interrupt ansible or shell script running, so I want to skip it. For example:

use -o StrictHostKeyChecking=no option, it will silently add remote host name to ~/.ssh/known_host file.

1
2

ssh-copy-id -i .ssh/id_dsa.pub -o StrictHostKeyChecking=no root@example.com
scp -o StrictHostKeyChecking=no -r ./source root@example.com:~

if you don’t want to add the host name, -o UserKnownHostsFile=/dev/null option can save you.

01/23/2019

scp or ssh without prompt input password

1
2
3

yum install -y sshpass
# Explicitly input password
sshpass -p <password> scp/ssh ...

It’s useful to set password-less at first time, combine all of these, no prompt will show up:

1

sshpass -p <password> ssh-copy-id -i ~/.ssh/id_rsa.pub -o StrictHostKeyChecking=no ...

02/21/2019

scp source directory and it’s content recursively to root user home directory in example.com.

1

scp -o StrictHostKeyChecking=no -r ~/source root@example.com:~

scp all files in source directory to target directory in example.com.

1

scp -o StrictHostKeyChecking=no ./source/* root@example.com:~/target

03/11/2019

The ssh command has -i option, you associate private key with this flags:

1

ssh -i ~/.ssh/id_rsa xxx

Note that SSH never send private key over the network, -i merely used to answer challenge that is generated using the corresponding public key from target machine, you don’t need to explicitly use -i if you use default private key in right location.

03/12/2019

If public key is lost, you can use existing private key to generate one:

1

ssh-keygen -y -f ~/.ssh/id_rsa > ~/.ssh/id_rsa.pub

Or just create new key pair

1

echo "yes" | ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa

09/05/2020

sftp server is up when install openssh-server package, and can be configured in /etc/ssh/sshd_config file: https://www.techrepublic.com/article/how-to-set-up-an-sftp-server-on-linux/

For the interactive commands, see man sftp, there are logs of regular commands can be used in sftp, for example: cd, chmod, ln, rm, etc.

The online free ftp server for testing purpose. Also, the mmnt.net can be used to find free ftp servers.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# Use password or ssh-public key to login to sftp server
sftp -o UserKnownHostsFile=/dev/null \
-o StrictHostKeyChecking=no \
demo@test.rebex.net[:path]

# print local working directory, the default place to hold download file
sftp> lpwd
# change local working directory
sftp> lcd [path]
# escape to local shell, type `exit` to back
sftp> !
# secape to local command
sftp> ![command]

# enable/disable progress meter
sftp> progress
# download file to local working directory
sftp> get <filename>
# download file to specified directory
sftp> get <filename> <local file path>

# upload file
# default file is in local working directory and upload to sftp current folder
# if no path is specified
sftp> put [local file path] [remote file path]

# quit sftp
sftp> bye

For non-interactive download/upload file:

1
2
3
4
5

# download
sftp user@hostname[:path] <local file path>
# upload, tricky
echo "put <local file path>" | sftp user@hostname[:path]
sftp user@hostname[:path] <<< $'put <local file path>'

Used in shell script:

1
2
3
4
5
6

sftp user@hostname <<EOF
cd /xxx/yyy/zzz
cd /aaa/bbb/ccc
put file.tgz
bye
EOF

01/20/2021

When the network connection is poor and ruining your SSH session, you can adjust the connection settings with larger interval probing and retry times:

1
2
3
4
5
6
7
8
9
10
11
12

Host myhostshortcut
    User XXX
    Port 22
    # or ip address
    HostName myhost.com
    User barthelemy
    # no-op probe to server interval 60s
    ServerAliveInterval 60
    # probe conunt max 10 times if noresponse
    ServerAliveCountMax 10
    # no tcp no-op probe
    TCPKeepAlive no

08/13/2021

When I ran git pull on Gitlab local repo from my cloudtop, I got error output:

1
2
3
4
5
6

Received disconnect from UNKNOWN port 65535:2: Too many authentication failures
Disconnected from UNKNOWN port 65535
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

This was resolved by adding IdentitiesOnly yes in ssh config file under gitlab config block, which instructs ssh to only use the authentication identity files specified on the command line or the configured in the ssh_config file.

Reference: https://www.tecmint.com/fix-ssh-too-many-authentication-failures-error/

03/17/2022

The Ansible playbook failed to ssh target VM and reported error:

1
2
3
4
5

module.backups-v2.null_resource.setup_cluster (local-exec): [WARNING]: Unhandled
error in Python interpreter discovery for host
module.backups-v2.null_resource.setup_cluster (local-exec): 10.10.16.205: Failed
to connect to the host via ssh: Bad owner or permissions
module.backups-v2.null_resource.setup_cluster (local-exec): on /root/.ssh/config

It turns out the permission and owner issue on /root/.ssh/config file, see this ticket for detail, the fix is to set 600 as chmod and chown by owner.

I want to introduce you two useful tools for kubernetes development, github link with detail.

Note: this is not an official Google product.

kubectx

kubectx helps you switching between clusters back and forth.

Install

Since kubectx/kubens are written in Bash, you should be able to install them to any POSIX environment that has Bash installed.

1

git clone https://github.com/ahmetb/kubectx /opt/kubectx

Make sure kubectx script is executable:

1
2
3
4
5
6
7
8
9
10

[root@myk8s1 ~] ls /opt/kubectx/ -ltr
total 40
-rw-r--r-- 1 root root 11357 Feb 17 21:25 LICENSE
-rw-r--r-- 1 root root   968 Feb 17 21:25 CONTRIBUTING.md
-rw-r--r-- 1 root root  7784 Feb 17 21:25 README.md
drwxr-xr-x 2 root root   121 Feb 17 21:25 completion
drwxr-xr-x 2 root root    84 Feb 17 21:25 img
drwxr-xr-x 3 root root   100 Feb 17 21:25 test
-rwxr-xr-x 1 root root  5273 Feb 17 21:25 kubens
-rwxr-xr-x 1 root root  5218 Feb 17 21:25 kubectx

Create symlinks to kubectx/kubens from somewhere in your PATH, like /usr/local/bin

1

ln -s /opt/kubectx/kubectx /usr/local/bin/kubectx

Usage

You should first understand how to switch among different clusters by using configuration files. please investigate this link, actually kubectx is built on top of it.

For example, I have one cluster on AWS and one cluster on Fyre, in each cluster there is a ~/.kube/config file, rename it as config.aws and config.fyre and put them to another client machine ~/.kube/ folder with kubectl installed.

1
2
3
4
5
6
7

[root@centctl1 .kube]# ls -ltr
total 16
drwxr-xr-x 3 root root   23 Nov 26 16:38 cache
-rw-r--r-- 1 root root 2214 Dec  6 10:16 config.aws
drwxr-xr-x 2 root root   73 Dec  6 10:16 kubens
drwxr-xr-x 3 root root 4096 Feb 17 22:05 http-cache
-rw------- 1 root root 5474 Feb 17 22:22 config.fyre

Append config files to environment variable KUBECONFIG , you can add export to .bashrc file.

1

export KUBECONFIG=$KUBECONFIG:$HOME/.kube/config.aws:$HOME/.kube/config.fyre

Now if you run kubectx you will see there are 2 contexts:

1
2
3

[root@centctl1 .kube] kubectx
arn:aws:eks:us-west-2:296744932886:cluster/IIS-Test-Cluster
kubernetes-admin@kubernetes

Jump to kubernetes-admin@kubernetes:

1
2

[root@centctl1 .kube] kubectx kubernetes-admin@kubernetes
Switched to context "kubernetes-admin@kubernetes".

Jump back:

1
2

[root@centctl1 .kube] kubectx -
Switched to context "arn:aws:eks:us-west-2:296744932886:cluster/IIS-Test-Cluster".

It is the same as you run below commands:

1
2

kubectl config view
kubectl config --kubeconfig=config.fyre use-context kubernetes-admin@kubernetes

kubens

kubens helps you switch between Kubernetes namespaces smoothly, so you don’t need to add -n <namespace> in every command.

Install

Download the same Github repository as kubectx, add symlinks:

1

ln -s /opt/kubectx/kubens /usr/local/bin/kubens

Usage

1
2
3
4
5
6
7
8
9
10

[root@myk8s1 ~] kubens
default
kube-public
kube-system
[root@myk8s1 ~] kubens kube-system
Context "kubernetes-admin@kubernetes" modified.
Active namespace is "kube-system".
[root@myk8s1 ~] kubens -
Context "kubernetes-admin@kubernetes" modified.
Active namespace is "default".

Kubernetes version 1.13.2

Issue description

I encountered the problem that when delete statefulset the execution hangs, for example:

1

kubectl delete statefulset is-en-conductor

it cannot delete the statefulset and associated pods at all

1
2
3
4
5
6
7
8
9

[root@opsft-lb-1 OpenShift] oc get all
NAME                             DESIRED   CURRENT   AGE
statefulsets/is-en-conductor     0         1         5h
statefulsets/is-engine-compute   0         2         5h

NAME                     READY     STATUS        RESTARTS   AGE
po/is-en-conductor-0     0/1       Terminating   2          1h
po/is-engine-compute-0   1/1       Running       0          5h
po/is-engine-compute-1   0/1       Terminating   0          5h

Solution

1

kubectl delete statefulsets <statefulset name> --force --grace-period=0 --cascade=false

now we only have hanging pods, force delete them

1
2
3
4

NAME                     READY     STATUS        RESTARTS   AGE
po/is-en-conductor-0     0/1       Terminating   2          1h
po/is-engine-compute-0   1/1       Running       0          5h
po/is-engine-compute-1   0/1       Terminating   0          5h

1

kubectl delete pod <pod name> --force --grace-period=0

This blog is about how to deploy Hexo with Next theme on your Github pages. It also mentions some common plugins such as search, markdown and Mermaid.

Kubernetes version 1.13.2

We want to scale the compute pods by calling k8s API from inside the engine conductor container, this definitely need to be authorized and we need to grant privilege for this action.

There are some concepts you need to know in order to achieve the goal.

Service Account

what is Service Account

Processes in containers inside pods can contact the apiserver. When they do, they are authenticated as a particular service account (for example, by default is default service account).

Once you create a namespace, for example test-1, there is a default service account automatically generated.

1

kubectl get sa -n test-1

1
2

NAME      SECRETS   AGE
default   1         139m

let’s see what is inside the service account

1

kubectl describe sa default -n test-1

1
2
3
4
5
6
7
8

Name:                default
Namespace:           test-1
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   default-token-mtv4n
Tokens:              default-token-mtv4n
Events:              <none>

Here we see there is a mountable secret default-token-mtv4n, that is the credentials to access the apiserver.

1

kubectl describe secret default-token-mtv4n -n test-1

1
2
3
4
5
6
7
8
9
10
11
12
13

Name:         default-token-mtv4n
Namespace:    test-1
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: default
              kubernetes.io/service-account.uid: 387381d3-2272-11e9-91a2-00163e0196e7

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  6 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrd...

ClusterRole

what is Role and ClusterRole

A ClusterRole can be used to grant the same permissions as a Role, but because they are cluster-scoped, they can also be used to grant access to

• cluster-scoped resources (like nodes)
• non-resource endpoints (like “/healthz”)
• namespaced resources (like pods) across all namespaces

Here we use a cluster role called cluster-admin, it’s generated by default

1

kubectl get clusterrole | grep cluster-admin

1

cluster-admin                        174m

ClusterRole Binding

what is RoleBinding and ClusterRole Binding

A role binding grants the permissions defined in a role to a user or set of users. It holds a list of subjects (users, groups, or service accounts), and a reference to the role being granted. Permissions can be granted within a namespace with a RoleBinding, or cluster-wide with a ClusterRoleBinding.

below we grant service account default in namespace test-1 the cluster-admin level privilege.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: <cluster role name>
  labels:
    <key:value>
subjects:
 - kind: ServiceAccount
   name: default
   namespace: <namespace of sa>
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

then we can write a script, using curl to call K8s API, for example, to scale the number of compute pods:

1
2
3
4
5

http_code=$(curl -w "%{http_code}" -sS  --cacert $CACERT  -XPATCH -H "Content-Type: application/strategic-merge-patch+json" -H "Accept: application/json" -H "Authorization: Bearer $TOKEN" "https://kubernetes.default/apis/apps/v1/namespaces/$NAMESPACE/statefulsets/is-engine-compute" --data "{\"spec\":{\"replicas\":$REP}}" -o $OUT_FILE)
if [[ $http_code -ne 200 ]]; then
    ${JQ} '{ result:.status, code: .code,  message: .message }' $OUT_FILE
    exit 1
fi

Where are these CACERT, TOEKN and NAMESPACE from? Actually each container has a default mount point reside in:

1

/var/run/secrets/kubernetes.io/serviceaccount

You can see this when you run kubectl describe pod. Just like other mount files, there are 3 files, for example:

1
2
3
4

total 0
lrwxrwxrwx 1 root root 13 Sep 14 16:21 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root 16 Sep 14 16:21 namespace -> ..data/namespace
lrwxrwxrwx 1 root root 12 Sep 14 16:21 token -> ..data/token

All of them are used in curl command above.

1
2
3

NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

Mac provides the command line for copy and paste, useful for some tasks.

reference link：http://osxdaily.com/2007/03/05/manipulating-the-clipboard-from-the-command-line/

Copy

copy the content of the file

1

pbcopy < file.txt

Now the content is in clipboard, ready to be pasted.

you can use pipe to combine command such as find, grep, awk and cut to filter and augment data.

for example:

1
2

echo "this is a demo" | pbcopy
docker images | grep "iis-" | pbcopy

Paste

Simply run this command:

1

pbpaste

Redirect content to a file:

1

pbpaste > file.txt

1

pbpaste | grep "xxx"

This is the blog incubator and TODO list 保持记忆的熟练的唯一方式就是复习 和 刷题，特别是语言类。 最近在做一些pipeline的工作，需要快速上手，主要涉及到的: Terraform Python Jenkins gradle (一种build tool, others like Blaze, Make) Groovy (JVM language, similar to Java, for declarative Jenkins pipeline) 现在不清楚的地方是各自文件的语法。 先从groovy，gradle入手，然后jenkins。 [x] 复习过去x天改动过的blog

CI/CD

[x] travis.yml and inspiration: 大概能看出来是怎么回事了

Envoy Migration

最近几天发现的缺失: [x] SSH config and proxy config file, proxy jump, ssh-add. [x] Gitlab, see linkedin learning, gitlab-ci and web dashboard general use. [x] Http header, CONNECT method, CRLF, browser debug [x] docker compose for testing, why not docker? [x] SSL/TLS, cloud computing use cases, see udemy ip route table vs iptables filter/nat table??? [x] curl flags and testing, pluralsight [x] linux tips weekly, proxy sections [x] Unix domain socket, websocket, co-routine? [x] proxy, http proxy, reverse proxy?? ssl/tls?

[x] envoy tcp proxy and how to test? 设置好socket server, client，然后client connect to proxy port, in proxy, it driect to server address and port. 解决了！upstream cluster IP 没设置好。。对于docker 不是127.0.0.1!! [x] for now, understand: http proxy (connect support), tcp proxy, socks proxy. [x] shell parameter expansion variable pattern match: ${1##*/} ${USAGE%/*} ${1:1:1}, etc [x] proxy的模拟测试环境 -> docker compose + flask + python other packages [x] pagerduty, for on-call

[ ] socks how to setup in gcloud or aws --> VPN

[ ] ip/iptables, vagrant + envoy docker does not work, linkedin learning

[ ] network traffic school, internally! see my SRE week note. [ ] 养老金扩展: https://www.transglobalus.com/zh/services_zh/

[ ] monit: is a utility for managing and monitoring processes, for example restart service when it cost too much memory and cpu

[x] systemd init file

1
2
3
4
5
6

## some fields
network-online.target
Environment=
OOMScoreAdjust=-1000 ~ 1000
WantedBy=multi-user.target
Type=forking | simple

[x] what are smoke & nightly testing [x] blue/green deployment

[ ] Linux LPIC-1/2/3 exams [ ] Linux LFCE training [ ] CCNA Cisco network certifiate, see pluralsight my channel network

[ ] 重读how linux works book [ ] rsyslogd, syslog services

[ ] Using golang write tcp, http server. [ ] container runtime varies, search youtube [ ] device42 [ ] tower [ ] packer [ ] jinja2 template syntax

[x] linux screen command [ ] serverspec vs testinfra, infra instance testing tools [ ] auditiong user account on linux, how? [ ] command history fzf plugin [ ] load average: http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html

Python

[x] python click必须掌握，很重要！对于advanced script!! click echo color support: click.echo(click.style('Some more text', bg='blue', fg='white')) 这样可以为以后log中重点高亮 [x] python logging 更高级的用法

如果要建造一个package 哪些是必要的流程，以及测试? [ ] setuptools understand [ ] pytest [ ] cookiecutter, creating new project [ ] pip install -e . (on the fly editing and take effect)

K8s

kubectl wait command: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#port-forward

kubectl-images: https://github.com/chenjiandongx/kubectl-images

kubectl krew: https://github.com/kubernetes-sigs/krew

[ ] k8s is deprecating docker in upcoming release. switch to other common docker runtime: containerd and CRI-O. There are lots of options out there for this specific use case including things like kaniko, img, and buildah.

Squid

这个对forward proxy支持挺好, long history proxy -_-|||, old [ ] squid proxy

Groovy

掌握基本data type, control structure, function definition 就可以了. 因为目前主要是用在Jenkins中。 和Java 通用的.

Groovy quick start: https://www.youtube.com/watch?v=B98jc8hdu9g

Books, course and presentations, recommend: https://groovy-lang.org/learn.html#books O’relly course

http://groovy-lang.org/syntax.html alternative java language

DSL: domain specific language sdkman very good installation toolkit (also for gradle!)

1
2
3
4

## must have jdk installed first
yum install java-1.8.0-openjdk-devel -y
## specify version 2.5.2
sdk install groovy 2.5.2

Utilize vagrant VM to setup groovy runtime with SDKMAN, sync folder to work with VSC. groovy closure: https://groovy-lang.org/closures.html

1
2
3
4

// each
def names = ["alice", "bob", "michael"];
// using closure
names.each { println it };

OOP, the class can be no constructor.

String interpolation: https://groovy-lang.org/syntax.html#_string_interpolation

Gradle

don’t see latest book, please refer to official doc https://gradle.org/guides/ Ant and Maven are also build tools, gradle wrapper, incremental builds.

1. build by convention
2. Groovy DSL
3. supports dependencies
4. support mutli-project build
5. easily customizable

To install Gradle, using sdkman tool.

有几个概念搞清楚:

1. gradle wrapper -> gradlew (version control)
2. build.gradle: syntax, plugin
3. gradle.properties
4. extract env variable

1
2
3
4
5
6
7
8

apply plugin: 'java'

task {
  doLast {
    // groovy syntax
    println "hello, gradle!"
  }
}

Git

using use/passowrd instead of ssh need to setup personal access credential: https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line

for example, mypersonal access token 456f6facec85415b91588f581sfsdfssfe56c0 I can embed the user/cred to push command: https://stackoverflow.com/questions/29776439/username-and-password-in-command-for-git-push

1

git push 'https://chengdol%40ibm.com:456f6facec85415b91588f58182dd1cfe27e56c0@github.ibm.com/repo/hello.git'

use %40 replace @. github.ibm.com/repo/hello.git is the url to your repo

[x] git tag, see my blog [ ] github CLI: gh, very similar to git command, can make alias to it.

[ ] pluralsight git training in my channel [ ] unfinished, fork operation

[ ] gitlab on GKE lab

[ ] git commands more git rewrite: https://jonhnnyweslley.net/blog/how-to-rewrite-git-urls-to-clone-faster-and-push-safer/ https://paste.googleplex.com/6716960572178432

1

git diff --name-only --diff-filter=ACMR HEAD~1 HEAD

deploy keys

https://github.blog/2015-06-16-read-only-deploy-keys/

Gitbook

https://www.gitbook.com/ 这个写东西还可以, but not free

承上启下

interpersonal skill, BQ https://mail.google.com/mail/u/1/#inbox/FMfcgxwHNMbchDhFLQBnSltwkxBJVLbX

As an “intro” project, I would recommend deploying Kubernetes using Terraform. You can use any provider such as GKE, AKS or EKS.

Once the Terraform template is working, extend it to install some applications inside Kubernetes, such as Vault, Elastic, Cassandra, etc. using the Helm and Helmfile.

You may want to integrate the database applications inside Kubernetes with a Vault instance such that they would use it for authentication backend.

You can either build the Helm Charts yourself or use existing ones (perhaps you would want to modify them…)

Further, you can extend the application deployment to emit some metrics and setup monitoring using Prometheus and Grafana.

Once you are happy with the runtime stack, introduce some CI/CD pipelines using Jenkins and/or Gitlab.

现在能接触到的基础打好 k8s, docker, jenkins (groovy + gradle), ansible, helm git -> gitlab (with jenkins)

• 常用工具: vim visual studio bash (中文版教程可以), zsh

• composing database layer: cassandra elasticsearch (monitoring + log system) consul vault

• monitoring: prometheus grafana kibana

• others: kafka zookeeper istio golang RabbitMQ redis Python

load balancer: nginx, haproxy

• cloud 基本结构，应用，组件: AWS, ok GCP, ok Azure

我猜工作流程: 用terraform 构造k8s 环境 或其他， 用ansible增加或更改配置，然后在k8s中部署应用(prometheus + grafana and elk for logs)，最后集成CI/CD development -> QA/UAT - > production

understand

如果不清楚这些则看不懂配置的含义, 要清楚各个cloud中resource的类型，配置，如何搭配: instance, vpc, lb, dns, storage, routing, subnet, gateway

terraform -> deploy k8s (EKS or GKS) -> install vault, elasticsearch, cassandra by helm (need modify though) -> Prometheus Grafana Kibana -> CI/CD: jenkins + gitlab

把完全不知道是什么的快速过一遍。 先学terraform -> elasticsearch -> vault, cassandra -> monitoring tools packer -> gitlab, tower -> helmfile

steps

1. terrform provision k8s on aws and gcp
2. k8s on fyre to test other components via helm
3. monitoring system: prometheus + grafana, deployed by operator
4. log system: elasticsearch, kibaba, logstash, beats

实验: 在k8s中部署prometheus + elastic stack, by helm or yaml，监控系统状态

5. Hashicorp: consul, vault
6. cassandra

How to be a contribution beginner on k8s? https://youtu.be/o68ff5NokR8

1. find issues, search label good first issue
2. communication
3. build relationships

Kubernetes version 1.13.2 This article mainly talks about setting up k8s cluster by kubeadm manually. As far as I know there are no coming changes that will significantly impact the validity of these steps.

Cluster Info

I have a 3 nodes bare-metal cluster called myk8s with CentOS version 7.5, the /etc/hosts file in each node:

1
2
3

172.16.158.44    myk8s1.fyre.ibm.com myk8s1
172.16.171.110   myk8s2.fyre.ibm.com myk8s2
172.16.171.227   myk8s3.fyre.ibm.com myk8s3

Let’ see the network interface on master node:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# ifconfig -a

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.158.44  netmask 255.255.0.0  broadcast 172.16.255.255
        ether 00:16:3e:01:9e:2c  txqueuelen 1000  (Ethernet)
        RX packets 1617615790  bytes 203050237209 (189.1 GiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 436  bytes 50037 (48.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 9.30.97.218  netmask 255.255.254.0  broadcast 9.30.97.255
        ether 00:20:09:1e:61:da  txqueuelen 1000  (Ethernet)
        RX packets 13350021  bytes 1424223654 (1.3 GiB)
        RX errors 0  dropped 5  overruns 0  frame 0
        TX packets 246436  bytes 45433438 (43.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
...

Configure

For every node in cluster, following instruction below

Install utilities

1
2
3

yum update -y
yum install -y vim
yum install -y git

Disable firewall

Check firewall status and disable it if active

1

systemctl status firewalld

1
2

systemctl disable firewalld
systemctl stop firewalld

Install kubeadm kubectl and kubelet

Install kubeadm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF

# Set SELinux in permissive mode (effectively disabling it)
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

systemctl enable --now kubelet

Setting SELinux in permissive mode by running setenforce 0 and sed ... effectively disables it. This is required to allow containers to access the host filesystem, which is needed by pod networks for example. You have to do this until SELinux support is improved in the kubelet.

Currently installed:

1
2

Installed:
  kubeadm.x86_64 0:1.13.3-0               kubectl.x86_64 0:1.13.3-0               kubelet.x86_64 0:1.13.3-0

check /etc/sysctl.conf file, for example:

1
2
3
4
5

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv4.ip_forward = 0
...

ensure that these 3 options exist and set to 1, because some users on RHEL/CentOS 7 have reported issues with traffic being routed incorrectly due to iptables being bypassed

1
2
3

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

if these 3 items not set, edit net.ipv4.ip_forward = 1 and append net.bridge.bridge-nf-call-ip6tables = 1 and net.bridge.bridge-nf-call-iptables = 1 in sysctl.conf file

then make sure that the net.bridge.bridge-nf-call is enabled, check if br_netfilter module is loaded. This can be done by running

1

lsmod | grep br_netfilter

if not, to load it explicitly call

1

modprobe br_netfilter

next run this command to reload setting

1

sysctl --system

then you can check the final setting:

1

sysctl -a | grep -E "net.bridge|net.ipv4.ip_forward"

Install docker

CRI installation in Kubernetes

Uninstall old versions

Older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies.

1
2
3
4
5
6
7
8

yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

Official Docker installation guides

Install Docker CE

currently Docker version 18.06.2 is recommended, but 1.11, 1.12, 1.13 and 17.03 are known to work as well. Keep track of the latest verified Docker version in the Kubernetes release notes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

# Set up the repository
yum install yum-utils device-mapper-persistent-data lvm2

# Add docker repository.
yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

# Install docker ce.
yum update && yum install docker-ce-18.06.2.ce

# Create /etc/docker directory.
mkdir /etc/docker

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker

check result

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

[root@myk8s1 ~] docker version
Client:
 Version:           18.06.2-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        6d37f41
 Built:             Sun Feb 10 03:46:03 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.2-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       6d37f41
  Built:            Sun Feb 10 03:48:29 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Disable swap

why we need to disable swap? Swap brings disk IO overhead, as well as breaking cgroups for pod memory control.

in /etc/fstab file, comment out swap setting

1

/dev/mapper/centos-swap swap                    swap    defaults        0 0

activate new configuration and check

1

swapoff -a

1
2
3
4
5

[root@myk8s3 ~] free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        189M        5.7G        136M        1.8G        7.0G
Swap:            0B          0B          0B

for worker nodes in cluster, stop here. Continue steps in master node:

Initialize kubernetes cluster

I will use Calico as the container network solution, in master node, run

1

kubeadm init --pod-network-cidr=192.168.0.0/16

you can specify the version by using --kubernetes-version v1.13.3, otherwise it will pull latest version from Internet.

you can see the output like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

...
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"

...
Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 9.30.97.218:6443 --token jjkiw2.n478eree0wrr3bmc --discovery-token-ca-cert-hash sha256:79659fb0b3fb0044f382ab5a5e317d4f775e821a61d0df4a401a4cbd8d8c5a7f

keep the last command for joining worker node later

1
2

kubeadm join 9.30.97.218:6443 --token jjkiw2.n478eree0wrr3bmc --discovery-token-ca-cert-hash sha256:79659fb0b3fb0044f382ab5a5e317d4f775e821a61d0df4a401a4cbd8d8c5a7f

then run following command in master node:

1
2
3

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

now if you run kubectl version, you will get something like below:

1
2
3

[root@myk8s1 ~] kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

let’s check what kind of docker images pulled from network to create the cluster in master

1
2
3
4
5
6
7
8
9

[root@myk8s1 ~] docker images
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-apiserver            v1.13.3             fe242e556a99        2 weeks ago         181MB
k8s.gcr.io/kube-controller-manager   v1.13.3             0482f6400933        2 weeks ago         146MB
k8s.gcr.io/kube-proxy                v1.13.3             98db19758ad4        2 weeks ago         80.3MB
k8s.gcr.io/kube-scheduler            v1.13.3             3a6f709e97a0        2 weeks ago         79.6MB
k8s.gcr.io/coredns                   1.2.6               f59dcacceff4        3 months ago        40MB
k8s.gcr.io/etcd                      3.2.24              3cab8e1b9802        4 months ago        220MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        14 months ago       742kB

Launch cluster network

1
2
3
4
5
6
7
8
9

# kubectl get pods --all-namespaces
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   coredns-86c58d9df4-5dfh9                      0/1     Pending   0          9m30s
kube-system   coredns-86c58d9df4-d9bfm                      0/1     Pending   0          9m30s
kube-system   etcd-myk8s1.fyre.ibm.com                      1/1     Running   0          8m52s
kube-system   kube-apiserver-myk8s1.fyre.ibm.com            1/1     Running   0          8m37s
kube-system   kube-controller-manager-myk8s1.fyre.ibm.com   1/1     Running   0          8m34s
kube-system   kube-proxy-wxjx8                              1/1     Running   0          9m31s
kube-system   kube-scheduler-myk8s1.fyre.ibm.com            1/1     Running   0          8m46s

you can find some pods are not ready, for example coredns-86c58d9df4-5dfh9 and coredns-86c58d9df4-d9bfm, also the master node

1
2
3

[root@myk8s1 ~] kubectl get nodes
NAME                  STATUS     ROLES    AGE   VERSION
myk8s1.fyre.ibm.com   NotReady   master   11m   v1.13.3

it’s time to set up network, you should first figure out which Calico version you need, check kubernetes release note, we see currently it support Calico version 3.3.1:

you can also refer this link to install, it’s the same setting as below:

1
2
3

kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

after applying rbac-kdd.yaml and calico.yaml, now you can see

1
2
3
4
5
6
7
8
9
10

# kubectl get pods --all-namespaces
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   calico-node-4vm2c                             2/2     Running   0          45s
kube-system   coredns-86c58d9df4-5dfh9                      1/1     Running   0          37m
kube-system   coredns-86c58d9df4-d9bfm                      1/1     Running   0          37m
kube-system   etcd-myk8s1.fyre.ibm.com                      1/1     Running   0          36m
kube-system   kube-apiserver-myk8s1.fyre.ibm.com            1/1     Running   0          36m
kube-system   kube-controller-manager-myk8s1.fyre.ibm.com   1/1     Running   0          36m
kube-system   kube-proxy-wxjx8                              1/1     Running   0          37m
kube-system   kube-scheduler-myk8s1.fyre.ibm.com            1/1     Running   0          36m

1
2
3

# kubectl get nodes
NAME                  STATUS   ROLES    AGE   VERSION
myk8s1.fyre.ibm.com   Ready    master   38m   v1.13.3

Note that I encountered the problem that when join the worker nodes, the calico-node becomes not ready

1
2
3
4
5
6

# kubectl get pods --all-namespaces
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   calico-node-4vm2c                             1/2     Running   0          11m
kube-system   calico-node-zsbjj                             1/2     Running   0          96s
kube-system   coredns-86c58d9df4-5dfh9                      1/1     Running   0          48m
...

The reason is my master node has multiple eth, I need to specify which one to use in order to be consistent among all nodes.

1

wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

delete the previous Calico deployment and then edit and apply yaml file again:

1
2
3
4
5
6

[root@myk8s1 ~] kubectl get pods --all-namespaces
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   calico-node-dpcsp                             2/2     Running   0          6m15s
kube-system   calico-node-gc5hs                             2/2     Running   0          6m15s
kube-system   coredns-86c58d9df4-5dfh9                      1/1     Running   0          81m
...

Join worker nodes

Join worker nodes is pretty easy, run this command on all worker nodes:

1

kubeadm join 9.30.97.218:6443 --token jjkiw2.n478eree0wrr3bmc --discovery-token-ca-cert-hash sha256:79659fb0b3fb0044f382ab5a5e317d4f775e821a61d0df4a401a4cbd8d8c5a7f

Check node status

1
2
3
4
5

# kubectl get nodes
NAME                  STATUS   ROLES    AGE   VERSION
myk8s1.fyre.ibm.com   Ready    master   83m   v1.13.3
myk8s2.fyre.ibm.com   Ready    <none>   36m   v1.13.3
myk8s3.fyre.ibm.com   Ready    <none>   33s   v1.13.3

By default, tokens expire after 24 hours. If you are joining a node to the cluster after the current token has expired, you can create a new token by running the following command on the master node

1

kubeadm token create

If you don’t have the value of --discovery-token-ca-cert-hash, you can get it by running the following command chain on the master node:

1
2

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
   openssl dgst -sha256 -hex | sed 's/^.* //'

Now a fresh kubernetes cluster with 1 master and 2 worker nodes is created.

How Does lookup operator work now

1. Process every record that would compose to be a lookup table, read one record at a time, and add the record to the memory (may overflow here). Until all the records are processed, then save the lookup table in the memory into disk.

2. Reads the lookup table file back to memory (may overflow here) to setup the hash bucket for each record (set next record offset) , then write to disk again.

3. Load the entire lookup table file from the disk to memory to do lookup(may overflow here).

As we can see, there are 3 parts that will cause memory overflow, the main reason is that we load the whole lookup table into memory.

To solve this problem, the easy and straightforward way to do is to divide the lookup table into several parts (we call this as section), then process these sections accordingly.

New Design

Let’s see the diagram to illustrate the workflow:

Header: meta data about lookup table: address, offset… Bucket vector: speed up lookup by chaining the records to different bucket using hash.

Here assume we have 5 records and 2 buckets, so when we lookup record 4, after hashing, we search from bucket2, so we will only walk through record 2, skip record 1,3 and 5. Of course, the pointer is stored in each record, actually it’s an offset value.

section file container: we fix the size of section file container in memory (can be configured by client or set by some logic according to the size of memory in client machine)

Current Design to minimize the memory usage

1. Set a max memory cap (section file container size).

2. Process each record as it got read in, add the size of the record, when the size will exceed or be the same as max memory cap, save the records that collected so far into a file (called section file, size of section file <= max memory cap size).

3. Repeat the step above until all records have been saved into section file. Therefore, one big lookup table file now has been divided into several chunks of section files.

4. Process the input record and use its key to get the hash bucket number, chain the records(namely add offect in placeholder left, then we need to wirte new content back to disk, need to swap section files with writing and reading operations).

5. when do lookup operation, hash the key and get the entry to lookup it, walk through and compare each record until find it or not, may need to read several section file from disk (now write operation).

If the size of input data <= 2 * available memory size (ignore other small overhead), we will only have 2 section files, the performance will be only corroded slightl(assume the read/write file operation is good).