海胆阶段's Blog

Introduction

Kubernetes 1.14: Local Persistent Volumes GA Recap: A local persistent volume represents a local disk directly-attached to a single Kubernetes Node. With the Local Persistent Volume plugin, Kubernetes workloads can now consume high performance local storage using the same volume APIs that app developers have become accustomed to.

一个和hostPath的重要区别: The biggest difference is that the Kubernetes scheduler understands which node a Local Persistent Volume belongs to. With HostPath volumes, a pod referencing a HostPath volume may be moved by the scheduler to a different node resulting in data loss. But with Local Persistent Volumes, the Kubernetes scheduler ensures that a pod using a Local Persistent Volume is always scheduled to the same node.

While HostPath volumes may be referenced via a Persistent Volume Claim (PVC) or directly inline in a pod definition, Local Persistent Volumes can only be referenced via a PVC. This provides additional security benefits since Persistent Volume objects are managed by the administrator, preventing Pods from being able to access any path on the host.

Additional benefits include support for formatting of block devices during mount, and volume ownership using fsGroup.

注意: 实际上emptyDir + fsGroup也可以实现类似hostPath的效果，emptyDir用的是/sysroot (RedHat Linux), 比如多个pods 使用emptyDir在同一个Node, 我在各自的emptyDir中touch了一个file: compute-0 和compute-3, 进入Node使用find command就可以看到了:

1
2
3

/sysroot/ostree/deploy/rhcos/var/lib/kubelet/pods/68e65ed4-4e62-4588-9269-8947dea9dd46/volumes/kubernetes.io~empty-dir/compute-dedicated-scratch/compute-0

/sysroot/ostree/deploy/rhcos/var/lib/kubelet/pods/92b26b37-92f3-4609-83cb-da8cb8727ca2/volumes/kubernetes.io~empty-dir/compute-dedicated-scratch/compute-3

还需要注意的是，local storage provisioning 在每个node上只会provision attach的disk个数一样的PV，并且这个PV会被一个PVC占据，尽管PV大小是500G但是PVC只请求5G。（不知道这个以后是否会有改进）

Steps

Only test on OCP 4.3 version

OpenShift persistent storage using local volumes Deploy local-storage Operator

1. install local storage operator (it is by default set in local-storage namespace)
2. provision the local storage
3. create local volume persistentVolumeClaim and attach to pod

After deploy the operator, then

1
2
3

## get hostname of each worker node
## can use label -l to filter worker node if needed
kbc describe node | grep hostname

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

apiVersion: "local.storage.openshift.io/v1"
kind: "LocalVolume"
metadata:
  name: "local-disks"
  namespace: "local-storage"
spec:
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          ## put hostname above here
          - worker0.jc-portworx.os.xx.com
          - worker1.jc-portworx.os.xx.com
          - worker2.jc-portworx.os.xx.com
  storageClassDevices:
    - storageClassName: "local-sc"
      volumeMode: Filesystem
      ## The file system that will be formatted when the local volume is mounted
      fsType: xfs
      devicePaths:
        ## use blkid command to get this devicePath
        - /dev/vdc

For example:

1
2
3
4

blkid
## for clarity I remove unrelated output
## First colume is the device path
/dev/vdc: LABEL="mdvol" UUID="fdac344f-8d5f-48bd-9101-99cb416bb93d" TYPE="xfs"

let’s check /dev/vdc

1
2
3
4

lsblk /dev/vdc

NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vdc  252:32   0  500G  0 disk

After apply the CR LocalVolume, let’s check local-storage namespace status, you should see lcoal diskmaker and provisioner pods are up and running, corresponding PVs are ready as well.

1
2
3
4
5
6
7
8
9
10
11
12
13

NAME                                          READY   STATUS    RESTARTS   AGE
pod/local-disks-local-diskmaker-6787r         1/1     Running   0          52m
pod/local-disks-local-diskmaker-jvwnq         1/1     Running   0          52m
pod/local-disks-local-diskmaker-lfzq9         1/1     Running   0          52m
pod/local-disks-local-provisioner-fzgs2       1/1     Running   0          52m
pod/local-disks-local-provisioner-mqd86       1/1     Running   0          52m
pod/local-disks-local-provisioner-t2bvz       1/1     Running   0          52m
pod/local-storage-operator-7f8dbfb95c-7brlv   1/1     Running   0          16h

## PV
local-pv-38162728                          500Gi      RWO            Delete           Available                                                     local-sc                7m45s
local-pv-64bcf276                          500Gi      RWO            Delete           Available                                                     local-sc                7m45s
local-pv-bd2d227                           500Gi      RWO            Delete           Available 

If things are all set, we can consume the local storage provisioned by local-sc. Here I use volumeClaimTemplates instead of create separate PVC (这里应该不能使用分开的PVC，因为PVC的创建和pod位于的node有关，事先并不知道).

Notice that if there is one PV per node, then one PVC will consume the whole PV. So if use statefulset with volume claim template, we will only have one pod per node.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test-local-sc
spec:
  selector:
    matchLabels:
      app: nginx # has to match .spec.template.metadata.labels
  replicas: 3 # by default is 1
  template:
    metadata:
      labels:
        app: nginx # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 2
      securityContext: {}
      serviceAccount: wkc-iis-sa
      serviceAccountName: wkc-iis-sa
      containers:
      - name: nginx
        image: xxx.swg.com/compute-image:b994-11_7_1_1-b191
        securityContext:
          allowPrivilegeEscalation: true
          privileged: false
          readOnlyRootFilesystem: false
          runAsNonRoot: true
          runAsUser: 10032
        command: ['/bin/sh', '-c', 'tail -f /dev/null']
        volumeMounts:
        - name: my-scratch
          mountPath: /opt/xx/Scratch2
  volumeClaimTemplates:
  - metadata:
      name: my-scratch
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: local-sc
      resources:
        requests:
          storage: 5Gi

Now let’s check /dev/vdc again by lsblk, you will see it is associated with the pod.

OOD 可以看成是 High Level System Design 后的具体实现。

This blog is for Object Oriented Design, please revisit frequently to refresh. The notes are mainly from https://www.educative.io/.

OOP Design video OOP: Object-oriented programming Design pattern: singleton, factory, etc Concurrency

The four principles of object-oriented programming are encapsulation, abstraction, inheritance, and polymorphism.

The process of OO analysis and design can be described as:

1. Identifying the objects in a system;
2. Defining relationships between objects;
3. Establishing the interface of each object;
4. Making a design, which can be converted to executables using OO languages.

要清楚如何定义enum, constants abstract class, interface. 如何选择他们.

UML stands for Unified Modeling Language and is used to model the Object-Oriented Analysis of a software system. UML is a way of visualizing and documenting a software system by using a collection of diagrams.

Be familiar with:

1. use case diagram (actors， 不是指所有的object: admin, customer, system…)
2. class diagram (relationships)
3. sequence diagram (emphasize on interaction between objects, time series)
4. activity diagram (emphasize on control of flow)

Summary: 总的来看，通过class diagram，设计好基本的abstract class, interface, base class，然后再延伸，实例化。实例化的class中设计好attributes, method的实现。

• 定义好enum, constants.

• 把不同的系统分开设计，比如notifiation, payment…

• 对于actor people，有一个account class, 被包含在person abstract class中，然后这个person被实例化为其他诸如guest, admin, member等class. (有类似关系的以此类推)

• 对于有查询需求的任务, search interface中声明search方法，然后被search的对象类implements, 在这个对象内部实现search的各种method, 通常需要与database连接。

• 可能会用到设计模式

Design a Library Management System

Library management systems help libraries keep track of the books and their checkouts, as well as members’ subscriptions and profiles.

1. get clarity of the requirements, Be sure to ask questions to find the exact scope of the system that the interviewer has in mind

找到actors之后，围绕actors就可以提出具体要求，可以做什么事情: member can search book by title, name, date, category, author. book has unique id, rack number, etc. member can checkout, reserve book copies. days a member can keep the book. numbers a member can checkout. collect fine if after due date. system can send notification to user. …

最后确认需具体要实现什么的功能？

2. use case diagram, define top use cases 从object角度, 看看各自可以有什么操作, 对具体的实现功能，画出具体的use case diagram。

3. class diagram 有几个问题: 要从high level考虑有哪些abstract class, enum, constant, interface等。 然后各自的依赖关系画一下。

interface vs inheritace，什么时候谁合适?

在实现代码的时候，class diagram可以作为指导原则

4. activity diagram 画出具体实现某一功能的状态图

5. code Since you are not required to write a fully executable code in an interview, you can assume parts of the code to interact with the database, payment system, etc…

主要写出基本的部分: Enums and Constants:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

// enums
public enum BookStatus
{
  AVAILABLE,
  RESERVED,
  LOANED,
  LOST
}

public enum ReservationStatus
{
  WAITING,
  PENDING,
  CANCELED,
  NONE
}
// constants
public class Constants {
  public static final int MAX_BOOKS_ISSUED_TO_A_USER = 5;
  public static final int MAX_LENDING_DAYS = 10;
}

然后是abstrace class, interface 和其他继承，实现的类，根据class diagram去实现，举几个例子:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

// For simplicity, we are not defining getter and setter functions. The reader can
// assume that all class attributes are private and accessed through their respective
// public getter methods and modified only through their public methods function.
public abstract class Book {
  private String ISBN;
  private String title;
  private String subject;
  private String publisher;
  private String language;
  private int numberOfPages;
  private List<Author> authors;
}

public class BookItem extends Book {
  private String barcode;
  private boolean isReferenceOnly;
  private Date borrowed;
  private Date dueDate;
  private double price;
  private BookFormat format;
  private BookStatus status;
  private Date dateOfPurchase;
  private Date publicationDate;
  private Rack placedAt;

  public boolean checkout(String memberId) {
    if(bookItem.getIsReferenceOnly()) {
      ShowError("This book is Reference only and can't be issued");
      return false;
    }
    if(!BookLending.lendBook(this.getBarCode(), memberId)){
      return false;
    }
    this.updateBookItemStatus(BookStatus.LOANED);
    return true;
  }
}

public class Rack {
  private int number;
  private String locationIdentifier;
}

// interface
public interface Search 
{
  public List<Book> searchByTitle(String title);
  public List<Book> searchByAuthor(String author);
  public List<Book> searchBySubject(String subject);
  public List<Book> searchByPubDate(Date publishDate);
}

public class Catalog implements Search 
{
  private HashMap<String, List<Book>> bookTitles;
  private HashMap<String, List<Book>> bookAuthors;
  private HashMap<String, List<Book>> bookSubjects;
  private HashMap<String, List<Book>> bookPublicationDates;

  public List<Book> searchByTitle(String query) {
    // return all books containing the string query in their title.
    return bookTitles.get(query);
  }

  public List<Book> searchByAuthor(String query) {
    // return all books containing the string query in their author's name.
    return bookAuthors.get(query);
  }
}

Design parking lot

https://www.youtube.com/watch?v=DSGsa0pu8-k&t=814s 评论里有一些批评意见，可以参考。

• Identify the problem scope: do you want me to come up a system design, or class hierarchy? or should we get into certain specific question and wirte some methods?

• How you approach the problem: show a clear and systematic approach how you tackle this problem how is the parking lot designed? building? open space? free or pay? how many spots are we talking about? floors? are there mutliple entrances or exits? should we first fill out floor from top? prices strategy? premium, general customer?

进口的识别系统会识别plate number + vehicle type，然后再ticket上标明应该停放到哪个building，哪一层的哪个位置。(如果有多个entrance, 则有concurrency问题需要解决)

code some part of the design? which part do you want me to implement? We have database backend but here for Simplicity can we assume we store the data in memory?

算法去实现spot -> vehicle的分配。

1. system requirements multi-floors multi-entries or exits parking ticket pay cash or credit card pay at info panel or at exit display message when full different parking spots for different type car electric car spots, charge station support step price hourly

2. use case diagram get clarity what functions we need to implement.

3. class diagram ParkingLot ParkingFloor ParkingSpot Account: admin and parking attendant ParkingTicket Vehicle: many types Payment: credit card or cash ParkingRate ParkingDisplayBoard ParkingAttendantPortal CustomerInfoPortal ElectricPanel

4. code see: https://www.educative.io/courses/grokking-the-object-oriented-design-interview/gxM3gRxmr8Z

Design Amazon online shopping system

1. get clarity

• what scope do you want me to focus on?
• high level design or dive into a specific function component, wirte class and method.

Objects: guest, member, admin and system 分别讨论一下各自可以做什么操作。

几个主要的system: register, search, review, order, shipment, payment, notification.

2. use case diagram
3. activity diagram
4. sequence diagram

Design shopify

shopify eCommerce platform 类似于淘宝, 有自己的网店portal.

Design stack overflow

Stack Overflow is one of the largest online communities for developers to learn and share their knowledge.

Users of Stack Overflow can earn reputation points and badges. For example, a person is awarded ten reputation points for receiving an “up” vote on an answer and five points for the “up” vote of a question. The can also receive badges for their valued contributions. A higher reputation lets users unlock new privileges like the ability to vote, comment on, and even edit other people’s posts.

Actors: admin, guest, member, system, moderator(close,delete,undelete question)

Design a Movie Ticket Booking System

1. high level overview movie theater -> halls -> moive -> shows search movie select show and boot ticket select seat notification payment concurrency issue when booking: 在数据库层面实现的，没有使用Java的concurrency package We can use transactions in SQL databases to avoid any clashes. lock the rows before we update them discount/coupon apply

2. use case diagram

3. class diagram

4. activity diagram

Design ATM

Automated teller machine (ATM) withdraw and deposit money

components of ATM: card reader kaypad screen Cash dispenser Deposit slot printer network

checking and saving account of user 都放在transaction中，保证原子性: Balance inquiry Deposit cash Deposit check Withdraw cash Transfer funds

The ATM will maintain an internal log of transactions that contains information about hardware failures; this log will be used by the ATM operator to resolve any issues.

actors: operator customer bank manager system

Design an Airline Management System

This system involves the scheduling of flights, air ticket reservations, flight cancellations, customer support, and staff management. Daily flights updates can also be retrieved by using the system.

roundtrip one way mutli-city

回忆一下预定的界面 departing returning

Design a Hotel Management System

和ticket booking类似 a online portal, keep track of available rooms, book rooms and generate bill. booking of different room types like standard, deluxe, family suite, etc housekeeping log to keep track of all housekeeping tasks

1. use case diagram guest manager system housekeeper Receptionist

2. class diagram

3. activity diagram

Restaurant Management system

同上，补充几个特点: The system allows the manager to keep track of available tables in the system as well as the reservation of tables and bill generation. 这里是西餐的形式，没人一个座位，单独点餐。

menu -> menu sections -> items rooms -> tables (reservation or walk-in)

Design Facebook

Facebook is an online social networking service where users can connect with other users to post and read messages. Users access Facebook through their website interface or mobile apps.

Design LinkedIn

Similar to Facebook design but carter to professionals. 各种功能几乎就一样。

A LinkedIn member’s profile page, which emphasizes their skills, employment history, and education, has professional network news feeds with customizable modules.

1. use case diagram member admin system

2. class diagram

3. activity diagram

这些都是我工作至今或多或少用到过的，单纯地陈列这些命令没什么用，关键是要在合适的场景选择合适的工具去解决问题，这里就是给自己提个醒儿，也是出于好奇统计一下自己的知识储备。

a

awk (data processing and extracting), alias, at(schedule job one time), arp(check layer2 to layer3 mapping), ab(apach HTTP server benchmark tool), apropos, aspell, ack(grep like search for source code)

b

base64(encode/decode), bg, basename(last name), blkid, bzip2, bc

c

cat, cd, cp, cut ,curl ,chown ,chmod ,chgrp ,cron ,clear, cal(calendar), comm

d

df(disk space check), dirname(path prefix), du(occupied space check), diff, date, dd(convert and copy), dnsdomainname, dig(dns lookup utility), disown(remove job from current job list)

e

echo, exit, export, env, exportfs, ethtool(physical card), eval, expand(tab to spaces)

f

file, free, find, fg, firewall-cmd, fallocate(fast allocate space), fuser, fsck, fold, fmt

g

gzip, grep, git, gitk, getfacl, getent(look up /etc/hosts)

h

hostname, host, htpasswd, history, hostnamectl(permenant hostname), hdparm d

i

ip(so powerful), ifconfig(obsolete), id, iperf3, iptables, iostat, ifdown, ifup, install(copy files and set attributes)

j

jq, jobs, journalctl, join

k

kill

l

ls, less, ln, lsblk, lvdisplay, lscpu, lastlog, last(reboot), losetup, lsof(list open files), loginctl, lvm, lvreate, lvextend, lvresize, lvreduce, lvemove, lvdisplay, locate

m

mv, mount, more, man, mkdir, mktemp, mdadm, make, mkfifo(named pipe), mpstat

n

nc, netstat, nslookup, nmap, nice(process priority), nmcli, nohup, nl, numfmt(unit convert)

o

openssl

p

pwd, ping, ps, perf, pvdisplay, pmap, pwdx, pv(throttle the progress of data through a pipe), pvcreate, pvscan, paste, patch, printf, pidstat

q

so far no!

r

rm, rmdir, route, rsync, readlink, runlevel, renice, rpm, rev(reverse seq), reset

s

ssh, scp, sftp, strace, sudo, su, sed(stream editor), setfacl, sleep, stat, systemctl, shutdown, sar(system activity report), ss(similar to netstat), stress, stress-ng, sort, seq, sync, sshfs, screen, script, set, split, strings(print printable chars in files)

t

tar, tcpdump(wireshark), tcpreplay, top(powerful!), trap, touch, tee, tail, tree, tracepath, traceroute, tload, tc(traffic control), tac(reversion of cat), truncate, tr(translate), timeout, tmux

u

uniq, uname, umount, unlink, uuidgen, uptime, udevadm, unalias

v

vim, vmstat, vgcreate, vgextend, vgreduce, vgremove, vgdisplay

w

w, who, wget, wc, wall, write, watch, wipefs

x

xargs

y

yum, yes

z

zip

What should be the layout for a good bug ticket:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
and verify the issue you are about to submit is not a duplicate.
--->

## Environment information
(cluster info, system info)

## Problem Description
(Summarize the bug encountered concisely)

## Steps to reproduce
(How one can reproduce the issue - this is very important)

## Expected behaviour
(what you should see?)

## Observed behaviour
(what actually happens?)

## Additional info or screenshots
(Paste any relevant logs - please use code blocks (```) to format console output,
logs, and code as it is very hard to read otherwise.)

## Workaround available
(If you can, link to the line of code that might be responsible for the problem)

## Per Meeting Minutes

## TO-DO's

More:

1. add right pipelines
2. add right labels
3. @people when reply

OpenShift version: 4.3

Create Internal Image Registry Route

From the Infra Node, run the following commands. This will create a accessable path for you to push image to internal image registry.

1
2
3
4

oc project openshift-image-registry
oc patch configs.imageregistry.operator.openshift.io/cluster --type merge -p '{"spec":{"defaultRoute":true}}'
## we need this output when tag and push image
CLUSTER_IMAGE_REGISTRY_ROUTE=$(oc get route)

Pull, Tag and Push Image

Here we use podman:

1
2
3
4
5
6
7
8
9
10
11
12
13

## pull original from other registry
## or use podman to load image archive
podman login -u <user> -p <password> docker.io
podman load -i <image>.tar.gz

podman pull <path>/<image>:<tag>

export PRIVATE_REGISTRY=${CLUSTER_IMAGE_REGISTRY_ROUTE}/<project>
## kubeadmin is the default cluster admin
podman login -u kubeadmin -p $(oc whoami -t)  $PRIVATE_REGISTRY --tls-verify=false

podman tag  <path>/<image>:<tag> $PRIVATE_REGISTRY/<image>:<tag>
podman push $PRIVATE_REGISTRY/<image>:<tag>

Create Role and Binding

You need to get authenicated when pull image from cluster image registry, here we create a dedicated service account under the target project, then grant privileges to this service account and specify it to yaml file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

oc apply -f - << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: <service account name>
  namespace: <projetc>
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: <cluster role name>
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: <couster role binding name>
subjects:
  - kind: ServiceAccount
    name: <service account name>
    namespace: <project>
roleRef:
  kind: ClusterRole
  name: <cluster role name>
  apiGroup: rbac.authorization.k8s.io

Example pod yaml file:

1
2
3
4
5
6
7
8
9
10
11

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec: 
  ## specify the service accout
  serviceAccountName: <service account name>
  containers:
  - name: test-cotainer
    image: image-registry.openshift-image-registry.svc:5000/<project>/<image>:<tag>
    command: ['sh', '-c', 'tail -f /dev/null']

Note that the default cluster registry path is image-registry.openshift-image-registry.svc:5000, consist of <svc name>.<project>.svc:<port>. don’t use that route path.

Podman is a pod manager tool, a daemonless container engine for developing, managing, and running OCI Containers on your Linux System. Containers can either be run as root or in rootless mode. Simply put: alias docker=podman.

实际使用中发现podman commit命令和docker格式有不同，且commit后的image使用上有不正常的地方，比如HOSTNAME不见了。

Container tool guide, it shows the difference between buildha and podman:

Both Buildah and Podman are command line tools that work on OCI images and containers. The two projects differentiate in their specialization.

Buildah specializes in building OCI images. Buildah’s commands replicate all of the commands that are found in a Dockerfile. Buildah’s goal is also to provide a lower level coreutils interface to build images, allowing people to build containers without requiring a Dockerfile. The intent with Buildah is to allow other scripting languages to build container images, without requiring a daemon.

Podman specializes in all of the commands and functions that help you to maintain and modify OCI images, such as pulling and tagging. It also allows you to create, run, and maintain containers created from those images.

A major difference between Podman and Buildah is their concept of a container. Podman allows users to create “traditional containers” where the intent of these containers is to be long lived. While Buildah containers are really just created to allow content to be added back to the container image. An easy way to think of it is the buildah run command emulates the RUN command in a Dockerfile while the podman run command emulates the docker run command in functionality. Because of this you cannot see Podman containers from within Buildah or vice versa.

so buildah mainly is used to build images (to build images you need to run containers before commit updates), podman is used to run container in production environment.

In short Buildah is an efficient way to create OCI images while Podman allows you to manage and maintain those images and containers in a production environment using familiar container cli commands.

Some commands overlaps between the projecs

Let’s see some example when I was working on deploy ds assembly on portworx:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

## install podman in redhat/centos
yum install podman -y

## the same as docker login
podman login -u <user name> -p <password> docker.io

## the same as docker pull/tag/push
podman pull k8s.gcr.io/pause:3.1
podman tag k8s.gcr.io/pause:3.1 <regisrty path>/pause:3.1

## --tls-verify=false 
## disable HTTPS and verify certificates when contacting registry
## you may also need is when login
podman push <registry path>/pause:3.1 --tls-verify=false

OpenShift version: 4.3

OpenShift current is evolved to version 4.3 (last time when I was working on it, it was version 3.11), I am assigned to try non-root install for DS assembly (a plugin) in CP4D cluster. what non-root means

1
2
3
4

No cluster admin privileges
No root processes in containers
No host access or ssh requirements
No elevated SCCs (other than the cpd defaults) 

We have met last 3 requirements, so focus on first one.

After doing research, the steps are clear but not that straightforward (相比3.11目前版本的配置变化还挺大的，支持的内容更丰富了)

1. Create regular user
2. Specify identity provider for OAuth
3. Bind necessary cluster role or local role
4. Run installation

4.3版本的一大变化是kubeadmin是默认的cluster-admin user，如同之前的systemadmin, kubeadmin is treated as the root user for the cluster. The password is dynamically generated and unique to your OpenShift Container Platform environment.

1

oc login -u kubeadmin -p IVfPS-FvJZI-Vagzw-nIpVA --server=https://api.dsocp43.os.fyre.ibm.com:6443

password is provided in output when install is done，记下来就行，之前3.11 systemadmin是不需要password login的。

Create user and specify identity provider

Understanding identity provider configuration。 By default, only a kubeadmin user exists on your cluster. To specify an identity provider, you must create a Custom Resource (CR) that describes that identity provider and add it to the cluster.

这里我选择htpasswd当作identity provider, Configuring an HTPasswd identity provider

1
2

## htpasswd -c -B -b </path/to/users.htpasswd> <user_name> <password>
htpasswd -c -B -b ~/.htpasswd demo demo

Then create htpasswd secret:

1

oc create secret generic htpass-secret --from-file=htpasswd=~/.htpasswd -n openshift-config

Create Custom Resource and add it to cluster

1
2
3
4
5
6
7
8
9
10
11
12

apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
  name: cluster
spec:
  identityProviders:
  - name: my_htpasswd_provider 
    mappingMethod: claim 
    type: HTPasswd
    htpasswd:
      fileData:
        name: htpass-secret 

1

oc apply -f <yaml file>

Verify:

1
2

oc login -u demo -p demo
oc whoami

Bind cluster role or local role

Using RBAC to define and apply permissions. Cluster administrators can use the cluster roles and bindings to control who has various access levels to the OpenShift Container Platform platform itself and all projects.

Use kubeadmin to create custom cluster role or local role and bind 目前为止demo user只能对自己创建的project都基本的project admin权限，如果要想操作其他project的内容，可以local bind一个project admin:

1

oc adm policy add-role-to-user admin demo -n <target project>

如果需要access其他resource，比如：

1
2
3

[ERROR] [2020-02-19 15:19:21-0617] Error verifying current oauth token - Error from server (Forbidden): 
oauthaccesstokens.oauth.openshift.io "XKRLOq8286U3YnRbf6lsv99Uk2rD1A6wanNVxgp5NNs" is forbidden:
User "demo" cannot get resource "oauthaccesstokens" in API group "oauth.openshift.io" at the cluster scope

这里提示user demo cannot get resource oauthaccesstokens at the cluster scope, 我们可以先根据这个resource创建一个cluster role，然后bind it to user demo. 创建cluster role时可以指定操作，verb有get, list, create, delete, patch, watch, deletecollection。然后把创建好的cluster role 用cluster role binding 绑定到demo上:

1
2
3
4
5

## new custom custerrole oauthaccesstokens
oc create clusterrole oauthaccesstoken_custom \
  --verb=get,list,create,delete,patch,watch \
  --resource=oauthaccesstokens
oc adm policy add-cluster-role-to-user oauthaccesstoken_custom demo

Check OpenShift Web can see what exactly bindings are there for user demo，这样就很方便了。

For updating components secret or configmap, you can do complete replace:

1
2
3
4
5
6

kubectl get secret docker-registry-auth -o yaml -n default \
        | sed -e "/htpasswd/c\  htpasswd: ${AUTH_BASE64}" \
        | kubectl replace -f -
kubectl get configmap repl-pod-map -n $NAMESPACE -o yaml 
        | sed -e "/${POD_NAME}/d" 
        | kubectl replace -f-

For patching pod controller like deployment and statefulset, see: https://kubernetes.io/docs/tasks/run-application/update-api-object-kubectl-patch/

Strategic merge patch

For example, add one more containers in the pod, give the yaml file patch-file.yaml:

1
2
3
4
5
6

spec:
  template:
    spec:
      containers:
      - name: patch-demo-ctr-2
        image: redis

1
2

# this command use default patch strategy for containers field: merge
kubectl patch deployment patch-demo --patch "$(cat patch-file.yaml)"

This will get merged not replaced (the original container is kept), but not all fields have merge strategy, some use default strategy that is replace, for example: tolerations field.

How to know the patch strategy of each field? see API document It seems not all fields have specified patch strategy, for example, I don’t see that on configmap.

Json merge patch

A strategic merge patch is different from a JSON merge patch. With a JSON merge patch, if you want to update a list, you have to specify the entire new list. And the new list completely replaces the existing list.

The kubectl patch command has a --type parameter that you can set to one of these values:

1. json (json patch)
2. merge (json merge patch)
3. strategic (default)

这里文档有问题，如果用json merge patch on configmap，其实还是做的merge操作(之前的数据被保留)，并不是replace.这个值得注意，比如以下2个commands效果是一样的:

1
2
3
4

## json merge patch
kubectl patch configmap repl-pod-map -n pines --type merge -p "{\"data\": {\"test1\":\"test1key\"}}"
## defaul strategic patch
kubectl patch configmap repl-pod-map -n pines  -p "{\"data\": {\"test1\":\"test1key\"}}"

我猜可能是configmap没有明确的patch strategy定义，但对于其他有明确定义的field，则json merge patch会replace之前的数据。

//TODO [ ] ip vs iptables 各自用在什么地方? my question: https://unix.stackexchange.com/questions/608560/what-is-the-difference-between-iptables-and-ip-route-table

ip command in Linux is present in the iproute package which is used for performing several network administration tasks. IP stands for Internet Protocol.

ifconfig is obsolete, ip is preferred.

It can perform several tasks like configuring and modifying the default and static routing, setting up tunnel over IP, listing IP addresses and property information, modifying the status of the interface, assigning, deleting and setting up IP addresses and routes.

Explanations: Linux ip Command with Examples Linux manual page: ip

Most frequently used subcommands:

• link (l) - Display and modify network interfaces.
• address (a) - Display and modify IP Addresses.
• route ® - Display and alter the routing table.
• neigh (n) - Display and manipulate neighbor objects (ARP table).
• netns - deal with network namespace

The configurations set with the ip command are not persistent. After a system restart, all changes are lost. For permanent settings, you need to edit the distro-specific configuration files or add the commands to a startup script.

address

Show all IP address associated with all interfaces that are available

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

## ip address
ip addr
ip a
## only show ipv4
ip -4 a
## show specified device
ip addr show eth0
## abbr
ip a s eth0
## assign an IP address to an interface. (need append netmask)
## can assign multiple ip address to an interface
## not persistent
ip addr add 192.168.1.50/24 dev eth0
## delete an assigned IP address to an interface. (need append netmask)
ip addr del 192.168.1.50/24 dev eth0

link

It is used to display link layer information, like MAC address, it will fetch characteristics of the link layer devices currently available. Any networking device which has a driver loaded can be classified as an available device.

1
2
3
4
5
6
7
8
9

## show statistic of device with human readable format
## 显示有多少error, drop packets来看是不是网络由问题
## double -s will show error details
ip -s -s -h link
ip -s -h link show eth0
## bring up
ip link set eth0 up
## bring down
ip link set eth0 down

route

This command helps you to see the route packets your network will take as set in your kernel routing table. The first entry is the default route. Other commands perform the same: route, netstat -r

1
2
3
4
5
6
7

ip route
## route for a specific network
ip route list 9.30.204.0/22
## add a route to 192.168.121.0/24 via the gateway at 192.168.121.1
ip route add 192.168.121.0/24 via 192.168.121.1
## add a route to 192.168.121.0/24 that can be reached on device eth0.
ip route add 192.168.121.0/24 dev eth0

neigh

netns

$$ is a special bash variable (special parameter $ with a preceding expansion mark $) that gets expanded to the pid of the shell.

/proc/self is a real symbolic link to the /proc/ subdirectory of the process that is currently making the call.

When you do ls /proc/$$ the shell expands it to ls /proc/pid-of-bash and that is what you see, the contents of the shell process.

But when you do ls /proc/self you see the contents of the short lived ls process. If you write code which uses /proc/self that code will see its own pid, namely the process making the system call with /proc/self as part of the pathname in one of its arguments.

The $$ is not limited to this usage, you can write echo $$ to see the bash pid; you can use it to kill yourself, etc.