There is a better alternative, please see <<VSC Developing inside a Container>>.

To use and manage third-party libraries without messing up python environment, organizing different project that has its own unique dependencies:

• pip: package management
• virtualenv: project and dependencies
• virtualenvwrapper: making venv more convenient

Does not talk the package that is the form of __init__.py under a folder, we are talking python distribution package.

Pip

Best practice:

1. always work inside a virtual environment, keep things nice and clean.
2. be careful using pip with sudo when install packages, otherwise the installation is system-wide.

Mac’s pre-installed Python is not meant for development, you can use Homebrew to install or download Python from python.org, that will go along with pip. For Linux, adhere to system manager to install pip or python. In Mac, try check if pip is there and it’s version.

To install pip(2/3) on Linux:

1
2
3
4
5
6
7
8
9
10
11

# search pip2 or pip3 package
sudo yum search python* | grep pip
# this will also install libtirpc
# python3, python3-libs, and python3-setuptools
sudo yum install python3-pip

pip3 -V
pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6)

pip2 -V
pip 9.0.3 from /usr/lib/python2.7/site-packages (python 2.7)

pip commonly use commands:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# local or global config info
# you will see the package repo
pip3 config [debug, edit, get, list, set, unset]

# search
pip3 search <package name>
# download package in current dir
pip3 download <package name>
# will auto install other dependencies
pip3 install <package name>

# list packages installed
pip3 list
# show outdate packages
pip3 list -o

# uninstall
# will not uninstall its dependencies
pip3 uninstall <package name>

# show package info
# you will see the location where the package is installed
# and its source code url
pip3 show <package name>

# seek help
pip3 help

pip is actually fetching packages from Python package index (or your own package repo) https://pypi.org/

How to work:

1. search key work directly.
2. go to Browse projects -> Operating system -> Linux, then select other classifier (but this is still hard to search what is exactly needed).
3. check development status, select package in production/stable version.

Pip install from specified repo:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# use additional repo
pip install --extra-index-url '<repo url>' vault-client==0.0.4

# or set by
# pip config set <ket> <value>
global.extra-index-url='<repo url>'
global.timeout='10'
global.trusted-host='registry.corp.xxx.com'
# then run
pip install vault-client==0.0.4

# 或者创建一个~/.pip/pip.conf 文件
[global]
timeout=10
trusted-host = egistry.corp.xxx.com
extra-index-url = <repo url>
# 然后
pip install --no-cache-dir vault-client==0.0.4

Virtual Environment

Combining with virtualenvwrapper is good, recommended.

Install virtualenv:

1
2
3
4
5
6

# install system-widely
# preferred way
# -m: run module
sudo python3 -m pip install virtualenv
# or
sudo pip3 install virtualenv

Create virtualenv:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

mkdir ~/virtualenvs && cd ~/virtualenvs

# create a virtual env called rates in python3
virtualenv -p python3 rates_py3
virtualenv -p python3.8.4 rates_py3
# python2 based
virtualenv -p python2 rates_py2

# activate
cd rates_py3
# after this you will see a prefix 
# 一旦激活，不管在其他任何地方，都是这个环境！
. ./bin/activate
# check
python -V
pip -V
# you will only see less packages installed
pip list

# then start your work in your project folder...

# deactivate
deactivate

Other similar tool, this venv may pre-installed or need to pip install globally:

1
2

# python >= 3.3, may popular in furture
python3 -m venv <virtual env name>

Syncing packages with colleagues, put this requirement file in version control to share and update:

1
2
3
4
5
6
7
8
9

# fist activate the virtual environment
# package list
python -m pip freeze > requirements.txt
# the condition can be ==, !=, >=, <=

# create another virtual environment with the same python verion like yours
# activeate this new environment
# then run
python -m pip install -r requirements.txt

You can specify version in pip install:

1
2
3
4
5
6
7
8

pip install flask==1.0.0
pip install 'Django<2.0'
# upgrade to latest version
pip install -U flask

# upgrade pip
# take care not to overwrite system pip using sudo
pip install -U pip

How to manage the project and virtual environment? Separating project with virtual environment! 放在不同的文件夹中，使用时激活就行了，一般一个venv对应一个project, 但如果要测试多个不同的环境，也可以多个venvs map to one project.

1
2
3
4
5
6
7

--dev
|  |-----my_game
|  |-----my_website
|
--virtual environment
   |-----my_game
   |-----my_website

Real-world example, when develop flask framework, use setup.py with editable pip to install packages in virtual environment, so you can edit the flask source code and it will reflect in real-time: When would the -e, --editable option be useful with pip install?

1
2
3
4
5

git clone https://github.com/pallets/flask

# activate virtual environment
# go to root level of flask directory
python -m pip install -e .

Now have developing env for flask.

You can also see tox.ini file in flask git repo, it is used for testing against different python versions in different virtual environments.

Virtualenvwrapper

A user-friently wrapper around virtualenv， easy creation and activation, bind projects to virtualenvs.

Setup:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# install system-widely
sudo python3 -m pip install virtualenvwrapper
sudo pip3 install virtualenvwrapper


# get path
which virtualenvwrapper.sh
/usr/local/bin/virtualenvwrapper.sh

# add below lines to ~/.bashrc
# point virtualenvwrapper to pyhton3 explicitly
# the path could be /usr/local/bin/python3, check the config
export python3=/usr/local/bin/python3
export VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python3
# the example path:
source /Library/Frameworks/Python.framework/Versions/3.10/bin/virtualenvwrapper.sh
# if you don't want to use default virtual env home
# use absolute path
export WORKON_HOME="/home/<user>/virtualenvs"
# set the project homes, when use mkproject will create project folder here
# use absolute path
export PROJECT_HOME="/home/<user>/dev"

Operations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# show virtual environments list
workon

# enter or switch virtual environment
workon <venv name>

# will create both venv and project
# if the project is bound with venv, use workon will auto switch to project folder
mkproject <pro name>
mkproject -p python3 <pro name>
mkproject -p python2 <pro name>

# create a venv only
mkvirtualenv <venv name>

# for old project does not bind with venv
# activate venv and go to old project folder, run below to bind them
setvirtualenvproject 

# remove a venv
# you need to manually remove project folder is you want
rmvirtualenv <venv name>

# deactivate venv
deactivate

Other Future Tools

New projects:

• pipenv: https://pipenv-fork.readthedocs.io/en/latest/
• poetry: https://python-poetry.org/

Introduction

Go users: https://github.com/golang/go/wiki/GoUsers web services, devops: docker + k8s

Visual Studio Code go-plugin, vim go-plugin

Install and set up a runnable demo project, a brief guidance can see here: Official Setup Go project

If forget, read through the tutorials one by one: https://go.dev/doc/

Other resources:

• The Go programming Language

• Effective Go 这个更深入了，特别是用法方面

• Go Blog

• Go by Example

• Other people web site

• Go语言中文网

What are go package and module is explained here: https://go.dev/doc/code A package is a collection of source files in the same directory that are compiled together. Functions, types, variables, and constants defined in one source file are visible to all other source files within the same package.

A repository contains one or more modules. A module is a collection of related Go packages that are released together. A Go repository typically contains only one module, located at the root of the repository. A file named go.mod there declares the module path: the import path prefix for all packages within the module. The module contains the packages in the directory containing its go.mod file as well as subdirectories of that directory, up to the next subdirectory containing another go.mod file (if any).

An import path is a string used to import a package. A package’s import path is its module path joined with its subdirectory within the module. For example, the module github.com/google/go-cmp contains a package in the directory cmp/. That package’s import path is github.com/google/go-cmp/cmp. Packages in the standard library do not have a module path prefix.

Example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

// A module is a collection of related Go packages that are released together
// in the first line of go.mod, you can see the module name

// main package and func is the entrypoint of app
// the file name can be others not necessary as main.go
package main

import (
  . "fmt" // call fmt funcs directly withou fmt prefix
  err "errors" // alias
  "os"
  "net/http"
  "regexp"
  "builtin" // no need to import
  _ "github.com/ziutek/mysql" // call init in the package
)

// auto append `;`, so let { at the same line
func main() {
  fmt.Println("Hello world")
}

package name in go file should be the same as folder name.

Important env variables:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

## https://maelvls.dev/go111module-everywhere/
## In 1.15, it's equivalent to auto. 
## In >= 1.16, it's equivalent to on
GO111MODULE=""

## you can export or use `go env -w`
## GOROOT is set automatically
export GOROOT=/usr/local/go
export GOBIN=$GOROOT/bin

## only need to set GOPATH
## go path is the project workspace
## it has src (user create), pkg and bin (auto create for you when build or install)
export GOPATH=$HOME/go
export PATH=$PATH:$GOBIN

## persistent set and unset
go env -w GOPATH=$HOME/go
go env GOPATH
go env -u GOPATH

Understand run, build, install, get subcommands. pluralsight has Go CLI playbook course.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

go version

## used GO >= 1.16
## has version suffix
## install binary without changing go.mod
go install sigs.k8s.io/kind@v0.9.0
go install sigs.k8s.io/kind@latest

## create a module
## link with your code repo url
## generate go.mod
go mod init example.com/example
## add missing and remove unused modules
## can edit the package version in require block
go mod tidy

# clean mod cache
go clean --modcache

## link dependency to local path
## ../greetings is a local package in relative path
## example.com/greetings is that packages module path
## see this tutorial:
## https://go.dev/doc/tutorial/call-module-code
go mod edit -replace example.com/greetings=../greetings
go mod tidy

# download dependencies for off-line go build
# see this ticket:
# https://stackoverflow.com/questions/68544611/what-is-the-purpose-of-go-mod-vendor-command
go mod vendor

## GO >=1.16, only for editing go.mod
## -x: verbose
## it will modify the go.mod
## or you edit go.mod manually
go get github.com/sirupsen/logrus@v1.8.0

## simliar to python dir()/help()
## case-insensitive
go doc fmt
go doc net/http
go doc time.Since
go doc fmt.Println
## local API server , browser your own package
godoc -http :8080

## linter, but VSC will auto does that when save go files
go fmt <package># it runs gofmt -l -w

## test
## file with _test.go suffix
go test [-v] [-run="Hello|Bye"] [-cover|-coverprofile]

## -n: dry run
go run -n main.go
## -work: print $WORK temp folder
## you will see the intermediary files created
go run -work main.go
## run main package
go run main.go
go run .
go run

## build(compile) not install
go build hello
## it will create executable in bin folder
cd $GOPATH/bin
## executable name is the same as folder name
./hello

## install dir is control by
## env GOPATH and GOBIN
go install hello

# check doc from CLI
go doc rand.Float64
go doc math.Sin
go doc strconv.Atoi

数据类型

基本数据类型: int32, float64, bool, string(没有char type) 复合数据类型: array, slice, map, function, pointer, struct, interface

&(取地址符，没有地址的算数运算), *t

变量赋值方式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

var xx = 23
var xx int = 23 
var xx float32 = 2.32

// var block
var (
  xx int = 34
  yy = 78.99
  zz = "123"
)
// 简短写法
a := "hello"

// swap
b := "world"
a, b = b, a

// 全局变量不支持简短写法
var GLOBAL int = 100

常量使用

Use all upper cases

字符串类型

1
2

str1 := "abc"
str2 := `abc`

强制类型转换

1

int()

算数运算符

关系运算符

<, >, <=, >=, ==, !=

逻辑运算符

&&, ||, !

位运算符

&, | , ^, &^(位清空), <<, >>

If-else

condition 没有括号, no ternary

1
2
3
4
5
6

// if 还可以有初值，和for loop类似了
// 也可以把初值写到外面
if err := file.Chmod(0664); err != nil {
    log.Print(err)
    return err
}

Switch

每个case 自带break。 关键字 fallthrough 只能放case最后一行, 连接执行2个case且一定执行。 case 后的数据类型必须和switch一致, case 可无序，多个condition可用逗号分开。 switch 省略条件相当于switch true(tagless switch), 也可省略后，把condition 写在case 后面，就像if-else if了 也可以有初值switch t = 100 {..}, t only use in switch block

For loop

no while loop 语法和C一样: for init; conditionl; post {} condition no bracket

1
2
3
4

for {}
for ;; {}
for key, value := range map {}
for index, value := range [slice, array, string] {}

Goto

前后跳都可以

Array

没赋值的默认值和C语言一样:

1
2
3
4
5
6
7
8
9
10
11

var a [10]int // element default is 0
var a = [10]int{}
var a = [3]float64 {1, 2, 3}
a := [3]float64 {1, 2, 3}
// 可以设置index对应的值
a := [3]float64 {0:32, 2:99}
a := [...]float64 {0:32, 2:99}
len(a)
cap(a)
// ascending sort 
sort.Ints(a)

对于数组 len(a) 长度= cap(a) 容量

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

// same address usage as C language
var arr = [3]int32{99, 100, 101}

// 数组是值传递, 改变arr2中的值不会改变arr, 函数中注意, 除非传递的是地址
// 可直接复制
arr2 := arr
// true
fmt.Println(arr2 == arr)
// type [3]int
fmt.Printf("%T\n", arr2)
// 注意在go中数组名不是地址了，要专门用取地址符号
// 这2个值一样的
fmt.Printf("%p\n", &arr)
fmt.Printf("%p\n", &arr[0])

// t1 其实是指针类型 *[3]int32
t1 := &arr
fmt.Printf("%d\n", (*t1)[0])

二维数组

1
2
3

var arr := [3][3]int{{},{},{}}
len(arr)
len(arr[0])

Slice

动态数组，大小可变，背后有一个底层数组.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

// 不写长度, slice = nil, no underlying array
// have to append to use it
var slice1 []int
var slice1 = []float64 {1, 2, 3}
// or
// make(type, len, capacity) len 必须小于或等于 cap
// make function is in builtin package
// used to create object
slice1 := make([]int, 2) // len, cap = 2
slice1 := make([]int, 0, 2) // len = 0, cap = 2
// slice1中存的就是切片的地址，不需要取地址符了
// 而数组的地址则是&arr
fmt.Printf("%p\n", slice1)

slice1 = append(slice1, 1, 2, 3, 4, 5, 6, 7)
slice2 := []int{10, 11}
// 注意这里用...，类似于python的分解操作，如果查看append的定义，其实它的末尾参数是个可变参数...type
// 当append超过容量后，生成新的切片的，地址就变了
// 自动扩容是之前的2倍
slice1 = append(slice1, slice2...)

// map, slice 是地址传递!
// 指向同一个地址
slice3 := slice1

// create slice from array
// slice1和arr1指向同一个地址，共享数据，除非slice1扩容则会新生成一个内存地址
arr1 := [10]int
// [start, end)
slice1 := arr1[:]
slice1 := arr1[0:10]
slice1 := arr1[2:4]
// 这时候slice1地址和arr1不一样了
slice1 = append(slice1, 4, 5, 6)

// deep copy, for []type
copy(dst, src)
copy(slice2[2:], slice[1:])

Map

To understand map, see this question and the comment: https://stackoverflow.com/questions/40680981/are-maps-passed-by-value-or-by-reference-in-go this is a go playground for map to help understand its behavior:

If a map isn’t a reference variable, what is it?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

// nil map, 不能直接用
var map1 map[int]float64
fmt.Println(map1 == nil)

map1 := make(map[int]string)
map1 := map[string]int{}
map1 := map[string]int{"hello": 100, "Java": 99}

map1["see"] = -34
val1 := map1["hello"]

// 如果map没有key，取出来的是value类型的"0"值
// 怎么判断呢? ok is bool, if true, then exist
val, ok := map["key"]

delete(map1["see"])
len(map1)

String

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

// UTF-8 coding
str := "hello"
str := `hello`
// string is acutally byte sequence
slice := []byte{65, 66, 67, 68}
str1 := string(slice1)
slice := []byte(str1)

substr := str[1:3]
// strings package 主要是字符串 操作函数
// strconv package 主要是字符串 和 基本数据类型之间相互转换
// go '+' operands must have same type, so use Sprintf to concat
// different type
"123" + strconv.Itoa(100)
b1, err := strconv.ParseBool("true")
str1 := strconv.FormatBool(b1)

defer

用于延迟函数或方法的执行, 用defer 控制的调用会等到它的containing/surrounding function执行完了之后再执行，并且当所有延迟的部分执行结束之后，containing function才执行return 语句。

如果有多个defer,则按照后进先出的顺序。 用法：

• 关闭连接，文件，比如defer close() 在函数的开头.
• panic, recover，defer函数执行完毕后，异常才会被抛出到上一层.

注意，延迟函数的arguments的值，在defer的时候就固定了，值也可能是地址，则复合对象的内部值可能被改变了。

1
2
3
4
5

func deferTest() {
  defer fmt.Println("1")
  defer fmt.Println("2")
  fmt.Println("3")
}

Function

函数也是一种复合数据类型，可以用%T 查看。 注意函数的参数类型和参数名是反过来写的，且返回值类型列表也写在最后。 函数名如果大写开头 表示公共函数，可被其他package调用，比如fmt.Println()，否则只能package内部访问。 参数传递没有python这么多形式。

参数传递也有值传递和引用(地址)传递. 值传递: int, float, string, bool, array, struct 引用传递: slice, map, chan

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

// (int, int) 对应了 return val1, val2的类型
// 注意这里p2是slice, p3是array
func funcName(p1 int, p2 []float, p3 [10]int, p4 map) (int, int) {
  // logic
  return val1, val2
}
// 或者只写return，返回列表里已经有了返回值名字了
func funcName(p1 int, p2 []float) (val1 int, val2 int) {
  val1 = p1 * 3
  val2 = len(p2)
  // return named values
  return
}
// 或者return 单纯用来结束函数
// 注意这里没有定义返回值列表
func funcName() {
  return
}

// 如果只有一个返回值，可以不写括号
// 多个参数类型一致，可以把类型写在后边一起
func funcName(p1, p2 int, p3 string) int {
  return val1
}

可变参数...，只能放到参数列表的最后，且只能有一个可变参数:

1
2
3
4
5
6
7
8
9

// ...type，0个到多个参数都可以，比如Println()就是
// nums 是个slice类型
func funcName(p1 int, nums ...int) {
  // pass
}
// 调用的时候注意，如果是复合类型数据需要分解
funcName(100, 1, 2, 3)
// split sign `...`
funcName(100, []int {1, 2, 3}...)

可以定义函数变量，然后赋值，其实赋值的是函数的地址:

1
2

var c func(a ...interface{}) (n int, err error)
c = fmt.Println

Anonymous functions and closures 匿名函数，没有函数名，可以赋值给变量 或 直接调用. 所以说Go是支持函数式编程的，匿名函数可以作为其他函数的参数(这个作为参数的函数就叫回调函数)，注意不是函数执行的返回值作为参数，是函数本身! 匿名函数也可以作为返回值(闭包)

1
2
3
4
5
6
7
8
9

// no function name and call it
func () {
  fmt.Println()
}()
// or
fun3 := func () {
  fmt.Println()
}
fun3()

回调函数:

1
2
3
4
5
6
7
8
9
10
11

// callback function
func add(a, b int) int {
  return a + b
}
// calling function
// here we can define func type to replace 'func(int, int)int' in argument list
func operate(a, b int, fun func(int, int)int) int {
  return fun(a, b)
}
// call
operate(1, 3, add)

闭包closure, 这里外部函数返回后，匿名函数把外部函数内部的资源i保留了下来。Python中local function一样的道理. A closure is a function value that references variables from outside its body.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

// `func() int` is treated as return type as a whole
// the increment() return a closure
func increment() func() int {
  // stay alive in closure function
  i := 0
  return func () int {
    i++
    return i
  }
}

res := increment()
// 1
fmt.Println(res())
// 2
fmt.Println(res())
// 3
fmt.Println(res())

Pointer

指针类型符号*, 和C语言一样.

1
2
3
4
5
6
7
8
9
10
11

// nil is pointer's default vaule
var p1 *int
a := 10
p1 = &a
// print pointer value
fmt.Println(p1)
fmt.Printf("%p\n", p1)

// 指针的指针
var pp1 **int
pp1 = &p1

数组指针，指向数组的指针

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

arr := [3]int {1, 2, 3}
var pa *[3]int
pa = &arr
// 这个输出注意，并不是输出的地址数值，而是一种形式
// 如果要输出地址数值，用%p
fmt.Println(pa) // &[1 2 3]

// the address of arr
fmt.Printf("%p\n", pa)
fmt.Printf("%p\n", &arr)

// 相同的表达
(*pa)[0]
pa[0] // (*pa)[0] 的简化写法
arr[0]

指针数组，元素是指针的数组

1
2
3
4
5

a := 1
b := 2
c := 3
arr := [3]*int {&a, &b, &c}
fmt.Println(arr)

For example:

1
2
3
4
5
6

// 指向 指针数组的指针
*[3]*int
// 指向 数组指针的指针
**[5]string
// 指向 指针数组指针的指针
**[3]*string

函数指针，指向函数的指针, Go中函数名就是函数的地址，如同slice, map一样:

1
2

var c func(a ...interface{}) (n int, err error)
c = fmt.Println

指针函数，返回指针值的函数:

1
2
3
4
5
6
7
8
9

// 2个地址输出是一样的
func getArr() *int[4] {
  arr := [3]int {1, 2, 3}
  fmt.Printf("%p\n", &arr)
  return &arr
}

arr := getArr()
fmt.Printf("%p\n", &arr)

Struct

Usually struct has struct tag.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

type Person struct {
  name string
  age int
  sex string
  address string
}

// 几种创建方式
// p1 is an empty struct object with default values
var p1 Person
// same as p1
// 可以拿到外面赋值， 比如p2.age = 34
p2 := Person{}

p3 := Person (name: "XXX", age: 23, sex: "female", address: "YYY")
p4 := Person {
  name: "XXX", 
  age: 23, 
  sex: "female", 
  address: "YYY"
}
// order matters
p3 := Person ("XXX", 23, "female", "YYY")

结构体是值类型的，和array一样，结构体名不是它的地址。

1
2
3
4
5
6
7
8
9

// 这是内容复制
p4 := p1

// 这是地址赋值
var pp1 *Person
pp1 = &p1
// 访问字段，以下都可以
pp1.name
(*pp1).name

make只能创建map, slice, channel等 make vs new: https://www.godesignpatterns.com/2014/04/new-vs-make.html new returns pointer:

1
2
3
4
5
6

// they are the same
var p1 Person
pp1 := &p1

pp1 := &Persion{}
pp1 := new(Persion)

new可以创建任意类型空间 和 返回任意类型的指针:

1

pint := new(int)

匿名结构体，和匿名函数类似的定义用法

1
2
3
4
5
6
7
8

// 这里定义了一个匿名结构体 同时进行赋值
s2 := struct {
  name string
  age int
}{
  name: "xxx",
  age: 18,
} 

匿名字段，字段没有名字，且字段类型不能重复:

1
2
3
4
5
6
7
8
9
10
11

type Worker struct {
  // 只能有一个string
  string
  // 只能有一个int
  int
}

w1 := Worker {"xxx", 11}
// 默认使用数据类型作为访问的名字
fmt.Println(w1.string)
fmt.Println(w1.int)

结构体嵌套:

1
2
3
4
5
6
7
8
9
10
11
12
13

type A struct {
  a string
}

type B struct {
  b string
  c A
}

s := B {b: "from B", c: A {a: "from A"}}
// access field
s.b
s.c.a

如果对嵌套结构体使用了匿名字段，则相当于融合到了当前结构体，访问的时候不用中间层了。这在OOP 中使用到了:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

type A struct {
  a string
}

type B struct {
  b string
  A
}

// 注意这里匿名字段初始化A: A{a: "from A"}
s := B{b: "from B", A: A{a: "from A"}}
// access field
s.b
// 中间层被省略了
s.a

Type

type 除了定义结构体，接口，还可以定义全新的类型 或者 别名: function type and value

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

// myint, mystr 就是全新的类型了 
type myint int
type mystr string

// 不能和原类型相互赋值！
var myint = 23
var mystr = "hello"

// myint, mystr 只是一个别名，注意和上面的区别
type myint = int
type mystr = string

// 可以和原类型相互赋值！
var a int = 10
var b myint = a

Go语言实现函数式编程的时候，如果函数复杂，可以用type 简化，这里的例子是作为返回值，也可以作为参数:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

// 把func(string, string) string 整体别名叫做myfun
type myfun func(string, string) string

// myfun 作为fun1 的返回值类型，就不用写一大堆了
func fun1() myfun {
  // 返回一个函数
  return func(a, b string) string {
    return a + b
  }
}

// 调用, res得到一个函数定义
res := fun1()
// "100100"
fmt.Println(res("100", "100"))

还需要注意的是，一个package中定义的别名，不能在其他包中为它添加method.

Error

Go中不要把错误 和 异常弄混了，这个还要单独看其他文章区分一下使用环境。

Go中错误也是一种数据类型error, 惯例是error 在函数中返回值的最后一个位置, error其实是一个接口:

1
2
3

type error interface {
    Error() string
}

使用:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

import {
  "os"
  "log"
  "fmt"
}

func main() {
  // 很多包中的函数都会有错误信息返回，特别是文件，网络操作
  f, err := os.Open("./test.txt")
  if err != nil {
    log.Fatal(err)
    // 得到具体的错误类型，输出其他字段
    // *os.PathError 是这个函数返回的错误类型
    if ins, ok := err.(*os.PathError); ok {
      // 错误中的字段
      fmt.Println(ins.Op)
      fmt.Println(ins.Path)
      fmt.Println(ins.Err)
      // 如果错误实现了自己的方法，调用方法也行
    }
  }
  // pass
}

自己如何创建错误呢, 通过errors.New这个package提供的函数 或者fmt.Errorf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

import {
  "errors"
  "fmt"
}

func main() {
  err1 := errors.New("my error")
  fmt.Println(err1)
  // 是 *errors.errorString 类型
  fmt.Printf("%T\n", err1)

  err2 := fmt.Errorf("my error is %d", 100)
  fmt.Println(err2)
  // 是 *errors.errorString 类型
  fmt.Printf("%T\n", err2)
}

以上只是简单的例子，复杂的error 是通过结构体实现的，里面包含了错误的具体信息，然后实现了error 这个接口.

Panic and Recover

和defer 一起使用较多，遇到panic，函数后续会停止执行，然后逆序执行所有已经遇到的当前层的defer函数，没遇到的不会执行，因为中断了，最后往上层抛出异常， defer 函数中可能包含recover 来恢复panic。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

func A() {
  fmt.Println("from A")
}

func B() {
  fmt.Println("from B")
  defer fmt.Println("from B defer 1")
  for i := 0; i <= 10; i++ {
    fmt.Println("i = ", i)
    if i == 5 {
      // panic throw
      panic("panic happened")
    }
  }
  // will not execute this defer
  defer fmt.Println("from B defer 2")
}

func main() {
  // 捕捉恢复了panic
  defer func() {
    if msg := recover(); msg != nil {
      fmt.Println("from recover")
    }
  }

  A()
  defer fmt.Println("from main defer 1")
  B()
  // 这后面不会被执行了
  defer fmt.Println("from main defer 2")
  fmt.Println("from main end")
}

哪些场景适合使用panic呢?

• 空指针
• 下标越界
• 除数为0
• 分支没有出现
• 错误输入

Why Python scripting? Easy to learn and write, interactively, powerful built-in data types and object oriented. Included packages to support huge range of tasks.

The demos of this blog: https://github.com/chengdol/python-scripting

Common Modules for Scripting

• math: Trig functions, sqrt, logarithms, exponentials
• pickle: Serialise / de-serialize objects for persistent storage
• random: Generate random numbers with various distributions
• re: Regular expression pattern matching and substitution
• string: Comprehensive string formatting
• configparser: Configuration file parser
• bz2: gzip, read and write compressed files
• tarfile: Read and write tar archives
• datetime: Represent and manipulate dates and times
• logging: Log message generator with various backends
• argparse: Parser for command-line options
• optparse: Parse command-line arguments
• click: command-line argument toolkit, decorator
• os: Interface to operating system services
• sys: Access argv, stdin, stdout, etc.
• socket: Python binding for the traditional BSD socket API
• http: Modules for client and server side http, and cookies
• shutil: Copy / remove files and directory trees
• glob: Shell-style wildcard expansion
• xml: Processing of XML data
• hashlib: common interface to many hash functions
• signal: single handling
• subprocess: execute command by spawn new processes, connect to their input/output/error pipes, and obtain their return codes
• shlex: parsing unix shell commands, for example, split long arguments
• smtplib: email handling
• threading: threading operations
• timeit: measure executing time
• pyyaml: parse yaml file
• requests: simple http library
• retrying: retrying flaky function
• python-terraform: terraform wrapper

Work Environment

REPL

REPL: the interactive console. 这是最基本的一个python interactive shell, can be used for testing purpose.

IPython

ipython: the enhanced python interpreter, can run shell commands + REPL, make alias for arbitrary shell commands and with TAB completion.

How to install: yum install -y ipython3 Then in terminal, run ipython3

可以直接在ipython中运行比如ls -ltr, cd 之类的命令，这些都属于magic function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

## off magic
automagic
## then you need % prefix to run shell commands
%ls -ltr

## open magic again
%automagic
## no need % prefix
ls -ltr

## helper
ls?

## list env variables
whos

Create alias:

1
2
3

## create alias 'findsymlinks'
alias findsymlinks ls -l /etc/ | grep '^l'
%findsymlinks

Run and edit files in IPython:

1
2
3

run script.py
## default using vi or vim, $EDITOR
edit script.py

对于不能直接运行的shell commads, use shell escape with prefix !, similar to vim feature:

1
2
3
4

## can store the output to a variable
out = !df
for line in out.grep("tmpfs"):
    print(line)

IDLE

How to install: yum install -y idle3 To run idle3, you need desktop environment.

Managing File System

找能实现Bash中功能的函数就行。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

## walk around file system, wrap around Linux system call
import os

os.getcwd()
## cd /tmp
os.chdir("/tmp")
## current level dir list
os.listdir("/")
## recursively to subdir, will return a tuple
## (current dir name, child dirs, child files)
os.walk("/tmp")
os.stat("/etc/hosts")

os.mkdir(path [, mode])
os.rename(src, dst)
os.remove(file)
## remove empty dir
os.rmdir(dir)

## 0o644, prefixed with '0o'
os.chown(file, mode)
os.chmod(file, uid, gid)
## hardlink
os.link(src, dst)
## softlink
os.symlink(src, dst)

## high-level file operations
import shutil as st
st.copy(src, dst)
## also copy attr
st.copy2(src, dst)
## mv
st.move(src, dst)
## rm -f
st.rmtree(path)
st.copytree(src, dst, ignore=None)
## which
st.which("java")

st.make_archive(basename, format)
st.unpack_archive(filename, extrac_dir, format)

Interacting with Linux System

1
2
3
4
5
6

import sys

## print command line parameters
for arg in sys.argv[:]:
    print(f"{arg}", end = ' ')
print()

To parse parameters, use optparse module, see example in git repo 1_optparse-demo.py. Besides optparse and argparse from the standard library, click module is a good alternative.

To get env varaible:

1
2
3
4
5
6

import os

## set default value when empty response
os.getenv("EDITOR", "/usr/bin/vim")
os.getenv("HOME")
os.environ.get("HOME")

这节的git repo例子很有意思5_signal-primes-v5.py, 6_timeout.py, 用signal handler 去改变条件变量的值，从而改变运行逻辑。之前一直在BASH中用trap去做终止前的处理。Linux has 2 signals set aside for user: SIGUSR1, SIGUSR2.

Executing Commands

Run external commands, for example, call other shell commands or executables by subprocess, similar to linux ().

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

import subprocess
import os

## 这是最简单的调用
## this will not disturb current env
env = os.environ.copy()
## the run method will call Popen under the nood
cmd = ["ls", "-ltr"]
## if does not set stdout/stderr, then command print result to console
process = subprocess.run(cmd, 
               env=env, 
               stdout=subprocess.PIPE, 
               stderr=subprocess.PIPE)


process.returncode
process.stdout.decode("utf-8")
process.stderr.decode("utf-8")


## 如果要运行和shell有关的命名且需要用到shell的特性 such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user’s home directory.
## If shell is True, it is recommended to pass args as a string rather than as a sequence.
env = os.environ.copy()
cmd = "ls -ltr ~ | grep -i download"
process = subprocess.run(cmd, 
               shell=True,
               env=env, 
               stdout=subprocess.PIPE, 
               stderr=subprocess.PIPE)

## the same as
cmd = ["/bin/sh", "-c", "--", "ls -ltr ~ | grep -i download"]
process = subprocess.run(cmd,
               env=env, 
               stdout=subprocess.PIPE, 
               stderr=subprocess.PIPE)

process.returncode
process.stdout.decode("utf-8")
process.stderr.decode("utf-8")



## 高级操作 Popen, run 在背后调用的它
env = os.environ.copy()
## the same as export KUBECONFIG=clusters.yaml
env['KUBECONFIG'] = "clusters.yaml"
## this kubectl will refer KUBECONFIG env variable above
cmd = ["kubectl", "get", "sts"]
process = subprocess.Popen(cmd, 
                 env=env, 
                 stdout=subprocess.PIPE, 
                 stderr=subprocess.PIPE)

## out, err is byte type
out, err = process.communicate()
## conver to string
out = out.decode("utf-8")
err = err.decode("utf-8")

if process.returncode != 0:
  raise subprocess.CalledProcessError(process.returncode, cmd)

关于python concurrency专门的总结: Python Concurrency

String Manipulation

Besides basic operation, it talks about datetime:

1
2
3
4
5
6
7
8
9
10
11
12

import datetime

## depends on your local time zone
rightnow = datetime.datetime.now()
utc_rightnow = datetime.datetime.utcnow()

rightnow.month
rightnow.hour

## lots of % format
rightnow.strftime("Today is %A")
## datetime.timedelta, to add and decrease time slot

然后讲了re module. 可以参考这节的git code.

Processing Text, Logging

For long running background service, logging is a must, we can log events into: file, syslog or systemd journal.

Logger has different handlers, for example: StreamHandler(stdout, stderr), FileHanlder, watchFileHandler, SysLogHandler, SockerHandler, JournalHandler, etc.

Logging levels: NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL.

未来可见的一段时期要switch 到 Python了。 这里主要记录一下用Python 刷题的总结，特别是一些不知道的module, funtions等等，还有就是知道了但没有熟练应用。

对于while, if 等，None, 0 等都视作False，不用再去显示比较了。

1
2
3
4
5
6

divmod(x, y)
"""
returns a pair of numbers consisting of their quotient and remainder.
"""
x and y : x is numerator and y is denominator
x and y must be non complex

L1480: 这题学到个新的function accumulate 依次叠加

1
2
3
4

from itertools import accumulate
itr = accumulate([1,2,3,4,5])
list(itr)
## [1, 3, 6, 10, 15]

L617: 如果是BFS, 则要使用queue:

1
2
3
4
5
6

## init
queue = collections.deque([])
## same as len(queue)
## canonical way for all collections (tuples, strings, lists, dicts and all their many subtypes) to check empty or not
while queue:
    pass

L1512: collecions.Counter()

1
2
3

## return a dict subclass object
## {object: count, ...}
collections.Counter("sbadbfasdbfab")

LC236: 注意这里的这里表达方式

1
2
3
4
5
6
7

def lowestCommonAncestor(self, root, p, q):
    ## 这个表达很省事
    if root in (None, p, q): return root
    ## generator 
    left, right = (self.lowestCommonAncestor(kid, p, q)
                   for kid in (root.left, root.right))
    return root if left and right else left or right

This course is from PluralSight Automating Jenkins with Groovy. 这里不是讲如何编写declarative pipeline, 而是对Jenkins进行本地化配置, by Groovy script.

使用 Groovy 操控Jenkins API 去替代一些重复的UI 操作，就是从API 层面去使用jenkins，比如schedule job等等，甚至改变一些系统设置，毕竟Jenkins 也是Java 应用。

除了对Jenkinsfile 进行 version control, 对Jenkins本身的配置，也可以，这时候就需要Groovy script帮忙了. 在哪里去Run groovy script呢? https://foxutech.com/how-to-run-groovy-script-in-jenkins/

• Groovy Plugin
• Jenkins Script Console
• Scriptler Plugins

第1, 2个接下来👇都提到了。

Jenkins Configure Groovy

Install Groovy plug-in from Jenkins Plugin Manager, search Groovy in Available section.

You can also run Groovy script in Manage Jenkins -> Script Console, you can write system script here, for example:

1
2
3
4
5
6

// default packages are imported
def svr = Jenkins.instance;
// get existing TEST job
def job = svr.getJob("TEST");
def sched= job.scheduleBuild2(0);
sched.get();

还可以用groovy 调用API 去读取设置的Parameters, then use them in script. The jenkins API documention is in: https://javadoc.jenkins.io/ Hudson的名字由来是历史原因.

You can put this inline script in freestyle project, go to configure -> Build, there are 2 options about Groovy:

• Execute Groovy Script: non-system script, has less security freight, do jobs that out of jenkins internals.
• Execute System Groovy Script: To run system script, you need to be admin (unchecked use groovy sandbox) or be approved by admin.

Groovy Startup Script

Configuration as code, the startup script will be executed after Jenkins starts immediately to set userful properties.

Create init.groovy.d folder under /var/lib/jenkins and put the script 1_config.groovy in it:

1
2
3
4
5
6
7
8
9
10
11
12

import jenkins.model.*;
import java.util.logging.Logger
// add change into logs
Logger logger = Logger.getLogger("")
logger.info "Executing init script"

// disable remember me checkbox
Jenkins.instance.setDisableRememberMe(true)
Jenkins.instance.setSystemMessage('Jenkins Server - Automating Jenkins with Groovy')
Jenkins.instance.save()

logger.info "Init script complete"

on the browser, type http://localhost:8080/restart, after restart Jenkins, you will see the difference, the checkbox is gone.

Grape is the package manager for Groovy: http://docs.groovy-lang.org/latest/html/documentation/grape.html

Creating Builds

这里主要说了declarative pipeline 的Jenkinsfile, 如同工作中用到的，大量的Groovy functions.

Credentials and Users

You can do the same job on UI, here show you how to do it via script.

• Credentials: used by Jenkins to access something outside
• Users: for users to login Jenkins

This Groovy script will create, delete and list Jenkins user:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

import hudson.model.User;
import hudson.security.*

class UserManager
{
  Set allUsers()
  {
    return User.getAll();
  }
  
  void createUser(String userName, String password){
    if (!this.userExists(userName))
    {
      Jenkins instance = Jenkins.getInstance();
      def realm = new HudsonPrivateSecurityRealm(false);
        
      realm.createAccount(userName, password);
      instance.setSecurityRealm(realm);
      instance.save();
    }
  }
  
  Boolean userExists(userName)
  {
	  return User.get(userName) != null;
  }
  
  void deleteUser(String userId)
  {
  	if (this.userExists(userId))
    {
	    User u = User.get(userId);
	    u.delete();
	  }
  }
}

def mgr = new UserManager();

mgr.createUser("test", "user");
mgr.deleteUser("test");

println(mgr.userExists("cbehrens"));

for (user in mgr.allUsers()){
	println(user.id); 
}

Create credentials in script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

import com.cloudbees.plugins.credentials.*;
import com.cloudbees.plugins.credentials.common.*;
import com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl;

class CredentialManager
{
  Set getAll()
  {
	return CredentialsProvider.lookupCredentials(StandardUsernameCredentials.class, Jenkins.instance, null, null);
  }  
  
  void changePassword(String credentialId, String password){
    println 'change password';
    def creds = CredentialsProvider.lookupCredentials(StandardUsernameCredentials.class, Jenkins.instance, null, null);
    
    def credential = creds.findResult { it.id == credentialId ? it : null };
      
    if (credential != null)
    {
      def credentials_store = jenkins.model.Jenkins.instance.getExtensionList(
            'com.cloudbees.plugins.credentials.SystemCredentialsProvider'
            )[0].getStore()

        def success = credentials_store.updateCredentials(
            com.cloudbees.plugins.credentials.domains.Domain.global(), 
            credential, 
            new UsernamePasswordCredentialsImpl(credential.scope, credential.id, credential.description, credential.username, password)
            )

        if (!success) 
        {
            throw new RuntimeException("Changing password failed.");
        }
      
      	println("password change complete");
    }
  }
}

def mgr = new CredentialManager()
mgr.changePassword("githubcreds", "password");

This post is about set up efficient, beautiful theme emulator with oh-my-zsh.

Reference from: https://gist.github.com/kevin-smets/8568070

Install iTerm2

Install iTerm2 on Mac
https://www.iterm2.com/

Configure iTerm2

Solarized Dark is bad for some syntax highlighting, to import other themes, download from there, zip file, you can uncompress and put the downloaded folder to the user home directory, for example /Users/chengdol: https://iterm2colorschemes.com/

The theme I use is Chalk, import Chalk.itermcolors file(s) in the scheme(s) folder:

Additionally，go and set iTerm2:

1
2
3
4
5

Preference ->
Advanced ->
mouse ->
Scroll wheel sends arrow keys when in alternate screen mode ->
yes

This fix the mess-up issue when you scroll cursor inside Vim.

iTerm2 Tips

1. Hotkeys, the floating terminal window setup:

Go to Preferences -> Keys -> Hotkey and create a Dedicated Hotkey Window. My customized hotkey is ctrl + shift + t. Set the hotkey window profiles text font the same as the default iTerm2 window, here I use MesloLGS NF.

2. Locating the cursor in iTerm2 terminal: command + /.

3. Send commands to mutliple panes in the same tab shift + command + i, disable use the same command.

4. Go to split pane by direction: command + shift + arrow key.

5. zsh cat command appends ‘%’ in line end if no newline at end.

Configure Oh My Zsh

Install ohmyz on Mac. Mac zsh is pre-installed, you can check by:

1

zsh --version

If not, install zsh first. Then install ohmyzsh:

1

sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Then install Powerlevel10k theme, this theme is what exactly I need:

1

git clone https://github.com/romkatv/powerlevel10k.git $ZSH_CUSTOM/themes/powerlevel10k

By default it uses robbyrussell theme, you can see it from ~/.zshrc file, the theme files are located in ~/.oh-my-zsh/themes folder.

1
2
3
4
5
6
7
8
9
10
11
12

# Set name of the theme to load --- if set to "random", it will
# load a random theme each time oh-my-zsh is loaded, in which case,
# to know which specific one was loaded, run: echo $RANDOM_THEME
# See https://github.com/ohmyzsh/ohmyzsh/wiki/Themes
ZSH_THEME="powerlevel10k/powerlevel10k"

# Which plugins would you like to load?
# Standard plugins can be found in $ZSH/plugins/
# Custom plugins may be added to $ZSH_CUSTOM/plugins/
# Example format: plugins=(rails git textmate ruby lighthouse)
# Add wisely, as too many plugins slow down shell startup.
plugins=(git kubectl docker docker-compose gcloud)

Next when you start a new terminal session, the Powerlevel10 configure wizard will be launched to set your prompt pattern, it will automatically check and install the font for you. When select encoding, choose Unicode, otherwise no icon will show.

If you want to reset the configuration, simply run:

1

p10k configure

Powerlevel10 project web page: https://github.com/romkatv/powerlevel10k#extremely-customizable

Then set zsh as the default shell` on Mac: https://askubuntu.com/questions/131823/how-to-make-zsh-the-default-shell

1
2
3
4
5
6
7

## verify default shell is zsh or not
echo $SHELL

## If not zsh, set it
chsh -s $(which zsh)
## show current running shell
echo $0

After changing the theme, relaunch iTerm2.

~./zshrc works the same as ~/.bashrc for bash, append other alias here, append following snippet in ~/.zshrc:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

# Example aliases
# alias zshconfig="mate ~/.zshrc"
# alias ohmyzsh="mate ~/.oh-my-zsh"
# ### some alias for convenience
alias cdd='cd ~/Desktop'
alias cdmb='cd ~/Desktop/chengdol.blog/'

## docker
alias di='docker images'
alias dp='docker ps'
alias drp='docker rm -f'
alias dri='docker rmi -f'
alias dl='docker logs -f'

function dexec()
{
 docker exec -it $1 sh
}

## python virtual env
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/virtualenvwrapper.sh
export WORKON_HOME="/Users/chengdol/virtualenvs"
export PROJECT_HOME="/Users/chengdol/virtualprojects"
alias svenv="source /Users/chengdol/virtualenvs/py3.10/bin/activate"

export OC_EDITOR=vim
export KUBE_EDITOR=vim

bindkey "^[^[[C" forward-word
bindkey "^[^[[D" backward-word
bindkey "^U" backward-kill-line

To list all shortcut in Zsh, run

1

bindkey

Reference:

• Ctrl-u in Bash
• Zsh key codes

Use cat command to see what exactly the key will present is a good idea.

Visual Studio Code Terminal

To make the VSC build-in terminal good with ohmyz theme, needs to add below setting to User VSC setting.json file(open by shift + command + P):

1
2
3

{
    "terminal.integrated.fontFamily": "MesloLGS NF",
}

Then restart the VSC, the theme should be good.

Bash with Starship

Still using iTerm2, if you want to stick to Bash shell, try this:

Starship: cross shell prompt. https://starship.rs/

Install by running:

1

curl -fsSL https://starship.rs/install.sh | bash

After install, append this line in ~/.bashrc

1

eval "$(starship init bash)"

My config file:

1

mkdir -p ~/.config && touch ~/.config/starship.toml

My current ~/.config/starship.toml file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

# Disable the newline at the start of the prompt
add_newline = true

[line_break]
disabled = false

[character]
symbol = "➜"
error_symbol = "✗"
use_symbol_for_status = true

[battery]
full_symbol = "🔋"
charging_symbol = "⚡️"
discharging_symbol = "💀"

[time]
disabled = false
format = "🕙[ %T ]"
utc_time_offset = "-5"
time_range = "10:00:00-14:00:00"

[directory]
truncation_length = 8

## show virtual environments
[python]
disabled = false

[docker_context]
disabled = false
symbol = "🐋 "
only_with_files = true

[kubernetes]
symbol = "⛵ "
style = "dimmed green"
disabled = false

[terraform]
symbol = "🏎💨 "
disabled = false

[package]
disabled = true

[ruby]
disabled = true

[rust]
disabled = true

[nodejs]
disabled = true

[java]
disabled = true

[golang]
disabled = false

[aws]
disabled = true

## add more

//TODO:

• https://app.pluralsight.com/course-player?clipId=8ada28fb-a440-477a-af89-84b1eceb5440

From Pluralsight: AWS Certified Developer – Associate (DVA-C01)

别人提到的:

1. 工作中经常用到， 需要花时间学习理解的: vpc, subset, security group, eni, route53, acm, IAM
2. 其次: load balancer, auto scaling, ddb throttle, api gateway。
3. 其他: lambda, step function, cloudformation, s3, ec2, sqs, sns, cloudwatch, ecs, ecr, code deploy 等等

Core Services

EC2: Elastic Cloud Compute AMI: Amazon Machine Image EBS: Elastic Block Storage, used for EC2 files systems Security Group: set of firewall rules that control the traffic for your single instance, for example, control who can ssh to EC2 instance, VPC is for groups of instance

S3: Simple Storage Service, maxmium file size is 5T, bucket is accessed via URL, the same as gcloud storage. Can be used for hosting static web site.

RDS: Relation Database Service Route53: Domain Name System (DNS) servics, you can register your domain name!

EC2

Enhancing Services

EB: Elastic Beanstalk, application service running on EC2

Lambda: Serverless option for executing code, function as a service, only pay when the code is running, significant cost savings if you have infrequent activity. Great for small, irregular tasks, for example, nightly ETL kickoffs, notification type functions

DynamoDB: a managed NoSQL database, supports both key-values and document models

VPC: for securing your services, components in the VPC can connect each through private IP. Multiple subnets can be in VPC, for example, you can configure public subnet and private subnet.

How does VPC work?

• route table: control what goes where
• network ACL(access control list): act as subnet-level firewalls, control who can come and go

CloudWatch: monitoring resources and acting on alerts, for example, CPU usage on EC2 instances, DynamoDB read/write throughput, estimated billing charges

CloudFront: super fast CDN, works seamlessly with S3, EC2, load balancing and route53

CloudWatch

For example, Increasing network traffic -> EC2 -> alarm CloudWatch -> action -> Auto Scaling Group -> EC2. SNS can also be integrated to CloudWatch.

SNS: simple notification service, Pub/sub messaging for microservices and serverless applications. First create topic, then subscribe this with from email or SMS, etc

IAM

MFA, multi-factor authentication, reuqire more than one factor to authenticate. MFA process: password + device code (app generated code refresh every 60 seconds) 类似将军令, 要先在手机上下载一个MFA app.

After loging aws console, click the account user name -> My security credentials -> MFA

IAM policy make it easy to assign permissions to users or groups in an administrative way. Users have no permission by default. Policy properties:

• Effect: allow, deny
• Action: operations user can perform
• Resources: user performs on

Root account permission is dangerious, follows amazon suggested best practices to have more securities. For example, create a admin grouo, attch policy to it, then add user to this group, use this user to login.

Access AWS

• Web console
• SDK, programming application uses: https://github.com/aws
• Command line, great for shell scripting: https://github.com/aws/aws-cli

To generate the access key for SDK and cli, after loging aws console, click the account user name -> My security credentials -> Access Keys.

Create ~/.aws/credentials file with content from your access key:

1
2
3
4
5

[default]
aws_access_key_id=AKIAIVHU6XLsd3J7IAKA
aws_secret_access_key=Vemdu3nD65uY1cWC0fznCEfUhvsUT9NIjMT790zqK
region=us-west-2
output=json

I use aws cli docker to run the commands, the docker container is ephemeral for each command (for convenience, set alias for docker run command), you need to mount the ~/.aws to container:

1

docker run --rm -ti -v ~/.aws:/root/.aws amazon/aws-cli s3 ls

For other methods installing

• https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html

To aviod installing dependencies, you can use virtual machine to setup environment.

Demo Components

A pizza web site:

• EC2, host web application
• DynamoDB, store users & toppings
• RDS, store pizza
• S3, store images & assets
• ElastiCache, store sessions

//TODO [ ] custom azure container: vim, zsh or startship, auto completion [ ] Azure subscription and tenant IDs? In ~/.azure/credentials

Setup Azure CLI

Using docker azure CLI container: https://docs.microsoft.com/en-us/cli/azure/run-azure-cli-docker

1
2
3
4
5
6
7

docker pull mcr.microsoft.com/azure-cli

docker run -d \
--name azure \
--entrypoint=/bin/bash \
mcr.microsoft.com/azure-cli:latest \
-c "tail -f /dev/null"

Commands

To access a aks cluster:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

# similar to company in gcp
# list subscriptions
az account list --output table

# set subscription
# you can see subscription from akz record
az account set -s "SUBSCRIPTION NAME"


# group, similar to project in gcp
# you can see group from akz record
# config default group
az configure --defaults group=<resource group name>


## or direct get zks credential by
az aks get-credentials -n <cluster name> -g <group name> --subscription

The command reference links:

• subscription
• resource group
• aks

The best way to learn is to read official document along with operating on glcoud console.

Some useful GCP projects:

• Networking 101 GCP Sheet
• Google Cloud 4 Words

Common Commands

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107

## after install gcloud SDK
## init
gcloud init --console-only

## can login multiple user accounts
gcloud auth login
## it is ADC(application default credential)
## for code to interact with GCP, such as terraform CLI
gcloud auth application-default login [--project]

## same as gcloud auth login but using SA credential
## and roles
gcloud auth activate-service-account [--key-file]

## list auth accounts or service account
gcloud auth list
## switch account
gcloud config set account <account name>
## revoke account
gcloud auth revoke <account name>


## show and install components, i.e alpha, beta, kubectl...
gcloud components list
gcloud components install [beta]


## all projects under my account, not the used one
gcloud projects list 


## set which project to use
gcloud config set project <project name>
## current project in use
gcloud config list project
## get project ID
gcloud config get-value project

## list service account in project
gcloud iam service-accounts list [--project <project ID>]
## create service-account after auth login and set project
gcloud iam service-accounts create <SA name>  [--display-name=<"description">] [--project <project id>]
## can update display name and description
gcloud iam service-accounts update ...
## disable service account
gcloud iam service-accounts enable/disable ... 
## delete: When a service account is deleted, its role bindings 
## are not immediately removed; they are automatically purged from 
## the system after a maximum of 60 days.
gcloud iam service-accounts delete ... 
## generate credentials json file for terrform
## can also delete it
gcloud iam service-accounts keys create ~/key.json \
  --iam-account <SA name>@<project ID>.iam.gserviceaccount.com
## see the roles bind to service account
gcloud iam service-accounts get-iam-policy <SA>



## see available context
## -o name: show context name
kubectl config get-contexts [-o name]
## switch context
kubectl config use-context <context name>
## rename context to human readable
kubectl config rename-context <old> <new>

## export current configuration to yaml file
kubectl config view --minify --flatten > cluster.yaml
## the same as this gcloud command
## KUBECONFIG=clusters.yaml: specify cluster.yaml to store the credentials
KUBECONFIG=clusters.yaml gcloud container clusters \
get-credentials <cluster name> --zone=<cluster zone>


## current enabled API list
gcloud services list [--porject <project ID>]
gcloud services enable <API>
gcloud services disable <API>


## create default VPC network
gcloud compute networks create default


## create K8s cluster in default network
gcloud container clusters create gke-eu --zone europe-west1-c \
  --release-channel stable --enable-ip-alias
## list cluster
gcloud container clusters list
gcloud container clusters list \
  --project <project name> \
  --filter "name:cluster-name" \
  --format "get(location)"
## describe
gcloud container clusters describe <cluster name> --region <region/zone>
## delete cluster
## -q: quiet
gcloud container clusters delete gke-eu --zone=europe-west1-cd [-q]


## grant IAM roles to end user in project
## member can be serviceAccount:email
gcloud projects add-iam-policy-binding <project ID> \
 --member user:<member> \
 --role=roles/gkehub.admin \
 --role=roles/resourcemanager.projectIamAdmin