Inprogress

Posted on Edited on In

This is the blog incubator and TODO list 保持记忆的熟练的唯一方式就是复习 和 刷题，特别是语言类。 最近在做一些pipeline的工作，需要快速上手，主要涉及到的: Terraform Python Jenkins gradle (一种build tool, others like Blaze, Make) Groovy (JVM language, similar to Java, for declarative Jenkins pipeline) 现在不清楚的地方是各自文件的语法。 先从groovy，gradle入手，然后jenkins。 [x] 复习过去x天改动过的blog

CI/CD

[x] travis.yml and inspiration: 大概能看出来是怎么回事了

Envoy Migration

最近几天发现的缺失: [x] SSH config and proxy config file, proxy jump, ssh-add. [x] Gitlab, see linkedin learning, gitlab-ci and web dashboard general use. [x] Http header, CONNECT method, CRLF, browser debug [x] docker compose for testing, why not docker? [x] SSL/TLS, cloud computing use cases, see udemy ip route table vs iptables filter/nat table??? [x] curl flags and testing, pluralsight [x] linux tips weekly, proxy sections [x] Unix domain socket, websocket, co-routine? [x] proxy, http proxy, reverse proxy?? ssl/tls?

[x] envoy tcp proxy and how to test? 设置好socket server, client，然后client connect to proxy port, in proxy, it driect to server address and port. 解决了！upstream cluster IP 没设置好。。对于docker 不是127.0.0.1!! [x] for now, understand: http proxy (connect support), tcp proxy, socks proxy. [x] shell parameter expansion variable pattern match: ${1##*/} ${USAGE%/*} ${1:1:1}, etc [x] proxy的模拟测试环境 -> docker compose + flask + python other packages [x] pagerduty, for on-call

[ ] socks how to setup in gcloud or aws --> VPN

[ ] ip/iptables, vagrant + envoy docker does not work, linkedin learning

[ ] network traffic school, internally! see my SRE week note. [ ] 养老金扩展: https://www.transglobalus.com/zh/services_zh/

[ ] monit: is a utility for managing and monitoring processes, for example restart service when it cost too much memory and cpu

[x] systemd init file

1
2
3
4
5
6
## some fields
network-online.target
Environment=
OOMScoreAdjust=-1000 ~ 1000
WantedBy=multi-user.target
Type=forking | simple

[x] what are smoke & nightly testing [x] blue/green deployment

[ ] Linux LPIC-1/2/3 exams [ ] Linux LFCE training [ ] CCNA Cisco network certifiate, see pluralsight my channel network

[ ] 重读how linux works book [ ] rsyslogd, syslog services

[ ] Using golang write tcp, http server. [ ] container runtime varies, search youtube [ ] device42 [ ] tower [ ] packer [ ] jinja2 template syntax

[x] linux screen command [ ] serverspec vs testinfra, infra instance testing tools [ ] auditiong user account on linux, how? [ ] command history fzf plugin [ ] load average: http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html

Python

[x] python click必须掌握，很重要！对于advanced script!! click echo color support: click.echo(click.style('Some more text', bg='blue', fg='white')) 这样可以为以后log中重点高亮 [x] python logging 更高级的用法

如果要建造一个package 哪些是必要的流程，以及测试? [ ] setuptools understand [ ] pytest [ ] cookiecutter, creating new project [ ] pip install -e . (on the fly editing and take effect)

K8s

kubectl wait command: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#port-forward

kubectl-images: https://github.com/chenjiandongx/kubectl-images

kubectl krew: https://github.com/kubernetes-sigs/krew

[ ] k8s is deprecating docker in upcoming release. switch to other common docker runtime: containerd and CRI-O. There are lots of options out there for this specific use case including things like kaniko, img, and buildah.

Squid

这个对forward proxy支持挺好, long history proxy -_-|||, old [ ] squid proxy

Groovy

掌握基本data type, control structure, function definition 就可以了. 因为目前主要是用在Jenkins中。 和Java 通用的.

Groovy quick start: https://www.youtube.com/watch?v=B98jc8hdu9g

Books, course and presentations, recommend: https://groovy-lang.org/learn.html#books O’relly course

http://groovy-lang.org/syntax.html alternative java language

DSL: domain specific language sdkman very good installation toolkit (also for gradle!)

1
2
3
4
## must have jdk installed first
yum install java-1.8.0-openjdk-devel -y
## specify version 2.5.2
sdk install groovy 2.5.2

Utilize vagrant VM to setup groovy runtime with SDKMAN, sync folder to work with VSC. groovy closure: https://groovy-lang.org/closures.html

1
2
3
4
// each
def names = ["alice", "bob", "michael"];
// using closure
names.each { println it };

OOP, the class can be no constructor.

String interpolation: https://groovy-lang.org/syntax.html#_string_interpolation

Gradle

don’t see latest book, please refer to official doc https://gradle.org/guides/ Ant and Maven are also build tools, gradle wrapper, incremental builds.

  1. build by convention
  2. Groovy DSL
  3. supports dependencies
  4. support mutli-project build
  5. easily customizable

To install Gradle, using sdkman tool.

有几个概念搞清楚:

  1. gradle wrapper -> gradlew (version control)
  2. build.gradle: syntax, plugin
  3. gradle.properties
  4. extract env variable
1
2
3
4
5
6
7
8
apply plugin: 'java'

task {
  doLast {
    // groovy syntax
    println "hello, gradle!"
  }
}

Git

using use/passowrd instead of ssh need to setup personal access credential: https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line

for example, mypersonal access token 456f6facec85415b91588f581sfsdfssfe56c0 I can embed the user/cred to push command: https://stackoverflow.com/questions/29776439/username-and-password-in-command-for-git-push

1
git push 'https://chengdol%40ibm.com:456f6facec85415b91588f58182dd1cfe27e56c0@github.ibm.com/repo/hello.git'

use %40 replace @. github.ibm.com/repo/hello.git is the url to your repo

[x] git tag, see my blog [ ] github CLI: gh, very similar to git command, can make alias to it.

[ ] pluralsight git training in my channel [ ] unfinished, fork operation

[ ] gitlab on GKE lab

[ ] git commands more git rewrite: https://jonhnnyweslley.net/blog/how-to-rewrite-git-urls-to-clone-faster-and-push-safer/ https://paste.googleplex.com/6716960572178432

1
git diff --name-only --diff-filter=ACMR HEAD~1 HEAD

deploy keys

https://github.blog/2015-06-16-read-only-deploy-keys/

Gitbook

https://www.gitbook.com/ 这个写东西还可以, but not free

承上启下

interpersonal skill, BQ https://mail.google.com/mail/u/1/#inbox/FMfcgxwHNMbchDhFLQBnSltwkxBJVLbX

As an “intro” project, I would recommend deploying Kubernetes using Terraform. You can use any provider such as GKE, AKS or EKS.

Once the Terraform template is working, extend it to install some applications inside Kubernetes, such as Vault, Elastic, Cassandra, etc. using the Helm and Helmfile.

You may want to integrate the database applications inside Kubernetes with a Vault instance such that they would use it for authentication backend.

You can either build the Helm Charts yourself or use existing ones (perhaps you would want to modify them…)

Further, you can extend the application deployment to emit some metrics and setup monitoring using Prometheus and Grafana.

Once you are happy with the runtime stack, introduce some CI/CD pipelines using Jenkins and/or Gitlab.

现在能接触到的基础打好 k8s, docker, jenkins (groovy + gradle), ansible, helm git -> gitlab (with jenkins)

  • 常用工具: vim visual studio bash (中文版教程可以), zsh

  • composing database layer: cassandra elasticsearch (monitoring + log system) consul vault

  • monitoring: prometheus grafana kibana

  • others: kafka zookeeper istio golang RabbitMQ redis Python

load balancer: nginx, haproxy

  • cloud 基本结构，应用，组件: AWS, ok GCP, ok Azure

我猜工作流程: 用terraform 构造k8s 环境 或其他， 用ansible增加或更改配置，然后在k8s中部署应用(prometheus + grafana and elk for logs)，最后集成CI/CD development -> QA/UAT - > production

understand

如果不清楚这些则看不懂配置的含义, 要清楚各个cloud中resource的类型，配置，如何搭配: instance, vpc, lb, dns, storage, routing, subnet, gateway

terraform -> deploy k8s (EKS or GKS) -> install vault, elasticsearch, cassandra by helm (need modify though) -> Prometheus Grafana Kibana -> CI/CD: jenkins + gitlab

把完全不知道是什么的快速过一遍。 先学terraform -> elasticsearch -> vault, cassandra -> monitoring tools packer -> gitlab, tower -> helmfile

steps

  1. terrform provision k8s on aws and gcp
  2. k8s on fyre to test other components via helm
  3. monitoring system: prometheus + grafana, deployed by operator
  4. log system: elasticsearch, kibaba, logstash, beats

实验: 在k8s中部署prometheus + elastic stack, by helm or yaml，监控系统状态

  1. Hashicorp: consul, vault
  2. cassandra