Vault Quick Start

Posted on | Edited on | In

Course git repo, this repo has useful Vault commands: https://github.com/ned1313/Getting-Started-Vault

My Vault vagrant demo: https://github.com/chengdol/InfraTree/tree/master/vagrant-vault

Questions

Vault vs K8s secrets? Examples of what Vault can do that k8s secrets cannot: With Vault you can rotate secrets and have secrets with short TTL With Vault you can access secrets across namespaces (or outside the k8s cluster) Vault can provide a PKI for signing certs (enabling for example automation of cert generation for mtls) Vault can use LDAP, oauth, IAM, etc as identity providers

Introduction

Vault web site: https://www.vaultproject.io/

Secure, store and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets and other sensitive data using a UI, CLI, or HTTP API. 注意API的path，并不是和UI上的path一样！

Vault works well with Consul, for example, set Consul as storage backend.

Start a Vault server:

1
2
3
## same as consul in development mode, don't use this in production!
## 0.0.0.0:8200, used for vagrant port fordwarding access from host
vault server -dev-listen-address 0.0.0.0:8200 -dev

Output as below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variable:

    $ export VAULT_ADDR='http://0.0.0.0:8200'

The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.

## these two are critical
Unseal Key: pLNBmZQRHvspdy5unZcTjm1jOVQ81Z0pO6ywYHNP1zQ=
Root Token: s.hfnmMkgG7cggWDOmPfHC1jIe

Development mode should NOT be used in production installations!

unseal key在production mode中用来解除对Vault server的封锁，否则无法login as root.

注意，这里VAULT_ADDR是本地的，vault server当然可以在其他地方，设置对应的地址即可.

1
2
3
4
5
6
## Vault API access
export VAULT_ADDR='http://0.0.0.0:8200'
export VAULT_TOKEN=s.ttlDcetbJe3uLt0FF5rSidg3

## login to vault server need VAULT_TOKEN to login
vault login

Vault web UI access: http://localhost:8200

So, just like Consul, there are 3 ways to interact with Vault server: UI, API, CLI (actually running API under the hood).

secret is a pre-existing secret engine folder in Vault storage path, you can see it in UI:

1
2
3
4
5
6
#Write a secret
vault kv put secret/hg2g answer=42
#For Linux
# marvin.json is a json file
curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST \
 --data @marvin.json $VAULT_ADDR/v1/secret/data/marvin

marvin.json is as follow:

1
2
3
4
5
6
{
    "data": {
        "paranoid": true,
        "status": "bored"
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#Get a secret
vault kv get secret/hg2g
#specify format
vault kv get -format=json secret/hg2g
vault kv get -format=yaml secret/hg2g
#For Linux
#Install jq if necessary
sudo yum install jq -y
curl --header "X-Vault-Token: $VAULT_TOKEN" $VAULT_ADDR/v1/secret/data/marvin | jq

#Put a new secret in and a new value for an existing secret
vault kv put secret/hg2g answer=54 ford=prefect
vault kv get secret/hg2g

#Delete the secrets
vault kv delete secret/hg2g
vault kv get secret/hg2g

#For Linux
curl --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/secret/data/marvin

Working with Secrets

Secret lifecycle:

  • Create
  • Read
  • Update
  • Delete (soft or hard unrecoverable)
  • Destroy

There is version 1 and version 2 of secret engine server, version 2 is more recoverable and versioning but less performance then version 1 if you need to scale. secret folder 就是默认创建的secret engine, version 2, 可以去UI 查看configuration.

Demo code about secrets lifecycle: https://github.com/ned1313/Getting-Started-Vault/blob/master/m3/m3-secretslifecycle.sh

Everytime you update the key, the version increment by 1:

1
2
3
4
## pick value by version
vault kv get -version=3 secret/hg2g
## from API
curl -X GET --header "X-Vault-Token: $VAULT_TOKEN" $VAULT_ADDR/v1/secret/data/hg2g?version=3 | jq .data.data

If you delete version 3, you still can get version 1 or 2:

1
2
3
4
5
6
7
8
9
10
vault kv delete secret/hg2g
vault kv get -version=2 secret/hg2g
## undelete version 3
## -versions not -version, you can undelete multiple: -versions=2,3
vault kv undelete -versions=3 secret/hg2g

## API
#For Linux
curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST \
  $VAULT_ADDR/v1/secret/undelete/hg2g  --data '{"versions": [2]}'

Destroy, can no longer undelete:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#Destroy the secrets
vault kv destroy -versions=1,2 secret/hg2g

#For Linux
## metadata is still there
curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST \
  $VAULT_ADDR/v1/secret/destroy/hg2g --data '{"versions": [1,2]}'

#Remove all data about secrets
vault kv metadata delete secret/hg2g
vault kv get secret/hg2g

#For Linux
curl --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE \
  $VAULT_ADDR/v1/secret/metadata/hg2g

Create New Secret Engine

Demo about secret engine, 我这里就不贴出来了, 如果用API，还有一些配置用的json文件，见这个的上层目录中的比如dev-b.json: https://github.com/ned1313/Getting-Started-Vault/blob/master/m3/m3-secretengine.sh

Create Mysql DB Secret Engine

Vault official reference: https://www.vaultproject.io/docs/secrets

Vault在这里类似一个中间层，在user 和 Mysql instance之间，保存和传递credential和请求. 这个demo在AZure上spin up了一个bitnami Mysql instance，和本地的vault mysql secret engine关联，然后通过Vault联系Mysql 产生一个dynamic user，并授予这个临时user权限，我们通过这个临时user就可以操作Mysql 数据库了。

这个dynamic user lifecycle also managed by Vault, as well as the permission the user has. https://github.com/ned1313/Getting-Started-Vault/blob/master/m3/m3-mysqlengine.sh

Besides Mysql example, Vault secrets can be used for certificates, for SSH credentials, etc. Secrets engine make Vault extensible, there are different plugin to handle different needs.

Controlling Access

Similar to RBAC in K8s, authentication method to control login, policies to control what is able to do, managing client tokens.

Vault Auth Method

Vault has internal authentication (called userpass) and support external, multiple authrntication methods. https://github.com/ned1313/Getting-Started-Vault/blob/master/m4/m4-basicauth.sh

1
2
3
4
5
6
7
8
9
## list current auth methods
## by default, we only have token auth method
vault auth list
## enable new auth method
vault auth enable userpass
## create user
vault write auth/userpass/users/arthur password=dent
## list user
vault list auth/userpass/users

You will see the changes in UI Access section.

Once you create a user/password, then you can run for example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
## 这里使用的是user/password login，不是用token
vault login -method=userpass username=arthur
## then input password
Password (will be hidden): 
## 这个warning是因为，做实验的时候之前export了root token
WARNING! The VAULT_TOKEN environment variable is set! This takes precedence
over the value set by this command. To use the value set by this command,
unset the VAULT_TOKEN environment variable or set it to the token displayed
below.

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  s.z52FGzn78XynKlxeS0Akt0t7
token_accessor         90LaRNNtsCUEg6yDd9ldRi3v
token_duration         768h
token_renewable        true
token_policies         ["default"]
identity_policies      []
policies               ["default"]
token_meta_username    arthur


## will show you where is the token from, not root token any more
## see `path` field where is the token from
vault token lookup

Token represents who you are and what you can do, for example, the user itself cannot update its password via its token:

1
2
3
4
5
6
7
8
9
10
## update password
vault write auth/userpass/users/arthur/password password=tricia
## output
Error writing data to auth/userpass/users/arthur/password: Error making API request.

URL: PUT http://0.0.0.0:8200/v1/auth/userpass/users/arthur/password
Code: 403. Errors:

* 1 error occurred:
	* permission denied

Can do this after export root token.

Delete auth:

1
2
#Remove account
vault delete auth/userpass/users/arthur

这里介绍了2个新概念: LDAP and Active Directory What are the differences between LDAP and Active Directory? Active Directory is a database based system that provides authentication, directory, policy, and other services in a Windows environment

LDAP (Lightweight Directory Access Protocol) is an application protocol for querying and modifying items in directory service providers like Active Directory, which supports a form of LDAP.

Short answer: AD is a directory services database, and LDAP is one of the protocols you can use to talk to it.

见下面vault policy章节的例子. Case: 在外部设置了AD as authentication method, enable LDAP auth in Vault. Then user login Vault with AD credentials against LDAP talk to AD, AD talk to Vault and determine the policy about what the user can do, then user will get the right token from vault to access the store.

Vault Policy

Similar to K8s role. https://github.com/ned1313/Getting-Started-Vault/blob/master/m4/m4-activedirectory.sh

这个例子，远程登录了一个Vault server, login as root, create devkv store with key and value pair, then create policy dev with HCL file for user to access the devkv.

Enable LDAP auth, configure it with Active Directory remotely. Then assign the developers group in LDAP with dev policy.

Then user adent login vault against ldap method. The user is in the developers group so it will get token with permission specified by dev policy.

Client Token

https://www.vaultproject.io/docs/concepts/tokens

Wrapping Response

https://github.com/ned1313/Getting-Started-Vault/blob/master/m4/m4-tokenwrapping.sh

Operating Vaule Server

主要讲了production Vault server setup.

  • Vault server architecture
  • Storage backend options
  • Vault server operations

这里例子用Consul作为 storage backend(不是说it’s not intended for this吗😂), 在Vault 主机上运行有Consul agent 和 远处的Consul server通信 (gossip tcp: 8301, RPC tcp: 8300) ，Consul server可以有多个以实现HA。Vault通过本地的Consul agent port 8500和其交互，外界访问Vault 用默认的port 8200: https://github.com/ned1313/Getting-Started-Vault/tree/master/m5

可以留意一下这里的Consul配置，之前用的一直是dev mode，这里是production mode了, 把Consul 包装成了一个service, run as daemon, in consul folder:

注意把 .hcl and .service 文件内容paste到 consul-deploy.sh 创建的文件中。script中的useradd command也值得借鉴。consul keygen 是用来产生encrypt string的，用在server 和 client的 .hcl中。

Consul agent setup:

好了，这里引出一个有意思的东西: Create Custom Systemd Service，之前从来没有这么做过，原来systemd是可以自己配置的！ https://medium.com/@benmorel/creating-a-linux-service-with-systemd-611b5c8b91d6

Vault server Config, also set Vault as systemd service, in vault folder:

在 .hcl 中就配置了Consul 为storage backend，看上去本来就有这种特性，所以不需要过多的设置。

Server Operations

Operator command: https://www.vaultproject.io/docs/commands/operator

Setup production Vault server, sealed and need to unseal: https://github.com/ned1313/Getting-Started-Vault/blob/master/m5/m5-serveroperations.sh

Auditing

Everything is audited, sensitive data is hashed unless you explicitly set false. https://github.com/ned1313/Getting-Started-Vault/tree/master/m6

Auditing device type:

  • File, JSON format
  • Syslog
  • Socket: TCP/UDP
1
2
3
4
5
6
7
8
9
10
11
12
13
## enable auditing device
vault audit enable [type] [-path=valut_path]
## log_raw=true means no encrypt sensitive data
vault audit enable file file_path=/var/log/vault/vault_audit.log log_raw=true
vault audit enable -path=file2 file file_path=/var/log/vault/vault_audit2.log
vault audit enable syslog tag="vault" facility="LOCAL7"

## disable
vault audit disable [vault_path]
## disable file2 created above
vault audit disable file2
## list
vault audit list -detailed
0%