Set up Secure Docker Registry in K8s

The step to set up secure docker registry service in K8s is different from docker. There are some adjustments and changes to apply.

Toolkits we need to achieve our goal:

  1. openssl
  2. htpasswd
  3. skopeo

Create SSL/TLS Certificate and Key

Use openssl command to generate certificate and private key for setup secure connection:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
## create certs
mkdir -p /root/registry-certs
## get from master

DOCKER_REGISTRY_URL=blair1.fyre.com
## make a copy of .crt and give suffix .cert
openssl req \
-newkey rsa:4096 -nodes -x509 -sha256 \
-keyout /root/registry-certs/tls.key \
-out /root/registry-certs/tls.cert \
-days 3650 \
-subj "/C=US/ST=CA/L=San Jose/O=IBM/OU=Org/CN=${DOCKER_REGISTRY_URL}"

cp /root/registry-certs/tls.cert /root/registry-certs/tls.crt

Then copy the crt file to every host under /etc/docker/certs.d/<${DOCKER_REGISTRY_URL}>:5000 folder for self-signed certificate trust.

Notice that if the docker daemon json file has enabled the insecure registry, it will not verify the ssl/tls cert! You get docker user account and password, then you can login without certs!

Create Docker User Info

1
2
3
4
5
6
##  create auth file
DOCKER_USER=demo
DOCKER_PASSWORD=demo

mkdir -p /tmp/registry-auth
htpasswd -Bbn ${DOCKER_USER} ${DOCKER_PASSWORD} > /tmp/registry-auth/htpasswd

Generate Secret

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
## create secrets
## we want to setup docker registry in default namespace
kubectl create secret tls docker-registry-tls \
--key=/root/registry-certs/tls.key \
--cert=/root/registry-certs/tls.cert \
-n default

kubetctl create secret generic docker-registry-auth \
--from-file=htpasswd=/tmp/registry-auth/htpasswd \
-n default

## Assume the working namespace is test-1
WORKING_NAME_SPACE=test-1
DOCKER_REGISTRY_SERVER="${DOCKER_REGISTRY_URL}:5000"

kubectl create namespace ${NAME_SPACE}
kubectl create secret docker-registry docker-registry-creds \
--docker-server=$DOCKER_REGISTRY_SERVER \
--docker-username=$DOCKER_USER \
--docker-password=$DOCKER_PASSWORD \
-n ${WORKING_NAME_SPACE}

Bind Image Pull Secret to Service Account

see document in K8s.

1
2
3
4
5
## patch creds to default service account in test-1
## assume we use default service account in yaml
kubectl patch serviceaccount default \
-p '{"imagePullSecrets": [{"name": "docker-registry-creds"}]}' \
-n ${WORKING_NAME_SPACE}

Or you can specify imagePullSecrets in yaml explicitly, for example:

1
2
3
4
5
6
7
8
9
10
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: private-reg-container
image: <your-private-image>
imagePullSecrets:
- name: <secret name>

Create Secure Docker Registry

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
## notice the 
## env field
## secret mount field

## deletion is enabled
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: docker-registry
labels:
app: docker-registry
spec:
replicas: 1
selector:
matchLabels:
app: docker-registry
template:
metadata:
labels:
app: docker-registry
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- {key: docker-registry, operator: In, values: ["true"]}
hostNetwork: true
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
containers:
- name: docker-registry
image: localhost:5000/registry:2.7.1
imagePullPolicy: IfNotPresent
env:
- name: REGISTRY_STORAGE_DELETE_ENABLED
value: "true"
- name: REGISTRY_AUTH
value: "htpasswd"
- name: REGISTRY_AUTH_HTPASSWD_REALM
value: "Registry Realm"
- name: REGISTRY_AUTH_HTPASSWD_PATH
value: "/auth/htpasswd"
- name: REGISTRY_HTTP_TLS_CERTIFICATE
value: /certs/tls.crt
- name: REGISTRY_HTTP_TLS_KEY
value: /certs/tls.key
ports:
- name: registry
containerPort: 5000
hostPort: 5000
volumeMounts:
- name: docker-data
mountPath: /var/lib/registry
- name: docker-tls
mountPath: /certs
readOnly: true
- name: docker-auth
mountPath: /auth
readOnly: true
volumes:
- name: docker-data
persistentVolumeClaim:
claimName: registry-pv-claim
- name: docker-tls
secret:
secretName: docker-registry-tls
- name: docker-auth
secret:
secretName: docker-registry-auth

So far the secure docker registry in K8s is up and running in default namespace, it’s host network true so can be accessed from remote. Later can expose it by ingress.

Update Docker User Info

See this post.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
## create new htpasswd file
DOCKER_USER=demonew
DOCKER_PASSWORD=demonew

mkdir -p /tmp/registry-auth
htpasswd -Bbn ${DOCKER_USER} ${DOCKER_PASSWORD} > /tmp/registry-auth/htpasswd
## then encode base64
AUTH_BASE64=$(cat /tmp/registry-auth/htpasswd | base64 -w 0)
## replace old auth secret
## the change will be populated to registry pod
kubectl get secret docker-registry-auth -o yaml -n default \
| sed -e "/htpasswd/c\ htpasswd: ${AUTH_BASE64}" \
| kubectl replace -f -

## replace old docker config creds secret in used working namespaces
NEW_REGISTRY_CREDS=$(kubectl create secret docker-registry docker-registry-creds \
--docker-server=$DOCKER_REGISTRY_SERVER \
--docker-username=$DOCKER_USER \
--docker-password=$DOCKER_PASSWORD \
-n default \
-o yaml --dry-run \
| grep "\.dockerconfigjson" | cut -d":" -f2)

kubectl get secret docker-registry-creds -o yaml -n ${WORKING_NAME_SPACE} \
| sed -e "/\.dockerconfigjson/c\ .dockerconfigjson: ${NEW_REGISTRY_CREDS}" \
| kubectl replace -f -

Skopeo Operation

Please refer my skopeo blog for more details.

1
2
3
4
5
6
7
8
9
10
skopeo copy \
--dest-creds ${DOCKER_USER}:${DOCKER_PASSWORD} \
--dest-cert-dir /root/registry-certs \
docker-archive:/root/busybox.tar.gz \
docker://${DOCKER_REGISTRY_SERVER}/busybox:latest

skopeo inspect \
--creds ${DOCKER_USER}:${DOCKER_PASSWORD} \
--cert-dir /root/registry-certs \
docker://${DOCKER_REGISTRY_SERVER}/busybox:latest
0%