Set up Secure Docker Registry in K8s

Posted on Edited on In

The step to set up secure docker registry service in K8s is different from docker. There are some adjustments and changes to apply.

Toolkits we need to achieve our goal:

  1. openssl
  2. htpasswd
  3. skopeo

Create SSL/TLS Certificate and Key

Use openssl command to generate certificate and private key for setup secure connection:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
## create certs
mkdir -p /root/registry-certs
## get from master

DOCKER_REGISTRY_URL=blair1.fyre.com
## make a copy of .crt and give suffix .cert
openssl req \
        -newkey rsa:4096 -nodes -x509 -sha256 \
        -keyout /root/registry-certs/tls.key \
        -out /root/registry-certs/tls.cert \
        -days 3650 \
        -subj "/C=US/ST=CA/L=San Jose/O=IBM/OU=Org/CN=${DOCKER_REGISTRY_URL}"

cp /root/registry-certs/tls.cert /root/registry-certs/tls.crt

Then copy the crt file to every host under /etc/docker/certs.d/<${DOCKER_REGISTRY_URL}>:5000 folder for self-signed certificate trust.

Notice that if the docker daemon json file has enabled the insecure registry, it will not verify the ssl/tls cert! You get docker user account and password, then you can login without certs!

Create Docker User Info

1
2
3
4
5
6
##  create auth file
DOCKER_USER=demo
DOCKER_PASSWORD=demo

mkdir -p /tmp/registry-auth
htpasswd -Bbn ${DOCKER_USER} ${DOCKER_PASSWORD} > /tmp/registry-auth/htpasswd

Generate Secret

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
## create secrets
## we want to setup docker registry in default namespace
kubectl create secret tls docker-registry-tls  \
    --key=/root/registry-certs/tls.key  \
    --cert=/root/registry-certs/tls.cert \
    -n default

kubetctl create secret generic docker-registry-auth \
    --from-file=htpasswd=/tmp/registry-auth/htpasswd \
    -n default

## Assume the working namespace is test-1
WORKING_NAME_SPACE=test-1
DOCKER_REGISTRY_SERVER="${DOCKER_REGISTRY_URL}:5000"

kubectl create namespace ${NAME_SPACE}
kubectl create secret docker-registry docker-registry-creds \
        --docker-server=$DOCKER_REGISTRY_SERVER \
        --docker-username=$DOCKER_USER \
        --docker-password=$DOCKER_PASSWORD \
        -n ${WORKING_NAME_SPACE}

Bind Image Pull Secret to Service Account

see document in K8s.

1
2
3
4
5
## patch creds to default service account in test-1
## assume we use default service account in yaml
kubectl patch serviceaccount default \
        -p '{"imagePullSecrets": [{"name": "docker-registry-creds"}]}' \
        -n ${WORKING_NAME_SPACE}

Or you can specify imagePullSecrets in yaml explicitly, for example:

1
2
3
4
5
6
7
8
9
10
apiVersion: v1
kind: Pod
metadata:
  name: private-reg
spec:
  containers:
  - name: private-reg-container
    image: <your-private-image>
  imagePullSecrets:
  - name: <secret name>

Create Secure Docker Registry

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
## notice the 
##        env field
##        secret mount field

## deletion is enabled
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: docker-registry
  labels:
    app: docker-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker-registry
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - {key: docker-registry, operator: In, values: ["true"]}
      hostNetwork: true
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
      - name: docker-registry
        image: localhost:5000/registry:2.7.1        
        imagePullPolicy: IfNotPresent
        env:
        - name: REGISTRY_STORAGE_DELETE_ENABLED
          value: "true"
        - name: REGISTRY_AUTH
          value: "htpasswd"
        - name: REGISTRY_AUTH_HTPASSWD_REALM
          value: "Registry Realm"
        - name: REGISTRY_AUTH_HTPASSWD_PATH
          value: "/auth/htpasswd"
        - name: REGISTRY_HTTP_TLS_CERTIFICATE
          value: /certs/tls.crt
        - name: REGISTRY_HTTP_TLS_KEY
          value: /certs/tls.key
        ports:
        - name: registry
          containerPort: 5000
          hostPort: 5000
        volumeMounts:
        - name: docker-data
          mountPath: /var/lib/registry
        - name: docker-tls
          mountPath: /certs
          readOnly: true
        - name: docker-auth
          mountPath: /auth
          readOnly: true
      volumes:
      - name: docker-data
        persistentVolumeClaim:
          claimName: registry-pv-claim
      - name: docker-tls
        secret:
          secretName: docker-registry-tls
      - name: docker-auth
        secret:
          secretName: docker-registry-auth

So far the secure docker registry in K8s is up and running in default namespace, it’s host network true so can be accessed from remote. Later can expose it by ingress.

Update Docker User Info

See this post.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
## create new htpasswd file
DOCKER_USER=demonew
DOCKER_PASSWORD=demonew

mkdir -p /tmp/registry-auth
htpasswd -Bbn ${DOCKER_USER} ${DOCKER_PASSWORD} > /tmp/registry-auth/htpasswd
## then encode base64
AUTH_BASE64=$(cat /tmp/registry-auth/htpasswd | base64 -w 0)
## replace old auth secret
## the change will be populated to registry pod
kubectl get secret docker-registry-auth -o yaml -n default \
        | sed -e "/htpasswd/c\  htpasswd: ${AUTH_BASE64}" \
        | kubectl replace -f -

## replace old docker config creds secret in used working namespaces
NEW_REGISTRY_CREDS=$(kubectl create secret docker-registry docker-registry-creds \
        --docker-server=$DOCKER_REGISTRY_SERVER \
        --docker-username=$DOCKER_USER \
        --docker-password=$DOCKER_PASSWORD \
        -n default \
        -o yaml --dry-run \
        | grep "\.dockerconfigjson" | cut -d":" -f2)

kubectl get secret docker-registry-creds -o yaml -n ${WORKING_NAME_SPACE} \
        | sed -e "/\.dockerconfigjson/c\  .dockerconfigjson: ${NEW_REGISTRY_CREDS}" \
        | kubectl replace -f -

Skopeo Operation

Please refer my skopeo blog for more details.

1
2
3
4
5
6
7
8
9
10
skopeo copy \
       --dest-creds ${DOCKER_USER}:${DOCKER_PASSWORD} \
       --dest-cert-dir /root/registry-certs \
       docker-archive:/root/busybox.tar.gz \
       docker://${DOCKER_REGISTRY_SERVER}/busybox:latest

skopeo inspect \
       --creds ${DOCKER_USER}:${DOCKER_PASSWORD} \
       --cert-dir /root/registry-certs \
       docker://${DOCKER_REGISTRY_SERVER}/busybox:latest