OpenShift Security Context Constraint

Posted on | Edited on | In

OpenShift version: 3.10

I have some doubts about Security Context Constraint (SCC) in OpenShift, for example, I give privileged SCC to service account, but some containers are still running as non-root user.

First what is SCC used for: control the actions that a pod can perform and what it has the ability to access, also very useful for managing access to persistent storage.

Prerequisite

Spin up a fresh OpenShift Enterprise cluster with version:

1
2
openshift v3.9.31
kubernetes v1.9.1+a0ce1bc657

Create regular user demo1

1
htpasswd -b /etc/origin/master/htpasswd demo1 demo1

After login as demo1, you have its records in cluster, if run as system:admin user:

1
2
oc get user
oc get identity

You will get demo1 information.

Fetch integrated docker registry address and port from system:admin user:

1
2
3
oc get svc -n default | grep -E "^docker-registry"

docker-registry                     ClusterIP   172.30.159.11    <none>        5000/TCP                  1h

Experiment

This experiment will show you:

  1. How to enable pulling image from other project.
  2. How to run container as root user.

Start with demo1, login by oc login -u demo1 and create 2 projects:

1
2
oc new-project demo1-proj ## this one is for deploying app
oc new-project demo1-ds   ## this one is for storing imagestream

Pull busybox and update entrypoint to tail /dev/null:

1
docker pull busybox
1
2
3
4
5
docker run -d \
  --name mybb \
  --entrypoint=/bin/sh \
  busybox \
  -c 'tail -f /dev/null'

Then commit:

1
docker commit <container id> busybox

Then docker tag to add docker registry address prefix:

1
docker tag docker.io/busybox 172.30.159.11:5000/demo1-ds/busybox:v1

Docker login to integrated docker registry and push:

1
2
docker login -u openshift -p `oc whoami -t` 172.30.159.11:5000
docker push 172.30.159.11:5000/demo1-ds/busybox:v1

Go back to demo1-proj project by oc project demo1-proj, write a simple deployment yaml bb-deploy.yml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bb-deployment
  labels:
    app: bb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bb
  template:
    metadata:
      labels:
        app: bb
    spec:
      containers:
      - name: bb
        image: 172.30.159.11:5000/demo1-ds/busybox:v1

Enable Pull from Other Projects

If now run:

1
oc apply -f bb-deploy.yml

It will fail to pull the image from demo1-ds (describe pod can see) because we deploy objects on project demo1-proj, it doesn’t have the permission to pull image from other project, let’s enable it:

1
2
3
oc policy add-role-to-user \
    system:image-puller system:serviceaccount:demo1-proj:default \
    --namespace=demo1-ds

Note that if you run this several times, it will create severl duplicate system:image-puller.

Then if you check rolebindings in demo1-ds, you see there is a new binding system:image-puller with service account demo1-proj/default:

1
2
3
4
5
6
7
8
oc get rolebindings -n demo1-ds

NAME                    ROLE                    USERS     GROUPS                            SERVICE ACCOUNTS     SUBJECTS
admin                   /admin                  demo1
system:deployers        /system:deployer                                                    deployer
system:image-builders   /system:image-builder                                               builder
system:image-puller     /system:image-puller                                                demo1-proj/default
system:image-pullers    /system:image-puller              system:serviceaccounts:demo1-ds

Ok, then we can deploy the busubox in demo1-proj project:

1
2
NAME                            READY     STATUS    RESTARTS   AGE
bb-deployment-78bdb8c4f-lzqfj   1/1       Running   0          6s

Let’s next see the container UID:

1
2
3
4
kbc exec -it bb-deployment-78bdb8c4f-lzqfj sh

/ $ id
uid=1000130000 gid=0(root) groups=1000130000

Set Security Context Constraint

Correct, OpenShift by default doesn’t spin up container run as root user due to security issue. Acutally this is about SCC, let’s dig deeper:

These 2 articles cover lots of things for SCC and service account: Managing Security Context Constraints Security Context Constraints official Configuring Service Accounts

you must have cluster-admin privilege to manage SCCs (you can grant cluster-admin privilege to regular user), there are 7 SCCs:

1
2
3
4
5
6
7
8
9
10
oc get scc

NAME               PRIV      CAPS      SELINUX     RUNASUSER          FSGROUP     SUPGROUP    PRIORITY   READONLYROOTFS   VOLUMES
anyuid             false     []        MustRunAs   RunAsAny           RunAsAny    RunAsAny    10         false            [configMap downwardAPI emptyDir persistentVolumeClaim projected secret]
hostaccess         false     []        MustRunAs   MustRunAsRange     MustRunAs   RunAsAny    <none>     false            [configMap downwardAPI emptyDir hostPath persistentVolumeClaim projected secret]
hostmount-anyuid   false     []        MustRunAs   RunAsAny           RunAsAny    RunAsAny    <none>     false            [configMap downwardAPI emptyDir hostPath nfs persistentVolumeClaim projected secret]
hostnetwork        false     []        MustRunAs   MustRunAsRange     MustRunAs   MustRunAs   <none>     false            [configMap downwardAPI emptyDir persistentVolumeClaim projected secret]
nonroot            false     []        MustRunAs   MustRunAsNonRoot   RunAsAny    RunAsAny    <none>     false            [configMap downwardAPI emptyDir persistentVolumeClaim projected secret]
privileged         true      [*]       RunAsAny    RunAsAny           RunAsAny    RunAsAny    <none>     false            [*]
restricted         false     []        MustRunAs   MustRunAsRange     MustRunAs   RunAsAny    <none>     false            [configMap downwardAPI emptyDir persistentVolumeClaim projected secret]

By default, when a container or pod does not request a user ID under which it should be run, the effective UID depends on the SCC that emits this pod. Because restricted SCC is granted to all authenticated users by default, it will be available to all users and service accounts and used in most cases.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
oc describe scc restricted

Name:                                           restricted
Priority:                                       <none>
Access:
  Users:                                        <none>
  Groups:                                       system:authenticated
Settings:
  Allow Privileged:                             false
  Default Add Capabilities:                     <none>
  Required Drop Capabilities:                   KILL,MKNOD,SETUID,SETGID
  Allowed Capabilities:                         <none>
  Allowed Seccomp Profiles:                     <none>
  Allowed Volume Types:                         configMap,downwardAPI,emptyDir,persistentVolumeClaim,projected,secret
  Allowed Flexvolumes:                          <all>
  Allow Host Network:                           false
  Allow Host Ports:                             false
  Allow Host PID:                               false
  Allow Host IPC:                               false
  Read Only Root Filesystem:                    false
  Run As User Strategy: MustRunAsRange
    UID:                                        <none>
    UID Range Min:                              <none>
    UID Range Max:                              <none>
  SELinux Context Strategy: MustRunAs
    User:                                       <none>
    Role:                                       <none>
    Type:                                       <none>
    Level:                                      <none>
  FSGroup Strategy: MustRunAs
    Ranges:                                     <none>
  Supplemental Groups Strategy: RunAsAny
    Ranges:                                     <none>

The restricted SCC uses MustRunAsRange strategy for constraining and defaulting the possible values of the securityContext.runAsUser field. The admission plug-in will look for the openshift.io/sa.scc.uid-range annotation on the current project to populate range fields, as it does not provide this range. In the end, a container will have runAsUser equal to the first value of the range that is hard to predict because every project has different ranges.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
oc describe project demo1-proj

Name:                   demo1-proj
Created:                3 hours ago
Labels:                 <none>
Annotations:            openshift.io/description=
                        openshift.io/display-name=
                        openshift.io/requester=demo1
                        openshift.io/sa.scc.mcs=s0:c11,c10
                        openshift.io/sa.scc.supplemental-groups=1000130000/10000
                        openshift.io/sa.scc.uid-range=1000130000/10000
Display Name:           <none>
Description:            <none>
Status:                 Active
Node Selector:          <none>
Quota:                  <none>
Resource limits:        <none>

You see, here openshift.io/sa.scc.uid-range start from 1000130000, is the UID of our busybox container.

SCCs are not granted directly to a project. Instead, you add a service account to an SCC and either specify the service account name on your pod or, when unspecified, run as the default service account.

Add and Remove SCC

Add service account default in project demo1-proj to SCC privileged

1
2
3
oc adm policy add-scc-to-user privileged system:serviceaccount:demo1-proj:default

scc "privileged" added to: ["system:serviceaccount:demo1-proj:default"]

Where to examine the result:

1
2
3
4
5
6
7
oc describe scc privileged

Name:                                           privileged
Priority:                                       <none>
Access:
  Users:                                        system:admin,system:serviceaccount:openshift-infra:build-controller,system:serviceaccount:management-infra:management-admin,system:serviceaccount:management-infra:inspector-admin,system:serviceaccount:glusterfs:default,system:serviceaccount:glusterfs:router,system:serviceaccount:glusterfs:heketi-storage-service-account,system:serviceaccount:demo1-proj:default
...

In the Users field, we now have system:serviceaccount:demo1-proj:default.

How to remove the SCC from a service account?

1
2
3
oc adm policy remove-scc-from-user privileged system:serviceaccount:demo1-proj:default

scc "privileged" removed from: ["system:serviceaccount:demo1-proj:default"]

Deploy Again

Then login as demo1, go to demo1-proj, deploy again:

1
oc apply -f bb-deploy.yml
1
2
3
4
oc exec -it bb-deployment-78bdb8c4f-rgj88 sh

/ $ id
uid=1000130000 gid=0(root) groups=1000130000

Why the UID is still 1000130000? We have applied privileged right? Because privileged is just a constraint, you need to Ensure that at least one of the pod’s containers is requesting a privileged mode in the security context.

So update the yaml file, add securityContext, and deploy again:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bb-deployment
  labels:
    app: bb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bb
  template:
    metadata:
      labels:
        app: bb
    spec:
      containers:
      - name: bb
        image: 172.30.159.11:5000/demo1-ds/busybox:v1
      securityContext:
        runAsUser: 0

Now if you check the UID, it’s 0:

1
2
3
4
oc rsh bb-deployment-58cb44b56b-zmdcw

/ # id
uid=0(root) gid=0(root) groups=10(wheel)

Note that oc rsh is the same as kubectl exec -it ... sh

Conclusion

Add privileged SCC to service account is not enough, need to specify runAsUser: 0 in yaml file.

0%