RBAC Authorization for K8s API Access

Posted on | Edited on | In

Kubernetes version 1.13.2

We want to scale the compute pods by calling k8s API from inside the engine conductor container, this definitely need to be authorized and we need to grant privilege for this action.

There are some concepts you need to know in order to achieve the goal.

Service Account

what is Service Account

Processes in containers inside pods can contact the apiserver. When they do, they are authenticated as a particular service account (for example, by default is default service account).

Once you create a namespace, for example test-1, there is a default service account automatically generated.

1
kubectl get sa -n test-1
1
2
NAME      SECRETS   AGE
default   1         139m

let’s see what is inside the service account

1
kubectl describe sa default -n test-1
1
2
3
4
5
6
7
8
Name:                default
Namespace:           test-1
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   default-token-mtv4n
Tokens:              default-token-mtv4n
Events:              <none>

Here we see there is a mountable secret default-token-mtv4n, that is the credentials to access the apiserver.

1
kubectl describe secret default-token-mtv4n -n test-1
1
2
3
4
5
6
7
8
9
10
11
12
13
Name:         default-token-mtv4n
Namespace:    test-1
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: default
              kubernetes.io/service-account.uid: 387381d3-2272-11e9-91a2-00163e0196e7

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  6 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrd...

ClusterRole

what is Role and ClusterRole

A ClusterRole can be used to grant the same permissions as a Role, but because they are cluster-scoped, they can also be used to grant access to

  • cluster-scoped resources (like nodes)
  • non-resource endpoints (like “/healthz”)
  • namespaced resources (like pods) across all namespaces

Here we use a cluster role called cluster-admin, it’s generated by default

1
kubectl get clusterrole | grep cluster-admin
1
cluster-admin                        174m

ClusterRole Binding

what is RoleBinding and ClusterRole Binding

A role binding grants the permissions defined in a role to a user or set of users. It holds a list of subjects (users, groups, or service accounts), and a reference to the role being granted. Permissions can be granted within a namespace with a RoleBinding, or cluster-wide with a ClusterRoleBinding.

below we grant service account default in namespace test-1 the cluster-admin level privilege.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: <cluster role name>
  labels:
    <key:value>
subjects:
 - kind: ServiceAccount
   name: default
   namespace: <namespace of sa>
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

then we can write a script, using curl to call K8s API, for example, to scale the number of compute pods:

1
2
3
4
5
http_code=$(curl -w "%{http_code}" -sS  --cacert $CACERT  -XPATCH -H "Content-Type: application/strategic-merge-patch+json" -H "Accept: application/json" -H "Authorization: Bearer $TOKEN" "https://kubernetes.default/apis/apps/v1/namespaces/$NAMESPACE/statefulsets/is-engine-compute" --data "{\"spec\":{\"replicas\":$REP}}" -o $OUT_FILE)
if [[ $http_code -ne 200 ]]; then
    ${JQ} '{ result:.status, code: .code,  message: .message }' $OUT_FILE
    exit 1
fi

Where are these CACERT, TOEKN and NAMESPACE from? Actually each container has a default mount point reside in:

1
/var/run/secrets/kubernetes.io/serviceaccount

You can see this when you run kubectl describe pod. Just like other mount files, there are 3 files, for example:

1
2
3
4
total 0
lrwxrwxrwx 1 root root 13 Sep 14 16:21 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root 16 Sep 14 16:21 namespace -> ..data/namespace
lrwxrwxrwx 1 root root 12 Sep 14 16:21 token -> ..data/token

All of them are used in curl command above.

1
2
3
NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
0%