Golang K8s Proj1

Posted on | Edited on | In

Porj1 description: build a small go docker app to fetch and display service A’s token from specified GKE cluster through port forwarding. The arguments in token request path are from configMap in the same GKE cluster.

I structure this blog in the order of the problem solving in development process.

Caveat

The VSC may have hiccups on import of go packages(annoying errors at package import statement which prevents code intelligent suggestions from using), troubleshooting could be:

  1. check the package is downloaded/installed, run go get
  2. open VSC editor at Go project root directory rather than others
  3. make sure Go tools are up-to-date
  4. try move the Go project out of GOPATH, and set go.mod for it
  5. restart VSC editor

How to get GKE cluster kubeconfig file?

To talk to GKE cluster, the first thing to do is running the gcloud command:

1
gcloud container clusters get-credentials <cluster> --region <region> --project <project>

I don’t want to install gcloud CLI, alternatively letting golang do this job. The idea is to figure out how does gcloud command generate the kubeconfig, for example:

1
KUBECONFIG=kubeconfig.yml gcloud container clusters get-credentials <cluster> --region <region> --project <project>

The result would be a single cluster kuebconfig.yml file in the current directory, and export this KUBECONFIG will make subsequent kubectl commands work on the cluster specified in this yaml file.

To understand in depth I use gcloud option --log-http to dump command log:

1
gcloud container clusters get-credentials <cluster> --region <region> --project <project> --log-http

Displaying redacted log here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
=======================
==== request start ====
uri: https://oauth2.googleapis.com/token
method: POST
== headers start ==
# header info
== headers end ==
== body start ==
Body redacted: Contains oauth token. Set log_http_redact_token property to false to print the body of this request.
== body end ==
==== request end ====
---- response start ----
status: 200
-- headers start --
# header info
-- headers end --
-- body start --
Body redacted: Contains oauth token. Set log_http_redact_token property to false to print the body of this response.
-- body end --
total round trip time (request+response): 0.084 secs
---- response end ----

Fetching cluster endpoint and auth data.
=======================
==== request start ====
uri: https://container.googleapis.com/v1/projects/<cluster>/locations/us-west1/clusters/<cluster>?alt=json
method: GET
== headers start ==
# header info
== headers end ==
== body start ==
 
== body end ==
==== request end ====
---- response start ----
status: 200
-- headers start --
# header info
-- headers end --
-- body start --
# big json file which contains necessary data to generate kubeconfig
-- body end --
total round trip time (request+response): 0.067 secs
---- response end ----
----------------------
kubeconfig entry generated for <cluster>.

The first http call is about OAuth2.0, used to authorize caller’s request, the gcloud client will do this automatically for you if env variable GOOGLE_APPLICATION_CREDENTIALS is set correctly, for example:

1
export GOOGLE_APPLICATION_CREDENTIALS="path to adc.json or service account key file"

I will mount this credential file into docker when I run container.

Note that I use GOOGLE_APPLICATION_CREDENTIALS because the app runs outside of the gcloud environment, if it is inside the attached service account will be used. See demo about Authenticating as a service account.

Next I go to figure out how to run gcloud K8s go client to get the GKE cluster info, from the log dump above the URL is known as:

1
uri: https://container.googleapis.com/v1/projects/<cluster>/locations/us-west1/clusters/<cluster>?alt=json

It is gcloud K8s engine REST API v1, the live experiment can play here. Once you fill the name field and click the EXECUTE button(uncheck the API key), the OAuth2.0 will pop up and let you authorize the request then the call will succeed.

Next find the corresponding go client call for this REST API: Go Cloud Client Libraries => search in search bar Kubernetes Engine API => search related func name func (*ClusterManagerClient) GetCluster

Note that gcloud K8s go client is for managing GKE cluster not the K8s resources, it is not the k8s/go-client mentioned later.

From the sample code, the required field structure is the same as the REST API path:

1
2
3
// The name (project, location, cluster) of the cluster to retrieve.
// Specified in the format `projects/*/locations/*/clusters/*`.
Name string `protobuf:"bytes,5,opt,name=name,proto3" json:"name,omitempty"`

Now ready, I have GOOGLE_APPLICATION_CREDENTIALS exported and gcloud K8s go client library, it is easy to get what gcloud container clusters get-credentials does for us and make the kubeconfig yaml file from template.

What is OAuth2.0

The OAuth(open authorization) 2.0 google doc and example.

Usage: authorization between services, OAuth access token is JWT.

  • Intro OAuth, access delegation, limited access.
  • OAuth deeper: terms: resource, resource owner, resource server, client, authorization server(issue access token)
  • JWT explained, vs session token(reference token). The JWT(value token) contains the complete request info, that’s why it uses JSON object. Session token is just a key from a session map on the server side.
  • JWT structure explained, encode and decode JWT object

How to use K8s go client to access K8s resource?

Note that K8s go client(kubernetes/go-client) is a standalone project used to talk to K8s, K8s itself is another go project(kubernetes/kubernetes).

I have made the kubeconfig yaml file ready and set KUBECONFIG env variable, then using the client to do API call, for example reference code, can also reference the go client example for out-of-k8s cluster.

In my project I need to get date from configMap, use go client for configmap.

Tutorial

Youtube: Getting Started with Kubernetes client-go Youtube: client-go K8s native development

How to port forward in pure golang?

Next, I need to query a K8s service, to make it easy I need to forward port onto localhost, the kubectl command is:

1
2
# service port forward
kubectl port-forward svc/example 9200:9200 -v=8 &> verbose.txt

I add -v=8 flag to dump the log into verbose.txt file.

Then I see there are consecutive API calls, it first gets service detail(GET), then looking for pod(GET) and pod details(GET) managed by that service and uses that pod to do port forwarding(POST). So it actually does:

1
2
# pod port forward
kubectl port-forward example-0 9200:9200

The go client has port forward package, to use it, import as k8s.io/client-go/tools/portforward(just follows the dir path layout).

The usage of port forward package is not obvious, I reference below 2 posts to make it work:

The go channels(start and stop port forward) and goroutine will be used here, you can check lsof or netstat to see the target localhost port is listening.

Additionally, You can also see how kubectl implement port-forward

How to convert curl to golang?

Next, I need to convert curl to golang, it is easy as the underlying is all about http request.

There is a interesting project has exactly what I need: https://mholt.github.io/curl-to-go/

How to unmarshal only small set of fields from HTTP response JSON?

I find 2 options:

  1. Use struct type which has only the target fields, have to construct multiple struct and embed them to reflect the JSON field nesting.
  2. Use interface + type assertion to receive and convert the target field, no struct is needed!

For single JSON object, don’t use json.Decoder, please use json.Unmarshal instead.

How to reduce go docker image size?

Lastly, I want to build a go docker app to simplify use and has minimum image size.

The docker official doc for building go docker image is not good, end up with a big size image.

The solution is easy: using multi-stage Dockerfile, build go binary in one go docker image and copy the binary to a new base image of the same Linux distro but has much less image size:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# syntax=docker/dockerfile:1

# Need to have google-cloud-sdk-gke-gcloud-auth-plugin
# for go-client to access GKE resource
# https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
FROM golang:1.18-buster AS auth-plugin
# version > 381.0.0-0 asks to install gcloud cli
# which is not desired and make image size bigger!
ENV AUTH_VERSION="381.0.0-0"
RUN echo "deb https://packages.cloud.google.com/apt cloud-sdk main" \
    | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list \
    && \
    curl https://packages.cloud.google.com/apt/doc/apt-key.gpg \
    | apt-key add -

RUN apt-get update \
    && \
    apt-get install google-cloud-sdk-gke-gcloud-auth-plugin=$AUTH_VERSION

# build binary executable base on alpine image
FROM golang:1.18-alpine AS binary
WORKDIR /deployment
COPY cmd ./cmd
COPY go.mod ./
COPY go.sum ./
RUN go mod download
RUN go build -o ./cmd/app ./cmd

# minimum app image creation
FROM alpine:3.15
WORKDIR /deployment
COPY --from=binary \
    /deployment/cmd/app \
    ./cmd/app
COPY --from=auth-plugin \
    /usr/lib/google-cloud-sdk/bin/gke-gcloud-auth-plugin \
    /usr/local/bin/gke-gcloud-auth-plugin
COPY config ./config
COPY template ./template

WORKDIR /deployment/cmd
ENTRYPOINT [ "./app"]
CMD ["--help"]

This approach helps me reduce the docker image size from near 2GB to 60MB.

There is a post has similar idea: Build a super minimalistic Docker Image to run your Golang App.

0%