Helm Upgrade Failed Due to Deprecated K8s API

Posted on | Edited on | In

PodSecurityPolicy was removed from K8s since v1.25.

Our GKE cluster had been forcely upgraded to v1.25 from v1.24 and we forgot to retire the PodSecurityPolicy resources in the chart, so the subsequent helm upgrade from pipeline failed due to the disappearance of PodSecurityPolicy kind and version policy/v1beta1.

Even if I disabled the PodSecurityPolicy from the chart but the helm upgrade still would reference to the resource from existing release and the “not found” showed up:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[2023-07-08T01:12:20.712Z] ARGS:
[2023-07-08T01:12:20.712Z]   0: helm (4 bytes)
[2023-07-08T01:12:20.712Z]   1: upgrade (7 bytes)
[2023-07-08T01:12:20.712Z]   2: --install (9 bytes)
[2023-07-08T01:12:20.712Z]   3: cert-manager (12 bytes)
[2023-07-08T01:12:20.712Z]   4: gcloud-helm/cert-manager (24 bytes)
[2023-07-08T01:12:20.712Z]   5: --version (9 bytes)
[2023-07-08T01:12:20.712Z]   6: 1.0.5 (5 bytes)
[2023-07-08T01:12:20.712Z]   7: --wait (6 bytes)
[2023-07-08T01:12:20.712Z]   8: --create-namespace (18 bytes)
[2023-07-08T01:12:20.712Z]   9: --namespace (11 bytes)
[2023-07-08T01:12:20.712Z]   10: cert-manager (12 bytes)
[2023-07-08T01:12:20.712Z]   11: --values (8 bytes)
[2023-07-08T01:12:20.712Z]   12: /tmp/helmfile4095048642/cert-manager-cert-manager-values-6d48855bcc (67 bytes)
[2023-07-08T01:12:20.712Z]   13: --reset-values (14 bytes)
[2023-07-08T01:12:20.712Z]   14: --history-max (13 bytes)
[2023-07-08T01:12:20.712Z]   15: 10 (2 bytes)
[2023-07-08T01:12:20.712Z] 
[2023-07-08T01:12:20.712Z] ERROR:
[2023-07-08T01:12:20.712Z]   exit status 1
[2023-07-08T01:12:20.712Z] 
[2023-07-08T01:12:20.712Z] EXIT STATUS
[2023-07-08T01:12:20.712Z]   1
[2023-07-08T01:12:20.712Z] 
[2023-07-08T01:12:20.712Z] STDERR:
[2023-07-08T01:12:20.712Z]   Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest: [resource mapping not found for name: "xxx" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"
...
[2023-07-08T01:12:20.712Z]   ensure CRDs are installed first]
[2023-07-08T01:12:20.712Z] 
[2023-07-08T01:12:20.712Z] COMBINED OUTPUT:

[2023-07-08T01:12:20.712Z]   Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest: [resource mapping not found for name: "xxx" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"

The helm uninstall on existing release also failed and got stuck with the same reason:

1
helm uninstall <release name> -n <ns> --no-hooks

The fix is simple by a helm plugin mapkubeapis:

1
2
helm plugin install https://github.com/helm/helm-mapkubeapis
helm mapkubeapis <release name> -n <ns>
1
2
3
4
5
6
7
8
9
10
helm history <release name> -n <ns>

REVISION	UPDATED                 	STATUS         	CHART             	APP VERSION	DESCRIPTION
1       	Sat Nov 20 00:21:31 2021	superseded     	xxx-1.0.3	v1.4.1     	Install complete
2       	Thu Aug 11 10:30:45 2022	superseded     	xxxr-1.0.5	v1.7.3     	Upgrade complete
3       	Thu Aug 11 10:58:10 2022	superseded     	xxx-1.0.3	v1.4.1     	Upgrade complete
# helm uninstall failed and hung
4       	Mon Aug 15 23:30:33 2022	superseded     	xxx-1.0.5	v1.7.3     	Deletion in progress (or silently failed)
# mapkubeapis helped re-map the deprecated API
5       	Sat Jul  8 01:56:17 2023	deployed       	xxx-1.0.5	v1.7.3     	Kubernetes deprecated API upgrade - DO NOT rollback from this version

Then the next helm uninstall/upgrade should be good.

0%