Rebinding Release PV with PVC

Posted on | Edited on | In

Reference Reuse existing Persistent Volume

Background Prod env upgrade failed, rolled back leads to historical monitoring data lost. The root cause is old PVC was removed accidently thus the corresponding PV got released, thanksfully the PV reclaim policy is retain so the data on disk was still preserved.

这里和以前遇到的不同的地方是使用了storage class, PV 定义中有claimRef 去做绑定.

Question How to access the historical data on released PV? 显而易见需要重新bound.

Solution The PV is actually provisioned dynamically by custom storage class gce-regional-ssd, in its definition, it is preserved for specific PVC by specifying the claimRef field:

1
2
3
4
5
6
7
claimRef:
  apiVersion: v1
  kind: PersistentVolumeClaim
  name: monitoring-alertmanager
  namespace: monitoring
  resourceVersion: "21865"
  uid: 4036d03f-fe2f-4d6f-bae8-dd67f33ad423

Since the PVC monitoring-alertmanager is alreay used another PV, to make this one available, kubectl edit to remove the uid and resourceVersion, modify the name, save and quit:

1
2
3
4
5
claimRef:
  apiVersion: v1
  kind: PersistentVolumeClaim
  name: monitoring-alertmanager-old
  namespace: monitoring

For now the PV becomes available only to PVC which is named monitoring-alertmanager-old. Or if you set claimRef to empty, PV will open to all PVCs.

Then creating that PVC to consume the PV (binding):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: monitoring
    component: alertmanager
  name: monitoring-alertmanager-old
  namespace: monitoring
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: gce-regional-ssd
  volumeMode: Filesystem

Then mount the PVC to your target resource to access the data.

其实如果describe PV, 可以找到在GCE中具体的Persistent Disk (under Compute Engine), 保险起见可以先snapshot 该persistent disk再做其他操作:

1
2
3
4
5
6
Source:
    Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName:     gke-us-east4-f3e3fedd--pvc-4036d03f-fe2f-4d6f-bae8-dd67f33ad423
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
0%