From 7bf5b507f7a9cec72f9ad74a962000b37260af21 Mon Sep 17 00:00:00 2001 From: Anshul Ahuja Date: Wed, 8 Feb 2023 16:31:24 +0530 Subject: [PATCH 01/28] Design to add support for Multiple VolumeSnapshotClasses in CSI Plugin Signed-off-by: Anshul Ahuja --- changelogs/unreleased/5774-anshulahuja98 | 1 + ...ultiple-csi-volumesnapshotclass-support.md | 216 ++++++++++++++++++ 2 files changed, 217 insertions(+) create mode 100644 changelogs/unreleased/5774-anshulahuja98 create mode 100644 design/multiple-csi-volumesnapshotclass-support.md diff --git a/changelogs/unreleased/5774-anshulahuja98 b/changelogs/unreleased/5774-anshulahuja98 new file mode 100644 index 000000000..00775f261 --- /dev/null +++ b/changelogs/unreleased/5774-anshulahuja98 @@ -0,0 +1 @@ +Design to add support for Multiple VolumeSnapshotClasses in CSI Plugin. \ No newline at end of file diff --git a/design/multiple-csi-volumesnapshotclass-support.md b/design/multiple-csi-volumesnapshotclass-support.md new file mode 100644 index 000000000..db17f1cae --- /dev/null +++ b/design/multiple-csi-volumesnapshotclass-support.md @@ -0,0 +1,216 @@ +# Proposal to add support for Multiple VolumeSnapshotClasses in CSI Plugin + +- [Proposal to add support for Multiple VolumeSnapshotClasses in CSI Plugin](#proposal-to-add-support-for-multiple-volumesnapshotclasses-in-csi-plugin) + - [Abstract](#abstract) + - [Background](#background) + - [Goals](#goals) + - [Non Goals](#non-goals) + - [Detailed Design](#detailed-design) + - [Plugin Inputs Contract Changes](#plugin-inputs-contract-changes) + - [Using Plugin Inputs for CSI Plugin](#using-plugin-inputs-for-csi-plugin) + - [Annotations overrides on PVC for CSI Plugin](#annotations-overrides-on-pvc-for-csi-plugin) + - [Using Plugin Inputs for Other Plugins](#using-plugin-inputs-for-other-plugins) + - [Alternatives Considered](#alternatives-considered) + - [Security Considerations](#security-considerations) + - [Compatibility](#compatibility) + - [Implementation](#implementation) + - [Open Issues](#open-issues) + + +## Abstract +Currently the Velero CSI plugin chooses the VolumeSnapshotClass in the cluster that has the same driver name and also has the velero.io/csi-volumesnapshot-class label set on it. This global selection is not sufficient for many use cases. This proposal is to add support for multiple VolumeSnapshotClasses in CSI Plugin where the user can specify the VolumeSnapshotClass to use for a particular driver and backup. + + +## Background +The Velero CSI plugin chooses the VolumeSnapshotClass in the cluster that has the same driver name and also has the velero.io/csi-volumesnapshot-class label set on it. This global selection is not sufficient for many use cases. For example, if a cluster has multiple VolumeSnapshotClasses for the same driver, the user may want to use a VolumeSnapshotClass that is different from the default one. The user might also have different schedules set up for backing up different parts of the cluster and might wish to use different VolumeSnapshotClasses for each of these backups. + +## Goals +- Allow the user to specify the VolumeSnapshotClass to use for a particular driver and backup. + +## Non Goals +- + + +## Detailed Design + +### Plugin Inputs Contract Changes +Approach is to introduce a new field `pluginInputs` in the velero CRs (Backup, Schedule, Restore). This field can be leveraged by all plugins for sending plugin specific settings rather than relying on annotations or global settings which hold across backups. + +```go +type PluginInput struct { + Name string `json:"name"` + Properties map[string][string] `json:"properties"` +} +``` + +### Using Plugin Inputs for CSI Plugin +The user can specify the VolumeSnapshotClass to use for a particular driver and backup using the plugin inputs. The CSI plugin will use the VolumeSnapshotClass specified in the plugin inputs. If the VolumeSnapshotClass is not specified for a driver, the CSI plugin will use the default VolumeSnapshotClass for the driver fetched using labels through older route. + +Example: +```yaml +apiVersion: velero.io/v1 +kind: Backup +metadata: + name: backup-1 +spec: + pluginInputs: + - name: velero.io/csi + - properties: + - key: csi.cloud.disk.driver + - value: csi-diskdriver-snapclass + - key: csi.cloud.file.driver + - value: csi-filedriver-snapclass +``` + +CLI Example + +```bash +velero backup create my-backup --plugin-inputs velero.io/csi:csi.cloud.disk.driver=csi-diskdriver-snapclass,csi.cloud.file.driver=csi-filedriver-snapclass +``` + +### Annotations overrides on PVC for CSI Plugin +The user can annotate the PVCs with VolumeSnapshotClass name. This will override whatever the user has passed in pluginInputs for that driver. + +- If annotation is not present or VolumeSnapshotClass referred is not present in cluster OR if the specified VSC does not have the same CSI driver as the PVC + - the CSI plugin will try to fallback to the pluginInputs value for that driver. If pluginInputs does not have the VSC for that driver, the CSI plugin will use the default VolumeSnapshotClass for the driver using the older label route. + +Example: +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: +name: pvc-1 +annotations: + velero.io/csi-volumesnapshot-class/disk.fg: csi-diskdriver-snapclass +``` + +### Using Plugin Inputs for Other Plugins +As of today various plugins such as StorageClass mapping plugin, use a global configmap which is discovered through labels/ annotations which is not ideal customer experience and prohibhit usage of specific settings for each backup/ schedule setup. + +The pluginInputs field can be used to pass in the reference the configmap/ secret to use for a particular plugin. This will allow the user to use different configmaps/ secrets for different backups/ schedules. + +Example: +```yaml +spec: + pluginInputs: + - name: velero.io/storageclass + - properties: + - key: storageclass-mapping-configmap + - value: configMapNamespace/configMapName +``` + + + +## Alternatives Considered + +1. **Through Annotations** + 1. **Support VolumeSnapshotClass selection at PVC level** + The user can annotate the PVCs with driver and VolumeSnapshotClass name. The CSI plugin will use the VolumeSnapshotClass specified in the annotation. If the annotation is not present, the CSI plugin will use the default VolumeSnapshotClass for the driver. If the VolumeSNapshotClass provided is of a different driver, the CSI plugin will use the default VolumeSnapshotClass for the driver. + + *example annotation on PVC:* + ```yaml + apiVersion: v1 + kind: PersistentVolumeClaim + metadata: + name: pvc-1 + annotations: + velero.io/csi-volumesnapshot-class: csi-diskdriver-snapclass + + ``` + + 2. **Support VolumeSnapshotClass selection at backup/schedule level** + The user can annotate the backup/ schedule with driver and VolumeSnapshotClass name. The CSI plugin will use the VolumeSnapshotClass specified in the annotation. If the annotation is not present, the CSI plugin will use the default VolumeSnapshotClass for the driver. + + *example annotation on backup/schedule:* + ```yaml + apiVersion: velero.io/v1 + kind: Backup + metadata: + name: backup-1 + annotations: + velero.io/csi-volumesnapshot-class/csi.cloud.disk.driver: csi-diskdriver-snapclass + velero.io/csi-volumesnapshot-class/csi.cloud.file.driver: csi-filedriver-snapclass + velero.io/csi-volumesnapshot-class/: csi-snapclass + ``` + + To query the annotations on a backup: "velero.io/csi-volumesnapshot-class/'driver name'" - where driver names comes from the PVC's driver. + + **Limitations of Annotations approach**: + - The user has to annotate the PVCs or backups with the VolumeSnapshotClass to use for each driver. This is not ideal for the user experience. + - Mitigation: We can extend Velero CLI to also annotate backups/schedules with the VolumeSnapshotClass to use for each driver. This will make it easier for the user to annotate the backups/schedules. This mitigation is not for the PVCs though, since PVCs is anyways a specific use case. + + +1. **Through CSI Specific Fields in Velero contracts** + + **Considerations** + - Since CSI snapshotting is done through the plugin, we don't intend to bloat up the Backup Spec with CSI specific fields. + - But considering that CSI Snapshotting is the way forward, we can debate if we should add a CSI section to the Backup Spec. + + + **Approach**: Similar to VolumeSnapshotLocation param in the Backup Spec, we can add a VolumeSnapshotClass param in the Backup Spec. This will allow the user to specify the VolumeSnapshotClass to use for the backup. The CSI plugin will use the VolumeSnapshotClass specified in the Backup Spec. If the VolumeSnapshotClass is not specified, the CSI plugin will use the default VolumeSnapshotClass for the driver. + + *example of VolumeSnapshotClass param in the Backup Spec:* + ```yaml + apiVersion: velero.io/v1 + kind: Backup + metadata: + name: backup-1 + spec: + csiParameters: + volumeSnapshotClasses: + driver: csi.cloud.disk.driver + snapClass: csi-diskdriver-snapclass + timeout: 10m + ``` + +1. **Through changes in velero contracts** + 1. **Through configmap references.** + Currently even the storageclass mapping plugin expects the user to create a configmap which is used globally, and fetched through labels. This behaviour has same issue as the VolumeSnapshotClass selection. We can introduce a field in the velero contracts which allow passing configmap references for each plugin. And then the plugin can honour the configmap passed in as reference. The configmap can be used to pass the VolumeSnapshotClass to use for the backup, and also other parameters to tweak. This can help in making plugins more flexible while not depending on global behaviour. + + + *example of configmap reference in the velero contracts:* + ```yaml + apiVersion: velero.io/v1 + kind: Backup + metadata: + name: backup-1 + spec: + configmapRefs: + - name: csi-volumesnapshotclass-configmap + - namespace: velero + - plugin: velero.io/csi + ``` + + 2. **Through generic property bag in the velero contracts**: We can introduce a field in the velero contracts which allow passing a generic property bag for each plugin. And then the plugin can honour the property bag passed in. + + + *example of property bag in the velero contracts:* + ```yaml + apiVersion: velero.io/v1 + kind: Backup + metadata: + name: backup-1 + spec: + pluginInputs: + - name: velero.io/csi + - properties: + - key: csi.cloud.disk.driver + - value: csi-diskdriver-snapclass + - key: csi.cloud.file.driver + - value: csi-filedriver-snapclass + ``` + + **Note**: Both these approaches can also be used to tweak other parameters such as CSI Snapshotting Timeout/intervals. And further can be used by other plugins. + + +## Security Considerations +No security impact. + +## Compatibility +Existing behaviour of csi plugin will be retained where it fetches the VolumeSnapshotClass through the label. This will be the default behaviour if the user does not specify the VolumeSnapshotClass. + +## Implementation +TBD based on closure of high level design proposals. + +## Open Issues +NA From 0b243bc4bc277347c75b00e2a30652e3a5ee4892 Mon Sep 17 00:00:00 2001 From: Anshul Ahuja Date: Tue, 14 Feb 2023 16:32:36 +0530 Subject: [PATCH 02/28] Address PR feedback Signed-off-by: Anshul Ahuja --- ...ultiple-csi-volumesnapshotclass-support.md | 25 ++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/design/multiple-csi-volumesnapshotclass-support.md b/design/multiple-csi-volumesnapshotclass-support.md index db17f1cae..f3879374a 100644 --- a/design/multiple-csi-volumesnapshotclass-support.md +++ b/design/multiple-csi-volumesnapshotclass-support.md @@ -5,6 +5,9 @@ - [Background](#background) - [Goals](#goals) - [Non Goals](#non-goals) + - [User Stories](#user-stories) + - [Scenario 1](#scenario-1) + - [Scenario 2](#scenario-2) - [Detailed Design](#detailed-design) - [Plugin Inputs Contract Changes](#plugin-inputs-contract-changes) - [Using Plugin Inputs for CSI Plugin](#using-plugin-inputs-for-csi-plugin) @@ -26,9 +29,27 @@ The Velero CSI plugin chooses the VolumeSnapshotClass in the cluster that has th ## Goals - Allow the user to specify the VolumeSnapshotClass to use for a particular driver and backup. +- Add pluginInputs field in the velero CRs (Backup, Schedule, Restore) to allow the user to specify the parameters which are specific to a plugin. ## Non Goals -- +- Deprecating existing VSC selection behaviour. (The current behaviour will remain the default behaviour if the user does not specify the VolumeSnapshotClass to use for a particular driver and backup.) + + +## User Stories + +### Scenario 1 +- Consider Alice is a cluster admin and has a cluster with multiple VolumeSnapshotClasses for the same driver. Each VSC stores the snapshots taken in different ResourceGroup(Azure equivalent). +- Alice has configured multiple scheduled backups each covering a different set of namespaces, representing different apps owned by different teams. +- Alice wants to use a different VolumeSnapshotClass for each backup such that each snapshot goes in it's respective ResourceGroup to simply management of snapshots(COGS, RBAC etc). +- In current velero, Alice can't achieve this as the CSI plugin will use the default VolumeSnapshotClass for the driver and all snapshots will go in the same ResourceGroup. +- Proposed design will allow Alice to achieve this by specifying the VolumeSnapshotClass to use for a particular driver and backup/schedule. + +## Scenario 2 +- Bob is a cluster admin has PVCs storing different types of data. +- Most of the PVCs are used for storing non senstive application data. But certain PVCs store critical financial data. +- For such PVCs Bob wants to use a VolumeSnapshotClass with certain encryption related parameters set. +- In current velero, Bob can't achieve this as the CSI plugin will use the default VolumeSnapshotClass for the driver and all snapshots will be taken using the same VolumeSnapshotClass. +- Proposed design will allow Bob to achieve this by overriding the VolumeSnapshotClass to use for a particular driver and backup/schedule using annotations on those specific PVCs. ## Detailed Design @@ -89,6 +110,8 @@ As of today various plugins such as StorageClass mapping plugin, use a global co The pluginInputs field can be used to pass in the reference the configmap/ secret to use for a particular plugin. This will allow the user to use different configmaps/ secrets for different backups/ schedules. +*Note*: The rule of thumb for plugin owners which leverage the pluginInputs field is to use the value specified in the pluginInputs field if present, else fallback to the global configmap/ secret / annotation based discovery which was existing behaviour or the plugin. + Example: ```yaml spec: From 2f3fa9699fd2b7c273f916525f93ea4345e79730 Mon Sep 17 00:00:00 2001 From: Anshul Ahuja Date: Tue, 14 Feb 2023 16:49:35 +0530 Subject: [PATCH 03/28] Spelling fix Signed-off-by: Anshul Ahuja --- design/multiple-csi-volumesnapshotclass-support.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design/multiple-csi-volumesnapshotclass-support.md b/design/multiple-csi-volumesnapshotclass-support.md index f3879374a..9f8507beb 100644 --- a/design/multiple-csi-volumesnapshotclass-support.md +++ b/design/multiple-csi-volumesnapshotclass-support.md @@ -46,7 +46,7 @@ The Velero CSI plugin chooses the VolumeSnapshotClass in the cluster that has th ## Scenario 2 - Bob is a cluster admin has PVCs storing different types of data. -- Most of the PVCs are used for storing non senstive application data. But certain PVCs store critical financial data. +- Most of the PVCs are used for storing non-sensitive application data. But certain PVCs store critical financial data. - For such PVCs Bob wants to use a VolumeSnapshotClass with certain encryption related parameters set. - In current velero, Bob can't achieve this as the CSI plugin will use the default VolumeSnapshotClass for the driver and all snapshots will be taken using the same VolumeSnapshotClass. - Proposed design will allow Bob to achieve this by overriding the VolumeSnapshotClass to use for a particular driver and backup/schedule using annotations on those specific PVCs. From c9ae1d4dc27e2b66e3636445d3245d6472452e64 Mon Sep 17 00:00:00 2001 From: Anshul Ahuja Date: Tue, 14 Mar 2023 09:29:38 +0000 Subject: [PATCH 04/28] change approach Signed-off-by: Anshul Ahuja --- ...ultiple-csi-volumesnapshotclass-support.md | 116 ++++-------------- 1 file changed, 26 insertions(+), 90 deletions(-) diff --git a/design/multiple-csi-volumesnapshotclass-support.md b/design/multiple-csi-volumesnapshotclass-support.md index 9f8507beb..075a474dd 100644 --- a/design/multiple-csi-volumesnapshotclass-support.md +++ b/design/multiple-csi-volumesnapshotclass-support.md @@ -29,7 +29,6 @@ The Velero CSI plugin chooses the VolumeSnapshotClass in the cluster that has th ## Goals - Allow the user to specify the VolumeSnapshotClass to use for a particular driver and backup. -- Add pluginInputs field in the velero CRs (Backup, Schedule, Restore) to allow the user to specify the parameters which are specific to a plugin. ## Non Goals - Deprecating existing VSC selection behaviour. (The current behaviour will remain the default behaviour if the user does not specify the VolumeSnapshotClass to use for a particular driver and backup.) @@ -54,94 +53,11 @@ The Velero CSI plugin chooses the VolumeSnapshotClass in the cluster that has th ## Detailed Design -### Plugin Inputs Contract Changes -Approach is to introduce a new field `pluginInputs` in the velero CRs (Backup, Schedule, Restore). This field can be leveraged by all plugins for sending plugin specific settings rather than relying on annotations or global settings which hold across backups. +### Staged Approach: -```go -type PluginInput struct { - Name string `json:"name"` - Properties map[string][string] `json:"properties"` -} -``` - -### Using Plugin Inputs for CSI Plugin -The user can specify the VolumeSnapshotClass to use for a particular driver and backup using the plugin inputs. The CSI plugin will use the VolumeSnapshotClass specified in the plugin inputs. If the VolumeSnapshotClass is not specified for a driver, the CSI plugin will use the default VolumeSnapshotClass for the driver fetched using labels through older route. - -Example: -```yaml -apiVersion: velero.io/v1 -kind: Backup -metadata: - name: backup-1 -spec: - pluginInputs: - - name: velero.io/csi - - properties: - - key: csi.cloud.disk.driver - - value: csi-diskdriver-snapclass - - key: csi.cloud.file.driver - - value: csi-filedriver-snapclass -``` - -CLI Example - -```bash -velero backup create my-backup --plugin-inputs velero.io/csi:csi.cloud.disk.driver=csi-diskdriver-snapclass,csi.cloud.file.driver=csi-filedriver-snapclass -``` - -### Annotations overrides on PVC for CSI Plugin -The user can annotate the PVCs with VolumeSnapshotClass name. This will override whatever the user has passed in pluginInputs for that driver. - -- If annotation is not present or VolumeSnapshotClass referred is not present in cluster OR if the specified VSC does not have the same CSI driver as the PVC - - the CSI plugin will try to fallback to the pluginInputs value for that driver. If pluginInputs does not have the VSC for that driver, the CSI plugin will use the default VolumeSnapshotClass for the driver using the older label route. - -Example: -```yaml -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: -name: pvc-1 -annotations: - velero.io/csi-volumesnapshot-class/disk.fg: csi-diskdriver-snapclass -``` - -### Using Plugin Inputs for Other Plugins -As of today various plugins such as StorageClass mapping plugin, use a global configmap which is discovered through labels/ annotations which is not ideal customer experience and prohibhit usage of specific settings for each backup/ schedule setup. - -The pluginInputs field can be used to pass in the reference the configmap/ secret to use for a particular plugin. This will allow the user to use different configmaps/ secrets for different backups/ schedules. - -*Note*: The rule of thumb for plugin owners which leverage the pluginInputs field is to use the value specified in the pluginInputs field if present, else fallback to the global configmap/ secret / annotation based discovery which was existing behaviour or the plugin. - -Example: -```yaml -spec: - pluginInputs: - - name: velero.io/storageclass - - properties: - - key: storageclass-mapping-configmap - - value: configMapNamespace/configMapName -``` - - - -## Alternatives Considered - -1. **Through Annotations** - 1. **Support VolumeSnapshotClass selection at PVC level** - The user can annotate the PVCs with driver and VolumeSnapshotClass name. The CSI plugin will use the VolumeSnapshotClass specified in the annotation. If the annotation is not present, the CSI plugin will use the default VolumeSnapshotClass for the driver. If the VolumeSNapshotClass provided is of a different driver, the CSI plugin will use the default VolumeSnapshotClass for the driver. - - *example annotation on PVC:* - ```yaml - apiVersion: v1 - kind: PersistentVolumeClaim - metadata: - name: pvc-1 - annotations: - velero.io/csi-volumesnapshot-class: csi-diskdriver-snapclass - - ``` - - 2. **Support VolumeSnapshotClass selection at backup/schedule level** +### Stage 1 Approach +#### Through Annotations + 1. **Support VolumeSnapshotClass selection at backup/schedule level** The user can annotate the backup/ schedule with driver and VolumeSnapshotClass name. The CSI plugin will use the VolumeSnapshotClass specified in the annotation. If the annotation is not present, the CSI plugin will use the default VolumeSnapshotClass for the driver. *example annotation on backup/schedule:* @@ -158,11 +74,31 @@ spec: To query the annotations on a backup: "velero.io/csi-volumesnapshot-class/'driver name'" - where driver names comes from the PVC's driver. - **Limitations of Annotations approach**: + 2. **Support VolumeSnapshotClass selection at PVC level** + The user can annotate the PVCs with driver and VolumeSnapshotClass name. The CSI plugin will use the VolumeSnapshotClass specified in the annotation. If the annotation is not present, the CSI plugin will use the default VolumeSnapshotClass for the driver. If the VolumeSnapshotClass provided is of a different driver, the CSI plugin will use the default VolumeSnapshotClass for the driver. + + *example annotation on PVC:* + ```yaml + apiVersion: v1 + kind: PersistentVolumeClaim + metadata: + name: pvc-1 + annotations: + velero.io/csi-volumesnapshot-class: csi-diskdriver-snapclass + + ``` + - The user has to annotate the PVCs or backups with the VolumeSnapshotClass to use for each driver. This is not ideal for the user experience. - - Mitigation: We can extend Velero CLI to also annotate backups/schedules with the VolumeSnapshotClass to use for each driver. This will make it easier for the user to annotate the backups/schedules. This mitigation is not for the PVCs though, since PVCs is anyways a specific use case. + - Mitigation: We can extend Velero CLI to also annotate backups/schedules with the VolumeSnapshotClass to use for each driver. This will make it easier for the user to annotate the backups/schedules. This mitigation is not for the PVCs though, since PVCs is anyways a specific use case. Similar to : " kubectl run --image myimage --annotations="foo=bar" --annotations="another=one" mypod" + We can add support for - velero backup create my-backup --annotations "velero.io/csi:csi.cloud.disk.driver=csi-diskdriver-snapclass" + +### Stage 2 Approach +The above annotations route is to get started and for initial design closure/ implementation, north star is to either introduce CSI specific fields (considering that CSI might be a very core part of velero going forward) in the backup/restore CR OR leverage the pluginInputs field as being tracked in: https://github.com/vmware-tanzu/velero/pull/5981 + +Refer section Alternatives 2. **Through generic property bag in the velero contracts**: in the design doc for more details on the pluginInputs field. +## Alternatives Considered 1. **Through CSI Specific Fields in Velero contracts** **Considerations** From 22c1f9f3d6d9adc92857edf8bbb9a90a01b425e8 Mon Sep 17 00:00:00 2001 From: Anshul Ahuja Date: Tue, 14 Mar 2023 09:34:12 +0000 Subject: [PATCH 05/28] cleanup Signed-off-by: Anshul Ahuja --- design/multiple-csi-volumesnapshotclass-support.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/design/multiple-csi-volumesnapshotclass-support.md b/design/multiple-csi-volumesnapshotclass-support.md index 075a474dd..cfe7a666e 100644 --- a/design/multiple-csi-volumesnapshotclass-support.md +++ b/design/multiple-csi-volumesnapshotclass-support.md @@ -87,10 +87,12 @@ The Velero CSI plugin chooses the VolumeSnapshotClass in the cluster that has th velero.io/csi-volumesnapshot-class: csi-diskdriver-snapclass ``` + + Consider this as a override option in conjunction to part 1. - - The user has to annotate the PVCs or backups with the VolumeSnapshotClass to use for each driver. This is not ideal for the user experience. - - Mitigation: We can extend Velero CLI to also annotate backups/schedules with the VolumeSnapshotClass to use for each driver. This will make it easier for the user to annotate the backups/schedules. This mitigation is not for the PVCs though, since PVCs is anyways a specific use case. Similar to : " kubectl run --image myimage --annotations="foo=bar" --annotations="another=one" mypod" - We can add support for - velero backup create my-backup --annotations "velero.io/csi:csi.cloud.disk.driver=csi-diskdriver-snapclass" +**Note**: The user has to annotate the PVCs or backups with the VolumeSnapshotClass to use for each driver. This is not ideal for the user experience. + - **Mitigation**: We can extend Velero CLI to also annotate backups/schedules with the VolumeSnapshotClass to use for each driver. This will make it easier for the user to annotate the backups/schedules. This mitigation is not for the PVCs though, since PVCs is anyways a specific use case. Similar to : " kubectl run --image myimage --annotations="foo=bar" --annotations="another=one" mypod" + We can add support for - velero backup create my-backup --annotations "velero.io/csi:csi.cloud.disk.driver=csi-diskdriver-snapclass" ### Stage 2 Approach The above annotations route is to get started and for initial design closure/ implementation, north star is to either introduce CSI specific fields (considering that CSI might be a very core part of velero going forward) in the backup/restore CR OR leverage the pluginInputs field as being tracked in: https://github.com/vmware-tanzu/velero/pull/5981 From 9c0562cb94316dc48c2c9b703c4a403928917ded Mon Sep 17 00:00:00 2001 From: Mateus Oliveira Date: Mon, 19 Jun 2023 10:49:25 -0300 Subject: [PATCH 06/28] fix: Lastest release link in website Signed-off-by: Mateus Oliveira --- Makefile | 2 +- site/Dockerfile | 6 +----- site/content/_index.md | 2 +- 3 files changed, 3 insertions(+), 7 deletions(-) diff --git a/Makefile b/Makefile index d188df41e..82b524106 100644 --- a/Makefile +++ b/Makefile @@ -344,7 +344,7 @@ serve-docs: build-image-hugo -v "$$(pwd)/site:/srv/hugo" \ -it -p 1313:1313 \ $(HUGO_IMAGE) \ - hugo server --bind=0.0.0.0 --enableGitInfo=false + server --bind=0.0.0.0 --enableGitInfo=false # gen-docs generates a new versioned docs directory under site/content/docs. # Please read the documentation in the script for instructions on how to use it. gen-docs: diff --git a/site/Dockerfile b/site/Dockerfile index 0915036ac..b39592c5e 100644 --- a/site/Dockerfile +++ b/site/Dockerfile @@ -1,9 +1,5 @@ -FROM ubuntu:20.04 - -RUN apt update -RUN apt install -y hugo +FROM klakegg/hugo:0.73.0-ext-ubuntu WORKDIR /srv/hugo EXPOSE 1313 - diff --git a/site/content/_index.md b/site/content/_index.md index 5689636e6..b4802d463 100644 --- a/site/content/_index.md +++ b/site/content/_index.md @@ -10,7 +10,7 @@ hero: content: Velero is an open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes. cta_link1: text: Latest Release Information - url: /blog/velero-1.6-bring-all-your-credentials/ + url: /blog/Velero-1.11/ cta_link2: text: Download Velero url: https://github.com/vmware-tanzu/velero/releases/latest From ee27cde39156ee5ebd1755edcbd6ed0ea25091fd Mon Sep 17 00:00:00 2001 From: Mateus Oliveira Date: Mon, 19 Jun 2023 10:54:18 -0300 Subject: [PATCH 07/28] fixup! fix: Lastest release link in website Signed-off-by: Mateus Oliveira --- site/content/docs/main/release-instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/docs/main/release-instructions.md b/site/content/docs/main/release-instructions.md index c8eb317dd..014653979 100644 --- a/site/content/docs/main/release-instructions.md +++ b/site/content/docs/main/release-instructions.md @@ -159,7 +159,7 @@ What to include in a release blog: Release blog post PR: * Prepare a PR containing the release blog post. Read the [website guidelines][2] for more information on creating a blog post. It's usually easiest to make a copy of the most recent existing post, then replace the content as appropriate. -* You also need to update `site/index.html` to have "Latest Release Information" contain a link to the new post. +* You also need to update `site/content/_index.md` to have "Latest Release Information" contain a link to the new post. * Plan to publish the blog post the same day as the release. ## Announce a release From 05da96384a59f65000ccf1ab0fff1b190114f833 Mon Sep 17 00:00:00 2001 From: Peter Pan Date: Tue, 20 Jun 2023 05:42:20 -0400 Subject: [PATCH 08/28] fix 404 link Signed-off-by: Peter Pan --- design/Implemented/csi-snapshots.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/design/Implemented/csi-snapshots.md b/design/Implemented/csi-snapshots.md index 36040b260..9b9a3d976 100644 --- a/design/Implemented/csi-snapshots.md +++ b/design/Implemented/csi-snapshots.md @@ -304,8 +304,8 @@ Without these objects, the provider-level snapshots cannot be located in order t [1]: https://kubernetes.io/blog/2018/10/09/introducing-volume-snapshot-alpha-for-kubernetes/ -[2]: https://github.com/kubernetes-csi/external-snapshotter/blob/master/pkg/apis/volumesnapshot/v1alpha1/types.go#L41 -[3]: https://github.com/kubernetes-csi/external-snapshotter/blob/master/pkg/apis/volumesnapshot/v1alpha1/types.go#L161 +[2]: https://github.com/kubernetes-csi/external-snapshotter/blob/master/client/apis/volumesnapshot/v1/types.go#L42 +[3]: https://github.com/kubernetes-csi/external-snapshotter/blob/master/client/apis/volumesnapshot/v1/types.go#L262 [4]: https://github.com/heptio/velero/blob/main/pkg/volume/snapshot.go#L21 [5]: https://github.com/heptio/velero/blob/main/pkg/apis/velero/v1/pod_volume_backup.go#L88 [6]: https://github.com/heptio/velero-csi-plugin/ From 4cea533865bf74afe37412a061c2d893c56c6bce Mon Sep 17 00:00:00 2001 From: Daniel Jiang Date: Mon, 26 Jun 2023 15:11:27 +0800 Subject: [PATCH 09/28] Add more unit test cases for cmd/util/output Signed-off-by: Daniel Jiang --- pkg/builder/delete_backup_request_builder.go | 59 +++++ pkg/builder/item_operation_builder.go | 193 ++++++++++++++ pkg/builder/object_meta.go | 7 + pkg/builder/pod_volume_backup_builder.go | 1 - pkg/builder/pod_volume_restore_builder.go | 83 ++++++ pkg/cmd/util/output/backup_describer_test.go | 241 +++++++++++++++++- .../backup_structured_describer_test.go | 208 ++++++++++++++- pkg/cmd/util/output/describe.go | 9 - pkg/cmd/util/output/describe_test.go | 34 +++ pkg/cmd/util/output/restore_describer_test.go | 183 +++++++++++++ pkg/cmd/util/output/schedule_describe_test.go | 123 +++++++++ 11 files changed, 1123 insertions(+), 18 deletions(-) create mode 100644 pkg/builder/delete_backup_request_builder.go create mode 100644 pkg/builder/item_operation_builder.go create mode 100644 pkg/builder/pod_volume_restore_builder.go create mode 100644 pkg/cmd/util/output/restore_describer_test.go create mode 100644 pkg/cmd/util/output/schedule_describe_test.go diff --git a/pkg/builder/delete_backup_request_builder.go b/pkg/builder/delete_backup_request_builder.go new file mode 100644 index 000000000..4788795a3 --- /dev/null +++ b/pkg/builder/delete_backup_request_builder.go @@ -0,0 +1,59 @@ +package builder + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" +) + +// DeleteBackupRequestBuilder builds DeleteBackupRequest objects +type DeleteBackupRequestBuilder struct { + object *velerov1api.DeleteBackupRequest +} + +// ForDeleteBackupRequest is the constructor for a DeleteBackupRequestBuilder. +func ForDeleteBackupRequest(ns, name string) *DeleteBackupRequestBuilder { + return &DeleteBackupRequestBuilder{ + object: &velerov1api.DeleteBackupRequest{ + TypeMeta: metav1.TypeMeta{ + APIVersion: velerov1api.SchemeGroupVersion.String(), + Kind: "DeleteBackupRequest", + }, + ObjectMeta: metav1.ObjectMeta{ + Namespace: ns, + Name: name, + }, + }, + } +} + +// Result returns the built DeleteBackupRequest. +func (b *DeleteBackupRequestBuilder) Result() *velerov1api.DeleteBackupRequest { + return b.object +} + +// ObjectMeta applies functional options to the DeleteBackupRequest's ObjectMeta. +func (b *DeleteBackupRequestBuilder) ObjectMeta(opts ...ObjectMetaOpt) *DeleteBackupRequestBuilder { + for _, opt := range opts { + opt(b.object) + } + return b +} + +// BackupName sets the DeleteBackupRequest's backup name. +func (b *DeleteBackupRequestBuilder) BackupName(name string) *DeleteBackupRequestBuilder { + b.object.Spec.BackupName = name + return b +} + +// Phase sets the DeleteBackupRequest's phase. +func (b *DeleteBackupRequestBuilder) Phase(phase velerov1api.DeleteBackupRequestPhase) *DeleteBackupRequestBuilder { + b.object.Status.Phase = phase + return b +} + +// Errors sets the DeleteBackupRequest's errors. +func (b *DeleteBackupRequestBuilder) Errors(errors ...string) *DeleteBackupRequestBuilder { + b.object.Status.Errors = errors + return b +} diff --git a/pkg/builder/item_operation_builder.go b/pkg/builder/item_operation_builder.go new file mode 100644 index 000000000..8ca9d7506 --- /dev/null +++ b/pkg/builder/item_operation_builder.go @@ -0,0 +1,193 @@ +package builder + +import ( + "time" + + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime/schema" + + "github.com/vmware-tanzu/velero/pkg/itemoperation" + "github.com/vmware-tanzu/velero/pkg/plugin/velero" +) + +// OperationStatusBuilder builds OperationStatus objects +type OperationStatusBuilder struct { + object *itemoperation.OperationStatus +} + +// ForOperationStatus is the constructor for a OperationStatusBuilder. +func ForOperationStatus() *OperationStatusBuilder { + return &OperationStatusBuilder{ + object: &itemoperation.OperationStatus{}, + } +} + +// Result returns the built OperationStatus. +func (osb *OperationStatusBuilder) Result() *itemoperation.OperationStatus { + return osb.object +} + +// Phase sets the OperationStatus's phase. +func (osb *OperationStatusBuilder) Phase(phase itemoperation.OperationPhase) *OperationStatusBuilder { + osb.object.Phase = phase + return osb +} + +// Error sets the OperationStatus's error. +func (osb *OperationStatusBuilder) Error(err string) *OperationStatusBuilder { + osb.object.Error = err + return osb +} + +// Progress sets the OperationStatus's progress. +func (osb *OperationStatusBuilder) Progress(nComplete int64, nTotal int64, operationUnits string) *OperationStatusBuilder { + osb.object.NCompleted = nComplete + osb.object.NTotal = nTotal + osb.object.OperationUnits = operationUnits + return osb +} + +// Description sets the OperationStatus's description. +func (osb *OperationStatusBuilder) Description(desc string) *OperationStatusBuilder { + osb.object.Description = desc + return osb +} + +// Created sets the OperationStatus's creation timestamp. +func (osb *OperationStatusBuilder) Created(t time.Time) *OperationStatusBuilder { + osb.object.Created = &metav1.Time{Time: t} + return osb +} + +// Updated sets the OperationStatus's last update timestamp. +func (osb *OperationStatusBuilder) Updated(t time.Time) *OperationStatusBuilder { + osb.object.Updated = &metav1.Time{Time: t} + return osb +} + +// Started sets the OperationStatus's start timestamp. +func (osb *OperationStatusBuilder) Started(t time.Time) *OperationStatusBuilder { + osb.object.Started = &metav1.Time{Time: t} + return osb +} + +// BackupOperationBuilder builds BackupOperation objects +type BackupOperationBuilder struct { + object *itemoperation.BackupOperation +} + +// ForBackupOperation is the constructor for a BackupOperationBuilder. +func ForBackupOperation() *BackupOperationBuilder { + return &BackupOperationBuilder{ + object: &itemoperation.BackupOperation{}, + } +} + +// Result returns the built BackupOperation. +func (bb *BackupOperationBuilder) Result() *itemoperation.BackupOperation { + return bb.object +} + +// BackupName sets the BackupOperation's backup name. +func (bb *BackupOperationBuilder) BackupName(name string) *BackupOperationBuilder { + bb.object.Spec.BackupName = name + return bb +} + +// OperationID sets the BackupOperation's operation ID. +func (bb *BackupOperationBuilder) OperationID(id string) *BackupOperationBuilder { + bb.object.Spec.OperationID = id + return bb +} + +// Status sets the BackupOperation's status. +func (bb *BackupOperationBuilder) Status(status itemoperation.OperationStatus) *BackupOperationBuilder { + bb.object.Status = status + return bb +} + +// ResourceIdentifier sets the BackupOperation's resource identifier. +func (bb *BackupOperationBuilder) ResourceIdentifier(group, resource, ns, name string) *BackupOperationBuilder { + bb.object.Spec.ResourceIdentifier = velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: group, + Resource: resource, + }, + Namespace: ns, + Name: name, + } + return bb +} + +// BackupItemAction sets the BackupOperation's backup item action. +func (bb *BackupOperationBuilder) BackupItemAction(bia string) *BackupOperationBuilder { + bb.object.Spec.BackupItemAction = bia + return bb +} + +// PostOperationItem adds a post-operation item to the BackupOperation's list of post-operation items. +func (bb *BackupOperationBuilder) PostOperationItem(group, resource, ns, name string) *BackupOperationBuilder { + bb.object.Spec.PostOperationItems = append(bb.object.Spec.PostOperationItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: group, + Resource: resource, + }, + Namespace: ns, + Name: name, + }) + return bb +} + +// RestoreOperationBuilder builds RestoreOperation objects +type RestoreOperationBuilder struct { + object *itemoperation.RestoreOperation +} + +// ForRestoreOperation is the constructor for a RestoreOperationBuilder. +func ForRestoreOperation() *RestoreOperationBuilder { + return &RestoreOperationBuilder{ + object: &itemoperation.RestoreOperation{}, + } +} + +// Result returns the built RestoreOperation. +func (rb *RestoreOperationBuilder) Result() *itemoperation.RestoreOperation { + return rb.object +} + +// RestoreName sets the RestoreOperation's restore name. +func (rb *RestoreOperationBuilder) RestoreName(name string) *RestoreOperationBuilder { + rb.object.Spec.RestoreName = name + return rb +} + +// OperationID sets the RestoreOperation's operation ID. +func (rb *RestoreOperationBuilder) OperationID(id string) *RestoreOperationBuilder { + rb.object.Spec.OperationID = id + return rb +} + +// RestoreItemAction sets the RestoreOperation's restore item action. +func (rb *RestoreOperationBuilder) RestoreItemAction(ria string) *RestoreOperationBuilder { + rb.object.Spec.RestoreItemAction = ria + return rb +} + +// Status sets the RestoreOperation's status. +func (rb *RestoreOperationBuilder) Status(status itemoperation.OperationStatus) *RestoreOperationBuilder { + rb.object.Status = status + return rb +} + +// ResourceIdentifier sets the RestoreOperation's resource identifier. +func (rb *RestoreOperationBuilder) ResourceIdentifier(group, resource, ns, name string) *RestoreOperationBuilder { + rb.object.Spec.ResourceIdentifier = velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: group, + Resource: resource, + }, + Namespace: ns, + Name: name, + } + return rb +} diff --git a/pkg/builder/object_meta.go b/pkg/builder/object_meta.go index 6df1afadc..90730e4be 100644 --- a/pkg/builder/object_meta.go +++ b/pkg/builder/object_meta.go @@ -153,3 +153,10 @@ func WithManagedFields(val []metav1.ManagedFieldsEntry) func(obj metav1.Object) obj.SetManagedFields(val) } } + +// WithCreationTimestamp is a functional option that applies the specified creationTimestamp +func WithCreationTimestamp(t time.Time) func(obj metav1.Object) { + return func(obj metav1.Object) { + obj.SetCreationTimestamp(metav1.Time{Time: t}) + } +} diff --git a/pkg/builder/pod_volume_backup_builder.go b/pkg/builder/pod_volume_backup_builder.go index 2b15d5e19..14e57a063 100644 --- a/pkg/builder/pod_volume_backup_builder.go +++ b/pkg/builder/pod_volume_backup_builder.go @@ -53,7 +53,6 @@ func (b *PodVolumeBackupBuilder) ObjectMeta(opts ...ObjectMetaOpt) *PodVolumeBac for _, opt := range opts { opt(b.object) } - return b } diff --git a/pkg/builder/pod_volume_restore_builder.go b/pkg/builder/pod_volume_restore_builder.go new file mode 100644 index 000000000..c131a0384 --- /dev/null +++ b/pkg/builder/pod_volume_restore_builder.go @@ -0,0 +1,83 @@ +package builder + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" +) + +// PodVolumeRestoreBuilder builds PodVolumeRestore objects. +type PodVolumeRestoreBuilder struct { + object *velerov1api.PodVolumeRestore +} + +// ForPodVolumeRestore is the constructor for a PodVolumeRestoreBuilder. +func ForPodVolumeRestore(ns, name string) *PodVolumeRestoreBuilder { + return &PodVolumeRestoreBuilder{ + object: &velerov1api.PodVolumeRestore{ + TypeMeta: metav1.TypeMeta{ + APIVersion: velerov1api.SchemeGroupVersion.String(), + Kind: "PodVolumeRestore", + }, + ObjectMeta: metav1.ObjectMeta{ + Namespace: ns, + Name: name, + }, + }, + } +} + +// Result returns the built PodVolumeRestore. +func (b *PodVolumeRestoreBuilder) Result() *velerov1api.PodVolumeRestore { + return b.object +} + +// ObjectMeta applies functional options to the PodVolumeRestore's ObjectMeta. +func (b *PodVolumeRestoreBuilder) ObjectMeta(opts ...ObjectMetaOpt) *PodVolumeRestoreBuilder { + for _, opt := range opts { + opt(b.object) + } + return b +} + +// Phase sets the PodVolumeRestore's phase. +func (b *PodVolumeRestoreBuilder) Phase(phase velerov1api.PodVolumeRestorePhase) *PodVolumeRestoreBuilder { + b.object.Status.Phase = phase + return b +} + +// BackupStorageLocation sets the PodVolumeRestore's backup storage location. +func (b *PodVolumeRestoreBuilder) BackupStorageLocation(name string) *PodVolumeRestoreBuilder { + b.object.Spec.BackupStorageLocation = name + return b +} + +// SnapshotID sets the PodVolumeRestore's snapshot ID. +func (b *PodVolumeRestoreBuilder) SnapshotID(snapshotID string) *PodVolumeRestoreBuilder { + b.object.Spec.SnapshotID = snapshotID + return b +} + +// PodName sets the name of the pod associated with this PodVolumeRestore. +func (b *PodVolumeRestoreBuilder) PodName(name string) *PodVolumeRestoreBuilder { + b.object.Spec.Pod.Name = name + return b +} + +// PodNamespace sets the name of the pod associated with this PodVolumeRestore. +func (b *PodVolumeRestoreBuilder) PodNamespace(ns string) *PodVolumeRestoreBuilder { + b.object.Spec.Pod.Namespace = ns + return b +} + +// Volume sets the name of the volume associated with this PodVolumeRestore. +func (b *PodVolumeRestoreBuilder) Volume(volume string) *PodVolumeRestoreBuilder { + b.object.Spec.Volume = volume + return b +} + +// UploaderType sets the type of uploader to use for this PodVolumeRestore. +func (b *PodVolumeRestoreBuilder) UploaderType(uploaderType string) *PodVolumeRestoreBuilder { + b.object.Spec.UploaderType = uploaderType + return b +} diff --git a/pkg/cmd/util/output/backup_describer_test.go b/pkg/cmd/util/output/backup_describer_test.go index 074c72bbe..f4ea3319f 100644 --- a/pkg/cmd/util/output/backup_describer_test.go +++ b/pkg/cmd/util/output/backup_describer_test.go @@ -6,6 +6,10 @@ import ( "text/tabwriter" "time" + "github.com/vmware-tanzu/velero/pkg/itemoperation" + + "github.com/stretchr/testify/require" + snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v4/apis/volumesnapshot/v1" "github.com/stretchr/testify/assert" v1 "k8s.io/api/core/v1" @@ -46,7 +50,31 @@ func TestDescribeBackupSpec(t *testing.T) { TTL(72 * time.Hour). CSISnapshotTimeout(10 * time.Minute). DataMover("mover"). - Result().Spec + Hooks(velerov1api.BackupHooks{ + Resources: []velerov1api.BackupResourceHookSpec{ + { + Name: "hook-1", + PreHooks: []velerov1api.BackupResourceHook{ + { + Exec: &velerov1api.ExecHook{ + Container: "hook-container-1", + Command: []string{"pre"}, + OnError: velerov1api.HookErrorModeContinue, + }, + }, + }, + PostHooks: []velerov1api.BackupResourceHook{ + { + Exec: &velerov1api.ExecHook{ + Container: "hook-container-1", + Command: []string{"post"}, + OnError: velerov1api.HookErrorModeContinue, + }, + }, + }, + }, + }, + }).Result().Spec expect1 := `Namespaces: Included: inc-ns-1, inc-ns-2 @@ -70,7 +98,30 @@ TTL: 72h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 0s -Hooks: +Hooks: + Resources: + hook-1: + Namespaces: + Included: inc-ns-1, inc-ns-2 + Excluded: exc-ns-1, exc-ns-2 + + Resources: + Included: inc-res-1, inc-res-2 + Excluded: exc-res-1, exc-res-2 + + Label selector: + + Pre Exec Hook: + Container: hook-container-1 + Command: pre + On Error: Continue + Timeout: 0s + + Post Exec Hook: + Container: hook-container-1 + Command: post + On Error: Continue + Timeout: 0s ` input2 := builder.ForBackup("test-ns", "test-backup-2"). @@ -112,13 +163,94 @@ ItemOperationTimeout: 0s Hooks: ` + input3 := builder.ForBackup("test-ns", "test-backup-3"). + StorageLocation("backup-location"). + OrderedResources(map[string]string{ + "kind1": "rs1-1, rs1-2", + }).Hooks(velerov1api.BackupHooks{ + Resources: []velerov1api.BackupResourceHookSpec{ + { + Name: "hook-1", + PreHooks: []velerov1api.BackupResourceHook{ + { + Exec: &velerov1api.ExecHook{ + Container: "hook-container-1", + Command: []string{"pre"}, + OnError: velerov1api.HookErrorModeContinue, + }, + }, + }, + PostHooks: []velerov1api.BackupResourceHook{ + { + Exec: &velerov1api.ExecHook{ + Container: "hook-container-1", + Command: []string{"post"}, + OnError: velerov1api.HookErrorModeContinue, + }, + }, + }, + }, + }, + }).Result().Spec + + expect3 := `Namespaces: + Included: * + Excluded: + +Resources: + Included: * + Excluded: + Cluster-scoped: auto + +Label selector: + +Storage Location: backup-location + +Velero-Native Snapshot PVs: auto +Snapshot Move Data: auto +Data Mover: + +TTL: 0s + +CSISnapshotTimeout: 0s +ItemOperationTimeout: 0s + +Hooks: + Resources: + hook-1: + Namespaces: + Included: * + Excluded: + + Resources: + Included: * + Excluded: + + Label selector: + + Pre Exec Hook: + Container: hook-container-1 + Command: pre + On Error: Continue + Timeout: 0s + + Post Exec Hook: + Container: hook-container-1 + Command: post + On Error: Continue + Timeout: 0s + +OrderedResources: + kind1: rs1-1, rs1-2 +` + testcases := []struct { name string input velerov1api.BackupSpec expect string }{ { - name: "old resource filter", + name: "old resource filter with hooks", input: input1, expect: expect1, }, @@ -127,6 +259,11 @@ Hooks: input: input2, expect: expect2, }, + { + name: "old resource filter with hooks and ordered resources", + input: input3, + expect: expect3, + }, } for _, tc := range testcases { @@ -164,7 +301,6 @@ func TestDescribeSnapshot(t *testing.T) { func TestDescribePodVolumeBackups(t *testing.T) { pvb1 := builder.ForPodVolumeBackup("test-ns", "test-pvb1"). - BackupStorageLocation("backup-location"). UploaderType("kopia"). Phase(velerov1api.PodVolumeBackupPhaseCompleted). BackupStorageLocation("bsl-1"). @@ -173,7 +309,6 @@ func TestDescribePodVolumeBackups(t *testing.T) { PodNamespace("pod-ns-1"). SnapshotID("snap-1").Result() pvb2 := builder.ForPodVolumeBackup("test-ns1", "test-pvb2"). - BackupStorageLocation("backup-location"). UploaderType("kopia"). Phase(velerov1api.PodVolumeBackupPhaseCompleted). BackupStorageLocation("bsl-1"). @@ -289,3 +424,99 @@ Snapshot Content Name: vsc-1 }) } } + +func TestDescribeDeleteBackupRequests(t *testing.T) { + t1, err1 := time.Parse("2006-Jan-02", "2023-Jun-26") + require.Nil(t, err1) + dbr1 := builder.ForDeleteBackupRequest("velero", "dbr1"). + ObjectMeta(builder.WithCreationTimestamp(t1)). + BackupName("bak-1"). + Phase(velerov1api.DeleteBackupRequestPhaseProcessed). + Errors("some error").Result() + t2, err2 := time.Parse("2006-Jan-02", "2023-Jun-25") + require.Nil(t, err2) + dbr2 := builder.ForDeleteBackupRequest("velero", "dbr2"). + ObjectMeta(builder.WithCreationTimestamp(t2)). + BackupName("bak-2"). + Phase(velerov1api.DeleteBackupRequestPhaseInProgress).Result() + + testcases := []struct { + name string + input []velerov1api.DeleteBackupRequest + expect string + }{ + { + name: "empty list", + input: []velerov1api.DeleteBackupRequest{}, + expect: `Deletion Attempts: +`, + }, + { + name: "list with one failed and one in-progress request", + input: []velerov1api.DeleteBackupRequest{*dbr1, *dbr2}, + expect: `Deletion Attempts (1 failed): + 2023-06-26 00:00:00 +0000 UTC: Processed + Errors: + some error + + 2023-06-25 00:00:00 +0000 UTC: InProgress +`, + }, + } + for _, tc := range testcases { + t.Run(tc.name, func(tt *testing.T) { + d := &Describer{ + Prefix: "", + out: &tabwriter.Writer{}, + buf: &bytes.Buffer{}, + } + d.out.Init(d.buf, 0, 8, 2, ' ', 0) + DescribeDeleteBackupRequests(d, tc.input) + d.out.Flush() + assert.Equal(tt, tc.expect, d.buf.String()) + }) + } +} + +func TestDescribeBackupItemOperation(t *testing.T) { + t1, err1 := time.Parse("2006-Jan-02", "2023-Jun-26") + require.Nil(t, err1) + t2, err2 := time.Parse("2006-Jan-02", "2023-Jun-25") + require.Nil(t, err2) + t3, err3 := time.Parse("2006-Jan-02", "2023-Jun-24") + require.Nil(t, err3) + input := builder.ForBackupOperation(). + BackupName("backup-1"). + OperationID("op-1"). + BackupItemAction("action-1"). + ResourceIdentifier("group", "rs-type", "ns", "rs-name"). + Status(*builder.ForOperationStatus(). + Phase(itemoperation.OperationPhaseFailed). + Error("operation error"). + Progress(50, 100, "bytes"). + Description("operation description"). + Created(t3). + Started(t2). + Updated(t1). + Result()).Result() + expected := ` Operation for rs-type.group ns/rs-name: + Backup Item Action Plugin: action-1 + Operation ID: op-1 + Phase: Failed + Operation Error: operation error + Progress: 50 of 100 complete (bytes) + Progress description: operation description + Created: 2023-06-24 00:00:00 +0000 UTC + Started: 2023-06-25 00:00:00 +0000 UTC + Updated: 2023-06-26 00:00:00 +0000 UTC +` + d := &Describer{ + Prefix: "", + out: &tabwriter.Writer{}, + buf: &bytes.Buffer{}, + } + d.out.Init(d.buf, 0, 8, 2, ' ', 0) + describeBackupItemOperation(d, input) + d.out.Flush() + assert.Equal(t, expected, d.buf.String()) +} diff --git a/pkg/cmd/util/output/backup_structured_describer_test.go b/pkg/cmd/util/output/backup_structured_describer_test.go index eb2aecd28..e58dc5d2a 100644 --- a/pkg/cmd/util/output/backup_structured_describer_test.go +++ b/pkg/cmd/util/output/backup_structured_describer_test.go @@ -5,6 +5,8 @@ import ( "testing" "time" + "github.com/stretchr/testify/require" + v1 "k8s.io/api/core/v1" "github.com/vmware-tanzu/velero/pkg/features" @@ -31,7 +33,32 @@ func TestDescribeBackupInSF(t *testing.T) { StorageLocation("backup-location"). TTL(72 * time.Hour). CSISnapshotTimeout(10 * time.Minute). - DataMover("mover") + DataMover("mover"). + Hooks(velerov1api.BackupHooks{ + Resources: []velerov1api.BackupResourceHookSpec{ + { + Name: "hook-1", + PreHooks: []velerov1api.BackupResourceHook{ + { + Exec: &velerov1api.ExecHook{ + Container: "hook-container-1", + Command: []string{"pre"}, + OnError: velerov1api.HookErrorModeContinue, + }, + }, + }, + PostHooks: []velerov1api.BackupResourceHook{ + { + Exec: &velerov1api.ExecHook{ + Container: "hook-container-1", + Command: []string{"post"}, + OnError: velerov1api.HookErrorModeContinue, + }, + }, + }, + }, + }, + }) expect1 := map[string]interface{}{ "spec": map[string]interface{}{ @@ -51,13 +78,73 @@ func TestDescribeBackupInSF(t *testing.T) { "TTL": "72h0m0s", "CSISnapshotTimeout": "10m0s", "veleroSnapshotMoveData": "auto", + "hooks": map[string]interface{}{ + "resources": map[string]interface{}{ + "hook-1": map[string]interface{}{ + "labelSelector": emptyDisplay, + "namespaces": map[string]string{ + "included": "inc-ns-1, inc-ns-2", + "excluded": "exc-ns-1, exc-ns-2", + }, + "preExecHook": []map[string]interface{}{ + { + "container": "hook-container-1", + "command": "pre", + "onError:": velerov1api.HookErrorModeContinue, + "timeout": "0s", + }, + }, + "postExecHook": []map[string]interface{}{ + { + "container": "hook-container-1", + "command": "post", + "onError:": velerov1api.HookErrorModeContinue, + "timeout": "0s", + }, + }, + "resources": map[string]string{ + "included": "inc-res-1, inc-res-2", + "excluded": "exc-res-1, exc-res-2", + }, + }, + }, + }, }, } DescribeBackupSpecInSF(sd, backupBuilder1.Result().Spec) assert.True(t, reflect.DeepEqual(sd.output, expect1)) - backupBuilder2 := builder.ForBackup("test-ns-2", "test-backup-2") - backupBuilder2.StorageLocation("backup-location") + backupBuilder2 := builder.ForBackup("test-ns-2", "test-backup-2"). + StorageLocation("backup-location"). + OrderedResources(map[string]string{ + "kind1": "rs1-1, rs1-2", + "kind2": "rs2-1, rs2-2", + }).Hooks(velerov1api.BackupHooks{ + Resources: []velerov1api.BackupResourceHookSpec{ + { + Name: "hook-1", + PreHooks: []velerov1api.BackupResourceHook{ + { + Exec: &velerov1api.ExecHook{ + Container: "hook-container-1", + Command: []string{"pre"}, + OnError: velerov1api.HookErrorModeContinue, + }, + }, + }, + PostHooks: []velerov1api.BackupResourceHook{ + { + Exec: &velerov1api.ExecHook{ + Container: "hook-container-1", + Command: []string{"post"}, + OnError: velerov1api.HookErrorModeContinue, + }, + }, + }, + }, + }, + }) + expect2 := map[string]interface{}{ "spec": map[string]interface{}{ "namespaces": map[string]interface{}{ @@ -76,6 +163,41 @@ func TestDescribeBackupInSF(t *testing.T) { "TTL": "0s", "CSISnapshotTimeout": "0s", "veleroSnapshotMoveData": "auto", + "hooks": map[string]interface{}{ + "resources": map[string]interface{}{ + "hook-1": map[string]interface{}{ + "labelSelector": emptyDisplay, + "namespaces": map[string]string{ + "included": "*", + "excluded": emptyDisplay, + }, + "preExecHook": []map[string]interface{}{ + { + "container": "hook-container-1", + "command": "pre", + "onError:": velerov1api.HookErrorModeContinue, + "timeout": "0s", + }, + }, + "postExecHook": []map[string]interface{}{ + { + "container": "hook-container-1", + "command": "post", + "onError:": velerov1api.HookErrorModeContinue, + "timeout": "0s", + }, + }, + "resources": map[string]string{ + "included": "*", + "excluded": emptyDisplay, + }, + }, + }, + }, + "orderedResources": map[string]string{ + "kind1": "rs1-1, rs1-2", + "kind2": "rs2-1, rs2-2", + }, }, } DescribeBackupSpecInSF(sd, backupBuilder2.Result().Spec) @@ -250,3 +372,83 @@ func TestDescribeBackupResultInSF(t *testing.T) { describeResultInSF(got, input) assert.True(t, reflect.DeepEqual(got, expect)) } + +func TestDescribeDeleteBackupRequestsInSF(t *testing.T) { + t1, err1 := time.Parse("2006-Jan-02", "2023-Jun-26") + require.Nil(t, err1) + dbr1 := builder.ForDeleteBackupRequest("velero", "dbr1"). + ObjectMeta(builder.WithCreationTimestamp(t1)). + BackupName("bak-1"). + Phase(velerov1api.DeleteBackupRequestPhaseProcessed). + Errors("some error").Result() + t2, err2 := time.Parse("2006-Jan-02", "2023-Jun-25") + require.Nil(t, err2) + dbr2 := builder.ForDeleteBackupRequest("velero", "dbr2"). + ObjectMeta(builder.WithCreationTimestamp(t2)). + BackupName("bak-2"). + Phase(velerov1api.DeleteBackupRequestPhaseInProgress).Result() + + testcases := []struct { + name string + input []velerov1api.DeleteBackupRequest + expect map[string]interface{} + }{ + { + name: "empty list", + input: []velerov1api.DeleteBackupRequest{}, + expect: map[string]interface{}{ + "deletionAttempts": map[string]interface{}{ + "deleteBackupRequests": []map[string]interface{}{}, + }, + }, + }, + { + name: "list with one failed and one in-progress request", + input: []velerov1api.DeleteBackupRequest{*dbr1, *dbr2}, + expect: map[string]interface{}{ + "deletionAttempts": map[string]interface{}{ + "failed": int(1), + "deleteBackupRequests": []map[string]interface{}{ + { + "creationTimestamp": t1.String(), + "phase": velerov1api.DeleteBackupRequestPhaseProcessed, + "errors": []string{ + "some error", + }, + }, + { + "creationTimestamp": t2.String(), + "phase": velerov1api.DeleteBackupRequestPhaseInProgress, + }, + }, + }, + }, + }, + } + for _, tc := range testcases { + t.Run(tc.name, func(tt *testing.T) { + sd := &StructuredDescriber{ + output: make(map[string]interface{}), + format: "", + } + DescribeDeleteBackupRequestsInSF(sd, tc.input) + assert.True(tt, reflect.DeepEqual(sd.output, tc.expect)) + }) + } + +} + +func TestDescribeSnapshotInSF(t *testing.T) { + res := map[string]interface{}{} + iops := int64(100) + describeSnapshotInSF("pv-1", "snapshot-1", "ebs", "us-east-2", &iops, res) + expect := map[string]interface{}{ + "pv-1": map[string]string{ + "snapshotID": "snapshot-1", + "type": "ebs", + "availabilityZone": "us-east-2", + "IOPS": "100", + }, + } + assert.True(t, reflect.DeepEqual(expect, res)) +} diff --git a/pkg/cmd/util/output/describe.go b/pkg/cmd/util/output/describe.go index b6a44b676..d4f6e9e4f 100644 --- a/pkg/cmd/util/output/describe.go +++ b/pkg/cmd/util/output/describe.go @@ -47,15 +47,6 @@ func Describe(fn func(d *Describer)) string { return d.buf.String() } -func NewDescriber(minwidth, tabwidth, padding int, padchar byte, flags uint) *Describer { - d := &Describer{ - out: new(tabwriter.Writer), - buf: new(bytes.Buffer), - } - d.out.Init(d.buf, minwidth, tabwidth, padding, padchar, flags) - return d -} - func (d *Describer) Printf(msg string, args ...interface{}) { fmt.Fprint(d.out, d.Prefix) fmt.Fprintf(d.out, msg, args...) diff --git a/pkg/cmd/util/output/describe_test.go b/pkg/cmd/util/output/describe_test.go index 9385aa107..45becf873 100644 --- a/pkg/cmd/util/output/describe_test.go +++ b/pkg/cmd/util/output/describe_test.go @@ -3,6 +3,7 @@ package output import ( "bytes" "fmt" + "reflect" "testing" "text/tabwriter" @@ -132,3 +133,36 @@ func TestStructuredDescriber_JSONEncode(t *testing.T) { }) } } + +func TestStructuredDescriber_DescribeMetadata(t *testing.T) { + d := NewStructuredDescriber("") + input := metav1.ObjectMeta{ + Name: "test", + Namespace: "test-ns", + Labels: map[string]string{ + "label-1": "v1", + "label-2": "v2", + }, + Annotations: map[string]string{ + "annotation-1": "v1", + "annotation-2": "v2", + }, + } + expect := map[string]interface{}{ + "metadata": map[string]interface{}{ + "name": "test", + "namespace": "test-ns", + "labels": map[string]string{ + "label-1": "v1", + "label-2": "v2", + }, + "annotations": map[string]string{ + "annotation-1": "v1", + "annotation-2": "v2", + }, + }, + } + d.DescribeMetadata(input) + + assert.True(t, reflect.DeepEqual(expect, d.output)) +} diff --git a/pkg/cmd/util/output/restore_describer_test.go b/pkg/cmd/util/output/restore_describer_test.go new file mode 100644 index 000000000..dff7aa96d --- /dev/null +++ b/pkg/cmd/util/output/restore_describer_test.go @@ -0,0 +1,183 @@ +package output + +import ( + "bytes" + "testing" + "text/tabwriter" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + "github.com/vmware-tanzu/velero/pkg/builder" + "github.com/vmware-tanzu/velero/pkg/itemoperation" + "github.com/vmware-tanzu/velero/pkg/util/results" +) + +func TestDescribeResult(t *testing.T) { + testcases := []struct { + name string + inputName string + inputResult results.Result + expect string + }{ + { + name: "result without ns warns", + inputName: "restore-1", + inputResult: results.Result{ + Velero: []string{"velero-msg-1", "velero-msg-2"}, + Cluster: []string{"cluster-msg-1", "cluster-msg-2"}, + Namespaces: map[string][]string{}, + }, + expect: `restore-1: + Velero: velero-msg-1 + velero-msg-2 + Cluster: cluster-msg-1 + cluster-msg-2 + Namespaces: +`, + }, + { + name: "result with ns warns", + inputName: "restore-2", + inputResult: results.Result{ + Velero: []string{"velero-msg-1", "velero-msg-2"}, + Cluster: []string{"cluster-msg-1", "cluster-msg-2"}, + Namespaces: map[string][]string{ + "ns-1": {"ns-1-warn-1", "ns-1-warn-2"}, + }, + }, + expect: `restore-2: + Velero: velero-msg-1 + velero-msg-2 + Cluster: cluster-msg-1 + cluster-msg-2 + Namespaces: + ns-1: ns-1-warn-1 + ns-1-warn-2 +`, + }, + } + for _, tc := range testcases { + t.Run(tc.name, func(tt *testing.T) { + d := &Describer{ + Prefix: "", + out: &tabwriter.Writer{}, + buf: &bytes.Buffer{}, + } + d.out.Init(d.buf, 0, 8, 2, ' ', 0) + describeResult(d, tc.inputName, tc.inputResult) + d.out.Flush() + assert.Equal(tt, tc.expect, d.buf.String()) + }) + } +} + +func TestDescribeRestoreItemOperation(t *testing.T) { + t1, err1 := time.Parse("2006-Jan-02", "2023-Jun-26") + require.Nil(t, err1) + t2, err2 := time.Parse("2006-Jan-02", "2023-Jun-25") + require.Nil(t, err2) + t3, err3 := time.Parse("2006-Jan-02", "2023-Jun-24") + require.Nil(t, err3) + input := builder.ForRestoreOperation(). + RestoreName("restore-1"). + OperationID("op-1"). + RestoreItemAction("action-1"). + ResourceIdentifier("group", "rs-type", "ns", "rs-name"). + Status(*builder.ForOperationStatus(). + Phase(itemoperation.OperationPhaseFailed). + Error("operation error"). + Progress(50, 100, "bytes"). + Description("operation description"). + Created(t3). + Started(t2). + Updated(t1). + Result()).Result() + expected := ` Operation for rs-type.group ns/rs-name: + Restore Item Action Plugin: action-1 + Operation ID: op-1 + Phase: Failed + Operation Error: operation error + Progress: 50 of 100 complete (bytes) + Progress description: operation description + Created: 2023-06-24 00:00:00 +0000 UTC + Started: 2023-06-25 00:00:00 +0000 UTC + Updated: 2023-06-26 00:00:00 +0000 UTC +` + d := &Describer{ + Prefix: "", + out: &tabwriter.Writer{}, + buf: &bytes.Buffer{}, + } + d.out.Init(d.buf, 0, 8, 2, ' ', 0) + describeRestoreItemOperation(d, input) + d.out.Flush() + assert.Equal(t, expected, d.buf.String()) +} + +func TestDescribePodVolumeRestores(t *testing.T) { + pvr1 := builder.ForPodVolumeRestore("velero", "pvr-1"). + UploaderType("kopia"). + Phase(velerov1api.PodVolumeRestorePhaseCompleted). + BackupStorageLocation("bsl-1"). + Volume("vol-1"). + PodName("pod-1"). + PodNamespace("pod-ns-1"). + SnapshotID("snap-1").Result() + pvr2 := builder.ForPodVolumeRestore("velero", "pvr-2"). + UploaderType("kopia"). + Phase(velerov1api.PodVolumeRestorePhaseCompleted). + BackupStorageLocation("bsl-1"). + Volume("vol-2"). + PodName("pod-2"). + PodNamespace("pod-ns-1"). + SnapshotID("snap-2").Result() + + testcases := []struct { + name string + inputPVRList []velerov1api.PodVolumeRestore + inputDetails bool + expect string + }{ + { + name: "empty list", + inputPVRList: []velerov1api.PodVolumeRestore{}, + inputDetails: true, + expect: ``, + }, + { + name: "2 completed pvrs no details", + inputPVRList: []velerov1api.PodVolumeRestore{*pvr1, *pvr2}, + inputDetails: false, + expect: `kopia Restores (specify --details for more information): + Completed: 2 +`, + }, + { + name: "2 completed pvrs with details", + inputPVRList: []velerov1api.PodVolumeRestore{*pvr1, *pvr2}, + inputDetails: true, + expect: `kopia Restores: + Completed: + pod-ns-1/pod-1: vol-1 + pod-ns-1/pod-2: vol-2 +`, + }, + } + + for _, tc := range testcases { + t.Run(tc.name, func(tt *testing.T) { + d := &Describer{ + Prefix: "", + out: &tabwriter.Writer{}, + buf: &bytes.Buffer{}, + } + d.out.Init(d.buf, 0, 8, 2, ' ', 0) + describePodVolumeRestores(d, tc.inputPVRList, tc.inputDetails) + d.out.Flush() + assert.Equal(tt, tc.expect, d.buf.String()) + }) + } +} diff --git a/pkg/cmd/util/output/schedule_describe_test.go b/pkg/cmd/util/output/schedule_describe_test.go new file mode 100644 index 000000000..bcf71ad0d --- /dev/null +++ b/pkg/cmd/util/output/schedule_describe_test.go @@ -0,0 +1,123 @@ +package output + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + "github.com/vmware-tanzu/velero/pkg/builder" +) + +func TestDescribeSchedule(t *testing.T) { + input1 := builder.ForSchedule("velero", "schedule-1"). + Phase(velerov1api.SchedulePhaseFailedValidation). + ValidationError("validation failed").Result() + expect1 := `Name: schedule-1 +Namespace: velero +Labels: +Annotations: + +Phase: FailedValidation + +Validation errors: validation failed + +Paused: false + +Schedule: + +Backup Template: + Namespaces: + Included: * + Excluded: + + Resources: + Included: * + Excluded: + Cluster-scoped: auto + + Label selector: + + Storage Location: + + Velero-Native Snapshot PVs: auto + Snapshot Move Data: auto + Data Mover: + + TTL: 0s + + CSISnapshotTimeout: 0s + ItemOperationTimeout: 0s + + Hooks: + +Last Backup: +` + + input2 := builder.ForSchedule("velero", "schedule-2"). + Phase(velerov1api.SchedulePhaseEnabled). + CronSchedule("0 0 * * *"). + Template(builder.ForBackup("velero", "backup-1").Result().Spec). + LastBackupTime("2023-06-25 15:04:05").Result() + expect2 := `Name: schedule-2 +Namespace: velero +Labels: +Annotations: + +Phase: Enabled + +Paused: false + +Schedule: 0 0 * * * + +Backup Template: + Namespaces: + Included: * + Excluded: + + Resources: + Included: * + Excluded: + Cluster-scoped: auto + + Label selector: + + Storage Location: + + Velero-Native Snapshot PVs: auto + Snapshot Move Data: auto + Data Mover: + + TTL: 0s + + CSISnapshotTimeout: 0s + ItemOperationTimeout: 0s + + Hooks: + +Last Backup: 2023-06-25 15:04:05 +0000 UTC +` + + testcases := []struct { + name string + input *velerov1api.Schedule + expect string + }{ + { + name: "schedule failed in validation", + input: input1, + expect: expect1, + }, + { + name: "schedule enabled", + input: input2, + expect: expect2, + }, + } + + for _, tc := range testcases { + t.Run(tc.name, func(tt *testing.T) { + assert.Equal(tt, tc.expect, DescribeSchedule(tc.input)) + }) + } +} From b8c234a0a71e4f780c01de52f85e53732b378376 Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Tue, 27 Jun 2023 09:01:35 +0800 Subject: [PATCH 10/28] fix main CI out of space problem Signed-off-by: Lyndon-Li --- .github/workflows/push.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/push.yml b/.github/workflows/push.yml index 8f289fb03..f93e96bec 100644 --- a/.github/workflows/push.yml +++ b/.github/workflows/push.yml @@ -64,6 +64,10 @@ jobs: - name: Publish container image if: github.repository == 'vmware-tanzu/velero' run: | + sudo swapoff -a + sudo rm -f /mnt/swapfile + docker image prune -a --force + # Build and push Velero image to docker registry docker login -u ${{ secrets.DOCKER_USER }} -p ${{ secrets.DOCKER_PASSWORD }} VERSION=$(./hack/docker-push.sh | grep 'VERSION:' | awk -F: '{print $2}' | xargs) From 38d5003c6b551b4115f131a5adf11e1983c9f3b3 Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Thu, 29 Jun 2023 11:17:40 +0800 Subject: [PATCH 11/28] add ut for pkg/repository Signed-off-by: Lyndon-Li --- pkg/repository/provider/unified_repo_test.go | 464 +++++++++++++++++ .../udmrepo/kopialib/backend/common_test.go | 198 +++++++ .../kopialib/backend/file_system_test.go | 80 +++ .../udmrepo/kopialib/backend/gcs_test.go | 47 +- .../udmrepo/kopialib/backend/mocks/Reader.go | 101 ++++ .../udmrepo/kopialib/backend/mocks/Writer.go | 114 ++++ .../udmrepo/kopialib/backend/s3_test.go | 83 ++- .../udmrepo/kopialib/lib_repo_test.go | 485 ++++++++++++++++++ 8 files changed, 1566 insertions(+), 6 deletions(-) create mode 100644 pkg/repository/udmrepo/kopialib/backend/common_test.go create mode 100644 pkg/repository/udmrepo/kopialib/backend/file_system_test.go create mode 100644 pkg/repository/udmrepo/kopialib/backend/mocks/Reader.go create mode 100644 pkg/repository/udmrepo/kopialib/backend/mocks/Writer.go diff --git a/pkg/repository/provider/unified_repo_test.go b/pkg/repository/provider/unified_repo_test.go index 80b2611df..8ca73127b 100644 --- a/pkg/repository/provider/unified_repo_test.go +++ b/pkg/repository/provider/unified_repo_test.go @@ -887,3 +887,467 @@ func TestForget(t *testing.T) { }) } } + +func TestInitRepo(t *testing.T) { + testCases := []struct { + name string + funcTable localFuncTable + getter *credmock.SecretStore + repoService *reposervicenmocks.BackupRepoService + retFuncInit interface{} + credStoreReturn string + credStoreError error + expectedErr string + }{ + { + name: "get repo option fail", + expectedErr: "error to get repo options: error to get repo password: invalid credentials interface", + }, + { + name: "repo init fail", + getter: new(credmock.SecretStore), + credStoreReturn: "fake-password", + funcTable: localFuncTable{ + getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) { + return map[string]string{}, nil + }, + getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) { + return map[string]string{}, nil + }, + }, + repoService: new(reposervicenmocks.BackupRepoService), + retFuncInit: func(context.Context, udmrepo.RepoOptions, bool) error { + return errors.New("fake-error-1") + }, + expectedErr: "error to init backup repo: fake-error-1", + }, + { + name: "succeed", + getter: new(credmock.SecretStore), + credStoreReturn: "fake-password", + funcTable: localFuncTable{ + getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) { + return map[string]string{}, nil + }, + getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) { + return map[string]string{}, nil + }, + }, + repoService: new(reposervicenmocks.BackupRepoService), + retFuncInit: func(context.Context, udmrepo.RepoOptions, bool) error { + return nil + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + funcTable = tc.funcTable + + var secretStore velerocredentials.SecretStore + if tc.getter != nil { + tc.getter.On("Get", mock.Anything, mock.Anything).Return(tc.credStoreReturn, tc.credStoreError) + secretStore = tc.getter + } + + urp := unifiedRepoProvider{ + credentialGetter: velerocredentials.CredentialGetter{ + FromSecret: secretStore, + }, + repoService: tc.repoService, + log: velerotest.NewLogger(), + } + + if tc.repoService != nil { + tc.repoService.On("Init", mock.Anything, mock.Anything, mock.Anything).Return(tc.retFuncInit) + } + + err := urp.InitRepo(context.Background(), RepoParam{ + BackupLocation: &velerov1api.BackupStorageLocation{}, + BackupRepo: &velerov1api.BackupRepository{}, + }) + + if tc.expectedErr == "" { + assert.NoError(t, err) + } else { + assert.EqualError(t, err, tc.expectedErr) + } + }) + } +} + +func TestConnectToRepo(t *testing.T) { + testCases := []struct { + name string + funcTable localFuncTable + getter *credmock.SecretStore + repoService *reposervicenmocks.BackupRepoService + retFuncInit interface{} + credStoreReturn string + credStoreError error + expectedErr string + }{ + { + name: "get repo option fail", + expectedErr: "error to get repo options: error to get repo password: invalid credentials interface", + }, + { + name: "repo init fail", + getter: new(credmock.SecretStore), + credStoreReturn: "fake-password", + funcTable: localFuncTable{ + getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) { + return map[string]string{}, nil + }, + getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) { + return map[string]string{}, nil + }, + }, + repoService: new(reposervicenmocks.BackupRepoService), + retFuncInit: func(context.Context, udmrepo.RepoOptions, bool) error { + return errors.New("fake-error-1") + }, + expectedErr: "error to connect backup repo: fake-error-1", + }, + { + name: "succeed", + getter: new(credmock.SecretStore), + credStoreReturn: "fake-password", + funcTable: localFuncTable{ + getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) { + return map[string]string{}, nil + }, + getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) { + return map[string]string{}, nil + }, + }, + repoService: new(reposervicenmocks.BackupRepoService), + retFuncInit: func(context.Context, udmrepo.RepoOptions, bool) error { + return nil + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + funcTable = tc.funcTable + + var secretStore velerocredentials.SecretStore + if tc.getter != nil { + tc.getter.On("Get", mock.Anything, mock.Anything).Return(tc.credStoreReturn, tc.credStoreError) + secretStore = tc.getter + } + + urp := unifiedRepoProvider{ + credentialGetter: velerocredentials.CredentialGetter{ + FromSecret: secretStore, + }, + repoService: tc.repoService, + log: velerotest.NewLogger(), + } + + if tc.repoService != nil { + tc.repoService.On("Init", mock.Anything, mock.Anything, mock.Anything).Return(tc.retFuncInit) + } + + err := urp.ConnectToRepo(context.Background(), RepoParam{ + BackupLocation: &velerov1api.BackupStorageLocation{}, + BackupRepo: &velerov1api.BackupRepository{}, + }) + + if tc.expectedErr == "" { + assert.NoError(t, err) + } else { + assert.EqualError(t, err, tc.expectedErr) + } + }) + } +} + +func TestBoostRepoConnect(t *testing.T) { + var backupRepo *reposervicenmocks.BackupRepo + + testCases := []struct { + name string + funcTable localFuncTable + getter *credmock.SecretStore + repoService *reposervicenmocks.BackupRepoService + backupRepo *reposervicenmocks.BackupRepo + retFuncInit interface{} + retFuncOpen []interface{} + credStoreReturn string + credStoreError error + expectedErr string + }{ + { + name: "get repo option fail", + expectedErr: "error to get repo options: error to get repo password: invalid credentials interface", + }, + { + name: "repo not opened and connect fail", + getter: new(credmock.SecretStore), + credStoreReturn: "fake-password", + funcTable: localFuncTable{ + getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) { + return map[string]string{}, nil + }, + getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) { + return map[string]string{}, nil + }, + }, + repoService: new(reposervicenmocks.BackupRepoService), + retFuncOpen: []interface{}{ + func(context.Context, udmrepo.RepoOptions) udmrepo.BackupRepo { + return backupRepo + }, + + func(context.Context, udmrepo.RepoOptions) error { + return errors.New("fake-error-1") + }, + }, + retFuncInit: func(context.Context, udmrepo.RepoOptions, bool) error { + return errors.New("fake-error-2") + }, + expectedErr: "error to connect backup repo: fake-error-2", + }, + { + name: "repo not opened and connect succeed", + getter: new(credmock.SecretStore), + credStoreReturn: "fake-password", + funcTable: localFuncTable{ + getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) { + return map[string]string{}, nil + }, + getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) { + return map[string]string{}, nil + }, + }, + repoService: new(reposervicenmocks.BackupRepoService), + retFuncOpen: []interface{}{ + func(context.Context, udmrepo.RepoOptions) udmrepo.BackupRepo { + return backupRepo + }, + + func(context.Context, udmrepo.RepoOptions) error { + return errors.New("fake-error-1") + }, + }, + retFuncInit: func(context.Context, udmrepo.RepoOptions, bool) error { + return nil + }, + }, + { + name: "repo is opened", + getter: new(credmock.SecretStore), + credStoreReturn: "fake-password", + funcTable: localFuncTable{ + getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) { + return map[string]string{}, nil + }, + getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) { + return map[string]string{}, nil + }, + }, + repoService: new(reposervicenmocks.BackupRepoService), + backupRepo: new(reposervicenmocks.BackupRepo), + retFuncOpen: []interface{}{ + func(context.Context, udmrepo.RepoOptions) udmrepo.BackupRepo { + return backupRepo + }, + + func(context.Context, udmrepo.RepoOptions) error { + return nil + }, + }, + retFuncInit: func(context.Context, udmrepo.RepoOptions, bool) error { + return nil + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + funcTable = tc.funcTable + + var secretStore velerocredentials.SecretStore + if tc.getter != nil { + tc.getter.On("Get", mock.Anything, mock.Anything).Return(tc.credStoreReturn, tc.credStoreError) + secretStore = tc.getter + } + + urp := unifiedRepoProvider{ + credentialGetter: velerocredentials.CredentialGetter{ + FromSecret: secretStore, + }, + repoService: tc.repoService, + log: velerotest.NewLogger(), + } + + backupRepo = tc.backupRepo + + if tc.repoService != nil { + tc.repoService.On("Open", mock.Anything, mock.Anything).Return(tc.retFuncOpen[0], tc.retFuncOpen[1]) + tc.repoService.On("Init", mock.Anything, mock.Anything, mock.Anything).Return(tc.retFuncInit) + } + + if tc.backupRepo != nil { + backupRepo.On("Close", mock.Anything).Return(nil) + } + + err := urp.BoostRepoConnect(context.Background(), RepoParam{ + BackupLocation: &velerov1api.BackupStorageLocation{}, + BackupRepo: &velerov1api.BackupRepository{}, + }) + + if tc.expectedErr == "" { + assert.NoError(t, err) + } else { + assert.EqualError(t, err, tc.expectedErr) + } + }) + } +} + +func TestPruneRepo(t *testing.T) { + testCases := []struct { + name string + funcTable localFuncTable + getter *credmock.SecretStore + repoService *reposervicenmocks.BackupRepoService + retFuncMaintain interface{} + credStoreReturn string + credStoreError error + expectedErr string + }{ + { + name: "get repo option fail", + expectedErr: "error to get repo options: error to get repo password: invalid credentials interface", + }, + { + name: "repo maintain fail", + getter: new(credmock.SecretStore), + credStoreReturn: "fake-password", + funcTable: localFuncTable{ + getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) { + return map[string]string{}, nil + }, + getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) { + return map[string]string{}, nil + }, + }, + repoService: new(reposervicenmocks.BackupRepoService), + retFuncMaintain: func(context.Context, udmrepo.RepoOptions) error { + return errors.New("fake-error-1") + }, + expectedErr: "error to prune backup repo: fake-error-1", + }, + { + name: "succeed", + getter: new(credmock.SecretStore), + credStoreReturn: "fake-password", + funcTable: localFuncTable{ + getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) { + return map[string]string{}, nil + }, + getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) { + return map[string]string{}, nil + }, + }, + repoService: new(reposervicenmocks.BackupRepoService), + retFuncMaintain: func(context.Context, udmrepo.RepoOptions) error { + return nil + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + funcTable = tc.funcTable + + var secretStore velerocredentials.SecretStore + if tc.getter != nil { + tc.getter.On("Get", mock.Anything, mock.Anything).Return(tc.credStoreReturn, tc.credStoreError) + secretStore = tc.getter + } + + urp := unifiedRepoProvider{ + credentialGetter: velerocredentials.CredentialGetter{ + FromSecret: secretStore, + }, + repoService: tc.repoService, + log: velerotest.NewLogger(), + } + + if tc.repoService != nil { + tc.repoService.On("Maintain", mock.Anything, mock.Anything).Return(tc.retFuncMaintain) + } + + err := urp.PruneRepo(context.Background(), RepoParam{ + BackupLocation: &velerov1api.BackupStorageLocation{}, + BackupRepo: &velerov1api.BackupRepository{}, + }) + + if tc.expectedErr == "" { + assert.NoError(t, err) + } else { + assert.EqualError(t, err, tc.expectedErr) + } + }) + } +} + +func TestGetStorageType(t *testing.T) { + testCases := []struct { + name string + backupLocation *velerov1api.BackupStorageLocation + expectedRet string + }{ + { + name: "wrong backend type", + backupLocation: &velerov1api.BackupStorageLocation{}, + }, + { + name: "aws provider", + backupLocation: &velerov1api.BackupStorageLocation{ + Spec: velerov1api.BackupStorageLocationSpec{ + Provider: "velero.io/aws", + }, + }, + expectedRet: "s3", + }, + { + name: "azure provider", + backupLocation: &velerov1api.BackupStorageLocation{ + Spec: velerov1api.BackupStorageLocationSpec{ + Provider: "velero.io/azure", + }, + }, + expectedRet: "azure", + }, + { + name: "gcp provider", + backupLocation: &velerov1api.BackupStorageLocation{ + Spec: velerov1api.BackupStorageLocationSpec{ + Provider: "velero.io/gcp", + }, + }, + expectedRet: "gcs", + }, + { + name: "fs provider", + backupLocation: &velerov1api.BackupStorageLocation{ + Spec: velerov1api.BackupStorageLocationSpec{ + Provider: "velero.io/fs", + }, + }, + expectedRet: "filesystem", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + ret := getStorageType(tc.backupLocation) + assert.Equal(t, tc.expectedRet, ret) + }) + } +} diff --git a/pkg/repository/udmrepo/kopialib/backend/common_test.go b/pkg/repository/udmrepo/kopialib/backend/common_test.go new file mode 100644 index 000000000..daf6e8479 --- /dev/null +++ b/pkg/repository/udmrepo/kopialib/backend/common_test.go @@ -0,0 +1,198 @@ +/* +Copyright the Velero contributors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package backend + +import ( + "context" + "testing" + "time" + + "github.com/kopia/kopia/repo" + "github.com/kopia/kopia/repo/content" + "github.com/kopia/kopia/repo/encryption" + "github.com/kopia/kopia/repo/format" + "github.com/kopia/kopia/repo/hashing" + "github.com/kopia/kopia/repo/splitter" + "github.com/stretchr/testify/assert" + + "github.com/vmware-tanzu/velero/pkg/repository/udmrepo" +) + +func TestSetupNewRepositoryOptions(t *testing.T) { + testCases := []struct { + name string + flags map[string]string + expected repo.NewRepositoryOptions + }{ + { + name: "with hash algo", + flags: map[string]string{ + udmrepo.StoreOptionGenHashAlgo: "fake-hash", + }, + expected: repo.NewRepositoryOptions{ + BlockFormat: format.ContentFormat{ + Hash: "fake-hash", + Encryption: encryption.DefaultAlgorithm, + }, + ObjectFormat: format.ObjectFormat{ + Splitter: splitter.DefaultAlgorithm, + }, + }, + }, + { + name: "with encrypt algo", + flags: map[string]string{ + udmrepo.StoreOptionGenEncryptAlgo: "fake-encrypt", + }, + expected: repo.NewRepositoryOptions{ + BlockFormat: format.ContentFormat{ + Hash: hashing.DefaultAlgorithm, + Encryption: "fake-encrypt", + }, + ObjectFormat: format.ObjectFormat{ + Splitter: splitter.DefaultAlgorithm, + }, + }, + }, + { + name: "with splitter algo", + flags: map[string]string{ + udmrepo.StoreOptionGenSplitAlgo: "fake-splitter", + }, + expected: repo.NewRepositoryOptions{ + BlockFormat: format.ContentFormat{ + Hash: hashing.DefaultAlgorithm, + Encryption: encryption.DefaultAlgorithm, + }, + ObjectFormat: format.ObjectFormat{ + Splitter: "fake-splitter", + }, + }, + }, + { + name: "with retention algo", + flags: map[string]string{ + udmrepo.StoreOptionGenRetentionMode: "fake-retention-mode", + }, + expected: repo.NewRepositoryOptions{ + BlockFormat: format.ContentFormat{ + Hash: hashing.DefaultAlgorithm, + Encryption: encryption.DefaultAlgorithm, + }, + ObjectFormat: format.ObjectFormat{ + Splitter: splitter.DefaultAlgorithm, + }, + RetentionMode: "fake-retention-mode", + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + ret := SetupNewRepositoryOptions(context.Background(), tc.flags) + assert.Equal(t, tc.expected, ret) + }) + } +} + +func TestSetupConnectOptions(t *testing.T) { + defaultCacheOption := content.CachingOptions{ + MaxCacheSizeBytes: 2000 << 20, + MaxMetadataCacheSizeBytes: 2000 << 20, + MaxListCacheDuration: content.DurationSeconds(time.Duration(30) * time.Second), + } + + testCases := []struct { + name string + repoOptions udmrepo.RepoOptions + expected repo.ConnectOptions + }{ + { + name: "with domain", + repoOptions: udmrepo.RepoOptions{ + GeneralOptions: map[string]string{ + udmrepo.GenOptionOwnerDomain: "fake-domain", + }, + }, + expected: repo.ConnectOptions{ + CachingOptions: defaultCacheOption, + ClientOptions: repo.ClientOptions{ + Hostname: "fake-domain", + }, + }, + }, + { + name: "with username", + repoOptions: udmrepo.RepoOptions{ + GeneralOptions: map[string]string{ + udmrepo.GenOptionOwnerName: "fake-user", + }, + }, + expected: repo.ConnectOptions{ + CachingOptions: defaultCacheOption, + ClientOptions: repo.ClientOptions{ + Username: "fake-user", + }, + }, + }, + { + name: "with wrong readonly", + repoOptions: udmrepo.RepoOptions{ + GeneralOptions: map[string]string{ + udmrepo.StoreOptionGenReadOnly: "fake-bool", + }, + }, + expected: repo.ConnectOptions{ + CachingOptions: defaultCacheOption, + ClientOptions: repo.ClientOptions{}, + }, + }, + { + name: "with correct readonly", + repoOptions: udmrepo.RepoOptions{ + GeneralOptions: map[string]string{ + udmrepo.StoreOptionGenReadOnly: "true", + }, + }, + expected: repo.ConnectOptions{ + CachingOptions: defaultCacheOption, + ClientOptions: repo.ClientOptions{ + ReadOnly: true, + }, + }, + }, + { + name: "with description", + repoOptions: udmrepo.RepoOptions{ + Description: "fake-description", + }, + expected: repo.ConnectOptions{ + CachingOptions: defaultCacheOption, + ClientOptions: repo.ClientOptions{ + Description: "fake-description", + }, + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + ret := SetupConnectOptions(context.Background(), tc.repoOptions) + assert.Equal(t, tc.expected, ret) + }) + } +} diff --git a/pkg/repository/udmrepo/kopialib/backend/file_system_test.go b/pkg/repository/udmrepo/kopialib/backend/file_system_test.go new file mode 100644 index 000000000..fe9b8e624 --- /dev/null +++ b/pkg/repository/udmrepo/kopialib/backend/file_system_test.go @@ -0,0 +1,80 @@ +/* +Copyright the Velero contributors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package backend + +import ( + "context" + "testing" + + "github.com/kopia/kopia/repo/blob/filesystem" + "github.com/stretchr/testify/assert" + + "github.com/vmware-tanzu/velero/pkg/repository/udmrepo" +) + +func TestFSSetup(t *testing.T) { + testCases := []struct { + name string + flags map[string]string + expectedOptions filesystem.Options + expectedErr string + }{ + { + name: "must have fs path", + flags: map[string]string{}, + expectedErr: "key " + udmrepo.StoreOptionFsPath + " not found", + }, + { + name: "with fs path only", + flags: map[string]string{ + udmrepo.StoreOptionFsPath: "fake/path", + }, + expectedOptions: filesystem.Options{ + Path: "fake/path", + FileMode: 0o600, + DirectoryMode: 0o700, + }, + }, + { + name: "with prefix", + flags: map[string]string{ + udmrepo.StoreOptionFsPath: "fake/path", + udmrepo.StoreOptionPrefix: "fake-prefix", + }, + expectedOptions: filesystem.Options{ + Path: "fake/path/fake-prefix", + FileMode: 0o600, + DirectoryMode: 0o700, + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + fsFlags := FsBackend{} + + err := fsFlags.Setup(context.Background(), tc.flags) + + if tc.expectedErr == "" { + assert.NoError(t, err) + assert.Equal(t, tc.expectedOptions, fsFlags.options) + } else { + assert.EqualError(t, err, tc.expectedErr) + } + }) + } +} diff --git a/pkg/repository/udmrepo/kopialib/backend/gcs_test.go b/pkg/repository/udmrepo/kopialib/backend/gcs_test.go index 7abdcab3e..759e9baae 100644 --- a/pkg/repository/udmrepo/kopialib/backend/gcs_test.go +++ b/pkg/repository/udmrepo/kopialib/backend/gcs_test.go @@ -20,6 +20,7 @@ import ( "context" "testing" + "github.com/kopia/kopia/repo/blob/gcs" "github.com/stretchr/testify/assert" "github.com/vmware-tanzu/velero/pkg/repository/udmrepo" @@ -27,9 +28,10 @@ import ( func TestGcsSetup(t *testing.T) { testCases := []struct { - name string - flags map[string]string - expectedErr string + name string + flags map[string]string + expectedOptions gcs.Options + expectedErr string }{ { name: "must have bucket name", @@ -43,6 +45,44 @@ func TestGcsSetup(t *testing.T) { }, expectedErr: "key " + udmrepo.StoreOptionCredentialFile + " not found", }, + { + name: "with prefix", + flags: map[string]string{ + udmrepo.StoreOptionOssBucket: "fake-bucket", + udmrepo.StoreOptionCredentialFile: "fake-credential", + udmrepo.StoreOptionPrefix: "fake-prefix", + }, + expectedOptions: gcs.Options{ + BucketName: "fake-bucket", + ServiceAccountCredentialsFile: "fake-credential", + Prefix: "fake-prefix", + }, + }, + { + name: "with wrong readonly", + flags: map[string]string{ + udmrepo.StoreOptionOssBucket: "fake-bucket", + udmrepo.StoreOptionCredentialFile: "fake-credential", + udmrepo.StoreOptionGcsReadonly: "fake-bool", + }, + expectedOptions: gcs.Options{ + BucketName: "fake-bucket", + ServiceAccountCredentialsFile: "fake-credential", + }, + }, + { + name: "with correct readonly", + flags: map[string]string{ + udmrepo.StoreOptionOssBucket: "fake-bucket", + udmrepo.StoreOptionCredentialFile: "fake-credential", + udmrepo.StoreOptionGcsReadonly: "true", + }, + expectedOptions: gcs.Options{ + BucketName: "fake-bucket", + ServiceAccountCredentialsFile: "fake-credential", + ReadOnly: true, + }, + }, } for _, tc := range testCases { @@ -53,6 +93,7 @@ func TestGcsSetup(t *testing.T) { if tc.expectedErr == "" { assert.NoError(t, err) + assert.Equal(t, tc.expectedOptions, gcsFlags.options) } else { assert.EqualError(t, err, tc.expectedErr) } diff --git a/pkg/repository/udmrepo/kopialib/backend/mocks/Reader.go b/pkg/repository/udmrepo/kopialib/backend/mocks/Reader.go new file mode 100644 index 000000000..8efe8ee66 --- /dev/null +++ b/pkg/repository/udmrepo/kopialib/backend/mocks/Reader.go @@ -0,0 +1,101 @@ +// Code generated by mockery v2.22.1. DO NOT EDIT. + +package mocks + +import mock "github.com/stretchr/testify/mock" + +// Reader is an autogenerated mock type for the Reader type +type Reader struct { + mock.Mock +} + +// Close provides a mock function with given fields: +func (_m *Reader) Close() error { + ret := _m.Called() + + var r0 error + if rf, ok := ret.Get(0).(func() error); ok { + r0 = rf() + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// Length provides a mock function with given fields: +func (_m *Reader) Length() int64 { + ret := _m.Called() + + var r0 int64 + if rf, ok := ret.Get(0).(func() int64); ok { + r0 = rf() + } else { + r0 = ret.Get(0).(int64) + } + + return r0 +} + +// Read provides a mock function with given fields: p +func (_m *Reader) Read(p []byte) (int, error) { + ret := _m.Called(p) + + var r0 int + var r1 error + if rf, ok := ret.Get(0).(func([]byte) (int, error)); ok { + return rf(p) + } + if rf, ok := ret.Get(0).(func([]byte) int); ok { + r0 = rf(p) + } else { + r0 = ret.Get(0).(int) + } + + if rf, ok := ret.Get(1).(func([]byte) error); ok { + r1 = rf(p) + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +// Seek provides a mock function with given fields: offset, whence +func (_m *Reader) Seek(offset int64, whence int) (int64, error) { + ret := _m.Called(offset, whence) + + var r0 int64 + var r1 error + if rf, ok := ret.Get(0).(func(int64, int) (int64, error)); ok { + return rf(offset, whence) + } + if rf, ok := ret.Get(0).(func(int64, int) int64); ok { + r0 = rf(offset, whence) + } else { + r0 = ret.Get(0).(int64) + } + + if rf, ok := ret.Get(1).(func(int64, int) error); ok { + r1 = rf(offset, whence) + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type mockConstructorTestingTNewReader interface { + mock.TestingT + Cleanup(func()) +} + +// NewReader creates a new instance of Reader. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations. +func NewReader(t mockConstructorTestingTNewReader) *Reader { + mock := &Reader{} + mock.Mock.Test(t) + + t.Cleanup(func() { mock.AssertExpectations(t) }) + + return mock +} diff --git a/pkg/repository/udmrepo/kopialib/backend/mocks/Writer.go b/pkg/repository/udmrepo/kopialib/backend/mocks/Writer.go new file mode 100644 index 000000000..21f66334e --- /dev/null +++ b/pkg/repository/udmrepo/kopialib/backend/mocks/Writer.go @@ -0,0 +1,114 @@ +// Code generated by mockery v2.22.1. DO NOT EDIT. + +package mocks + +import ( + object "github.com/kopia/kopia/repo/object" + mock "github.com/stretchr/testify/mock" +) + +// Writer is an autogenerated mock type for the Writer type +type Writer struct { + mock.Mock +} + +// Checkpoint provides a mock function with given fields: +func (_m *Writer) Checkpoint() (object.ID, error) { + ret := _m.Called() + + var r0 object.ID + var r1 error + if rf, ok := ret.Get(0).(func() (object.ID, error)); ok { + return rf() + } + if rf, ok := ret.Get(0).(func() object.ID); ok { + r0 = rf() + } else { + r0 = ret.Get(0).(object.ID) + } + + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +// Close provides a mock function with given fields: +func (_m *Writer) Close() error { + ret := _m.Called() + + var r0 error + if rf, ok := ret.Get(0).(func() error); ok { + r0 = rf() + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// Result provides a mock function with given fields: +func (_m *Writer) Result() (object.ID, error) { + ret := _m.Called() + + var r0 object.ID + var r1 error + if rf, ok := ret.Get(0).(func() (object.ID, error)); ok { + return rf() + } + if rf, ok := ret.Get(0).(func() object.ID); ok { + r0 = rf() + } else { + r0 = ret.Get(0).(object.ID) + } + + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +// Write provides a mock function with given fields: p +func (_m *Writer) Write(p []byte) (int, error) { + ret := _m.Called(p) + + var r0 int + var r1 error + if rf, ok := ret.Get(0).(func([]byte) (int, error)); ok { + return rf(p) + } + if rf, ok := ret.Get(0).(func([]byte) int); ok { + r0 = rf(p) + } else { + r0 = ret.Get(0).(int) + } + + if rf, ok := ret.Get(1).(func([]byte) error); ok { + r1 = rf(p) + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type mockConstructorTestingTNewWriter interface { + mock.TestingT + Cleanup(func()) +} + +// NewWriter creates a new instance of Writer. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations. +func NewWriter(t mockConstructorTestingTNewWriter) *Writer { + mock := &Writer{} + mock.Mock.Test(t) + + t.Cleanup(func() { mock.AssertExpectations(t) }) + + return mock +} diff --git a/pkg/repository/udmrepo/kopialib/backend/s3_test.go b/pkg/repository/udmrepo/kopialib/backend/s3_test.go index c1f5e036b..43a761688 100644 --- a/pkg/repository/udmrepo/kopialib/backend/s3_test.go +++ b/pkg/repository/udmrepo/kopialib/backend/s3_test.go @@ -20,6 +20,7 @@ import ( "context" "testing" + "github.com/kopia/kopia/repo/blob/s3" "github.com/stretchr/testify/assert" "github.com/vmware-tanzu/velero/pkg/repository/udmrepo" @@ -27,15 +28,91 @@ import ( func TestS3Setup(t *testing.T) { testCases := []struct { - name string - flags map[string]string - expectedErr string + name string + flags map[string]string + expectedOptions s3.Options + expectedErr string }{ { name: "must have bucket name", flags: map[string]string{}, expectedErr: "key " + udmrepo.StoreOptionOssBucket + " not found", }, + { + name: "with bucket only", + flags: map[string]string{ + udmrepo.StoreOptionOssBucket: "fake-bucket", + }, + expectedOptions: s3.Options{ + BucketName: "fake-bucket", + }, + }, + { + name: "with others", + flags: map[string]string{ + udmrepo.StoreOptionOssBucket: "fake-bucket", + udmrepo.StoreOptionS3KeyID: "fake-ak", + udmrepo.StoreOptionS3SecretKey: "fake-sk", + udmrepo.StoreOptionS3Endpoint: "fake-endpoint", + udmrepo.StoreOptionOssRegion: "fake-region", + udmrepo.StoreOptionPrefix: "fake-prefix", + udmrepo.StoreOptionS3Token: "fake-token", + }, + expectedOptions: s3.Options{ + BucketName: "fake-bucket", + AccessKeyID: "fake-ak", + SecretAccessKey: "fake-sk", + Endpoint: "fake-endpoint", + Region: "fake-region", + Prefix: "fake-prefix", + SessionToken: "fake-token", + }, + }, + { + name: "with wrong tls", + flags: map[string]string{ + udmrepo.StoreOptionOssBucket: "fake-bucket", + udmrepo.StoreOptionS3DisableTLS: "fake-bool", + udmrepo.StoreOptionS3DisableTLSVerify: "fake-bool", + }, + expectedOptions: s3.Options{ + BucketName: "fake-bucket", + }, + }, + { + name: "with correct tls", + flags: map[string]string{ + udmrepo.StoreOptionOssBucket: "fake-bucket", + udmrepo.StoreOptionS3DisableTLS: "true", + udmrepo.StoreOptionS3DisableTLSVerify: "false", + }, + expectedOptions: s3.Options{ + BucketName: "fake-bucket", + DoNotUseTLS: true, + DoNotVerifyTLS: false, + }, + }, + { + name: "with wrong ca", + flags: map[string]string{ + udmrepo.StoreOptionOssBucket: "fake-bucket", + udmrepo.StoreOptionS3CustomCA: "fake-base-64", + }, + expectedOptions: s3.Options{ + BucketName: "fake-bucket", + }, + }, + { + name: "with correct ca", + flags: map[string]string{ + udmrepo.StoreOptionOssBucket: "fake-bucket", + udmrepo.StoreOptionS3CustomCA: "ZmFrZS1jYQ==", + }, + expectedOptions: s3.Options{ + BucketName: "fake-bucket", + RootCA: []byte{'f', 'a', 'k', 'e', '-', 'c', 'a'}, + }, + }, } for _, tc := range testCases { diff --git a/pkg/repository/udmrepo/kopialib/lib_repo_test.go b/pkg/repository/udmrepo/kopialib/lib_repo_test.go index 8c01827ae..e444c20d6 100644 --- a/pkg/repository/udmrepo/kopialib/lib_repo_test.go +++ b/pkg/repository/udmrepo/kopialib/lib_repo_test.go @@ -18,12 +18,14 @@ package kopialib import ( "context" + "math" "os" "testing" "time" "github.com/kopia/kopia/repo" "github.com/kopia/kopia/repo/manifest" + "github.com/kopia/kopia/repo/object" "github.com/pkg/errors" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/mock" @@ -713,3 +715,486 @@ func TestFlush(t *testing.T) { }) } } + +func TestNewObjectWriter(t *testing.T) { + rawObjWriter := repomocks.NewWriter(t) + testCases := []struct { + name string + rawWriter *repomocks.DirectRepositoryWriter + rawWriterRet object.Writer + expectedRet udmrepo.ObjectWriter + }{ + { + name: "raw writer is nil", + }, + { + name: "new object writer fail", + rawWriter: repomocks.NewDirectRepositoryWriter(t), + }, + { + name: "succeed", + rawWriter: repomocks.NewDirectRepositoryWriter(t), + rawWriterRet: rawObjWriter, + expectedRet: &kopiaObjectWriter{rawWriter: rawObjWriter}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + kr := &kopiaRepository{} + + if tc.rawWriter != nil { + tc.rawWriter.On("NewObjectWriter", mock.Anything, mock.Anything).Return(tc.rawWriterRet) + kr.rawWriter = tc.rawWriter + } + + ret := kr.NewObjectWriter(context.Background(), udmrepo.ObjectWriteOptions{}) + + assert.Equal(t, tc.expectedRet, ret) + }) + } +} + +func TestUpdateProgress(t *testing.T) { + testCases := []struct { + name string + progress int64 + uploaded int64 + throttle logThrottle + logMessage string + }{ + { + name: "should not output", + throttle: logThrottle{ + lastTime: math.MaxInt64, + }, + }, + { + name: "should output", + progress: 100, + uploaded: 200, + logMessage: "Repo uploaded 300 bytes.", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + logMessage := "" + kr := &kopiaRepository{ + logger: velerotest.NewSingleLogger(&logMessage), + throttle: tc.throttle, + uploaded: tc.uploaded, + } + + kr.updateProgress(tc.progress) + + if len(tc.logMessage) > 0 { + assert.Contains(t, logMessage, tc.logMessage) + } else { + assert.Equal(t, "", logMessage) + } + }) + } +} + +func TestReaderRead(t *testing.T) { + testCases := []struct { + name string + rawObjReader *repomocks.Reader + rawReaderRetErr error + expectedErr string + }{ + { + name: "raw reader is nil", + expectedErr: "object reader is closed or not open", + }, + { + name: "raw read fail", + rawObjReader: repomocks.NewReader(t), + rawReaderRetErr: errors.New("fake-read-error"), + expectedErr: "fake-read-error", + }, + { + name: "succeed", + rawObjReader: repomocks.NewReader(t), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + kr := &kopiaObjectReader{} + + if tc.rawObjReader != nil { + tc.rawObjReader.On("Read", mock.Anything).Return(0, tc.rawReaderRetErr) + kr.rawReader = tc.rawObjReader + } + + _, err := kr.Read(nil) + + if tc.expectedErr == "" { + assert.NoError(t, err) + } else { + assert.EqualError(t, err, tc.expectedErr) + } + }) + } +} + +func TestReaderSeek(t *testing.T) { + testCases := []struct { + name string + rawObjReader *repomocks.Reader + rawReaderRet int64 + rawReaderRetErr error + expectedRet int64 + expectedErr string + }{ + { + name: "raw reader is nil", + expectedErr: "object reader is closed or not open", + }, + { + name: "raw seek fail", + rawObjReader: repomocks.NewReader(t), + rawReaderRetErr: errors.New("fake-seek-error"), + expectedErr: "fake-seek-error", + }, + { + name: "succeed", + rawObjReader: repomocks.NewReader(t), + rawReaderRet: 100, + expectedRet: 100, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + kr := &kopiaObjectReader{} + + if tc.rawObjReader != nil { + tc.rawObjReader.On("Seek", mock.Anything, mock.Anything).Return(tc.rawReaderRet, tc.rawReaderRetErr) + kr.rawReader = tc.rawObjReader + } + + ret, err := kr.Seek(0, 0) + + if tc.expectedErr == "" { + assert.NoError(t, err) + assert.Equal(t, tc.expectedRet, ret) + } else { + assert.EqualError(t, err, tc.expectedErr) + } + }) + } +} + +func TestReaderClose(t *testing.T) { + testCases := []struct { + name string + rawObjReader *repomocks.Reader + rawReaderRetErr error + expectedErr string + }{ + { + name: "raw reader is nil", + }, + { + name: "raw close fail", + rawObjReader: repomocks.NewReader(t), + rawReaderRetErr: errors.New("fake-close-error"), + expectedErr: "fake-close-error", + }, + { + name: "succeed", + rawObjReader: repomocks.NewReader(t), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + kr := &kopiaObjectReader{} + + if tc.rawObjReader != nil { + tc.rawObjReader.On("Close").Return(tc.rawReaderRetErr) + kr.rawReader = tc.rawObjReader + } + + err := kr.Close() + + if tc.expectedErr == "" { + assert.NoError(t, err) + } else { + assert.EqualError(t, err, tc.expectedErr) + } + }) + } +} + +func TestReaderLength(t *testing.T) { + testCases := []struct { + name string + rawObjReader *repomocks.Reader + rawReaderRet int64 + expectedRet int64 + }{ + { + name: "raw reader is nil", + expectedRet: -1, + }, + { + name: "raw length fail", + rawObjReader: repomocks.NewReader(t), + rawReaderRet: 0, + expectedRet: 0, + }, + { + name: "succeed", + rawObjReader: repomocks.NewReader(t), + rawReaderRet: 200, + expectedRet: 200, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + kr := &kopiaObjectReader{} + + if tc.rawObjReader != nil { + tc.rawObjReader.On("Length").Return(tc.rawReaderRet) + kr.rawReader = tc.rawObjReader + } + + ret := kr.Length() + + assert.Equal(t, tc.expectedRet, ret) + }) + } +} + +func TestWriterWrite(t *testing.T) { + testCases := []struct { + name string + rawObjWriter *repomocks.Writer + rawWrtierRet int + rawWriterRetErr error + expectedRet int + expectedErr string + }{ + { + name: "raw writer is nil", + expectedErr: "object writer is closed or not open", + }, + { + name: "raw read fail", + rawObjWriter: repomocks.NewWriter(t), + rawWriterRetErr: errors.New("fake-write-error"), + expectedErr: "fake-write-error", + }, + { + name: "succeed", + rawObjWriter: repomocks.NewWriter(t), + rawWrtierRet: 200, + expectedRet: 200, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + kr := &kopiaObjectWriter{} + + if tc.rawObjWriter != nil { + tc.rawObjWriter.On("Write", mock.Anything).Return(tc.rawWrtierRet, tc.rawWriterRetErr) + kr.rawWriter = tc.rawObjWriter + } + + ret, err := kr.Write(nil) + + if tc.expectedErr == "" { + assert.NoError(t, err) + assert.Equal(t, tc.expectedRet, ret) + } else { + assert.EqualError(t, err, tc.expectedErr) + } + }) + } +} + +func TestWriterCheckpoint(t *testing.T) { + testCases := []struct { + name string + rawObjWriter *repomocks.Writer + rawWrtierRet object.ID + rawWriterRetErr error + expectedRet udmrepo.ID + expectedErr string + }{ + { + name: "raw writer is nil", + expectedErr: "object writer is closed or not open", + }, + { + name: "raw checkpoint fail", + rawObjWriter: repomocks.NewWriter(t), + rawWriterRetErr: errors.New("fake-checkpoint-error"), + expectedErr: "error to checkpoint object: fake-checkpoint-error", + }, + { + name: "succeed", + rawObjWriter: repomocks.NewWriter(t), + rawWrtierRet: object.ID{}, + expectedRet: udmrepo.ID(""), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + kr := &kopiaObjectWriter{} + + if tc.rawObjWriter != nil { + tc.rawObjWriter.On("Checkpoint").Return(tc.rawWrtierRet, tc.rawWriterRetErr) + kr.rawWriter = tc.rawObjWriter + } + + ret, err := kr.Checkpoint() + + if tc.expectedErr == "" { + assert.NoError(t, err) + assert.Equal(t, tc.expectedRet, ret) + } else { + assert.EqualError(t, err, tc.expectedErr) + } + }) + } +} + +func TestWriterResult(t *testing.T) { + testCases := []struct { + name string + rawObjWriter *repomocks.Writer + rawWrtierRet object.ID + rawWriterRetErr error + expectedRet udmrepo.ID + expectedErr string + }{ + { + name: "raw writer is nil", + expectedErr: "object writer is closed or not open", + }, + { + name: "raw result fail", + rawObjWriter: repomocks.NewWriter(t), + rawWriterRetErr: errors.New("fake-result-error"), + expectedErr: "error to wait object: fake-result-error", + }, + { + name: "succeed", + rawObjWriter: repomocks.NewWriter(t), + rawWrtierRet: object.ID{}, + expectedRet: udmrepo.ID(""), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + kr := &kopiaObjectWriter{} + + if tc.rawObjWriter != nil { + tc.rawObjWriter.On("Result").Return(tc.rawWrtierRet, tc.rawWriterRetErr) + kr.rawWriter = tc.rawObjWriter + } + + ret, err := kr.Result() + + if tc.expectedErr == "" { + assert.NoError(t, err) + assert.Equal(t, tc.expectedRet, ret) + } else { + assert.EqualError(t, err, tc.expectedErr) + } + }) + } +} + +func TestWriterClose(t *testing.T) { + testCases := []struct { + name string + rawObjWriter *repomocks.Writer + rawWriterRetErr error + expectedErr string + }{ + { + name: "raw writer is nil", + }, + { + name: "raw close fail", + rawObjWriter: repomocks.NewWriter(t), + rawWriterRetErr: errors.New("fake-close-error"), + expectedErr: "fake-close-error", + }, + { + name: "succeed", + rawObjWriter: repomocks.NewWriter(t), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + kr := &kopiaObjectWriter{} + + if tc.rawObjWriter != nil { + tc.rawObjWriter.On("Close").Return(tc.rawWriterRetErr) + kr.rawWriter = tc.rawObjWriter + } + + err := kr.Close() + + if tc.expectedErr == "" { + assert.NoError(t, err) + } else { + assert.EqualError(t, err, tc.expectedErr) + } + }) + } +} + +func TestMaintainProgress(t *testing.T) { + testCases := []struct { + name string + progress int64 + uploaded int64 + throttle logThrottle + logMessage string + }{ + { + name: "should not output", + throttle: logThrottle{ + lastTime: math.MaxInt64, + }, + }, + { + name: "should output", + progress: 100, + uploaded: 200, + logMessage: "Repo maintenance uploaded 300 bytes.", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + logMessage := "" + km := &kopiaMaintenance{ + logger: velerotest.NewSingleLogger(&logMessage), + throttle: tc.throttle, + uploaded: tc.uploaded, + } + + km.maintainProgress(tc.progress) + + if len(tc.logMessage) > 0 { + assert.Contains(t, logMessage, tc.logMessage) + } else { + assert.Equal(t, "", logMessage) + } + }) + } +} From 75833eaa5b9758c84b7f26c9e0b5a1f0786c3cf2 Mon Sep 17 00:00:00 2001 From: allenxu404 Date: Thu, 29 Jun 2023 14:10:03 +0800 Subject: [PATCH 12/28] fix hook filter display issue for backup describer Signed-off-by: allenxu404 --- changelogs/unreleased/6434-allenxu404 | 1 + pkg/cmd/util/output/backup_describer.go | 16 ++++++++-------- pkg/cmd/util/output/backup_describer_test.go | 12 ++++++++---- .../util/output/backup_structured_describer.go | 16 ++++++++-------- .../output/backup_structured_describer_test.go | 12 ++++++++---- 5 files changed, 33 insertions(+), 24 deletions(-) create mode 100644 changelogs/unreleased/6434-allenxu404 diff --git a/changelogs/unreleased/6434-allenxu404 b/changelogs/unreleased/6434-allenxu404 new file mode 100644 index 000000000..3c5b87b22 --- /dev/null +++ b/changelogs/unreleased/6434-allenxu404 @@ -0,0 +1 @@ +Fix hook filter display issue for backup describer \ No newline at end of file diff --git a/pkg/cmd/util/output/backup_describer.go b/pkg/cmd/util/output/backup_describer.go index 0dc42e147..23ca38976 100644 --- a/pkg/cmd/util/output/backup_describer.go +++ b/pkg/cmd/util/output/backup_describer.go @@ -231,31 +231,31 @@ func DescribeBackupSpec(d *Describer, spec velerov1api.BackupSpec) { d.Printf("\t\t%s:\n", backupResourceHookSpec.Name) d.Printf("\t\t\tNamespaces:\n") var s string - if len(spec.IncludedNamespaces) == 0 { + if len(backupResourceHookSpec.IncludedNamespaces) == 0 { s = "*" } else { - s = strings.Join(spec.IncludedNamespaces, ", ") + s = strings.Join(backupResourceHookSpec.IncludedNamespaces, ", ") } d.Printf("\t\t\t\tIncluded:\t%s\n", s) - if len(spec.ExcludedNamespaces) == 0 { + if len(backupResourceHookSpec.ExcludedNamespaces) == 0 { s = emptyDisplay } else { - s = strings.Join(spec.ExcludedNamespaces, ", ") + s = strings.Join(backupResourceHookSpec.ExcludedNamespaces, ", ") } d.Printf("\t\t\t\tExcluded:\t%s\n", s) d.Println() d.Printf("\t\t\tResources:\n") - if len(spec.IncludedResources) == 0 { + if len(backupResourceHookSpec.IncludedResources) == 0 { s = "*" } else { - s = strings.Join(spec.IncludedResources, ", ") + s = strings.Join(backupResourceHookSpec.IncludedResources, ", ") } d.Printf("\t\t\t\tIncluded:\t%s\n", s) - if len(spec.ExcludedResources) == 0 { + if len(backupResourceHookSpec.ExcludedResources) == 0 { s = emptyDisplay } else { - s = strings.Join(spec.ExcludedResources, ", ") + s = strings.Join(backupResourceHookSpec.ExcludedResources, ", ") } d.Printf("\t\t\t\tExcluded:\t%s\n", s) diff --git a/pkg/cmd/util/output/backup_describer_test.go b/pkg/cmd/util/output/backup_describer_test.go index f4ea3319f..7f76ac4da 100644 --- a/pkg/cmd/util/output/backup_describer_test.go +++ b/pkg/cmd/util/output/backup_describer_test.go @@ -72,6 +72,10 @@ func TestDescribeBackupSpec(t *testing.T) { }, }, }, + IncludedNamespaces: []string{"hook-inc-ns-1", "hook-inc-ns-2"}, + ExcludedNamespaces: []string{"hook-exc-ns-1", "hook-exc-ns-2"}, + IncludedResources: []string{"hook-inc-res-1", "hook-inc-res-2"}, + ExcludedResources: []string{"hook-exc-res-1", "hook-exc-res-2"}, }, }, }).Result().Spec @@ -102,12 +106,12 @@ Hooks: Resources: hook-1: Namespaces: - Included: inc-ns-1, inc-ns-2 - Excluded: exc-ns-1, exc-ns-2 + Included: hook-inc-ns-1, hook-inc-ns-2 + Excluded: hook-exc-ns-1, hook-exc-ns-2 Resources: - Included: inc-res-1, inc-res-2 - Excluded: exc-res-1, exc-res-2 + Included: hook-inc-res-1, hook-inc-res-2 + Excluded: hook-exc-res-1, hook-exc-res-2 Label selector: diff --git a/pkg/cmd/util/output/backup_structured_describer.go b/pkg/cmd/util/output/backup_structured_describer.go index 896eeafdd..cecab22cb 100644 --- a/pkg/cmd/util/output/backup_structured_describer.go +++ b/pkg/cmd/util/output/backup_structured_describer.go @@ -160,31 +160,31 @@ func DescribeBackupSpecInSF(d *StructuredDescriber, spec velerov1api.BackupSpec) ResourceDetails := make(map[string]interface{}) var s string namespaceInfo := make(map[string]string) - if len(spec.IncludedNamespaces) == 0 { + if len(backupResourceHookSpec.IncludedNamespaces) == 0 { s = "*" } else { - s = strings.Join(spec.IncludedNamespaces, ", ") + s = strings.Join(backupResourceHookSpec.IncludedNamespaces, ", ") } namespaceInfo["included"] = s - if len(spec.ExcludedNamespaces) == 0 { + if len(backupResourceHookSpec.ExcludedNamespaces) == 0 { s = emptyDisplay } else { - s = strings.Join(spec.ExcludedNamespaces, ", ") + s = strings.Join(backupResourceHookSpec.ExcludedNamespaces, ", ") } namespaceInfo["excluded"] = s ResourceDetails["namespaces"] = namespaceInfo resourcesInfo := make(map[string]string) - if len(spec.IncludedResources) == 0 { + if len(backupResourceHookSpec.IncludedResources) == 0 { s = "*" } else { - s = strings.Join(spec.IncludedResources, ", ") + s = strings.Join(backupResourceHookSpec.IncludedResources, ", ") } resourcesInfo["included"] = s - if len(spec.ExcludedResources) == 0 { + if len(backupResourceHookSpec.ExcludedResources) == 0 { s = emptyDisplay } else { - s = strings.Join(spec.ExcludedResources, ", ") + s = strings.Join(backupResourceHookSpec.ExcludedResources, ", ") } resourcesInfo["excluded"] = s ResourceDetails["resources"] = resourcesInfo diff --git a/pkg/cmd/util/output/backup_structured_describer_test.go b/pkg/cmd/util/output/backup_structured_describer_test.go index e58dc5d2a..2a0247bf7 100644 --- a/pkg/cmd/util/output/backup_structured_describer_test.go +++ b/pkg/cmd/util/output/backup_structured_describer_test.go @@ -56,6 +56,10 @@ func TestDescribeBackupInSF(t *testing.T) { }, }, }, + IncludedNamespaces: []string{"hook-inc-ns-1", "hook-inc-ns-2"}, + ExcludedNamespaces: []string{"hook-exc-ns-1", "hook-exc-ns-2"}, + IncludedResources: []string{"hook-inc-res-1", "hook-inc-res-2"}, + ExcludedResources: []string{"hook-exc-res-1", "hook-exc-res-2"}, }, }, }) @@ -83,8 +87,8 @@ func TestDescribeBackupInSF(t *testing.T) { "hook-1": map[string]interface{}{ "labelSelector": emptyDisplay, "namespaces": map[string]string{ - "included": "inc-ns-1, inc-ns-2", - "excluded": "exc-ns-1, exc-ns-2", + "included": "hook-inc-ns-1, hook-inc-ns-2", + "excluded": "hook-exc-ns-1, hook-exc-ns-2", }, "preExecHook": []map[string]interface{}{ { @@ -103,8 +107,8 @@ func TestDescribeBackupInSF(t *testing.T) { }, }, "resources": map[string]string{ - "included": "inc-res-1, inc-res-2", - "excluded": "exc-res-1, exc-res-2", + "included": "hook-inc-res-1, hook-inc-res-2", + "excluded": "hook-exc-res-1, hook-exc-res-2", }, }, }, From 1bfcee776c43a026b9310dd5e049cea940051eb9 Mon Sep 17 00:00:00 2001 From: Ming Date: Mon, 26 Jun 2023 15:53:21 +0000 Subject: [PATCH 13/28] Add data download controller Signed-off-by: Ming --- changelogs/unreleased/6436-qiuming-best | 1 + pkg/builder/data_download_builder.go | 103 ++++ pkg/cmd/cli/nodeagent/server.go | 4 + pkg/controller/data_download_controller.go | 487 ++++++++++++++++++ .../data_download_controller_test.go | 211 ++++++++ pkg/datapath/mocks/types.go | 86 ++++ pkg/exposer/mocks/generic_restore.go | 96 ++++ 7 files changed, 988 insertions(+) create mode 100644 changelogs/unreleased/6436-qiuming-best create mode 100644 pkg/builder/data_download_builder.go create mode 100644 pkg/controller/data_download_controller.go create mode 100644 pkg/controller/data_download_controller_test.go create mode 100644 pkg/datapath/mocks/types.go create mode 100644 pkg/exposer/mocks/generic_restore.go diff --git a/changelogs/unreleased/6436-qiuming-best b/changelogs/unreleased/6436-qiuming-best new file mode 100644 index 000000000..adb564bb9 --- /dev/null +++ b/changelogs/unreleased/6436-qiuming-best @@ -0,0 +1 @@ +Add data download controller for data mover diff --git a/pkg/builder/data_download_builder.go b/pkg/builder/data_download_builder.go new file mode 100644 index 000000000..0842c0c32 --- /dev/null +++ b/pkg/builder/data_download_builder.go @@ -0,0 +1,103 @@ +/* +Copyright The Velero Contributors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package builder + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + velerov2alpha1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1" +) + +// DataDownloadBuilder builds DataDownload objects +type DataDownloadBuilder struct { + object *velerov2alpha1api.DataDownload +} + +// ForDataDownload is the constructor for a DataDownloadBuilder. +func ForDataDownload(ns, name string) *DataDownloadBuilder { + return &DataDownloadBuilder{ + object: &velerov2alpha1api.DataDownload{ + TypeMeta: metav1.TypeMeta{ + APIVersion: velerov2alpha1api.SchemeGroupVersion.String(), + Kind: "DataDownloadload", + }, + ObjectMeta: metav1.ObjectMeta{ + Namespace: ns, + Name: name, + }, + }, + } +} + +// Result returns the built DataDownload. +func (d *DataDownloadBuilder) Result() *velerov2alpha1api.DataDownload { + return d.object +} + +// BackupStorageLocation sets the DataDownload's backup storage location. +func (d *DataDownloadBuilder) BackupStorageLocation(name string) *DataDownloadBuilder { + d.object.Spec.BackupStorageLocation = name + return d +} + +// Phase sets the DataDownload's phase. +func (d *DataDownloadBuilder) Phase(phase velerov2alpha1api.DataDownloadPhase) *DataDownloadBuilder { + d.object.Status.Phase = phase + return d +} + +// SnapshotID sets the DataDownload's SnapshotID. +func (d *DataDownloadBuilder) SnapshotID(id string) *DataDownloadBuilder { + d.object.Spec.SnapshotID = id + return d +} + +// DataMover sets the DataDownload's DataMover. +func (d *DataDownloadBuilder) DataMover(dataMover string) *DataDownloadBuilder { + d.object.Spec.DataMover = dataMover + return d +} + +// SourceNamespace sets the DataDownload's SourceNamespace. +func (d *DataDownloadBuilder) SourceNamespace(sourceNamespace string) *DataDownloadBuilder { + d.object.Spec.SourceNamespace = sourceNamespace + return d +} + +// TargetVolume sets the DataDownload's TargetVolume. +func (d *DataDownloadBuilder) TargetVolume(targetVolume velerov2alpha1api.TargetVolumeSpec) *DataDownloadBuilder { + d.object.Spec.TargetVolume = targetVolume + return d +} + +// Cancel sets the DataDownload's Cancel. +func (d *DataDownloadBuilder) Cancel(cancel bool) *DataDownloadBuilder { + d.object.Spec.Cancel = cancel + return d +} + +// OperationTimeout sets the DataDownload's OperationTimeout. +func (d *DataDownloadBuilder) OperationTimeout(timeout metav1.Duration) *DataDownloadBuilder { + d.object.Spec.OperationTimeout = timeout + return d +} + +// DataMoverConfig sets the DataDownload's DataMoverConfig. +func (d *DataDownloadBuilder) DataMoverConfig(config *map[string]string) *DataDownloadBuilder { + d.object.Spec.DataMoverConfig = *config + return d +} diff --git a/pkg/cmd/cli/nodeagent/server.go b/pkg/cmd/cli/nodeagent/server.go index 5e138b250..3f635602e 100644 --- a/pkg/cmd/cli/nodeagent/server.go +++ b/pkg/cmd/cli/nodeagent/server.go @@ -260,6 +260,10 @@ func (s *nodeAgentServer) run() { s.logger.WithError(err).Fatal("Unable to create the data upload controller") } + if err = controller.NewDataDownloadReconciler(s.mgr.GetClient(), s.kubeClient, repoEnsurer, credentialGetter, s.nodeName, s.logger).SetupWithManager(s.mgr); err != nil { + s.logger.WithError(err).Fatal("Unable to create the data download controller") + } + s.logger.Info("Controllers starting...") if err := s.mgr.Start(ctrl.SetupSignalHandler()); err != nil { diff --git a/pkg/controller/data_download_controller.go b/pkg/controller/data_download_controller.go new file mode 100644 index 000000000..1f0e78c69 --- /dev/null +++ b/pkg/controller/data_download_controller.go @@ -0,0 +1,487 @@ +/* +Copyright The Velero Contributors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package controller + +import ( + "context" + "fmt" + "time" + + "github.com/pkg/errors" + "github.com/sirupsen/logrus" + v1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/kubernetes" + "k8s.io/utils/clock" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/builder" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/predicate" + "sigs.k8s.io/controller-runtime/pkg/reconcile" + "sigs.k8s.io/controller-runtime/pkg/source" + + "github.com/vmware-tanzu/velero/internal/credentials" + "github.com/vmware-tanzu/velero/pkg/apis/velero/shared" + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + velerov2alpha1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1" + datamover "github.com/vmware-tanzu/velero/pkg/datamover" + "github.com/vmware-tanzu/velero/pkg/datapath" + "github.com/vmware-tanzu/velero/pkg/exposer" + repository "github.com/vmware-tanzu/velero/pkg/repository" + "github.com/vmware-tanzu/velero/pkg/uploader" + "github.com/vmware-tanzu/velero/pkg/util/filesystem" + "github.com/vmware-tanzu/velero/pkg/util/kube" +) + +// DataDownloadReconciler reconciles a DataDownload object +type DataDownloadReconciler struct { + client client.Client + kubeClient kubernetes.Interface + logger logrus.FieldLogger + credentialGetter *credentials.CredentialGetter + fileSystem filesystem.Interface + clock clock.WithTickerAndDelayedExecution + restoreExposer exposer.GenericRestoreExposer + nodeName string + repositoryEnsurer *repository.Ensurer + dataPathMgr *datapath.Manager +} + +func NewDataDownloadReconciler(client client.Client, kubeClient kubernetes.Interface, + repoEnsurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter, nodeName string, logger logrus.FieldLogger) *DataDownloadReconciler { + return &DataDownloadReconciler{ + client: client, + kubeClient: kubeClient, + logger: logger.WithField("controller", "DataDownload"), + credentialGetter: credentialGetter, + fileSystem: filesystem.NewFileSystem(), + clock: &clock.RealClock{}, + nodeName: nodeName, + repositoryEnsurer: repoEnsurer, + restoreExposer: exposer.NewGenericRestoreExposer(kubeClient, logger), + dataPathMgr: datapath.NewManager(1), + } +} + +// +kubebuilder:rbac:groups=velero.io,resources=datadownloads,verbs=get;list;watch;create;update;patch;delete +// +kubebuilder:rbac:groups=velero.io,resources=datadownloads/status,verbs=get;update;patch +// +kubebuilder:rbac:groups="",resources=pods,verbs=get +// +kubebuilder:rbac:groups="",resources=persistentvolumes,verbs=get +// +kubebuilder:rbac:groups="",resources=persistentvolumerclaims,verbs=get + +func (r *DataDownloadReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { + log := r.logger.WithFields(logrus.Fields{ + "controller": "datadownload", + "datadownload": req.NamespacedName, + }) + + log.Infof("Reconcile %s", req.Name) + + dd := &velerov2alpha1api.DataDownload{} + if err := r.client.Get(ctx, types.NamespacedName{Namespace: req.Namespace, Name: req.Name}, dd); err != nil { + if apierrors.IsNotFound(err) { + log.Warn("DataDownload not found, skip") + return ctrl.Result{}, nil + } + log.WithError(err).Error("Unable to get the DataDownload") + return ctrl.Result{}, err + } + + if dd.Spec.DataMover != "" && dd.Spec.DataMover != dataMoverType { + log.WithField("data mover", dd.Spec.DataMover).Info("it is not one built-in data mover which is not supported by Velero") + return ctrl.Result{}, nil + } + + if r.restoreExposer == nil { + return r.errorOut(ctx, dd, errors.New("uninitialized generic exposer"), "uninitialized exposer", log) + } + + if dd.Status.Phase == "" || dd.Status.Phase == velerov2alpha1api.DataDownloadPhaseNew { + log.Info("Data download starting") + + if _, err := r.getTargetPVC(ctx, dd); err != nil { + return ctrl.Result{Requeue: true}, nil + } + + accepted, err := r.acceptDataDownload(ctx, dd) + if err != nil { + return r.errorOut(ctx, dd, err, "error to accept the data download", log) + } + + if !accepted { + log.Debug("Data download is not accepted") + return ctrl.Result{}, nil + } + + log.Info("Data download is accepted") + + hostingPodLabels := map[string]string{velerov1api.DataDownloadLabel: dd.Name} + + // ep.Expose() will trigger to create one pod whose volume is restored by a given volume snapshot, + // but the pod maybe is not in the same node of the current controller, so we need to return it here. + // And then only the controller who is in the same node could do the rest work. + err = r.restoreExposer.Expose(ctx, getDataDownloadOwnerObject(dd), dd.Spec.TargetVolume.PVC, dd.Spec.TargetVolume.Namespace, hostingPodLabels, dd.Spec.OperationTimeout.Duration) + if err != nil { + return r.errorOut(ctx, dd, err, "error to start restore expose", log) + } + log.Info("Restore is exposed") + + return ctrl.Result{}, nil + } else if dd.Status.Phase == velerov2alpha1api.DataDownloadPhasePrepared { + log.Info("Data download is prepared") + fsRestore := r.dataPathMgr.GetAsyncBR(dd.Name) + + if fsRestore != nil { + log.Info("Cancellable data path is already started") + return ctrl.Result{}, nil + } + + result, err := r.restoreExposer.GetExposed(ctx, getDataDownloadOwnerObject(dd), r.client, r.nodeName, dd.Spec.OperationTimeout.Duration) + if err != nil { + return r.errorOut(ctx, dd, err, "restore exposer is not ready", log) + } else if result == nil { + log.Debug("Get empty restore exposer") + return ctrl.Result{}, nil + } + + log.Info("Restore PVC is ready") + + // Update status to InProgress + original := dd.DeepCopy() + dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseInProgress + dd.Status.StartTimestamp = &metav1.Time{Time: r.clock.Now()} + if err := r.client.Patch(ctx, dd, client.MergeFrom(original)); err != nil { + log.WithError(err).Error("Unable to update status to in progress") + return ctrl.Result{}, err + } + + log.Info("Data download is marked as in progress") + + return r.runCancelableDataPath(ctx, dd, result, log) + } else if dd.Status.Phase == velerov2alpha1api.DataDownloadPhaseInProgress { + log.Info("Data download is in progress") + if dd.Spec.Cancel { + fsRestore := r.dataPathMgr.GetAsyncBR(dd.Name) + if fsRestore == nil { + return ctrl.Result{}, nil + } + + log.Info("Data download is being canceled") + // Update status to Canceling. + original := dd.DeepCopy() + dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseCanceling + if err := r.client.Patch(ctx, dd, client.MergeFrom(original)); err != nil { + log.WithError(err).Error("error updating data download status") + return ctrl.Result{}, err + } + + fsRestore.Cancel() + return ctrl.Result{}, nil + } + + return ctrl.Result{}, nil + } else { + log.Debugf("Data download now is in %s phase and do nothing by current %s controller", dd.Status.Phase, r.nodeName) + return ctrl.Result{}, nil + } +} + +func (r *DataDownloadReconciler) runCancelableDataPath(ctx context.Context, dd *velerov2alpha1api.DataDownload, res *exposer.ExposeResult, log logrus.FieldLogger) (reconcile.Result, error) { + log.Info("Creating data path routine") + callbacks := datapath.Callbacks{ + OnCompleted: r.OnDataDownloadCompleted, + OnFailed: r.OnDataDownloadFailed, + OnCancelled: r.OnDataDownloadCancelled, + OnProgress: r.OnDataDownloadProgress, + } + + fsRestore, err := r.dataPathMgr.CreateFileSystemBR(dd.Name, dataUploadDownloadRequestor, ctx, r.client, dd.Namespace, callbacks, log) + if err != nil { + if err == datapath.ConcurrentLimitExceed { + log.Info("runCancelableDataDownload is concurrent limited") + return ctrl.Result{Requeue: true, RequeueAfter: time.Minute}, nil + } else { + return r.errorOut(ctx, dd, err, "error to create data path", log) + } + } + + path, err := exposer.GetPodVolumeHostPath(ctx, res.ByPod.HostingPod, res.ByPod.PVC, r.client, r.fileSystem, log) + if err != nil { + return r.errorOut(ctx, dd, err, "error exposing host path for pod volume", log) + } + + log.WithField("path", path.ByPath).Debug("Found host path") + if err := fsRestore.Init(ctx, dd.Spec.BackupStorageLocation, dd.Spec.SourceNamespace, datamover.GetUploaderType(dd.Spec.DataMover), + velerov1api.BackupRepositoryTypeKopia, "", r.repositoryEnsurer, r.credentialGetter); err != nil { + return r.errorOut(ctx, dd, err, "error to initialize data path", log) + } + log.WithField("path", path.ByPath).Info("fs init") + + if err := fsRestore.StartRestore(dd.Spec.SnapshotID, path); err != nil { + return r.errorOut(ctx, dd, err, fmt.Sprintf("error starting data path %s restore", path.ByPath), log) + } + + log.WithField("path", path.ByPath).Info("Async fs restore data path started") + return ctrl.Result{}, nil +} + +func (r *DataDownloadReconciler) OnDataDownloadCompleted(ctx context.Context, namespace string, ddName string, result datapath.Result) { + defer r.closeDataPath(ctx, ddName) + + log := r.logger.WithField("datadownload", ddName) + log.Info("Async fs restore data path completed") + + var dd velerov2alpha1api.DataDownload + if err := r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, &dd); err != nil { + log.WithError(err).Warn("Failed to get datadownload on completion") + return + } + + objRef := getDataDownloadOwnerObject(&dd) + err := r.restoreExposer.RebindVolume(ctx, objRef, dd.Spec.TargetVolume.PVC, dd.Spec.TargetVolume.Namespace, dd.Spec.OperationTimeout.Duration) + if err != nil { + log.WithError(err).Error("Failed to rebind PV to target PVC on completion") + return + } + + log.Info("Cleaning up exposed environment") + r.restoreExposer.CleanUp(ctx, objRef) + + original := dd.DeepCopy() + dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseCompleted + dd.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()} + if err := r.client.Patch(ctx, &dd, client.MergeFrom(original)); err != nil { + log.WithError(err).Error("error updating data download status") + } else { + log.Infof("Data download is marked as %s", dd.Status.Phase) + } +} + +func (r *DataDownloadReconciler) OnDataDownloadFailed(ctx context.Context, namespace string, ddName string, err error) { + defer r.closeDataPath(ctx, ddName) + + log := r.logger.WithField("datadownload", ddName) + + log.WithError(err).Error("Async fs restore data path failed") + + var dd velerov2alpha1api.DataDownload + if getErr := r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, &dd); getErr != nil { + log.WithError(getErr).Warn("Failed to get data download on failure") + } else { + if _, errOut := r.errorOut(ctx, &dd, err, "data path restore failed", log); err != nil { + log.WithError(err).Warnf("Failed to patch data download with err %v", errOut) + } + } +} + +func (r *DataDownloadReconciler) OnDataDownloadCancelled(ctx context.Context, namespace string, ddName string) { + defer r.closeDataPath(ctx, ddName) + + log := r.logger.WithField("datadownload", ddName) + + log.Warn("Async fs backup data path canceled") + + var dd velerov2alpha1api.DataDownload + if getErr := r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, &dd); getErr != nil { + log.WithError(getErr).Warn("Failed to get datadownload on cancel") + } else { + // cleans up any objects generated during the snapshot expose + r.restoreExposer.CleanUp(ctx, getDataDownloadOwnerObject(&dd)) + + original := dd.DeepCopy() + dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseCanceled + if dd.Status.StartTimestamp.IsZero() { + dd.Status.StartTimestamp = &metav1.Time{Time: r.clock.Now()} + } + dd.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()} + if err := r.client.Patch(ctx, &dd, client.MergeFrom(original)); err != nil { + log.WithError(err).Error("error updating data download status") + } + } +} + +func (r *DataDownloadReconciler) OnDataDownloadProgress(ctx context.Context, namespace string, ddName string, progress *uploader.Progress) { + log := r.logger.WithField("datadownload", ddName) + + var dd velerov2alpha1api.DataDownload + if err := r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, &dd); err != nil { + log.WithError(err).Warn("Failed to get data download on progress") + return + } + + original := dd.DeepCopy() + dd.Status.Progress = shared.DataMoveOperationProgress{TotalBytes: progress.TotalBytes, BytesDone: progress.BytesDone} + + if err := r.client.Patch(ctx, &dd, client.MergeFrom(original)); err != nil { + log.WithError(err).Error("Failed to update restore snapshot progress") + } +} + +// SetupWithManager registers the DataDownload controller. +// The fresh new DataDownload CR first created will trigger to create one pod (long time, maybe failure or unknown status) by one of the datadownload controllers +// then the request will get out of the Reconcile queue immediately by not blocking others' CR handling, in order to finish the rest data download process we need to +// re-enqueue the previous related request once the related pod is in running status to keep going on the rest logic. and below logic will avoid handling the unwanted +// pod status and also avoid block others CR handling +func (r *DataDownloadReconciler) SetupWithManager(mgr ctrl.Manager) error { + return ctrl.NewControllerManagedBy(mgr). + For(&velerov2alpha1api.DataDownload{}). + Watches(&source.Kind{Type: &v1.Pod{}}, kube.EnqueueRequestsFromMapUpdateFunc(r.findSnapshotRestoreForPod), + builder.WithPredicates(predicate.Funcs{ + UpdateFunc: func(ue event.UpdateEvent) bool { + newObj := ue.ObjectNew.(*v1.Pod) + + if _, ok := newObj.Labels[velerov1api.DataDownloadLabel]; !ok { + return false + } + + if newObj.Status.Phase != v1.PodRunning { + return false + } + + if newObj.Spec.NodeName == "" { + return false + } + + return true + }, + CreateFunc: func(event.CreateEvent) bool { + return false + }, + DeleteFunc: func(de event.DeleteEvent) bool { + return false + }, + GenericFunc: func(ge event.GenericEvent) bool { + return false + }, + })). + Complete(r) +} + +func (r *DataDownloadReconciler) findSnapshotRestoreForPod(podObj client.Object) []reconcile.Request { + pod := podObj.(*v1.Pod) + + dd := &velerov2alpha1api.DataDownload{} + err := r.client.Get(context.Background(), types.NamespacedName{ + Namespace: pod.Namespace, + Name: pod.Labels[velerov1api.DataDownloadLabel], + }, dd) + + if err != nil { + r.logger.WithField("Restore pod", pod.Name).WithError(err).Error("unable to get DataDownload") + return []reconcile.Request{} + } + + if dd.Status.Phase != velerov2alpha1api.DataDownloadPhaseAccepted { + return []reconcile.Request{} + } + + requests := make([]reconcile.Request, 1) + + r.logger.WithField("Restore pod", pod.Name).Infof("Preparing data download %s", dd.Name) + err = r.patchDataDownload(context.Background(), dd, prepareDataDownload) + if err != nil { + r.logger.WithField("Restore pod", pod.Name).WithError(err).Error("unable to patch data download") + return []reconcile.Request{} + } + + requests[0] = reconcile.Request{ + NamespacedName: types.NamespacedName{ + Namespace: dd.Namespace, + Name: dd.Name, + }, + } + + return requests +} + +func (r *DataDownloadReconciler) patchDataDownload(ctx context.Context, req *velerov2alpha1api.DataDownload, mutate func(*velerov2alpha1api.DataDownload)) error { + original := req.DeepCopy() + mutate(req) + if err := r.client.Patch(ctx, req, client.MergeFrom(original)); err != nil { + return errors.Wrap(err, "error patching data download") + } + + return nil +} + +func prepareDataDownload(ssb *velerov2alpha1api.DataDownload) { + ssb.Status.Phase = velerov2alpha1api.DataDownloadPhasePrepared +} + +func (r *DataDownloadReconciler) errorOut(ctx context.Context, dd *velerov2alpha1api.DataDownload, err error, msg string, log logrus.FieldLogger) (ctrl.Result, error) { + r.restoreExposer.CleanUp(ctx, getDataDownloadOwnerObject(dd)) + return ctrl.Result{}, r.updateStatusToFailed(ctx, dd, err, msg, log) +} + +func (r *DataDownloadReconciler) updateStatusToFailed(ctx context.Context, dd *velerov2alpha1api.DataDownload, err error, msg string, log logrus.FieldLogger) error { + log.Infof("update data download status to %v", dd.Status.Phase) + original := dd.DeepCopy() + dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseFailed + dd.Status.Message = errors.WithMessage(err, msg).Error() + dd.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()} + + if err = r.client.Patch(ctx, dd, client.MergeFrom(original)); err != nil { + log.WithError(err).Error("error updating DataDownload status") + return err + } + + return nil +} + +func (r *DataDownloadReconciler) acceptDataDownload(ctx context.Context, dd *velerov2alpha1api.DataDownload) (bool, error) { + updated := dd.DeepCopy() + updated.Status.Phase = velerov2alpha1api.DataDownloadPhaseAccepted + + r.logger.Infof("Accepting snapshot restore %s", dd.Name) + // For all data download controller in each node-agent will try to update download CR, and only one controller will success, + // and the success one could handle later logic + err := r.client.Update(ctx, updated) + if err == nil { + return true, nil + } else if apierrors.IsConflict(err) { + r.logger.WithField("DataDownload", dd.Name).Error("This data download restore has been accepted by others") + return false, nil + } else { + return false, err + } +} + +func (r *DataDownloadReconciler) getTargetPVC(ctx context.Context, dd *velerov2alpha1api.DataDownload) (*v1.PersistentVolumeClaim, error) { + return r.kubeClient.CoreV1().PersistentVolumeClaims(dd.Spec.TargetVolume.Namespace).Get(ctx, dd.Spec.TargetVolume.PVC, metav1.GetOptions{}) +} + +func (r *DataDownloadReconciler) closeDataPath(ctx context.Context, ddName string) { + fsBackup := r.dataPathMgr.GetAsyncBR(ddName) + if fsBackup != nil { + fsBackup.Close(ctx) + } + + r.dataPathMgr.RemoveAsyncBR(ddName) +} + +func getDataDownloadOwnerObject(dd *velerov2alpha1api.DataDownload) v1.ObjectReference { + return v1.ObjectReference{ + Kind: dd.Kind, + Namespace: dd.Namespace, + Name: dd.Name, + UID: dd.UID, + APIVersion: dd.APIVersion, + } +} diff --git a/pkg/controller/data_download_controller_test.go b/pkg/controller/data_download_controller_test.go new file mode 100644 index 000000000..5db1170c7 --- /dev/null +++ b/pkg/controller/data_download_controller_test.go @@ -0,0 +1,211 @@ +/* +Copyright The Velero Contributors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package controller + +import ( + "context" + "fmt" + "testing" + + "github.com/pkg/errors" + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/mock" + "github.com/stretchr/testify/require" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + clientgofake "k8s.io/client-go/kubernetes/fake" + ctrl "sigs.k8s.io/controller-runtime" + kbclient "sigs.k8s.io/controller-runtime/pkg/client" + + "github.com/vmware-tanzu/velero/internal/credentials" + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + velerov2alpha1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1" + "github.com/vmware-tanzu/velero/pkg/builder" + "github.com/vmware-tanzu/velero/pkg/datapath" + "github.com/vmware-tanzu/velero/pkg/exposer" + velerotest "github.com/vmware-tanzu/velero/pkg/test" + + "sigs.k8s.io/controller-runtime/pkg/client/fake" + + datapathmockes "github.com/vmware-tanzu/velero/pkg/datapath/mocks" + exposermockes "github.com/vmware-tanzu/velero/pkg/exposer/mocks" +) + +const dataDownloadName string = "datadownload-1" + +func dataDownloadBuilder() *builder.DataDownloadBuilder { + return builder.ForDataDownload(velerov1api.DefaultNamespace, dataDownloadName). + BackupStorageLocation("bsl-loc"). + DataMover("velero"). + SnapshotID("test-snapshot-id").TargetVolume(velerov2alpha1api.TargetVolumeSpec{ + PV: "test-pv", + PVC: "test-pvc", + Namespace: "test-ns", + }) +} + +func initDataDownloadReconciler(objects []runtime.Object, needError ...bool) (*DataDownloadReconciler, error) { + scheme := runtime.NewScheme() + err := velerov1api.AddToScheme(scheme) + if err != nil { + return nil, err + } + err = velerov2alpha1api.AddToScheme(scheme) + if err != nil { + return nil, err + } + err = corev1.AddToScheme(scheme) + if err != nil { + return nil, err + } + + fakeClient := &FakeClient{ + Client: fake.NewClientBuilder().WithScheme(scheme).Build(), + } + + if len(needError) == 4 { + fakeClient.getError = needError[0] + fakeClient.createError = needError[1] + fakeClient.updateError = needError[2] + fakeClient.patchError = needError[3] + } + + fakeKubeClient := clientgofake.NewSimpleClientset(objects...) + fakeFS := velerotest.NewFakeFileSystem() + pathGlob := fmt.Sprintf("/host_pods/%s/volumes/*/%s", "", dataDownloadName) + _, err = fakeFS.Create(pathGlob) + if err != nil { + return nil, err + } + + credentialFileStore, err := credentials.NewNamespacedFileStore( + fakeClient, + velerov1api.DefaultNamespace, + "/tmp/credentials", + fakeFS, + ) + if err != nil { + return nil, err + } + return NewDataDownloadReconciler(fakeClient, fakeKubeClient, nil, &credentials.CredentialGetter{FromFile: credentialFileStore}, "test_node", velerotest.NewLogger()), nil +} + +func TestDataDownloadReconcile(t *testing.T) { + tests := []struct { + name string + dd *velerov2alpha1api.DataDownload + targetPVC *corev1.PersistentVolumeClaim + dataMgr *datapath.Manager + needErrs []bool + isExposeErr bool + isGetExposeErr bool + expectedStatusMsg string + }{ + { + name: "Restore is exposed", + dd: dataDownloadBuilder().Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + }, + { + name: "Get empty restore exposer", + dd: dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhasePrepared).Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + }, + { + name: "Failed to get restore exposer", + dd: dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhasePrepared).Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + expectedStatusMsg: "Error to get restore exposer", + isGetExposeErr: true, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + r, err := initDataDownloadReconciler([]runtime.Object{test.targetPVC}, test.needErrs...) + require.NoError(t, err) + defer func() { + r.client.Delete(ctx, test.dd, &kbclient.DeleteOptions{}) + if test.targetPVC != nil { + r.client.Delete(ctx, test.targetPVC, &kbclient.DeleteOptions{}) + } + }() + + ctx := context.Background() + if test.dd.Namespace == velerov1api.DefaultNamespace { + err = r.client.Create(ctx, test.dd) + require.NoError(t, err) + } + + if test.dataMgr != nil { + r.dataPathMgr = test.dataMgr + } else { + r.dataPathMgr = datapath.NewManager(1) + } + + datapath.FSBRCreator = func(string, string, kbclient.Client, string, datapath.Callbacks, logrus.FieldLogger) datapath.AsyncBR { + return datapathmockes.NewAsyncBR(t) + } + + if test.isExposeErr || test.isGetExposeErr { + r.restoreExposer = func() exposer.GenericRestoreExposer { + ep := exposermockes.NewGenericRestoreExposer(t) + if test.isExposeErr { + ep.On("Expose", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(errors.New("Error to expose restore exposer")) + } + + if test.isGetExposeErr { + ep.On("GetExposed", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(nil, errors.New("Error to get restore exposer")) + } + + ep.On("CleanUp", mock.Anything, mock.Anything).Return() + return ep + }() + } + + if test.dd.Status.Phase == velerov2alpha1api.DataDownloadPhaseInProgress { + if fsBR := r.dataPathMgr.GetAsyncBR(test.dd.Name); fsBR == nil { + _, err := r.dataPathMgr.CreateFileSystemBR(test.dd.Name, pVBRRequestor, ctx, r.client, velerov1api.DefaultNamespace, datapath.Callbacks{OnCancelled: r.OnDataDownloadCancelled}, velerotest.NewLogger()) + require.NoError(t, err) + } + } + actualResult, err := r.Reconcile(ctx, ctrl.Request{ + NamespacedName: types.NamespacedName{ + Namespace: velerov1api.DefaultNamespace, + Name: test.dd.Name, + }, + }) + + require.Nil(t, err) + require.NotNil(t, actualResult) + + dd := velerov2alpha1api.DataDownload{} + err = r.client.Get(ctx, kbclient.ObjectKey{ + Name: test.dd.Name, + Namespace: test.dd.Namespace, + }, &dd) + + if test.isGetExposeErr { + assert.Contains(t, dd.Status.Message, test.expectedStatusMsg) + } + require.Nil(t, err) + t.Logf("%s: \n %v \n", test.name, dd) + }) + } +} diff --git a/pkg/datapath/mocks/types.go b/pkg/datapath/mocks/types.go new file mode 100644 index 000000000..ecf655df0 --- /dev/null +++ b/pkg/datapath/mocks/types.go @@ -0,0 +1,86 @@ +// Code generated by mockery v2.20.0. DO NOT EDIT. + +package mocks + +import ( + context "context" + + credentials "github.com/vmware-tanzu/velero/internal/credentials" + datapath "github.com/vmware-tanzu/velero/pkg/datapath" + + mock "github.com/stretchr/testify/mock" + + repository "github.com/vmware-tanzu/velero/pkg/repository" +) + +// AsyncBR is an autogenerated mock type for the AsyncBR type +type AsyncBR struct { + mock.Mock +} + +// Cancel provides a mock function with given fields: +func (_m *AsyncBR) Cancel() { + _m.Called() +} + +// Close provides a mock function with given fields: ctx +func (_m *AsyncBR) Close(ctx context.Context) { + _m.Called(ctx) +} + +// Init provides a mock function with given fields: ctx, bslName, sourceNamespace, uploaderType, repositoryType, repoIdentifier, repositoryEnsurer, credentialGetter +func (_m *AsyncBR) Init(ctx context.Context, bslName string, sourceNamespace string, uploaderType string, repositoryType string, repoIdentifier string, repositoryEnsurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter) error { + ret := _m.Called(ctx, bslName, sourceNamespace, uploaderType, repositoryType, repoIdentifier, repositoryEnsurer, credentialGetter) + + var r0 error + if rf, ok := ret.Get(0).(func(context.Context, string, string, string, string, string, *repository.Ensurer, *credentials.CredentialGetter) error); ok { + r0 = rf(ctx, bslName, sourceNamespace, uploaderType, repositoryType, repoIdentifier, repositoryEnsurer, credentialGetter) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// StartBackup provides a mock function with given fields: source, realSource, parentSnapshot, forceFull, tags +func (_m *AsyncBR) StartBackup(source datapath.AccessPoint, realSource string, parentSnapshot string, forceFull bool, tags map[string]string) error { + ret := _m.Called(source, realSource, parentSnapshot, forceFull, tags) + + var r0 error + if rf, ok := ret.Get(0).(func(datapath.AccessPoint, string, string, bool, map[string]string) error); ok { + r0 = rf(source, realSource, parentSnapshot, forceFull, tags) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// StartRestore provides a mock function with given fields: snapshotID, target +func (_m *AsyncBR) StartRestore(snapshotID string, target datapath.AccessPoint) error { + ret := _m.Called(snapshotID, target) + + var r0 error + if rf, ok := ret.Get(0).(func(string, datapath.AccessPoint) error); ok { + r0 = rf(snapshotID, target) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type mockConstructorTestingTNewAsyncBR interface { + mock.TestingT + Cleanup(func()) +} + +// NewAsyncBR creates a new instance of AsyncBR. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations. +func NewAsyncBR(t mockConstructorTestingTNewAsyncBR) *AsyncBR { + mock := &AsyncBR{} + mock.Mock.Test(t) + + t.Cleanup(func() { mock.AssertExpectations(t) }) + + return mock +} diff --git a/pkg/exposer/mocks/generic_restore.go b/pkg/exposer/mocks/generic_restore.go new file mode 100644 index 000000000..a7d20f87c --- /dev/null +++ b/pkg/exposer/mocks/generic_restore.go @@ -0,0 +1,96 @@ +// Code generated by mockery v2.20.0. DO NOT EDIT. + +package mocks + +import ( + context "context" + + client "sigs.k8s.io/controller-runtime/pkg/client" + + exposer "github.com/vmware-tanzu/velero/pkg/exposer" + + mock "github.com/stretchr/testify/mock" + + time "time" + + v1 "k8s.io/api/core/v1" +) + +// GenericRestoreExposer is an autogenerated mock type for the GenericRestoreExposer type +type GenericRestoreExposer struct { + mock.Mock +} + +// CleanUp provides a mock function with given fields: _a0, _a1 +func (_m *GenericRestoreExposer) CleanUp(_a0 context.Context, _a1 v1.ObjectReference) { + _m.Called(_a0, _a1) +} + +// Expose provides a mock function with given fields: _a0, _a1, _a2, _a3, _a4, _a5 +func (_m *GenericRestoreExposer) Expose(_a0 context.Context, _a1 v1.ObjectReference, _a2 string, _a3 string, _a4 map[string]string, _a5 time.Duration) error { + ret := _m.Called(_a0, _a1, _a2, _a3, _a4, _a5) + + var r0 error + if rf, ok := ret.Get(0).(func(context.Context, v1.ObjectReference, string, string, map[string]string, time.Duration) error); ok { + r0 = rf(_a0, _a1, _a2, _a3, _a4, _a5) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// GetExposed provides a mock function with given fields: _a0, _a1, _a2, _a3, _a4 +func (_m *GenericRestoreExposer) GetExposed(_a0 context.Context, _a1 v1.ObjectReference, _a2 client.Client, _a3 string, _a4 time.Duration) (*exposer.ExposeResult, error) { + ret := _m.Called(_a0, _a1, _a2, _a3, _a4) + + var r0 *exposer.ExposeResult + var r1 error + if rf, ok := ret.Get(0).(func(context.Context, v1.ObjectReference, client.Client, string, time.Duration) (*exposer.ExposeResult, error)); ok { + return rf(_a0, _a1, _a2, _a3, _a4) + } + if rf, ok := ret.Get(0).(func(context.Context, v1.ObjectReference, client.Client, string, time.Duration) *exposer.ExposeResult); ok { + r0 = rf(_a0, _a1, _a2, _a3, _a4) + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(*exposer.ExposeResult) + } + } + + if rf, ok := ret.Get(1).(func(context.Context, v1.ObjectReference, client.Client, string, time.Duration) error); ok { + r1 = rf(_a0, _a1, _a2, _a3, _a4) + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +// RebindVolume provides a mock function with given fields: _a0, _a1, _a2, _a3, _a4 +func (_m *GenericRestoreExposer) RebindVolume(_a0 context.Context, _a1 v1.ObjectReference, _a2 string, _a3 string, _a4 time.Duration) error { + ret := _m.Called(_a0, _a1, _a2, _a3, _a4) + + var r0 error + if rf, ok := ret.Get(0).(func(context.Context, v1.ObjectReference, string, string, time.Duration) error); ok { + r0 = rf(_a0, _a1, _a2, _a3, _a4) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type mockConstructorTestingTNewGenericRestoreExposer interface { + mock.TestingT + Cleanup(func()) +} + +// NewGenericRestoreExposer creates a new instance of GenericRestoreExposer. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations. +func NewGenericRestoreExposer(t mockConstructorTestingTNewGenericRestoreExposer) *GenericRestoreExposer { + mock := &GenericRestoreExposer{} + mock.Mock.Test(t) + + t.Cleanup(func() { mock.AssertExpectations(t) }) + + return mock +} From a97d01f7e6924fcd5ec74bf400f4276ac7750aac Mon Sep 17 00:00:00 2001 From: Peter Pan Date: Fri, 30 Jun 2023 08:14:10 +0800 Subject: [PATCH 14/28] Fix the snapshot log wording to be more accurate (#6395) Signed-off-by: Peter Pan --- pkg/backup/item_backupper.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pkg/backup/item_backupper.go b/pkg/backup/item_backupper.go index fbf43df42..a9a3ae440 100644 --- a/pkg/backup/item_backupper.go +++ b/pkg/backup/item_backupper.go @@ -545,7 +545,8 @@ func (ib *itemBackupper) takePVSnapshot(obj runtime.Unstructured, log logrus.Fie } if volumeSnapshotter == nil { - log.Info("Persistent volume is not a supported volume type for snapshots, skipping.") + // the PV may still has change to be snapshotted by CSI plugin's `PVCBackupItemAction` in PVC backup logic + log.Info("Persistent volume is not a supported volume type for Velero-native volumeSnapshotter snapshot, skipping.") return nil } From f2f479fe3ab17486eee3d2f91bf430f5a3634e88 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Wenkai=20Yin=28=E5=B0=B9=E6=96=87=E5=BC=80=29?= Date: Fri, 30 Jun 2023 09:48:21 +0800 Subject: [PATCH 15/28] Add more unit test cases for pkg/persistence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add more unit test cases for pkg/persistence Fixes #6340 Signed-off-by: Wenkai Yin(尹文开) --- pkg/persistence/object_store_test.go | 204 +++++++++++++++++++++++++++ 1 file changed, 204 insertions(+) diff --git a/pkg/persistence/object_store_test.go b/pkg/persistence/object_store_test.go index 896e5d435..de632d924 100644 --- a/pkg/persistence/object_store_test.go +++ b/pkg/persistence/object_store_test.go @@ -27,6 +27,7 @@ import ( "strings" "testing" + snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v4/apis/volumesnapshot/v1" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -493,6 +494,55 @@ func TestGetBackupItemOperations(t *testing.T) { assert.EqualValues(t, operations, res) } +func TestGetRestoreItemOperations(t *testing.T) { + harness := newObjectBackupStoreTestHarness("test-bucket", "") + + // itemoperations file not found should not error + res, err := harness.GetRestoreItemOperations("test-restore") + assert.NoError(t, err) + assert.Nil(t, res) + + // itemoperations file containing invalid data should error + harness.objectStore.PutObject(harness.bucket, "restores/test-restore/restore-test-restore-itemoperations.json.gz", newStringReadSeeker("foo")) + _, err = harness.GetRestoreItemOperations("test-restore") + assert.NotNil(t, err) + + // itemoperations file containing gzipped json data should return correctly + operations := []*itemoperation.RestoreOperation{ + { + Spec: itemoperation.RestoreOperationSpec{ + RestoreName: "test-restore", + ResourceIdentifier: velero.ResourceIdentifier{ + GroupResource: kuberesource.Pods, + Namespace: "ns", + Name: "item-1", + }, + }, + }, + { + Spec: itemoperation.RestoreOperationSpec{ + RestoreName: "test-restore", + ResourceIdentifier: velero.ResourceIdentifier{ + GroupResource: kuberesource.Pods, + Namespace: "ns", + Name: "item-2", + }, + }, + }, + } + + obj := new(bytes.Buffer) + gzw := gzip.NewWriter(obj) + + require.NoError(t, json.NewEncoder(gzw).Encode(operations)) + require.NoError(t, gzw.Close()) + require.NoError(t, harness.objectStore.PutObject(harness.bucket, "restores/test-restore/restore-test-restore-itemoperations.json.gz", obj)) + + res, err = harness.GetRestoreItemOperations("test-restore") + assert.NoError(t, err) + assert.EqualValues(t, operations, res) +} + func TestGetBackupContents(t *testing.T) { harness := newObjectBackupStoreTestHarness("test-bucket", "") @@ -559,6 +609,58 @@ func TestDeleteBackup(t *testing.T) { } } +func TestDeleteRestore(t *testing.T) { + tests := []struct { + name string + prefix string + listObjectsError error + deleteErrors []error + expectedErr string + }{ + { + name: "normal case", + }, + { + name: "normal case with backup store prefix", + prefix: "velero-backups/", + }, + { + name: "some delete errors, do as much as we can", + deleteErrors: []error{errors.New("a"), nil, errors.New("c")}, + expectedErr: "[a, c]", + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + objectStore := new(providermocks.ObjectStore) + backupStore := &objectBackupStore{ + objectStore: objectStore, + bucket: "test-bucket", + layout: NewObjectStoreLayout(test.prefix), + logger: velerotest.NewLogger(), + } + defer objectStore.AssertExpectations(t) + + objects := []string{test.prefix + "restores/bak/velero-restore.json", test.prefix + "restores/bak/bak.tar.gz", test.prefix + "restores/bak/bak.log.gz"} + + objectStore.On("ListObjects", backupStore.bucket, test.prefix+"restores/bak/").Return(objects, test.listObjectsError) + for i, obj := range objects { + var err error + if i < len(test.deleteErrors) { + err = test.deleteErrors[i] + } + + objectStore.On("DeleteObject", backupStore.bucket, obj).Return(err) + } + + err := backupStore.DeleteRestore("bak") + + velerotest.AssertErrorMatches(t, test.expectedErr, err) + }) + } +} + func TestGetDownloadURL(t *testing.T) { tests := []struct { name string @@ -673,6 +775,108 @@ func TestGetDownloadURL(t *testing.T) { } } +func TestGetCSIVolumeSnapshotClasses(t *testing.T) { + harness := newObjectBackupStoreTestHarness("test-bucket", "") + + // file not found should not error + res, err := harness.GetCSIVolumeSnapshotClasses("test-backup") + assert.NoError(t, err) + assert.Nil(t, res) + + // file containing invalid data should error + harness.objectStore.PutObject(harness.bucket, "backups/test-backup/test-backup-csi-volumesnapshotclasses.json.gz", newStringReadSeeker("foo")) + _, err = harness.GetCSIVolumeSnapshotClasses("test-backup") + assert.NotNil(t, err) + + // file containing gzipped json data should return correctly + classes := []*snapshotv1api.VolumeSnapshotClass{ + { + Driver: "driver", + }, + } + + obj := new(bytes.Buffer) + gzw := gzip.NewWriter(obj) + + require.NoError(t, json.NewEncoder(gzw).Encode(classes)) + require.NoError(t, gzw.Close()) + require.NoError(t, harness.objectStore.PutObject(harness.bucket, "backups/test-backup/test-backup-csi-volumesnapshotclasses.json.gz", obj)) + + res, err = harness.GetCSIVolumeSnapshotClasses("test-backup") + assert.NoError(t, err) + assert.EqualValues(t, classes, res) +} + +func TestGetCSIVolumeSnapshots(t *testing.T) { + harness := newObjectBackupStoreTestHarness("test-bucket", "") + + // file not found should not error + res, err := harness.GetCSIVolumeSnapshots("test-backup") + assert.NoError(t, err) + assert.Nil(t, res) + + // file containing invalid data should error + harness.objectStore.PutObject(harness.bucket, "backups/test-backup/test-backup-csi-volumesnapshots.json.gz", newStringReadSeeker("foo")) + _, err = harness.GetCSIVolumeSnapshots("test-backup") + assert.NotNil(t, err) + + // file containing gzipped json data should return correctly + snapshots := []*snapshotv1api.VolumeSnapshot{ + { + Spec: snapshotv1api.VolumeSnapshotSpec{ + Source: snapshotv1api.VolumeSnapshotSource{ + VolumeSnapshotContentName: nil, + }, + }, + }, + } + + obj := new(bytes.Buffer) + gzw := gzip.NewWriter(obj) + + require.NoError(t, json.NewEncoder(gzw).Encode(snapshots)) + require.NoError(t, gzw.Close()) + require.NoError(t, harness.objectStore.PutObject(harness.bucket, "backups/test-backup/test-backup-csi-volumesnapshots.json.gz", obj)) + + res, err = harness.GetCSIVolumeSnapshots("test-backup") + assert.NoError(t, err) + assert.EqualValues(t, snapshots, res) +} + +func TestGetCSIVolumeSnapshotContents(t *testing.T) { + harness := newObjectBackupStoreTestHarness("test-bucket", "") + + // file not found should not error + res, err := harness.GetCSIVolumeSnapshotContents("test-backup") + assert.NoError(t, err) + assert.Nil(t, res) + + // file containing invalid data should error + harness.objectStore.PutObject(harness.bucket, "backups/test-backup/test-backup-csi-volumesnapshotcontents.json.gz", newStringReadSeeker("foo")) + _, err = harness.GetCSIVolumeSnapshotContents("test-backup") + assert.NotNil(t, err) + + // file containing gzipped json data should return correctly + contents := []*snapshotv1api.VolumeSnapshotContent{ + { + Spec: snapshotv1api.VolumeSnapshotContentSpec{ + Driver: "driver", + }, + }, + } + + obj := new(bytes.Buffer) + gzw := gzip.NewWriter(obj) + + require.NoError(t, json.NewEncoder(gzw).Encode(contents)) + require.NoError(t, gzw.Close()) + require.NoError(t, harness.objectStore.PutObject(harness.bucket, "backups/test-backup/test-backup-csi-volumesnapshotcontents.json.gz", obj)) + + res, err = harness.GetCSIVolumeSnapshotContents("test-backup") + assert.NoError(t, err) + assert.EqualValues(t, contents, res) +} + type objectStoreGetter map[string]velero.ObjectStore func (osg objectStoreGetter) GetObjectStore(provider string) (velero.ObjectStore, error) { From ec4bb421173a47315a0c3ec8e9f50e84c498b9e6 Mon Sep 17 00:00:00 2001 From: Xun Jiang Date: Fri, 16 Jun 2023 17:37:10 +0800 Subject: [PATCH 16/28] Retrieve DataUpload into backup result ConfigMap during volume snapshot restore. Fix issue #6117. Add CSI plugin needs builder functions. Signed-off-by: Xun Jiang --- changelogs/unreleased/6410-blackpiglet | 1 + changelogs/unreleased/6436-qiuming-best | 1 + pkg/apis/velero/v1/labels_annotations.go | 23 + pkg/builder/data_download_builder.go | 112 ++++ pkg/builder/object_meta.go | 7 + .../persistent_volume_claim_builder.go | 37 ++ pkg/cmd/cli/nodeagent/server.go | 4 + pkg/cmd/server/plugin/plugin.go | 16 +- pkg/cmd/server/server.go | 2 + pkg/controller/data_download_controller.go | 487 ++++++++++++++++++ .../data_download_controller_test.go | 211 ++++++++ pkg/datapath/mocks/types.go | 86 ++++ pkg/exposer/mocks/generic_restore.go | 96 ++++ pkg/restore/dataupload_retrieve_action.go | 105 ++++ .../dataupload_retrieve_action_test.go | 88 ++++ pkg/restore/restore.go | 15 + pkg/restore/restore_test.go | 12 + pkg/test/resources.go | 10 + 18 files changed, 1310 insertions(+), 3 deletions(-) create mode 100644 changelogs/unreleased/6410-blackpiglet create mode 100644 changelogs/unreleased/6436-qiuming-best create mode 100644 pkg/builder/data_download_builder.go create mode 100644 pkg/controller/data_download_controller.go create mode 100644 pkg/controller/data_download_controller_test.go create mode 100644 pkg/datapath/mocks/types.go create mode 100644 pkg/exposer/mocks/generic_restore.go create mode 100644 pkg/restore/dataupload_retrieve_action.go create mode 100644 pkg/restore/dataupload_retrieve_action_test.go diff --git a/changelogs/unreleased/6410-blackpiglet b/changelogs/unreleased/6410-blackpiglet new file mode 100644 index 000000000..a72bab343 --- /dev/null +++ b/changelogs/unreleased/6410-blackpiglet @@ -0,0 +1 @@ +Retrieve DataUpload into backup result ConfigMap during volume snapshot restore. \ No newline at end of file diff --git a/changelogs/unreleased/6436-qiuming-best b/changelogs/unreleased/6436-qiuming-best new file mode 100644 index 000000000..adb564bb9 --- /dev/null +++ b/changelogs/unreleased/6436-qiuming-best @@ -0,0 +1 @@ +Add data download controller for data mover diff --git a/pkg/apis/velero/v1/labels_annotations.go b/pkg/apis/velero/v1/labels_annotations.go index 15bd57d78..defd56421 100644 --- a/pkg/apis/velero/v1/labels_annotations.go +++ b/pkg/apis/velero/v1/labels_annotations.go @@ -78,4 +78,27 @@ const ( // AsyncOperationIDLabel is the label key used to identify the async operation ID AsyncOperationIDLabel = "velero.io/async-operation-id" + + // PVCNameLabel is the label key used to identify the the PVC's namespace and name. + // The format is /. + PVCNamespaceNameLabel = "velero.io/pvc-namespace-name" + + // DynamicPVRestoreLabel is the label key for dynamic PV restore + DynamicPVRestoreLabel = "velero.io/dynamic-pv-restore" + + // ResourceUsageLabel is the label key to explain the Velero resource usage. + ResourceUsageLabel = "velero.io/resource-usage" +) + +type AsyncOperationIDPrefix string + +const ( + AsyncOperationIDPrefixDataDownload AsyncOperationIDPrefix = "dd-" + AsyncOperationIDPrefixDataUpload AsyncOperationIDPrefix = "du-" +) + +type VeleroResourceUsage string + +const ( + VeleroResourceUsageDataUploadResult VeleroResourceUsage = "DataUpload" ) diff --git a/pkg/builder/data_download_builder.go b/pkg/builder/data_download_builder.go new file mode 100644 index 000000000..f46e5b4e6 --- /dev/null +++ b/pkg/builder/data_download_builder.go @@ -0,0 +1,112 @@ +/* +Copyright The Velero Contributors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package builder + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + velerov2alpha1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1" +) + +// DataDownloadBuilder builds DataDownload objects. +type DataDownloadBuilder struct { + object *velerov2alpha1api.DataDownload +} + +// ForDataDownload is the constructor of DataDownloadBuilder +func ForDataDownload(namespace, name string) *DataDownloadBuilder { + return &DataDownloadBuilder{ + object: &velerov2alpha1api.DataDownload{ + TypeMeta: metav1.TypeMeta{ + Kind: "DataDownload", + APIVersion: velerov2alpha1api.SchemeGroupVersion.String(), + }, + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: namespace, + }, + }, + } +} + +// Result returns the built DataDownload. +func (d *DataDownloadBuilder) Result() *velerov2alpha1api.DataDownload { + return d.object +} + +// BackupStorageLocation sets the DataDownload's backup storage location. +func (d *DataDownloadBuilder) BackupStorageLocation(name string) *DataDownloadBuilder { + d.object.Spec.BackupStorageLocation = name + return d +} + +// Phase sets the DataDownload's phase. +func (d *DataDownloadBuilder) Phase(phase velerov2alpha1api.DataDownloadPhase) *DataDownloadBuilder { + d.object.Status.Phase = phase + return d +} + +// SnapshotID sets the DataDownload's SnapshotID. +func (d *DataDownloadBuilder) SnapshotID(id string) *DataDownloadBuilder { + d.object.Spec.SnapshotID = id + return d +} + +// DataMover sets the DataDownload's DataMover. +func (d *DataDownloadBuilder) DataMover(dataMover string) *DataDownloadBuilder { + d.object.Spec.DataMover = dataMover + return d +} + +// SourceNamespace sets the DataDownload's SourceNamespace. +func (d *DataDownloadBuilder) SourceNamespace(sourceNamespace string) *DataDownloadBuilder { + d.object.Spec.SourceNamespace = sourceNamespace + return d +} + +// TargetVolume sets the DataDownload's TargetVolume. +func (d *DataDownloadBuilder) TargetVolume(targetVolume velerov2alpha1api.TargetVolumeSpec) *DataDownloadBuilder { + d.object.Spec.TargetVolume = targetVolume + return d +} + +// Cancel sets the DataDownload's Cancel. +func (d *DataDownloadBuilder) Cancel(cancel bool) *DataDownloadBuilder { + d.object.Spec.Cancel = cancel + return d +} + +// OperationTimeout sets the DataDownload's OperationTimeout. +func (d *DataDownloadBuilder) OperationTimeout(timeout metav1.Duration) *DataDownloadBuilder { + d.object.Spec.OperationTimeout = timeout + return d +} + +// DataMoverConfig sets the DataDownload's DataMoverConfig. +func (d *DataDownloadBuilder) DataMoverConfig(config *map[string]string) *DataDownloadBuilder { + d.object.Spec.DataMoverConfig = *config + return d +} + +// ObjectMeta applies functional options to the DataDownload's ObjectMeta. +func (b *DataDownloadBuilder) ObjectMeta(opts ...ObjectMetaOpt) *DataDownloadBuilder { + for _, opt := range opts { + opt(b.object) + } + + return b +} diff --git a/pkg/builder/object_meta.go b/pkg/builder/object_meta.go index 90730e4be..561187f9e 100644 --- a/pkg/builder/object_meta.go +++ b/pkg/builder/object_meta.go @@ -160,3 +160,10 @@ func WithCreationTimestamp(t time.Time) func(obj metav1.Object) { obj.SetCreationTimestamp(metav1.Time{Time: t}) } } + +// WithOwnerReference is a functional option that applies the specified OwnerReference to an object. +func WithOwnerReference(val []metav1.OwnerReference) func(obj metav1.Object) { + return func(obj metav1.Object) { + obj.SetOwnerReferences(val) + } +} diff --git a/pkg/builder/persistent_volume_claim_builder.go b/pkg/builder/persistent_volume_claim_builder.go index 376d71444..569277dd3 100644 --- a/pkg/builder/persistent_volume_claim_builder.go +++ b/pkg/builder/persistent_volume_claim_builder.go @@ -18,6 +18,7 @@ package builder import ( corev1api "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) @@ -73,3 +74,39 @@ func (b *PersistentVolumeClaimBuilder) Phase(phase corev1api.PersistentVolumeCla b.object.Status.Phase = phase return b } + +// RequestResource sets the PersistentVolumeClaim's spec.Resources.Requests. +func (b *PersistentVolumeClaimBuilder) RequestResource(requests corev1api.ResourceList) *PersistentVolumeClaimBuilder { + if b.object.Spec.Resources.Requests == nil { + b.object.Spec.Resources.Requests = make(map[corev1api.ResourceName]resource.Quantity) + } + b.object.Spec.Resources.Requests = requests + return b +} + +// LimitResource sets the PersistentVolumeClaim's spec.Resources.Limits. +func (b *PersistentVolumeClaimBuilder) LimitResource(limits corev1api.ResourceList) *PersistentVolumeClaimBuilder { + if b.object.Spec.Resources.Limits == nil { + b.object.Spec.Resources.Limits = make(map[corev1api.ResourceName]resource.Quantity) + } + b.object.Spec.Resources.Limits = limits + return b +} + +// DataSource sets the PersistentVolumeClaim's spec.DataSource. +func (b *PersistentVolumeClaimBuilder) DataSource(dataSource *corev1api.TypedLocalObjectReference) *PersistentVolumeClaimBuilder { + b.object.Spec.DataSource = dataSource + return b +} + +// DataSourceRef sets the PersistentVolumeClaim's spec.DataSourceRef. +func (b *PersistentVolumeClaimBuilder) DataSourceRef(dataSourceRef *corev1api.TypedLocalObjectReference) *PersistentVolumeClaimBuilder { + b.object.Spec.DataSourceRef = dataSourceRef + return b +} + +// Selector sets the PersistentVolumeClaim's spec.Selector. +func (b *PersistentVolumeClaimBuilder) Selector(labelSelector *metav1.LabelSelector) *PersistentVolumeClaimBuilder { + b.object.Spec.Selector = labelSelector + return b +} diff --git a/pkg/cmd/cli/nodeagent/server.go b/pkg/cmd/cli/nodeagent/server.go index 5e138b250..3f635602e 100644 --- a/pkg/cmd/cli/nodeagent/server.go +++ b/pkg/cmd/cli/nodeagent/server.go @@ -260,6 +260,10 @@ func (s *nodeAgentServer) run() { s.logger.WithError(err).Fatal("Unable to create the data upload controller") } + if err = controller.NewDataDownloadReconciler(s.mgr.GetClient(), s.kubeClient, repoEnsurer, credentialGetter, s.nodeName, s.logger).SetupWithManager(s.mgr); err != nil { + s.logger.WithError(err).Fatal("Unable to create the data download controller") + } + s.logger.Info("Controllers starting...") if err := s.mgr.Start(ctrl.SetupSignalHandler()); err != nil { diff --git a/pkg/cmd/server/plugin/plugin.go b/pkg/cmd/server/plugin/plugin.go index a42f88730..45e45389a 100644 --- a/pkg/cmd/server/plugin/plugin.go +++ b/pkg/cmd/server/plugin/plugin.go @@ -22,11 +22,10 @@ import ( apiextensions "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" - "github.com/vmware-tanzu/velero/pkg/features" - "github.com/vmware-tanzu/velero/pkg/backup" "github.com/vmware-tanzu/velero/pkg/client" velerodiscovery "github.com/vmware-tanzu/velero/pkg/discovery" + "github.com/vmware-tanzu/velero/pkg/features" veleroplugin "github.com/vmware-tanzu/velero/pkg/plugin/framework" plugincommon "github.com/vmware-tanzu/velero/pkg/plugin/framework/common" "github.com/vmware-tanzu/velero/pkg/restore" @@ -59,7 +58,8 @@ func NewCommand(f client.Factory) *cobra.Command { RegisterRestoreItemAction("velero.io/change-pvc-node-selector", newChangePVCNodeSelectorItemAction(f)). RegisterRestoreItemAction("velero.io/apiservice", newAPIServiceRestoreItemAction). RegisterRestoreItemAction("velero.io/admission-webhook-configuration", newAdmissionWebhookConfigurationAction). - RegisterRestoreItemAction("velero.io/secret", newSecretRestoreItemAction(f)) + RegisterRestoreItemAction("velero.io/secret", newSecretRestoreItemAction(f)). + RegisterRestoreItemAction("velero.io/dataupload", newDataUploadRetrieveAction(f)) if !features.IsEnabled(velerov1api.APIGroupVersionsFeatureFlag) { // Do not register crd-remap-version BIA if the API Group feature flag is enabled, so that the v1 CRD can be backed up pluginServer = pluginServer.RegisterBackupItemAction("velero.io/crd-remap-version", newRemapCRDVersionAction(f)) @@ -245,3 +245,13 @@ func newSecretRestoreItemAction(f client.Factory) plugincommon.HandlerInitialize return restore.NewSecretAction(logger, client), nil } } + +func newDataUploadRetrieveAction(f client.Factory) plugincommon.HandlerInitializer { + return func(logger logrus.FieldLogger) (interface{}, error) { + client, err := f.KubeClient() + if err != nil { + return nil, err + } + return restore.NewDataUploadRetrieveAction(logger, client.CoreV1().ConfigMaps(f.Namespace())), nil + } +} diff --git a/pkg/cmd/server/server.go b/pkg/cmd/server/server.go index d2c596f29..00a4456f0 100644 --- a/pkg/cmd/server/server.go +++ b/pkg/cmd/server/server.go @@ -521,6 +521,7 @@ High priorities: - VolumeSnapshotContents are needed as they contain the handle to the volume snapshot in the storage provider - VolumeSnapshots are needed to create PVCs using the VolumeSnapshot as their data source. + - DataUploads need to restore before PVC for Snapshot DataMover to work, because PVC needs the DataUploadResults to create DataDownloads. - PVs go before PVCs because PVCs depend on them. - PVCs go before pods or controllers so they can be mounted as volumes. - Service accounts go before secrets so service account token secrets can be filled automatically. @@ -551,6 +552,7 @@ var defaultRestorePriorities = restore.Priorities{ "volumesnapshotclass.snapshot.storage.k8s.io", "volumesnapshotcontents.snapshot.storage.k8s.io", "volumesnapshots.snapshot.storage.k8s.io", + "datauploads.velero.io", "persistentvolumes", "persistentvolumeclaims", "serviceaccounts", diff --git a/pkg/controller/data_download_controller.go b/pkg/controller/data_download_controller.go new file mode 100644 index 000000000..1f0e78c69 --- /dev/null +++ b/pkg/controller/data_download_controller.go @@ -0,0 +1,487 @@ +/* +Copyright The Velero Contributors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package controller + +import ( + "context" + "fmt" + "time" + + "github.com/pkg/errors" + "github.com/sirupsen/logrus" + v1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/kubernetes" + "k8s.io/utils/clock" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/builder" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/predicate" + "sigs.k8s.io/controller-runtime/pkg/reconcile" + "sigs.k8s.io/controller-runtime/pkg/source" + + "github.com/vmware-tanzu/velero/internal/credentials" + "github.com/vmware-tanzu/velero/pkg/apis/velero/shared" + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + velerov2alpha1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1" + datamover "github.com/vmware-tanzu/velero/pkg/datamover" + "github.com/vmware-tanzu/velero/pkg/datapath" + "github.com/vmware-tanzu/velero/pkg/exposer" + repository "github.com/vmware-tanzu/velero/pkg/repository" + "github.com/vmware-tanzu/velero/pkg/uploader" + "github.com/vmware-tanzu/velero/pkg/util/filesystem" + "github.com/vmware-tanzu/velero/pkg/util/kube" +) + +// DataDownloadReconciler reconciles a DataDownload object +type DataDownloadReconciler struct { + client client.Client + kubeClient kubernetes.Interface + logger logrus.FieldLogger + credentialGetter *credentials.CredentialGetter + fileSystem filesystem.Interface + clock clock.WithTickerAndDelayedExecution + restoreExposer exposer.GenericRestoreExposer + nodeName string + repositoryEnsurer *repository.Ensurer + dataPathMgr *datapath.Manager +} + +func NewDataDownloadReconciler(client client.Client, kubeClient kubernetes.Interface, + repoEnsurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter, nodeName string, logger logrus.FieldLogger) *DataDownloadReconciler { + return &DataDownloadReconciler{ + client: client, + kubeClient: kubeClient, + logger: logger.WithField("controller", "DataDownload"), + credentialGetter: credentialGetter, + fileSystem: filesystem.NewFileSystem(), + clock: &clock.RealClock{}, + nodeName: nodeName, + repositoryEnsurer: repoEnsurer, + restoreExposer: exposer.NewGenericRestoreExposer(kubeClient, logger), + dataPathMgr: datapath.NewManager(1), + } +} + +// +kubebuilder:rbac:groups=velero.io,resources=datadownloads,verbs=get;list;watch;create;update;patch;delete +// +kubebuilder:rbac:groups=velero.io,resources=datadownloads/status,verbs=get;update;patch +// +kubebuilder:rbac:groups="",resources=pods,verbs=get +// +kubebuilder:rbac:groups="",resources=persistentvolumes,verbs=get +// +kubebuilder:rbac:groups="",resources=persistentvolumerclaims,verbs=get + +func (r *DataDownloadReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { + log := r.logger.WithFields(logrus.Fields{ + "controller": "datadownload", + "datadownload": req.NamespacedName, + }) + + log.Infof("Reconcile %s", req.Name) + + dd := &velerov2alpha1api.DataDownload{} + if err := r.client.Get(ctx, types.NamespacedName{Namespace: req.Namespace, Name: req.Name}, dd); err != nil { + if apierrors.IsNotFound(err) { + log.Warn("DataDownload not found, skip") + return ctrl.Result{}, nil + } + log.WithError(err).Error("Unable to get the DataDownload") + return ctrl.Result{}, err + } + + if dd.Spec.DataMover != "" && dd.Spec.DataMover != dataMoverType { + log.WithField("data mover", dd.Spec.DataMover).Info("it is not one built-in data mover which is not supported by Velero") + return ctrl.Result{}, nil + } + + if r.restoreExposer == nil { + return r.errorOut(ctx, dd, errors.New("uninitialized generic exposer"), "uninitialized exposer", log) + } + + if dd.Status.Phase == "" || dd.Status.Phase == velerov2alpha1api.DataDownloadPhaseNew { + log.Info("Data download starting") + + if _, err := r.getTargetPVC(ctx, dd); err != nil { + return ctrl.Result{Requeue: true}, nil + } + + accepted, err := r.acceptDataDownload(ctx, dd) + if err != nil { + return r.errorOut(ctx, dd, err, "error to accept the data download", log) + } + + if !accepted { + log.Debug("Data download is not accepted") + return ctrl.Result{}, nil + } + + log.Info("Data download is accepted") + + hostingPodLabels := map[string]string{velerov1api.DataDownloadLabel: dd.Name} + + // ep.Expose() will trigger to create one pod whose volume is restored by a given volume snapshot, + // but the pod maybe is not in the same node of the current controller, so we need to return it here. + // And then only the controller who is in the same node could do the rest work. + err = r.restoreExposer.Expose(ctx, getDataDownloadOwnerObject(dd), dd.Spec.TargetVolume.PVC, dd.Spec.TargetVolume.Namespace, hostingPodLabels, dd.Spec.OperationTimeout.Duration) + if err != nil { + return r.errorOut(ctx, dd, err, "error to start restore expose", log) + } + log.Info("Restore is exposed") + + return ctrl.Result{}, nil + } else if dd.Status.Phase == velerov2alpha1api.DataDownloadPhasePrepared { + log.Info("Data download is prepared") + fsRestore := r.dataPathMgr.GetAsyncBR(dd.Name) + + if fsRestore != nil { + log.Info("Cancellable data path is already started") + return ctrl.Result{}, nil + } + + result, err := r.restoreExposer.GetExposed(ctx, getDataDownloadOwnerObject(dd), r.client, r.nodeName, dd.Spec.OperationTimeout.Duration) + if err != nil { + return r.errorOut(ctx, dd, err, "restore exposer is not ready", log) + } else if result == nil { + log.Debug("Get empty restore exposer") + return ctrl.Result{}, nil + } + + log.Info("Restore PVC is ready") + + // Update status to InProgress + original := dd.DeepCopy() + dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseInProgress + dd.Status.StartTimestamp = &metav1.Time{Time: r.clock.Now()} + if err := r.client.Patch(ctx, dd, client.MergeFrom(original)); err != nil { + log.WithError(err).Error("Unable to update status to in progress") + return ctrl.Result{}, err + } + + log.Info("Data download is marked as in progress") + + return r.runCancelableDataPath(ctx, dd, result, log) + } else if dd.Status.Phase == velerov2alpha1api.DataDownloadPhaseInProgress { + log.Info("Data download is in progress") + if dd.Spec.Cancel { + fsRestore := r.dataPathMgr.GetAsyncBR(dd.Name) + if fsRestore == nil { + return ctrl.Result{}, nil + } + + log.Info("Data download is being canceled") + // Update status to Canceling. + original := dd.DeepCopy() + dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseCanceling + if err := r.client.Patch(ctx, dd, client.MergeFrom(original)); err != nil { + log.WithError(err).Error("error updating data download status") + return ctrl.Result{}, err + } + + fsRestore.Cancel() + return ctrl.Result{}, nil + } + + return ctrl.Result{}, nil + } else { + log.Debugf("Data download now is in %s phase and do nothing by current %s controller", dd.Status.Phase, r.nodeName) + return ctrl.Result{}, nil + } +} + +func (r *DataDownloadReconciler) runCancelableDataPath(ctx context.Context, dd *velerov2alpha1api.DataDownload, res *exposer.ExposeResult, log logrus.FieldLogger) (reconcile.Result, error) { + log.Info("Creating data path routine") + callbacks := datapath.Callbacks{ + OnCompleted: r.OnDataDownloadCompleted, + OnFailed: r.OnDataDownloadFailed, + OnCancelled: r.OnDataDownloadCancelled, + OnProgress: r.OnDataDownloadProgress, + } + + fsRestore, err := r.dataPathMgr.CreateFileSystemBR(dd.Name, dataUploadDownloadRequestor, ctx, r.client, dd.Namespace, callbacks, log) + if err != nil { + if err == datapath.ConcurrentLimitExceed { + log.Info("runCancelableDataDownload is concurrent limited") + return ctrl.Result{Requeue: true, RequeueAfter: time.Minute}, nil + } else { + return r.errorOut(ctx, dd, err, "error to create data path", log) + } + } + + path, err := exposer.GetPodVolumeHostPath(ctx, res.ByPod.HostingPod, res.ByPod.PVC, r.client, r.fileSystem, log) + if err != nil { + return r.errorOut(ctx, dd, err, "error exposing host path for pod volume", log) + } + + log.WithField("path", path.ByPath).Debug("Found host path") + if err := fsRestore.Init(ctx, dd.Spec.BackupStorageLocation, dd.Spec.SourceNamespace, datamover.GetUploaderType(dd.Spec.DataMover), + velerov1api.BackupRepositoryTypeKopia, "", r.repositoryEnsurer, r.credentialGetter); err != nil { + return r.errorOut(ctx, dd, err, "error to initialize data path", log) + } + log.WithField("path", path.ByPath).Info("fs init") + + if err := fsRestore.StartRestore(dd.Spec.SnapshotID, path); err != nil { + return r.errorOut(ctx, dd, err, fmt.Sprintf("error starting data path %s restore", path.ByPath), log) + } + + log.WithField("path", path.ByPath).Info("Async fs restore data path started") + return ctrl.Result{}, nil +} + +func (r *DataDownloadReconciler) OnDataDownloadCompleted(ctx context.Context, namespace string, ddName string, result datapath.Result) { + defer r.closeDataPath(ctx, ddName) + + log := r.logger.WithField("datadownload", ddName) + log.Info("Async fs restore data path completed") + + var dd velerov2alpha1api.DataDownload + if err := r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, &dd); err != nil { + log.WithError(err).Warn("Failed to get datadownload on completion") + return + } + + objRef := getDataDownloadOwnerObject(&dd) + err := r.restoreExposer.RebindVolume(ctx, objRef, dd.Spec.TargetVolume.PVC, dd.Spec.TargetVolume.Namespace, dd.Spec.OperationTimeout.Duration) + if err != nil { + log.WithError(err).Error("Failed to rebind PV to target PVC on completion") + return + } + + log.Info("Cleaning up exposed environment") + r.restoreExposer.CleanUp(ctx, objRef) + + original := dd.DeepCopy() + dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseCompleted + dd.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()} + if err := r.client.Patch(ctx, &dd, client.MergeFrom(original)); err != nil { + log.WithError(err).Error("error updating data download status") + } else { + log.Infof("Data download is marked as %s", dd.Status.Phase) + } +} + +func (r *DataDownloadReconciler) OnDataDownloadFailed(ctx context.Context, namespace string, ddName string, err error) { + defer r.closeDataPath(ctx, ddName) + + log := r.logger.WithField("datadownload", ddName) + + log.WithError(err).Error("Async fs restore data path failed") + + var dd velerov2alpha1api.DataDownload + if getErr := r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, &dd); getErr != nil { + log.WithError(getErr).Warn("Failed to get data download on failure") + } else { + if _, errOut := r.errorOut(ctx, &dd, err, "data path restore failed", log); err != nil { + log.WithError(err).Warnf("Failed to patch data download with err %v", errOut) + } + } +} + +func (r *DataDownloadReconciler) OnDataDownloadCancelled(ctx context.Context, namespace string, ddName string) { + defer r.closeDataPath(ctx, ddName) + + log := r.logger.WithField("datadownload", ddName) + + log.Warn("Async fs backup data path canceled") + + var dd velerov2alpha1api.DataDownload + if getErr := r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, &dd); getErr != nil { + log.WithError(getErr).Warn("Failed to get datadownload on cancel") + } else { + // cleans up any objects generated during the snapshot expose + r.restoreExposer.CleanUp(ctx, getDataDownloadOwnerObject(&dd)) + + original := dd.DeepCopy() + dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseCanceled + if dd.Status.StartTimestamp.IsZero() { + dd.Status.StartTimestamp = &metav1.Time{Time: r.clock.Now()} + } + dd.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()} + if err := r.client.Patch(ctx, &dd, client.MergeFrom(original)); err != nil { + log.WithError(err).Error("error updating data download status") + } + } +} + +func (r *DataDownloadReconciler) OnDataDownloadProgress(ctx context.Context, namespace string, ddName string, progress *uploader.Progress) { + log := r.logger.WithField("datadownload", ddName) + + var dd velerov2alpha1api.DataDownload + if err := r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, &dd); err != nil { + log.WithError(err).Warn("Failed to get data download on progress") + return + } + + original := dd.DeepCopy() + dd.Status.Progress = shared.DataMoveOperationProgress{TotalBytes: progress.TotalBytes, BytesDone: progress.BytesDone} + + if err := r.client.Patch(ctx, &dd, client.MergeFrom(original)); err != nil { + log.WithError(err).Error("Failed to update restore snapshot progress") + } +} + +// SetupWithManager registers the DataDownload controller. +// The fresh new DataDownload CR first created will trigger to create one pod (long time, maybe failure or unknown status) by one of the datadownload controllers +// then the request will get out of the Reconcile queue immediately by not blocking others' CR handling, in order to finish the rest data download process we need to +// re-enqueue the previous related request once the related pod is in running status to keep going on the rest logic. and below logic will avoid handling the unwanted +// pod status and also avoid block others CR handling +func (r *DataDownloadReconciler) SetupWithManager(mgr ctrl.Manager) error { + return ctrl.NewControllerManagedBy(mgr). + For(&velerov2alpha1api.DataDownload{}). + Watches(&source.Kind{Type: &v1.Pod{}}, kube.EnqueueRequestsFromMapUpdateFunc(r.findSnapshotRestoreForPod), + builder.WithPredicates(predicate.Funcs{ + UpdateFunc: func(ue event.UpdateEvent) bool { + newObj := ue.ObjectNew.(*v1.Pod) + + if _, ok := newObj.Labels[velerov1api.DataDownloadLabel]; !ok { + return false + } + + if newObj.Status.Phase != v1.PodRunning { + return false + } + + if newObj.Spec.NodeName == "" { + return false + } + + return true + }, + CreateFunc: func(event.CreateEvent) bool { + return false + }, + DeleteFunc: func(de event.DeleteEvent) bool { + return false + }, + GenericFunc: func(ge event.GenericEvent) bool { + return false + }, + })). + Complete(r) +} + +func (r *DataDownloadReconciler) findSnapshotRestoreForPod(podObj client.Object) []reconcile.Request { + pod := podObj.(*v1.Pod) + + dd := &velerov2alpha1api.DataDownload{} + err := r.client.Get(context.Background(), types.NamespacedName{ + Namespace: pod.Namespace, + Name: pod.Labels[velerov1api.DataDownloadLabel], + }, dd) + + if err != nil { + r.logger.WithField("Restore pod", pod.Name).WithError(err).Error("unable to get DataDownload") + return []reconcile.Request{} + } + + if dd.Status.Phase != velerov2alpha1api.DataDownloadPhaseAccepted { + return []reconcile.Request{} + } + + requests := make([]reconcile.Request, 1) + + r.logger.WithField("Restore pod", pod.Name).Infof("Preparing data download %s", dd.Name) + err = r.patchDataDownload(context.Background(), dd, prepareDataDownload) + if err != nil { + r.logger.WithField("Restore pod", pod.Name).WithError(err).Error("unable to patch data download") + return []reconcile.Request{} + } + + requests[0] = reconcile.Request{ + NamespacedName: types.NamespacedName{ + Namespace: dd.Namespace, + Name: dd.Name, + }, + } + + return requests +} + +func (r *DataDownloadReconciler) patchDataDownload(ctx context.Context, req *velerov2alpha1api.DataDownload, mutate func(*velerov2alpha1api.DataDownload)) error { + original := req.DeepCopy() + mutate(req) + if err := r.client.Patch(ctx, req, client.MergeFrom(original)); err != nil { + return errors.Wrap(err, "error patching data download") + } + + return nil +} + +func prepareDataDownload(ssb *velerov2alpha1api.DataDownload) { + ssb.Status.Phase = velerov2alpha1api.DataDownloadPhasePrepared +} + +func (r *DataDownloadReconciler) errorOut(ctx context.Context, dd *velerov2alpha1api.DataDownload, err error, msg string, log logrus.FieldLogger) (ctrl.Result, error) { + r.restoreExposer.CleanUp(ctx, getDataDownloadOwnerObject(dd)) + return ctrl.Result{}, r.updateStatusToFailed(ctx, dd, err, msg, log) +} + +func (r *DataDownloadReconciler) updateStatusToFailed(ctx context.Context, dd *velerov2alpha1api.DataDownload, err error, msg string, log logrus.FieldLogger) error { + log.Infof("update data download status to %v", dd.Status.Phase) + original := dd.DeepCopy() + dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseFailed + dd.Status.Message = errors.WithMessage(err, msg).Error() + dd.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()} + + if err = r.client.Patch(ctx, dd, client.MergeFrom(original)); err != nil { + log.WithError(err).Error("error updating DataDownload status") + return err + } + + return nil +} + +func (r *DataDownloadReconciler) acceptDataDownload(ctx context.Context, dd *velerov2alpha1api.DataDownload) (bool, error) { + updated := dd.DeepCopy() + updated.Status.Phase = velerov2alpha1api.DataDownloadPhaseAccepted + + r.logger.Infof("Accepting snapshot restore %s", dd.Name) + // For all data download controller in each node-agent will try to update download CR, and only one controller will success, + // and the success one could handle later logic + err := r.client.Update(ctx, updated) + if err == nil { + return true, nil + } else if apierrors.IsConflict(err) { + r.logger.WithField("DataDownload", dd.Name).Error("This data download restore has been accepted by others") + return false, nil + } else { + return false, err + } +} + +func (r *DataDownloadReconciler) getTargetPVC(ctx context.Context, dd *velerov2alpha1api.DataDownload) (*v1.PersistentVolumeClaim, error) { + return r.kubeClient.CoreV1().PersistentVolumeClaims(dd.Spec.TargetVolume.Namespace).Get(ctx, dd.Spec.TargetVolume.PVC, metav1.GetOptions{}) +} + +func (r *DataDownloadReconciler) closeDataPath(ctx context.Context, ddName string) { + fsBackup := r.dataPathMgr.GetAsyncBR(ddName) + if fsBackup != nil { + fsBackup.Close(ctx) + } + + r.dataPathMgr.RemoveAsyncBR(ddName) +} + +func getDataDownloadOwnerObject(dd *velerov2alpha1api.DataDownload) v1.ObjectReference { + return v1.ObjectReference{ + Kind: dd.Kind, + Namespace: dd.Namespace, + Name: dd.Name, + UID: dd.UID, + APIVersion: dd.APIVersion, + } +} diff --git a/pkg/controller/data_download_controller_test.go b/pkg/controller/data_download_controller_test.go new file mode 100644 index 000000000..5db1170c7 --- /dev/null +++ b/pkg/controller/data_download_controller_test.go @@ -0,0 +1,211 @@ +/* +Copyright The Velero Contributors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package controller + +import ( + "context" + "fmt" + "testing" + + "github.com/pkg/errors" + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/mock" + "github.com/stretchr/testify/require" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + clientgofake "k8s.io/client-go/kubernetes/fake" + ctrl "sigs.k8s.io/controller-runtime" + kbclient "sigs.k8s.io/controller-runtime/pkg/client" + + "github.com/vmware-tanzu/velero/internal/credentials" + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + velerov2alpha1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1" + "github.com/vmware-tanzu/velero/pkg/builder" + "github.com/vmware-tanzu/velero/pkg/datapath" + "github.com/vmware-tanzu/velero/pkg/exposer" + velerotest "github.com/vmware-tanzu/velero/pkg/test" + + "sigs.k8s.io/controller-runtime/pkg/client/fake" + + datapathmockes "github.com/vmware-tanzu/velero/pkg/datapath/mocks" + exposermockes "github.com/vmware-tanzu/velero/pkg/exposer/mocks" +) + +const dataDownloadName string = "datadownload-1" + +func dataDownloadBuilder() *builder.DataDownloadBuilder { + return builder.ForDataDownload(velerov1api.DefaultNamespace, dataDownloadName). + BackupStorageLocation("bsl-loc"). + DataMover("velero"). + SnapshotID("test-snapshot-id").TargetVolume(velerov2alpha1api.TargetVolumeSpec{ + PV: "test-pv", + PVC: "test-pvc", + Namespace: "test-ns", + }) +} + +func initDataDownloadReconciler(objects []runtime.Object, needError ...bool) (*DataDownloadReconciler, error) { + scheme := runtime.NewScheme() + err := velerov1api.AddToScheme(scheme) + if err != nil { + return nil, err + } + err = velerov2alpha1api.AddToScheme(scheme) + if err != nil { + return nil, err + } + err = corev1.AddToScheme(scheme) + if err != nil { + return nil, err + } + + fakeClient := &FakeClient{ + Client: fake.NewClientBuilder().WithScheme(scheme).Build(), + } + + if len(needError) == 4 { + fakeClient.getError = needError[0] + fakeClient.createError = needError[1] + fakeClient.updateError = needError[2] + fakeClient.patchError = needError[3] + } + + fakeKubeClient := clientgofake.NewSimpleClientset(objects...) + fakeFS := velerotest.NewFakeFileSystem() + pathGlob := fmt.Sprintf("/host_pods/%s/volumes/*/%s", "", dataDownloadName) + _, err = fakeFS.Create(pathGlob) + if err != nil { + return nil, err + } + + credentialFileStore, err := credentials.NewNamespacedFileStore( + fakeClient, + velerov1api.DefaultNamespace, + "/tmp/credentials", + fakeFS, + ) + if err != nil { + return nil, err + } + return NewDataDownloadReconciler(fakeClient, fakeKubeClient, nil, &credentials.CredentialGetter{FromFile: credentialFileStore}, "test_node", velerotest.NewLogger()), nil +} + +func TestDataDownloadReconcile(t *testing.T) { + tests := []struct { + name string + dd *velerov2alpha1api.DataDownload + targetPVC *corev1.PersistentVolumeClaim + dataMgr *datapath.Manager + needErrs []bool + isExposeErr bool + isGetExposeErr bool + expectedStatusMsg string + }{ + { + name: "Restore is exposed", + dd: dataDownloadBuilder().Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + }, + { + name: "Get empty restore exposer", + dd: dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhasePrepared).Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + }, + { + name: "Failed to get restore exposer", + dd: dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhasePrepared).Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + expectedStatusMsg: "Error to get restore exposer", + isGetExposeErr: true, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + r, err := initDataDownloadReconciler([]runtime.Object{test.targetPVC}, test.needErrs...) + require.NoError(t, err) + defer func() { + r.client.Delete(ctx, test.dd, &kbclient.DeleteOptions{}) + if test.targetPVC != nil { + r.client.Delete(ctx, test.targetPVC, &kbclient.DeleteOptions{}) + } + }() + + ctx := context.Background() + if test.dd.Namespace == velerov1api.DefaultNamespace { + err = r.client.Create(ctx, test.dd) + require.NoError(t, err) + } + + if test.dataMgr != nil { + r.dataPathMgr = test.dataMgr + } else { + r.dataPathMgr = datapath.NewManager(1) + } + + datapath.FSBRCreator = func(string, string, kbclient.Client, string, datapath.Callbacks, logrus.FieldLogger) datapath.AsyncBR { + return datapathmockes.NewAsyncBR(t) + } + + if test.isExposeErr || test.isGetExposeErr { + r.restoreExposer = func() exposer.GenericRestoreExposer { + ep := exposermockes.NewGenericRestoreExposer(t) + if test.isExposeErr { + ep.On("Expose", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(errors.New("Error to expose restore exposer")) + } + + if test.isGetExposeErr { + ep.On("GetExposed", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(nil, errors.New("Error to get restore exposer")) + } + + ep.On("CleanUp", mock.Anything, mock.Anything).Return() + return ep + }() + } + + if test.dd.Status.Phase == velerov2alpha1api.DataDownloadPhaseInProgress { + if fsBR := r.dataPathMgr.GetAsyncBR(test.dd.Name); fsBR == nil { + _, err := r.dataPathMgr.CreateFileSystemBR(test.dd.Name, pVBRRequestor, ctx, r.client, velerov1api.DefaultNamespace, datapath.Callbacks{OnCancelled: r.OnDataDownloadCancelled}, velerotest.NewLogger()) + require.NoError(t, err) + } + } + actualResult, err := r.Reconcile(ctx, ctrl.Request{ + NamespacedName: types.NamespacedName{ + Namespace: velerov1api.DefaultNamespace, + Name: test.dd.Name, + }, + }) + + require.Nil(t, err) + require.NotNil(t, actualResult) + + dd := velerov2alpha1api.DataDownload{} + err = r.client.Get(ctx, kbclient.ObjectKey{ + Name: test.dd.Name, + Namespace: test.dd.Namespace, + }, &dd) + + if test.isGetExposeErr { + assert.Contains(t, dd.Status.Message, test.expectedStatusMsg) + } + require.Nil(t, err) + t.Logf("%s: \n %v \n", test.name, dd) + }) + } +} diff --git a/pkg/datapath/mocks/types.go b/pkg/datapath/mocks/types.go new file mode 100644 index 000000000..ecf655df0 --- /dev/null +++ b/pkg/datapath/mocks/types.go @@ -0,0 +1,86 @@ +// Code generated by mockery v2.20.0. DO NOT EDIT. + +package mocks + +import ( + context "context" + + credentials "github.com/vmware-tanzu/velero/internal/credentials" + datapath "github.com/vmware-tanzu/velero/pkg/datapath" + + mock "github.com/stretchr/testify/mock" + + repository "github.com/vmware-tanzu/velero/pkg/repository" +) + +// AsyncBR is an autogenerated mock type for the AsyncBR type +type AsyncBR struct { + mock.Mock +} + +// Cancel provides a mock function with given fields: +func (_m *AsyncBR) Cancel() { + _m.Called() +} + +// Close provides a mock function with given fields: ctx +func (_m *AsyncBR) Close(ctx context.Context) { + _m.Called(ctx) +} + +// Init provides a mock function with given fields: ctx, bslName, sourceNamespace, uploaderType, repositoryType, repoIdentifier, repositoryEnsurer, credentialGetter +func (_m *AsyncBR) Init(ctx context.Context, bslName string, sourceNamespace string, uploaderType string, repositoryType string, repoIdentifier string, repositoryEnsurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter) error { + ret := _m.Called(ctx, bslName, sourceNamespace, uploaderType, repositoryType, repoIdentifier, repositoryEnsurer, credentialGetter) + + var r0 error + if rf, ok := ret.Get(0).(func(context.Context, string, string, string, string, string, *repository.Ensurer, *credentials.CredentialGetter) error); ok { + r0 = rf(ctx, bslName, sourceNamespace, uploaderType, repositoryType, repoIdentifier, repositoryEnsurer, credentialGetter) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// StartBackup provides a mock function with given fields: source, realSource, parentSnapshot, forceFull, tags +func (_m *AsyncBR) StartBackup(source datapath.AccessPoint, realSource string, parentSnapshot string, forceFull bool, tags map[string]string) error { + ret := _m.Called(source, realSource, parentSnapshot, forceFull, tags) + + var r0 error + if rf, ok := ret.Get(0).(func(datapath.AccessPoint, string, string, bool, map[string]string) error); ok { + r0 = rf(source, realSource, parentSnapshot, forceFull, tags) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// StartRestore provides a mock function with given fields: snapshotID, target +func (_m *AsyncBR) StartRestore(snapshotID string, target datapath.AccessPoint) error { + ret := _m.Called(snapshotID, target) + + var r0 error + if rf, ok := ret.Get(0).(func(string, datapath.AccessPoint) error); ok { + r0 = rf(snapshotID, target) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type mockConstructorTestingTNewAsyncBR interface { + mock.TestingT + Cleanup(func()) +} + +// NewAsyncBR creates a new instance of AsyncBR. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations. +func NewAsyncBR(t mockConstructorTestingTNewAsyncBR) *AsyncBR { + mock := &AsyncBR{} + mock.Mock.Test(t) + + t.Cleanup(func() { mock.AssertExpectations(t) }) + + return mock +} diff --git a/pkg/exposer/mocks/generic_restore.go b/pkg/exposer/mocks/generic_restore.go new file mode 100644 index 000000000..a7d20f87c --- /dev/null +++ b/pkg/exposer/mocks/generic_restore.go @@ -0,0 +1,96 @@ +// Code generated by mockery v2.20.0. DO NOT EDIT. + +package mocks + +import ( + context "context" + + client "sigs.k8s.io/controller-runtime/pkg/client" + + exposer "github.com/vmware-tanzu/velero/pkg/exposer" + + mock "github.com/stretchr/testify/mock" + + time "time" + + v1 "k8s.io/api/core/v1" +) + +// GenericRestoreExposer is an autogenerated mock type for the GenericRestoreExposer type +type GenericRestoreExposer struct { + mock.Mock +} + +// CleanUp provides a mock function with given fields: _a0, _a1 +func (_m *GenericRestoreExposer) CleanUp(_a0 context.Context, _a1 v1.ObjectReference) { + _m.Called(_a0, _a1) +} + +// Expose provides a mock function with given fields: _a0, _a1, _a2, _a3, _a4, _a5 +func (_m *GenericRestoreExposer) Expose(_a0 context.Context, _a1 v1.ObjectReference, _a2 string, _a3 string, _a4 map[string]string, _a5 time.Duration) error { + ret := _m.Called(_a0, _a1, _a2, _a3, _a4, _a5) + + var r0 error + if rf, ok := ret.Get(0).(func(context.Context, v1.ObjectReference, string, string, map[string]string, time.Duration) error); ok { + r0 = rf(_a0, _a1, _a2, _a3, _a4, _a5) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// GetExposed provides a mock function with given fields: _a0, _a1, _a2, _a3, _a4 +func (_m *GenericRestoreExposer) GetExposed(_a0 context.Context, _a1 v1.ObjectReference, _a2 client.Client, _a3 string, _a4 time.Duration) (*exposer.ExposeResult, error) { + ret := _m.Called(_a0, _a1, _a2, _a3, _a4) + + var r0 *exposer.ExposeResult + var r1 error + if rf, ok := ret.Get(0).(func(context.Context, v1.ObjectReference, client.Client, string, time.Duration) (*exposer.ExposeResult, error)); ok { + return rf(_a0, _a1, _a2, _a3, _a4) + } + if rf, ok := ret.Get(0).(func(context.Context, v1.ObjectReference, client.Client, string, time.Duration) *exposer.ExposeResult); ok { + r0 = rf(_a0, _a1, _a2, _a3, _a4) + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(*exposer.ExposeResult) + } + } + + if rf, ok := ret.Get(1).(func(context.Context, v1.ObjectReference, client.Client, string, time.Duration) error); ok { + r1 = rf(_a0, _a1, _a2, _a3, _a4) + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +// RebindVolume provides a mock function with given fields: _a0, _a1, _a2, _a3, _a4 +func (_m *GenericRestoreExposer) RebindVolume(_a0 context.Context, _a1 v1.ObjectReference, _a2 string, _a3 string, _a4 time.Duration) error { + ret := _m.Called(_a0, _a1, _a2, _a3, _a4) + + var r0 error + if rf, ok := ret.Get(0).(func(context.Context, v1.ObjectReference, string, string, time.Duration) error); ok { + r0 = rf(_a0, _a1, _a2, _a3, _a4) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type mockConstructorTestingTNewGenericRestoreExposer interface { + mock.TestingT + Cleanup(func()) +} + +// NewGenericRestoreExposer creates a new instance of GenericRestoreExposer. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations. +func NewGenericRestoreExposer(t mockConstructorTestingTNewGenericRestoreExposer) *GenericRestoreExposer { + mock := &GenericRestoreExposer{} + mock.Mock.Test(t) + + t.Cleanup(func() { mock.AssertExpectations(t) }) + + return mock +} diff --git a/pkg/restore/dataupload_retrieve_action.go b/pkg/restore/dataupload_retrieve_action.go new file mode 100644 index 000000000..f42837310 --- /dev/null +++ b/pkg/restore/dataupload_retrieve_action.go @@ -0,0 +1,105 @@ +/* +Copyright 2020 the Velero contributors. + + Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package restore + +import ( + "context" + "encoding/json" + + "github.com/pkg/errors" + "github.com/sirupsen/logrus" + corev1api "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + corev1client "k8s.io/client-go/kubernetes/typed/core/v1" + + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + velerov2alpha1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1" + "github.com/vmware-tanzu/velero/pkg/label" + "github.com/vmware-tanzu/velero/pkg/plugin/velero" +) + +type DataUploadRetrieveAction struct { + logger logrus.FieldLogger + configMapClient corev1client.ConfigMapInterface +} + +func NewDataUploadRetrieveAction(logger logrus.FieldLogger, configMapClient corev1client.ConfigMapInterface) *DataUploadRetrieveAction { + return &DataUploadRetrieveAction{ + logger: logger, + configMapClient: configMapClient, + } +} + +func (d *DataUploadRetrieveAction) AppliesTo() (velero.ResourceSelector, error) { + return velero.ResourceSelector{ + IncludedResources: []string{"datauploads.velero.io"}, + }, nil +} + +func (d *DataUploadRetrieveAction) Execute(input *velero.RestoreItemActionExecuteInput) (*velero.RestoreItemActionExecuteOutput, error) { + d.logger.Info("Executing DataUploadRetrieveAction") + + dataUpload := velerov2alpha1.DataUpload{} + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(input.ItemFromBackup.UnstructuredContent(), &dataUpload); err != nil { + d.logger.Errorf("unable to convert unstructured item to DataUpload: %s", err.Error()) + return nil, errors.Wrap(err, "unable to convert unstructured item to DataUpload.") + } + + dataUploadResult := velerov2alpha1.DataUploadResult{ + BackupStorageLocation: dataUpload.Spec.BackupStorageLocation, + DataMover: dataUpload.Spec.DataMover, + SnapshotID: dataUpload.Status.SnapshotID, + SourceNamespace: dataUpload.Spec.SourceNamespace, + DataMoverResult: dataUpload.Status.DataMoverResult, + } + + jsonBytes, err := json.Marshal(dataUploadResult) + if err != nil { + d.logger.Errorf("fail to convert DataUploadResult to JSON: %s", err.Error()) + return nil, errors.Wrap(err, "fail to convert DataUploadResult to JSON") + } + + cm := corev1api.ConfigMap{ + TypeMeta: metav1.TypeMeta{ + Kind: "ConfigMap", + APIVersion: corev1api.SchemeGroupVersion.String(), + }, + ObjectMeta: metav1.ObjectMeta{ + GenerateName: dataUpload.Name + "-", + Namespace: dataUpload.Namespace, + Labels: map[string]string{ + velerov1api.RestoreUIDLabel: label.GetValidName(string(input.Restore.UID)), + velerov1api.PVCNamespaceNameLabel: dataUpload.Spec.SourceNamespace + "." + dataUpload.Spec.SourcePVC, + velerov1api.ResourceUsageLabel: string(velerov1api.VeleroResourceUsageDataUploadResult), + }, + }, + Data: map[string]string{ + string(input.Restore.UID): string(jsonBytes), + }, + } + + _, err = d.configMapClient.Create(context.Background(), &cm, metav1.CreateOptions{}) + if err != nil { + d.logger.Errorf("fail to create DataUploadResult ConfigMap %s/%s: %s", cm.Namespace, cm.Name, err.Error()) + return nil, errors.Wrap(err, "fail to create DataUploadResult ConfigMap") + } + + return &velero.RestoreItemActionExecuteOutput{ + SkipRestore: true, + }, nil +} diff --git a/pkg/restore/dataupload_retrieve_action_test.go b/pkg/restore/dataupload_retrieve_action_test.go new file mode 100644 index 000000000..e04050e6d --- /dev/null +++ b/pkg/restore/dataupload_retrieve_action_test.go @@ -0,0 +1,88 @@ +/* +Copyright 2020 the Velero contributors. + + Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package restore + +import ( + "context" + "fmt" + "testing" + + "github.com/stretchr/testify/require" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/kubernetes/fake" + + velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + velerov2alpha1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1" + "github.com/vmware-tanzu/velero/pkg/builder" + "github.com/vmware-tanzu/velero/pkg/plugin/velero" + velerotest "github.com/vmware-tanzu/velero/pkg/test" +) + +func TestDataUploadRetrieveActionExectue(t *testing.T) { + tests := []struct { + name string + dataUpload *velerov2alpha1.DataUpload + restore *velerov1.Restore + expectedDataUploadResult *corev1.ConfigMap + expectedErr string + }{ + { + name: "DataUploadRetrieve Action test", + dataUpload: builder.ForDataUpload("velero", "testDU").SourceNamespace("testNamespace").SourcePVC("testPVC").Result(), + restore: builder.ForRestore("velero", "testRestore").ObjectMeta(builder.WithUID("testingUID")).Result(), + expectedDataUploadResult: builder.ForConfigMap("velero", "").ObjectMeta(builder.WithGenerateName("testDU-"), builder.WithLabels(velerov1.PVCNamespaceNameLabel, "testNamespace.testPVC", velerov1.RestoreUIDLabel, "testingUID", velerov1.ResourceUsageLabel, string(velerov1.VeleroResourceUsageDataUploadResult))).Data("testingUID", `{"backupStorageLocation":"","sourceNamespace":"testNamespace"}`).Result(), + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + logger := velerotest.NewLogger() + cmClient := fake.NewSimpleClientset() + + var unstructuredDataUpload map[string]interface{} + if tc.dataUpload != nil { + var err error + unstructuredDataUpload, err = runtime.DefaultUnstructuredConverter.ToUnstructured(tc.dataUpload) + require.NoError(t, err) + } + input := velero.RestoreItemActionExecuteInput{ + Restore: tc.restore, + ItemFromBackup: &unstructured.Unstructured{Object: unstructuredDataUpload}, + } + + action := NewDataUploadRetrieveAction(logger, cmClient.CoreV1().ConfigMaps("velero")) + _, err := action.Execute(&input) + if tc.expectedErr != "" { + require.Equal(t, tc.expectedErr, err.Error()) + } + require.NoError(t, err) + + if tc.expectedDataUploadResult != nil { + cmList, err := cmClient.CoreV1().ConfigMaps("velero").List(context.Background(), metav1.ListOptions{ + LabelSelector: fmt.Sprintf("%s=%s,%s=%s", velerov1.RestoreUIDLabel, "testingUID", velerov1.PVCNamespaceNameLabel, tc.dataUpload.Spec.SourceNamespace+"."+tc.dataUpload.Spec.SourcePVC), + }) + require.NoError(t, err) + // debug + fmt.Printf("CM: %s\n", &cmList.Items[0]) + require.Equal(t, *tc.expectedDataUploadResult, cmList.Items[0]) + } + }) + } +} diff --git a/pkg/restore/restore.go b/pkg/restore/restore.go index 1288f69f8..98eaf90cf 100644 --- a/pkg/restore/restore.go +++ b/pkg/restore/restore.go @@ -535,6 +535,21 @@ func (ctx *restoreContext) execute() (results.Result, results.Result) { // Close the progress update channel. quit <- struct{}{} + // Clean the DataUploadResult ConfigMaps + defer func() { + opts := []crclient.DeleteAllOfOption{ + crclient.InNamespace(ctx.restore.Namespace), + crclient.MatchingLabels{ + velerov1api.RestoreUIDLabel: string(ctx.restore.UID), + velerov1api.ResourceUsageLabel: string(velerov1api.VeleroResourceUsageDataUploadResult), + }, + } + err := ctx.kbClient.DeleteAllOf(go_context.Background(), &v1.ConfigMap{}, opts...) + if err != nil { + ctx.log.Errorf("Fail to batch delete DataUploadResult ConfigMaps for restore %s: %s", ctx.restore.Name, err.Error()) + } + }() + // Do a final progress update as stopping the ticker might have left last few // updates from taking place. updated := ctx.restore.DeepCopy() diff --git a/pkg/restore/restore_test.go b/pkg/restore/restore_test.go index 20bdcc13f..2a399726f 100644 --- a/pkg/restore/restore_test.go +++ b/pkg/restore/restore_test.go @@ -1397,6 +1397,18 @@ func TestRestoreActionsRunForCorrectItems(t *testing.T) { new(recordResourcesAction).ForNamespace("ns-2").ForResource("pods"): nil, }, }, + { + name: "actions run for datauploads resource", + restore: defaultRestore().Result(), + backup: defaultBackup().Result(), + tarball: test.NewTarWriter(t). + AddItems("datauploads.velero.io", builder.ForDataUpload("velero", "du").Result()). + Done(), + apiResources: []*test.APIResource{test.DataUploads()}, + actions: map[*recordResourcesAction][]string{ + new(recordResourcesAction).ForNamespace("velero").ForResource("datauploads.velero.io"): {"velero/du"}, + }, + }, } for _, tc := range tests { diff --git a/pkg/test/resources.go b/pkg/test/resources.go index dfe22278d..7c2fa17f6 100644 --- a/pkg/test/resources.go +++ b/pkg/test/resources.go @@ -183,3 +183,13 @@ func Services(items ...metav1.Object) *APIResource { Items: items, } } + +func DataUploads(items ...metav1.Object) *APIResource { + return &APIResource{ + Group: "velero.io", + Version: "v2alpha1", + Name: "datauploads", + Namespaced: true, + Items: items, + } +} From e205e2122d6c045231d76e63556fc3b6e55234e2 Mon Sep 17 00:00:00 2001 From: Xun Jiang Date: Fri, 16 Jun 2023 17:37:10 +0800 Subject: [PATCH 17/28] Retrieve DataUpload into backup result ConfigMap during volume snapshot restore. Fix issue #6117. Add CSI plugin needs builder functions. Signed-off-by: Xun Jiang --- pkg/builder/data_download_builder.go | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/pkg/builder/data_download_builder.go b/pkg/builder/data_download_builder.go index 5ba726cf2..c564c80cf 100644 --- a/pkg/builder/data_download_builder.go +++ b/pkg/builder/data_download_builder.go @@ -5,10 +5,8 @@ Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. @@ -24,7 +22,6 @@ import ( velerov2alpha1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1" ) - // DataDownloadBuilder builds DataDownload objects. type DataDownloadBuilder struct { object *velerov2alpha1api.DataDownload @@ -106,10 +103,10 @@ func (d *DataDownloadBuilder) DataMoverConfig(config *map[string]string) *DataDo } // ObjectMeta applies functional options to the DataDownload's ObjectMeta. -func (b *DataDownloadBuilder) ObjectMeta(opts ...ObjectMetaOpt) *DataDownloadBuilder { +func (d *DataDownloadBuilder) ObjectMeta(opts ...ObjectMetaOpt) *DataDownloadBuilder { for _, opt := range opts { - opt(b.object) + opt(d.object) } - return b + return d } From 65cb25a74c8df306b7b8c459f7ee94446a2b9961 Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Fri, 30 Jun 2023 14:48:51 +0800 Subject: [PATCH 18/28] fix concurrent repo ensure problem Signed-off-by: Lyndon-Li --- pkg/exposer/csi_snapshot.go | 2 +- pkg/exposer/generic_restore.go | 2 +- pkg/repository/ensurer.go | 22 ++++++++++++---------- pkg/repository/ensurer_test.go | 25 +++++++++++++++++++++---- 4 files changed, 35 insertions(+), 16 deletions(-) diff --git a/pkg/exposer/csi_snapshot.go b/pkg/exposer/csi_snapshot.go index 9c984de53..4d452a39e 100644 --- a/pkg/exposer/csi_snapshot.go +++ b/pkg/exposer/csi_snapshot.go @@ -202,7 +202,7 @@ func (e *csiSnapshotExposer) GetExposed(ctx context.Context, ownerObject corev1. }, pod) if err != nil { if apierrors.IsNotFound(err) { - curLog.WithField("backup pod", backupPodName).Errorf("Backup pod is not running in the current node %s", exposeWaitParam.NodeName) + curLog.WithField("backup pod", backupPodName).Debugf("Backup pod is not running in the current node %s", exposeWaitParam.NodeName) return nil, nil } else { return nil, errors.Wrapf(err, "error to get backup pod %s", backupPodName) diff --git a/pkg/exposer/generic_restore.go b/pkg/exposer/generic_restore.go index c5c693a2f..3617c53a8 100644 --- a/pkg/exposer/generic_restore.go +++ b/pkg/exposer/generic_restore.go @@ -123,7 +123,7 @@ func (e *genericRestoreExposer) GetExposed(ctx context.Context, ownerObject core }, pod) if err != nil { if apierrors.IsNotFound(err) { - curLog.WithField("backup pod", restorePodName).Error("Backup pod is not running in the current node") + curLog.WithField("backup pod", restorePodName).Debug("Backup pod is not running in the current node") return nil, nil } else { return nil, errors.Wrapf(err, "error to get backup pod %s", restorePodName) diff --git a/pkg/repository/ensurer.go b/pkg/repository/ensurer.go index 4bdd05fba..885363328 100644 --- a/pkg/repository/ensurer.go +++ b/pkg/repository/ensurer.go @@ -80,20 +80,18 @@ func (r *Ensurer) EnsureRepo(ctx context.Context, namespace, volumeNamespace, ba log.Debug("Released lock") }() - repo, err := GetBackupRepository(ctx, r.repoClient, namespace, backupRepoKey, true) + _, err := GetBackupRepository(ctx, r.repoClient, namespace, backupRepoKey, false) if err == nil { - log.Debug("Ready repository found") - return repo, nil - } + log.Info("Founding existing repo") + return r.waitBackupRepository(ctx, namespace, backupRepoKey) + } else if isBackupRepositoryNotFoundError(err) { + log.Info("No repository found, creating one") - if !isBackupRepositoryNotFoundError(err) { + // no repo found: create one and wait for it to be ready + return r.createBackupRepositoryAndWait(ctx, namespace, backupRepoKey) + } else { return nil, errors.WithStack(err) } - - log.Debug("No repository found, creating one") - - // no repo found: create one and wait for it to be ready - return r.createBackupRepositoryAndWait(ctx, namespace, backupRepoKey) } func (r *Ensurer) repoLock(key BackupRepositoryKey) *sync.Mutex { @@ -113,6 +111,10 @@ func (r *Ensurer) createBackupRepositoryAndWait(ctx context.Context, namespace s return nil, errors.Wrap(err, "unable to create backup repository resource") } + return r.waitBackupRepository(ctx, namespace, backupRepoKey) +} + +func (r *Ensurer) waitBackupRepository(ctx context.Context, namespace string, backupRepoKey BackupRepositoryKey) (*velerov1api.BackupRepository, error) { var repo *velerov1api.BackupRepository checkFunc := func(ctx context.Context) (bool, error) { found, err := GetBackupRepository(ctx, r.repoClient, namespace, backupRepoKey, true) diff --git a/pkg/repository/ensurer_test.go b/pkg/repository/ensurer_test.go index 72dff8a3a..e0d6a0593 100644 --- a/pkg/repository/ensurer_test.go +++ b/pkg/repository/ensurer_test.go @@ -30,13 +30,19 @@ import ( ) func TestEnsureRepo(t *testing.T) { - bkRepoObj := NewBackupRepository(velerov1.DefaultNamespace, BackupRepositoryKey{ + bkRepoObjReady := NewBackupRepository(velerov1.DefaultNamespace, BackupRepositoryKey{ VolumeNamespace: "fake-ns", BackupLocation: "fake-bsl", RepositoryType: "fake-repo-type", }) - bkRepoObj.Status.Phase = velerov1.BackupRepositoryPhaseReady + bkRepoObjReady.Status.Phase = velerov1.BackupRepositoryPhaseReady + + bkRepoObjNotReady := NewBackupRepository(velerov1.DefaultNamespace, BackupRepositoryKey{ + VolumeNamespace: "fake-ns", + BackupLocation: "fake-bsl", + RepositoryType: "fake-repo-type", + }) scheme := runtime.NewScheme() velerov1.AddToScheme(scheme) @@ -82,10 +88,21 @@ func TestEnsureRepo(t *testing.T) { bsl: "fake-bsl", repositoryType: "fake-repo-type", kubeClientObj: []runtime.Object{ - bkRepoObj, + bkRepoObjReady, }, runtimeScheme: scheme, - expectedRepo: bkRepoObj, + expectedRepo: bkRepoObjReady, + }, + { + name: "wait existing repo fail", + namespace: "fake-ns", + bsl: "fake-bsl", + repositoryType: "fake-repo-type", + kubeClientObj: []runtime.Object{ + bkRepoObjNotReady, + }, + runtimeScheme: scheme, + err: "failed to wait BackupRepository: timed out waiting for the condition", }, { name: "create fail", From 22a99c34b99186702d07209b78e672352196f3f0 Mon Sep 17 00:00:00 2001 From: Ming Date: Fri, 30 Jun 2023 07:28:24 +0000 Subject: [PATCH 19/28] Fix data path concurrent Signed-off-by: Ming --- pkg/controller/data_download_controller.go | 54 +++++++++--------- .../data_download_controller_test.go | 7 ++- pkg/controller/data_upload_controller.go | 55 ++++++++++--------- pkg/controller/data_upload_controller_test.go | 8 +-- 4 files changed, 69 insertions(+), 55 deletions(-) diff --git a/pkg/controller/data_download_controller.go b/pkg/controller/data_download_controller.go index 1f0e78c69..95531cb6b 100644 --- a/pkg/controller/data_download_controller.go +++ b/pkg/controller/data_download_controller.go @@ -161,7 +161,25 @@ func (r *DataDownloadReconciler) Reconcile(ctx context.Context, req ctrl.Request return ctrl.Result{}, nil } - log.Info("Restore PVC is ready") + log.Info("Restore PVC is ready and creating data path routine") + + // Need to first create file system BR and get data path instance then update data upload status + callbacks := datapath.Callbacks{ + OnCompleted: r.OnDataDownloadCompleted, + OnFailed: r.OnDataDownloadFailed, + OnCancelled: r.OnDataDownloadCancelled, + OnProgress: r.OnDataDownloadProgress, + } + + fsRestore, err = r.dataPathMgr.CreateFileSystemBR(dd.Name, dataUploadDownloadRequestor, ctx, r.client, dd.Namespace, callbacks, log) + if err != nil { + if err == datapath.ConcurrentLimitExceed { + log.Info("Data path instance is concurrent limited requeue later") + return ctrl.Result{Requeue: true, RequeueAfter: time.Minute}, nil + } else { + return r.errorOut(ctx, dd, err, "error to create data path", log) + } + } // Update status to InProgress original := dd.DeepCopy() @@ -174,7 +192,12 @@ func (r *DataDownloadReconciler) Reconcile(ctx context.Context, req ctrl.Request log.Info("Data download is marked as in progress") - return r.runCancelableDataPath(ctx, dd, result, log) + reconcileResult, err := r.runCancelableDataPath(ctx, fsRestore, dd, result, log) + if err != nil { + log.Errorf("Failed to run cancelable data path for %s with err %v", dd.Name, err) + r.closeDataPath(ctx, dd.Name) + } + return reconcileResult, err } else if dd.Status.Phase == velerov2alpha1api.DataDownloadPhaseInProgress { log.Info("Data download is in progress") if dd.Spec.Cancel { @@ -203,25 +226,7 @@ func (r *DataDownloadReconciler) Reconcile(ctx context.Context, req ctrl.Request } } -func (r *DataDownloadReconciler) runCancelableDataPath(ctx context.Context, dd *velerov2alpha1api.DataDownload, res *exposer.ExposeResult, log logrus.FieldLogger) (reconcile.Result, error) { - log.Info("Creating data path routine") - callbacks := datapath.Callbacks{ - OnCompleted: r.OnDataDownloadCompleted, - OnFailed: r.OnDataDownloadFailed, - OnCancelled: r.OnDataDownloadCancelled, - OnProgress: r.OnDataDownloadProgress, - } - - fsRestore, err := r.dataPathMgr.CreateFileSystemBR(dd.Name, dataUploadDownloadRequestor, ctx, r.client, dd.Namespace, callbacks, log) - if err != nil { - if err == datapath.ConcurrentLimitExceed { - log.Info("runCancelableDataDownload is concurrent limited") - return ctrl.Result{Requeue: true, RequeueAfter: time.Minute}, nil - } else { - return r.errorOut(ctx, dd, err, "error to create data path", log) - } - } - +func (r *DataDownloadReconciler) runCancelableDataPath(ctx context.Context, fsRestore datapath.AsyncBR, dd *velerov2alpha1api.DataDownload, res *exposer.ExposeResult, log logrus.FieldLogger) (reconcile.Result, error) { path, err := exposer.GetPodVolumeHostPath(ctx, res.ByPod.HostingPod, res.ByPod.PVC, r.client, r.fileSystem, log) if err != nil { return r.errorOut(ctx, dd, err, "error exposing host path for pod volume", log) @@ -437,12 +442,11 @@ func (r *DataDownloadReconciler) updateStatusToFailed(ctx context.Context, dd *v dd.Status.Message = errors.WithMessage(err, msg).Error() dd.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()} - if err = r.client.Patch(ctx, dd, client.MergeFrom(original)); err != nil { - log.WithError(err).Error("error updating DataDownload status") - return err + if patchErr := r.client.Patch(ctx, dd, client.MergeFrom(original)); patchErr != nil { + log.WithError(patchErr).Error("error updating DataDownload status") } - return nil + return err } func (r *DataDownloadReconciler) acceptDataDownload(ctx context.Context, dd *velerov2alpha1api.DataDownload) (bool, error) { diff --git a/pkg/controller/data_download_controller_test.go b/pkg/controller/data_download_controller_test.go index 5db1170c7..7ccf20e4e 100644 --- a/pkg/controller/data_download_controller_test.go +++ b/pkg/controller/data_download_controller_test.go @@ -192,7 +192,12 @@ func TestDataDownloadReconcile(t *testing.T) { }, }) - require.Nil(t, err) + if test.isGetExposeErr { + assert.Contains(t, err.Error(), test.expectedStatusMsg) + } else { + require.Nil(t, err) + } + require.NotNil(t, actualResult) dd := velerov2alpha1api.DataDownload{} diff --git a/pkg/controller/data_upload_controller.go b/pkg/controller/data_upload_controller.go index 759b297a1..735026cda 100644 --- a/pkg/controller/data_upload_controller.go +++ b/pkg/controller/data_upload_controller.go @@ -160,7 +160,25 @@ func (r *DataUploadReconciler) Reconcile(ctx context.Context, req ctrl.Request) return ctrl.Result{}, nil } - log.Info("Exposed snapshot is ready") + log.Info("Exposed snapshot is ready and creating data path routine") + + // Need to first create file system BR and get data path instance then update data upload status + callbacks := datapath.Callbacks{ + OnCompleted: r.OnDataUploadCompleted, + OnFailed: r.OnDataUploadFailed, + OnCancelled: r.OnDataUploadCancelled, + OnProgress: r.OnDataUploadProgress, + } + + fsBackup, err = r.dataPathMgr.CreateFileSystemBR(du.Name, dataUploadDownloadRequestor, ctx, r.client, du.Namespace, callbacks, log) + if err != nil { + if err == datapath.ConcurrentLimitExceed { + log.Info("Data path instance is concurrent limited requeue later") + return ctrl.Result{Requeue: true, RequeueAfter: time.Minute}, nil + } else { + return r.errorOut(ctx, &du, err, "error to create data path", log) + } + } // Update status to InProgress original := du.DeepCopy() @@ -171,7 +189,12 @@ func (r *DataUploadReconciler) Reconcile(ctx context.Context, req ctrl.Request) } log.Info("Data upload is marked as in progress") - return r.runCancelableDataUpload(ctx, &du, res, log) + result, err := r.runCancelableDataUpload(ctx, fsBackup, &du, res, log) + if err != nil { + log.Errorf("Failed to run cancelable data path for %s with err %v", du.Name, err) + r.closeDataPath(ctx, du.Name) + } + return result, err } else if du.Status.Phase == velerov2alpha1api.DataUploadPhaseInProgress { log.Info("Data upload is in progress") if du.Spec.Cancel { @@ -198,25 +221,8 @@ func (r *DataUploadReconciler) Reconcile(ctx context.Context, req ctrl.Request) } } -func (r *DataUploadReconciler) runCancelableDataUpload(ctx context.Context, du *velerov2alpha1api.DataUpload, res *exposer.ExposeResult, log logrus.FieldLogger) (reconcile.Result, error) { - log.Info("Creating data path routine") - callbacks := datapath.Callbacks{ - OnCompleted: r.OnDataUploadCompleted, - OnFailed: r.OnDataUploadFailed, - OnCancelled: r.OnDataUploadCancelled, - OnProgress: r.OnDataUploadProgress, - } - - fsBackup, err := r.dataPathMgr.CreateFileSystemBR(du.Name, dataUploadDownloadRequestor, ctx, r.client, du.Namespace, callbacks, log) - if err != nil { - if err == datapath.ConcurrentLimitExceed { - log.Info("runCancelableDataUpload is concurrent limited") - return ctrl.Result{Requeue: true, RequeueAfter: time.Minute}, nil - } else { - return r.errorOut(ctx, du, err, "error to create data path", log) - } - } - +func (r *DataUploadReconciler) runCancelableDataUpload(ctx context.Context, fsBackup datapath.AsyncBR, du *velerov2alpha1api.DataUpload, res *exposer.ExposeResult, log logrus.FieldLogger) (reconcile.Result, error) { + log.Info("Run cancelable dataUpload") path, err := exposer.GetPodVolumeHostPath(ctx, res.ByPod.HostingPod, res.ByPod.PVC, r.client, r.fileSystem, log) if err != nil { return r.errorOut(ctx, du, err, "error exposing host path for pod volume", log) @@ -460,12 +466,11 @@ func (r *DataUploadReconciler) updateStatusToFailed(ctx context.Context, du *vel } du.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()} - if err = r.client.Patch(ctx, du, client.MergeFrom(original)); err != nil { - log.WithError(err).Error("error updating DataUpload status") - return err + if patchErr := r.client.Patch(ctx, du, client.MergeFrom(original)); patchErr != nil { + log.WithError(patchErr).Error("error updating DataUpload status") } - return nil + return err } func (r *DataUploadReconciler) acceptDataUpload(ctx context.Context, du *velerov2alpha1api.DataUpload) (bool, error) { diff --git a/pkg/controller/data_upload_controller_test.go b/pkg/controller/data_upload_controller_test.go index 0515a05a4..654e07531 100644 --- a/pkg/controller/data_upload_controller_test.go +++ b/pkg/controller/data_upload_controller_test.go @@ -291,8 +291,8 @@ func TestReconcile(t *testing.T) { expectedProcessed: true, expected: dataUploadBuilder().Phase(velerov2alpha1api.DataUploadPhaseFailed).Result(), expectedRequeue: ctrl.Result{}, - }, - { + expectedErrMsg: "unknown type type of snapshot exposer is not exist", + }, { name: "Dataupload should be accepted", du: dataUploadBuilder().Result(), pod: builder.ForPod(velerov1api.DefaultNamespace, dataUploadName).Volumes(&corev1.Volume{Name: "dataupload-1"}).Result(), @@ -336,7 +336,7 @@ func TestReconcile(t *testing.T) { pod: builder.ForPod(velerov1api.DefaultNamespace, dataUploadName).Volumes(&corev1.Volume{Name: "dataupload-1"}).Result(), du: dataUploadBuilder().Phase(velerov2alpha1api.DataUploadPhasePrepared).SnapshotType(fakeSnapshotType).Cancel(true).Result(), expectedProcessed: false, - expected: dataUploadBuilder().Phase(velerov2alpha1api.DataUploadPhaseInProgress).Result(), + expected: dataUploadBuilder().Phase(velerov2alpha1api.DataUploadPhasePrepared).Result(), expectedRequeue: ctrl.Result{Requeue: true, RequeueAfter: time.Minute}, }, } @@ -400,7 +400,7 @@ func TestReconcile(t *testing.T) { if test.expectedErrMsg == "" { require.NoError(t, err) } else { - assert.Equal(t, err.Error(), test.expectedErrMsg) + assert.Contains(t, err.Error(), test.expectedErrMsg) } du := velerov2alpha1api.DataUpload{} From dcdd5f99d6b8fc04e20da88e731b4b392fae1cc7 Mon Sep 17 00:00:00 2001 From: Mateus Oliveira <66965232+mateusoliveira43@users.noreply.github.com> Date: Fri, 30 Jun 2023 16:15:37 -0300 Subject: [PATCH 20/28] fix: Remove duplicated stale job (#6416) * fix: Remove duplicated stale job Signed-off-by: Mateus Oliveira * fixup! fix: Remove duplicated stale job Signed-off-by: Mateus Oliveira * fixup! fix: Remove duplicated stale job Signed-off-by: Mateus Oliveira --------- Signed-off-by: Mateus Oliveira --- .github/stale.yml | 44 ------------------------------ .github/workflows/stale-issues.yml | 15 +++++----- 2 files changed, 7 insertions(+), 52 deletions(-) delete mode 100644 .github/stale.yml diff --git a/.github/stale.yml b/.github/stale.yml deleted file mode 100644 index ce96631aa..000000000 --- a/.github/stale.yml +++ /dev/null @@ -1,44 +0,0 @@ -# Number of days of inactivity before an issue becomes stale -daysUntilStale: 60 -# Number of days of inactivity before a stale issue is closed -daysUntilClose: 14 -# Issues with these labels will never be considered stale -exemptLabels: - - Epic - - Area/CLI - - Area/Cloud/AWS - - Area/Cloud/Azure - - Area/Cloud/GCP - - Area/Cloud/vSphere - - Area/CSI - - Area/Design - - Area/Documentation - - Area/Plugins - - Bug - - Enhancement/User - - kind/requirement - - kind/refactor - - kind/tech-debt - - limitation - - Needs investigation - - Needs triage - - Needs Product - - P0 - Hair on fire - - P1 - Important - - P2 - Long-term important - - P3 - Wouldn't it be nice if... - - Product Requirements - - Restic - GA - - Restic - - release-blocker - - Security -# Label to use when marking an issue as stale -staleLabel: staled -# Comment to post when marking an issue as stale. Set to `false` to disable -markComment: > - This issue has been automatically marked as stale because it has not had - recent activity. It will be closed if no further activity occurs. Thank you - for your contributions. -# Comment to post when closing a stale issue. Set to `false` to disable -closeComment: > - Closing the stale issue. diff --git a/.github/workflows/stale-issues.yml b/.github/workflows/stale-issues.yml index 80944e8cf..df5fe1fc7 100644 --- a/.github/workflows/stale-issues.yml +++ b/.github/workflows/stale-issues.yml @@ -1,8 +1,7 @@ name: "Close stale issues and PRs" on: schedule: - # First of every month - - cron: "30 1 * * *" + - cron: "30 1 * * *" # Every day at 1:30 UTC jobs: stale: @@ -11,14 +10,14 @@ jobs: - uses: actions/stale@v3 with: repo-token: ${{ secrets.GITHUB_TOKEN }} - stale-issue-message: "This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. If a Velero team member has requested log or more information, please provide the output of the shared commands." - close-issue-message: "This issue was closed because it has been stalled for 5 days with no activity." - days-before-issue-stale: 30 - days-before-issue-close: 5 + stale-issue-message: "This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands." + close-issue-message: "This issue was closed because it has been stalled for 14 days with no activity." + days-before-issue-stale: 60 + days-before-issue-close: 14 + stale-issue-label: staled # Disable stale PRs for now; they can remain open. days-before-pr-stale: -1 days-before-pr-close: -1 # Only issues made after Feb 09 2021. start-date: "2021-09-02T00:00:00" - # Only make issues stale if they have these labels. Comma separated. - only-labels: "Needs info,Duplicate" + exempt-issue-labels: "Epic,Area/CLI,Area/Cloud/AWS,Area/Cloud/Azure,Area/Cloud/GCP,Area/Cloud/vSphere,Area/CSI,Area/Design,Area/Documentation,Area/Plugins,Bug,Enhancement/User,kind/requirement,kind/refactor,kind/tech-debt,limitation,Needs investigation,Needs triage,Needs Product,P0 - Hair on fire,P1 - Important,P2 - Long-term important,P3 - Wouldn't it be nice if...,Product Requirements,Restic - GA,Restic,release-blocker,Security" From 0416b93b07ba3d902eb12f95d6ba21ae86e23cc3 Mon Sep 17 00:00:00 2001 From: Zhiqiang Zhang Date: Sat, 1 Jul 2023 21:50:20 +0800 Subject: [PATCH 21/28] fix doc typo Signed-off-by: zhangzhiqiang02 --- site/content/docs/main/file-system-backup.md | 2 +- site/content/docs/v1.10/file-system-backup.md | 2 +- site/content/docs/v1.11/file-system-backup.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/site/content/docs/main/file-system-backup.md b/site/content/docs/main/file-system-backup.md index 89c43ce7b..e6a1d0e8f 100644 --- a/site/content/docs/main/file-system-backup.md +++ b/site/content/docs/main/file-system-backup.md @@ -74,7 +74,7 @@ Integrated Edition (formerly VMware Enterprise PKS), or Microsoft Azure. **RancherOS** -Update the host path for volumes in the nonde-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to +Update the host path for volumes in the node-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to `/opt/rke/var/lib/kubelet/pods`. ```yaml diff --git a/site/content/docs/v1.10/file-system-backup.md b/site/content/docs/v1.10/file-system-backup.md index 6fcd1a33b..f8badb109 100644 --- a/site/content/docs/v1.10/file-system-backup.md +++ b/site/content/docs/v1.10/file-system-backup.md @@ -74,7 +74,7 @@ Integrated Edition (formerly VMware Enterprise PKS), or Microsoft Azure. **RancherOS** -Update the host path for volumes in the nonde-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to +Update the host path for volumes in the node-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to `/opt/rke/var/lib/kubelet/pods`. ```yaml diff --git a/site/content/docs/v1.11/file-system-backup.md b/site/content/docs/v1.11/file-system-backup.md index 881747895..dc51cab84 100644 --- a/site/content/docs/v1.11/file-system-backup.md +++ b/site/content/docs/v1.11/file-system-backup.md @@ -74,7 +74,7 @@ Integrated Edition (formerly VMware Enterprise PKS), or Microsoft Azure. **RancherOS** -Update the host path for volumes in the nonde-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to +Update the host path for volumes in the node-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to `/opt/rke/var/lib/kubelet/pods`. ```yaml From 40b2ee1323a4155e5c37161cad04391c5059e5ad Mon Sep 17 00:00:00 2001 From: Xun Jiang Date: Thu, 29 Jun 2023 13:48:22 +0800 Subject: [PATCH 22/28] Modify DownloadRequest controller logic 1. Avoid patch DownloadRequest when it's deleted. 2. Add periodic enqueue resource for reconcile. Signed-off-by: Xun Jiang --- changelogs/unreleased/6433-blackpiglet | 1 + pkg/controller/download_request_controller.go | 74 +++++++++++++------ .../download_request_controller_test.go | 42 +++++------ 3 files changed, 75 insertions(+), 42 deletions(-) create mode 100644 changelogs/unreleased/6433-blackpiglet diff --git a/changelogs/unreleased/6433-blackpiglet b/changelogs/unreleased/6433-blackpiglet new file mode 100644 index 000000000..a804fc890 --- /dev/null +++ b/changelogs/unreleased/6433-blackpiglet @@ -0,0 +1 @@ +Modify DownloadRequest controller logic \ No newline at end of file diff --git a/pkg/controller/download_request_controller.go b/pkg/controller/download_request_controller.go index 479c6d407..1f3de3955 100644 --- a/pkg/controller/download_request_controller.go +++ b/pkg/controller/download_request_controller.go @@ -18,6 +18,7 @@ package controller import ( "context" + "time" "github.com/pkg/errors" "github.com/sirupsen/logrus" @@ -25,12 +26,18 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" clocks "k8s.io/utils/clock" ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/builder" kbclient "sigs.k8s.io/controller-runtime/pkg/client" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" "github.com/vmware-tanzu/velero/pkg/itemoperationmap" "github.com/vmware-tanzu/velero/pkg/persistence" "github.com/vmware-tanzu/velero/pkg/plugin/clientmgmt" + "github.com/vmware-tanzu/velero/pkg/util/kube" +) + +const ( + defaultDownloadRequestSyncPeriod = time.Minute ) // downloadRequestReconciler reconciles a DownloadRequest object @@ -93,15 +100,6 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ return ctrl.Result{}, errors.WithStack(err) } - original := downloadRequest.DeepCopy() - defer func() { - // Always attempt to Patch the downloadRequest object and status after each reconciliation. - if err := r.client.Patch(ctx, downloadRequest, kbclient.MergeFrom(original)); err != nil { - log.WithError(err).Error("Error updating download request") - return - } - }() - if downloadRequest.Status != (velerov1api.DownloadRequestStatus{}) && downloadRequest.Status.Expiration != nil { if downloadRequest.Status.Expiration.Time.Before(r.clock.Now()) { // Delete any request that is expired, regardless of the phase: it is not @@ -111,19 +109,25 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ log.WithError(err).Error("Error deleting an expired download request") return ctrl.Result{}, errors.WithStack(err) } - return ctrl.Result{Requeue: false}, nil + return ctrl.Result{}, nil } else if downloadRequest.Status.Phase == velerov1api.DownloadRequestPhaseProcessed { - // Requeue the request if is not yet expired and has already been processed before, - // since it might still be in use by the logs streaming and shouldn't - // be deleted until after its expiration. - log.Debug("DownloadRequest has not yet expired - requeueing") - return ctrl.Result{Requeue: true}, nil + log.Debug("DownloadRequest has not yet expired.") + return ctrl.Result{}, nil } } // Process a brand new request. - backupName := downloadRequest.Spec.Target.Name if downloadRequest.Status.Phase == "" || downloadRequest.Status.Phase == velerov1api.DownloadRequestPhaseNew { + backupName := downloadRequest.Spec.Target.Name + original := downloadRequest.DeepCopy() + defer func() { + // Always attempt to Patch the downloadRequest object and status for new DownloadRequest. + if err := r.client.Patch(ctx, downloadRequest, kbclient.MergeFrom(original)); err != nil { + log.WithError(err).Error("Error updating download request") + return + } + }() + // Update the expiration. downloadRequest.Status.Expiration = &metav1.Time{Time: r.clock.Now().Add(persistence.DownloadURLTTL)} @@ -136,6 +140,11 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ Namespace: downloadRequest.Namespace, Name: downloadRequest.Spec.Target.Name, }, restore); err != nil { + if apierrors.IsNotFound(err) { + log.WithError(err).Error("fail to get restore for DownloadRequest") + return ctrl.Result{}, nil + } + log.Warnf("Fail to get restore for DownloadRequest %s. Retry later.", err.Error()) return ctrl.Result{}, errors.WithStack(err) } backupName = restore.Spec.BackupName @@ -146,6 +155,11 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ Namespace: downloadRequest.Namespace, Name: backupName, }, backup); err != nil { + if apierrors.IsNotFound(err) { + log.WithError(err).Error("fail to get backup for DownloadRequest") + return ctrl.Result{}, nil + } + log.Warnf("fail to get backup for DownloadRequest %s. Retry later.", err.Error()) return ctrl.Result{}, errors.WithStack(err) } @@ -154,6 +168,11 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ Namespace: backup.Namespace, Name: backup.Spec.StorageLocation, }, location); err != nil { + if apierrors.IsNotFound(err) { + log.Errorf("BSL for DownloadRequest cannot be found") + return ctrl.Result{}, nil + } + log.Warnf("Fail to get BSL for DownloadRequest: %s", err.Error()) return ctrl.Result{}, errors.WithStack(err) } @@ -163,7 +182,9 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ backupStore, err := r.backupStoreGetter.Get(location, pluginManager, log) if err != nil { log.WithError(err).Error("Error getting a backup store") - return ctrl.Result{}, errors.WithStack(err) + // Fail to get backup store is due to BSL setting issue or credential issue. + // It cannot be recovered. No need to retry. + return ctrl.Result{}, nil } // If this is a request for backup item operations, force upload of in-memory operations that @@ -180,8 +201,10 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ // ignore errors here. If we can't upload anything here, process the download as usual _ = r.restoreItemOperationsMap.UpdateForRestore(backupStore, downloadRequest.Spec.Target.Name) } + if downloadRequest.Status.DownloadURL, err = backupStore.GetDownloadURL(downloadRequest.Spec.Target); err != nil { - return ctrl.Result{Requeue: true}, errors.WithStack(err) + log.Warnf("fail to get Backup metadata file's download URL %s, retry later: %s", downloadRequest.Spec.Target, err) + return ctrl.Result{}, errors.WithStack(err) } downloadRequest.Status.Phase = velerov1api.DownloadRequestPhaseProcessed @@ -190,13 +213,22 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ downloadRequest.Status.Expiration = &metav1.Time{Time: r.clock.Now().Add(persistence.DownloadURLTTL)} } - // Requeue is mostly to handle deleting any expired requests that were not - // deleted as part of the normal client flow for whatever reason. - return ctrl.Result{Requeue: true}, nil + return ctrl.Result{}, nil } func (r *downloadRequestReconciler) SetupWithManager(mgr ctrl.Manager) error { + downloadRequestSource := kube.NewPeriodicalEnqueueSource(r.log, mgr.GetClient(), + &velerov1api.DownloadRequestList{}, defaultDownloadRequestSyncPeriod, kube.PeriodicalEnqueueSourceOption{}) + downloadRequestPredicates := kube.NewGenericEventPredicate(func(object kbclient.Object) bool { + downloadRequest := object.(*velerov1api.DownloadRequest) + if downloadRequest.Status != (velerov1api.DownloadRequestStatus{}) && downloadRequest.Status.Expiration != nil { + return downloadRequest.Status.Expiration.Time.Before(r.clock.Now()) + } + return true + }) + return ctrl.NewControllerManagedBy(mgr). For(&velerov1api.DownloadRequest{}). + Watches(downloadRequestSource, nil, builder.WithPredicates(downloadRequestPredicates)). Complete(r) } diff --git a/pkg/controller/download_request_controller_test.go b/pkg/controller/download_request_controller_test.go index ad53b6972..184771991 100644 --- a/pkg/controller/download_request_controller_test.go +++ b/pkg/controller/download_request_controller_test.go @@ -156,55 +156,55 @@ var _ = Describe("Download Request Reconciler", func() { } }, - Entry("backup contents request for nonexistent backup returns an error", request{ + Entry("backup contents request for nonexistent backup returns nil", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindBackupContents, "a1-backup").Result(), backup: builder.ForBackup(velerov1api.DefaultNamespace, "non-matching-backup").StorageLocation("a-location").Result(), backupLocation: builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(), - expectedReconcileErr: "backups.velero.io \"a1-backup\" not found", - expectedRequeue: ctrl.Result{Requeue: false}, + expectedReconcileErr: "", + expectedRequeue: ctrl.Result{}, }), - Entry("restore log request for nonexistent restore returns an error", request{ + Entry("restore log request for nonexistent restore returns nil", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindRestoreLog, "a-backup-20170912150214").Result(), restore: builder.ForRestore(velerov1api.DefaultNamespace, "non-matching-restore").Phase(velerov1api.RestorePhaseCompleted).Backup("a-backup").Result(), backup: defaultBackup(), backupLocation: builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(), - expectedReconcileErr: "restores.velero.io \"a-backup-20170912150214\" not found", - expectedRequeue: ctrl.Result{Requeue: false}, + expectedReconcileErr: "", + expectedRequeue: ctrl.Result{}, }), - Entry("backup contents request for backup with nonexistent location returns an error", request{ + Entry("backup contents request for backup with nonexistent location returns nil", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindBackupContents, "a-backup").Result(), backup: defaultBackup(), backupLocation: builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "non-matching-location").Provider("a-provider").Bucket("a-bucket").Result(), - expectedReconcileErr: "backupstoragelocations.velero.io \"a-location\" not found", - expectedRequeue: ctrl.Result{Requeue: false}, + expectedReconcileErr: "", + expectedRequeue: ctrl.Result{}, }), Entry("backup contents request with phase '' gets a url", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindBackupContents, "a-backup").Result(), backup: defaultBackup(), backupLocation: builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(), expectGetsURL: true, - expectedRequeue: ctrl.Result{Requeue: true}, + expectedRequeue: ctrl.Result{}, }), Entry("backup contents request with phase 'New' gets a url", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase(velerov1api.DownloadRequestPhaseNew).Target(velerov1api.DownloadTargetKindBackupContents, "a-backup").Result(), backup: defaultBackup(), backupLocation: builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(), expectGetsURL: true, - expectedRequeue: ctrl.Result{Requeue: true}, + expectedRequeue: ctrl.Result{}, }), Entry("backup log request with phase '' gets a url", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindBackupLog, "a-backup").Result(), backup: defaultBackup(), backupLocation: builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(), expectGetsURL: true, - expectedRequeue: ctrl.Result{Requeue: true}, + expectedRequeue: ctrl.Result{}, }), Entry("backup log request with phase 'New' gets a url", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase(velerov1api.DownloadRequestPhaseNew).Target(velerov1api.DownloadTargetKindBackupLog, "a-backup").Result(), backup: defaultBackup(), backupLocation: builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(), expectGetsURL: true, - expectedRequeue: ctrl.Result{Requeue: true}, + expectedRequeue: ctrl.Result{}, }), Entry("restore log request with phase '' gets a url", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindRestoreLog, "a-backup-20170912150214").Result(), @@ -212,7 +212,7 @@ var _ = Describe("Download Request Reconciler", func() { backup: defaultBackup(), backupLocation: builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(), expectGetsURL: true, - expectedRequeue: ctrl.Result{Requeue: true}, + expectedRequeue: ctrl.Result{}, }), Entry("restore log request with phase 'New' gets a url", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase(velerov1api.DownloadRequestPhaseNew).Target(velerov1api.DownloadTargetKindRestoreLog, "a-backup-20170912150214").Result(), @@ -220,7 +220,7 @@ var _ = Describe("Download Request Reconciler", func() { restore: builder.ForRestore(velerov1api.DefaultNamespace, "a-backup-20170912150214").Phase(velerov1api.RestorePhaseCompleted).Backup("a-backup").Result(), backupLocation: builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(), expectGetsURL: true, - expectedRequeue: ctrl.Result{Requeue: true}, + expectedRequeue: ctrl.Result{}, }), Entry("restore results request with phase '' gets a url", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindRestoreResults, "a-backup-20170912150214").Result(), @@ -228,7 +228,7 @@ var _ = Describe("Download Request Reconciler", func() { backup: defaultBackup(), backupLocation: builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(), expectGetsURL: true, - expectedRequeue: ctrl.Result{Requeue: true}, + expectedRequeue: ctrl.Result{}, }), Entry("restore results request with phase 'New' gets a url", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase(velerov1api.DownloadRequestPhaseNew).Target(velerov1api.DownloadTargetKindRestoreResults, "a-backup-20170912150214").Result(), @@ -236,30 +236,30 @@ var _ = Describe("Download Request Reconciler", func() { backup: defaultBackup(), backupLocation: builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(), expectGetsURL: true, - expectedRequeue: ctrl.Result{Requeue: true}, + expectedRequeue: ctrl.Result{}, }), Entry("request with phase 'Processed' and not expired is not deleted", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase(velerov1api.DownloadRequestPhaseProcessed).Target(velerov1api.DownloadTargetKindBackupLog, "a-backup-20170912150214").Result(), backup: defaultBackup(), - expectedRequeue: ctrl.Result{Requeue: true}, + expectedRequeue: ctrl.Result{}, }), Entry("request with phase 'Processed' and expired is deleted", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase(velerov1api.DownloadRequestPhaseProcessed).Target(velerov1api.DownloadTargetKindBackupLog, "a-backup-20170912150214").Result(), backup: defaultBackup(), expired: true, - expectedRequeue: ctrl.Result{Requeue: false}, + expectedRequeue: ctrl.Result{}, }), Entry("request with phase '' and expired is deleted", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindBackupLog, "a-backup-20170912150214").Result(), backup: defaultBackup(), expired: true, - expectedRequeue: ctrl.Result{Requeue: false}, + expectedRequeue: ctrl.Result{}, }), Entry("request with phase 'New' and expired is deleted", request{ downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase(velerov1api.DownloadRequestPhaseNew).Target(velerov1api.DownloadTargetKindBackupLog, "a-backup-20170912150214").Result(), backup: defaultBackup(), expired: true, - expectedRequeue: ctrl.Result{Requeue: false}, + expectedRequeue: ctrl.Result{}, }), ) }) From 7b4d4c7275fac765f794863e4d60ee292f0ce9ce Mon Sep 17 00:00:00 2001 From: Xun Jiang Date: Tue, 4 Jul 2023 15:10:34 +0800 Subject: [PATCH 23/28] Limit label "velero.io/pvc-namespace-name" length to 63. 1. Limit label length. 2. Modify UT accordingly. 3. Remove unnecessary const variable. Signed-off-by: Xun Jiang --- pkg/apis/velero/v1/labels_annotations.go | 3 --- pkg/restore/dataupload_retrieve_action.go | 4 ++-- pkg/restore/dataupload_retrieve_action_test.go | 9 ++++++++- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/pkg/apis/velero/v1/labels_annotations.go b/pkg/apis/velero/v1/labels_annotations.go index defd56421..b35cd6c6f 100644 --- a/pkg/apis/velero/v1/labels_annotations.go +++ b/pkg/apis/velero/v1/labels_annotations.go @@ -83,9 +83,6 @@ const ( // The format is /. PVCNamespaceNameLabel = "velero.io/pvc-namespace-name" - // DynamicPVRestoreLabel is the label key for dynamic PV restore - DynamicPVRestoreLabel = "velero.io/dynamic-pv-restore" - // ResourceUsageLabel is the label key to explain the Velero resource usage. ResourceUsageLabel = "velero.io/resource-usage" ) diff --git a/pkg/restore/dataupload_retrieve_action.go b/pkg/restore/dataupload_retrieve_action.go index f42837310..a691d653c 100644 --- a/pkg/restore/dataupload_retrieve_action.go +++ b/pkg/restore/dataupload_retrieve_action.go @@ -84,8 +84,8 @@ func (d *DataUploadRetrieveAction) Execute(input *velero.RestoreItemActionExecut Namespace: dataUpload.Namespace, Labels: map[string]string{ velerov1api.RestoreUIDLabel: label.GetValidName(string(input.Restore.UID)), - velerov1api.PVCNamespaceNameLabel: dataUpload.Spec.SourceNamespace + "." + dataUpload.Spec.SourcePVC, - velerov1api.ResourceUsageLabel: string(velerov1api.VeleroResourceUsageDataUploadResult), + velerov1api.PVCNamespaceNameLabel: label.GetValidName(dataUpload.Spec.SourceNamespace + "." + dataUpload.Spec.SourcePVC), + velerov1api.ResourceUsageLabel: label.GetValidName(string(velerov1api.VeleroResourceUsageDataUploadResult)), }, }, Data: map[string]string{ diff --git a/pkg/restore/dataupload_retrieve_action_test.go b/pkg/restore/dataupload_retrieve_action_test.go index e04050e6d..be8c65368 100644 --- a/pkg/restore/dataupload_retrieve_action_test.go +++ b/pkg/restore/dataupload_retrieve_action_test.go @@ -31,6 +31,7 @@ import ( velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" velerov2alpha1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1" "github.com/vmware-tanzu/velero/pkg/builder" + "github.com/vmware-tanzu/velero/pkg/label" "github.com/vmware-tanzu/velero/pkg/plugin/velero" velerotest "github.com/vmware-tanzu/velero/pkg/test" ) @@ -49,6 +50,12 @@ func TestDataUploadRetrieveActionExectue(t *testing.T) { restore: builder.ForRestore("velero", "testRestore").ObjectMeta(builder.WithUID("testingUID")).Result(), expectedDataUploadResult: builder.ForConfigMap("velero", "").ObjectMeta(builder.WithGenerateName("testDU-"), builder.WithLabels(velerov1.PVCNamespaceNameLabel, "testNamespace.testPVC", velerov1.RestoreUIDLabel, "testingUID", velerov1.ResourceUsageLabel, string(velerov1.VeleroResourceUsageDataUploadResult))).Data("testingUID", `{"backupStorageLocation":"","sourceNamespace":"testNamespace"}`).Result(), }, + { + name: "Long source namespace and PVC name should also work", + dataUpload: builder.ForDataUpload("velero", "testDU").SourceNamespace("migre209d0da-49c7-45ba-8d5a-3e59fd591ec1").SourcePVC("kibishii-data-kibishii-deployment-0").Result(), + restore: builder.ForRestore("velero", "testRestore").ObjectMeta(builder.WithUID("testingUID")).Result(), + expectedDataUploadResult: builder.ForConfigMap("velero", "").ObjectMeta(builder.WithGenerateName("testDU-"), builder.WithLabels(velerov1.PVCNamespaceNameLabel, "migre209d0da-49c7-45ba-8d5a-3e59fd591ec1.kibishii-data-ki152333", velerov1.RestoreUIDLabel, "testingUID", velerov1.ResourceUsageLabel, string(velerov1.VeleroResourceUsageDataUploadResult))).Data("testingUID", `{"backupStorageLocation":"","sourceNamespace":"migre209d0da-49c7-45ba-8d5a-3e59fd591ec1"}`).Result(), + }, } for _, tc := range tests { @@ -76,7 +83,7 @@ func TestDataUploadRetrieveActionExectue(t *testing.T) { if tc.expectedDataUploadResult != nil { cmList, err := cmClient.CoreV1().ConfigMaps("velero").List(context.Background(), metav1.ListOptions{ - LabelSelector: fmt.Sprintf("%s=%s,%s=%s", velerov1.RestoreUIDLabel, "testingUID", velerov1.PVCNamespaceNameLabel, tc.dataUpload.Spec.SourceNamespace+"."+tc.dataUpload.Spec.SourcePVC), + LabelSelector: fmt.Sprintf("%s=%s,%s=%s", velerov1.RestoreUIDLabel, "testingUID", velerov1.PVCNamespaceNameLabel, label.GetValidName(tc.dataUpload.Spec.SourceNamespace+"."+tc.dataUpload.Spec.SourcePVC)), }) require.NoError(t, err) // debug From 2f667f519191850e452ce4856f35d7d7a16fcecf Mon Sep 17 00:00:00 2001 From: Ming Qiu Date: Tue, 4 Jul 2023 16:32:56 +0800 Subject: [PATCH 24/28] Add data download controller UT Signed-off-by: Ming Qiu --- pkg/controller/data_download_controller.go | 4 +- .../data_download_controller_test.go | 412 +++++++++++++++++- 2 files changed, 392 insertions(+), 24 deletions(-) diff --git a/pkg/controller/data_download_controller.go b/pkg/controller/data_download_controller.go index 95531cb6b..4750cb0ed 100644 --- a/pkg/controller/data_download_controller.go +++ b/pkg/controller/data_download_controller.go @@ -431,7 +431,9 @@ func prepareDataDownload(ssb *velerov2alpha1api.DataDownload) { } func (r *DataDownloadReconciler) errorOut(ctx context.Context, dd *velerov2alpha1api.DataDownload, err error, msg string, log logrus.FieldLogger) (ctrl.Result, error) { - r.restoreExposer.CleanUp(ctx, getDataDownloadOwnerObject(dd)) + if r.restoreExposer != nil { + r.restoreExposer.CleanUp(ctx, getDataDownloadOwnerObject(dd)) + } return ctrl.Result{}, r.updateStatusToFailed(ctx, dd, err, msg, log) } diff --git a/pkg/controller/data_download_controller_test.go b/pkg/controller/data_download_controller_test.go index 7ccf20e4e..773112207 100644 --- a/pkg/controller/data_download_controller_test.go +++ b/pkg/controller/data_download_controller_test.go @@ -20,6 +20,7 @@ import ( "context" "fmt" "testing" + "time" "github.com/pkg/errors" "github.com/sirupsen/logrus" @@ -27,23 +28,26 @@ import ( "github.com/stretchr/testify/mock" "github.com/stretchr/testify/require" corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" clientgofake "k8s.io/client-go/kubernetes/fake" ctrl "sigs.k8s.io/controller-runtime" kbclient "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/reconcile" + + "sigs.k8s.io/controller-runtime/pkg/client/fake" "github.com/vmware-tanzu/velero/internal/credentials" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" velerov2alpha1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1" "github.com/vmware-tanzu/velero/pkg/builder" "github.com/vmware-tanzu/velero/pkg/datapath" + datapathmockes "github.com/vmware-tanzu/velero/pkg/datapath/mocks" "github.com/vmware-tanzu/velero/pkg/exposer" velerotest "github.com/vmware-tanzu/velero/pkg/test" + "github.com/vmware-tanzu/velero/pkg/uploader" - "sigs.k8s.io/controller-runtime/pkg/client/fake" - - datapathmockes "github.com/vmware-tanzu/velero/pkg/datapath/mocks" exposermockes "github.com/vmware-tanzu/velero/pkg/exposer/mocks" ) @@ -85,10 +89,15 @@ func initDataDownloadReconciler(objects []runtime.Object, needError ...bool) (*D fakeClient.updateError = needError[2] fakeClient.patchError = needError[3] } + var fakeKubeClient *clientgofake.Clientset + if len(objects) != 0 { + fakeKubeClient = clientgofake.NewSimpleClientset(objects...) + } else { + fakeKubeClient = clientgofake.NewSimpleClientset() + } - fakeKubeClient := clientgofake.NewSimpleClientset(objects...) fakeFS := velerotest.NewFakeFileSystem() - pathGlob := fmt.Sprintf("/host_pods/%s/volumes/*/%s", "", dataDownloadName) + pathGlob := fmt.Sprintf("/host_pods/%s/volumes/*/%s", "test-uid", "test-pvc") _, err = fakeFS.Create(pathGlob) if err != nil { return nil, err @@ -113,10 +122,106 @@ func TestDataDownloadReconcile(t *testing.T) { targetPVC *corev1.PersistentVolumeClaim dataMgr *datapath.Manager needErrs []bool + needCreateFSBR bool isExposeErr bool isGetExposeErr bool + isNilExposer bool + isFSBRInitErr bool + isFSBRRestoreErr bool + notNilExpose bool + notMockCleanUp bool + mockCancel bool + mockClose bool expectedStatusMsg string + expectedResult *ctrl.Result }{ + { + name: "Unknown data download status", + dd: dataDownloadBuilder().Phase("Unknown").Cancel(true).Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + }, + { + name: "Cancel data downloand in progress and patch data download error", + dd: dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhaseInProgress).Cancel(true).Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + needErrs: []bool{false, false, false, true}, + needCreateFSBR: true, + expectedStatusMsg: "Patch error", + }, + { + name: "Cancel data downloand in progress with empty FSBR", + dd: dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhaseInProgress).Cancel(true).Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + mockCancel: true, + }, + { + name: "Cancel data downloand in progress", + dd: dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhaseInProgress).Cancel(true).Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + needCreateFSBR: true, + mockCancel: true, + }, + { + name: "Error in data path is concurrent limited", + dd: dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhasePrepared).Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + dataMgr: datapath.NewManager(0), + notNilExpose: true, + notMockCleanUp: true, + expectedResult: &ctrl.Result{Requeue: true, RequeueAfter: time.Minute}, + }, + { + name: "Error getting volume directory name for pvc in pod", + dd: dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhasePrepared).Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + notNilExpose: true, + mockClose: true, + expectedStatusMsg: "error identifying unique volume path on host", + }, + { + name: "Unable to update status to in progress for data download", + dd: dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhasePrepared).Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + needErrs: []bool{false, false, false, true}, + notNilExpose: true, + notMockCleanUp: true, + expectedStatusMsg: "Patch error", + }, + { + name: "accept DataDownload error", + dd: dataDownloadBuilder().Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + needErrs: []bool{false, false, true, false}, + expectedStatusMsg: "Update error", + }, + { + name: "Not create target pvc", + dd: dataDownloadBuilder().Result(), + }, + { + name: "Uninitialized dataDownload", + dd: dataDownloadBuilder().Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + isNilExposer: true, + expectedStatusMsg: "uninitialized generic exposer", + }, + { + name: "DataDownload not created in velero default namespace", + dd: builder.ForDataDownload("test-ns", dataDownloadName).Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + }, + { + name: "Failed to get dataDownload", + dd: builder.ForDataDownload("test-ns", dataDownloadName).Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + needErrs: []bool{true, false, false, false}, + expectedStatusMsg: "Create error", + }, + { + name: "Unsupported dataDownload type", + dd: dataDownloadBuilder().DataMover("Unsuppoorted type").Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + }, { name: "Restore is exposed", dd: dataDownloadBuilder().Result(), @@ -134,11 +239,22 @@ func TestDataDownloadReconcile(t *testing.T) { expectedStatusMsg: "Error to get restore exposer", isGetExposeErr: true, }, + { + name: "Error to start restore expose", + dd: dataDownloadBuilder().Result(), + targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(), + expectedStatusMsg: "Error to expose restore exposer", + isExposeErr: true, + }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { - r, err := initDataDownloadReconciler([]runtime.Object{test.targetPVC}, test.needErrs...) + var objs []runtime.Object + if test.targetPVC != nil { + objs = []runtime.Object{test.targetPVC} + } + r, err := initDataDownloadReconciler(objs, test.needErrs...) require.NoError(t, err) defer func() { r.client.Delete(ctx, test.dd, &kbclient.DeleteOptions{}) @@ -160,31 +276,49 @@ func TestDataDownloadReconcile(t *testing.T) { } datapath.FSBRCreator = func(string, string, kbclient.Client, string, datapath.Callbacks, logrus.FieldLogger) datapath.AsyncBR { - return datapathmockes.NewAsyncBR(t) + fsBR := datapathmockes.NewAsyncBR(t) + if test.mockCancel { + fsBR.On("Cancel").Return() + } + + if test.mockClose { + fsBR.On("Close", mock.Anything).Return() + } + + return fsBR } - if test.isExposeErr || test.isGetExposeErr { - r.restoreExposer = func() exposer.GenericRestoreExposer { - ep := exposermockes.NewGenericRestoreExposer(t) - if test.isExposeErr { - ep.On("Expose", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(errors.New("Error to expose restore exposer")) - } + if test.isExposeErr || test.isGetExposeErr || test.isNilExposer || test.notNilExpose { + if test.isNilExposer { + r.restoreExposer = nil + } else { + r.restoreExposer = func() exposer.GenericRestoreExposer { + ep := exposermockes.NewGenericRestoreExposer(t) + if test.isExposeErr { + ep.On("Expose", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(errors.New("Error to expose restore exposer")) + } else if test.notNilExpose { + hostingPod := builder.ForPod("test-ns", "test-name").Volumes(&corev1.Volume{Name: "test-pvc"}).Result() + hostingPod.ObjectMeta.SetUID("test-uid") + ep.On("GetExposed", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(&exposer.ExposeResult{ByPod: exposer.ExposeByPod{HostingPod: hostingPod, PVC: "test-pvc"}}, nil) + } else if test.isGetExposeErr { + ep.On("GetExposed", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(nil, errors.New("Error to get restore exposer")) + } - if test.isGetExposeErr { - ep.On("GetExposed", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(nil, errors.New("Error to get restore exposer")) - } - - ep.On("CleanUp", mock.Anything, mock.Anything).Return() - return ep - }() + if !test.notMockCleanUp { + ep.On("CleanUp", mock.Anything, mock.Anything).Return() + } + return ep + }() + } } - if test.dd.Status.Phase == velerov2alpha1api.DataDownloadPhaseInProgress { + if test.needCreateFSBR { if fsBR := r.dataPathMgr.GetAsyncBR(test.dd.Name); fsBR == nil { _, err := r.dataPathMgr.CreateFileSystemBR(test.dd.Name, pVBRRequestor, ctx, r.client, velerov1api.DefaultNamespace, datapath.Callbacks{OnCancelled: r.OnDataDownloadCancelled}, velerotest.NewLogger()) require.NoError(t, err) } } + actualResult, err := r.Reconcile(ctx, ctrl.Request{ NamespacedName: types.NamespacedName{ Namespace: velerov1api.DefaultNamespace, @@ -192,7 +326,7 @@ func TestDataDownloadReconcile(t *testing.T) { }, }) - if test.isGetExposeErr { + if test.expectedStatusMsg != "" { assert.Contains(t, err.Error(), test.expectedStatusMsg) } else { require.Nil(t, err) @@ -200,6 +334,11 @@ func TestDataDownloadReconcile(t *testing.T) { require.NotNil(t, actualResult) + if test.expectedResult != nil { + assert.Equal(t, test.expectedResult.Requeue, test.expectedResult.Requeue) + assert.Equal(t, test.expectedResult.RequeueAfter, test.expectedResult.RequeueAfter) + } + dd := velerov2alpha1api.DataDownload{} err = r.client.Get(ctx, kbclient.ObjectKey{ Name: test.dd.Name, @@ -209,8 +348,235 @@ func TestDataDownloadReconcile(t *testing.T) { if test.isGetExposeErr { assert.Contains(t, dd.Status.Message, test.expectedStatusMsg) } - require.Nil(t, err) + if test.dd.Namespace == velerov1api.DefaultNamespace { + require.Nil(t, err) + } else { + assert.True(t, true, apierrors.IsNotFound(err)) + } + t.Logf("%s: \n %v \n", test.name, dd) }) } } + +func TestOnDataDownloadFailed(t *testing.T) { + for _, getErr := range []bool{true, false} { + ctx := context.TODO() + needErrs := []bool{getErr, false, false, false} + r, err := initDataDownloadReconciler(nil, needErrs...) + require.NoError(t, err) + + dd := dataDownloadBuilder().Result() + namespace := dd.Namespace + ddName := dd.Name + // Add the DataDownload object to the fake client + assert.NoError(t, r.client.Create(ctx, dd)) + r.OnDataDownloadFailed(ctx, namespace, ddName, fmt.Errorf("Failed to handle %v", ddName)) + updatedDD := &velerov2alpha1api.DataDownload{} + if getErr { + assert.Error(t, r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, updatedDD)) + assert.NotEqual(t, velerov2alpha1api.DataDownloadPhaseFailed, updatedDD.Status.Phase) + assert.Equal(t, updatedDD.Status.StartTimestamp.IsZero(), true) + } else { + assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, updatedDD)) + assert.Equal(t, velerov2alpha1api.DataDownloadPhaseFailed, updatedDD.Status.Phase) + assert.Equal(t, updatedDD.Status.StartTimestamp.IsZero(), true) + } + } +} + +func TestOnDataDownloadCancelled(t *testing.T) { + for _, getErr := range []bool{true, false} { + ctx := context.TODO() + needErrs := []bool{getErr, false, false, false} + r, err := initDataDownloadReconciler(nil, needErrs...) + require.NoError(t, err) + + dd := dataDownloadBuilder().Result() + namespace := dd.Namespace + ddName := dd.Name + // Add the DataDownload object to the fake client + assert.NoError(t, r.client.Create(ctx, dd)) + r.OnDataDownloadCancelled(ctx, namespace, ddName) + updatedDD := &velerov2alpha1api.DataDownload{} + if getErr { + assert.Error(t, r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, updatedDD)) + assert.NotEqual(t, velerov2alpha1api.DataDownloadPhaseFailed, updatedDD.Status.Phase) + assert.Equal(t, updatedDD.Status.StartTimestamp.IsZero(), true) + } else { + assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, updatedDD)) + assert.Equal(t, velerov2alpha1api.DataDownloadPhaseCanceled, updatedDD.Status.Phase) + assert.Equal(t, updatedDD.Status.StartTimestamp.IsZero(), false) + assert.Equal(t, updatedDD.Status.CompletionTimestamp.IsZero(), false) + } + } +} + +func TestOnDataDownloadCompleted(t *testing.T) { + tests := []struct { + name string + emptyFSBR bool + isGetErr bool + rebindVolumeErr bool + }{ + { + name: "Data download complete", + emptyFSBR: false, + isGetErr: false, + rebindVolumeErr: false, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + ctx := context.TODO() + needErrs := []bool{test.isGetErr, false, false, false} + r, err := initDataDownloadReconciler(nil, needErrs...) + r.restoreExposer = func() exposer.GenericRestoreExposer { + ep := exposermockes.NewGenericRestoreExposer(t) + if test.rebindVolumeErr { + ep.On("RebindVolume", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(errors.New("Error to rebind volume")) + + } else { + ep.On("RebindVolume", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(nil) + } + ep.On("CleanUp", mock.Anything, mock.Anything).Return() + return ep + }() + + require.NoError(t, err) + dd := dataDownloadBuilder().Result() + namespace := dd.Namespace + ddName := dd.Name + // Add the DataDownload object to the fake client + assert.NoError(t, r.client.Create(ctx, dd)) + r.OnDataDownloadCompleted(ctx, namespace, ddName, datapath.Result{}) + updatedDD := &velerov2alpha1api.DataDownload{} + if test.isGetErr { + assert.Error(t, r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, updatedDD)) + assert.Equal(t, velerov2alpha1api.DataDownloadPhase(""), updatedDD.Status.Phase) + assert.Equal(t, updatedDD.Status.CompletionTimestamp.IsZero(), true) + } else { + assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, updatedDD)) + assert.Equal(t, velerov2alpha1api.DataDownloadPhaseCompleted, updatedDD.Status.Phase) + assert.Equal(t, updatedDD.Status.CompletionTimestamp.IsZero(), false) + } + }) + } +} + +func TestOnDataDownloadProgress(t *testing.T) { + totalBytes := int64(1024) + bytesDone := int64(512) + tests := []struct { + name string + dd *velerov2alpha1api.DataDownload + progress uploader.Progress + needErrs []bool + }{ + { + name: "patch in progress phase success", + dd: dataDownloadBuilder().Result(), + progress: uploader.Progress{ + TotalBytes: totalBytes, + BytesDone: bytesDone, + }, + }, + { + name: "failed to get datadownload", + dd: dataDownloadBuilder().Result(), + needErrs: []bool{true, false, false, false}, + }, + { + name: "failed to patch datadownload", + dd: dataDownloadBuilder().Result(), + needErrs: []bool{false, false, false, true}, + }, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + ctx := context.TODO() + + r, err := initDataDownloadReconciler(nil, test.needErrs...) + require.NoError(t, err) + defer func() { + r.client.Delete(ctx, test.dd, &kbclient.DeleteOptions{}) + }() + // Create a DataDownload object + dd := dataDownloadBuilder().Result() + namespace := dd.Namespace + duName := dd.Name + // Add the DataDownload object to the fake client + assert.NoError(t, r.client.Create(context.Background(), dd)) + + // Create a Progress object + progress := &uploader.Progress{ + TotalBytes: totalBytes, + BytesDone: bytesDone, + } + + // Call the OnDataDownloadProgress function + r.OnDataDownloadProgress(ctx, namespace, duName, progress) + if len(test.needErrs) != 0 && !test.needErrs[0] { + // Get the updated DataDownload object from the fake client + updatedDu := &velerov2alpha1api.DataDownload{} + assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: duName, Namespace: namespace}, updatedDu)) + // Assert that the DataDownload object has been updated with the progress + assert.Equal(t, test.progress.TotalBytes, updatedDu.Status.Progress.TotalBytes) + assert.Equal(t, test.progress.BytesDone, updatedDu.Status.Progress.BytesDone) + } + }) + } +} + +func TestFindDataDownloadForPod(t *testing.T) { + needErrs := []bool{false, false, false, false} + r, err := initDataDownloadReconciler(nil, needErrs...) + require.NoError(t, err) + tests := []struct { + name string + du *velerov2alpha1api.DataDownload + pod *corev1.Pod + checkFunc func(*velerov2alpha1api.DataDownload, []reconcile.Request) + }{ + { + name: "find dataDownload for pod", + du: dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhaseAccepted).Result(), + pod: builder.ForPod(velerov1api.DefaultNamespace, dataDownloadName).Labels(map[string]string{velerov1api.DataDownloadLabel: dataDownloadName}).Result(), + checkFunc: func(du *velerov2alpha1api.DataDownload, requests []reconcile.Request) { + // Assert that the function returns a single request + assert.Len(t, requests, 1) + // Assert that the request contains the correct namespaced name + assert.Equal(t, du.Namespace, requests[0].Namespace) + assert.Equal(t, du.Name, requests[0].Name) + }, + }, { + name: "no matched pod", + du: dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhaseAccepted).Result(), + pod: builder.ForPod(velerov1api.DefaultNamespace, dataDownloadName).Labels(map[string]string{velerov1api.DataDownloadLabel: "non-existing-datadownload"}).Result(), + checkFunc: func(du *velerov2alpha1api.DataDownload, requests []reconcile.Request) { + assert.Empty(t, requests) + }, + }, + { + name: "dataDownload not accepte", + du: dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhaseInProgress).Result(), + pod: builder.ForPod(velerov1api.DefaultNamespace, dataDownloadName).Labels(map[string]string{velerov1api.DataDownloadLabel: dataDownloadName}).Result(), + checkFunc: func(du *velerov2alpha1api.DataDownload, requests []reconcile.Request) { + assert.Empty(t, requests) + }, + }, + } + for _, test := range tests { + ctx := context.Background() + assert.NoError(t, r.client.Create(ctx, test.pod)) + assert.NoError(t, r.client.Create(ctx, test.du)) + // Call the findSnapshotRestoreForPod function + requests := r.findSnapshotRestoreForPod(test.pod) + test.checkFunc(test.du, requests) + r.client.Delete(ctx, test.du, &kbclient.DeleteOptions{}) + if test.pod != nil { + r.client.Delete(ctx, test.pod, &kbclient.DeleteOptions{}) + } + } +} From d7f1ea4fbd5ccdbf4327f855e461bca9caa89507 Mon Sep 17 00:00:00 2001 From: Xun Jiang Date: Fri, 30 Jun 2023 17:53:27 +0800 Subject: [PATCH 25/28] Add exit code log and possible memory shortage warning log for Restic command failure. Signed-off-by: Xun Jiang --- changelogs/unreleased/6459-blackpiglet | 1 + pkg/repository/restic/repository.go | 2 +- pkg/restic/exec_commands.go | 4 +++- pkg/util/exec/exec.go | 23 +++++++++++++++++++++++ 4 files changed, 28 insertions(+), 2 deletions(-) create mode 100644 changelogs/unreleased/6459-blackpiglet diff --git a/changelogs/unreleased/6459-blackpiglet b/changelogs/unreleased/6459-blackpiglet new file mode 100644 index 000000000..26e0e3856 --- /dev/null +++ b/changelogs/unreleased/6459-blackpiglet @@ -0,0 +1 @@ +Add exit code log and possible memory shortage warning log for Restic command failure. \ No newline at end of file diff --git a/pkg/repository/restic/repository.go b/pkg/repository/restic/repository.go index 392caf284..3c15e0b37 100644 --- a/pkg/repository/restic/repository.go +++ b/pkg/repository/restic/repository.go @@ -112,7 +112,7 @@ func (r *RepositoryService) exec(cmd *restic.Command, bsl *velerov1api.BackupSto cmd.ExtraFlags = append(cmd.ExtraFlags, skipTLSRet) } - stdout, stderr, err := veleroexec.RunCommand(cmd.Cmd()) + stdout, stderr, err := veleroexec.RunCommandWithLog(cmd.Cmd(), r.log) r.log.WithFields(logrus.Fields{ "repository": cmd.RepoName(), "command": cmd.String(), diff --git a/pkg/restic/exec_commands.go b/pkg/restic/exec_commands.go index 0cbc42802..94c17c04a 100644 --- a/pkg/restic/exec_commands.go +++ b/pkg/restic/exec_commands.go @@ -86,6 +86,7 @@ func RunBackup(backupCmd *Command, log logrus.FieldLogger, updater uploader.Prog err := cmd.Start() if err != nil { + exec.LogErrorAsExitCode(err, log) return stdoutBuf.String(), stderrBuf.String(), err } @@ -119,6 +120,7 @@ func RunBackup(backupCmd *Command, log logrus.FieldLogger, updater uploader.Prog err = cmd.Wait() if err != nil { + exec.LogErrorAsExitCode(err, log) return stdoutBuf.String(), stderrBuf.String(), err } quit <- struct{}{} @@ -229,7 +231,7 @@ func RunRestore(restoreCmd *Command, log logrus.FieldLogger, updater uploader.Pr } }() - stdout, stderr, err := exec.RunCommand(restoreCmd.Cmd()) + stdout, stderr, err := exec.RunCommandWithLog(restoreCmd.Cmd(), log) quit <- struct{}{} // update progress to 100% diff --git a/pkg/util/exec/exec.go b/pkg/util/exec/exec.go index 84bffb257..109118d58 100644 --- a/pkg/util/exec/exec.go +++ b/pkg/util/exec/exec.go @@ -22,6 +22,7 @@ import ( "os/exec" "github.com/pkg/errors" + "github.com/sirupsen/logrus" ) // RunCommand runs a command and returns its stdout, stderr, and its returned @@ -52,3 +53,25 @@ func RunCommand(cmd *exec.Cmd) (string, string, error) { return stdout, stderr, runErr } + +func RunCommandWithLog(cmd *exec.Cmd, log logrus.FieldLogger) (string, string, error) { + stdout, stderr, err := RunCommand(cmd) + LogErrorAsExitCode(err, log) + return stdout, stderr, err +} + +func LogErrorAsExitCode(err error, log logrus.FieldLogger) { + if err != nil { + if exitError, ok := err.(*exec.ExitError); ok { + log.Errorf("Restic command fail with ExitCode: %d. Process ID is %d, Exit error is: %s", exitError.ExitCode(), exitError.Pid(), exitError.String()) + // Golang's os.exec -1 ExitCode means signal kill. Usually this is caused + // by CGroup's OOM. Log a warning to notice user. + // https://github.com/golang/go/blob/master/src/os/exec_posix.go#L128-L136 + if exitError.ExitCode() == -1 { + log.Warnf("The ExitCode is -1, which means the process is terminated by signal. Usually this is caused by CGroup kill due to out of memory. Please check whether there is such information in the work nodes' dmesg log.") + } + } else { + log.WithError(err).Info("Error cannot be convert to ExitError format.") + } + } +} From e71ee0cc5f1d9ed1b8354a91914101b1b74a85e7 Mon Sep 17 00:00:00 2001 From: kayrus Date: Wed, 5 Jul 2023 17:23:16 +0200 Subject: [PATCH 26/28] Add support for OpenStack CSI drivers topology keys Signed-off-by: kayrus --- changelogs/unreleased/6464-openstack-csi-topology-keys | 1 + pkg/backup/item_backupper.go | 6 +++++- 2 files changed, 6 insertions(+), 1 deletion(-) create mode 100644 changelogs/unreleased/6464-openstack-csi-topology-keys diff --git a/changelogs/unreleased/6464-openstack-csi-topology-keys b/changelogs/unreleased/6464-openstack-csi-topology-keys new file mode 100644 index 000000000..d4c3c976c --- /dev/null +++ b/changelogs/unreleased/6464-openstack-csi-topology-keys @@ -0,0 +1 @@ +Add support for OpenStack CSI drivers topology keys diff --git a/pkg/backup/item_backupper.go b/pkg/backup/item_backupper.go index a9a3ae440..d6cd5ba6b 100644 --- a/pkg/backup/item_backupper.go +++ b/pkg/backup/item_backupper.go @@ -445,6 +445,10 @@ const ( azureCsiZoneKey = "topology.disk.csi.azure.com/zone" gkeCsiZoneKey = "topology.gke.io/zone" gkeZoneSeparator = "__" + + // OpenStack CSI drivers topology keys + cinderCsiZoneKey = "topology.manila.csi.openstack.org/zone" + manilaCsiZoneKey = "topology.cinder.csi.openstack.org/zone" ) // takePVSnapshot triggers a snapshot for the volume/disk underlying a PersistentVolume if the provided @@ -506,7 +510,7 @@ func (ib *itemBackupper) takePVSnapshot(obj runtime.Unstructured, log logrus.Fie if !labelFound { var k string log.Infof("label %q is not present on PersistentVolume", zoneLabelDeprecated) - k, pvFailureDomainZone = zoneFromPVNodeAffinity(pv, awsEbsCsiZoneKey, azureCsiZoneKey, gkeCsiZoneKey, zoneLabel, zoneLabelDeprecated) + k, pvFailureDomainZone = zoneFromPVNodeAffinity(pv, awsEbsCsiZoneKey, azureCsiZoneKey, gkeCsiZoneKey, cinderCsiZoneKey, manilaCsiZoneKey, zoneLabel, zoneLabelDeprecated) if pvFailureDomainZone != "" { log.Infof("zone info from nodeAffinity requirements: %s, key: %s", pvFailureDomainZone, k) } else { From ff83d5e0c99bb299e718655aa87f50b2cde1bbca Mon Sep 17 00:00:00 2001 From: Tiger Kaovilai Date: Wed, 5 Jul 2023 12:36:07 -0400 Subject: [PATCH 27/28] typo: s/inokes/invokes Signed-off-by: Tiger Kaovilai --- site/content/docs/main/file-system-backup.md | 4 ++-- site/content/docs/v1.10/file-system-backup.md | 4 ++-- site/content/docs/v1.11/file-system-backup.md | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/site/content/docs/main/file-system-backup.md b/site/content/docs/main/file-system-backup.md index 89c43ce7b..be32119a4 100644 --- a/site/content/docs/main/file-system-backup.md +++ b/site/content/docs/main/file-system-backup.md @@ -539,7 +539,7 @@ that it's backing up for the volumes to be backed up using FSB. 5. Meanwhile, each `PodVolumeBackup` is handled by the controller on the appropriate node, which: - has a hostPath volume mount of `/var/lib/kubelet/pods` to access the pod volume data - finds the pod volume's subdirectory within the above volume - - based on the path selection, Velero inokes restic or kopia for backup + - based on the path selection, Velero invokes restic or kopia for backup - updates the status of the custom resource to `Completed` or `Failed` 6. As each `PodVolumeBackup` finishes, the main Velero process adds it to the Velero backup in a file named `-podvolumebackups.json.gz`. This file gets uploaded to object storage alongside the backup tarball. @@ -564,7 +564,7 @@ some reason (i.e. lack of cluster resources), the FSB restore will not be done. - has a hostPath volume mount of `/var/lib/kubelet/pods` to access the pod volume data - waits for the pod to be running the init container - finds the pod volume's subdirectory within the above volume - - based on the path selection, Velero inokes restic or kopia for restore + - based on the path selection, Velero invokes restic or kopia for restore - on success, writes a file into the pod volume, in a `.velero` subdirectory, whose name is the UID of the Velero restore that this pod volume restore is for - updates the status of the custom resource to `Completed` or `Failed` diff --git a/site/content/docs/v1.10/file-system-backup.md b/site/content/docs/v1.10/file-system-backup.md index 6fcd1a33b..96e861cf3 100644 --- a/site/content/docs/v1.10/file-system-backup.md +++ b/site/content/docs/v1.10/file-system-backup.md @@ -539,7 +539,7 @@ that it's backing up for the volumes to be backed up using FSB. 5. Meanwhile, each `PodVolumeBackup` is handled by the controller on the appropriate node, which: - has a hostPath volume mount of `/var/lib/kubelet/pods` to access the pod volume data - finds the pod volume's subdirectory within the above volume - - based on the path selection, Velero inokes restic or kopia for backup + - based on the path selection, Velero invokes restic or kopia for backup - updates the status of the custom resource to `Completed` or `Failed` 6. As each `PodVolumeBackup` finishes, the main Velero process adds it to the Velero backup in a file named `-podvolumebackups.json.gz`. This file gets uploaded to object storage alongside the backup tarball. @@ -564,7 +564,7 @@ some reason (i.e. lack of cluster resources), the FSB restore will not be done. - has a hostPath volume mount of `/var/lib/kubelet/pods` to access the pod volume data - waits for the pod to be running the init container - finds the pod volume's subdirectory within the above volume - - based on the path selection, Velero inokes restic or kopia for restore + - based on the path selection, Velero invokes restic or kopia for restore - on success, writes a file into the pod volume, in a `.velero` subdirectory, whose name is the UID of the Velero restore that this pod volume restore is for - updates the status of the custom resource to `Completed` or `Failed` diff --git a/site/content/docs/v1.11/file-system-backup.md b/site/content/docs/v1.11/file-system-backup.md index 881747895..5022adbcd 100644 --- a/site/content/docs/v1.11/file-system-backup.md +++ b/site/content/docs/v1.11/file-system-backup.md @@ -539,7 +539,7 @@ that it's backing up for the volumes to be backed up using FSB. 5. Meanwhile, each `PodVolumeBackup` is handled by the controller on the appropriate node, which: - has a hostPath volume mount of `/var/lib/kubelet/pods` to access the pod volume data - finds the pod volume's subdirectory within the above volume - - based on the path selection, Velero inokes restic or kopia for backup + - based on the path selection, Velero invokes restic or kopia for backup - updates the status of the custom resource to `Completed` or `Failed` 6. As each `PodVolumeBackup` finishes, the main Velero process adds it to the Velero backup in a file named `-podvolumebackups.json.gz`. This file gets uploaded to object storage alongside the backup tarball. @@ -564,7 +564,7 @@ some reason (i.e. lack of cluster resources), the FSB restore will not be done. - has a hostPath volume mount of `/var/lib/kubelet/pods` to access the pod volume data - waits for the pod to be running the init container - finds the pod volume's subdirectory within the above volume - - based on the path selection, Velero inokes restic or kopia for restore + - based on the path selection, Velero invokes restic or kopia for restore - on success, writes a file into the pod volume, in a `.velero` subdirectory, whose name is the UID of the Velero restore that this pod volume restore is for - updates the status of the custom resource to `Completed` or `Failed` From 8a7aa2051ca3f8caa00e3ac39f3f3e73ca8de9de Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Thu, 6 Jul 2023 12:10:57 +0800 Subject: [PATCH 28/28] add node name data mover CR Signed-off-by: Lyndon-Li --- config/crd/v2alpha1/bases/velero.io_datadownloads.yaml | 7 +++++++ config/crd/v2alpha1/bases/velero.io_datauploads.yaml | 7 +++++++ config/crd/v2alpha1/crds/crds.go | 4 ++-- pkg/apis/velero/v2alpha1/data_download_types.go | 5 +++++ pkg/apis/velero/v2alpha1/data_upload_types.go | 5 +++++ pkg/controller/data_download_controller.go | 5 +++-- pkg/controller/data_upload_controller.go | 5 +++-- 7 files changed, 32 insertions(+), 6 deletions(-) diff --git a/config/crd/v2alpha1/bases/velero.io_datadownloads.yaml b/config/crd/v2alpha1/bases/velero.io_datadownloads.yaml index 20d604153..8389028f7 100644 --- a/config/crd/v2alpha1/bases/velero.io_datadownloads.yaml +++ b/config/crd/v2alpha1/bases/velero.io_datadownloads.yaml @@ -41,6 +41,10 @@ spec: jsonPath: .metadata.creationTimestamp name: Age type: date + - description: Name of the node where the DataDownload is processed + jsonPath: .status.node + name: Node + type: string name: v2alpha1 schema: openAPIV3Schema: @@ -132,6 +136,9 @@ spec: message: description: Message is a message about the DataDownload's status. type: string + node: + description: Node is name of the node where the DataDownload is processed. + type: string phase: description: Phase is the current state of the DataDownload. enum: diff --git a/config/crd/v2alpha1/bases/velero.io_datauploads.yaml b/config/crd/v2alpha1/bases/velero.io_datauploads.yaml index ea354f1d4..ffc279958 100644 --- a/config/crd/v2alpha1/bases/velero.io_datauploads.yaml +++ b/config/crd/v2alpha1/bases/velero.io_datauploads.yaml @@ -42,6 +42,10 @@ spec: jsonPath: .metadata.creationTimestamp name: Age type: date + - description: Name of the node where the DataUpload is processed + jsonPath: .status.node + name: Node + type: string name: v2alpha1 schema: openAPIV3Schema: @@ -147,6 +151,9 @@ spec: message: description: Message is a message about the DataUpload's status. type: string + node: + description: Node is name of the node where the DataUpload is processed. + type: string path: description: Path is the full path of the snapshot volume being backed up. diff --git a/config/crd/v2alpha1/crds/crds.go b/config/crd/v2alpha1/crds/crds.go index 00f9ec3fd..65981a8b6 100644 --- a/config/crd/v2alpha1/crds/crds.go +++ b/config/crd/v2alpha1/crds/crds.go @@ -29,8 +29,8 @@ import ( ) var rawCRDs = [][]byte{ - []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcX\xcdr\xe4\xb8\r\xbe\xfb)P\x93\x83/#yg\x93J\xa5\xfa6\xd3N\xaa\\ٙt\xad\xa7\xfaNI\x90\x9a;\x14\xc9\xf0\xa7\x1d'\x95wO\x81?jI\xcdvۻ\x99\xd5M$\b|\x04\xc0\x0f \xab\xaa\xbaa\x9a\xef\xd1X\xae\xe4\x06\x98\xe6\xf8/\x87\x92\xfel\xfd\xed/\xb6\xe6\xea\xee\xf8\xe1\xe6\x1b\x97\xdd\x06\xb6\xde:5\xfe\x8cVy\xd3\xe2=\xf6\\rǕ\xbc\x19ѱ\x8e9\xb6\xb9\x01`R*\xc7h\xd8\xd2/@\xab\xa43J\b4Հ\xb2\xfe\xe6\x1bl<\x17\x1d\x9a\xa0<\x9b>\xfeP\x7f\xf8\xb1\xfe\xe1\x06@\xb2\x117@\xfa:\xf5$\x85b\x9d\xad\x8f(Ш\x9a\xab\x1b\xab\xb1%ŃQ^o\xe04\x11\x17&\xa3\x11\xf0=s\xec>\xe9\bÂ[\xf7\xf7\xb3\xa9\x9f\xb8uaZ\vo\x98X\xd9\x0e3\x96\xcb\xc1\vf\x96s7\x00\xb6U\x1a7\xf0\x85Lk\xd6\"\x8d\xa5=\x05(\x15\xb0\xae\v^bbg\xb8th\xb6J\xf81{\xa7\x82\x0emk\xb8v\xc1\vsX`\x1dsނ\xf5\xed\x01\x98\x85/\xf8t\xf7 wF\r\x06m\x84\x05\xf0\x8bUr\xc7\xdca\x03u\x14\xaf\xf5\x81YL\xb3ѕ\x8fa\"\r\xb9g\xc2k\x9d\xe1r(!\xf8\xcaG\x84Λ\x10B\xdaw\x8b\xe0\x0e\xdc.\xa1=1K\xf0\x8c\xc3\xee\"\x900O\xea\xacc\xa3^#\x9a-\x8d\x90:\xe6\xb0\x04h\xabF-\xd0a\aͳü\x8d^\x99\x91\xb9\rp\xe9\xfe\xfc\xa7˾HΪ\xc3\xd2{%\x97\x8e\xf9D\xa30\x1b\x8eH(J\x03\x9a\xa2w\x94c\xe2\xb7\x00q\xa4\xe0\xd3l}D\x12\xf5\xceǯB\xa1\x94\x03Ճ; |b\xed7\xaf\xe1\xd1)\xc3\x06\x84\x9fT\x1b\xc3\xf7t@\x83A\xa2\x89\x12\x94\xbd\xc0)v\xca\x14C\xa7\xb1\xad\xa3lR\x96u\xad\xe2\xb74\xf4\x7fϭ\xd6 +\xe6V\xa6\x9a:Hp%\xcb\t\xf6q\xc0rr\xc5\xe9\xe3\x8fL\xe8\x03\xfb\x10\xcfv{\xc0\x91m\x92\xbc\xd2(?\xee\x1e\xf6\x7f|\\\f\x03h\xa34\x1a\xc73\xc5\xc4oƞ\xb3QX\xee\xfb\x96\x14F)\xe8\x886ц\xa0$\xa2\xc0.a\x88\xe1\xe4\x16\fj\x83\x16\xa5\x9b{7\x7f\xaa\a&A5\xbf`\xebjxDCj\xc0\x1e\x94\x17\x1d\xb1\xed\x11\x8d\x03\x83\xad\x1a$\xff\xf7\xa4ۂS\xc1\xa8`\x0e\x13ߝ\xbe@L\x92\t82\xe1\xf1=0\xd9\xc1Ȟ\xc1 Y\x01/g\xfa\x82\x88\xad\xe1\xb32\b\\\xf6j\x03\a\xe7\xb4\xdd\xdc\xdd\r\xdc\xe5\xaaѪq\xf4\x92\xbb\xe7\xbbP\x00x\xe3\x9d2\xf6\xae\xc3#\x8a;ˇ\x8a\x99\xf6\xc0\x1d\xb6\xce\x1b\xbcc\x9aW\x01\xba\f\x95\xa3\x1e\xbb?\x98Tg\xec\xed\x02\xebY\x8e\xc5/\x10\xfe\v\x11 ֧\xc4gii\xdc\xc5\xc9\xd14D\xde\xf9\xf9\xaf\x8f_!\x9b\x0e\xc1X{?\xf8\xfd\xb4ОB@\x0e\xe3\xb2G\x13\x83\xd8\x1b5\x06\x9d(;\xad\xb8t\xe1\xa7\x15\x1c\xe5\xda\xfd\xd67#w\x14\xf7\x7fz\xb4\x8ebU\xc36\x94Rh\x10\xbc\xa6\xfc\xedjx\x90\xb0e#\x8a-\xb3\xf8\xdd\x03@\x9e\xb6\x159\xf6u!\x98w\x01k\xe1\xe8\xb5\xd9D.\xe3\x17\xe25'\x84G\x8d-\x85\x8e\xbcG\xcbx\xcf\x13\xbd\xf5\xca\x00[\xc8\xd6\v\x95\xe5#K_\x91\xe2\xd6B+L\x9fJk209c\xe3ĵ6J\x9e)\x05\x10\x17\xf9٠V\x96;e\x9eO,]\x9fi\xb8\x10\x00\xfaZ&[\x14Wv\xb2\rB\xc0eG\x9e\xc4)\xef\x88\"\xa2\x82\x80I\xc9Aѹ\xb8\xec\xe0\xf8=8ZE\x89j\xd1ўdX\xbe un\x81K8\xb5/0oS\xd6;k\x94\x12\xc8ּG\xb9\xf5Y\x1d\xa9\x81\x92=\x1f\xce\xf78\xef\xb4.\x05\xfe\x8a\xfb\ni83I\xbb\xa0\x9c#$\xd5H\xe3UNH\"ޞ\x0f\xa9\xb6\x15\x8c\xf6\x1cEg/\xc5\xf2\xec|\xe4\r\a+W\xc29\xa1\xcc\xc7#\x95\x97P\xec\x83\x02\n,\xf1\x88\r]\x14M\x16\x10\xc6\x14\xacᡟi\xe4\x16\u07bd\x03e\xe0]\xec\xb4߽\x8f\xe9\xea\xb9p\x15\x973\x1b\x05\x8dO\\\x88l\xf7MYLћ\xaa\xbb\xf2\xee\x8a\x03\xfe\xb1\x12_\xf9\xc1Q\xdb\x11\xf6\xee\x14<1\xee\xa6rW\xc0<\x99\xb6\xef\xa1\xc1\x9e(֠\xf3F\xd2I@c\x88rlP\xa9\xbc{Ӧ\xacd\xda\x1e\x94{\xb8\xbf\xb2\x9d\xc7I0\xb3\xcb\xc3}\xe6\x96}\x88\xc2D1I\x12\x9c*\x05\x94\xa0G\x0e\t\xc5\xe8mhC\x05\x9c\xee5\xd7 /\xa53ne\xf8\xc0\xa9\xad\x90\xd3̉\xf2\x8et\x0f*%\"\xb7a\x7f\u0601\xd7\x118Q\fU\xd7\x06\xa1\xe3}\x8f\x06\xa5\x8b\xf55\x1a\xde\xed\xb7\xb7\xf6d\xa4\xa4\xb3\x9fa\b\x1d\xd6ȴƎ\xdaQ\x8alrԛ\\\xe4\x98\x19\xd0\xed\xc36\xae\xf8\xe7\xebL4;\x87*7\xdd\x1d\xa8\x10\xa4\xe8F\x8d\xb0\xdbo\xa9\x03+lc\xb7?Gx\xb9\xcaAjx/D\xf0\f\xe5Y\xfc\x12\x9e\x97\x1c{\x85N\x01\xf4\xf1\x15\x96w\xfbR!\x9d\xdc\x01\xee\xc0\x1cI\xa4{\x014\xcfE\x9d\x90\xcfG\n\xe7\xaf\xc3۾\n\xf0\xf6E\xc4\xdb5\xe4\vx\x9b\xe7\xdf\f\x99\x8a77\u061d\xa3\xae^\x88\\\x05\xfaX\x1cl__\xa2ʖ\xabrw\xb5\x92YS\xfcj\xfaD\x96\xeb\x89%Ӭf\xe7G\xf2Umh\xb8\x9e\xbf\xb6\x11\x8d\xef1)\xec\xad7\x81\x86\xd2+\r\xdd\xca~U+\xda\xc6\xf7\x8d\xf9U\xf6Z\xfbv\xbe\"\xdc\xf7L7\xabw,'T\xbcO\xe7G\x94R\xffv\xd2\x17\x97\x06z$u\xd8\x01\x1eQ\x02\xb5ڌ\v\xec\xb2N[\xc3W\xea\xc6\xc3\xc5\xe7v}E\n\xfeN\x8aB٥\x9e\xa9\x00\xfa|]~L\xa1\xebNE*\xce$\xa4\x17\x825\x027\xe0\x8c\xbf\xd4?\x16\x0fʈֲ\xe1\x1aQ\x7f\x8eR\U0006a616\x00k\xa8\xa9X\xf7\xb4\xb76\xc5\xfeME#\xf4\xbdW@\xecH\xa6\x94g\x13ϼܒ\xa3\xf4c\x89\r\xbe\xe0Sa\xf4cۢ.1T\x05;\x83\x9a\x99\xe2\xd4\xd9K\xe4|2^0Jd\x95\xe7\x8a:\xa7\xa7\xbe\xc2\xdc\xdfB\x02\xbe\xc9\xd3\t\xdf5g\xe7\x1b\xc9A\x89|\x80\xc2k\x9c\xf4c\x83\x86<\x1e\xde\xfb\xb2\xeb33\x15\x92\x9e\xc9n\x11\xb2\x93\x86\xa9\x0f\v\xaa\xe8\xf4Pe\x88\x97\xa6ܙv\xdcj\xc1J\x85-\xefd\xd12\x9c\x922\x1f\xf4\x89R\xdf\xda#L\xaf\xa3\xe5\xc2Wz\xe2,Ea\xfeX\xb9\x9a\x9f^=\xbf\x8f\x85\x17.Q\xcbW\xe8km\xecB\xf8\x1a\xa9\xa6\a\xf0\x12\xa5\xce\xd9\xf1\x9c\v\x97f~O\x1a,:\xeal0 \xeff\xba\xd3S\xc6|\xc47\xd3\x03\xdd\x06\xfe\xf3ߛ\xff\x05\x00\x00\xff\xff\xd8T?\xb3K\x1a\x00\x00"), - []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcY\xcbsۼ\x11\xbf\xfb\xaf\xd8I\x0f\xb9X\xf4\x97\xaf\x9dNG\xb7Dng4\xfd\x92j\"\xd7w\x88\\\x91\x88A\x80\xc5C\xae\xdb\xe9\xff\xdeY<(> \xd1N\x93ꐉ\x01\xecb\x9f\xbf\xdd\x05W\xab\xd5\r\xeb\xf8#jÕ\\\x03\xeb8\xfeӢ\xa4\xbfL\xf1\xf4'Spuw\xfap\xf3\xc4e\xb5\x86\x8d3V\xb5_\xd1(\xa7K\xbc\xc7#\x97\xdcr%oZ\xb4\xacb\x96\xado\x00\x98\x94\xca2Z6\xf4'@\xa9\xa4\xd5J\bԫ\x1ae\xf1\xe4\x0exp\\T\xa8=\xf3t\xf5\xe9\x97\xe2ï\xc5/7\x00\x92\xb5\xb8\x06\xe2\xe7:\xa1Xe\x8a\x13\nԪ\xe0\xea\xc6tX\x12\xdbZ+\u05ed\xe1\xbc\x11\xc8\xe2\x95A\xdc{f\xd9\xdf=\a\xbf(\xb8\xb1\x7f\x9dl\xfcƍ\xf5\x9b\x9dp\x9a\x89ѭ~\xddpY;\xc1\xf4p\xe7\x06\xc0\x94\xaa\xc35|\xa1+;V\"\xadEM\xbc\b+`U\xe5m\xc3\xc4NsiQo\x94pm\xb2\xc9\n*4\xa5\xe6\x9d\xf5\xba\x9f\x05\x02c\x99u\x06\x8c+\x1b`\x06\xbe\xe0\xf3\xddV\ued2a5\x9a \x12\xc07\xa3\xe4\x8e\xd9f\rE8^t\r3\x18w\x83\xf9\xf6~#.\xd9\x17\x92\xd6X\xcde\x9d\xbb\xff\x81\xb7\b\x95\xd3\xdem\xa4s\x89`\x1bn\x86\x82=3C\xc2i\x8b\xd5E1\xfc>13\x96\xb5\xddT\x9e\x01i\x10\xa8b\x16s\xe2lT\xdb\t\xb4X\xc1\xe1\xc5bR\xe2\xa8t\xcb\xec\x1a\xb8\xb4\x7f\xfc\xc3eKDS\x15\x9e\xf4^ɱY>\xd1*\f\x96\x83$\xe4\xa1\x1au\xd66\xca2\xf1\xbf\bb\x89\xc1\xa7\x01}\x90$\xf0\x1d\xae/\x8aB\xe1\x06\xea\b\xb6A\xf8\xc4\xca'\xd7\xc1\xde*\xcdj\x84\xdfT\x19\x9c\xf7ܠ\x8e\xce;\x84#\xa6QNTpH\x1a\x03\x18\xabt\u058b\x1d\x96E\xa0\x8a|\x13ۉ+\xc7w\xfe\xe0 +5\xb2l\x90%\x94)\xfc\t\xaed>\xd2>֘\x8f\xb2\xb0}\xfa\x95\x89\xaea\x1fBz\x97\r\xb6l\x1dϫ\x0e\xe5\xc7\xdd\xf6\xf1\xf7\xfb\xd12@\xa7U\x87\xda\xf2\x84/\xe17\x00\xce\xc1*\x8c\xb5~O\f\xc3)\xa8\b1\xd1x\xffE\xb4\xc0*\xca\x10\xfc\xca\rh\xec4\x1a\x94vh\xdb\xf4SG`\x12\xd4\xe1\x1b\x96\xb6\x80=jb\x93<\\*yBmAc\xa9j\xc9\xff\xd5\xf36`\x95\xbfT0\x8b\x11\xf0\xce?\x8fN\x92\t81\xe1\xf0\x16\x98\xac\xa0e/\xa0\x91n\x01'\a\xfc\xfc\x11S\xc0g\xa5\x11\xb8<\xaa54\xd6vf}wWs\x9b\nF\xa9\xda\xd6In_\xee<\xf6\xf3\x83\xb3J\x9b\xbb\nO(\xee\f\xafWL\x97\r\xb7XZ\xa7\xf1\x8eu|\xe5E\x97\xbeh\x14m\xf5;\x1dK\x8cy?\x92u\x16a\xe1\xe7\xd1\xfe\x8a\a\b\xf4\x81\x1b`\x914hq64-\x91u\xbe\xfey\xff\x00\xe9j\uf329\xf5\xbd\xddτ\xe6\xec\x022\x18\x97G\xd4\xc1\x89G\xadZ\xcf\x13e\xd5).\xad\xff\xa3\x14\x1c\xe5\xd4\xfc\xc6\x1dZn\xc9\xef\xffph,\xf9\xaa\x80\x8d\xaf\xa2p@p\x1d\xc5oU\xc0V\u0086\xb5(6\xcc\xe0Ow\x00YڬȰ\xafs\xc1\xb0\x01\x98\x1e\x0eV\x1bl\xa4\x1a~\xc1_g8\xd8wX\x92\xe3\xc8vDď<\xa2\xdcQi`\x83\x93ň]>]\xe9\x97\x05\xb7顉<\x9fr4I,9\x80䄷\xe1\xe4\x8c)\x80\x98\x82tO\xa3\xb1S\x86[\xa5_\x88q\xc0\xe7b\xc6\xe1\x82\xf1\xe9W2Y\xa2X\xd0d\xe3\x0f\x01\x97\x15\xd9\x11\xfb\x98#x\b\f\xbcLJ֊r\xe2\x92y\xc3ok\x89\x86BԠ%\x8d\xa4'\x1e\x8097\xc0%\x9c\xbb\x17\x18v)S\xad\x0eJ\tdS\xbc+\r\xdfK֙F\xd9\x05ݶGH'\x1f^:\xa4\xcb7\xfb\xed-\xfd\x93\xd6).N\xbc\x8a\x00L\xc9Cu|\x0e\xb2\x10\x80\x96\x0em\xf6[0\x91|n\x04\xe9\x84`\a\x81k\xb0\xda\xcd\x15\xbb\x1c\x86\xf4Kl7\x82\x99쁉\x821\x00\xfd\xf1\\\xf4%~P\xfa\x13\xb6aS\xa4\xe9\rN出\xd1\x01\x11\xef\xeb.\xab\x13\x8d<\xf2\xc8\xeb\xf9\xdd\xc3\xd9\xe8Z\x8a\\UmV3\x06W\x92ũD\x90$\xab\x96\xd6W\xa9~P\x97t\xe4ulC3\x97\x1e9\x8aʼ9\xd9\x17\xec\xe1\x85X\x80\xb0^\x89T\xec\"R\x11=x\x061 \x9c\xf1\xa3\x11mf\x14\b%\xa5 Ds!ҽo\xaaJ\xe4ܾSWn\t\xc3\xff69>\xb1\x83\xa5\x01\xc2\xebn\x15<3n\xfb\xd65\a\xe0\x89\x97\xb9\x85\x03\x1e\xa9]\xd2h\x9d\x96T\xd9Pkj \x8cg\xa9\\\x06گ(e\x06efA\xa1iE\xf2Z\xd0\xff\xa7\x98=L\xf4\x8c2\xae{\x9b\x84\xbe\x83\xed\x1f'\x96\x84\x1c\x9fNr*\xcdkNc\x81\xecw\xcemK\x00\x87\x8c\xa4q\xec\xf4p\xe5\xf1\xb6\xa0F!\xf5p\x04\x80gv\x94\xa1\xe1r\x02p\x1a;6\xfbm\x86gOQ\xc5\xfc\xcad\xe7\xa25v\x8f\x9bWفD\xc9\xe05-?7\xbcl\xc6~\x9b\x8d\b^\x16\xf6\x84\xbeE}\x83\x98y\xa0^\xe5\x1b\xd6əi\x96M\xb6\x87\xf1:\xdd\x1a\xbb>\xbb\xbb{ܼ\xaa\xa9\xf7\xaf\x1e\xafk\xeb\xc3\x13W\xb4r\xe9\xb4Fi\xd3\xc3\x17M\xb8\xdf\xd1ؗ\xe1\xc9h\xf8(\xb0\xd4\f\xcf)\xfc䬫\x01ڰԠ\xfb\x87\x89\xf4,\x95k\x87\xcf\xec\x02\xa5\x9f\xe4\x89\x1bV\x80'\x94@S\vゐ۳4Ŕ&\x9fN=\x97\x88b\xe1\r2ͬQ\xbc\xf4\"\xf0@\xc1\xe9G\xd2\xf7\xe6\nO\x0f\xa2\x94~\x19#\xcc#:\xbdw\xd1 \xba\xca2}Um\xcc&g\xdf+|E\xe3D\xa6@\xfc\xc4^!\\\x19\xa6-\x93\xed\x15\xae\xcf\b\xcc\x00\x03\x1d\x98D\x98\xb862}\x7f\x03Ѣ1\xac^\xc2\xf1\xcf\xe1Tx\xe9\x88$\xc0\x0eTGǢ\xbd71\xd9\xde\x04\xa3\x1d\xb3͂\x04;f\x9b\x94\xd6G'\x84\xa7\x99պ\xd8\xfa\x1e\x90b\xf8G\x95\x1a\xf5\u0093D\xea\x12+n:\xc1^2\x8c\x93\"\xc3\x1c\x1fdK\xc2\xd5TZ\xe7\xf1p}\xba\xef\xbf>\xe4\xa7\xc5\xdc'\x84\x9c\x0f\x86\x1f\x03&\xfb\xfdW\x85\x9fs\xc3\x158J\x99\xbc\xbd\x7fe\xfb\xbb\xbdOY\xc7+\x94\x96:z\xed\xcbϸ\x97\x92W\a\x9a\xc1\x1b\xd9\xdbڿ\xd17\xa9%\x89G\x87\x17\xfa\x81\xf85,\xd7\r\xec)\xc5\tX\xfcC\xf0f\xfa\xbd\xe2\xb6\xff\xfc\xc1l|\x8d.\x1b&kJ\bI%ŗ\xa4\x1c\xe3Y\x81\x1f\x95\xf3\xb1\xf8\xff\xcfJ\x9e\r\x97٢\x97\xbc\x1a\xf0\x8eo\x10\xc3\x15w\xe8_\xff\xd7\xf0\xef\xff\xdc\xfc7\x00\x00\xff\xffgK\fV\xa3\x1e\x00\x00"), + []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcY_o\xe4\xb8\r\x7fϧ \xb6\x0fyY;\xb7ע(\xe6mw\xd2\x02Ao\xd3\xc1e\x91w٢=\xbaȒ*ɓ\xa6E\xbf{A\xc9\xf2\xf8\x8f&N\xf6z緑(\xf2'\x92\xfa\x91\xd2\x14EqŌxD\xeb\x84V;`F\xe0\xbf<*\xfa\xe5ʧ\xbf\xb8R\xe8\x9bӧ\xab'\xa1\xf8\x0e\xf6\xbd\xf3\xba\xfb\x19\x9d\xeem\x8d\xb7\xd8\b%\xbc\xd0\xea\xaaC\xcf8\xf3lw\x05\xc0\x94ҞѰ\xa3\x9f\x00\xb5V\xdej)\xd1\x16-\xaa\U000a9bf0\xea\x85\xe4h\x83\xf2d\xfa\xf4C\xf9\xe9\xc7\xf2\x87+\x00\xc5:\xdc\x01\xe9\xe3\xfaYI\u0378+O(\xd1\xeaR\xe8+g\xb0&ŭս\xd9\xc1y\".\x1c\x8cF\xc0\xb7̳\xdbAG\x18\x96\xc2\xf9\xbf\xaf\xa6~\x12·i#{\xcb\xe4\xc2v\x98qB\xb5\xbddv>w\x05\xe0jmp\a\xf7dڰ\x1ail\xd8S\x80R\x00\xe3\xdd\x00S\xa0\xab_\xb0\xf6%<\xa0%5\xe0\x8e\xba\x97\x9ch\xff\x84փ\xc5Z\xb7J\xfc{\xd4\xed\xc0\xeb`T2\x8f\x03\xf1\x9e\xbf\xc0\x90\x8aI81\xd9\xe3G`\x8aC\xc7^\xc0\"Y\x81^M\xf4\x05\x11W\xc2Wm\x11\x84j\xf4\x0e\x8e\xde\x1b\xb7\xbb\xb9i\x85O\xe5\xab\xd6]\xd7+\xe1_nB%\x12U\xef\xb5u7\x1cO(o\x9ch\vf\xeb\xa3\xf0X\xfb\xde\xe2\r3\xa2\b\xd0U(ae\xc7\xff`\x87\x82\xe7\xaegXW\x11\x8d_\xa8<\xafD\x80\xca\x0f\xa5\x13\x1b\x96\xc6]\x9c\x1dMC䝟\xff\xfa\xf0\r\x92\xe9\x10\x8c\xa5\xf7\x83\xdf\xcf\v\xdd9\x04\xe40\xa1\x1a\xb41\x88\x8d\xd5]Љ\x8a\x1b-\x94\x0f?j)P-\xdd\xef\xfa\xaa\x13\x9e\xe2\xfe\xcf\x1e\x9d\xa7X\x95\xb0\x0f5\x1d*\x84\xde\xd0A\xe2%\xdc)س\x0e\xe5\x9e9\xfc\xcd\x03@\x9ev\x059\xf6m!\x98\xb6#K\xe1\xe8\xb5\xc9D\xea'.\xc4k\xca\x02\x0f\x06k\n\x1dy\x8f\x96\x89F\f<\xdbh\vl&[\xceT\xe6\x8f,}Y\xae]\n-0}ɭI\xc0Ԅ\xd1\x06\xd2wQr\xa5\x14@^,\x14\x16\x8dv\xc2k\xfbr.\x17\xe5JÅ\x00\xd0W3U\xa3\xdc\xd8\xc9>\b\x81P\x9c<\x89c\xde\x11ED\x05\x01\x93V\xad\xa6sq\xd9\xc1\xf1\xbb\xf3\xb4\x8a\x12ա\xa7=\xa9,\x93\v\x05\xe7>\n\xa6\xfd\xd2rg\x95\xd6\x12ْ\xf7(\xb7\xbe\xea\x13ur\xaa\x11\xedz\x8fӖ\xefR\xe07ܗIÉI\xda\x05\xe5\x1c!):\x1a/RB\x12\xf16\xa2\x1d\x8al\xc6h#Prw)\x96\xab\xf3\x916\x1c\xacl\x84sD\x99\x8e\xc7P^B\xd7\x11\x14P`\x89G\\h\xe7h2\x830\xa6`\tw\xcdD\xa3p\xf0\xe1\x03h\v\x1fb\xcb\xff\xe1cL\xd7^H_\b5\xb1\x91\xd1\xf8,\xa4Lvߕ\xc5\x14\xbd\xb1\xcdн\xdfp\xc0?\x16\xe2\v?x\xea\x7f\xc2\u07bd\x86g&\xfcX\xee2\x98G\xd3\xee#T\xd8\x10\xc5Z\xf4\xbdUt\x12\xd0Z\xa2\x1c\x17T\xea\u07bfkSN1\xe3\x8e\xda\xdf\xddnl\xe7a\x14L\xecrw\x9b\xb8\xe51Da\xa4\x98A\x12\xbc\xce\x05\x94\xa0G\x0e\t\xc5\xe8}hC\x05\x1c/X[\x90\xe7\xd2\t\xb7\xb6\xa2\x15\xd4V\xa8q\xe6Ly'\xba\x90\xe5\x12Q\xb8\xb0?\xe4Л\b\x9c(\x86\xaak\x85\xc0EӠE\xe5c}\x8d\x86\x0f\x8f\xfbkw6\x92\xd3\xd9L0\x84\x0e\xabc\xc6 \xa7\xbe\x98\";8\xea].\xf2̶\xe8\x1f\xc366\xfc\xf3m\"\x9a\x9cC\x95\x9b.1T\b\x86\xe8F\x8dpx\xdcS\a\x96\xd9\xc6\xe1q\x8d\xf0r\x95\x83\xa1\xe1\xbd\x10\xc1\x15\xcaU\xfc\x06<\xaf9v\x83N\x01\xcc\xe9\r\x96\x0f\x8f\xb9B:\xba\x03\xfc\x91y\x92\x18.(P\xbdduB:\x1fC8\xbf\x0fo\xfd&\xc0\xfbW\x11\uf5d0/\xe0\xad^~5d*\xde\xc2\"_\xa3.^\x89\\\x01\xe6\x94\x1d\xac\xdf^\xa2\xf2\x96\x8b|w\xb5\x90YR\xfcb\xfaL\x96ˉ9\xd3,f\xa7G\xf2Mmh\xb8B\xbe\xb5\x11\x8d\x0fCC\xd8\xeb\xde\x06\x1a\x1a\x9e\x8b\xe8V\xf6]\xadh\x1d\x1fZ\xa6w\xea\xad\xf6m\xbd\"\xdc\xf7,\x9f\xd4;\x96\x12*^\xec\xd3kN\xae\x7f;\xeb\x8bK\x03=\x92:\xe4\x80'T@\xad6\x13\x12y\xd2\xe9J\xf8F\xddx\xb8\xf8\\/\xafH\xc1߃\xa2Pv\xa9gʀ^\xafK\xaf:t\xdd)H\xc5JB\xf5R\xb2J\xe2\x0e\xbc\xed/\xf5\x8fكҡs\xac\xdd\"\xea\xafQ*^\x15\x87%\xc0*j*\x96=\xed\xb5\x1bb\xff\xae\xa2\xa14\xdf\xc2p\xafy\x00\xa0\xbe\xe3\x95\xe4]XB\x0f\xbe\x01\xe6@2\xb9\x9c\x1f\xa1\xbd~=@\xd5w9f\xba\xc7\xe7\xcc\xe8\xe7\xbaF\x93c\xcb\x02\x0e\x16\r\xb3٩\xd5\xf3\xect2^vrę\xe6\xb2:\xc7\xf7\xcf\xcc\xdc\xdf\xc2ax\x97\xa7\a|[\xceN\xb7\xa3\xa3\x96\xe90\x87'J\xd5w\x15Z\xf2xx\x04M\xaeO,\x999\x80L\xf1Y\xc8\xce\x1aƞ0\xa8\xa2\x93LU*^\xe0R\x97̅3\x92\xe5\x8al\xdaɬ}9\x1f\x90D:#\xbd\xbf\xb7_\x19\x9f\x8c\xf3E8\xf7\ue6cb\xc2\xf4\x05w1?>\x05\xff6\x16^\xb9\xd0͟\xe6\xb7Z\xea\x99\xf0\x16\xc1\x0f\xff\n\xe4\xe8}\xca\xd4k^\x9e\x9b\xf9=)9\xeb\xa8\xd5`@\xce'\xba\x87g\x95\xe9H_\x8d\x8f\x85;\xf8\xcf\x7f\xaf\xfe\x17\x00\x00\xff\xff\x9f\xc23\x7f`\x1b\x00\x00"), + []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcY\xcbsۼ\x11\xbf\xfb\xaf\xd8I\x0f\xb9X\xf4\x97\xaf\x9dNG\xb7Dng4\xfd\xe2z\xa2\xd4w\x88\\\x91\x88A\x80\xc5C\xae\xdb\xe9\xff\xdeY<(> \xd1J\x93\xf2\xe0\xb1\x00\xecb\x1f\xd8\xdf\xee\x02\xab\xd5\xea\x86u\xfc\t\xb5\xe1J\xae\x81u\x1c\xffiQ\xd2/S<\xff\xc9\x14\\\xdd\x1d?\xdc\xfeR|\xf8\xb5\xf8\xe5\x06@\xb2\x16\xd7@\xfc\\'\x14\xabLqD\x81Z\x15\\ݘ\x0eKb[k\xe5\xba5\x9c&\x02Y\xdc2\x88{\xcf,\xfb\xbb\xe7\xe0\a\x057\xf6\xaf\x93\x89߸\xb1~\xb2\x13N31\xdaՏ\x1b.k'\x98\x1e\xce\xdc\x00\x98Ru\xb8\x86\aڲc%\xd2X\xd4ċ\xb0\x02VU\xde6LH\x12\xf8\x0e\xc7\x17E\xa1\xe3\x06\xea\x00\xb6A\xf8\xc4\xcag\xd7\xc1\xce*\xcdj\x84\xdfT\x19\x9c\xf7Ҡ\x8e\xceۇ%\xa6QNT\xb0O\x1a\x03\x18\xabt\u058b\x1d\x96E\xa0\x8a|\x13ۉ+\xc7{\xfe\xe0CVjd\xd9C\x96P\xa6\xf0+\xb8\x92\xf9\x93\xf6\xb1\xc67\x9d\xb2\xa15\xa5\xaa\xb07\x1d\x0e%\xe2\x06:\xadJ4\xe6¹'\xf2\x91\f\x0f\xa7\x81\x99Y\u008a\xe3\xafLt\r\xfb\x10P\xa6l\xb0e\xebH\xa1:\x94\x1f\x1f\xb7O\xbfߍ\x86\x81\x04\xe9P[\x9e`.|\x03\xfc\x1e\x8c\xc2X\xd9\xf7\xc40\xac\x82\x8a\x80\x1b\x8d\xd74\x82\x16VQ\x86`\x10n@c\xa7Ѡ\xb4C\x17\xa7O\x1d\x80IP\xfboX\xda\x02v\xa8\x89M:h\xa5\x92G\xd4\x164\x96\xaa\x96\xfc_=o\x03V\xf9M\x05\xb3\x18q\xf7\xf4y\x90\x94L\xc0\x91\t\x87\xb7\xc0d\x05-{\x05\x8d\xb4\v89\xe0痘\x02>+\x8d\xc0\xe5A\xad\xa1\xb1\xb63뻻\x9a۔\xb7JնNr\xfbz\xe7S\x10\xdf;\xab\xb4\xb9\xab\xf0\x88\xe2\xce\xf0z\xc5t\xd9p\x8b\xa5u\x1a\xefX\xc7W^t\xe9sW\xd1V\xbf\xd31ә\xf7#Yg\x1e\r\x9fO:\x17<@\xb9\x87\x8e\x13\x8b\xa4A\x8b\x93\xa1i\x88\xac\xf3\xe5ϻ\xaf\x90\xb6\xf6ΘZ\xdf\xdb\xfdDhN. \x83qy@\x1d\x9cxЪ\xf5}%\x93%\x8a\x05M6~\x11pY\x91\x1d\xb1?s\x04\x0f\x81\x81\x97I\xc9ZQL\x9c3o\xf8\xb6\x96h\xe8\x88\x1a\xb4\xa4\x91\xcc 8\x97p*\xa2`X,M\xb5\xda+%\x90M\xf1\xae4|'Yg\x1ae\x17t\xdb\x1e \xad\xfc\xfa\xda!m\xbe\xd9mo\xe9O\x1a\xa7sq\xe4U\x04`\n\x1e*'\xe6 \v\x01hi\xd1f\xb7\x05\x13\xc9\xe7F\x90N\b\xb6\x17\xb8\x06\xab\xdd\\\xb1\xf3ǐ\xbe\xc4v#\x98\xc9.\x98(\x18\x0f\xa0_\x9e;}\x89\x1f\x94~\x85m\xd8\x14iz\x83S\xfa\xa1\xa2x@\xc4\xfb\xf4\x0f/\xdc6Y\xca\v\xc7\x0fbq\x93\x04\xfc\x11\xfa\xc4b'\xa8\xa3\x0e\x17\x94y|\xdax}\x974#T\xfe\x1e\xcd\x02\xcb\xf3\aq\xa6\xdbӈ \xa7\xddD\xcas\xca)\x8a/\xc2\b\xac\xc0u\xd7\xcbN\x01\xce5Vs\x99W#\x7fe\xa6\xc7J\x9f\x89\xda\x19\xb6{S0\xcb>\xab#u^\xf2\xc0\xeb\xf9\xde\xc3\x16\xedR\x88\\Tm\x963\x06[\x92\xc5)E\x90$\xab\x96\xc6W)\x7fP\x95t\xe0u\xac\x863\x9b\x1e8\x8a\xca\\\x1d\xec\v\xf6\xf0B,@X\xafDJv\x11\xa9\x88\x1e<\x83x \x9c\xf1\x1d\x1aMf\x14\b)\xa5 D>m\xdeT\xd4\xfb\x86\xfcme}\xb8i\x8bV.\x9d\xd6(m\xba\x7f\xa3\x0e\xf7;\n\xfb2\xdc\\\r\xef&\x96\x8a\xe19\x85\xef\x9cu5@\x1b\x96\nt\x7f?\x92n\xc7r\xe5\xf0\x89]\xa0\xf4\x9d