Merge branch 'main' into resourcemodifier

.github/stale.yml

@@ -1,44 +0,0 @@
-# Number of days of inactivity before an issue becomes stale
-daysUntilStale: 60
-# Number of days of inactivity before a stale issue is closed
-daysUntilClose: 14
-# Issues with these labels will never be considered stale
-exemptLabels:
-  - Epic
-  - Area/CLI
-  - Area/Cloud/AWS
-  - Area/Cloud/Azure
-  - Area/Cloud/GCP
-  - Area/Cloud/vSphere
-  - Area/CSI
-  - Area/Design
-  - Area/Documentation
-  - Area/Plugins
-  - Bug
-  - Enhancement/User
-  - kind/requirement
-  - kind/refactor
-  - kind/tech-debt
-  - limitation
-  - Needs investigation
-  - Needs triage
-  - Needs Product
-  - P0 - Hair on fire
-  - P1 - Important
-  - P2 - Long-term important
-  - P3 - Wouldn't it be nice if...
-  - Product Requirements
-  - Restic - GA
-  - Restic
-  - release-blocker
-  - Security
-# Label to use when marking an issue as stale
-staleLabel: staled
-# Comment to post when marking an issue as stale. Set to `false` to disable
-markComment: >
-  This issue has been automatically marked as stale because it has not had
-  recent activity. It will be closed if no further activity occurs. Thank you
-  for your contributions.
-# Comment to post when closing a stale issue. Set to `false` to disable
-closeComment: >
-  Closing the stale issue.

.github/workflows/push.yml

@@ -64,6 +64,10 @@ jobs:
     - name: Publish container image
       if: github.repository == 'vmware-tanzu/velero'
       run: |
+        sudo swapoff -a
+        sudo rm -f /mnt/swapfile
+        docker image prune -a --force
+              
         # Build and push Velero image to docker registry
         docker login -u ${{ secrets.DOCKER_USER }} -p ${{ secrets.DOCKER_PASSWORD }}
         VERSION=$(./hack/docker-push.sh | grep 'VERSION:' | awk -F: '{print $2}' | xargs)

.github/workflows/stale-issues.yml

@@ -1,8 +1,7 @@
 name: "Close stale issues and PRs"
 on:
   schedule:
-    # First of every month
-    - cron: "30 1 * * *"
+    - cron: "30 1 * * *" # Every day at 1:30 UTC
 
 jobs:
   stale:
@@ -11,14 +10,14 @@ jobs:
       - uses: actions/stale@v3
         with:
           repo-token: ${{ secrets.GITHUB_TOKEN }}
-          stale-issue-message: "This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. If a Velero team member has requested log or more information, please provide the output of the shared commands."
-          close-issue-message: "This issue was closed because it has been stalled for 5 days with no activity."
-          days-before-issue-stale: 30
-          days-before-issue-close: 5
+          stale-issue-message: "This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands."
+          close-issue-message: "This issue was closed because it has been stalled for 14 days with no activity."
+          days-before-issue-stale: 60
+          days-before-issue-close: 14
+          stale-issue-label: staled
           # Disable stale PRs for now; they can remain open.
           days-before-pr-stale: -1
           days-before-pr-close: -1
           # Only issues made after Feb 09 2021.
           start-date: "2021-09-02T00:00:00"
-          # Only make issues stale if they have these labels. Comma separated.
-          only-labels: "Needs info,Duplicate"
+          exempt-issue-labels: "Epic,Area/CLI,Area/Cloud/AWS,Area/Cloud/Azure,Area/Cloud/GCP,Area/Cloud/vSphere,Area/CSI,Area/Design,Area/Documentation,Area/Plugins,Bug,Enhancement/User,kind/requirement,kind/refactor,kind/tech-debt,limitation,Needs investigation,Needs triage,Needs Product,P0 - Hair on fire,P1 - Important,P2 - Long-term important,P3 - Wouldn't it be nice if...,Product Requirements,Restic - GA,Restic,release-blocker,Security"

Makefile

@@ -344,7 +344,7 @@ serve-docs: build-image-hugo
 	-v "$$(pwd)/site:/srv/hugo" \
 	-it -p 1313:1313 \
 	$(HUGO_IMAGE) \
-	hugo server --bind=0.0.0.0 --enableGitInfo=false
+	server --bind=0.0.0.0 --enableGitInfo=false
 # gen-docs generates a new versioned docs directory under site/content/docs.
 # Please read the documentation in the script for instructions on how to use it.
 gen-docs:

changelogs/unreleased/5774-anshulahuja98

@@ -0,0 +1 @@
+Design to add support for Multiple VolumeSnapshotClasses in CSI Plugin.

changelogs/unreleased/6410-blackpiglet

@@ -0,0 +1 @@
+Retrieve DataUpload into backup result ConfigMap during volume snapshot restore.

changelogs/unreleased/6433-blackpiglet

@@ -0,0 +1 @@
+Modify DownloadRequest controller logic

changelogs/unreleased/6434-allenxu404

@@ -0,0 +1 @@
+Fix hook filter display issue for backup describer

changelogs/unreleased/6436-qiuming-best

@@ -0,0 +1 @@
+Add data download controller for data mover

changelogs/unreleased/6459-blackpiglet

@@ -0,0 +1 @@
+Add exit code log and possible memory shortage warning log for Restic command failure.

changelogs/unreleased/6464-openstack-csi-topology-keys

@@ -0,0 +1 @@
+Add support for OpenStack CSI drivers topology keys

config/crd/v2alpha1/bases/velero.io_datadownloads.yaml

@@ -41,6 +41,10 @@ spec:
       jsonPath: .metadata.creationTimestamp
       name: Age
       type: date
+    - description: Name of the node where the DataDownload is processed
+      jsonPath: .status.node
+      name: Node
+      type: string
     name: v2alpha1
     schema:
       openAPIV3Schema:
@@ -132,6 +136,9 @@ spec:
               message:
                 description: Message is a message about the DataDownload's status.
                 type: string
+              node:
+                description: Node is name of the node where the DataDownload is processed.
+                type: string
               phase:
                 description: Phase is the current state of the DataDownload.
                 enum:

config/crd/v2alpha1/bases/velero.io_datauploads.yaml

@@ -42,6 +42,10 @@ spec:
       jsonPath: .metadata.creationTimestamp
       name: Age
       type: date
+    - description: Name of the node where the DataUpload is processed
+      jsonPath: .status.node
+      name: Node
+      type: string
     name: v2alpha1
     schema:
       openAPIV3Schema:
@@ -147,6 +151,9 @@ spec:
               message:
                 description: Message is a message about the DataUpload's status.
                 type: string
+              node:
+                description: Node is name of the node where the DataUpload is processed.
+                type: string
               path:
                 description: Path is the full path of the snapshot volume being backed
                   up.

design/Implemented/csi-snapshots.md

@@ -304,8 +304,8 @@ Without these objects, the provider-level snapshots cannot be located in order t
 
 
 [1]: https://kubernetes.io/blog/2018/10/09/introducing-volume-snapshot-alpha-for-kubernetes/
-[2]: https://github.com/kubernetes-csi/external-snapshotter/blob/master/pkg/apis/volumesnapshot/v1alpha1/types.go#L41
-[3]: https://github.com/kubernetes-csi/external-snapshotter/blob/master/pkg/apis/volumesnapshot/v1alpha1/types.go#L161
+[2]: https://github.com/kubernetes-csi/external-snapshotter/blob/master/client/apis/volumesnapshot/v1/types.go#L42
+[3]: https://github.com/kubernetes-csi/external-snapshotter/blob/master/client/apis/volumesnapshot/v1/types.go#L262
 [4]: https://github.com/heptio/velero/blob/main/pkg/volume/snapshot.go#L21
 [5]: https://github.com/heptio/velero/blob/main/pkg/apis/velero/v1/pod_volume_backup.go#L88
 [6]: https://github.com/heptio/velero-csi-plugin/

design/multiple-csi-volumesnapshotclass-support.md

@@ -0,0 +1,177 @@
+# Proposal to add support for Multiple VolumeSnapshotClasses in CSI Plugin
+
+- [Proposal to add support for Multiple VolumeSnapshotClasses in CSI Plugin](#proposal-to-add-support-for-multiple-volumesnapshotclasses-in-csi-plugin)
+	- [Abstract](#abstract)
+	- [Background](#background)
+	- [Goals](#goals)
+	- [Non Goals](#non-goals)
+    - [User Stories](#user-stories)
+        - [Scenario 1](#scenario-1)
+        - [Scenario 2](#scenario-2)
+    - [Detailed Design](#detailed-design)
+        - [Plugin Inputs Contract Changes](#plugin-inputs-contract-changes)
+        - [Using Plugin Inputs for CSI Plugin](#using-plugin-inputs-for-csi-plugin)
+        - [Annotations overrides on PVC for CSI Plugin](#annotations-overrides-on-pvc-for-csi-plugin)
+        - [Using Plugin Inputs for Other Plugins](#using-plugin-inputs-for-other-plugins)
+    - [Alternatives Considered](#alternatives-considered)
+    - [Security Considerations](#security-considerations)
+    - [Compatibility](#compatibility)
+    - [Implementation](#implementation)
+    - [Open Issues](#open-issues)
+
+
+## Abstract
+Currently the Velero CSI plugin chooses the VolumeSnapshotClass in the cluster that has the same driver name and also has the velero.io/csi-volumesnapshot-class label set on it. This global selection is not sufficient for many use cases. This proposal is to add support for multiple VolumeSnapshotClasses in CSI Plugin where the user can specify the VolumeSnapshotClass to use for a particular driver and backup.
+
+
+## Background
+The Velero CSI plugin chooses the VolumeSnapshotClass in the cluster that has the same driver name and also has the velero.io/csi-volumesnapshot-class label set on it. This global selection is not sufficient for many use cases. For example, if a cluster has multiple VolumeSnapshotClasses for the same driver, the user may want to use a VolumeSnapshotClass that is different from the default one. The user might also have different schedules set up for backing up different parts of the cluster and might wish to use different VolumeSnapshotClasses for each of these backups. 
+
+## Goals
+- Allow the user to specify the VolumeSnapshotClass to use for a particular driver and backup.
+
+## Non Goals
+- Deprecating existing VSC selection behaviour. (The current behaviour will remain the default behaviour if the user does not specify the VolumeSnapshotClass to use for a particular driver and backup.)
+
+
+## User Stories
+
+### Scenario 1
+- Consider Alice is a cluster admin and has a cluster with multiple VolumeSnapshotClasses for the same driver. Each VSC stores the snapshots taken in different ResourceGroup(Azure equivalent). 
+- Alice has configured multiple scheduled backups each covering a different set of namespaces, representing different apps owned by different teams. 
+- Alice wants to use a different VolumeSnapshotClass for each backup such that each snapshot goes in it's respective ResourceGroup to simply management of snapshots(COGS, RBAC etc).
+- In current velero, Alice can't achieve this as the CSI plugin will use the default VolumeSnapshotClass for the driver and all snapshots will go in the same ResourceGroup.
+- Proposed design will allow Alice to achieve this by specifying the VolumeSnapshotClass to use for a particular driver and backup/schedule.
+
+## Scenario 2
+- Bob is a cluster admin has PVCs storing different types of data. 
+- Most of the PVCs are used for storing non-sensitive application data. But certain PVCs store critical financial data.
+- For such PVCs Bob wants to use a VolumeSnapshotClass with certain encryption related parameters set. 
+- In current velero, Bob can't achieve this as the CSI plugin will use the default VolumeSnapshotClass for the driver and all snapshots will be taken using the same VolumeSnapshotClass.
+- Proposed design will allow Bob to achieve this by overriding the VolumeSnapshotClass to use for a particular driver and backup/schedule using annotations on those specific PVCs.
+
+
+## Detailed Design
+
+### Staged Approach: 
+
+### Stage 1 Approach
+#### Through Annotations
+    1. **Support VolumeSnapshotClass selection at backup/schedule level**
+    The user can  annotate the backup/ schedule with driver and VolumeSnapshotClass name. The CSI plugin will use the VolumeSnapshotClass specified in the annotation. If the annotation is not present, the CSI plugin will use the default VolumeSnapshotClass for the driver.
+
+        *example annotation on backup/schedule:*
+        ```yaml
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+        name: backup-1
+        annotations:
+            velero.io/csi-volumesnapshot-class/csi.cloud.disk.driver: csi-diskdriver-snapclass
+            velero.io/csi-volumesnapshot-class/csi.cloud.file.driver: csi-filedriver-snapclass
+            velero.io/csi-volumesnapshot-class/<driver name>: csi-snapclass
+        ```
+
+         To query the annotations on a backup: "velero.io/csi-volumesnapshot-class/'driver name'" - where driver names comes from the PVC's driver.
+
+    2. **Support VolumeSnapshotClass selection at PVC level**
+    The user can annotate the PVCs with driver and VolumeSnapshotClass name. The CSI plugin will use the VolumeSnapshotClass specified in the annotation. If the annotation is not present, the CSI plugin will use the default VolumeSnapshotClass for the driver. If the VolumeSnapshotClass provided is of a different driver, the CSI plugin will use the default VolumeSnapshotClass for the driver.
+
+        *example annotation on PVC:*
+        ```yaml
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+        name: pvc-1
+        annotations:
+            velero.io/csi-volumesnapshot-class: csi-diskdriver-snapclass
+            
+        ```
+    
+    Consider this as a override option in conjunction to part 1.
+
+**Note**: The user has to annotate the PVCs or backups with the VolumeSnapshotClass to use for each driver. This is not ideal for the user experience.
+    - **Mitigation**: We can extend Velero CLI to also annotate backups/schedules with the VolumeSnapshotClass to use for each driver. This will make it easier for the user to annotate the backups/schedules. This mitigation is not for the PVCs though, since PVCs is anyways a specific use case. Similar to : " kubectl run --image myimage --annotations="foo=bar" --annotations="another=one" mypod"
+    We can add support for  - velero backup create my-backup --annotations "velero.io/csi:csi.cloud.disk.driver=csi-diskdriver-snapclass"
+
+### Stage 2 Approach
+The above annotations route is to get started and for initial design closure/ implementation, north star is to either introduce CSI specific fields (considering that CSI might be a very core part of velero going forward) in the backup/restore CR OR leverage the pluginInputs field as being tracked in: https://github.com/vmware-tanzu/velero/pull/5981
+
+Refer section Alternatives 2. **Through generic property bag in the velero contracts**: in the design doc for more details on the pluginInputs field.
+
+
+## Alternatives Considered
+1. **Through CSI Specific Fields in Velero contracts**
+
+    **Considerations**
+    - Since CSI snapshotting is done through the plugin, we don't intend to bloat up the Backup Spec with CSI specific fields. 
+    - But considering that CSI Snapshotting is the way forward, we can debate if we should add a CSI section to the Backup Spec.
+
+
+    **Approach**: Similar to VolumeSnapshotLocation param in the Backup Spec, we can add a VolumeSnapshotClass param in the Backup Spec. This will allow the user to specify the VolumeSnapshotClass to use for the backup. The CSI plugin will use the VolumeSnapshotClass specified in the Backup Spec. If the VolumeSnapshotClass is not specified, the CSI plugin will use the default VolumeSnapshotClass for the driver.
+
+    *example of VolumeSnapshotClass param in the Backup Spec:*
+    ```yaml
+    apiVersion: velero.io/v1
+    kind: Backup
+    metadata:
+    name: backup-1
+    spec:
+        csiParameters: 
+            volumeSnapshotClasses: 
+                driver: csi.cloud.disk.driver
+                snapClass: csi-diskdriver-snapclass
+            timeout: 10m
+    ```
+
+1. **Through changes in velero contracts**
+    1. **Through configmap references.**
+    Currently even the storageclass mapping plugin expects the user to create a configmap which is used globally, and fetched through labels. This behaviour has same issue as the VolumeSnapshotClass selection. We can introduce a field in the velero contracts which allow passing configmap references for each plugin. And then the plugin can honour the configmap passed in as reference. The configmap can be used to pass the VolumeSnapshotClass to use for the backup, and also other parameters to tweak. This can help in making plugins more flexible while not depending on global behaviour.
+    
+
+        *example of configmap reference in the velero contracts:*
+        ```yaml
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+        name: backup-1
+        spec:
+            configmapRefs:
+            - name: csi-volumesnapshotclass-configmap
+            - namespace: velero
+            - plugin: velero.io/csi
+         ```
+
+    2. **Through generic property bag in the velero contracts**: We can introduce a field in the velero contracts which allow passing a generic property bag for each plugin. And then the plugin can honour the property bag passed in.
+
+
+        *example of property bag in the velero contracts:*
+        ```yaml
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+        name: backup-1
+        spec:
+            pluginInputs:
+            - name: velero.io/csi
+            - properties:
+                - key: csi.cloud.disk.driver
+                - value: csi-diskdriver-snapclass
+                - key: csi.cloud.file.driver
+                - value: csi-filedriver-snapclass
+        ```
+
+    **Note**: Both these approaches can also be used to tweak other parameters such as CSI Snapshotting Timeout/intervals. And further can be used by other plugins.
+
+
+## Security Considerations
+No security impact.
+
+## Compatibility
+Existing behaviour of csi plugin will be retained where it fetches the VolumeSnapshotClass through the label. This will be the default behaviour if the user does not specify the VolumeSnapshotClass.
+
+## Implementation
+TBD based on closure of high level design proposals.
+
+## Open Issues
+NA

pkg/apis/velero/v1/labels_annotations.go

@@ -78,4 +78,24 @@ const (
 
 	// AsyncOperationIDLabel is the label key used to identify the async operation ID
 	AsyncOperationIDLabel = "velero.io/async-operation-id"
+
+	// PVCNameLabel is the label key used to identify the the PVC's namespace and name.
+	// The format is <namespace>/<name>.
+	PVCNamespaceNameLabel = "velero.io/pvc-namespace-name"
+
+	// ResourceUsageLabel is the label key to explain the Velero resource usage.
+	ResourceUsageLabel = "velero.io/resource-usage"
+)
+
+type AsyncOperationIDPrefix string
+
+const (
+	AsyncOperationIDPrefixDataDownload AsyncOperationIDPrefix = "dd-"
+	AsyncOperationIDPrefixDataUpload   AsyncOperationIDPrefix = "du-"
+)
+
+type VeleroResourceUsage string
+
+const (
+	VeleroResourceUsageDataUploadResult VeleroResourceUsage = "DataUpload"
 )

pkg/apis/velero/v2alpha1/data_download_types.go

@@ -111,6 +111,10 @@ type DataDownloadStatus struct {
 	// about the restore operation.
 	// +optional
 	Progress shared.DataMoveOperationProgress `json:"progress,omitempty"`
+
+	// Node is name of the node where the DataDownload is processed.
+	// +optional
+	Node string `json:"node,omitempty"`
 }
 
 // TODO(2.0) After converting all resources to use the runtime-controller client, the genclient and k8s:deepcopy markers will no longer be needed and should be removed.
@@ -125,6 +129,7 @@ type DataDownloadStatus struct {
 // +kubebuilder:printcolumn:name="Total Bytes",type="integer",format="int64",JSONPath=".status.progress.totalBytes",description="Total bytes"
 // +kubebuilder:printcolumn:name="Storage Location",type="string",JSONPath=".spec.backupStorageLocation",description="Name of the Backup Storage Location where the backup data is stored"
 // +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp",description="Time duration since this DataDownload was created"
+// +kubebuilder:printcolumn:name="Node",type="string",JSONPath=".status.node",description="Name of the node where the DataDownload is processed"
 
 type DataDownload struct {
 	metav1.TypeMeta `json:",inline"`

pkg/apis/velero/v2alpha1/data_upload_types.go

@@ -140,6 +140,10 @@ type DataUploadStatus struct {
 	// about the backup operation.
 	// +optional
 	Progress shared.DataMoveOperationProgress `json:"progress,omitempty"`
+
+	// Node is name of the node where the DataUpload is processed.
+	// +optional
+	Node string `json:"node,omitempty"`
 }
 
 // TODO(2.0) After converting all resources to use the runttime-controller client,
@@ -155,6 +159,7 @@ type DataUploadStatus struct {
 // +kubebuilder:printcolumn:name="Total Bytes",type="integer",format="int64",JSONPath=".status.progress.totalBytes",description="Total bytes"
 // +kubebuilder:printcolumn:name="Storage Location",type="string",JSONPath=".spec.backupStorageLocation",description="Name of the Backup Storage Location where this backup should be stored"
 // +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp",description="Time duration since this DataUpload was created"
+// +kubebuilder:printcolumn:name="Node",type="string",JSONPath=".status.node",description="Name of the node where the DataUpload is processed"
 
 type DataUpload struct {
 	metav1.TypeMeta `json:",inline"`

pkg/backup/item_backupper.go

@@ -445,6 +445,10 @@ const (
 	azureCsiZoneKey  = "topology.disk.csi.azure.com/zone"
 	gkeCsiZoneKey    = "topology.gke.io/zone"
 	gkeZoneSeparator = "__"
+
+	// OpenStack CSI drivers topology keys
+	cinderCsiZoneKey = "topology.manila.csi.openstack.org/zone"
+	manilaCsiZoneKey = "topology.cinder.csi.openstack.org/zone"
 )
 
 // takePVSnapshot triggers a snapshot for the volume/disk underlying a PersistentVolume if the provided
@@ -506,7 +510,7 @@ func (ib *itemBackupper) takePVSnapshot(obj runtime.Unstructured, log logrus.Fie
 		if !labelFound {
 			var k string
 			log.Infof("label %q is not present on PersistentVolume", zoneLabelDeprecated)
-			k, pvFailureDomainZone = zoneFromPVNodeAffinity(pv, awsEbsCsiZoneKey, azureCsiZoneKey, gkeCsiZoneKey, zoneLabel, zoneLabelDeprecated)
+			k, pvFailureDomainZone = zoneFromPVNodeAffinity(pv, awsEbsCsiZoneKey, azureCsiZoneKey, gkeCsiZoneKey, cinderCsiZoneKey, manilaCsiZoneKey, zoneLabel, zoneLabelDeprecated)
 			if pvFailureDomainZone != "" {
 				log.Infof("zone info from nodeAffinity requirements: %s, key: %s", pvFailureDomainZone, k)
 			} else {
@@ -545,7 +549,8 @@ func (ib *itemBackupper) takePVSnapshot(obj runtime.Unstructured, log logrus.Fie
 	}
 
 	if volumeSnapshotter == nil {
-		log.Info("Persistent volume is not a supported volume type for snapshots, skipping.")
+		// the PV may still has change to be snapshotted by CSI plugin's `PVCBackupItemAction` in PVC backup logic
+		log.Info("Persistent volume is not a supported volume type for Velero-native volumeSnapshotter snapshot, skipping.")
 		return nil
 	}
 

pkg/builder/data_download_builder.go

@@ -0,0 +1,112 @@
+/*
+Copyright The Velero Contributors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package builder
+
+import (
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	velerov2alpha1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1"
+)
+
+// DataDownloadBuilder builds DataDownload objects.
+type DataDownloadBuilder struct {
+	object *velerov2alpha1api.DataDownload
+}
+
+// ForDataDownload is the constructor of DataDownloadBuilder
+func ForDataDownload(namespace, name string) *DataDownloadBuilder {
+	return &DataDownloadBuilder{
+		object: &velerov2alpha1api.DataDownload{
+			TypeMeta: metav1.TypeMeta{
+				Kind:       "DataDownload",
+				APIVersion: velerov2alpha1api.SchemeGroupVersion.String(),
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      name,
+				Namespace: namespace,
+			},
+		},
+	}
+}
+
+// Result returns the built DataDownload.
+func (d *DataDownloadBuilder) Result() *velerov2alpha1api.DataDownload {
+	return d.object
+}
+
+// BackupStorageLocation sets the DataDownload's backup storage location.
+func (d *DataDownloadBuilder) BackupStorageLocation(name string) *DataDownloadBuilder {
+	d.object.Spec.BackupStorageLocation = name
+	return d
+}
+
+// Phase sets the DataDownload's phase.
+func (d *DataDownloadBuilder) Phase(phase velerov2alpha1api.DataDownloadPhase) *DataDownloadBuilder {
+	d.object.Status.Phase = phase
+	return d
+}
+
+// SnapshotID sets the DataDownload's SnapshotID.
+func (d *DataDownloadBuilder) SnapshotID(id string) *DataDownloadBuilder {
+	d.object.Spec.SnapshotID = id
+	return d
+}
+
+// DataMover sets the DataDownload's DataMover.
+func (d *DataDownloadBuilder) DataMover(dataMover string) *DataDownloadBuilder {
+	d.object.Spec.DataMover = dataMover
+	return d
+}
+
+// SourceNamespace sets the DataDownload's SourceNamespace.
+func (d *DataDownloadBuilder) SourceNamespace(sourceNamespace string) *DataDownloadBuilder {
+	d.object.Spec.SourceNamespace = sourceNamespace
+	return d
+}
+
+// TargetVolume sets the DataDownload's TargetVolume.
+func (d *DataDownloadBuilder) TargetVolume(targetVolume velerov2alpha1api.TargetVolumeSpec) *DataDownloadBuilder {
+	d.object.Spec.TargetVolume = targetVolume
+	return d
+}
+
+// Cancel sets the DataDownload's Cancel.
+func (d *DataDownloadBuilder) Cancel(cancel bool) *DataDownloadBuilder {
+	d.object.Spec.Cancel = cancel
+	return d
+}
+
+// OperationTimeout sets the DataDownload's OperationTimeout.
+func (d *DataDownloadBuilder) OperationTimeout(timeout metav1.Duration) *DataDownloadBuilder {
+	d.object.Spec.OperationTimeout = timeout
+	return d
+}
+
+// DataMoverConfig sets the DataDownload's DataMoverConfig.
+func (d *DataDownloadBuilder) DataMoverConfig(config *map[string]string) *DataDownloadBuilder {
+	d.object.Spec.DataMoverConfig = *config
+	return d
+}
+
+// ObjectMeta applies functional options to the DataDownload's ObjectMeta.
+func (d *DataDownloadBuilder) ObjectMeta(opts ...ObjectMetaOpt) *DataDownloadBuilder {
+	for _, opt := range opts {
+		opt(d.object)
+	}
+
+	return d
+}

pkg/builder/delete_backup_request_builder.go

@@ -0,0 +1,59 @@
+package builder
+
+import (
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
+)
+
+// DeleteBackupRequestBuilder builds DeleteBackupRequest objects
+type DeleteBackupRequestBuilder struct {
+	object *velerov1api.DeleteBackupRequest
+}
+
+// ForDeleteBackupRequest is the constructor for a DeleteBackupRequestBuilder.
+func ForDeleteBackupRequest(ns, name string) *DeleteBackupRequestBuilder {
+	return &DeleteBackupRequestBuilder{
+		object: &velerov1api.DeleteBackupRequest{
+			TypeMeta: metav1.TypeMeta{
+				APIVersion: velerov1api.SchemeGroupVersion.String(),
+				Kind:       "DeleteBackupRequest",
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Namespace: ns,
+				Name:      name,
+			},
+		},
+	}
+}
+
+// Result returns the built DeleteBackupRequest.
+func (b *DeleteBackupRequestBuilder) Result() *velerov1api.DeleteBackupRequest {
+	return b.object
+}
+
+// ObjectMeta applies functional options to the DeleteBackupRequest's ObjectMeta.
+func (b *DeleteBackupRequestBuilder) ObjectMeta(opts ...ObjectMetaOpt) *DeleteBackupRequestBuilder {
+	for _, opt := range opts {
+		opt(b.object)
+	}
+	return b
+}
+
+// BackupName sets the DeleteBackupRequest's backup name.
+func (b *DeleteBackupRequestBuilder) BackupName(name string) *DeleteBackupRequestBuilder {
+	b.object.Spec.BackupName = name
+	return b
+}
+
+// Phase sets the DeleteBackupRequest's phase.
+func (b *DeleteBackupRequestBuilder) Phase(phase velerov1api.DeleteBackupRequestPhase) *DeleteBackupRequestBuilder {
+	b.object.Status.Phase = phase
+	return b
+}
+
+// Errors sets the DeleteBackupRequest's errors.
+func (b *DeleteBackupRequestBuilder) Errors(errors ...string) *DeleteBackupRequestBuilder {
+	b.object.Status.Errors = errors
+	return b
+}

pkg/builder/item_operation_builder.go

@@ -0,0 +1,193 @@
+package builder
+
+import (
+	"time"
+
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+
+	"github.com/vmware-tanzu/velero/pkg/itemoperation"
+	"github.com/vmware-tanzu/velero/pkg/plugin/velero"
+)
+
+// OperationStatusBuilder builds OperationStatus objects
+type OperationStatusBuilder struct {
+	object *itemoperation.OperationStatus
+}
+
+// ForOperationStatus is the constructor for a OperationStatusBuilder.
+func ForOperationStatus() *OperationStatusBuilder {
+	return &OperationStatusBuilder{
+		object: &itemoperation.OperationStatus{},
+	}
+}
+
+// Result returns the built OperationStatus.
+func (osb *OperationStatusBuilder) Result() *itemoperation.OperationStatus {
+	return osb.object
+}
+
+// Phase sets the OperationStatus's phase.
+func (osb *OperationStatusBuilder) Phase(phase itemoperation.OperationPhase) *OperationStatusBuilder {
+	osb.object.Phase = phase
+	return osb
+}
+
+// Error sets the OperationStatus's error.
+func (osb *OperationStatusBuilder) Error(err string) *OperationStatusBuilder {
+	osb.object.Error = err
+	return osb
+}
+
+// Progress sets the OperationStatus's progress.
+func (osb *OperationStatusBuilder) Progress(nComplete int64, nTotal int64, operationUnits string) *OperationStatusBuilder {
+	osb.object.NCompleted = nComplete
+	osb.object.NTotal = nTotal
+	osb.object.OperationUnits = operationUnits
+	return osb
+}
+
+// Description sets the OperationStatus's description.
+func (osb *OperationStatusBuilder) Description(desc string) *OperationStatusBuilder {
+	osb.object.Description = desc
+	return osb
+}
+
+// Created sets the OperationStatus's creation timestamp.
+func (osb *OperationStatusBuilder) Created(t time.Time) *OperationStatusBuilder {
+	osb.object.Created = &metav1.Time{Time: t}
+	return osb
+}
+
+// Updated sets the OperationStatus's last update timestamp.
+func (osb *OperationStatusBuilder) Updated(t time.Time) *OperationStatusBuilder {
+	osb.object.Updated = &metav1.Time{Time: t}
+	return osb
+}
+
+// Started sets the OperationStatus's start timestamp.
+func (osb *OperationStatusBuilder) Started(t time.Time) *OperationStatusBuilder {
+	osb.object.Started = &metav1.Time{Time: t}
+	return osb
+}
+
+// BackupOperationBuilder builds BackupOperation objects
+type BackupOperationBuilder struct {
+	object *itemoperation.BackupOperation
+}
+
+// ForBackupOperation is the constructor for a BackupOperationBuilder.
+func ForBackupOperation() *BackupOperationBuilder {
+	return &BackupOperationBuilder{
+		object: &itemoperation.BackupOperation{},
+	}
+}
+
+// Result returns the built BackupOperation.
+func (bb *BackupOperationBuilder) Result() *itemoperation.BackupOperation {
+	return bb.object
+}
+
+// BackupName sets the BackupOperation's backup name.
+func (bb *BackupOperationBuilder) BackupName(name string) *BackupOperationBuilder {
+	bb.object.Spec.BackupName = name
+	return bb
+}
+
+// OperationID sets the BackupOperation's operation ID.
+func (bb *BackupOperationBuilder) OperationID(id string) *BackupOperationBuilder {
+	bb.object.Spec.OperationID = id
+	return bb
+}
+
+// Status sets the BackupOperation's status.
+func (bb *BackupOperationBuilder) Status(status itemoperation.OperationStatus) *BackupOperationBuilder {
+	bb.object.Status = status
+	return bb
+}
+
+// ResourceIdentifier sets the BackupOperation's resource identifier.
+func (bb *BackupOperationBuilder) ResourceIdentifier(group, resource, ns, name string) *BackupOperationBuilder {
+	bb.object.Spec.ResourceIdentifier = velero.ResourceIdentifier{
+		GroupResource: schema.GroupResource{
+			Group:    group,
+			Resource: resource,
+		},
+		Namespace: ns,
+		Name:      name,
+	}
+	return bb
+}
+
+// BackupItemAction sets the BackupOperation's backup item action.
+func (bb *BackupOperationBuilder) BackupItemAction(bia string) *BackupOperationBuilder {
+	bb.object.Spec.BackupItemAction = bia
+	return bb
+}
+
+// PostOperationItem adds a post-operation item to the BackupOperation's list of post-operation items.
+func (bb *BackupOperationBuilder) PostOperationItem(group, resource, ns, name string) *BackupOperationBuilder {
+	bb.object.Spec.PostOperationItems = append(bb.object.Spec.PostOperationItems, velero.ResourceIdentifier{
+		GroupResource: schema.GroupResource{
+			Group:    group,
+			Resource: resource,
+		},
+		Namespace: ns,
+		Name:      name,
+	})
+	return bb
+}
+
+// RestoreOperationBuilder builds RestoreOperation objects
+type RestoreOperationBuilder struct {
+	object *itemoperation.RestoreOperation
+}
+
+// ForRestoreOperation is the constructor for a RestoreOperationBuilder.
+func ForRestoreOperation() *RestoreOperationBuilder {
+	return &RestoreOperationBuilder{
+		object: &itemoperation.RestoreOperation{},
+	}
+}
+
+// Result returns the built RestoreOperation.
+func (rb *RestoreOperationBuilder) Result() *itemoperation.RestoreOperation {
+	return rb.object
+}
+
+// RestoreName sets the RestoreOperation's restore name.
+func (rb *RestoreOperationBuilder) RestoreName(name string) *RestoreOperationBuilder {
+	rb.object.Spec.RestoreName = name
+	return rb
+}
+
+// OperationID sets the RestoreOperation's operation ID.
+func (rb *RestoreOperationBuilder) OperationID(id string) *RestoreOperationBuilder {
+	rb.object.Spec.OperationID = id
+	return rb
+}
+
+// RestoreItemAction sets the RestoreOperation's restore item action.
+func (rb *RestoreOperationBuilder) RestoreItemAction(ria string) *RestoreOperationBuilder {
+	rb.object.Spec.RestoreItemAction = ria
+	return rb
+}
+
+// Status sets the RestoreOperation's status.
+func (rb *RestoreOperationBuilder) Status(status itemoperation.OperationStatus) *RestoreOperationBuilder {
+	rb.object.Status = status
+	return rb
+}
+
+// ResourceIdentifier sets the RestoreOperation's resource identifier.
+func (rb *RestoreOperationBuilder) ResourceIdentifier(group, resource, ns, name string) *RestoreOperationBuilder {
+	rb.object.Spec.ResourceIdentifier = velero.ResourceIdentifier{
+		GroupResource: schema.GroupResource{
+			Group:    group,
+			Resource: resource,
+		},
+		Namespace: ns,
+		Name:      name,
+	}
+	return rb
+}

pkg/builder/object_meta.go

@@ -153,3 +153,17 @@ func WithManagedFields(val []metav1.ManagedFieldsEntry) func(obj metav1.Object)
 		obj.SetManagedFields(val)
 	}
 }
+
+// WithCreationTimestamp is a functional option that applies the specified creationTimestamp
+func WithCreationTimestamp(t time.Time) func(obj metav1.Object) {
+	return func(obj metav1.Object) {
+		obj.SetCreationTimestamp(metav1.Time{Time: t})
+	}
+}
+
+// WithOwnerReference is a functional option that applies the specified OwnerReference to an object.
+func WithOwnerReference(val []metav1.OwnerReference) func(obj metav1.Object) {
+	return func(obj metav1.Object) {
+		obj.SetOwnerReferences(val)
+	}
+}

pkg/builder/persistent_volume_claim_builder.go

@@ -18,6 +18,7 @@ package builder
 
 import (
 	corev1api "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
@@ -73,3 +74,39 @@ func (b *PersistentVolumeClaimBuilder) Phase(phase corev1api.PersistentVolumeCla
 	b.object.Status.Phase = phase
 	return b
 }
+
+// RequestResource sets the PersistentVolumeClaim's spec.Resources.Requests.
+func (b *PersistentVolumeClaimBuilder) RequestResource(requests corev1api.ResourceList) *PersistentVolumeClaimBuilder {
+	if b.object.Spec.Resources.Requests == nil {
+		b.object.Spec.Resources.Requests = make(map[corev1api.ResourceName]resource.Quantity)
+	}
+	b.object.Spec.Resources.Requests = requests
+	return b
+}
+
+// LimitResource sets the PersistentVolumeClaim's spec.Resources.Limits.
+func (b *PersistentVolumeClaimBuilder) LimitResource(limits corev1api.ResourceList) *PersistentVolumeClaimBuilder {
+	if b.object.Spec.Resources.Limits == nil {
+		b.object.Spec.Resources.Limits = make(map[corev1api.ResourceName]resource.Quantity)
+	}
+	b.object.Spec.Resources.Limits = limits
+	return b
+}
+
+// DataSource sets the PersistentVolumeClaim's spec.DataSource.
+func (b *PersistentVolumeClaimBuilder) DataSource(dataSource *corev1api.TypedLocalObjectReference) *PersistentVolumeClaimBuilder {
+	b.object.Spec.DataSource = dataSource
+	return b
+}
+
+// DataSourceRef sets the PersistentVolumeClaim's spec.DataSourceRef.
+func (b *PersistentVolumeClaimBuilder) DataSourceRef(dataSourceRef *corev1api.TypedLocalObjectReference) *PersistentVolumeClaimBuilder {
+	b.object.Spec.DataSourceRef = dataSourceRef
+	return b
+}
+
+// Selector sets the PersistentVolumeClaim's spec.Selector.
+func (b *PersistentVolumeClaimBuilder) Selector(labelSelector *metav1.LabelSelector) *PersistentVolumeClaimBuilder {
+	b.object.Spec.Selector = labelSelector
+	return b
+}

pkg/builder/pod_volume_backup_builder.go

@@ -53,7 +53,6 @@ func (b *PodVolumeBackupBuilder) ObjectMeta(opts ...ObjectMetaOpt) *PodVolumeBac
 	for _, opt := range opts {
 		opt(b.object)
 	}
-
 	return b
 }
 

pkg/builder/pod_volume_restore_builder.go

@@ -0,0 +1,83 @@
+package builder
+
+import (
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
+)
+
+// PodVolumeRestoreBuilder builds PodVolumeRestore objects.
+type PodVolumeRestoreBuilder struct {
+	object *velerov1api.PodVolumeRestore
+}
+
+// ForPodVolumeRestore is the constructor for a PodVolumeRestoreBuilder.
+func ForPodVolumeRestore(ns, name string) *PodVolumeRestoreBuilder {
+	return &PodVolumeRestoreBuilder{
+		object: &velerov1api.PodVolumeRestore{
+			TypeMeta: metav1.TypeMeta{
+				APIVersion: velerov1api.SchemeGroupVersion.String(),
+				Kind:       "PodVolumeRestore",
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Namespace: ns,
+				Name:      name,
+			},
+		},
+	}
+}
+
+// Result returns the built PodVolumeRestore.
+func (b *PodVolumeRestoreBuilder) Result() *velerov1api.PodVolumeRestore {
+	return b.object
+}
+
+// ObjectMeta applies functional options to the PodVolumeRestore's ObjectMeta.
+func (b *PodVolumeRestoreBuilder) ObjectMeta(opts ...ObjectMetaOpt) *PodVolumeRestoreBuilder {
+	for _, opt := range opts {
+		opt(b.object)
+	}
+	return b
+}
+
+// Phase sets the PodVolumeRestore's phase.
+func (b *PodVolumeRestoreBuilder) Phase(phase velerov1api.PodVolumeRestorePhase) *PodVolumeRestoreBuilder {
+	b.object.Status.Phase = phase
+	return b
+}
+
+// BackupStorageLocation sets the PodVolumeRestore's backup storage location.
+func (b *PodVolumeRestoreBuilder) BackupStorageLocation(name string) *PodVolumeRestoreBuilder {
+	b.object.Spec.BackupStorageLocation = name
+	return b
+}
+
+// SnapshotID sets the PodVolumeRestore's snapshot ID.
+func (b *PodVolumeRestoreBuilder) SnapshotID(snapshotID string) *PodVolumeRestoreBuilder {
+	b.object.Spec.SnapshotID = snapshotID
+	return b
+}
+
+// PodName sets the name of the pod associated with this PodVolumeRestore.
+func (b *PodVolumeRestoreBuilder) PodName(name string) *PodVolumeRestoreBuilder {
+	b.object.Spec.Pod.Name = name
+	return b
+}
+
+// PodNamespace sets the name of the pod associated with this PodVolumeRestore.
+func (b *PodVolumeRestoreBuilder) PodNamespace(ns string) *PodVolumeRestoreBuilder {
+	b.object.Spec.Pod.Namespace = ns
+	return b
+}
+
+// Volume sets the name of the volume associated with this PodVolumeRestore.
+func (b *PodVolumeRestoreBuilder) Volume(volume string) *PodVolumeRestoreBuilder {
+	b.object.Spec.Volume = volume
+	return b
+}
+
+// UploaderType sets the type of uploader to use for this PodVolumeRestore.
+func (b *PodVolumeRestoreBuilder) UploaderType(uploaderType string) *PodVolumeRestoreBuilder {
+	b.object.Spec.UploaderType = uploaderType
+	return b
+}

pkg/cmd/cli/nodeagent/server.go

@@ -260,6 +260,10 @@ func (s *nodeAgentServer) run() {
 		s.logger.WithError(err).Fatal("Unable to create the data upload controller")
 	}
 
+	if err = controller.NewDataDownloadReconciler(s.mgr.GetClient(), s.kubeClient, repoEnsurer, credentialGetter, s.nodeName, s.logger).SetupWithManager(s.mgr); err != nil {
+		s.logger.WithError(err).Fatal("Unable to create the data download controller")
+	}
+
 	s.logger.Info("Controllers starting...")
 
 	if err := s.mgr.Start(ctrl.SetupSignalHandler()); err != nil {

pkg/cmd/server/plugin/plugin.go

@@ -22,11 +22,10 @@ import (
 	apiextensions "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
 
 	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
-	"github.com/vmware-tanzu/velero/pkg/features"
-
 	"github.com/vmware-tanzu/velero/pkg/backup"
 	"github.com/vmware-tanzu/velero/pkg/client"
 	velerodiscovery "github.com/vmware-tanzu/velero/pkg/discovery"
+	"github.com/vmware-tanzu/velero/pkg/features"
 	veleroplugin "github.com/vmware-tanzu/velero/pkg/plugin/framework"
 	plugincommon "github.com/vmware-tanzu/velero/pkg/plugin/framework/common"
 	"github.com/vmware-tanzu/velero/pkg/restore"
@@ -59,7 +58,8 @@ func NewCommand(f client.Factory) *cobra.Command {
 				RegisterRestoreItemAction("velero.io/change-pvc-node-selector", newChangePVCNodeSelectorItemAction(f)).
 				RegisterRestoreItemAction("velero.io/apiservice", newAPIServiceRestoreItemAction).
 				RegisterRestoreItemAction("velero.io/admission-webhook-configuration", newAdmissionWebhookConfigurationAction).
-				RegisterRestoreItemAction("velero.io/secret", newSecretRestoreItemAction(f))
+				RegisterRestoreItemAction("velero.io/secret", newSecretRestoreItemAction(f)).
+				RegisterRestoreItemAction("velero.io/dataupload", newDataUploadRetrieveAction(f))
 			if !features.IsEnabled(velerov1api.APIGroupVersionsFeatureFlag) {
 				// Do not register crd-remap-version BIA if the API Group feature flag is enabled, so that the v1 CRD can be backed up
 				pluginServer = pluginServer.RegisterBackupItemAction("velero.io/crd-remap-version", newRemapCRDVersionAction(f))
@@ -245,3 +245,13 @@ func newSecretRestoreItemAction(f client.Factory) plugincommon.HandlerInitialize
 		return restore.NewSecretAction(logger, client), nil
 	}
 }
+
+func newDataUploadRetrieveAction(f client.Factory) plugincommon.HandlerInitializer {
+	return func(logger logrus.FieldLogger) (interface{}, error) {
+		client, err := f.KubeClient()
+		if err != nil {
+			return nil, err
+		}
+		return restore.NewDataUploadRetrieveAction(logger, client.CoreV1().ConfigMaps(f.Namespace())), nil
+	}
+}

pkg/cmd/server/server.go

@@ -521,6 +521,7 @@ High priorities:
   - VolumeSnapshotContents are needed as they contain the handle to the volume snapshot in the
     storage provider
   - VolumeSnapshots are needed to create PVCs using the VolumeSnapshot as their data source.
+  - DataUploads need to restore before PVC for Snapshot DataMover to work, because PVC needs the DataUploadResults to create DataDownloads.
   - PVs go before PVCs because PVCs depend on them.
   - PVCs go before pods or controllers so they can be mounted as volumes.
   - Service accounts go before secrets so service account token secrets can be filled automatically.
@@ -551,6 +552,7 @@ var defaultRestorePriorities = restore.Priorities{
 		"volumesnapshotclass.snapshot.storage.k8s.io",
 		"volumesnapshotcontents.snapshot.storage.k8s.io",
 		"volumesnapshots.snapshot.storage.k8s.io",
+		"datauploads.velero.io",
 		"persistentvolumes",
 		"persistentvolumeclaims",
 		"serviceaccounts",

pkg/cmd/util/output/backup_describer.go

@@ -231,31 +231,31 @@ func DescribeBackupSpec(d *Describer, spec velerov1api.BackupSpec) {
 			d.Printf("\t\t%s:\n", backupResourceHookSpec.Name)
 			d.Printf("\t\t\tNamespaces:\n")
 			var s string
-			if len(spec.IncludedNamespaces) == 0 {
+			if len(backupResourceHookSpec.IncludedNamespaces) == 0 {
 				s = "*"
 			} else {
-				s = strings.Join(spec.IncludedNamespaces, ", ")
+				s = strings.Join(backupResourceHookSpec.IncludedNamespaces, ", ")
 			}
 			d.Printf("\t\t\t\tIncluded:\t%s\n", s)
-			if len(spec.ExcludedNamespaces) == 0 {
+			if len(backupResourceHookSpec.ExcludedNamespaces) == 0 {
 				s = emptyDisplay
 			} else {
-				s = strings.Join(spec.ExcludedNamespaces, ", ")
+				s = strings.Join(backupResourceHookSpec.ExcludedNamespaces, ", ")
 			}
 			d.Printf("\t\t\t\tExcluded:\t%s\n", s)
 
 			d.Println()
 			d.Printf("\t\t\tResources:\n")
-			if len(spec.IncludedResources) == 0 {
+			if len(backupResourceHookSpec.IncludedResources) == 0 {
 				s = "*"
 			} else {
-				s = strings.Join(spec.IncludedResources, ", ")
+				s = strings.Join(backupResourceHookSpec.IncludedResources, ", ")
 			}
 			d.Printf("\t\t\t\tIncluded:\t%s\n", s)
-			if len(spec.ExcludedResources) == 0 {
+			if len(backupResourceHookSpec.ExcludedResources) == 0 {
 				s = emptyDisplay
 			} else {
-				s = strings.Join(spec.ExcludedResources, ", ")
+				s = strings.Join(backupResourceHookSpec.ExcludedResources, ", ")
 			}
 			d.Printf("\t\t\t\tExcluded:\t%s\n", s)
 

pkg/cmd/util/output/backup_describer_test.go

@@ -6,6 +6,10 @@ import (
 	"text/tabwriter"
 	"time"
 
+	"github.com/vmware-tanzu/velero/pkg/itemoperation"
+
+	"github.com/stretchr/testify/require"
+
 	snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v4/apis/volumesnapshot/v1"
 	"github.com/stretchr/testify/assert"
 	v1 "k8s.io/api/core/v1"
@@ -46,7 +50,35 @@ func TestDescribeBackupSpec(t *testing.T) {
 		TTL(72 * time.Hour).
 		CSISnapshotTimeout(10 * time.Minute).
 		DataMover("mover").
-		Result().Spec
+		Hooks(velerov1api.BackupHooks{
+			Resources: []velerov1api.BackupResourceHookSpec{
+				{
+					Name: "hook-1",
+					PreHooks: []velerov1api.BackupResourceHook{
+						{
+							Exec: &velerov1api.ExecHook{
+								Container: "hook-container-1",
+								Command:   []string{"pre"},
+								OnError:   velerov1api.HookErrorModeContinue,
+							},
+						},
+					},
+					PostHooks: []velerov1api.BackupResourceHook{
+						{
+							Exec: &velerov1api.ExecHook{
+								Container: "hook-container-1",
+								Command:   []string{"post"},
+								OnError:   velerov1api.HookErrorModeContinue,
+							},
+						},
+					},
+					IncludedNamespaces: []string{"hook-inc-ns-1", "hook-inc-ns-2"},
+					ExcludedNamespaces: []string{"hook-exc-ns-1", "hook-exc-ns-2"},
+					IncludedResources:  []string{"hook-inc-res-1", "hook-inc-res-2"},
+					ExcludedResources:  []string{"hook-exc-res-1", "hook-exc-res-2"},
+				},
+			},
+		}).Result().Spec
 
 	expect1 := `Namespaces:
   Included:  inc-ns-1, inc-ns-2
@@ -70,7 +102,30 @@ TTL:  72h0m0s
 CSISnapshotTimeout:    10m0s
 ItemOperationTimeout:  0s
 
-Hooks:  <none>
+Hooks:
+  Resources:
+    hook-1:
+      Namespaces:
+        Included:  hook-inc-ns-1, hook-inc-ns-2
+        Excluded:  hook-exc-ns-1, hook-exc-ns-2
+
+      Resources:
+        Included:  hook-inc-res-1, hook-inc-res-2
+        Excluded:  hook-exc-res-1, hook-exc-res-2
+
+      Label selector:  <none>
+
+      Pre Exec Hook:
+        Container:  hook-container-1
+        Command:    pre
+        On Error:   Continue
+        Timeout:    0s
+
+      Post Exec Hook:
+        Container:  hook-container-1
+        Command:    post
+        On Error:   Continue
+        Timeout:    0s
 `
 
 	input2 := builder.ForBackup("test-ns", "test-backup-2").
@@ -112,13 +167,94 @@ ItemOperationTimeout:  0s
 Hooks:  <none>
 `
 
+	input3 := builder.ForBackup("test-ns", "test-backup-3").
+		StorageLocation("backup-location").
+		OrderedResources(map[string]string{
+			"kind1": "rs1-1, rs1-2",
+		}).Hooks(velerov1api.BackupHooks{
+		Resources: []velerov1api.BackupResourceHookSpec{
+			{
+				Name: "hook-1",
+				PreHooks: []velerov1api.BackupResourceHook{
+					{
+						Exec: &velerov1api.ExecHook{
+							Container: "hook-container-1",
+							Command:   []string{"pre"},
+							OnError:   velerov1api.HookErrorModeContinue,
+						},
+					},
+				},
+				PostHooks: []velerov1api.BackupResourceHook{
+					{
+						Exec: &velerov1api.ExecHook{
+							Container: "hook-container-1",
+							Command:   []string{"post"},
+							OnError:   velerov1api.HookErrorModeContinue,
+						},
+					},
+				},
+			},
+		},
+	}).Result().Spec
+
+	expect3 := `Namespaces:
+  Included:  *
+  Excluded:  <none>
+
+Resources:
+  Included:        *
+  Excluded:        <none>
+  Cluster-scoped:  auto
+
+Label selector:  <none>
+
+Storage Location:  backup-location
+
+Velero-Native Snapshot PVs:  auto
+Snapshot Move Data:          auto
+Data Mover:                  <none>
+
+TTL:  0s
+
+CSISnapshotTimeout:    0s
+ItemOperationTimeout:  0s
+
+Hooks:
+  Resources:
+    hook-1:
+      Namespaces:
+        Included:  *
+        Excluded:  <none>
+
+      Resources:
+        Included:  *
+        Excluded:  <none>
+
+      Label selector:  <none>
+
+      Pre Exec Hook:
+        Container:  hook-container-1
+        Command:    pre
+        On Error:   Continue
+        Timeout:    0s
+
+      Post Exec Hook:
+        Container:  hook-container-1
+        Command:    post
+        On Error:   Continue
+        Timeout:    0s
+
+OrderedResources:
+  kind1: rs1-1, rs1-2
+`
+
 	testcases := []struct {
 		name   string
 		input  velerov1api.BackupSpec
 		expect string
 	}{
 		{
-			name:   "old resource filter",
+			name:   "old resource filter with hooks",
 			input:  input1,
 			expect: expect1,
 		},
@@ -127,6 +263,11 @@ Hooks:  <none>
 			input:  input2,
 			expect: expect2,
 		},
+		{
+			name:   "old resource filter with hooks and ordered resources",
+			input:  input3,
+			expect: expect3,
+		},
 	}
 
 	for _, tc := range testcases {
@@ -164,7 +305,6 @@ func TestDescribeSnapshot(t *testing.T) {
 
 func TestDescribePodVolumeBackups(t *testing.T) {
 	pvb1 := builder.ForPodVolumeBackup("test-ns", "test-pvb1").
-		BackupStorageLocation("backup-location").
 		UploaderType("kopia").
 		Phase(velerov1api.PodVolumeBackupPhaseCompleted).
 		BackupStorageLocation("bsl-1").
@@ -173,7 +313,6 @@ func TestDescribePodVolumeBackups(t *testing.T) {
 		PodNamespace("pod-ns-1").
 		SnapshotID("snap-1").Result()
 	pvb2 := builder.ForPodVolumeBackup("test-ns1", "test-pvb2").
-		BackupStorageLocation("backup-location").
 		UploaderType("kopia").
 		Phase(velerov1api.PodVolumeBackupPhaseCompleted).
 		BackupStorageLocation("bsl-1").
@@ -289,3 +428,99 @@ Snapshot Content Name: vsc-1
 		})
 	}
 }
+
+func TestDescribeDeleteBackupRequests(t *testing.T) {
+	t1, err1 := time.Parse("2006-Jan-02", "2023-Jun-26")
+	require.Nil(t, err1)
+	dbr1 := builder.ForDeleteBackupRequest("velero", "dbr1").
+		ObjectMeta(builder.WithCreationTimestamp(t1)).
+		BackupName("bak-1").
+		Phase(velerov1api.DeleteBackupRequestPhaseProcessed).
+		Errors("some error").Result()
+	t2, err2 := time.Parse("2006-Jan-02", "2023-Jun-25")
+	require.Nil(t, err2)
+	dbr2 := builder.ForDeleteBackupRequest("velero", "dbr2").
+		ObjectMeta(builder.WithCreationTimestamp(t2)).
+		BackupName("bak-2").
+		Phase(velerov1api.DeleteBackupRequestPhaseInProgress).Result()
+
+	testcases := []struct {
+		name   string
+		input  []velerov1api.DeleteBackupRequest
+		expect string
+	}{
+		{
+			name:  "empty list",
+			input: []velerov1api.DeleteBackupRequest{},
+			expect: `Deletion Attempts:
+`,
+		},
+		{
+			name:  "list with one failed and one in-progress request",
+			input: []velerov1api.DeleteBackupRequest{*dbr1, *dbr2},
+			expect: `Deletion Attempts (1 failed):
+  2023-06-26 00:00:00 +0000 UTC: Processed
+  Errors:
+    some error
+
+  2023-06-25 00:00:00 +0000 UTC: InProgress
+`,
+		},
+	}
+	for _, tc := range testcases {
+		t.Run(tc.name, func(tt *testing.T) {
+			d := &Describer{
+				Prefix: "",
+				out:    &tabwriter.Writer{},
+				buf:    &bytes.Buffer{},
+			}
+			d.out.Init(d.buf, 0, 8, 2, ' ', 0)
+			DescribeDeleteBackupRequests(d, tc.input)
+			d.out.Flush()
+			assert.Equal(tt, tc.expect, d.buf.String())
+		})
+	}
+}
+
+func TestDescribeBackupItemOperation(t *testing.T) {
+	t1, err1 := time.Parse("2006-Jan-02", "2023-Jun-26")
+	require.Nil(t, err1)
+	t2, err2 := time.Parse("2006-Jan-02", "2023-Jun-25")
+	require.Nil(t, err2)
+	t3, err3 := time.Parse("2006-Jan-02", "2023-Jun-24")
+	require.Nil(t, err3)
+	input := builder.ForBackupOperation().
+		BackupName("backup-1").
+		OperationID("op-1").
+		BackupItemAction("action-1").
+		ResourceIdentifier("group", "rs-type", "ns", "rs-name").
+		Status(*builder.ForOperationStatus().
+			Phase(itemoperation.OperationPhaseFailed).
+			Error("operation error").
+			Progress(50, 100, "bytes").
+			Description("operation description").
+			Created(t3).
+			Started(t2).
+			Updated(t1).
+			Result()).Result()
+	expected := `  Operation for rs-type.group ns/rs-name:
+    Backup Item Action Plugin:  action-1
+    Operation ID:               op-1
+    Phase:                      Failed
+    Operation Error:            operation error
+    Progress:                   50 of 100 complete (bytes)
+    Progress description:       operation description
+    Created:                    2023-06-24 00:00:00 +0000 UTC
+    Started:                    2023-06-25 00:00:00 +0000 UTC
+    Updated:                    2023-06-26 00:00:00 +0000 UTC
+`
+	d := &Describer{
+		Prefix: "",
+		out:    &tabwriter.Writer{},
+		buf:    &bytes.Buffer{},
+	}
+	d.out.Init(d.buf, 0, 8, 2, ' ', 0)
+	describeBackupItemOperation(d, input)
+	d.out.Flush()
+	assert.Equal(t, expected, d.buf.String())
+}

pkg/cmd/util/output/backup_structured_describer.go

@@ -160,31 +160,31 @@ func DescribeBackupSpecInSF(d *StructuredDescriber, spec velerov1api.BackupSpec)
 		ResourceDetails := make(map[string]interface{})
 		var s string
 		namespaceInfo := make(map[string]string)
-		if len(spec.IncludedNamespaces) == 0 {
+		if len(backupResourceHookSpec.IncludedNamespaces) == 0 {
 			s = "*"
 		} else {
-			s = strings.Join(spec.IncludedNamespaces, ", ")
+			s = strings.Join(backupResourceHookSpec.IncludedNamespaces, ", ")
 		}
 		namespaceInfo["included"] = s
-		if len(spec.ExcludedNamespaces) == 0 {
+		if len(backupResourceHookSpec.ExcludedNamespaces) == 0 {
 			s = emptyDisplay
 		} else {
-			s = strings.Join(spec.ExcludedNamespaces, ", ")
+			s = strings.Join(backupResourceHookSpec.ExcludedNamespaces, ", ")
 		}
 		namespaceInfo["excluded"] = s
 		ResourceDetails["namespaces"] = namespaceInfo
 
 		resourcesInfo := make(map[string]string)
-		if len(spec.IncludedResources) == 0 {
+		if len(backupResourceHookSpec.IncludedResources) == 0 {
 			s = "*"
 		} else {
-			s = strings.Join(spec.IncludedResources, ", ")
+			s = strings.Join(backupResourceHookSpec.IncludedResources, ", ")
 		}
 		resourcesInfo["included"] = s
-		if len(spec.ExcludedResources) == 0 {
+		if len(backupResourceHookSpec.ExcludedResources) == 0 {
 			s = emptyDisplay
 		} else {
-			s = strings.Join(spec.ExcludedResources, ", ")
+			s = strings.Join(backupResourceHookSpec.ExcludedResources, ", ")
 		}
 		resourcesInfo["excluded"] = s
 		ResourceDetails["resources"] = resourcesInfo

pkg/cmd/util/output/backup_structured_describer_test.go

@@ -5,6 +5,8 @@ import (
 	"testing"
 	"time"
 
+	"github.com/stretchr/testify/require"
+
 	v1 "k8s.io/api/core/v1"
 
 	"github.com/vmware-tanzu/velero/pkg/features"
@@ -31,7 +33,36 @@ func TestDescribeBackupInSF(t *testing.T) {
 		StorageLocation("backup-location").
 		TTL(72 * time.Hour).
 		CSISnapshotTimeout(10 * time.Minute).
-		DataMover("mover")
+		DataMover("mover").
+		Hooks(velerov1api.BackupHooks{
+			Resources: []velerov1api.BackupResourceHookSpec{
+				{
+					Name: "hook-1",
+					PreHooks: []velerov1api.BackupResourceHook{
+						{
+							Exec: &velerov1api.ExecHook{
+								Container: "hook-container-1",
+								Command:   []string{"pre"},
+								OnError:   velerov1api.HookErrorModeContinue,
+							},
+						},
+					},
+					PostHooks: []velerov1api.BackupResourceHook{
+						{
+							Exec: &velerov1api.ExecHook{
+								Container: "hook-container-1",
+								Command:   []string{"post"},
+								OnError:   velerov1api.HookErrorModeContinue,
+							},
+						},
+					},
+					IncludedNamespaces: []string{"hook-inc-ns-1", "hook-inc-ns-2"},
+					ExcludedNamespaces: []string{"hook-exc-ns-1", "hook-exc-ns-2"},
+					IncludedResources:  []string{"hook-inc-res-1", "hook-inc-res-2"},
+					ExcludedResources:  []string{"hook-exc-res-1", "hook-exc-res-2"},
+				},
+			},
+		})
 
 	expect1 := map[string]interface{}{
 		"spec": map[string]interface{}{
@@ -51,13 +82,73 @@ func TestDescribeBackupInSF(t *testing.T) {
 			"TTL":                     "72h0m0s",
 			"CSISnapshotTimeout":      "10m0s",
 			"veleroSnapshotMoveData":  "auto",
+			"hooks": map[string]interface{}{
+				"resources": map[string]interface{}{
+					"hook-1": map[string]interface{}{
+						"labelSelector": emptyDisplay,
+						"namespaces": map[string]string{
+							"included": "hook-inc-ns-1, hook-inc-ns-2",
+							"excluded": "hook-exc-ns-1, hook-exc-ns-2",
+						},
+						"preExecHook": []map[string]interface{}{
+							{
+								"container": "hook-container-1",
+								"command":   "pre",
+								"onError:":  velerov1api.HookErrorModeContinue,
+								"timeout":   "0s",
+							},
+						},
+						"postExecHook": []map[string]interface{}{
+							{
+								"container": "hook-container-1",
+								"command":   "post",
+								"onError:":  velerov1api.HookErrorModeContinue,
+								"timeout":   "0s",
+							},
+						},
+						"resources": map[string]string{
+							"included": "hook-inc-res-1, hook-inc-res-2",
+							"excluded": "hook-exc-res-1, hook-exc-res-2",
+						},
+					},
+				},
+			},
 		},
 	}
 	DescribeBackupSpecInSF(sd, backupBuilder1.Result().Spec)
 	assert.True(t, reflect.DeepEqual(sd.output, expect1))
 
-	backupBuilder2 := builder.ForBackup("test-ns-2", "test-backup-2")
-	backupBuilder2.StorageLocation("backup-location")
+	backupBuilder2 := builder.ForBackup("test-ns-2", "test-backup-2").
+		StorageLocation("backup-location").
+		OrderedResources(map[string]string{
+			"kind1": "rs1-1, rs1-2",
+			"kind2": "rs2-1, rs2-2",
+		}).Hooks(velerov1api.BackupHooks{
+		Resources: []velerov1api.BackupResourceHookSpec{
+			{
+				Name: "hook-1",
+				PreHooks: []velerov1api.BackupResourceHook{
+					{
+						Exec: &velerov1api.ExecHook{
+							Container: "hook-container-1",
+							Command:   []string{"pre"},
+							OnError:   velerov1api.HookErrorModeContinue,
+						},
+					},
+				},
+				PostHooks: []velerov1api.BackupResourceHook{
+					{
+						Exec: &velerov1api.ExecHook{
+							Container: "hook-container-1",
+							Command:   []string{"post"},
+							OnError:   velerov1api.HookErrorModeContinue,
+						},
+					},
+				},
+			},
+		},
+	})
+
 	expect2 := map[string]interface{}{
 		"spec": map[string]interface{}{
 			"namespaces": map[string]interface{}{
@@ -76,6 +167,41 @@ func TestDescribeBackupInSF(t *testing.T) {
 			"TTL":                     "0s",
 			"CSISnapshotTimeout":      "0s",
 			"veleroSnapshotMoveData":  "auto",
+			"hooks": map[string]interface{}{
+				"resources": map[string]interface{}{
+					"hook-1": map[string]interface{}{
+						"labelSelector": emptyDisplay,
+						"namespaces": map[string]string{
+							"included": "*",
+							"excluded": emptyDisplay,
+						},
+						"preExecHook": []map[string]interface{}{
+							{
+								"container": "hook-container-1",
+								"command":   "pre",
+								"onError:":  velerov1api.HookErrorModeContinue,
+								"timeout":   "0s",
+							},
+						},
+						"postExecHook": []map[string]interface{}{
+							{
+								"container": "hook-container-1",
+								"command":   "post",
+								"onError:":  velerov1api.HookErrorModeContinue,
+								"timeout":   "0s",
+							},
+						},
+						"resources": map[string]string{
+							"included": "*",
+							"excluded": emptyDisplay,
+						},
+					},
+				},
+			},
+			"orderedResources": map[string]string{
+				"kind1": "rs1-1, rs1-2",
+				"kind2": "rs2-1, rs2-2",
+			},
 		},
 	}
 	DescribeBackupSpecInSF(sd, backupBuilder2.Result().Spec)
@@ -250,3 +376,83 @@ func TestDescribeBackupResultInSF(t *testing.T) {
 	describeResultInSF(got, input)
 	assert.True(t, reflect.DeepEqual(got, expect))
 }
+
+func TestDescribeDeleteBackupRequestsInSF(t *testing.T) {
+	t1, err1 := time.Parse("2006-Jan-02", "2023-Jun-26")
+	require.Nil(t, err1)
+	dbr1 := builder.ForDeleteBackupRequest("velero", "dbr1").
+		ObjectMeta(builder.WithCreationTimestamp(t1)).
+		BackupName("bak-1").
+		Phase(velerov1api.DeleteBackupRequestPhaseProcessed).
+		Errors("some error").Result()
+	t2, err2 := time.Parse("2006-Jan-02", "2023-Jun-25")
+	require.Nil(t, err2)
+	dbr2 := builder.ForDeleteBackupRequest("velero", "dbr2").
+		ObjectMeta(builder.WithCreationTimestamp(t2)).
+		BackupName("bak-2").
+		Phase(velerov1api.DeleteBackupRequestPhaseInProgress).Result()
+
+	testcases := []struct {
+		name   string
+		input  []velerov1api.DeleteBackupRequest
+		expect map[string]interface{}
+	}{
+		{
+			name:  "empty list",
+			input: []velerov1api.DeleteBackupRequest{},
+			expect: map[string]interface{}{
+				"deletionAttempts": map[string]interface{}{
+					"deleteBackupRequests": []map[string]interface{}{},
+				},
+			},
+		},
+		{
+			name:  "list with one failed and one in-progress request",
+			input: []velerov1api.DeleteBackupRequest{*dbr1, *dbr2},
+			expect: map[string]interface{}{
+				"deletionAttempts": map[string]interface{}{
+					"failed": int(1),
+					"deleteBackupRequests": []map[string]interface{}{
+						{
+							"creationTimestamp": t1.String(),
+							"phase":             velerov1api.DeleteBackupRequestPhaseProcessed,
+							"errors": []string{
+								"some error",
+							},
+						},
+						{
+							"creationTimestamp": t2.String(),
+							"phase":             velerov1api.DeleteBackupRequestPhaseInProgress,
+						},
+					},
+				},
+			},
+		},
+	}
+	for _, tc := range testcases {
+		t.Run(tc.name, func(tt *testing.T) {
+			sd := &StructuredDescriber{
+				output: make(map[string]interface{}),
+				format: "",
+			}
+			DescribeDeleteBackupRequestsInSF(sd, tc.input)
+			assert.True(tt, reflect.DeepEqual(sd.output, tc.expect))
+		})
+	}
+
+}
+
+func TestDescribeSnapshotInSF(t *testing.T) {
+	res := map[string]interface{}{}
+	iops := int64(100)
+	describeSnapshotInSF("pv-1", "snapshot-1", "ebs", "us-east-2", &iops, res)
+	expect := map[string]interface{}{
+		"pv-1": map[string]string{
+			"snapshotID":       "snapshot-1",
+			"type":             "ebs",
+			"availabilityZone": "us-east-2",
+			"IOPS":             "100",
+		},
+	}
+	assert.True(t, reflect.DeepEqual(expect, res))
+}

pkg/cmd/util/output/describe.go

@@ -47,15 +47,6 @@ func Describe(fn func(d *Describer)) string {
 	return d.buf.String()
 }
 
-func NewDescriber(minwidth, tabwidth, padding int, padchar byte, flags uint) *Describer {
-	d := &Describer{
-		out: new(tabwriter.Writer),
-		buf: new(bytes.Buffer),
-	}
-	d.out.Init(d.buf, minwidth, tabwidth, padding, padchar, flags)
-	return d
-}
-
 func (d *Describer) Printf(msg string, args ...interface{}) {
 	fmt.Fprint(d.out, d.Prefix)
 	fmt.Fprintf(d.out, msg, args...)

pkg/cmd/util/output/describe_test.go

@@ -3,6 +3,7 @@ package output
 import (
 	"bytes"
 	"fmt"
+	"reflect"
 	"testing"
 	"text/tabwriter"
 
@@ -132,3 +133,36 @@ func TestStructuredDescriber_JSONEncode(t *testing.T) {
 		})
 	}
 }
+
+func TestStructuredDescriber_DescribeMetadata(t *testing.T) {
+	d := NewStructuredDescriber("")
+	input := metav1.ObjectMeta{
+		Name:      "test",
+		Namespace: "test-ns",
+		Labels: map[string]string{
+			"label-1": "v1",
+			"label-2": "v2",
+		},
+		Annotations: map[string]string{
+			"annotation-1": "v1",
+			"annotation-2": "v2",
+		},
+	}
+	expect := map[string]interface{}{
+		"metadata": map[string]interface{}{
+			"name":      "test",
+			"namespace": "test-ns",
+			"labels": map[string]string{
+				"label-1": "v1",
+				"label-2": "v2",
+			},
+			"annotations": map[string]string{
+				"annotation-1": "v1",
+				"annotation-2": "v2",
+			},
+		},
+	}
+	d.DescribeMetadata(input)
+
+	assert.True(t, reflect.DeepEqual(expect, d.output))
+}

pkg/cmd/util/output/restore_describer_test.go

@@ -0,0 +1,183 @@
+package output
+
+import (
+	"bytes"
+	"testing"
+	"text/tabwriter"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
+	"github.com/vmware-tanzu/velero/pkg/builder"
+	"github.com/vmware-tanzu/velero/pkg/itemoperation"
+	"github.com/vmware-tanzu/velero/pkg/util/results"
+)
+
+func TestDescribeResult(t *testing.T) {
+	testcases := []struct {
+		name        string
+		inputName   string
+		inputResult results.Result
+		expect      string
+	}{
+		{
+			name:      "result without ns warns",
+			inputName: "restore-1",
+			inputResult: results.Result{
+				Velero:     []string{"velero-msg-1", "velero-msg-2"},
+				Cluster:    []string{"cluster-msg-1", "cluster-msg-2"},
+				Namespaces: map[string][]string{},
+			},
+			expect: `restore-1:
+  Velero:   velero-msg-1
+            velero-msg-2
+  Cluster:  cluster-msg-1
+            cluster-msg-2
+  Namespaces: <none>
+`,
+		},
+		{
+			name:      "result with ns warns",
+			inputName: "restore-2",
+			inputResult: results.Result{
+				Velero:  []string{"velero-msg-1", "velero-msg-2"},
+				Cluster: []string{"cluster-msg-1", "cluster-msg-2"},
+				Namespaces: map[string][]string{
+					"ns-1": {"ns-1-warn-1", "ns-1-warn-2"},
+				},
+			},
+			expect: `restore-2:
+  Velero:   velero-msg-1
+            velero-msg-2
+  Cluster:  cluster-msg-1
+            cluster-msg-2
+  Namespaces:
+    ns-1:  ns-1-warn-1
+           ns-1-warn-2
+`,
+		},
+	}
+	for _, tc := range testcases {
+		t.Run(tc.name, func(tt *testing.T) {
+			d := &Describer{
+				Prefix: "",
+				out:    &tabwriter.Writer{},
+				buf:    &bytes.Buffer{},
+			}
+			d.out.Init(d.buf, 0, 8, 2, ' ', 0)
+			describeResult(d, tc.inputName, tc.inputResult)
+			d.out.Flush()
+			assert.Equal(tt, tc.expect, d.buf.String())
+		})
+	}
+}
+
+func TestDescribeRestoreItemOperation(t *testing.T) {
+	t1, err1 := time.Parse("2006-Jan-02", "2023-Jun-26")
+	require.Nil(t, err1)
+	t2, err2 := time.Parse("2006-Jan-02", "2023-Jun-25")
+	require.Nil(t, err2)
+	t3, err3 := time.Parse("2006-Jan-02", "2023-Jun-24")
+	require.Nil(t, err3)
+	input := builder.ForRestoreOperation().
+		RestoreName("restore-1").
+		OperationID("op-1").
+		RestoreItemAction("action-1").
+		ResourceIdentifier("group", "rs-type", "ns", "rs-name").
+		Status(*builder.ForOperationStatus().
+			Phase(itemoperation.OperationPhaseFailed).
+			Error("operation error").
+			Progress(50, 100, "bytes").
+			Description("operation description").
+			Created(t3).
+			Started(t2).
+			Updated(t1).
+			Result()).Result()
+	expected := `  Operation for rs-type.group ns/rs-name:
+    Restore Item Action Plugin:  action-1
+    Operation ID:                op-1
+    Phase:                       Failed
+    Operation Error:             operation error
+    Progress:                    50 of 100 complete (bytes)
+    Progress description:        operation description
+    Created:                     2023-06-24 00:00:00 +0000 UTC
+    Started:                     2023-06-25 00:00:00 +0000 UTC
+    Updated:                     2023-06-26 00:00:00 +0000 UTC
+`
+	d := &Describer{
+		Prefix: "",
+		out:    &tabwriter.Writer{},
+		buf:    &bytes.Buffer{},
+	}
+	d.out.Init(d.buf, 0, 8, 2, ' ', 0)
+	describeRestoreItemOperation(d, input)
+	d.out.Flush()
+	assert.Equal(t, expected, d.buf.String())
+}
+
+func TestDescribePodVolumeRestores(t *testing.T) {
+	pvr1 := builder.ForPodVolumeRestore("velero", "pvr-1").
+		UploaderType("kopia").
+		Phase(velerov1api.PodVolumeRestorePhaseCompleted).
+		BackupStorageLocation("bsl-1").
+		Volume("vol-1").
+		PodName("pod-1").
+		PodNamespace("pod-ns-1").
+		SnapshotID("snap-1").Result()
+	pvr2 := builder.ForPodVolumeRestore("velero", "pvr-2").
+		UploaderType("kopia").
+		Phase(velerov1api.PodVolumeRestorePhaseCompleted).
+		BackupStorageLocation("bsl-1").
+		Volume("vol-2").
+		PodName("pod-2").
+		PodNamespace("pod-ns-1").
+		SnapshotID("snap-2").Result()
+
+	testcases := []struct {
+		name         string
+		inputPVRList []velerov1api.PodVolumeRestore
+		inputDetails bool
+		expect       string
+	}{
+		{
+			name:         "empty list",
+			inputPVRList: []velerov1api.PodVolumeRestore{},
+			inputDetails: true,
+			expect:       ``,
+		},
+		{
+			name:         "2 completed pvrs no details",
+			inputPVRList: []velerov1api.PodVolumeRestore{*pvr1, *pvr2},
+			inputDetails: false,
+			expect: `kopia Restores (specify --details for more information):
+  Completed:  2
+`,
+		},
+		{
+			name:         "2 completed pvrs with details",
+			inputPVRList: []velerov1api.PodVolumeRestore{*pvr1, *pvr2},
+			inputDetails: true,
+			expect: `kopia Restores:
+  Completed:
+    pod-ns-1/pod-1: vol-1
+    pod-ns-1/pod-2: vol-2
+`,
+		},
+	}
+
+	for _, tc := range testcases {
+		t.Run(tc.name, func(tt *testing.T) {
+			d := &Describer{
+				Prefix: "",
+				out:    &tabwriter.Writer{},
+				buf:    &bytes.Buffer{},
+			}
+			d.out.Init(d.buf, 0, 8, 2, ' ', 0)
+			describePodVolumeRestores(d, tc.inputPVRList, tc.inputDetails)
+			d.out.Flush()
+			assert.Equal(tt, tc.expect, d.buf.String())
+		})
+	}
+}

pkg/cmd/util/output/schedule_describe_test.go

@@ -0,0 +1,123 @@
+package output
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+
+	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
+	"github.com/vmware-tanzu/velero/pkg/builder"
+)
+
+func TestDescribeSchedule(t *testing.T) {
+	input1 := builder.ForSchedule("velero", "schedule-1").
+		Phase(velerov1api.SchedulePhaseFailedValidation).
+		ValidationError("validation failed").Result()
+	expect1 := `Name:         schedule-1
+Namespace:    velero
+Labels:       <none>
+Annotations:  <none>
+
+Phase:  FailedValidation
+
+Validation errors:  validation failed
+
+Paused:  false
+
+Schedule:  
+
+Backup Template:
+  Namespaces:
+    Included:  *
+    Excluded:  <none>
+  
+  Resources:
+    Included:        *
+    Excluded:        <none>
+    Cluster-scoped:  auto
+  
+  Label selector:  <none>
+  
+  Storage Location:  
+  
+  Velero-Native Snapshot PVs:  auto
+  Snapshot Move Data:          auto
+  Data Mover:                  <none>
+  
+  TTL:  0s
+  
+  CSISnapshotTimeout:    0s
+  ItemOperationTimeout:  0s
+  
+  Hooks:  <none>
+
+Last Backup:  <never>
+`
+
+	input2 := builder.ForSchedule("velero", "schedule-2").
+		Phase(velerov1api.SchedulePhaseEnabled).
+		CronSchedule("0 0 * * *").
+		Template(builder.ForBackup("velero", "backup-1").Result().Spec).
+		LastBackupTime("2023-06-25 15:04:05").Result()
+	expect2 := `Name:         schedule-2
+Namespace:    velero
+Labels:       <none>
+Annotations:  <none>
+
+Phase:  Enabled
+
+Paused:  false
+
+Schedule:  0 0 * * *
+
+Backup Template:
+  Namespaces:
+    Included:  *
+    Excluded:  <none>
+  
+  Resources:
+    Included:        *
+    Excluded:        <none>
+    Cluster-scoped:  auto
+  
+  Label selector:  <none>
+  
+  Storage Location:  
+  
+  Velero-Native Snapshot PVs:  auto
+  Snapshot Move Data:          auto
+  Data Mover:                  <none>
+  
+  TTL:  0s
+  
+  CSISnapshotTimeout:    0s
+  ItemOperationTimeout:  0s
+  
+  Hooks:  <none>
+
+Last Backup:  2023-06-25 15:04:05 +0000 UTC
+`
+
+	testcases := []struct {
+		name   string
+		input  *velerov1api.Schedule
+		expect string
+	}{
+		{
+			name:   "schedule failed in validation",
+			input:  input1,
+			expect: expect1,
+		},
+		{
+			name:   "schedule enabled",
+			input:  input2,
+			expect: expect2,
+		},
+	}
+
+	for _, tc := range testcases {
+		t.Run(tc.name, func(tt *testing.T) {
+			assert.Equal(tt, tc.expect, DescribeSchedule(tc.input))
+		})
+	}
+}

pkg/controller/data_download_controller.go

@@ -0,0 +1,494 @@
+/*
+Copyright The Velero Contributors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package controller
+
+import (
+	"context"
+	"fmt"
+	"time"
+
+	"github.com/pkg/errors"
+	"github.com/sirupsen/logrus"
+	v1 "k8s.io/api/core/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/utils/clock"
+	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/builder"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/event"
+	"sigs.k8s.io/controller-runtime/pkg/predicate"
+	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+	"sigs.k8s.io/controller-runtime/pkg/source"
+
+	"github.com/vmware-tanzu/velero/internal/credentials"
+	"github.com/vmware-tanzu/velero/pkg/apis/velero/shared"
+	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
+	velerov2alpha1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1"
+	datamover "github.com/vmware-tanzu/velero/pkg/datamover"
+	"github.com/vmware-tanzu/velero/pkg/datapath"
+	"github.com/vmware-tanzu/velero/pkg/exposer"
+	repository "github.com/vmware-tanzu/velero/pkg/repository"
+	"github.com/vmware-tanzu/velero/pkg/uploader"
+	"github.com/vmware-tanzu/velero/pkg/util/filesystem"
+	"github.com/vmware-tanzu/velero/pkg/util/kube"
+)
+
+// DataDownloadReconciler reconciles a DataDownload object
+type DataDownloadReconciler struct {
+	client            client.Client
+	kubeClient        kubernetes.Interface
+	logger            logrus.FieldLogger
+	credentialGetter  *credentials.CredentialGetter
+	fileSystem        filesystem.Interface
+	clock             clock.WithTickerAndDelayedExecution
+	restoreExposer    exposer.GenericRestoreExposer
+	nodeName          string
+	repositoryEnsurer *repository.Ensurer
+	dataPathMgr       *datapath.Manager
+}
+
+func NewDataDownloadReconciler(client client.Client, kubeClient kubernetes.Interface,
+	repoEnsurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter, nodeName string, logger logrus.FieldLogger) *DataDownloadReconciler {
+	return &DataDownloadReconciler{
+		client:            client,
+		kubeClient:        kubeClient,
+		logger:            logger.WithField("controller", "DataDownload"),
+		credentialGetter:  credentialGetter,
+		fileSystem:        filesystem.NewFileSystem(),
+		clock:             &clock.RealClock{},
+		nodeName:          nodeName,
+		repositoryEnsurer: repoEnsurer,
+		restoreExposer:    exposer.NewGenericRestoreExposer(kubeClient, logger),
+		dataPathMgr:       datapath.NewManager(1),
+	}
+}
+
+// +kubebuilder:rbac:groups=velero.io,resources=datadownloads,verbs=get;list;watch;create;update;patch;delete
+// +kubebuilder:rbac:groups=velero.io,resources=datadownloads/status,verbs=get;update;patch
+// +kubebuilder:rbac:groups="",resources=pods,verbs=get
+// +kubebuilder:rbac:groups="",resources=persistentvolumes,verbs=get
+// +kubebuilder:rbac:groups="",resources=persistentvolumerclaims,verbs=get
+
+func (r *DataDownloadReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
+	log := r.logger.WithFields(logrus.Fields{
+		"controller":   "datadownload",
+		"datadownload": req.NamespacedName,
+	})
+
+	log.Infof("Reconcile %s", req.Name)
+
+	dd := &velerov2alpha1api.DataDownload{}
+	if err := r.client.Get(ctx, types.NamespacedName{Namespace: req.Namespace, Name: req.Name}, dd); err != nil {
+		if apierrors.IsNotFound(err) {
+			log.Warn("DataDownload not found, skip")
+			return ctrl.Result{}, nil
+		}
+		log.WithError(err).Error("Unable to get the DataDownload")
+		return ctrl.Result{}, err
+	}
+
+	if dd.Spec.DataMover != "" && dd.Spec.DataMover != dataMoverType {
+		log.WithField("data mover", dd.Spec.DataMover).Info("it is not one built-in data mover which is not supported by Velero")
+		return ctrl.Result{}, nil
+	}
+
+	if r.restoreExposer == nil {
+		return r.errorOut(ctx, dd, errors.New("uninitialized generic exposer"), "uninitialized exposer", log)
+	}
+
+	if dd.Status.Phase == "" || dd.Status.Phase == velerov2alpha1api.DataDownloadPhaseNew {
+		log.Info("Data download starting")
+
+		if _, err := r.getTargetPVC(ctx, dd); err != nil {
+			return ctrl.Result{Requeue: true}, nil
+		}
+
+		accepted, err := r.acceptDataDownload(ctx, dd)
+		if err != nil {
+			return r.errorOut(ctx, dd, err, "error to accept the data download", log)
+		}
+
+		if !accepted {
+			log.Debug("Data download is not accepted")
+			return ctrl.Result{}, nil
+		}
+
+		log.Info("Data download is accepted")
+
+		hostingPodLabels := map[string]string{velerov1api.DataDownloadLabel: dd.Name}
+
+		// ep.Expose() will trigger to create one pod whose volume is restored by a given volume snapshot,
+		// but the pod maybe is not in the same node of the current controller, so we need to return it here.
+		// And then only the controller who is in the same node could do the rest work.
+		err = r.restoreExposer.Expose(ctx, getDataDownloadOwnerObject(dd), dd.Spec.TargetVolume.PVC, dd.Spec.TargetVolume.Namespace, hostingPodLabels, dd.Spec.OperationTimeout.Duration)
+		if err != nil {
+			return r.errorOut(ctx, dd, err, "error to start restore expose", log)
+		}
+		log.Info("Restore is exposed")
+
+		return ctrl.Result{}, nil
+	} else if dd.Status.Phase == velerov2alpha1api.DataDownloadPhasePrepared {
+		log.Info("Data download is prepared")
+		fsRestore := r.dataPathMgr.GetAsyncBR(dd.Name)
+
+		if fsRestore != nil {
+			log.Info("Cancellable data path is already started")
+			return ctrl.Result{}, nil
+		}
+
+		result, err := r.restoreExposer.GetExposed(ctx, getDataDownloadOwnerObject(dd), r.client, r.nodeName, dd.Spec.OperationTimeout.Duration)
+		if err != nil {
+			return r.errorOut(ctx, dd, err, "restore exposer is not ready", log)
+		} else if result == nil {
+			log.Debug("Get empty restore exposer")
+			return ctrl.Result{}, nil
+		}
+
+		log.Info("Restore PVC is ready and creating data path routine")
+
+		// Need to first create file system BR and get data path instance then update data upload status
+		callbacks := datapath.Callbacks{
+			OnCompleted: r.OnDataDownloadCompleted,
+			OnFailed:    r.OnDataDownloadFailed,
+			OnCancelled: r.OnDataDownloadCancelled,
+			OnProgress:  r.OnDataDownloadProgress,
+		}
+
+		fsRestore, err = r.dataPathMgr.CreateFileSystemBR(dd.Name, dataUploadDownloadRequestor, ctx, r.client, dd.Namespace, callbacks, log)
+		if err != nil {
+			if err == datapath.ConcurrentLimitExceed {
+				log.Info("Data path instance is concurrent limited requeue later")
+				return ctrl.Result{Requeue: true, RequeueAfter: time.Minute}, nil
+			} else {
+				return r.errorOut(ctx, dd, err, "error to create data path", log)
+			}
+		}
+
+		// Update status to InProgress
+		original := dd.DeepCopy()
+		dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseInProgress
+		dd.Status.StartTimestamp = &metav1.Time{Time: r.clock.Now()}
+		if err := r.client.Patch(ctx, dd, client.MergeFrom(original)); err != nil {
+			log.WithError(err).Error("Unable to update status to in progress")
+			return ctrl.Result{}, err
+		}
+
+		log.Info("Data download is marked as in progress")
+
+		reconcileResult, err := r.runCancelableDataPath(ctx, fsRestore, dd, result, log)
+		if err != nil {
+			log.Errorf("Failed to run cancelable data path for %s with err %v", dd.Name, err)
+			r.closeDataPath(ctx, dd.Name)
+		}
+		return reconcileResult, err
+	} else if dd.Status.Phase == velerov2alpha1api.DataDownloadPhaseInProgress {
+		log.Info("Data download is in progress")
+		if dd.Spec.Cancel {
+			fsRestore := r.dataPathMgr.GetAsyncBR(dd.Name)
+			if fsRestore == nil {
+				return ctrl.Result{}, nil
+			}
+
+			log.Info("Data download is being canceled")
+			// Update status to Canceling.
+			original := dd.DeepCopy()
+			dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseCanceling
+			if err := r.client.Patch(ctx, dd, client.MergeFrom(original)); err != nil {
+				log.WithError(err).Error("error updating data download status")
+				return ctrl.Result{}, err
+			}
+
+			fsRestore.Cancel()
+			return ctrl.Result{}, nil
+		}
+
+		return ctrl.Result{}, nil
+	} else {
+		log.Debugf("Data download now is in %s phase and do nothing by current %s controller", dd.Status.Phase, r.nodeName)
+		return ctrl.Result{}, nil
+	}
+}
+
+func (r *DataDownloadReconciler) runCancelableDataPath(ctx context.Context, fsRestore datapath.AsyncBR, dd *velerov2alpha1api.DataDownload, res *exposer.ExposeResult, log logrus.FieldLogger) (reconcile.Result, error) {
+	path, err := exposer.GetPodVolumeHostPath(ctx, res.ByPod.HostingPod, res.ByPod.PVC, r.client, r.fileSystem, log)
+	if err != nil {
+		return r.errorOut(ctx, dd, err, "error exposing host path for pod volume", log)
+	}
+
+	log.WithField("path", path.ByPath).Debug("Found host path")
+	if err := fsRestore.Init(ctx, dd.Spec.BackupStorageLocation, dd.Spec.SourceNamespace, datamover.GetUploaderType(dd.Spec.DataMover),
+		velerov1api.BackupRepositoryTypeKopia, "", r.repositoryEnsurer, r.credentialGetter); err != nil {
+		return r.errorOut(ctx, dd, err, "error to initialize data path", log)
+	}
+	log.WithField("path", path.ByPath).Info("fs init")
+
+	if err := fsRestore.StartRestore(dd.Spec.SnapshotID, path); err != nil {
+		return r.errorOut(ctx, dd, err, fmt.Sprintf("error starting data path %s restore", path.ByPath), log)
+	}
+
+	log.WithField("path", path.ByPath).Info("Async fs restore data path started")
+	return ctrl.Result{}, nil
+}
+
+func (r *DataDownloadReconciler) OnDataDownloadCompleted(ctx context.Context, namespace string, ddName string, result datapath.Result) {
+	defer r.closeDataPath(ctx, ddName)
+
+	log := r.logger.WithField("datadownload", ddName)
+	log.Info("Async fs restore data path completed")
+
+	var dd velerov2alpha1api.DataDownload
+	if err := r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, &dd); err != nil {
+		log.WithError(err).Warn("Failed to get datadownload on completion")
+		return
+	}
+
+	objRef := getDataDownloadOwnerObject(&dd)
+	err := r.restoreExposer.RebindVolume(ctx, objRef, dd.Spec.TargetVolume.PVC, dd.Spec.TargetVolume.Namespace, dd.Spec.OperationTimeout.Duration)
+	if err != nil {
+		log.WithError(err).Error("Failed to rebind PV to target PVC on completion")
+		return
+	}
+
+	log.Info("Cleaning up exposed environment")
+	r.restoreExposer.CleanUp(ctx, objRef)
+
+	original := dd.DeepCopy()
+	dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseCompleted
+	dd.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()}
+	if err := r.client.Patch(ctx, &dd, client.MergeFrom(original)); err != nil {
+		log.WithError(err).Error("error updating data download status")
+	} else {
+		log.Infof("Data download is marked as %s", dd.Status.Phase)
+	}
+}
+
+func (r *DataDownloadReconciler) OnDataDownloadFailed(ctx context.Context, namespace string, ddName string, err error) {
+	defer r.closeDataPath(ctx, ddName)
+
+	log := r.logger.WithField("datadownload", ddName)
+
+	log.WithError(err).Error("Async fs restore data path failed")
+
+	var dd velerov2alpha1api.DataDownload
+	if getErr := r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, &dd); getErr != nil {
+		log.WithError(getErr).Warn("Failed to get data download on failure")
+	} else {
+		if _, errOut := r.errorOut(ctx, &dd, err, "data path restore failed", log); err != nil {
+			log.WithError(err).Warnf("Failed to patch data download with err %v", errOut)
+		}
+	}
+}
+
+func (r *DataDownloadReconciler) OnDataDownloadCancelled(ctx context.Context, namespace string, ddName string) {
+	defer r.closeDataPath(ctx, ddName)
+
+	log := r.logger.WithField("datadownload", ddName)
+
+	log.Warn("Async fs backup data path canceled")
+
+	var dd velerov2alpha1api.DataDownload
+	if getErr := r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, &dd); getErr != nil {
+		log.WithError(getErr).Warn("Failed to get datadownload on cancel")
+	} else {
+		// cleans up any objects generated during the snapshot expose
+		r.restoreExposer.CleanUp(ctx, getDataDownloadOwnerObject(&dd))
+
+		original := dd.DeepCopy()
+		dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseCanceled
+		if dd.Status.StartTimestamp.IsZero() {
+			dd.Status.StartTimestamp = &metav1.Time{Time: r.clock.Now()}
+		}
+		dd.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()}
+		if err := r.client.Patch(ctx, &dd, client.MergeFrom(original)); err != nil {
+			log.WithError(err).Error("error updating data download status")
+		}
+	}
+}
+
+func (r *DataDownloadReconciler) OnDataDownloadProgress(ctx context.Context, namespace string, ddName string, progress *uploader.Progress) {
+	log := r.logger.WithField("datadownload", ddName)
+
+	var dd velerov2alpha1api.DataDownload
+	if err := r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, &dd); err != nil {
+		log.WithError(err).Warn("Failed to get data download on progress")
+		return
+	}
+
+	original := dd.DeepCopy()
+	dd.Status.Progress = shared.DataMoveOperationProgress{TotalBytes: progress.TotalBytes, BytesDone: progress.BytesDone}
+
+	if err := r.client.Patch(ctx, &dd, client.MergeFrom(original)); err != nil {
+		log.WithError(err).Error("Failed to update restore snapshot progress")
+	}
+}
+
+// SetupWithManager registers the DataDownload controller.
+// The fresh new DataDownload CR first created will trigger to create one pod (long time, maybe failure or unknown status) by one of the datadownload controllers
+// then the request will get out of the Reconcile queue immediately by not blocking others' CR handling, in order to finish the rest data download process we need to
+// re-enqueue the previous related request once the related pod is in running status to keep going on the rest logic. and below logic will avoid handling the unwanted
+// pod status and also avoid block others CR handling
+func (r *DataDownloadReconciler) SetupWithManager(mgr ctrl.Manager) error {
+	return ctrl.NewControllerManagedBy(mgr).
+		For(&velerov2alpha1api.DataDownload{}).
+		Watches(&source.Kind{Type: &v1.Pod{}}, kube.EnqueueRequestsFromMapUpdateFunc(r.findSnapshotRestoreForPod),
+			builder.WithPredicates(predicate.Funcs{
+				UpdateFunc: func(ue event.UpdateEvent) bool {
+					newObj := ue.ObjectNew.(*v1.Pod)
+
+					if _, ok := newObj.Labels[velerov1api.DataDownloadLabel]; !ok {
+						return false
+					}
+
+					if newObj.Status.Phase != v1.PodRunning {
+						return false
+					}
+
+					if newObj.Spec.NodeName == "" {
+						return false
+					}
+
+					return true
+				},
+				CreateFunc: func(event.CreateEvent) bool {
+					return false
+				},
+				DeleteFunc: func(de event.DeleteEvent) bool {
+					return false
+				},
+				GenericFunc: func(ge event.GenericEvent) bool {
+					return false
+				},
+			})).
+		Complete(r)
+}
+
+func (r *DataDownloadReconciler) findSnapshotRestoreForPod(podObj client.Object) []reconcile.Request {
+	pod := podObj.(*v1.Pod)
+
+	dd := &velerov2alpha1api.DataDownload{}
+	err := r.client.Get(context.Background(), types.NamespacedName{
+		Namespace: pod.Namespace,
+		Name:      pod.Labels[velerov1api.DataDownloadLabel],
+	}, dd)
+
+	if err != nil {
+		r.logger.WithField("Restore pod", pod.Name).WithError(err).Error("unable to get DataDownload")
+		return []reconcile.Request{}
+	}
+
+	if dd.Status.Phase != velerov2alpha1api.DataDownloadPhaseAccepted {
+		return []reconcile.Request{}
+	}
+
+	requests := make([]reconcile.Request, 1)
+
+	r.logger.WithField("Restore pod", pod.Name).Infof("Preparing data download %s", dd.Name)
+	err = r.patchDataDownload(context.Background(), dd, r.prepareDataDownload)
+	if err != nil {
+		r.logger.WithField("Restore pod", pod.Name).WithError(err).Error("unable to patch data download")
+		return []reconcile.Request{}
+	}
+
+	requests[0] = reconcile.Request{
+		NamespacedName: types.NamespacedName{
+			Namespace: dd.Namespace,
+			Name:      dd.Name,
+		},
+	}
+
+	return requests
+}
+
+func (r *DataDownloadReconciler) patchDataDownload(ctx context.Context, req *velerov2alpha1api.DataDownload, mutate func(*velerov2alpha1api.DataDownload)) error {
+	original := req.DeepCopy()
+	mutate(req)
+	if err := r.client.Patch(ctx, req, client.MergeFrom(original)); err != nil {
+		return errors.Wrap(err, "error patching data download")
+	}
+
+	return nil
+}
+
+func (r *DataDownloadReconciler) prepareDataDownload(ssb *velerov2alpha1api.DataDownload) {
+	ssb.Status.Phase = velerov2alpha1api.DataDownloadPhasePrepared
+	ssb.Status.Node = r.nodeName
+}
+
+func (r *DataDownloadReconciler) errorOut(ctx context.Context, dd *velerov2alpha1api.DataDownload, err error, msg string, log logrus.FieldLogger) (ctrl.Result, error) {
+	if r.restoreExposer != nil {
+		r.restoreExposer.CleanUp(ctx, getDataDownloadOwnerObject(dd))
+	}
+	return ctrl.Result{}, r.updateStatusToFailed(ctx, dd, err, msg, log)
+}
+
+func (r *DataDownloadReconciler) updateStatusToFailed(ctx context.Context, dd *velerov2alpha1api.DataDownload, err error, msg string, log logrus.FieldLogger) error {
+	log.Infof("update data download status to %v", dd.Status.Phase)
+	original := dd.DeepCopy()
+	dd.Status.Phase = velerov2alpha1api.DataDownloadPhaseFailed
+	dd.Status.Message = errors.WithMessage(err, msg).Error()
+	dd.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()}
+
+	if patchErr := r.client.Patch(ctx, dd, client.MergeFrom(original)); patchErr != nil {
+		log.WithError(patchErr).Error("error updating DataDownload status")
+	}
+
+	return err
+}
+
+func (r *DataDownloadReconciler) acceptDataDownload(ctx context.Context, dd *velerov2alpha1api.DataDownload) (bool, error) {
+	updated := dd.DeepCopy()
+	updated.Status.Phase = velerov2alpha1api.DataDownloadPhaseAccepted
+
+	r.logger.Infof("Accepting snapshot restore %s", dd.Name)
+	// For all data download controller in each node-agent will try to update download CR, and only one controller will success,
+	// and the success one could handle later logic
+	err := r.client.Update(ctx, updated)
+	if err == nil {
+		return true, nil
+	} else if apierrors.IsConflict(err) {
+		r.logger.WithField("DataDownload", dd.Name).Error("This data download restore has been accepted by others")
+		return false, nil
+	} else {
+		return false, err
+	}
+}
+
+func (r *DataDownloadReconciler) getTargetPVC(ctx context.Context, dd *velerov2alpha1api.DataDownload) (*v1.PersistentVolumeClaim, error) {
+	return r.kubeClient.CoreV1().PersistentVolumeClaims(dd.Spec.TargetVolume.Namespace).Get(ctx, dd.Spec.TargetVolume.PVC, metav1.GetOptions{})
+}
+
+func (r *DataDownloadReconciler) closeDataPath(ctx context.Context, ddName string) {
+	fsBackup := r.dataPathMgr.GetAsyncBR(ddName)
+	if fsBackup != nil {
+		fsBackup.Close(ctx)
+	}
+
+	r.dataPathMgr.RemoveAsyncBR(ddName)
+}
+
+func getDataDownloadOwnerObject(dd *velerov2alpha1api.DataDownload) v1.ObjectReference {
+	return v1.ObjectReference{
+		Kind:       dd.Kind,
+		Namespace:  dd.Namespace,
+		Name:       dd.Name,
+		UID:        dd.UID,
+		APIVersion: dd.APIVersion,
+	}
+}

pkg/controller/data_download_controller_test.go

@@ -0,0 +1,582 @@
+/*
+Copyright The Velero Contributors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package controller
+
+import (
+	"context"
+	"fmt"
+	"testing"
+	"time"
+
+	"github.com/pkg/errors"
+	"github.com/sirupsen/logrus"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/mock"
+	"github.com/stretchr/testify/require"
+	corev1 "k8s.io/api/core/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/types"
+	clientgofake "k8s.io/client-go/kubernetes/fake"
+	ctrl "sigs.k8s.io/controller-runtime"
+	kbclient "sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+
+	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+
+	"github.com/vmware-tanzu/velero/internal/credentials"
+	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
+	velerov2alpha1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1"
+	"github.com/vmware-tanzu/velero/pkg/builder"
+	"github.com/vmware-tanzu/velero/pkg/datapath"
+	datapathmockes "github.com/vmware-tanzu/velero/pkg/datapath/mocks"
+	"github.com/vmware-tanzu/velero/pkg/exposer"
+	velerotest "github.com/vmware-tanzu/velero/pkg/test"
+	"github.com/vmware-tanzu/velero/pkg/uploader"
+
+	exposermockes "github.com/vmware-tanzu/velero/pkg/exposer/mocks"
+)
+
+const dataDownloadName string = "datadownload-1"
+
+func dataDownloadBuilder() *builder.DataDownloadBuilder {
+	return builder.ForDataDownload(velerov1api.DefaultNamespace, dataDownloadName).
+		BackupStorageLocation("bsl-loc").
+		DataMover("velero").
+		SnapshotID("test-snapshot-id").TargetVolume(velerov2alpha1api.TargetVolumeSpec{
+		PV:        "test-pv",
+		PVC:       "test-pvc",
+		Namespace: "test-ns",
+	})
+}
+
+func initDataDownloadReconciler(objects []runtime.Object, needError ...bool) (*DataDownloadReconciler, error) {
+	scheme := runtime.NewScheme()
+	err := velerov1api.AddToScheme(scheme)
+	if err != nil {
+		return nil, err
+	}
+	err = velerov2alpha1api.AddToScheme(scheme)
+	if err != nil {
+		return nil, err
+	}
+	err = corev1.AddToScheme(scheme)
+	if err != nil {
+		return nil, err
+	}
+
+	fakeClient := &FakeClient{
+		Client: fake.NewClientBuilder().WithScheme(scheme).Build(),
+	}
+
+	if len(needError) == 4 {
+		fakeClient.getError = needError[0]
+		fakeClient.createError = needError[1]
+		fakeClient.updateError = needError[2]
+		fakeClient.patchError = needError[3]
+	}
+	var fakeKubeClient *clientgofake.Clientset
+	if len(objects) != 0 {
+		fakeKubeClient = clientgofake.NewSimpleClientset(objects...)
+	} else {
+		fakeKubeClient = clientgofake.NewSimpleClientset()
+	}
+
+	fakeFS := velerotest.NewFakeFileSystem()
+	pathGlob := fmt.Sprintf("/host_pods/%s/volumes/*/%s", "test-uid", "test-pvc")
+	_, err = fakeFS.Create(pathGlob)
+	if err != nil {
+		return nil, err
+	}
+
+	credentialFileStore, err := credentials.NewNamespacedFileStore(
+		fakeClient,
+		velerov1api.DefaultNamespace,
+		"/tmp/credentials",
+		fakeFS,
+	)
+	if err != nil {
+		return nil, err
+	}
+	return NewDataDownloadReconciler(fakeClient, fakeKubeClient, nil, &credentials.CredentialGetter{FromFile: credentialFileStore}, "test_node", velerotest.NewLogger()), nil
+}
+
+func TestDataDownloadReconcile(t *testing.T) {
+	tests := []struct {
+		name              string
+		dd                *velerov2alpha1api.DataDownload
+		targetPVC         *corev1.PersistentVolumeClaim
+		dataMgr           *datapath.Manager
+		needErrs          []bool
+		needCreateFSBR    bool
+		isExposeErr       bool
+		isGetExposeErr    bool
+		isNilExposer      bool
+		isFSBRInitErr     bool
+		isFSBRRestoreErr  bool
+		notNilExpose      bool
+		notMockCleanUp    bool
+		mockCancel        bool
+		mockClose         bool
+		expectedStatusMsg string
+		expectedResult    *ctrl.Result
+	}{
+		{
+			name:      "Unknown data download status",
+			dd:        dataDownloadBuilder().Phase("Unknown").Cancel(true).Result(),
+			targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+		},
+		{
+			name:              "Cancel data downloand in progress and patch data download error",
+			dd:                dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhaseInProgress).Cancel(true).Result(),
+			targetPVC:         builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+			needErrs:          []bool{false, false, false, true},
+			needCreateFSBR:    true,
+			expectedStatusMsg: "Patch error",
+		},
+		{
+			name:       "Cancel data downloand in progress with empty FSBR",
+			dd:         dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhaseInProgress).Cancel(true).Result(),
+			targetPVC:  builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+			mockCancel: true,
+		},
+		{
+			name:           "Cancel data downloand in progress",
+			dd:             dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhaseInProgress).Cancel(true).Result(),
+			targetPVC:      builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+			needCreateFSBR: true,
+			mockCancel:     true,
+		},
+		{
+			name:           "Error in data path is concurrent limited",
+			dd:             dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhasePrepared).Result(),
+			targetPVC:      builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+			dataMgr:        datapath.NewManager(0),
+			notNilExpose:   true,
+			notMockCleanUp: true,
+			expectedResult: &ctrl.Result{Requeue: true, RequeueAfter: time.Minute},
+		},
+		{
+			name:              "Error getting volume directory name for pvc in pod",
+			dd:                dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhasePrepared).Result(),
+			targetPVC:         builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+			notNilExpose:      true,
+			mockClose:         true,
+			expectedStatusMsg: "error identifying unique volume path on host",
+		},
+		{
+			name:              "Unable to update status to in progress for data download",
+			dd:                dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhasePrepared).Result(),
+			targetPVC:         builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+			needErrs:          []bool{false, false, false, true},
+			notNilExpose:      true,
+			notMockCleanUp:    true,
+			expectedStatusMsg: "Patch error",
+		},
+		{
+			name:              "accept DataDownload error",
+			dd:                dataDownloadBuilder().Result(),
+			targetPVC:         builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+			needErrs:          []bool{false, false, true, false},
+			expectedStatusMsg: "Update error",
+		},
+		{
+			name: "Not create target pvc",
+			dd:   dataDownloadBuilder().Result(),
+		},
+		{
+			name:              "Uninitialized dataDownload",
+			dd:                dataDownloadBuilder().Result(),
+			targetPVC:         builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+			isNilExposer:      true,
+			expectedStatusMsg: "uninitialized generic exposer",
+		},
+		{
+			name:      "DataDownload not created in velero default namespace",
+			dd:        builder.ForDataDownload("test-ns", dataDownloadName).Result(),
+			targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+		},
+		{
+			name:              "Failed to get dataDownload",
+			dd:                builder.ForDataDownload("test-ns", dataDownloadName).Result(),
+			targetPVC:         builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+			needErrs:          []bool{true, false, false, false},
+			expectedStatusMsg: "Create error",
+		},
+		{
+			name:      "Unsupported dataDownload type",
+			dd:        dataDownloadBuilder().DataMover("Unsuppoorted type").Result(),
+			targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+		},
+		{
+			name:      "Restore is exposed",
+			dd:        dataDownloadBuilder().Result(),
+			targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+		},
+		{
+			name:      "Get empty restore exposer",
+			dd:        dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhasePrepared).Result(),
+			targetPVC: builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+		},
+		{
+			name:              "Failed to get restore exposer",
+			dd:                dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhasePrepared).Result(),
+			targetPVC:         builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+			expectedStatusMsg: "Error to get restore exposer",
+			isGetExposeErr:    true,
+		},
+		{
+			name:              "Error to start restore expose",
+			dd:                dataDownloadBuilder().Result(),
+			targetPVC:         builder.ForPersistentVolumeClaim("test-ns", "test-pvc").Result(),
+			expectedStatusMsg: "Error to expose restore exposer",
+			isExposeErr:       true,
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			var objs []runtime.Object
+			if test.targetPVC != nil {
+				objs = []runtime.Object{test.targetPVC}
+			}
+			r, err := initDataDownloadReconciler(objs, test.needErrs...)
+			require.NoError(t, err)
+			defer func() {
+				r.client.Delete(ctx, test.dd, &kbclient.DeleteOptions{})
+				if test.targetPVC != nil {
+					r.client.Delete(ctx, test.targetPVC, &kbclient.DeleteOptions{})
+				}
+			}()
+
+			ctx := context.Background()
+			if test.dd.Namespace == velerov1api.DefaultNamespace {
+				err = r.client.Create(ctx, test.dd)
+				require.NoError(t, err)
+			}
+
+			if test.dataMgr != nil {
+				r.dataPathMgr = test.dataMgr
+			} else {
+				r.dataPathMgr = datapath.NewManager(1)
+			}
+
+			datapath.FSBRCreator = func(string, string, kbclient.Client, string, datapath.Callbacks, logrus.FieldLogger) datapath.AsyncBR {
+				fsBR := datapathmockes.NewAsyncBR(t)
+				if test.mockCancel {
+					fsBR.On("Cancel").Return()
+				}
+
+				if test.mockClose {
+					fsBR.On("Close", mock.Anything).Return()
+				}
+
+				return fsBR
+			}
+
+			if test.isExposeErr || test.isGetExposeErr || test.isNilExposer || test.notNilExpose {
+				if test.isNilExposer {
+					r.restoreExposer = nil
+				} else {
+					r.restoreExposer = func() exposer.GenericRestoreExposer {
+						ep := exposermockes.NewGenericRestoreExposer(t)
+						if test.isExposeErr {
+							ep.On("Expose", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(errors.New("Error to expose restore exposer"))
+						} else if test.notNilExpose {
+							hostingPod := builder.ForPod("test-ns", "test-name").Volumes(&corev1.Volume{Name: "test-pvc"}).Result()
+							hostingPod.ObjectMeta.SetUID("test-uid")
+							ep.On("GetExposed", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(&exposer.ExposeResult{ByPod: exposer.ExposeByPod{HostingPod: hostingPod, PVC: "test-pvc"}}, nil)
+						} else if test.isGetExposeErr {
+							ep.On("GetExposed", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(nil, errors.New("Error to get restore exposer"))
+						}
+
+						if !test.notMockCleanUp {
+							ep.On("CleanUp", mock.Anything, mock.Anything).Return()
+						}
+						return ep
+					}()
+				}
+			}
+
+			if test.needCreateFSBR {
+				if fsBR := r.dataPathMgr.GetAsyncBR(test.dd.Name); fsBR == nil {
+					_, err := r.dataPathMgr.CreateFileSystemBR(test.dd.Name, pVBRRequestor, ctx, r.client, velerov1api.DefaultNamespace, datapath.Callbacks{OnCancelled: r.OnDataDownloadCancelled}, velerotest.NewLogger())
+					require.NoError(t, err)
+				}
+			}
+
+			actualResult, err := r.Reconcile(ctx, ctrl.Request{
+				NamespacedName: types.NamespacedName{
+					Namespace: velerov1api.DefaultNamespace,
+					Name:      test.dd.Name,
+				},
+			})
+
+			if test.expectedStatusMsg != "" {
+				assert.Contains(t, err.Error(), test.expectedStatusMsg)
+			} else {
+				require.Nil(t, err)
+			}
+
+			require.NotNil(t, actualResult)
+
+			if test.expectedResult != nil {
+				assert.Equal(t, test.expectedResult.Requeue, test.expectedResult.Requeue)
+				assert.Equal(t, test.expectedResult.RequeueAfter, test.expectedResult.RequeueAfter)
+			}
+
+			dd := velerov2alpha1api.DataDownload{}
+			err = r.client.Get(ctx, kbclient.ObjectKey{
+				Name:      test.dd.Name,
+				Namespace: test.dd.Namespace,
+			}, &dd)
+
+			if test.isGetExposeErr {
+				assert.Contains(t, dd.Status.Message, test.expectedStatusMsg)
+			}
+			if test.dd.Namespace == velerov1api.DefaultNamespace {
+				require.Nil(t, err)
+			} else {
+				assert.True(t, true, apierrors.IsNotFound(err))
+			}
+
+			t.Logf("%s: \n %v \n", test.name, dd)
+		})
+	}
+}
+
+func TestOnDataDownloadFailed(t *testing.T) {
+	for _, getErr := range []bool{true, false} {
+		ctx := context.TODO()
+		needErrs := []bool{getErr, false, false, false}
+		r, err := initDataDownloadReconciler(nil, needErrs...)
+		require.NoError(t, err)
+
+		dd := dataDownloadBuilder().Result()
+		namespace := dd.Namespace
+		ddName := dd.Name
+		// Add the DataDownload object to the fake client
+		assert.NoError(t, r.client.Create(ctx, dd))
+		r.OnDataDownloadFailed(ctx, namespace, ddName, fmt.Errorf("Failed to handle %v", ddName))
+		updatedDD := &velerov2alpha1api.DataDownload{}
+		if getErr {
+			assert.Error(t, r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, updatedDD))
+			assert.NotEqual(t, velerov2alpha1api.DataDownloadPhaseFailed, updatedDD.Status.Phase)
+			assert.Equal(t, updatedDD.Status.StartTimestamp.IsZero(), true)
+		} else {
+			assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, updatedDD))
+			assert.Equal(t, velerov2alpha1api.DataDownloadPhaseFailed, updatedDD.Status.Phase)
+			assert.Equal(t, updatedDD.Status.StartTimestamp.IsZero(), true)
+		}
+	}
+}
+
+func TestOnDataDownloadCancelled(t *testing.T) {
+	for _, getErr := range []bool{true, false} {
+		ctx := context.TODO()
+		needErrs := []bool{getErr, false, false, false}
+		r, err := initDataDownloadReconciler(nil, needErrs...)
+		require.NoError(t, err)
+
+		dd := dataDownloadBuilder().Result()
+		namespace := dd.Namespace
+		ddName := dd.Name
+		// Add the DataDownload object to the fake client
+		assert.NoError(t, r.client.Create(ctx, dd))
+		r.OnDataDownloadCancelled(ctx, namespace, ddName)
+		updatedDD := &velerov2alpha1api.DataDownload{}
+		if getErr {
+			assert.Error(t, r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, updatedDD))
+			assert.NotEqual(t, velerov2alpha1api.DataDownloadPhaseFailed, updatedDD.Status.Phase)
+			assert.Equal(t, updatedDD.Status.StartTimestamp.IsZero(), true)
+		} else {
+			assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, updatedDD))
+			assert.Equal(t, velerov2alpha1api.DataDownloadPhaseCanceled, updatedDD.Status.Phase)
+			assert.Equal(t, updatedDD.Status.StartTimestamp.IsZero(), false)
+			assert.Equal(t, updatedDD.Status.CompletionTimestamp.IsZero(), false)
+		}
+	}
+}
+
+func TestOnDataDownloadCompleted(t *testing.T) {
+	tests := []struct {
+		name            string
+		emptyFSBR       bool
+		isGetErr        bool
+		rebindVolumeErr bool
+	}{
+		{
+			name:            "Data download complete",
+			emptyFSBR:       false,
+			isGetErr:        false,
+			rebindVolumeErr: false,
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			ctx := context.TODO()
+			needErrs := []bool{test.isGetErr, false, false, false}
+			r, err := initDataDownloadReconciler(nil, needErrs...)
+			r.restoreExposer = func() exposer.GenericRestoreExposer {
+				ep := exposermockes.NewGenericRestoreExposer(t)
+				if test.rebindVolumeErr {
+					ep.On("RebindVolume", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(errors.New("Error to rebind volume"))
+
+				} else {
+					ep.On("RebindVolume", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(nil)
+				}
+				ep.On("CleanUp", mock.Anything, mock.Anything).Return()
+				return ep
+			}()
+
+			require.NoError(t, err)
+			dd := dataDownloadBuilder().Result()
+			namespace := dd.Namespace
+			ddName := dd.Name
+			// Add the DataDownload object to the fake client
+			assert.NoError(t, r.client.Create(ctx, dd))
+			r.OnDataDownloadCompleted(ctx, namespace, ddName, datapath.Result{})
+			updatedDD := &velerov2alpha1api.DataDownload{}
+			if test.isGetErr {
+				assert.Error(t, r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, updatedDD))
+				assert.Equal(t, velerov2alpha1api.DataDownloadPhase(""), updatedDD.Status.Phase)
+				assert.Equal(t, updatedDD.Status.CompletionTimestamp.IsZero(), true)
+			} else {
+				assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, updatedDD))
+				assert.Equal(t, velerov2alpha1api.DataDownloadPhaseCompleted, updatedDD.Status.Phase)
+				assert.Equal(t, updatedDD.Status.CompletionTimestamp.IsZero(), false)
+			}
+		})
+	}
+}
+
+func TestOnDataDownloadProgress(t *testing.T) {
+	totalBytes := int64(1024)
+	bytesDone := int64(512)
+	tests := []struct {
+		name     string
+		dd       *velerov2alpha1api.DataDownload
+		progress uploader.Progress
+		needErrs []bool
+	}{
+		{
+			name: "patch in progress phase success",
+			dd:   dataDownloadBuilder().Result(),
+			progress: uploader.Progress{
+				TotalBytes: totalBytes,
+				BytesDone:  bytesDone,
+			},
+		},
+		{
+			name:     "failed to get datadownload",
+			dd:       dataDownloadBuilder().Result(),
+			needErrs: []bool{true, false, false, false},
+		},
+		{
+			name:     "failed to patch datadownload",
+			dd:       dataDownloadBuilder().Result(),
+			needErrs: []bool{false, false, false, true},
+		},
+	}
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			ctx := context.TODO()
+
+			r, err := initDataDownloadReconciler(nil, test.needErrs...)
+			require.NoError(t, err)
+			defer func() {
+				r.client.Delete(ctx, test.dd, &kbclient.DeleteOptions{})
+			}()
+			// Create a DataDownload object
+			dd := dataDownloadBuilder().Result()
+			namespace := dd.Namespace
+			duName := dd.Name
+			// Add the DataDownload object to the fake client
+			assert.NoError(t, r.client.Create(context.Background(), dd))
+
+			// Create a Progress object
+			progress := &uploader.Progress{
+				TotalBytes: totalBytes,
+				BytesDone:  bytesDone,
+			}
+
+			// Call the OnDataDownloadProgress function
+			r.OnDataDownloadProgress(ctx, namespace, duName, progress)
+			if len(test.needErrs) != 0 && !test.needErrs[0] {
+				// Get the updated DataDownload object from the fake client
+				updatedDu := &velerov2alpha1api.DataDownload{}
+				assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: duName, Namespace: namespace}, updatedDu))
+				// Assert that the DataDownload object has been updated with the progress
+				assert.Equal(t, test.progress.TotalBytes, updatedDu.Status.Progress.TotalBytes)
+				assert.Equal(t, test.progress.BytesDone, updatedDu.Status.Progress.BytesDone)
+			}
+		})
+	}
+}
+
+func TestFindDataDownloadForPod(t *testing.T) {
+	needErrs := []bool{false, false, false, false}
+	r, err := initDataDownloadReconciler(nil, needErrs...)
+	require.NoError(t, err)
+	tests := []struct {
+		name      string
+		du        *velerov2alpha1api.DataDownload
+		pod       *corev1.Pod
+		checkFunc func(*velerov2alpha1api.DataDownload, []reconcile.Request)
+	}{
+		{
+			name: "find dataDownload for pod",
+			du:   dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhaseAccepted).Result(),
+			pod:  builder.ForPod(velerov1api.DefaultNamespace, dataDownloadName).Labels(map[string]string{velerov1api.DataDownloadLabel: dataDownloadName}).Result(),
+			checkFunc: func(du *velerov2alpha1api.DataDownload, requests []reconcile.Request) {
+				// Assert that the function returns a single request
+				assert.Len(t, requests, 1)
+				// Assert that the request contains the correct namespaced name
+				assert.Equal(t, du.Namespace, requests[0].Namespace)
+				assert.Equal(t, du.Name, requests[0].Name)
+			},
+		}, {
+			name: "no matched pod",
+			du:   dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhaseAccepted).Result(),
+			pod:  builder.ForPod(velerov1api.DefaultNamespace, dataDownloadName).Labels(map[string]string{velerov1api.DataDownloadLabel: "non-existing-datadownload"}).Result(),
+			checkFunc: func(du *velerov2alpha1api.DataDownload, requests []reconcile.Request) {
+				assert.Empty(t, requests)
+			},
+		},
+		{
+			name: "dataDownload not accepte",
+			du:   dataDownloadBuilder().Phase(velerov2alpha1api.DataDownloadPhaseInProgress).Result(),
+			pod:  builder.ForPod(velerov1api.DefaultNamespace, dataDownloadName).Labels(map[string]string{velerov1api.DataDownloadLabel: dataDownloadName}).Result(),
+			checkFunc: func(du *velerov2alpha1api.DataDownload, requests []reconcile.Request) {
+				assert.Empty(t, requests)
+			},
+		},
+	}
+	for _, test := range tests {
+		ctx := context.Background()
+		assert.NoError(t, r.client.Create(ctx, test.pod))
+		assert.NoError(t, r.client.Create(ctx, test.du))
+		// Call the findSnapshotRestoreForPod function
+		requests := r.findSnapshotRestoreForPod(test.pod)
+		test.checkFunc(test.du, requests)
+		r.client.Delete(ctx, test.du, &kbclient.DeleteOptions{})
+		if test.pod != nil {
+			r.client.Delete(ctx, test.pod, &kbclient.DeleteOptions{})
+		}
+	}
+}

pkg/controller/data_upload_controller.go

@@ -160,7 +160,25 @@ func (r *DataUploadReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 			return ctrl.Result{}, nil
 		}
 
-		log.Info("Exposed snapshot is ready")
+		log.Info("Exposed snapshot is ready and creating data path routine")
+
+		// Need to first create file system BR and get data path instance then update data upload status
+		callbacks := datapath.Callbacks{
+			OnCompleted: r.OnDataUploadCompleted,
+			OnFailed:    r.OnDataUploadFailed,
+			OnCancelled: r.OnDataUploadCancelled,
+			OnProgress:  r.OnDataUploadProgress,
+		}
+
+		fsBackup, err = r.dataPathMgr.CreateFileSystemBR(du.Name, dataUploadDownloadRequestor, ctx, r.client, du.Namespace, callbacks, log)
+		if err != nil {
+			if err == datapath.ConcurrentLimitExceed {
+				log.Info("Data path instance is concurrent limited requeue later")
+				return ctrl.Result{Requeue: true, RequeueAfter: time.Minute}, nil
+			} else {
+				return r.errorOut(ctx, &du, err, "error to create data path", log)
+			}
+		}
 
 		// Update status to InProgress
 		original := du.DeepCopy()
@@ -171,7 +189,12 @@ func (r *DataUploadReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 		}
 
 		log.Info("Data upload is marked as in progress")
-		return r.runCancelableDataUpload(ctx, &du, res, log)
+		result, err := r.runCancelableDataUpload(ctx, fsBackup, &du, res, log)
+		if err != nil {
+			log.Errorf("Failed to run cancelable data path for %s with err %v", du.Name, err)
+			r.closeDataPath(ctx, du.Name)
+		}
+		return result, err
 	} else if du.Status.Phase == velerov2alpha1api.DataUploadPhaseInProgress {
 		log.Info("Data upload is in progress")
 		if du.Spec.Cancel {
@@ -198,25 +221,8 @@ func (r *DataUploadReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 	}
 }
 
-func (r *DataUploadReconciler) runCancelableDataUpload(ctx context.Context, du *velerov2alpha1api.DataUpload, res *exposer.ExposeResult, log logrus.FieldLogger) (reconcile.Result, error) {
-	log.Info("Creating data path routine")
-	callbacks := datapath.Callbacks{
-		OnCompleted: r.OnDataUploadCompleted,
-		OnFailed:    r.OnDataUploadFailed,
-		OnCancelled: r.OnDataUploadCancelled,
-		OnProgress:  r.OnDataUploadProgress,
-	}
-
-	fsBackup, err := r.dataPathMgr.CreateFileSystemBR(du.Name, dataUploadDownloadRequestor, ctx, r.client, du.Namespace, callbacks, log)
-	if err != nil {
-		if err == datapath.ConcurrentLimitExceed {
-			log.Info("runCancelableDataUpload is concurrent limited")
-			return ctrl.Result{Requeue: true, RequeueAfter: time.Minute}, nil
-		} else {
-			return r.errorOut(ctx, du, err, "error to create data path", log)
-		}
-	}
-
+func (r *DataUploadReconciler) runCancelableDataUpload(ctx context.Context, fsBackup datapath.AsyncBR, du *velerov2alpha1api.DataUpload, res *exposer.ExposeResult, log logrus.FieldLogger) (reconcile.Result, error) {
+	log.Info("Run cancelable dataUpload")
 	path, err := exposer.GetPodVolumeHostPath(ctx, res.ByPod.HostingPod, res.ByPod.PVC, r.client, r.fileSystem, log)
 	if err != nil {
 		return r.errorOut(ctx, du, err, "error exposing host path for pod volume", log)
@@ -410,7 +416,7 @@ func (r *DataUploadReconciler) findDataUploadForPod(podObj client.Object) []reco
 	}
 
 	r.logger.WithField("Backup pod", pod.Name).Infof("Preparing dataupload %s", du.Name)
-	if err := r.patchDataUpload(context.Background(), du, prepareDataUpload); err != nil {
+	if err := r.patchDataUpload(context.Background(), du, r.prepareDataUpload); err != nil {
 		r.logger.WithField("Backup pod", pod.Name).WithError(err).Error("failed to patch dataupload")
 		return []reconcile.Request{}
 	}
@@ -434,8 +440,9 @@ func (r *DataUploadReconciler) patchDataUpload(ctx context.Context, req *velerov
 	return nil
 }
 
-func prepareDataUpload(du *velerov2alpha1api.DataUpload) {
+func (r *DataUploadReconciler) prepareDataUpload(du *velerov2alpha1api.DataUpload) {
 	du.Status.Phase = velerov2alpha1api.DataUploadPhasePrepared
+	du.Status.Node = r.nodeName
 }
 
 func (r *DataUploadReconciler) errorOut(ctx context.Context, du *velerov2alpha1api.DataUpload, err error, msg string, log logrus.FieldLogger) (ctrl.Result, error) {
@@ -460,12 +467,11 @@ func (r *DataUploadReconciler) updateStatusToFailed(ctx context.Context, du *vel
 	}
 
 	du.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()}
-	if err = r.client.Patch(ctx, du, client.MergeFrom(original)); err != nil {
-		log.WithError(err).Error("error updating DataUpload status")
-		return err
+	if patchErr := r.client.Patch(ctx, du, client.MergeFrom(original)); patchErr != nil {
+		log.WithError(patchErr).Error("error updating DataUpload status")
 	}
 
-	return nil
+	return err
 }
 
 func (r *DataUploadReconciler) acceptDataUpload(ctx context.Context, du *velerov2alpha1api.DataUpload) (bool, error) {

pkg/controller/data_upload_controller_test.go

@@ -291,8 +291,8 @@ func TestReconcile(t *testing.T) {
 			expectedProcessed: true,
 			expected:          dataUploadBuilder().Phase(velerov2alpha1api.DataUploadPhaseFailed).Result(),
 			expectedRequeue:   ctrl.Result{},
-		},
-		{
+			expectedErrMsg:    "unknown type type of snapshot exposer is not exist",
+		}, {
 			name:              "Dataupload should be accepted",
 			du:                dataUploadBuilder().Result(),
 			pod:               builder.ForPod(velerov1api.DefaultNamespace, dataUploadName).Volumes(&corev1.Volume{Name: "dataupload-1"}).Result(),
@@ -336,7 +336,7 @@ func TestReconcile(t *testing.T) {
 			pod:               builder.ForPod(velerov1api.DefaultNamespace, dataUploadName).Volumes(&corev1.Volume{Name: "dataupload-1"}).Result(),
 			du:                dataUploadBuilder().Phase(velerov2alpha1api.DataUploadPhasePrepared).SnapshotType(fakeSnapshotType).Cancel(true).Result(),
 			expectedProcessed: false,
-			expected:          dataUploadBuilder().Phase(velerov2alpha1api.DataUploadPhaseInProgress).Result(),
+			expected:          dataUploadBuilder().Phase(velerov2alpha1api.DataUploadPhasePrepared).Result(),
 			expectedRequeue:   ctrl.Result{Requeue: true, RequeueAfter: time.Minute},
 		},
 	}
@@ -400,7 +400,7 @@ func TestReconcile(t *testing.T) {
 			if test.expectedErrMsg == "" {
 				require.NoError(t, err)
 			} else {
-				assert.Equal(t, err.Error(), test.expectedErrMsg)
+				assert.Contains(t, err.Error(), test.expectedErrMsg)
 			}
 
 			du := velerov2alpha1api.DataUpload{}

pkg/controller/download_request_controller.go

@@ -18,6 +18,7 @@ package controller
 
 import (
 	"context"
+	"time"
 
 	"github.com/pkg/errors"
 	"github.com/sirupsen/logrus"
@@ -25,12 +26,18 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	clocks "k8s.io/utils/clock"
 	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/builder"
 	kbclient "sigs.k8s.io/controller-runtime/pkg/client"
 
 	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
 	"github.com/vmware-tanzu/velero/pkg/itemoperationmap"
 	"github.com/vmware-tanzu/velero/pkg/persistence"
 	"github.com/vmware-tanzu/velero/pkg/plugin/clientmgmt"
+	"github.com/vmware-tanzu/velero/pkg/util/kube"
+)
+
+const (
+	defaultDownloadRequestSyncPeriod = time.Minute
 )
 
 // downloadRequestReconciler reconciles a DownloadRequest object
@@ -93,15 +100,6 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ
 		return ctrl.Result{}, errors.WithStack(err)
 	}
 
-	original := downloadRequest.DeepCopy()
-	defer func() {
-		// Always attempt to Patch the downloadRequest object and status after each reconciliation.
-		if err := r.client.Patch(ctx, downloadRequest, kbclient.MergeFrom(original)); err != nil {
-			log.WithError(err).Error("Error updating download request")
-			return
-		}
-	}()
-
 	if downloadRequest.Status != (velerov1api.DownloadRequestStatus{}) && downloadRequest.Status.Expiration != nil {
 		if downloadRequest.Status.Expiration.Time.Before(r.clock.Now()) {
 			// Delete any request that is expired, regardless of the phase: it is not
@@ -111,19 +109,25 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ
 				log.WithError(err).Error("Error deleting an expired download request")
 				return ctrl.Result{}, errors.WithStack(err)
 			}
-			return ctrl.Result{Requeue: false}, nil
+			return ctrl.Result{}, nil
 		} else if downloadRequest.Status.Phase == velerov1api.DownloadRequestPhaseProcessed {
-			// Requeue the request if is not yet expired and has already been processed before,
-			// since it might still be in use by the logs streaming and shouldn't
-			// be deleted until after its expiration.
-			log.Debug("DownloadRequest has not yet expired - requeueing")
-			return ctrl.Result{Requeue: true}, nil
+			log.Debug("DownloadRequest has not yet expired.")
+			return ctrl.Result{}, nil
 		}
 	}
 
 	// Process a brand new request.
-	backupName := downloadRequest.Spec.Target.Name
 	if downloadRequest.Status.Phase == "" || downloadRequest.Status.Phase == velerov1api.DownloadRequestPhaseNew {
+		backupName := downloadRequest.Spec.Target.Name
+		original := downloadRequest.DeepCopy()
+		defer func() {
+			// Always attempt to Patch the downloadRequest object and status for new DownloadRequest.
+			if err := r.client.Patch(ctx, downloadRequest, kbclient.MergeFrom(original)); err != nil {
+				log.WithError(err).Error("Error updating download request")
+				return
+			}
+		}()
+
 		// Update the expiration.
 		downloadRequest.Status.Expiration = &metav1.Time{Time: r.clock.Now().Add(persistence.DownloadURLTTL)}
 
@@ -136,6 +140,11 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ
 				Namespace: downloadRequest.Namespace,
 				Name:      downloadRequest.Spec.Target.Name,
 			}, restore); err != nil {
+				if apierrors.IsNotFound(err) {
+					log.WithError(err).Error("fail to get restore for DownloadRequest")
+					return ctrl.Result{}, nil
+				}
+				log.Warnf("Fail to get restore for DownloadRequest %s. Retry later.", err.Error())
 				return ctrl.Result{}, errors.WithStack(err)
 			}
 			backupName = restore.Spec.BackupName
@@ -146,6 +155,11 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ
 			Namespace: downloadRequest.Namespace,
 			Name:      backupName,
 		}, backup); err != nil {
+			if apierrors.IsNotFound(err) {
+				log.WithError(err).Error("fail to get backup for DownloadRequest")
+				return ctrl.Result{}, nil
+			}
+			log.Warnf("fail to get backup for DownloadRequest %s. Retry later.", err.Error())
 			return ctrl.Result{}, errors.WithStack(err)
 		}
 
@@ -154,6 +168,11 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ
 			Namespace: backup.Namespace,
 			Name:      backup.Spec.StorageLocation,
 		}, location); err != nil {
+			if apierrors.IsNotFound(err) {
+				log.Errorf("BSL for DownloadRequest cannot be found")
+				return ctrl.Result{}, nil
+			}
+			log.Warnf("Fail to get BSL for DownloadRequest: %s", err.Error())
 			return ctrl.Result{}, errors.WithStack(err)
 		}
 
@@ -163,7 +182,9 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ
 		backupStore, err := r.backupStoreGetter.Get(location, pluginManager, log)
 		if err != nil {
 			log.WithError(err).Error("Error getting a backup store")
-			return ctrl.Result{}, errors.WithStack(err)
+			// Fail to get backup store is due to BSL setting issue or credential issue.
+			// It cannot be recovered. No need to retry.
+			return ctrl.Result{}, nil
 		}
 
 		// If this is a request for backup item operations, force upload of in-memory operations that
@@ -180,8 +201,10 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ
 			// ignore errors here. If we can't upload anything here, process the download as usual
 			_ = r.restoreItemOperationsMap.UpdateForRestore(backupStore, downloadRequest.Spec.Target.Name)
 		}
+
 		if downloadRequest.Status.DownloadURL, err = backupStore.GetDownloadURL(downloadRequest.Spec.Target); err != nil {
-			return ctrl.Result{Requeue: true}, errors.WithStack(err)
+			log.Warnf("fail to get Backup metadata file's download URL %s, retry later: %s", downloadRequest.Spec.Target, err)
+			return ctrl.Result{}, errors.WithStack(err)
 		}
 
 		downloadRequest.Status.Phase = velerov1api.DownloadRequestPhaseProcessed
@@ -190,13 +213,22 @@ func (r *downloadRequestReconciler) Reconcile(ctx context.Context, req ctrl.Requ
 		downloadRequest.Status.Expiration = &metav1.Time{Time: r.clock.Now().Add(persistence.DownloadURLTTL)}
 	}
 
-	// Requeue is mostly to handle deleting any expired requests that were not
-	// deleted as part of the normal client flow for whatever reason.
-	return ctrl.Result{Requeue: true}, nil
+	return ctrl.Result{}, nil
 }
 
 func (r *downloadRequestReconciler) SetupWithManager(mgr ctrl.Manager) error {
+	downloadRequestSource := kube.NewPeriodicalEnqueueSource(r.log, mgr.GetClient(),
+		&velerov1api.DownloadRequestList{}, defaultDownloadRequestSyncPeriod, kube.PeriodicalEnqueueSourceOption{})
+	downloadRequestPredicates := kube.NewGenericEventPredicate(func(object kbclient.Object) bool {
+		downloadRequest := object.(*velerov1api.DownloadRequest)
+		if downloadRequest.Status != (velerov1api.DownloadRequestStatus{}) && downloadRequest.Status.Expiration != nil {
+			return downloadRequest.Status.Expiration.Time.Before(r.clock.Now())
+		}
+		return true
+	})
+
 	return ctrl.NewControllerManagedBy(mgr).
 		For(&velerov1api.DownloadRequest{}).
+		Watches(downloadRequestSource, nil, builder.WithPredicates(downloadRequestPredicates)).
 		Complete(r)
 }

pkg/controller/download_request_controller_test.go

@@ -156,55 +156,55 @@ var _ = Describe("Download Request Reconciler", func() {
 			}
 		},
 
-		Entry("backup contents request for nonexistent backup returns an error", request{
+		Entry("backup contents request for nonexistent backup returns nil", request{
 			downloadRequest:      builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindBackupContents, "a1-backup").Result(),
 			backup:               builder.ForBackup(velerov1api.DefaultNamespace, "non-matching-backup").StorageLocation("a-location").Result(),
 			backupLocation:       builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(),
-			expectedReconcileErr: "backups.velero.io \"a1-backup\" not found",
-			expectedRequeue:      ctrl.Result{Requeue: false},
+			expectedReconcileErr: "",
+			expectedRequeue:      ctrl.Result{},
 		}),
-		Entry("restore log request for nonexistent restore returns an error", request{
+		Entry("restore log request for nonexistent restore returns nil", request{
 			downloadRequest:      builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindRestoreLog, "a-backup-20170912150214").Result(),
 			restore:              builder.ForRestore(velerov1api.DefaultNamespace, "non-matching-restore").Phase(velerov1api.RestorePhaseCompleted).Backup("a-backup").Result(),
 			backup:               defaultBackup(),
 			backupLocation:       builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(),
-			expectedReconcileErr: "restores.velero.io \"a-backup-20170912150214\" not found",
-			expectedRequeue:      ctrl.Result{Requeue: false},
+			expectedReconcileErr: "",
+			expectedRequeue:      ctrl.Result{},
 		}),
-		Entry("backup contents request for backup with nonexistent location returns an error", request{
+		Entry("backup contents request for backup with nonexistent location returns nil", request{
 			downloadRequest:      builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindBackupContents, "a-backup").Result(),
 			backup:               defaultBackup(),
 			backupLocation:       builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "non-matching-location").Provider("a-provider").Bucket("a-bucket").Result(),
-			expectedReconcileErr: "backupstoragelocations.velero.io \"a-location\" not found",
-			expectedRequeue:      ctrl.Result{Requeue: false},
+			expectedReconcileErr: "",
+			expectedRequeue:      ctrl.Result{},
 		}),
 		Entry("backup contents request with phase '' gets a url", request{
 			downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindBackupContents, "a-backup").Result(),
 			backup:          defaultBackup(),
 			backupLocation:  builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(),
 			expectGetsURL:   true,
-			expectedRequeue: ctrl.Result{Requeue: true},
+			expectedRequeue: ctrl.Result{},
 		}),
 		Entry("backup contents request with phase 'New' gets a url", request{
 			downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase(velerov1api.DownloadRequestPhaseNew).Target(velerov1api.DownloadTargetKindBackupContents, "a-backup").Result(),
 			backup:          defaultBackup(),
 			backupLocation:  builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(),
 			expectGetsURL:   true,
-			expectedRequeue: ctrl.Result{Requeue: true},
+			expectedRequeue: ctrl.Result{},
 		}),
 		Entry("backup log request with phase '' gets a url", request{
 			downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindBackupLog, "a-backup").Result(),
 			backup:          defaultBackup(),
 			backupLocation:  builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(),
 			expectGetsURL:   true,
-			expectedRequeue: ctrl.Result{Requeue: true},
+			expectedRequeue: ctrl.Result{},
 		}),
 		Entry("backup log request with phase 'New' gets a url", request{
 			downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase(velerov1api.DownloadRequestPhaseNew).Target(velerov1api.DownloadTargetKindBackupLog, "a-backup").Result(),
 			backup:          defaultBackup(),
 			backupLocation:  builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(),
 			expectGetsURL:   true,
-			expectedRequeue: ctrl.Result{Requeue: true},
+			expectedRequeue: ctrl.Result{},
 		}),
 		Entry("restore log request with phase '' gets a url", request{
 			downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindRestoreLog, "a-backup-20170912150214").Result(),
@@ -212,7 +212,7 @@ var _ = Describe("Download Request Reconciler", func() {
 			backup:          defaultBackup(),
 			backupLocation:  builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(),
 			expectGetsURL:   true,
-			expectedRequeue: ctrl.Result{Requeue: true},
+			expectedRequeue: ctrl.Result{},
 		}),
 		Entry("restore log request with phase 'New' gets a url", request{
 			downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase(velerov1api.DownloadRequestPhaseNew).Target(velerov1api.DownloadTargetKindRestoreLog, "a-backup-20170912150214").Result(),
@@ -220,7 +220,7 @@ var _ = Describe("Download Request Reconciler", func() {
 			restore:         builder.ForRestore(velerov1api.DefaultNamespace, "a-backup-20170912150214").Phase(velerov1api.RestorePhaseCompleted).Backup("a-backup").Result(),
 			backupLocation:  builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(),
 			expectGetsURL:   true,
-			expectedRequeue: ctrl.Result{Requeue: true},
+			expectedRequeue: ctrl.Result{},
 		}),
 		Entry("restore results request with phase '' gets a url", request{
 			downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindRestoreResults, "a-backup-20170912150214").Result(),
@@ -228,7 +228,7 @@ var _ = Describe("Download Request Reconciler", func() {
 			backup:          defaultBackup(),
 			backupLocation:  builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(),
 			expectGetsURL:   true,
-			expectedRequeue: ctrl.Result{Requeue: true},
+			expectedRequeue: ctrl.Result{},
 		}),
 		Entry("restore results request with phase 'New' gets a url", request{
 			downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase(velerov1api.DownloadRequestPhaseNew).Target(velerov1api.DownloadTargetKindRestoreResults, "a-backup-20170912150214").Result(),
@@ -236,30 +236,30 @@ var _ = Describe("Download Request Reconciler", func() {
 			backup:          defaultBackup(),
 			backupLocation:  builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "a-location").Provider("a-provider").Bucket("a-bucket").Result(),
 			expectGetsURL:   true,
-			expectedRequeue: ctrl.Result{Requeue: true},
+			expectedRequeue: ctrl.Result{},
 		}),
 		Entry("request with phase 'Processed' and not expired is not deleted", request{
 			downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase(velerov1api.DownloadRequestPhaseProcessed).Target(velerov1api.DownloadTargetKindBackupLog, "a-backup-20170912150214").Result(),
 			backup:          defaultBackup(),
-			expectedRequeue: ctrl.Result{Requeue: true},
+			expectedRequeue: ctrl.Result{},
 		}),
 		Entry("request with phase 'Processed' and expired is deleted", request{
 			downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase(velerov1api.DownloadRequestPhaseProcessed).Target(velerov1api.DownloadTargetKindBackupLog, "a-backup-20170912150214").Result(),
 			backup:          defaultBackup(),
 			expired:         true,
-			expectedRequeue: ctrl.Result{Requeue: false},
+			expectedRequeue: ctrl.Result{},
 		}),
 		Entry("request with phase '' and expired is deleted", request{
 			downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase("").Target(velerov1api.DownloadTargetKindBackupLog, "a-backup-20170912150214").Result(),
 			backup:          defaultBackup(),
 			expired:         true,
-			expectedRequeue: ctrl.Result{Requeue: false},
+			expectedRequeue: ctrl.Result{},
 		}),
 		Entry("request with phase 'New' and expired is deleted", request{
 			downloadRequest: builder.ForDownloadRequest(velerov1api.DefaultNamespace, "a-download-request").Phase(velerov1api.DownloadRequestPhaseNew).Target(velerov1api.DownloadTargetKindBackupLog, "a-backup-20170912150214").Result(),
 			backup:          defaultBackup(),
 			expired:         true,
-			expectedRequeue: ctrl.Result{Requeue: false},
+			expectedRequeue: ctrl.Result{},
 		}),
 	)
 })

pkg/datapath/mocks/types.go

@@ -0,0 +1,86 @@
+// Code generated by mockery v2.20.0. DO NOT EDIT.
+
+package mocks
+
+import (
+	context "context"
+
+	credentials "github.com/vmware-tanzu/velero/internal/credentials"
+	datapath "github.com/vmware-tanzu/velero/pkg/datapath"
+
+	mock "github.com/stretchr/testify/mock"
+
+	repository "github.com/vmware-tanzu/velero/pkg/repository"
+)
+
+// AsyncBR is an autogenerated mock type for the AsyncBR type
+type AsyncBR struct {
+	mock.Mock
+}
+
+// Cancel provides a mock function with given fields:
+func (_m *AsyncBR) Cancel() {
+	_m.Called()
+}
+
+// Close provides a mock function with given fields: ctx
+func (_m *AsyncBR) Close(ctx context.Context) {
+	_m.Called(ctx)
+}
+
+// Init provides a mock function with given fields: ctx, bslName, sourceNamespace, uploaderType, repositoryType, repoIdentifier, repositoryEnsurer, credentialGetter
+func (_m *AsyncBR) Init(ctx context.Context, bslName string, sourceNamespace string, uploaderType string, repositoryType string, repoIdentifier string, repositoryEnsurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter) error {
+	ret := _m.Called(ctx, bslName, sourceNamespace, uploaderType, repositoryType, repoIdentifier, repositoryEnsurer, credentialGetter)
+
+	var r0 error
+	if rf, ok := ret.Get(0).(func(context.Context, string, string, string, string, string, *repository.Ensurer, *credentials.CredentialGetter) error); ok {
+		r0 = rf(ctx, bslName, sourceNamespace, uploaderType, repositoryType, repoIdentifier, repositoryEnsurer, credentialGetter)
+	} else {
+		r0 = ret.Error(0)
+	}
+
+	return r0
+}
+
+// StartBackup provides a mock function with given fields: source, realSource, parentSnapshot, forceFull, tags
+func (_m *AsyncBR) StartBackup(source datapath.AccessPoint, realSource string, parentSnapshot string, forceFull bool, tags map[string]string) error {
+	ret := _m.Called(source, realSource, parentSnapshot, forceFull, tags)
+
+	var r0 error
+	if rf, ok := ret.Get(0).(func(datapath.AccessPoint, string, string, bool, map[string]string) error); ok {
+		r0 = rf(source, realSource, parentSnapshot, forceFull, tags)
+	} else {
+		r0 = ret.Error(0)
+	}
+
+	return r0
+}
+
+// StartRestore provides a mock function with given fields: snapshotID, target
+func (_m *AsyncBR) StartRestore(snapshotID string, target datapath.AccessPoint) error {
+	ret := _m.Called(snapshotID, target)
+
+	var r0 error
+	if rf, ok := ret.Get(0).(func(string, datapath.AccessPoint) error); ok {
+		r0 = rf(snapshotID, target)
+	} else {
+		r0 = ret.Error(0)
+	}
+
+	return r0
+}
+
+type mockConstructorTestingTNewAsyncBR interface {
+	mock.TestingT
+	Cleanup(func())
+}
+
+// NewAsyncBR creates a new instance of AsyncBR. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
+func NewAsyncBR(t mockConstructorTestingTNewAsyncBR) *AsyncBR {
+	mock := &AsyncBR{}
+	mock.Mock.Test(t)
+
+	t.Cleanup(func() { mock.AssertExpectations(t) })
+
+	return mock
+}

pkg/exposer/csi_snapshot.go

@@ -202,7 +202,7 @@ func (e *csiSnapshotExposer) GetExposed(ctx context.Context, ownerObject corev1.
 	}, pod)
 	if err != nil {
 		if apierrors.IsNotFound(err) {
-			curLog.WithField("backup pod", backupPodName).Errorf("Backup pod is not running in the current node %s", exposeWaitParam.NodeName)
+			curLog.WithField("backup pod", backupPodName).Debugf("Backup pod is not running in the current node %s", exposeWaitParam.NodeName)
 			return nil, nil
 		} else {
 			return nil, errors.Wrapf(err, "error to get backup pod %s", backupPodName)

pkg/exposer/generic_restore.go

@@ -123,7 +123,7 @@ func (e *genericRestoreExposer) GetExposed(ctx context.Context, ownerObject core
 	}, pod)
 	if err != nil {
 		if apierrors.IsNotFound(err) {
-			curLog.WithField("backup pod", restorePodName).Error("Backup pod is not running in the current node")
+			curLog.WithField("backup pod", restorePodName).Debug("Backup pod is not running in the current node")
 			return nil, nil
 		} else {
 			return nil, errors.Wrapf(err, "error to get backup pod %s", restorePodName)

pkg/exposer/mocks/generic_restore.go

@@ -0,0 +1,96 @@
+// Code generated by mockery v2.20.0. DO NOT EDIT.
+
+package mocks
+
+import (
+	context "context"
+
+	client "sigs.k8s.io/controller-runtime/pkg/client"
+
+	exposer "github.com/vmware-tanzu/velero/pkg/exposer"
+
+	mock "github.com/stretchr/testify/mock"
+
+	time "time"
+
+	v1 "k8s.io/api/core/v1"
+)
+
+// GenericRestoreExposer is an autogenerated mock type for the GenericRestoreExposer type
+type GenericRestoreExposer struct {
+	mock.Mock
+}
+
+// CleanUp provides a mock function with given fields: _a0, _a1
+func (_m *GenericRestoreExposer) CleanUp(_a0 context.Context, _a1 v1.ObjectReference) {
+	_m.Called(_a0, _a1)
+}
+
+// Expose provides a mock function with given fields: _a0, _a1, _a2, _a3, _a4, _a5
+func (_m *GenericRestoreExposer) Expose(_a0 context.Context, _a1 v1.ObjectReference, _a2 string, _a3 string, _a4 map[string]string, _a5 time.Duration) error {
+	ret := _m.Called(_a0, _a1, _a2, _a3, _a4, _a5)
+
+	var r0 error
+	if rf, ok := ret.Get(0).(func(context.Context, v1.ObjectReference, string, string, map[string]string, time.Duration) error); ok {
+		r0 = rf(_a0, _a1, _a2, _a3, _a4, _a5)
+	} else {
+		r0 = ret.Error(0)
+	}
+
+	return r0
+}
+
+// GetExposed provides a mock function with given fields: _a0, _a1, _a2, _a3, _a4
+func (_m *GenericRestoreExposer) GetExposed(_a0 context.Context, _a1 v1.ObjectReference, _a2 client.Client, _a3 string, _a4 time.Duration) (*exposer.ExposeResult, error) {
+	ret := _m.Called(_a0, _a1, _a2, _a3, _a4)
+
+	var r0 *exposer.ExposeResult
+	var r1 error
+	if rf, ok := ret.Get(0).(func(context.Context, v1.ObjectReference, client.Client, string, time.Duration) (*exposer.ExposeResult, error)); ok {
+		return rf(_a0, _a1, _a2, _a3, _a4)
+	}
+	if rf, ok := ret.Get(0).(func(context.Context, v1.ObjectReference, client.Client, string, time.Duration) *exposer.ExposeResult); ok {
+		r0 = rf(_a0, _a1, _a2, _a3, _a4)
+	} else {
+		if ret.Get(0) != nil {
+			r0 = ret.Get(0).(*exposer.ExposeResult)
+		}
+	}
+
+	if rf, ok := ret.Get(1).(func(context.Context, v1.ObjectReference, client.Client, string, time.Duration) error); ok {
+		r1 = rf(_a0, _a1, _a2, _a3, _a4)
+	} else {
+		r1 = ret.Error(1)
+	}
+
+	return r0, r1
+}
+
+// RebindVolume provides a mock function with given fields: _a0, _a1, _a2, _a3, _a4
+func (_m *GenericRestoreExposer) RebindVolume(_a0 context.Context, _a1 v1.ObjectReference, _a2 string, _a3 string, _a4 time.Duration) error {
+	ret := _m.Called(_a0, _a1, _a2, _a3, _a4)
+
+	var r0 error
+	if rf, ok := ret.Get(0).(func(context.Context, v1.ObjectReference, string, string, time.Duration) error); ok {
+		r0 = rf(_a0, _a1, _a2, _a3, _a4)
+	} else {
+		r0 = ret.Error(0)
+	}
+
+	return r0
+}
+
+type mockConstructorTestingTNewGenericRestoreExposer interface {
+	mock.TestingT
+	Cleanup(func())
+}
+
+// NewGenericRestoreExposer creates a new instance of GenericRestoreExposer. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
+func NewGenericRestoreExposer(t mockConstructorTestingTNewGenericRestoreExposer) *GenericRestoreExposer {
+	mock := &GenericRestoreExposer{}
+	mock.Mock.Test(t)
+
+	t.Cleanup(func() { mock.AssertExpectations(t) })
+
+	return mock
+}

pkg/persistence/object_store_test.go

@@ -27,6 +27,7 @@ import (
 	"strings"
 	"testing"
 
+	snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v4/apis/volumesnapshot/v1"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -493,6 +494,55 @@ func TestGetBackupItemOperations(t *testing.T) {
 	assert.EqualValues(t, operations, res)
 }
 
+func TestGetRestoreItemOperations(t *testing.T) {
+	harness := newObjectBackupStoreTestHarness("test-bucket", "")
+
+	// itemoperations file not found should not error
+	res, err := harness.GetRestoreItemOperations("test-restore")
+	assert.NoError(t, err)
+	assert.Nil(t, res)
+
+	// itemoperations file containing invalid data should error
+	harness.objectStore.PutObject(harness.bucket, "restores/test-restore/restore-test-restore-itemoperations.json.gz", newStringReadSeeker("foo"))
+	_, err = harness.GetRestoreItemOperations("test-restore")
+	assert.NotNil(t, err)
+
+	// itemoperations file containing gzipped json data should return correctly
+	operations := []*itemoperation.RestoreOperation{
+		{
+			Spec: itemoperation.RestoreOperationSpec{
+				RestoreName: "test-restore",
+				ResourceIdentifier: velero.ResourceIdentifier{
+					GroupResource: kuberesource.Pods,
+					Namespace:     "ns",
+					Name:          "item-1",
+				},
+			},
+		},
+		{
+			Spec: itemoperation.RestoreOperationSpec{
+				RestoreName: "test-restore",
+				ResourceIdentifier: velero.ResourceIdentifier{
+					GroupResource: kuberesource.Pods,
+					Namespace:     "ns",
+					Name:          "item-2",
+				},
+			},
+		},
+	}
+
+	obj := new(bytes.Buffer)
+	gzw := gzip.NewWriter(obj)
+
+	require.NoError(t, json.NewEncoder(gzw).Encode(operations))
+	require.NoError(t, gzw.Close())
+	require.NoError(t, harness.objectStore.PutObject(harness.bucket, "restores/test-restore/restore-test-restore-itemoperations.json.gz", obj))
+
+	res, err = harness.GetRestoreItemOperations("test-restore")
+	assert.NoError(t, err)
+	assert.EqualValues(t, operations, res)
+}
+
 func TestGetBackupContents(t *testing.T) {
 	harness := newObjectBackupStoreTestHarness("test-bucket", "")
 
@@ -559,6 +609,58 @@ func TestDeleteBackup(t *testing.T) {
 	}
 }
 
+func TestDeleteRestore(t *testing.T) {
+	tests := []struct {
+		name             string
+		prefix           string
+		listObjectsError error
+		deleteErrors     []error
+		expectedErr      string
+	}{
+		{
+			name: "normal case",
+		},
+		{
+			name:   "normal case with backup store prefix",
+			prefix: "velero-backups/",
+		},
+		{
+			name:         "some delete errors, do as much as we can",
+			deleteErrors: []error{errors.New("a"), nil, errors.New("c")},
+			expectedErr:  "[a, c]",
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			objectStore := new(providermocks.ObjectStore)
+			backupStore := &objectBackupStore{
+				objectStore: objectStore,
+				bucket:      "test-bucket",
+				layout:      NewObjectStoreLayout(test.prefix),
+				logger:      velerotest.NewLogger(),
+			}
+			defer objectStore.AssertExpectations(t)
+
+			objects := []string{test.prefix + "restores/bak/velero-restore.json", test.prefix + "restores/bak/bak.tar.gz", test.prefix + "restores/bak/bak.log.gz"}
+
+			objectStore.On("ListObjects", backupStore.bucket, test.prefix+"restores/bak/").Return(objects, test.listObjectsError)
+			for i, obj := range objects {
+				var err error
+				if i < len(test.deleteErrors) {
+					err = test.deleteErrors[i]
+				}
+
+				objectStore.On("DeleteObject", backupStore.bucket, obj).Return(err)
+			}
+
+			err := backupStore.DeleteRestore("bak")
+
+			velerotest.AssertErrorMatches(t, test.expectedErr, err)
+		})
+	}
+}
+
 func TestGetDownloadURL(t *testing.T) {
 	tests := []struct {
 		name              string
@@ -673,6 +775,108 @@ func TestGetDownloadURL(t *testing.T) {
 	}
 }
 
+func TestGetCSIVolumeSnapshotClasses(t *testing.T) {
+	harness := newObjectBackupStoreTestHarness("test-bucket", "")
+
+	// file not found should not error
+	res, err := harness.GetCSIVolumeSnapshotClasses("test-backup")
+	assert.NoError(t, err)
+	assert.Nil(t, res)
+
+	// file containing invalid data should error
+	harness.objectStore.PutObject(harness.bucket, "backups/test-backup/test-backup-csi-volumesnapshotclasses.json.gz", newStringReadSeeker("foo"))
+	_, err = harness.GetCSIVolumeSnapshotClasses("test-backup")
+	assert.NotNil(t, err)
+
+	// file containing gzipped json data should return correctly
+	classes := []*snapshotv1api.VolumeSnapshotClass{
+		{
+			Driver: "driver",
+		},
+	}
+
+	obj := new(bytes.Buffer)
+	gzw := gzip.NewWriter(obj)
+
+	require.NoError(t, json.NewEncoder(gzw).Encode(classes))
+	require.NoError(t, gzw.Close())
+	require.NoError(t, harness.objectStore.PutObject(harness.bucket, "backups/test-backup/test-backup-csi-volumesnapshotclasses.json.gz", obj))
+
+	res, err = harness.GetCSIVolumeSnapshotClasses("test-backup")
+	assert.NoError(t, err)
+	assert.EqualValues(t, classes, res)
+}
+
+func TestGetCSIVolumeSnapshots(t *testing.T) {
+	harness := newObjectBackupStoreTestHarness("test-bucket", "")
+
+	// file not found should not error
+	res, err := harness.GetCSIVolumeSnapshots("test-backup")
+	assert.NoError(t, err)
+	assert.Nil(t, res)
+
+	// file containing invalid data should error
+	harness.objectStore.PutObject(harness.bucket, "backups/test-backup/test-backup-csi-volumesnapshots.json.gz", newStringReadSeeker("foo"))
+	_, err = harness.GetCSIVolumeSnapshots("test-backup")
+	assert.NotNil(t, err)
+
+	// file containing gzipped json data should return correctly
+	snapshots := []*snapshotv1api.VolumeSnapshot{
+		{
+			Spec: snapshotv1api.VolumeSnapshotSpec{
+				Source: snapshotv1api.VolumeSnapshotSource{
+					VolumeSnapshotContentName: nil,
+				},
+			},
+		},
+	}
+
+	obj := new(bytes.Buffer)
+	gzw := gzip.NewWriter(obj)
+
+	require.NoError(t, json.NewEncoder(gzw).Encode(snapshots))
+	require.NoError(t, gzw.Close())
+	require.NoError(t, harness.objectStore.PutObject(harness.bucket, "backups/test-backup/test-backup-csi-volumesnapshots.json.gz", obj))
+
+	res, err = harness.GetCSIVolumeSnapshots("test-backup")
+	assert.NoError(t, err)
+	assert.EqualValues(t, snapshots, res)
+}
+
+func TestGetCSIVolumeSnapshotContents(t *testing.T) {
+	harness := newObjectBackupStoreTestHarness("test-bucket", "")
+
+	// file not found should not error
+	res, err := harness.GetCSIVolumeSnapshotContents("test-backup")
+	assert.NoError(t, err)
+	assert.Nil(t, res)
+
+	// file containing invalid data should error
+	harness.objectStore.PutObject(harness.bucket, "backups/test-backup/test-backup-csi-volumesnapshotcontents.json.gz", newStringReadSeeker("foo"))
+	_, err = harness.GetCSIVolumeSnapshotContents("test-backup")
+	assert.NotNil(t, err)
+
+	// file containing gzipped json data should return correctly
+	contents := []*snapshotv1api.VolumeSnapshotContent{
+		{
+			Spec: snapshotv1api.VolumeSnapshotContentSpec{
+				Driver: "driver",
+			},
+		},
+	}
+
+	obj := new(bytes.Buffer)
+	gzw := gzip.NewWriter(obj)
+
+	require.NoError(t, json.NewEncoder(gzw).Encode(contents))
+	require.NoError(t, gzw.Close())
+	require.NoError(t, harness.objectStore.PutObject(harness.bucket, "backups/test-backup/test-backup-csi-volumesnapshotcontents.json.gz", obj))
+
+	res, err = harness.GetCSIVolumeSnapshotContents("test-backup")
+	assert.NoError(t, err)
+	assert.EqualValues(t, contents, res)
+}
+
 type objectStoreGetter map[string]velero.ObjectStore
 
 func (osg objectStoreGetter) GetObjectStore(provider string) (velero.ObjectStore, error) {

pkg/repository/ensurer.go

@@ -80,20 +80,18 @@ func (r *Ensurer) EnsureRepo(ctx context.Context, namespace, volumeNamespace, ba
 		log.Debug("Released lock")
 	}()
 
-	repo, err := GetBackupRepository(ctx, r.repoClient, namespace, backupRepoKey, true)
+	_, err := GetBackupRepository(ctx, r.repoClient, namespace, backupRepoKey, false)
 	if err == nil {
-		log.Debug("Ready repository found")
-		return repo, nil
-	}
+		log.Info("Founding existing repo")
+		return r.waitBackupRepository(ctx, namespace, backupRepoKey)
+	} else if isBackupRepositoryNotFoundError(err) {
+		log.Info("No repository found, creating one")
 
-	if !isBackupRepositoryNotFoundError(err) {
+		// no repo found: create one and wait for it to be ready
+		return r.createBackupRepositoryAndWait(ctx, namespace, backupRepoKey)
+	} else {
 		return nil, errors.WithStack(err)
 	}
-
-	log.Debug("No repository found, creating one")
-
-	// no repo found: create one and wait for it to be ready
-	return r.createBackupRepositoryAndWait(ctx, namespace, backupRepoKey)
 }
 
 func (r *Ensurer) repoLock(key BackupRepositoryKey) *sync.Mutex {
@@ -113,6 +111,10 @@ func (r *Ensurer) createBackupRepositoryAndWait(ctx context.Context, namespace s
 		return nil, errors.Wrap(err, "unable to create backup repository resource")
 	}
 
+	return r.waitBackupRepository(ctx, namespace, backupRepoKey)
+}
+
+func (r *Ensurer) waitBackupRepository(ctx context.Context, namespace string, backupRepoKey BackupRepositoryKey) (*velerov1api.BackupRepository, error) {
 	var repo *velerov1api.BackupRepository
 	checkFunc := func(ctx context.Context) (bool, error) {
 		found, err := GetBackupRepository(ctx, r.repoClient, namespace, backupRepoKey, true)

pkg/repository/ensurer_test.go

@@ -30,13 +30,19 @@ import (
 )
 
 func TestEnsureRepo(t *testing.T) {
-	bkRepoObj := NewBackupRepository(velerov1.DefaultNamespace, BackupRepositoryKey{
+	bkRepoObjReady := NewBackupRepository(velerov1.DefaultNamespace, BackupRepositoryKey{
 		VolumeNamespace: "fake-ns",
 		BackupLocation:  "fake-bsl",
 		RepositoryType:  "fake-repo-type",
 	})
 
-	bkRepoObj.Status.Phase = velerov1.BackupRepositoryPhaseReady
+	bkRepoObjReady.Status.Phase = velerov1.BackupRepositoryPhaseReady
+
+	bkRepoObjNotReady := NewBackupRepository(velerov1.DefaultNamespace, BackupRepositoryKey{
+		VolumeNamespace: "fake-ns",
+		BackupLocation:  "fake-bsl",
+		RepositoryType:  "fake-repo-type",
+	})
 
 	scheme := runtime.NewScheme()
 	velerov1.AddToScheme(scheme)
@@ -82,10 +88,21 @@ func TestEnsureRepo(t *testing.T) {
 			bsl:            "fake-bsl",
 			repositoryType: "fake-repo-type",
 			kubeClientObj: []runtime.Object{
-				bkRepoObj,
+				bkRepoObjReady,
 			},
 			runtimeScheme: scheme,
-			expectedRepo:  bkRepoObj,
+			expectedRepo:  bkRepoObjReady,
+		},
+		{
+			name:           "wait existing repo fail",
+			namespace:      "fake-ns",
+			bsl:            "fake-bsl",
+			repositoryType: "fake-repo-type",
+			kubeClientObj: []runtime.Object{
+				bkRepoObjNotReady,
+			},
+			runtimeScheme: scheme,
+			err:           "failed to wait BackupRepository: timed out waiting for the condition",
 		},
 		{
 			name:           "create fail",

pkg/repository/provider/unified_repo_test.go

@@ -887,3 +887,467 @@ func TestForget(t *testing.T) {
 		})
 	}
 }
+
+func TestInitRepo(t *testing.T) {
+	testCases := []struct {
+		name            string
+		funcTable       localFuncTable
+		getter          *credmock.SecretStore
+		repoService     *reposervicenmocks.BackupRepoService
+		retFuncInit     interface{}
+		credStoreReturn string
+		credStoreError  error
+		expectedErr     string
+	}{
+		{
+			name:        "get repo option fail",
+			expectedErr: "error to get repo options: error to get repo password: invalid credentials interface",
+		},
+		{
+			name:            "repo init fail",
+			getter:          new(credmock.SecretStore),
+			credStoreReturn: "fake-password",
+			funcTable: localFuncTable{
+				getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+				getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+			},
+			repoService: new(reposervicenmocks.BackupRepoService),
+			retFuncInit: func(context.Context, udmrepo.RepoOptions, bool) error {
+				return errors.New("fake-error-1")
+			},
+			expectedErr: "error to init backup repo: fake-error-1",
+		},
+		{
+			name:            "succeed",
+			getter:          new(credmock.SecretStore),
+			credStoreReturn: "fake-password",
+			funcTable: localFuncTable{
+				getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+				getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+			},
+			repoService: new(reposervicenmocks.BackupRepoService),
+			retFuncInit: func(context.Context, udmrepo.RepoOptions, bool) error {
+				return nil
+			},
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			funcTable = tc.funcTable
+
+			var secretStore velerocredentials.SecretStore
+			if tc.getter != nil {
+				tc.getter.On("Get", mock.Anything, mock.Anything).Return(tc.credStoreReturn, tc.credStoreError)
+				secretStore = tc.getter
+			}
+
+			urp := unifiedRepoProvider{
+				credentialGetter: velerocredentials.CredentialGetter{
+					FromSecret: secretStore,
+				},
+				repoService: tc.repoService,
+				log:         velerotest.NewLogger(),
+			}
+
+			if tc.repoService != nil {
+				tc.repoService.On("Init", mock.Anything, mock.Anything, mock.Anything).Return(tc.retFuncInit)
+			}
+
+			err := urp.InitRepo(context.Background(), RepoParam{
+				BackupLocation: &velerov1api.BackupStorageLocation{},
+				BackupRepo:     &velerov1api.BackupRepository{},
+			})
+
+			if tc.expectedErr == "" {
+				assert.NoError(t, err)
+			} else {
+				assert.EqualError(t, err, tc.expectedErr)
+			}
+		})
+	}
+}
+
+func TestConnectToRepo(t *testing.T) {
+	testCases := []struct {
+		name            string
+		funcTable       localFuncTable
+		getter          *credmock.SecretStore
+		repoService     *reposervicenmocks.BackupRepoService
+		retFuncInit     interface{}
+		credStoreReturn string
+		credStoreError  error
+		expectedErr     string
+	}{
+		{
+			name:        "get repo option fail",
+			expectedErr: "error to get repo options: error to get repo password: invalid credentials interface",
+		},
+		{
+			name:            "repo init fail",
+			getter:          new(credmock.SecretStore),
+			credStoreReturn: "fake-password",
+			funcTable: localFuncTable{
+				getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+				getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+			},
+			repoService: new(reposervicenmocks.BackupRepoService),
+			retFuncInit: func(context.Context, udmrepo.RepoOptions, bool) error {
+				return errors.New("fake-error-1")
+			},
+			expectedErr: "error to connect backup repo: fake-error-1",
+		},
+		{
+			name:            "succeed",
+			getter:          new(credmock.SecretStore),
+			credStoreReturn: "fake-password",
+			funcTable: localFuncTable{
+				getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+				getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+			},
+			repoService: new(reposervicenmocks.BackupRepoService),
+			retFuncInit: func(context.Context, udmrepo.RepoOptions, bool) error {
+				return nil
+			},
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			funcTable = tc.funcTable
+
+			var secretStore velerocredentials.SecretStore
+			if tc.getter != nil {
+				tc.getter.On("Get", mock.Anything, mock.Anything).Return(tc.credStoreReturn, tc.credStoreError)
+				secretStore = tc.getter
+			}
+
+			urp := unifiedRepoProvider{
+				credentialGetter: velerocredentials.CredentialGetter{
+					FromSecret: secretStore,
+				},
+				repoService: tc.repoService,
+				log:         velerotest.NewLogger(),
+			}
+
+			if tc.repoService != nil {
+				tc.repoService.On("Init", mock.Anything, mock.Anything, mock.Anything).Return(tc.retFuncInit)
+			}
+
+			err := urp.ConnectToRepo(context.Background(), RepoParam{
+				BackupLocation: &velerov1api.BackupStorageLocation{},
+				BackupRepo:     &velerov1api.BackupRepository{},
+			})
+
+			if tc.expectedErr == "" {
+				assert.NoError(t, err)
+			} else {
+				assert.EqualError(t, err, tc.expectedErr)
+			}
+		})
+	}
+}
+
+func TestBoostRepoConnect(t *testing.T) {
+	var backupRepo *reposervicenmocks.BackupRepo
+
+	testCases := []struct {
+		name            string
+		funcTable       localFuncTable
+		getter          *credmock.SecretStore
+		repoService     *reposervicenmocks.BackupRepoService
+		backupRepo      *reposervicenmocks.BackupRepo
+		retFuncInit     interface{}
+		retFuncOpen     []interface{}
+		credStoreReturn string
+		credStoreError  error
+		expectedErr     string
+	}{
+		{
+			name:        "get repo option fail",
+			expectedErr: "error to get repo options: error to get repo password: invalid credentials interface",
+		},
+		{
+			name:            "repo not opened and connect fail",
+			getter:          new(credmock.SecretStore),
+			credStoreReturn: "fake-password",
+			funcTable: localFuncTable{
+				getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+				getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+			},
+			repoService: new(reposervicenmocks.BackupRepoService),
+			retFuncOpen: []interface{}{
+				func(context.Context, udmrepo.RepoOptions) udmrepo.BackupRepo {
+					return backupRepo
+				},
+
+				func(context.Context, udmrepo.RepoOptions) error {
+					return errors.New("fake-error-1")
+				},
+			},
+			retFuncInit: func(context.Context, udmrepo.RepoOptions, bool) error {
+				return errors.New("fake-error-2")
+			},
+			expectedErr: "error to connect backup repo: fake-error-2",
+		},
+		{
+			name:            "repo not opened and connect succeed",
+			getter:          new(credmock.SecretStore),
+			credStoreReturn: "fake-password",
+			funcTable: localFuncTable{
+				getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+				getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+			},
+			repoService: new(reposervicenmocks.BackupRepoService),
+			retFuncOpen: []interface{}{
+				func(context.Context, udmrepo.RepoOptions) udmrepo.BackupRepo {
+					return backupRepo
+				},
+
+				func(context.Context, udmrepo.RepoOptions) error {
+					return errors.New("fake-error-1")
+				},
+			},
+			retFuncInit: func(context.Context, udmrepo.RepoOptions, bool) error {
+				return nil
+			},
+		},
+		{
+			name:            "repo is opened",
+			getter:          new(credmock.SecretStore),
+			credStoreReturn: "fake-password",
+			funcTable: localFuncTable{
+				getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+				getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+			},
+			repoService: new(reposervicenmocks.BackupRepoService),
+			backupRepo:  new(reposervicenmocks.BackupRepo),
+			retFuncOpen: []interface{}{
+				func(context.Context, udmrepo.RepoOptions) udmrepo.BackupRepo {
+					return backupRepo
+				},
+
+				func(context.Context, udmrepo.RepoOptions) error {
+					return nil
+				},
+			},
+			retFuncInit: func(context.Context, udmrepo.RepoOptions, bool) error {
+				return nil
+			},
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			funcTable = tc.funcTable
+
+			var secretStore velerocredentials.SecretStore
+			if tc.getter != nil {
+				tc.getter.On("Get", mock.Anything, mock.Anything).Return(tc.credStoreReturn, tc.credStoreError)
+				secretStore = tc.getter
+			}
+
+			urp := unifiedRepoProvider{
+				credentialGetter: velerocredentials.CredentialGetter{
+					FromSecret: secretStore,
+				},
+				repoService: tc.repoService,
+				log:         velerotest.NewLogger(),
+			}
+
+			backupRepo = tc.backupRepo
+
+			if tc.repoService != nil {
+				tc.repoService.On("Open", mock.Anything, mock.Anything).Return(tc.retFuncOpen[0], tc.retFuncOpen[1])
+				tc.repoService.On("Init", mock.Anything, mock.Anything, mock.Anything).Return(tc.retFuncInit)
+			}
+
+			if tc.backupRepo != nil {
+				backupRepo.On("Close", mock.Anything).Return(nil)
+			}
+
+			err := urp.BoostRepoConnect(context.Background(), RepoParam{
+				BackupLocation: &velerov1api.BackupStorageLocation{},
+				BackupRepo:     &velerov1api.BackupRepository{},
+			})
+
+			if tc.expectedErr == "" {
+				assert.NoError(t, err)
+			} else {
+				assert.EqualError(t, err, tc.expectedErr)
+			}
+		})
+	}
+}
+
+func TestPruneRepo(t *testing.T) {
+	testCases := []struct {
+		name            string
+		funcTable       localFuncTable
+		getter          *credmock.SecretStore
+		repoService     *reposervicenmocks.BackupRepoService
+		retFuncMaintain interface{}
+		credStoreReturn string
+		credStoreError  error
+		expectedErr     string
+	}{
+		{
+			name:        "get repo option fail",
+			expectedErr: "error to get repo options: error to get repo password: invalid credentials interface",
+		},
+		{
+			name:            "repo maintain fail",
+			getter:          new(credmock.SecretStore),
+			credStoreReturn: "fake-password",
+			funcTable: localFuncTable{
+				getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+				getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+			},
+			repoService: new(reposervicenmocks.BackupRepoService),
+			retFuncMaintain: func(context.Context, udmrepo.RepoOptions) error {
+				return errors.New("fake-error-1")
+			},
+			expectedErr: "error to prune backup repo: fake-error-1",
+		},
+		{
+			name:            "succeed",
+			getter:          new(credmock.SecretStore),
+			credStoreReturn: "fake-password",
+			funcTable: localFuncTable{
+				getStorageVariables: func(*velerov1api.BackupStorageLocation, string, string) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+				getStorageCredentials: func(*velerov1api.BackupStorageLocation, velerocredentials.FileStore) (map[string]string, error) {
+					return map[string]string{}, nil
+				},
+			},
+			repoService: new(reposervicenmocks.BackupRepoService),
+			retFuncMaintain: func(context.Context, udmrepo.RepoOptions) error {
+				return nil
+			},
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			funcTable = tc.funcTable
+
+			var secretStore velerocredentials.SecretStore
+			if tc.getter != nil {
+				tc.getter.On("Get", mock.Anything, mock.Anything).Return(tc.credStoreReturn, tc.credStoreError)
+				secretStore = tc.getter
+			}
+
+			urp := unifiedRepoProvider{
+				credentialGetter: velerocredentials.CredentialGetter{
+					FromSecret: secretStore,
+				},
+				repoService: tc.repoService,
+				log:         velerotest.NewLogger(),
+			}
+
+			if tc.repoService != nil {
+				tc.repoService.On("Maintain", mock.Anything, mock.Anything).Return(tc.retFuncMaintain)
+			}
+
+			err := urp.PruneRepo(context.Background(), RepoParam{
+				BackupLocation: &velerov1api.BackupStorageLocation{},
+				BackupRepo:     &velerov1api.BackupRepository{},
+			})
+
+			if tc.expectedErr == "" {
+				assert.NoError(t, err)
+			} else {
+				assert.EqualError(t, err, tc.expectedErr)
+			}
+		})
+	}
+}
+
+func TestGetStorageType(t *testing.T) {
+	testCases := []struct {
+		name           string
+		backupLocation *velerov1api.BackupStorageLocation
+		expectedRet    string
+	}{
+		{
+			name:           "wrong backend type",
+			backupLocation: &velerov1api.BackupStorageLocation{},
+		},
+		{
+			name: "aws provider",
+			backupLocation: &velerov1api.BackupStorageLocation{
+				Spec: velerov1api.BackupStorageLocationSpec{
+					Provider: "velero.io/aws",
+				},
+			},
+			expectedRet: "s3",
+		},
+		{
+			name: "azure provider",
+			backupLocation: &velerov1api.BackupStorageLocation{
+				Spec: velerov1api.BackupStorageLocationSpec{
+					Provider: "velero.io/azure",
+				},
+			},
+			expectedRet: "azure",
+		},
+		{
+			name: "gcp provider",
+			backupLocation: &velerov1api.BackupStorageLocation{
+				Spec: velerov1api.BackupStorageLocationSpec{
+					Provider: "velero.io/gcp",
+				},
+			},
+			expectedRet: "gcs",
+		},
+		{
+			name: "fs provider",
+			backupLocation: &velerov1api.BackupStorageLocation{
+				Spec: velerov1api.BackupStorageLocationSpec{
+					Provider: "velero.io/fs",
+				},
+			},
+			expectedRet: "filesystem",
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			ret := getStorageType(tc.backupLocation)
+			assert.Equal(t, tc.expectedRet, ret)
+		})
+	}
+}

pkg/repository/restic/repository.go

@@ -112,7 +112,7 @@ func (r *RepositoryService) exec(cmd *restic.Command, bsl *velerov1api.BackupSto
 		cmd.ExtraFlags = append(cmd.ExtraFlags, skipTLSRet)
 	}
 
-	stdout, stderr, err := veleroexec.RunCommand(cmd.Cmd())
+	stdout, stderr, err := veleroexec.RunCommandWithLog(cmd.Cmd(), r.log)
 	r.log.WithFields(logrus.Fields{
 		"repository": cmd.RepoName(),
 		"command":    cmd.String(),

pkg/repository/udmrepo/kopialib/backend/common_test.go

@@ -0,0 +1,198 @@
+/*
+Copyright the Velero contributors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package backend
+
+import (
+	"context"
+	"testing"
+	"time"
+
+	"github.com/kopia/kopia/repo"
+	"github.com/kopia/kopia/repo/content"
+	"github.com/kopia/kopia/repo/encryption"
+	"github.com/kopia/kopia/repo/format"
+	"github.com/kopia/kopia/repo/hashing"
+	"github.com/kopia/kopia/repo/splitter"
+	"github.com/stretchr/testify/assert"
+
+	"github.com/vmware-tanzu/velero/pkg/repository/udmrepo"
+)
+
+func TestSetupNewRepositoryOptions(t *testing.T) {
+	testCases := []struct {
+		name     string
+		flags    map[string]string
+		expected repo.NewRepositoryOptions
+	}{
+		{
+			name: "with hash algo",
+			flags: map[string]string{
+				udmrepo.StoreOptionGenHashAlgo: "fake-hash",
+			},
+			expected: repo.NewRepositoryOptions{
+				BlockFormat: format.ContentFormat{
+					Hash:       "fake-hash",
+					Encryption: encryption.DefaultAlgorithm,
+				},
+				ObjectFormat: format.ObjectFormat{
+					Splitter: splitter.DefaultAlgorithm,
+				},
+			},
+		},
+		{
+			name: "with encrypt algo",
+			flags: map[string]string{
+				udmrepo.StoreOptionGenEncryptAlgo: "fake-encrypt",
+			},
+			expected: repo.NewRepositoryOptions{
+				BlockFormat: format.ContentFormat{
+					Hash:       hashing.DefaultAlgorithm,
+					Encryption: "fake-encrypt",
+				},
+				ObjectFormat: format.ObjectFormat{
+					Splitter: splitter.DefaultAlgorithm,
+				},
+			},
+		},
+		{
+			name: "with splitter algo",
+			flags: map[string]string{
+				udmrepo.StoreOptionGenSplitAlgo: "fake-splitter",
+			},
+			expected: repo.NewRepositoryOptions{
+				BlockFormat: format.ContentFormat{
+					Hash:       hashing.DefaultAlgorithm,
+					Encryption: encryption.DefaultAlgorithm,
+				},
+				ObjectFormat: format.ObjectFormat{
+					Splitter: "fake-splitter",
+				},
+			},
+		},
+		{
+			name: "with retention algo",
+			flags: map[string]string{
+				udmrepo.StoreOptionGenRetentionMode: "fake-retention-mode",
+			},
+			expected: repo.NewRepositoryOptions{
+				BlockFormat: format.ContentFormat{
+					Hash:       hashing.DefaultAlgorithm,
+					Encryption: encryption.DefaultAlgorithm,
+				},
+				ObjectFormat: format.ObjectFormat{
+					Splitter: splitter.DefaultAlgorithm,
+				},
+				RetentionMode: "fake-retention-mode",
+			},
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			ret := SetupNewRepositoryOptions(context.Background(), tc.flags)
+			assert.Equal(t, tc.expected, ret)
+		})
+	}
+}
+
+func TestSetupConnectOptions(t *testing.T) {
+	defaultCacheOption := content.CachingOptions{
+		MaxCacheSizeBytes:         2000 << 20,
+		MaxMetadataCacheSizeBytes: 2000 << 20,
+		MaxListCacheDuration:      content.DurationSeconds(time.Duration(30) * time.Second),
+	}
+
+	testCases := []struct {
+		name        string
+		repoOptions udmrepo.RepoOptions
+		expected    repo.ConnectOptions
+	}{
+		{
+			name: "with domain",
+			repoOptions: udmrepo.RepoOptions{
+				GeneralOptions: map[string]string{
+					udmrepo.GenOptionOwnerDomain: "fake-domain",
+				},
+			},
+			expected: repo.ConnectOptions{
+				CachingOptions: defaultCacheOption,
+				ClientOptions: repo.ClientOptions{
+					Hostname: "fake-domain",
+				},
+			},
+		},
+		{
+			name: "with username",
+			repoOptions: udmrepo.RepoOptions{
+				GeneralOptions: map[string]string{
+					udmrepo.GenOptionOwnerName: "fake-user",
+				},
+			},
+			expected: repo.ConnectOptions{
+				CachingOptions: defaultCacheOption,
+				ClientOptions: repo.ClientOptions{
+					Username: "fake-user",
+				},
+			},
+		},
+		{
+			name: "with wrong readonly",
+			repoOptions: udmrepo.RepoOptions{
+				GeneralOptions: map[string]string{
+					udmrepo.StoreOptionGenReadOnly: "fake-bool",
+				},
+			},
+			expected: repo.ConnectOptions{
+				CachingOptions: defaultCacheOption,
+				ClientOptions:  repo.ClientOptions{},
+			},
+		},
+		{
+			name: "with correct readonly",
+			repoOptions: udmrepo.RepoOptions{
+				GeneralOptions: map[string]string{
+					udmrepo.StoreOptionGenReadOnly: "true",
+				},
+			},
+			expected: repo.ConnectOptions{
+				CachingOptions: defaultCacheOption,
+				ClientOptions: repo.ClientOptions{
+					ReadOnly: true,
+				},
+			},
+		},
+		{
+			name: "with description",
+			repoOptions: udmrepo.RepoOptions{
+				Description: "fake-description",
+			},
+			expected: repo.ConnectOptions{
+				CachingOptions: defaultCacheOption,
+				ClientOptions: repo.ClientOptions{
+					Description: "fake-description",
+				},
+			},
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			ret := SetupConnectOptions(context.Background(), tc.repoOptions)
+			assert.Equal(t, tc.expected, ret)
+		})
+	}
+}

pkg/repository/udmrepo/kopialib/backend/file_system_test.go

@@ -0,0 +1,80 @@
+/*
+Copyright the Velero contributors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package backend
+
+import (
+	"context"
+	"testing"
+
+	"github.com/kopia/kopia/repo/blob/filesystem"
+	"github.com/stretchr/testify/assert"
+
+	"github.com/vmware-tanzu/velero/pkg/repository/udmrepo"
+)
+
+func TestFSSetup(t *testing.T) {
+	testCases := []struct {
+		name            string
+		flags           map[string]string
+		expectedOptions filesystem.Options
+		expectedErr     string
+	}{
+		{
+			name:        "must have fs path",
+			flags:       map[string]string{},
+			expectedErr: "key " + udmrepo.StoreOptionFsPath + " not found",
+		},
+		{
+			name: "with fs path only",
+			flags: map[string]string{
+				udmrepo.StoreOptionFsPath: "fake/path",
+			},
+			expectedOptions: filesystem.Options{
+				Path:          "fake/path",
+				FileMode:      0o600,
+				DirectoryMode: 0o700,
+			},
+		},
+		{
+			name: "with prefix",
+			flags: map[string]string{
+				udmrepo.StoreOptionFsPath: "fake/path",
+				udmrepo.StoreOptionPrefix: "fake-prefix",
+			},
+			expectedOptions: filesystem.Options{
+				Path:          "fake/path/fake-prefix",
+				FileMode:      0o600,
+				DirectoryMode: 0o700,
+			},
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			fsFlags := FsBackend{}
+
+			err := fsFlags.Setup(context.Background(), tc.flags)
+
+			if tc.expectedErr == "" {
+				assert.NoError(t, err)
+				assert.Equal(t, tc.expectedOptions, fsFlags.options)
+			} else {
+				assert.EqualError(t, err, tc.expectedErr)
+			}
+		})
+	}
+}

pkg/repository/udmrepo/kopialib/backend/gcs_test.go

@@ -20,6 +20,7 @@ import (
 	"context"
 	"testing"
 
+	"github.com/kopia/kopia/repo/blob/gcs"
 	"github.com/stretchr/testify/assert"
 
 	"github.com/vmware-tanzu/velero/pkg/repository/udmrepo"
@@ -27,9 +28,10 @@ import (
 
 func TestGcsSetup(t *testing.T) {
 	testCases := []struct {
-		name        string
-		flags       map[string]string
-		expectedErr string
+		name            string
+		flags           map[string]string
+		expectedOptions gcs.Options
+		expectedErr     string
 	}{
 		{
 			name:        "must have bucket name",
@@ -43,6 +45,44 @@ func TestGcsSetup(t *testing.T) {
 			},
 			expectedErr: "key " + udmrepo.StoreOptionCredentialFile + " not found",
 		},
+		{
+			name: "with prefix",
+			flags: map[string]string{
+				udmrepo.StoreOptionOssBucket:      "fake-bucket",
+				udmrepo.StoreOptionCredentialFile: "fake-credential",
+				udmrepo.StoreOptionPrefix:         "fake-prefix",
+			},
+			expectedOptions: gcs.Options{
+				BucketName:                    "fake-bucket",
+				ServiceAccountCredentialsFile: "fake-credential",
+				Prefix:                        "fake-prefix",
+			},
+		},
+		{
+			name: "with wrong readonly",
+			flags: map[string]string{
+				udmrepo.StoreOptionOssBucket:      "fake-bucket",
+				udmrepo.StoreOptionCredentialFile: "fake-credential",
+				udmrepo.StoreOptionGcsReadonly:    "fake-bool",
+			},
+			expectedOptions: gcs.Options{
+				BucketName:                    "fake-bucket",
+				ServiceAccountCredentialsFile: "fake-credential",
+			},
+		},
+		{
+			name: "with correct readonly",
+			flags: map[string]string{
+				udmrepo.StoreOptionOssBucket:      "fake-bucket",
+				udmrepo.StoreOptionCredentialFile: "fake-credential",
+				udmrepo.StoreOptionGcsReadonly:    "true",
+			},
+			expectedOptions: gcs.Options{
+				BucketName:                    "fake-bucket",
+				ServiceAccountCredentialsFile: "fake-credential",
+				ReadOnly:                      true,
+			},
+		},
 	}
 
 	for _, tc := range testCases {
@@ -53,6 +93,7 @@ func TestGcsSetup(t *testing.T) {
 
 			if tc.expectedErr == "" {
 				assert.NoError(t, err)
+				assert.Equal(t, tc.expectedOptions, gcsFlags.options)
 			} else {
 				assert.EqualError(t, err, tc.expectedErr)
 			}

pkg/repository/udmrepo/kopialib/backend/mocks/Reader.go

@@ -0,0 +1,101 @@
+// Code generated by mockery v2.22.1. DO NOT EDIT.
+
+package mocks
+
+import mock "github.com/stretchr/testify/mock"
+
+// Reader is an autogenerated mock type for the Reader type
+type Reader struct {
+	mock.Mock
+}
+
+// Close provides a mock function with given fields:
+func (_m *Reader) Close() error {
+	ret := _m.Called()
+
+	var r0 error
+	if rf, ok := ret.Get(0).(func() error); ok {
+		r0 = rf()
+	} else {
+		r0 = ret.Error(0)
+	}
+
+	return r0
+}
+
+// Length provides a mock function with given fields:
+func (_m *Reader) Length() int64 {
+	ret := _m.Called()
+
+	var r0 int64
+	if rf, ok := ret.Get(0).(func() int64); ok {
+		r0 = rf()
+	} else {
+		r0 = ret.Get(0).(int64)
+	}
+
+	return r0
+}
+
+// Read provides a mock function with given fields: p
+func (_m *Reader) Read(p []byte) (int, error) {
+	ret := _m.Called(p)
+
+	var r0 int
+	var r1 error
+	if rf, ok := ret.Get(0).(func([]byte) (int, error)); ok {
+		return rf(p)
+	}
+	if rf, ok := ret.Get(0).(func([]byte) int); ok {
+		r0 = rf(p)
+	} else {
+		r0 = ret.Get(0).(int)
+	}
+
+	if rf, ok := ret.Get(1).(func([]byte) error); ok {
+		r1 = rf(p)
+	} else {
+		r1 = ret.Error(1)
+	}
+
+	return r0, r1
+}
+
+// Seek provides a mock function with given fields: offset, whence
+func (_m *Reader) Seek(offset int64, whence int) (int64, error) {
+	ret := _m.Called(offset, whence)
+
+	var r0 int64
+	var r1 error
+	if rf, ok := ret.Get(0).(func(int64, int) (int64, error)); ok {
+		return rf(offset, whence)
+	}
+	if rf, ok := ret.Get(0).(func(int64, int) int64); ok {
+		r0 = rf(offset, whence)
+	} else {
+		r0 = ret.Get(0).(int64)
+	}
+
+	if rf, ok := ret.Get(1).(func(int64, int) error); ok {
+		r1 = rf(offset, whence)
+	} else {
+		r1 = ret.Error(1)
+	}
+
+	return r0, r1
+}
+
+type mockConstructorTestingTNewReader interface {
+	mock.TestingT
+	Cleanup(func())
+}
+
+// NewReader creates a new instance of Reader. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
+func NewReader(t mockConstructorTestingTNewReader) *Reader {
+	mock := &Reader{}
+	mock.Mock.Test(t)
+
+	t.Cleanup(func() { mock.AssertExpectations(t) })
+
+	return mock
+}

pkg/repository/udmrepo/kopialib/backend/mocks/Writer.go

@@ -0,0 +1,114 @@
+// Code generated by mockery v2.22.1. DO NOT EDIT.
+
+package mocks
+
+import (
+	object "github.com/kopia/kopia/repo/object"
+	mock "github.com/stretchr/testify/mock"
+)
+
+// Writer is an autogenerated mock type for the Writer type
+type Writer struct {
+	mock.Mock
+}
+
+// Checkpoint provides a mock function with given fields:
+func (_m *Writer) Checkpoint() (object.ID, error) {
+	ret := _m.Called()
+
+	var r0 object.ID
+	var r1 error
+	if rf, ok := ret.Get(0).(func() (object.ID, error)); ok {
+		return rf()
+	}
+	if rf, ok := ret.Get(0).(func() object.ID); ok {
+		r0 = rf()
+	} else {
+		r0 = ret.Get(0).(object.ID)
+	}
+
+	if rf, ok := ret.Get(1).(func() error); ok {
+		r1 = rf()
+	} else {
+		r1 = ret.Error(1)
+	}
+
+	return r0, r1
+}
+
+// Close provides a mock function with given fields:
+func (_m *Writer) Close() error {
+	ret := _m.Called()
+
+	var r0 error
+	if rf, ok := ret.Get(0).(func() error); ok {
+		r0 = rf()
+	} else {
+		r0 = ret.Error(0)
+	}
+
+	return r0
+}
+
+// Result provides a mock function with given fields:
+func (_m *Writer) Result() (object.ID, error) {
+	ret := _m.Called()
+
+	var r0 object.ID
+	var r1 error
+	if rf, ok := ret.Get(0).(func() (object.ID, error)); ok {
+		return rf()
+	}
+	if rf, ok := ret.Get(0).(func() object.ID); ok {
+		r0 = rf()
+	} else {
+		r0 = ret.Get(0).(object.ID)
+	}
+
+	if rf, ok := ret.Get(1).(func() error); ok {
+		r1 = rf()
+	} else {
+		r1 = ret.Error(1)
+	}
+
+	return r0, r1
+}
+
+// Write provides a mock function with given fields: p
+func (_m *Writer) Write(p []byte) (int, error) {
+	ret := _m.Called(p)
+
+	var r0 int
+	var r1 error
+	if rf, ok := ret.Get(0).(func([]byte) (int, error)); ok {
+		return rf(p)
+	}
+	if rf, ok := ret.Get(0).(func([]byte) int); ok {
+		r0 = rf(p)
+	} else {
+		r0 = ret.Get(0).(int)
+	}
+
+	if rf, ok := ret.Get(1).(func([]byte) error); ok {
+		r1 = rf(p)
+	} else {
+		r1 = ret.Error(1)
+	}
+
+	return r0, r1
+}
+
+type mockConstructorTestingTNewWriter interface {
+	mock.TestingT
+	Cleanup(func())
+}
+
+// NewWriter creates a new instance of Writer. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
+func NewWriter(t mockConstructorTestingTNewWriter) *Writer {
+	mock := &Writer{}
+	mock.Mock.Test(t)
+
+	t.Cleanup(func() { mock.AssertExpectations(t) })
+
+	return mock
+}

pkg/repository/udmrepo/kopialib/backend/s3_test.go

@@ -20,6 +20,7 @@ import (
 	"context"
 	"testing"
 
+	"github.com/kopia/kopia/repo/blob/s3"
 	"github.com/stretchr/testify/assert"
 
 	"github.com/vmware-tanzu/velero/pkg/repository/udmrepo"
@@ -27,15 +28,91 @@ import (
 
 func TestS3Setup(t *testing.T) {
 	testCases := []struct {
-		name        string
-		flags       map[string]string
-		expectedErr string
+		name            string
+		flags           map[string]string
+		expectedOptions s3.Options
+		expectedErr     string
 	}{
 		{
 			name:        "must have bucket name",
 			flags:       map[string]string{},
 			expectedErr: "key " + udmrepo.StoreOptionOssBucket + " not found",
 		},
+		{
+			name: "with bucket only",
+			flags: map[string]string{
+				udmrepo.StoreOptionOssBucket: "fake-bucket",
+			},
+			expectedOptions: s3.Options{
+				BucketName: "fake-bucket",
+			},
+		},
+		{
+			name: "with others",
+			flags: map[string]string{
+				udmrepo.StoreOptionOssBucket:   "fake-bucket",
+				udmrepo.StoreOptionS3KeyID:     "fake-ak",
+				udmrepo.StoreOptionS3SecretKey: "fake-sk",
+				udmrepo.StoreOptionS3Endpoint:  "fake-endpoint",
+				udmrepo.StoreOptionOssRegion:   "fake-region",
+				udmrepo.StoreOptionPrefix:      "fake-prefix",
+				udmrepo.StoreOptionS3Token:     "fake-token",
+			},
+			expectedOptions: s3.Options{
+				BucketName:      "fake-bucket",
+				AccessKeyID:     "fake-ak",
+				SecretAccessKey: "fake-sk",
+				Endpoint:        "fake-endpoint",
+				Region:          "fake-region",
+				Prefix:          "fake-prefix",
+				SessionToken:    "fake-token",
+			},
+		},
+		{
+			name: "with wrong tls",
+			flags: map[string]string{
+				udmrepo.StoreOptionOssBucket:          "fake-bucket",
+				udmrepo.StoreOptionS3DisableTLS:       "fake-bool",
+				udmrepo.StoreOptionS3DisableTLSVerify: "fake-bool",
+			},
+			expectedOptions: s3.Options{
+				BucketName: "fake-bucket",
+			},
+		},
+		{
+			name: "with correct tls",
+			flags: map[string]string{
+				udmrepo.StoreOptionOssBucket:          "fake-bucket",
+				udmrepo.StoreOptionS3DisableTLS:       "true",
+				udmrepo.StoreOptionS3DisableTLSVerify: "false",
+			},
+			expectedOptions: s3.Options{
+				BucketName:     "fake-bucket",
+				DoNotUseTLS:    true,
+				DoNotVerifyTLS: false,
+			},
+		},
+		{
+			name: "with wrong ca",
+			flags: map[string]string{
+				udmrepo.StoreOptionOssBucket:  "fake-bucket",
+				udmrepo.StoreOptionS3CustomCA: "fake-base-64",
+			},
+			expectedOptions: s3.Options{
+				BucketName: "fake-bucket",
+			},
+		},
+		{
+			name: "with correct ca",
+			flags: map[string]string{
+				udmrepo.StoreOptionOssBucket:  "fake-bucket",
+				udmrepo.StoreOptionS3CustomCA: "ZmFrZS1jYQ==",
+			},
+			expectedOptions: s3.Options{
+				BucketName: "fake-bucket",
+				RootCA:     []byte{'f', 'a', 'k', 'e', '-', 'c', 'a'},
+			},
+		},
 	}
 
 	for _, tc := range testCases {

pkg/repository/udmrepo/kopialib/lib_repo_test.go

@@ -18,12 +18,14 @@ package kopialib
 
 import (
 	"context"
+	"math"
 	"os"
 	"testing"
 	"time"
 
 	"github.com/kopia/kopia/repo"
 	"github.com/kopia/kopia/repo/manifest"
+	"github.com/kopia/kopia/repo/object"
 	"github.com/pkg/errors"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/mock"
@@ -713,3 +715,486 @@ func TestFlush(t *testing.T) {
 		})
 	}
 }
+
+func TestNewObjectWriter(t *testing.T) {
+	rawObjWriter := repomocks.NewWriter(t)
+	testCases := []struct {
+		name         string
+		rawWriter    *repomocks.DirectRepositoryWriter
+		rawWriterRet object.Writer
+		expectedRet  udmrepo.ObjectWriter
+	}{
+		{
+			name: "raw writer is nil",
+		},
+		{
+			name:      "new object writer fail",
+			rawWriter: repomocks.NewDirectRepositoryWriter(t),
+		},
+		{
+			name:         "succeed",
+			rawWriter:    repomocks.NewDirectRepositoryWriter(t),
+			rawWriterRet: rawObjWriter,
+			expectedRet:  &kopiaObjectWriter{rawWriter: rawObjWriter},
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			kr := &kopiaRepository{}
+
+			if tc.rawWriter != nil {
+				tc.rawWriter.On("NewObjectWriter", mock.Anything, mock.Anything).Return(tc.rawWriterRet)
+				kr.rawWriter = tc.rawWriter
+			}
+
+			ret := kr.NewObjectWriter(context.Background(), udmrepo.ObjectWriteOptions{})
+
+			assert.Equal(t, tc.expectedRet, ret)
+		})
+	}
+}
+
+func TestUpdateProgress(t *testing.T) {
+	testCases := []struct {
+		name       string
+		progress   int64
+		uploaded   int64
+		throttle   logThrottle
+		logMessage string
+	}{
+		{
+			name: "should not output",
+			throttle: logThrottle{
+				lastTime: math.MaxInt64,
+			},
+		},
+		{
+			name:       "should output",
+			progress:   100,
+			uploaded:   200,
+			logMessage: "Repo uploaded 300 bytes.",
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			logMessage := ""
+			kr := &kopiaRepository{
+				logger:   velerotest.NewSingleLogger(&logMessage),
+				throttle: tc.throttle,
+				uploaded: tc.uploaded,
+			}
+
+			kr.updateProgress(tc.progress)
+
+			if len(tc.logMessage) > 0 {
+				assert.Contains(t, logMessage, tc.logMessage)
+			} else {
+				assert.Equal(t, "", logMessage)
+			}
+		})
+	}
+}
+
+func TestReaderRead(t *testing.T) {
+	testCases := []struct {
+		name            string
+		rawObjReader    *repomocks.Reader
+		rawReaderRetErr error
+		expectedErr     string
+	}{
+		{
+			name:        "raw reader is nil",
+			expectedErr: "object reader is closed or not open",
+		},
+		{
+			name:            "raw read fail",
+			rawObjReader:    repomocks.NewReader(t),
+			rawReaderRetErr: errors.New("fake-read-error"),
+			expectedErr:     "fake-read-error",
+		},
+		{
+			name:         "succeed",
+			rawObjReader: repomocks.NewReader(t),
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			kr := &kopiaObjectReader{}
+
+			if tc.rawObjReader != nil {
+				tc.rawObjReader.On("Read", mock.Anything).Return(0, tc.rawReaderRetErr)
+				kr.rawReader = tc.rawObjReader
+			}
+
+			_, err := kr.Read(nil)
+
+			if tc.expectedErr == "" {
+				assert.NoError(t, err)
+			} else {
+				assert.EqualError(t, err, tc.expectedErr)
+			}
+		})
+	}
+}
+
+func TestReaderSeek(t *testing.T) {
+	testCases := []struct {
+		name            string
+		rawObjReader    *repomocks.Reader
+		rawReaderRet    int64
+		rawReaderRetErr error
+		expectedRet     int64
+		expectedErr     string
+	}{
+		{
+			name:        "raw reader is nil",
+			expectedErr: "object reader is closed or not open",
+		},
+		{
+			name:            "raw seek fail",
+			rawObjReader:    repomocks.NewReader(t),
+			rawReaderRetErr: errors.New("fake-seek-error"),
+			expectedErr:     "fake-seek-error",
+		},
+		{
+			name:         "succeed",
+			rawObjReader: repomocks.NewReader(t),
+			rawReaderRet: 100,
+			expectedRet:  100,
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			kr := &kopiaObjectReader{}
+
+			if tc.rawObjReader != nil {
+				tc.rawObjReader.On("Seek", mock.Anything, mock.Anything).Return(tc.rawReaderRet, tc.rawReaderRetErr)
+				kr.rawReader = tc.rawObjReader
+			}
+
+			ret, err := kr.Seek(0, 0)
+
+			if tc.expectedErr == "" {
+				assert.NoError(t, err)
+				assert.Equal(t, tc.expectedRet, ret)
+			} else {
+				assert.EqualError(t, err, tc.expectedErr)
+			}
+		})
+	}
+}
+
+func TestReaderClose(t *testing.T) {
+	testCases := []struct {
+		name            string
+		rawObjReader    *repomocks.Reader
+		rawReaderRetErr error
+		expectedErr     string
+	}{
+		{
+			name: "raw reader is nil",
+		},
+		{
+			name:            "raw close fail",
+			rawObjReader:    repomocks.NewReader(t),
+			rawReaderRetErr: errors.New("fake-close-error"),
+			expectedErr:     "fake-close-error",
+		},
+		{
+			name:         "succeed",
+			rawObjReader: repomocks.NewReader(t),
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			kr := &kopiaObjectReader{}
+
+			if tc.rawObjReader != nil {
+				tc.rawObjReader.On("Close").Return(tc.rawReaderRetErr)
+				kr.rawReader = tc.rawObjReader
+			}
+
+			err := kr.Close()
+
+			if tc.expectedErr == "" {
+				assert.NoError(t, err)
+			} else {
+				assert.EqualError(t, err, tc.expectedErr)
+			}
+		})
+	}
+}
+
+func TestReaderLength(t *testing.T) {
+	testCases := []struct {
+		name         string
+		rawObjReader *repomocks.Reader
+		rawReaderRet int64
+		expectedRet  int64
+	}{
+		{
+			name:        "raw reader is nil",
+			expectedRet: -1,
+		},
+		{
+			name:         "raw length fail",
+			rawObjReader: repomocks.NewReader(t),
+			rawReaderRet: 0,
+			expectedRet:  0,
+		},
+		{
+			name:         "succeed",
+			rawObjReader: repomocks.NewReader(t),
+			rawReaderRet: 200,
+			expectedRet:  200,
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			kr := &kopiaObjectReader{}
+
+			if tc.rawObjReader != nil {
+				tc.rawObjReader.On("Length").Return(tc.rawReaderRet)
+				kr.rawReader = tc.rawObjReader
+			}
+
+			ret := kr.Length()
+
+			assert.Equal(t, tc.expectedRet, ret)
+		})
+	}
+}
+
+func TestWriterWrite(t *testing.T) {
+	testCases := []struct {
+		name            string
+		rawObjWriter    *repomocks.Writer
+		rawWrtierRet    int
+		rawWriterRetErr error
+		expectedRet     int
+		expectedErr     string
+	}{
+		{
+			name:        "raw writer is nil",
+			expectedErr: "object writer is closed or not open",
+		},
+		{
+			name:            "raw read fail",
+			rawObjWriter:    repomocks.NewWriter(t),
+			rawWriterRetErr: errors.New("fake-write-error"),
+			expectedErr:     "fake-write-error",
+		},
+		{
+			name:         "succeed",
+			rawObjWriter: repomocks.NewWriter(t),
+			rawWrtierRet: 200,
+			expectedRet:  200,
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			kr := &kopiaObjectWriter{}
+
+			if tc.rawObjWriter != nil {
+				tc.rawObjWriter.On("Write", mock.Anything).Return(tc.rawWrtierRet, tc.rawWriterRetErr)
+				kr.rawWriter = tc.rawObjWriter
+			}
+
+			ret, err := kr.Write(nil)
+
+			if tc.expectedErr == "" {
+				assert.NoError(t, err)
+				assert.Equal(t, tc.expectedRet, ret)
+			} else {
+				assert.EqualError(t, err, tc.expectedErr)
+			}
+		})
+	}
+}
+
+func TestWriterCheckpoint(t *testing.T) {
+	testCases := []struct {
+		name            string
+		rawObjWriter    *repomocks.Writer
+		rawWrtierRet    object.ID
+		rawWriterRetErr error
+		expectedRet     udmrepo.ID
+		expectedErr     string
+	}{
+		{
+			name:        "raw writer is nil",
+			expectedErr: "object writer is closed or not open",
+		},
+		{
+			name:            "raw checkpoint fail",
+			rawObjWriter:    repomocks.NewWriter(t),
+			rawWriterRetErr: errors.New("fake-checkpoint-error"),
+			expectedErr:     "error to checkpoint object: fake-checkpoint-error",
+		},
+		{
+			name:         "succeed",
+			rawObjWriter: repomocks.NewWriter(t),
+			rawWrtierRet: object.ID{},
+			expectedRet:  udmrepo.ID(""),
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			kr := &kopiaObjectWriter{}
+
+			if tc.rawObjWriter != nil {
+				tc.rawObjWriter.On("Checkpoint").Return(tc.rawWrtierRet, tc.rawWriterRetErr)
+				kr.rawWriter = tc.rawObjWriter
+			}
+
+			ret, err := kr.Checkpoint()
+
+			if tc.expectedErr == "" {
+				assert.NoError(t, err)
+				assert.Equal(t, tc.expectedRet, ret)
+			} else {
+				assert.EqualError(t, err, tc.expectedErr)
+			}
+		})
+	}
+}
+
+func TestWriterResult(t *testing.T) {
+	testCases := []struct {
+		name            string
+		rawObjWriter    *repomocks.Writer
+		rawWrtierRet    object.ID
+		rawWriterRetErr error
+		expectedRet     udmrepo.ID
+		expectedErr     string
+	}{
+		{
+			name:        "raw writer is nil",
+			expectedErr: "object writer is closed or not open",
+		},
+		{
+			name:            "raw result fail",
+			rawObjWriter:    repomocks.NewWriter(t),
+			rawWriterRetErr: errors.New("fake-result-error"),
+			expectedErr:     "error to wait object: fake-result-error",
+		},
+		{
+			name:         "succeed",
+			rawObjWriter: repomocks.NewWriter(t),
+			rawWrtierRet: object.ID{},
+			expectedRet:  udmrepo.ID(""),
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			kr := &kopiaObjectWriter{}
+
+			if tc.rawObjWriter != nil {
+				tc.rawObjWriter.On("Result").Return(tc.rawWrtierRet, tc.rawWriterRetErr)
+				kr.rawWriter = tc.rawObjWriter
+			}
+
+			ret, err := kr.Result()
+
+			if tc.expectedErr == "" {
+				assert.NoError(t, err)
+				assert.Equal(t, tc.expectedRet, ret)
+			} else {
+				assert.EqualError(t, err, tc.expectedErr)
+			}
+		})
+	}
+}
+
+func TestWriterClose(t *testing.T) {
+	testCases := []struct {
+		name            string
+		rawObjWriter    *repomocks.Writer
+		rawWriterRetErr error
+		expectedErr     string
+	}{
+		{
+			name: "raw writer is nil",
+		},
+		{
+			name:            "raw close fail",
+			rawObjWriter:    repomocks.NewWriter(t),
+			rawWriterRetErr: errors.New("fake-close-error"),
+			expectedErr:     "fake-close-error",
+		},
+		{
+			name:         "succeed",
+			rawObjWriter: repomocks.NewWriter(t),
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			kr := &kopiaObjectWriter{}
+
+			if tc.rawObjWriter != nil {
+				tc.rawObjWriter.On("Close").Return(tc.rawWriterRetErr)
+				kr.rawWriter = tc.rawObjWriter
+			}
+
+			err := kr.Close()
+
+			if tc.expectedErr == "" {
+				assert.NoError(t, err)
+			} else {
+				assert.EqualError(t, err, tc.expectedErr)
+			}
+		})
+	}
+}
+
+func TestMaintainProgress(t *testing.T) {
+	testCases := []struct {
+		name       string
+		progress   int64
+		uploaded   int64
+		throttle   logThrottle
+		logMessage string
+	}{
+		{
+			name: "should not output",
+			throttle: logThrottle{
+				lastTime: math.MaxInt64,
+			},
+		},
+		{
+			name:       "should output",
+			progress:   100,
+			uploaded:   200,
+			logMessage: "Repo maintenance uploaded 300 bytes.",
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			logMessage := ""
+			km := &kopiaMaintenance{
+				logger:   velerotest.NewSingleLogger(&logMessage),
+				throttle: tc.throttle,
+				uploaded: tc.uploaded,
+			}
+
+			km.maintainProgress(tc.progress)
+
+			if len(tc.logMessage) > 0 {
+				assert.Contains(t, logMessage, tc.logMessage)
+			} else {
+				assert.Equal(t, "", logMessage)
+			}
+		})
+	}
+}

pkg/restic/exec_commands.go

@@ -86,6 +86,7 @@ func RunBackup(backupCmd *Command, log logrus.FieldLogger, updater uploader.Prog
 
 	err := cmd.Start()
 	if err != nil {
+		exec.LogErrorAsExitCode(err, log)
 		return stdoutBuf.String(), stderrBuf.String(), err
 	}
 
@@ -119,6 +120,7 @@ func RunBackup(backupCmd *Command, log logrus.FieldLogger, updater uploader.Prog
 
 	err = cmd.Wait()
 	if err != nil {
+		exec.LogErrorAsExitCode(err, log)
 		return stdoutBuf.String(), stderrBuf.String(), err
 	}
 	quit <- struct{}{}
@@ -229,7 +231,7 @@ func RunRestore(restoreCmd *Command, log logrus.FieldLogger, updater uploader.Pr
 		}
 	}()
 
-	stdout, stderr, err := exec.RunCommand(restoreCmd.Cmd())
+	stdout, stderr, err := exec.RunCommandWithLog(restoreCmd.Cmd(), log)
 	quit <- struct{}{}
 
 	// update progress to 100%

pkg/restore/dataupload_retrieve_action.go

@@ -0,0 +1,105 @@
+/*
+Copyright 2020 the Velero contributors.
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package restore
+
+import (
+	"context"
+	"encoding/json"
+
+	"github.com/pkg/errors"
+	"github.com/sirupsen/logrus"
+	corev1api "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	corev1client "k8s.io/client-go/kubernetes/typed/core/v1"
+
+	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
+	velerov2alpha1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1"
+	"github.com/vmware-tanzu/velero/pkg/label"
+	"github.com/vmware-tanzu/velero/pkg/plugin/velero"
+)
+
+type DataUploadRetrieveAction struct {
+	logger          logrus.FieldLogger
+	configMapClient corev1client.ConfigMapInterface
+}
+
+func NewDataUploadRetrieveAction(logger logrus.FieldLogger, configMapClient corev1client.ConfigMapInterface) *DataUploadRetrieveAction {
+	return &DataUploadRetrieveAction{
+		logger:          logger,
+		configMapClient: configMapClient,
+	}
+}
+
+func (d *DataUploadRetrieveAction) AppliesTo() (velero.ResourceSelector, error) {
+	return velero.ResourceSelector{
+		IncludedResources: []string{"datauploads.velero.io"},
+	}, nil
+}
+
+func (d *DataUploadRetrieveAction) Execute(input *velero.RestoreItemActionExecuteInput) (*velero.RestoreItemActionExecuteOutput, error) {
+	d.logger.Info("Executing DataUploadRetrieveAction")
+
+	dataUpload := velerov2alpha1.DataUpload{}
+	if err := runtime.DefaultUnstructuredConverter.FromUnstructured(input.ItemFromBackup.UnstructuredContent(), &dataUpload); err != nil {
+		d.logger.Errorf("unable to convert unstructured item to DataUpload: %s", err.Error())
+		return nil, errors.Wrap(err, "unable to convert unstructured item to DataUpload.")
+	}
+
+	dataUploadResult := velerov2alpha1.DataUploadResult{
+		BackupStorageLocation: dataUpload.Spec.BackupStorageLocation,
+		DataMover:             dataUpload.Spec.DataMover,
+		SnapshotID:            dataUpload.Status.SnapshotID,
+		SourceNamespace:       dataUpload.Spec.SourceNamespace,
+		DataMoverResult:       dataUpload.Status.DataMoverResult,
+	}
+
+	jsonBytes, err := json.Marshal(dataUploadResult)
+	if err != nil {
+		d.logger.Errorf("fail to convert DataUploadResult to JSON: %s", err.Error())
+		return nil, errors.Wrap(err, "fail to convert DataUploadResult to JSON")
+	}
+
+	cm := corev1api.ConfigMap{
+		TypeMeta: metav1.TypeMeta{
+			Kind:       "ConfigMap",
+			APIVersion: corev1api.SchemeGroupVersion.String(),
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			GenerateName: dataUpload.Name + "-",
+			Namespace:    dataUpload.Namespace,
+			Labels: map[string]string{
+				velerov1api.RestoreUIDLabel:       label.GetValidName(string(input.Restore.UID)),
+				velerov1api.PVCNamespaceNameLabel: label.GetValidName(dataUpload.Spec.SourceNamespace + "." + dataUpload.Spec.SourcePVC),
+				velerov1api.ResourceUsageLabel:    label.GetValidName(string(velerov1api.VeleroResourceUsageDataUploadResult)),
+			},
+		},
+		Data: map[string]string{
+			string(input.Restore.UID): string(jsonBytes),
+		},
+	}
+
+	_, err = d.configMapClient.Create(context.Background(), &cm, metav1.CreateOptions{})
+	if err != nil {
+		d.logger.Errorf("fail to create DataUploadResult ConfigMap %s/%s: %s", cm.Namespace, cm.Name, err.Error())
+		return nil, errors.Wrap(err, "fail to create DataUploadResult ConfigMap")
+	}
+
+	return &velero.RestoreItemActionExecuteOutput{
+		SkipRestore: true,
+	}, nil
+}

pkg/restore/dataupload_retrieve_action_test.go

@@ -0,0 +1,95 @@
+/*
+Copyright 2020 the Velero contributors.
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package restore
+
+import (
+	"context"
+	"fmt"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/client-go/kubernetes/fake"
+
+	velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
+	velerov2alpha1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1"
+	"github.com/vmware-tanzu/velero/pkg/builder"
+	"github.com/vmware-tanzu/velero/pkg/label"
+	"github.com/vmware-tanzu/velero/pkg/plugin/velero"
+	velerotest "github.com/vmware-tanzu/velero/pkg/test"
+)
+
+func TestDataUploadRetrieveActionExectue(t *testing.T) {
+	tests := []struct {
+		name                     string
+		dataUpload               *velerov2alpha1.DataUpload
+		restore                  *velerov1.Restore
+		expectedDataUploadResult *corev1.ConfigMap
+		expectedErr              string
+	}{
+		{
+			name:                     "DataUploadRetrieve Action test",
+			dataUpload:               builder.ForDataUpload("velero", "testDU").SourceNamespace("testNamespace").SourcePVC("testPVC").Result(),
+			restore:                  builder.ForRestore("velero", "testRestore").ObjectMeta(builder.WithUID("testingUID")).Result(),
+			expectedDataUploadResult: builder.ForConfigMap("velero", "").ObjectMeta(builder.WithGenerateName("testDU-"), builder.WithLabels(velerov1.PVCNamespaceNameLabel, "testNamespace.testPVC", velerov1.RestoreUIDLabel, "testingUID", velerov1.ResourceUsageLabel, string(velerov1.VeleroResourceUsageDataUploadResult))).Data("testingUID", `{"backupStorageLocation":"","sourceNamespace":"testNamespace"}`).Result(),
+		},
+		{
+			name:                     "Long source namespace and PVC name should also work",
+			dataUpload:               builder.ForDataUpload("velero", "testDU").SourceNamespace("migre209d0da-49c7-45ba-8d5a-3e59fd591ec1").SourcePVC("kibishii-data-kibishii-deployment-0").Result(),
+			restore:                  builder.ForRestore("velero", "testRestore").ObjectMeta(builder.WithUID("testingUID")).Result(),
+			expectedDataUploadResult: builder.ForConfigMap("velero", "").ObjectMeta(builder.WithGenerateName("testDU-"), builder.WithLabels(velerov1.PVCNamespaceNameLabel, "migre209d0da-49c7-45ba-8d5a-3e59fd591ec1.kibishii-data-ki152333", velerov1.RestoreUIDLabel, "testingUID", velerov1.ResourceUsageLabel, string(velerov1.VeleroResourceUsageDataUploadResult))).Data("testingUID", `{"backupStorageLocation":"","sourceNamespace":"migre209d0da-49c7-45ba-8d5a-3e59fd591ec1"}`).Result(),
+		},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			logger := velerotest.NewLogger()
+			cmClient := fake.NewSimpleClientset()
+
+			var unstructuredDataUpload map[string]interface{}
+			if tc.dataUpload != nil {
+				var err error
+				unstructuredDataUpload, err = runtime.DefaultUnstructuredConverter.ToUnstructured(tc.dataUpload)
+				require.NoError(t, err)
+			}
+			input := velero.RestoreItemActionExecuteInput{
+				Restore:        tc.restore,
+				ItemFromBackup: &unstructured.Unstructured{Object: unstructuredDataUpload},
+			}
+
+			action := NewDataUploadRetrieveAction(logger, cmClient.CoreV1().ConfigMaps("velero"))
+			_, err := action.Execute(&input)
+			if tc.expectedErr != "" {
+				require.Equal(t, tc.expectedErr, err.Error())
+			}
+			require.NoError(t, err)
+
+			if tc.expectedDataUploadResult != nil {
+				cmList, err := cmClient.CoreV1().ConfigMaps("velero").List(context.Background(), metav1.ListOptions{
+					LabelSelector: fmt.Sprintf("%s=%s,%s=%s", velerov1.RestoreUIDLabel, "testingUID", velerov1.PVCNamespaceNameLabel, label.GetValidName(tc.dataUpload.Spec.SourceNamespace+"."+tc.dataUpload.Spec.SourcePVC)),
+				})
+				require.NoError(t, err)
+				// debug
+				fmt.Printf("CM: %s\n", &cmList.Items[0])
+				require.Equal(t, *tc.expectedDataUploadResult, cmList.Items[0])
+			}
+		})
+	}
+}

pkg/restore/restore.go

@@ -538,6 +538,21 @@ func (ctx *restoreContext) execute() (results.Result, results.Result) {
 	// Close the progress update channel.
 	quit <- struct{}{}
 
+	// Clean the DataUploadResult ConfigMaps
+	defer func() {
+		opts := []crclient.DeleteAllOfOption{
+			crclient.InNamespace(ctx.restore.Namespace),
+			crclient.MatchingLabels{
+				velerov1api.RestoreUIDLabel:    string(ctx.restore.UID),
+				velerov1api.ResourceUsageLabel: string(velerov1api.VeleroResourceUsageDataUploadResult),
+			},
+		}
+		err := ctx.kbClient.DeleteAllOf(go_context.Background(), &v1.ConfigMap{}, opts...)
+		if err != nil {
+			ctx.log.Errorf("Fail to batch delete DataUploadResult ConfigMaps for restore %s: %s", ctx.restore.Name, err.Error())
+		}
+	}()
+
 	// Do a final progress update as stopping the ticker might have left last few
 	// updates from taking place.
 	updated := ctx.restore.DeepCopy()

pkg/restore/restore_test.go

@@ -1397,6 +1397,18 @@ func TestRestoreActionsRunForCorrectItems(t *testing.T) {
 				new(recordResourcesAction).ForNamespace("ns-2").ForResource("pods"):                   nil,
 			},
 		},
+		{
+			name:    "actions run for datauploads resource",
+			restore: defaultRestore().Result(),
+			backup:  defaultBackup().Result(),
+			tarball: test.NewTarWriter(t).
+				AddItems("datauploads.velero.io", builder.ForDataUpload("velero", "du").Result()).
+				Done(),
+			apiResources: []*test.APIResource{test.DataUploads()},
+			actions: map[*recordResourcesAction][]string{
+				new(recordResourcesAction).ForNamespace("velero").ForResource("datauploads.velero.io"): {"velero/du"},
+			},
+		},
 	}
 
 	for _, tc := range tests {

pkg/test/resources.go

@@ -183,3 +183,13 @@ func Services(items ...metav1.Object) *APIResource {
 		Items:      items,
 	}
 }
+
+func DataUploads(items ...metav1.Object) *APIResource {
+	return &APIResource{
+		Group:      "velero.io",
+		Version:    "v2alpha1",
+		Name:       "datauploads",
+		Namespaced: true,
+		Items:      items,
+	}
+}

pkg/util/exec/exec.go

@@ -22,6 +22,7 @@ import (
 	"os/exec"
 
 	"github.com/pkg/errors"
+	"github.com/sirupsen/logrus"
 )
 
 // RunCommand runs a command and returns its stdout, stderr, and its returned
@@ -52,3 +53,25 @@ func RunCommand(cmd *exec.Cmd) (string, string, error) {
 
 	return stdout, stderr, runErr
 }
+
+func RunCommandWithLog(cmd *exec.Cmd, log logrus.FieldLogger) (string, string, error) {
+	stdout, stderr, err := RunCommand(cmd)
+	LogErrorAsExitCode(err, log)
+	return stdout, stderr, err
+}
+
+func LogErrorAsExitCode(err error, log logrus.FieldLogger) {
+	if err != nil {
+		if exitError, ok := err.(*exec.ExitError); ok {
+			log.Errorf("Restic command fail with ExitCode: %d. Process ID is %d, Exit error is: %s", exitError.ExitCode(), exitError.Pid(), exitError.String())
+			// Golang's os.exec -1 ExitCode means signal kill. Usually this is caused
+			// by CGroup's OOM. Log a warning to notice user.
+			// https://github.com/golang/go/blob/master/src/os/exec_posix.go#L128-L136
+			if exitError.ExitCode() == -1 {
+				log.Warnf("The ExitCode is -1, which means the process is terminated by signal. Usually this is caused by CGroup kill due to out of memory. Please check whether there is such information in the work nodes' dmesg log.")
+			}
+		} else {
+			log.WithError(err).Info("Error cannot be convert to ExitError format.")
+		}
+	}
+}

site/Dockerfile

@@ -1,9 +1,5 @@
-FROM ubuntu:20.04
-
-RUN apt update
-RUN apt install -y hugo
+FROM klakegg/hugo:0.73.0-ext-ubuntu
 
 WORKDIR /srv/hugo
 
 EXPOSE 1313
-

site/content/_index.md

@@ -10,7 +10,7 @@ hero:
   content: Velero is an open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.
   cta_link1:
     text: Latest Release Information
-    url: /blog/velero-1.6-bring-all-your-credentials/
+    url: /blog/Velero-1.11/
   cta_link2:
     text:  Download Velero
     url: https://github.com/vmware-tanzu/velero/releases/latest

site/content/docs/main/file-system-backup.md

@@ -74,7 +74,7 @@ Integrated Edition (formerly VMware Enterprise PKS), or Microsoft Azure.
 **RancherOS**
 
 
-Update the host path for volumes in the nonde-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to 
+Update the host path for volumes in the node-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to 
 `/opt/rke/var/lib/kubelet/pods`.  
 
 ```yaml
@@ -539,7 +539,7 @@ that it's backing up for the volumes to be backed up using FSB.
 5. Meanwhile, each `PodVolumeBackup` is handled by the controller on the appropriate node, which:
     - has a hostPath volume mount of `/var/lib/kubelet/pods` to access the pod volume data
     - finds the pod volume's subdirectory within the above volume
-    - based on the path selection, Velero inokes restic or kopia for backup
+    - based on the path selection, Velero invokes restic or kopia for backup
     - updates the status of the custom resource to `Completed` or `Failed`
 6. As each `PodVolumeBackup` finishes, the main Velero process adds it to the Velero backup in a file named 
 `<backup-name>-podvolumebackups.json.gz`. This file gets uploaded to object storage alongside the backup tarball. 
@@ -564,7 +564,7 @@ some reason (i.e. lack of cluster resources), the FSB restore will not be done.
     - has a hostPath volume mount of `/var/lib/kubelet/pods` to access the pod volume data
     - waits for the pod to be running the init container
     - finds the pod volume's subdirectory within the above volume
-    - based on the path selection, Velero inokes restic or kopia for restore
+    - based on the path selection, Velero invokes restic or kopia for restore
     - on success, writes a file into the pod volume, in a `.velero` subdirectory, whose name is the UID of the Velero 
     restore that this pod volume restore is for
     - updates the status of the custom resource to `Completed` or `Failed`

site/content/docs/main/release-instructions.md

@@ -159,7 +159,7 @@ What to include in a release blog:
 
 Release blog post PR:
 * Prepare a PR containing the release blog post. Read the [website guidelines][2] for more information on creating a blog post. It's usually easiest to make a copy of the most recent existing post, then replace the content as appropriate.
-* You also need to update `site/index.html` to have "Latest Release Information" contain a link to the new post.
+* You also need to update `site/content/_index.md` to have "Latest Release Information" contain a link to the new post.
 * Plan to publish the blog post the same day as the release.
 
 ## Announce a release

site/content/docs/v1.10/file-system-backup.md

@@ -74,7 +74,7 @@ Integrated Edition (formerly VMware Enterprise PKS), or Microsoft Azure.
 **RancherOS**
 
 
-Update the host path for volumes in the nonde-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to 
+Update the host path for volumes in the node-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to 
 `/opt/rke/var/lib/kubelet/pods`.  
 
 ```yaml
@@ -539,7 +539,7 @@ that it's backing up for the volumes to be backed up using FSB.
 5. Meanwhile, each `PodVolumeBackup` is handled by the controller on the appropriate node, which:
     - has a hostPath volume mount of `/var/lib/kubelet/pods` to access the pod volume data
     - finds the pod volume's subdirectory within the above volume
-    - based on the path selection, Velero inokes restic or kopia for backup
+    - based on the path selection, Velero invokes restic or kopia for backup
     - updates the status of the custom resource to `Completed` or `Failed`
 6. As each `PodVolumeBackup` finishes, the main Velero process adds it to the Velero backup in a file named 
 `<backup-name>-podvolumebackups.json.gz`. This file gets uploaded to object storage alongside the backup tarball. 
@@ -564,7 +564,7 @@ some reason (i.e. lack of cluster resources), the FSB restore will not be done.
     - has a hostPath volume mount of `/var/lib/kubelet/pods` to access the pod volume data
     - waits for the pod to be running the init container
     - finds the pod volume's subdirectory within the above volume
-    - based on the path selection, Velero inokes restic or kopia for restore
+    - based on the path selection, Velero invokes restic or kopia for restore
     - on success, writes a file into the pod volume, in a `.velero` subdirectory, whose name is the UID of the Velero 
     restore that this pod volume restore is for
     - updates the status of the custom resource to `Completed` or `Failed`

site/content/docs/v1.11/file-system-backup.md

@@ -74,7 +74,7 @@ Integrated Edition (formerly VMware Enterprise PKS), or Microsoft Azure.
 **RancherOS**
 
 
-Update the host path for volumes in the nonde-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to 
+Update the host path for volumes in the node-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to 
 `/opt/rke/var/lib/kubelet/pods`.  
 
 ```yaml
@@ -539,7 +539,7 @@ that it's backing up for the volumes to be backed up using FSB.
 5. Meanwhile, each `PodVolumeBackup` is handled by the controller on the appropriate node, which:
     - has a hostPath volume mount of `/var/lib/kubelet/pods` to access the pod volume data
     - finds the pod volume's subdirectory within the above volume
-    - based on the path selection, Velero inokes restic or kopia for backup
+    - based on the path selection, Velero invokes restic or kopia for backup
     - updates the status of the custom resource to `Completed` or `Failed`
 6. As each `PodVolumeBackup` finishes, the main Velero process adds it to the Velero backup in a file named 
 `<backup-name>-podvolumebackups.json.gz`. This file gets uploaded to object storage alongside the backup tarball. 
@@ -564,7 +564,7 @@ some reason (i.e. lack of cluster resources), the FSB restore will not be done.
     - has a hostPath volume mount of `/var/lib/kubelet/pods` to access the pod volume data
     - waits for the pod to be running the init container
     - finds the pod volume's subdirectory within the above volume
-    - based on the path selection, Velero inokes restic or kopia for restore
+    - based on the path selection, Velero invokes restic or kopia for restore
     - on success, writes a file into the pod volume, in a `.velero` subdirectory, whose name is the UID of the Velero 
     restore that this pod volume restore is for
     - updates the status of the custom resource to `Completed` or `Failed`