Merge branch 'main' into data-mover-ms-new-exposer

2026-07-20 06:52:40 +00:00 · 2024-07-16 15:54:29 +08:00
parent 49097744ee 2b018272e6
commit 7f88d631a9
83 changed files with 1247 additions and 514 deletions
@@ -13,7 +13,6 @@ reviewers:
      - reasonerjt
      - ywk253100
      - blackpiglet
-      - qiuming-best
      - shubham-pampattiwar
      - Lyndon-Li
      - anshulahuja98
@@ -107,6 +107,29 @@ Lazy consensus does _not_ apply to the process of:

 * Removal of maintainers from Velero

+## Deprecation Policy
+
+### Deprecation Process
+
+Any contributor may introduce a request to deprecate a feature or an option of a feature by opening a feature request issue in the vmware-tanzu/velero GitHub project. The issue should describe why the feature is no longer needed or has become detrimental to Velero, as well as whether and how it has been superseded. The submitter should give as much detail as possible.
+
+Once the issue is filed, a one-month discussion period begins. Discussions take place within the issue itself as well as in the community meetings. The person who opens the issue, or a maintainer, should add the date and time marking the end of the discussion period in a comment on the issue as soon as possible after it is opened. A decision on the issue needs to be made within this one-month period.
+
+The feature will be deprecated by a supermajority vote of 50% plus one of the project maintainers at the time of the vote tallying, which is 72 hours after the end of the community meeting that is the end of the comment period. (Maintainers are permitted to vote in advance of the deadline, but should hold their votes until as close as possible to hear all possible discussion.) Votes will be tallied in comments on the issue. 
+
+Non-maintainers may add non-binding votes in comments to the issue as well; these are opinions to be taken into consideration by maintainers, but they do not count as votes. 
+
+If the vote passes, the deprecation window takes effect in the subsequent release, and the removal follows the schedule. 
+
+### Schedule
+If depreciation proposal passes by supermajority votes, the feature is deprecated in the next minor release and the feature can be removed completely after two minor version or equivalent major version e.g., if feature gets deprecated in Nth minor version, then feature can be removed after N+2 minor version or its equivalent if the major version number changes.
+
+### Deprecation Window
+
+The deprecation window is the period from the release in which the deprecation takes effect through the release in which the feature is removed. During this period, only critical security vulnerabilities and catastrophic bugs should be fixed.
+
+**Note:** If a backup relies on a deprecated feature, then backups made with the last Velero release before this feature is removed must still be restorable in version `n+2`. For instance, something like restic feature support, that might mean that restic is removed from the list of supported uploader types in version `n` but the underlying implementation required to restore from a restic backup won't be removed until release `n+2`.
+
 ## Updating Governance

 All substantive changes in Governance require a supermajority agreement by all maintainers.
@@ -14,7 +14,7 @@ https://velero.io/docs/v1.14/upgrade-to-1.14/

 ### Highlights

-#### The maintenance work for kopia backup repositories is run in jobs
+#### The maintenance work for kopia/restic backup repositories is run in jobs
 Since velero started using kopia as the approach for filesystem-level backup/restore, we've noticed an issue when velero connects to the kopia backup repositories and performs maintenance, it sometimes consumes excessive memory that can cause the velero pod to get OOM Killed.  To mitigate this issue, the maintenance work will be moved out of velero pod to a separate kubernetes job, and the user will be able to specify the resource request in "velero install".
 #### Volume Policies are extended to support more actions to handle volumes
 In an earlier release, a flexible volume policy was introduced to skip certain volumes from a backup.  In v1.14 we've made enhancement to this policy to allow the user to set how the volumes should be backed up.  The user will be able to set "fs-backup" or "snapshot" as value of “action" in the policy and velero will backup the volumes accordingly.  This enhancement allows the user to achieve a fine-grained control like "opt-in/out" without having to update the target workload.  For more details please refer to https://velero.io/docs/v1.14/resource-filtering/#supported-volumepolicy-actions
@@ -38,6 +38,7 @@ Besides the service principal with secret(password)-based authentication, Velero
 * CSI plugin has been merged into velero repo in v1.14 release.  It will be installed by default as an internal plugin, and should not be installed via "–plugins " parameter in "velero install" command.
 * The default resource requests and limitations for node agent are removed in v1.14, to make the node agent pods have the QoS class of "BestEffort", more details please refer to #7391
 * There's a change in namespace filtering behavior during backup:  In v1.14, when the includedNamespaces/excludedNamespaces fields are not set and the labelSelector/OrLabelSelectors are set in the backup spec, the backup will only include the namespaces which contain the resources that match the label selectors, while in previous releases all namespaces will be included in the backup with such settings.  More details refer to #7105
+* Patching the PV in the "Finalizing" state may cause the restore to be in "PartiallyFailed" state when the PV is blocked in "Pending" state, while in the previous release the restore may end up being in "Complete" state. For more details refer to #7866

 ### All Changes
 * Fix backup log to show error string, not index (#7805, @piny940)
@@ -0,0 +1 @@
+Propose a deprecation process for velero
@@ -0,0 +1 @@
+Ignore missing path error in conditional match
@@ -0,0 +1 @@
+Add the design for data mover micro service
@@ -0,0 +1 @@
+Add the result in the backup's VolumeInfo.
@@ -0,0 +1 @@
+Don't consider unschedulable pods unrecoverable
@@ -0,0 +1 @@
+Expose the VolumeHelper to third-party plugins.
@@ -0,0 +1 @@
+Fix issue #7904, add the deprecation and limitation clarification for change PVC selected-node feature
@@ -0,0 +1 @@
+Skip PV patch step in Restoe workflow for WaitForFirstConsumer VolumeBindingMode Pending state PVCs
@@ -0,0 +1 @@
+New data path for data mover ms according to design #7574
@@ -0,0 +1 @@
+Check whether the namespaces specified in namespace filter exist.
@@ -0,0 +1 @@
+Check whether the volume's source is PVC before fetching its PV.
@@ -0,0 +1 @@
+Expose the VolumeHelper to third-party plugins.
@@ -0,0 +1 @@
+Fix issue #7972, sync the backupPVC deletion in expose clean up
@@ -0,0 +1 @@
+Reuse existing plugin manager for get/put volume info
@@ -76,7 +76,7 @@ volumePolicies:
 - Update VolumePolicy action type validation to account for `fs-backup` and `snapshot` as valid VolumePolicy actions.
 - Modifications needed for `fs-backup` action:
  - Now based on the specification of volume policy on backup request we will decide whether to go via legacy pod annotations approach or the newer volume policy based fs-backup action approach.
-  - If there is a presence of volume policy(fs-backup/snapshot)  on the backup request that matches as an action for a volume we use the newer volume policy approach to get the list of the volumes for `fs-backup` action
+  - If there is a presence of volume policy(fs-backup/snapshot) on the backup request that matches as an action for a volume we use the newer volume policy approach to get the list of the volumes for `fs-backup` action
  - Else continue with the annotation based legacy approach workflow.

 - Modifications needed for `snapshot` action:
@@ -111,15 +111,7 @@ It is possible that node-agent pods, as a daemonset, don't run in every worker n
 Otherwise, if a backupPod are scheduled to a node where node-agent pod is absent, the corresponding DataUpload CR will stay in `Accepted` phase until the prepare timeout (by default 30min).  

 At present, as part of the expose operations, the exposer creates a volume, represented by backupPVC, from the snapshot. The backupPVC uses the same storageClass with the source volume. If the `volumeBindingMode` in the storageClass is `Immediate`, the volume is immediately allocated from the underlying storage without waiting for the backupPod. On the other hand, the loadAffinity is set to the backupPod's affinity. If the backupPod is scheduled to a node where the snapshot volume is not accessible, e.g., because of storage topologies, the backupPod won't get into Running state, concequently, the data movement won't complete.  
-Once this problem happens, the backupPod stays in `Pending` phase, and the corresponding DataUpload CR stays in `Accepted` phase until the prepare timeout (by default 30min).   
-
-There is a common solution for the both problems:
- We have an existing logic to periodically enqueue the dataupload CRs which are in the `Accepted` phase for timeout and cancel checks
- We add a new logic to this existing logic to check if the corresponding backupPods are in unrecoverable status
- The above problems could be covered by this check, because in both cases the backupPods are in abnormal and unrecoverable status
- If a backupPod is unrecoverable, the dataupload controller cancels the dataupload and deletes the backupPod
-
-Specifically, when the above problems happen, the status of a backupPod is like below:
+Once this problem happens, the backupPod stays in `Pending` phase, and the corresponding DataUpload CR stays in `Accepted` phase until the prepare timeout (by default 30min). Below is an example of the backupPod's status when the problem happens:   
 ```
  status:
    conditions:
@@ -133,5 +125,8 @@ Specifically, when the above problems happen, the status of a backupPod is like
    phase: Pending
 ```    

+On the other hand, the backupPod is deleted after the prepare timeout, so there is no way to tell the cause is one of the above problems or others.  
+To help the troubleshooting, we can add some diagnostic mechanism to discover the status of the backupPod and node-agent in the same node before deleting it as a result of the prepare timeout.  
+
 [1]: Implemented/unified-repo-and-kopia-integration/unified-repo-and-kopia-integration.md
 [2]: volume-snapshot-data-movement/volume-snapshot-data-movement.md
@@ -0,0 +1,201 @@
+# VGDP Micro Service For Volume Snapshot Data Movement
+
+## Glossary & Abbreviation
+
+**VGDP**: The collective of modules that is introduced in [Unified Repository design][1]. Velero uses these modules to finish data transmission for various purposes. It includes uploaders and the backup repository.  
+**Volume Snapshot Data Movement**: The backup/restore method introduced in [Volume Snapshot Data Movement design][2]. It backs up snapshot data from the volatile and limited production environment into the durable, heterogeneous and scalable backup storage.  
+**VBDM**: Velero Built-in Data Mover as introduced in [Volume Snapshot Data Movement design][2], it is the built-in data mover shipped along with Velero.  
+**Exposer**: Exposer is introduced in [Volume Snapshot Data Movement design][2] and is used to expose the volume snapshots/target volumes for VGDP to access locally.  
+
+## Background
+As the architecture introduced in [Volume Snapshot Data Movement design][2], VGDP instances are running inside the node-agent pods, however, more and more use cases require to run the VGDP instances in dedicated pods, or in another word, make them as micro services, the benefits are as below:
+- This avoids VGDP to access volume data through host path, while host path access involves privilege escalations in some environments (e.g., must run under privileged mode), which makes challenge to users.    
+- This enable users to to control resource (i.e., cpu, memory) request/limit in a granular manner, e.g., control them per backup/restore of a volume
+- This increases the resilience, crash of one VGDP activity won't affect others
+- In the cases that the backup storage must be represented by a Kubernetes persistent volumes (i.e., nfs storage, [COSI][3]), this avoids to dynamically mount the persistent volumes to node-agent pods and cause node-agent pods to restart (this is not accepted since node-agent lose it current state after its pods restart)  
+- This prevents unnecessary full backup. Velero's fs uploaders support file level incremental backup by comparing the file name and metadata. However, at present the files are visited by host path, while pod and PVC's ID are part of the host path, so once the pod is recreated, the same file is regarded as a different file since the pod's ID has been changed. If the fs uploader is in a dedicated pod and files are visited by pod's volume path, files' full path are not changed after pod restarts, so incremental backups could continue.  
+
+## Goals
+- Create a solution to make VGDP instances as micro services
+- Modify the VBDM to offload the VGDP work from node-agent to the VGDP micro service
+- Create the mechanism for VBDM to control and monitor the VGDP micro services in various scenarios
+
+## Non-Goals
+- The current solution covers Volume Snapshot Data Movement backup/restore type only, even though VGDP is also used by pod volume backup. It is less possible to do this for pod volume backup, since it must run inside the source workload pods.  
+- The current solution covers VBDM only. 3rd data movers still follow the **Replacement** section of [Volume Snapshot Data Movement design][2]. That is, 3rd data movers handle the DUCR/DDCR on their own and they are free to make themselves micro service style or monolith service style.  
+
+
+## Overview
+The solution is based on [Volume Snapshot Data Movement design][2], the architecture is followed as is and existing components are not changed unless it is necessary.  
+Below lists the changed components, why and how:  
+**Exposer**: Exposer is to expose the snapshot/target volume as a path/device name/endpoint that are recognizable by VGDP. Varying from the type of snapshot/target volume, a pod may be created as part of the expose. Now, since we run the VGDP instance in a separate pod, a pod is created anyway, we assume exposer creates a pod all the time and make the appropriate exposing configurations to the pod so that VGDP instance could access the snapshot/target volume locally inside the pod. The pod is still called as backupPod or restorePod.  
+Then we need to change the command the backupPod/restorePod is running, the command launches VGDP-MS (VGDP Micro Service, see below) when the container starts up.  
+For CSI snapshot, the backupPod/restorePod is created as the result of expose, the only thing left is to change the backupPod/restorePod's image.  
+**VBDM**: VBDM contains the data mover controller, while the controller calls the Exposer and launches the VGDP instances.  Now, since the VGDP instance is launched by the backupPod/restorePod, the controller should not launch the VGDP instance again. However, the controller still needs to monitor and control the VGDP instance. Moreover, in order to avoid any contest situations, the controller is still the only place to update DUCRs and DDCRs.    
+
+Besides the changes to above existing components, we need to add below new components:  
+**VGDP Watcher**: We create a new module to help the data mover controller to watch activities of the VGDP instance in the backupPod/restorePod. VGDP Watcher is a part of VBDM.  
+**VGDP-MS**: VGDP Micro Service is the binary for the command backupPod/restorePod runs. It accepts the parameters and then launches the VGDP instance according to the request type, specifically, backup or restore. VGDP-MS also runs other modules to sync-up with the data mover controller. VGDP-MS is also a part of VBDM.  
+
+Below diagram shows how these components work together:  
+![vgdp-ms-1.png](vgdp-ms-1.png)  
+
+The [Node-agent concurrency][4] is still used to control the concurrency of VGDP micro services. When there are too many volumes in the backup/restore, which takes too much computing resources(CPU, memory, etc.) or Kubernetes resources(pods, PVCs, PVs, etc.), users could set the concurrency in each node so as to control the total number of concurrent VGDP micro services in the cluster.  
+
+## Detailed Design
+### Exposer
+At present, the exposer creates backupPod/restorePod and sets ```velero-helper pause``` as the command run by backupPod/restorePod.  
+Now, VGDP-MS command will be used, and the ```velero``` image will be running inside the backupPod/restorePod. The command is like below:  
+```velero data-mover backup --volume-path xxx --volume-mode xxx --data-upload xxx --resource-timeout xxx --log-format xxx --log-level xxx```  
+Or:  
+```velero data-mover restore --volume-path xxx --volume-mode xxx --data-download xxx --resource-timeout xxx --log-format xxx --log-level xxx```  
+
+The first one is for backup and the other one is for restore.  
+Below are the parameters of the commands:  
+**volume-path**: Deliver the full path inside the backupPod/restorePod for the volume to be backed up/restored.  
+**volume-mode**: Deliver the mode for the volume be backed up/restored, at present either ```Filesystem``` mode or ```Block``` mode.  
+**data-upload**: DUCR for this backup.  
+**data-download**: DDCR for this backup. 
+**resource-timeout**: resource-timeout is used to control the timeout for operations related to resources. It has the same meaning with the resource-timeout for node-agent.  
+**log-format** and **log-level**: This is to control the behavior of log generation inside VGDP-MS. 
+
+In order to have the same capability and permission with node-agent, below pod configurations are inherited from node-agent and set to backupPod/restorePod's spec:
+- Volumes: Some configMaps will be mapped as volumes to node-agent, so we add the same volumes of node-agent to the backupPod/restorePod
+- Environment Variables
+- Security Contexts  
+We may not actually need all the capabilities in the VGDP-MS as the node-agent. At present, we just duplicate all of them, if we find any problem in future, we can filter out the capabilities that are not required by VGDP-MS.  
+The backupPod/restorePod is not run in Privileged mode as it is not required since the volumes are visisted by pod path.  
+The root user is still required, especially by the restore (in order to restore the file system attributes, owners, etc.), so we will use root user for backupPod/restorePod.  
+We set backupPod/restorePod's ```RestartPolicy``` to ```RestartPolicyNever```, so that once VGDP-MS terminates in any reason, backupPod/restorePod won't restart and the DUCR/DDCR is marked as one of the terminal phases (Completed/Failed/Cancelled) accordingly.  
+
+
+### VGDP Watcher
+#### Dual mode event watch
+The primary task of VGDP Watcher is to watch the status change from backupPod/restorePod or the VGDP instance, so as to inform the data mover controller in below situations:
+- backupPod/restorePod starts
+- VGDP instance starts
+- Progress update
+- VGDP instance completes/fails/cancelled
+- backupPod/restorePod stops
+
+We use two mechanism to make the watch:  
+**Pod Phases**: VGDP Watcher watches the backupPod/restorePod's phases updated by Kubernetes. That is, VGDP Watcher creates an informer to watch the pod resource for the backupPod/restorePod and detect that the pod reaches to one of the terminated phases (i.e., PodSucceeded, PodFailed). We also check the availability & status of the backupPod/restorePod at the beginning of the watch so as to detect the starting of the backupPod/restorePod.  
+**Custom Kubernetes Events**: VGDP-MS generates Kubernetes events and associates them to the DUCR/DDCR at the time of VGDP instance starting/stopping and progress update, then VGDP Watcher creates another informer to watch the Event resource associated to the DUCR/DDCR.  
+
+Pod Phases watch covers the entire lifecycle of the backupPod/restorePod, but we don't know the status of the VGDP instance through it; and it can only deliver information by then end of the pod lifecycle.   
+Custom Event watch generates details of the VGDP instances and the events could be generated any time; but it cannot generate notifications before VGDP starts or in the case that VGDP crashes or shutdown abnormally.  
+
+Therefore, we adopt the both mechanisms to VGDP Watcher. In the end, there will be two sources generating the result of VGDP-MS:  
+- The termination message of backupPod/restorePod
+- The message along with the VGDP Instance Completes/Fails/Cancelled event  
+
+On the one hand, in some cases only the backupPod/restorePod's termination message is available, e.g., the backupPod/restorePod crashes or or backupPod/restorePod quits before VGDP instance is started. So we refer to the first mechanism to get the notifications.  
+On the other hand, if they are both available, we have the results from them for mutual verification.   
+
+Conclusively, under the help of VGDP Watcher, data mover controller starts VGDP-MS controllably and waits until VGDP-MS ends under any circumstances.   
+
+#### AsyncBR adapter
+VGDP Watcher needs to notify the data mover controller when one of the watched event happens, so that the controller could do the operations as if it receives the same callbacks from VGDP as the current behavior. In order not to break the existing code logics of data mover controllers, we make VGDP Watcher as an adapter of AsyncBR which is the interface implemented by VGDP and called by the data mover controller.  
+Since the parameters to call VGDP Watcher is different from the ones to call VGDP, we change the AsyncBR interface to hide some parameters from one another, the new interface is as below:  
+```
+type AsyncBR interface {
+	// Init initializes an asynchronous data path instance
+	Init(ctx context.Context, res *exposer.ExposeResult, param interface{}) error
+
+	// StartBackup starts an asynchronous data path instance for backup
+	StartBackup(dataMoverConfig map[string]string, param interface{}) error
+
+	// StartRestore starts an asynchronous data path instance for restore
+	StartRestore(snapshotID string, dataMoverConfig map[string]string) error
+
+	// Cancel cancels an asynchronous data path instance
+	Cancel()
+
+	// Close closes an asynchronous data path instance
+	Close(ctx context.Context)
+}
+```
+Some parameters are hidden into ```param```, but the functions and calling logics are not changed.  
+
+VGDP Watcher should be launched by the data mover controller before VGDP instance starts, otherwise, multiple corner problems may happen. E.g., VGDP-MS may run the VGDP instance immediately after the backupPod/restorePod is launched and completes it before the data mover controller starts VGDP Watcher, as a result, multiple informs are missed from VGDP Watcher.  
+Therefore, the controller launches VGDP Watcher first and then set the DUCR/DDCR to ```InProgress```; on the other hand, VGDP-MS waits DUCR/DDCR turns to ```InProgress``` before running the VGDP instance.  
+
+### VGDP-MS
+VGDP-MS is represented by ```velero data-mover``` subcommand and has its own subcommand ```backup``` and ```restore```.  
+Below diagram shows the VGDP-MS workflow:  
+![vgdp-ms-2.png](vgdp-ms-2.png)  
+
+**Start DUCR/DDCR Watcher**: VGDP-MS needs to watch the corresponding DUCR/DDCR so as to react on some events happening to the DUCR/DDCR. E.g., when the data movement is cancelled, a ```Cancel``` flag is set to the DUCR/DDCR, by watching the DUCR/DDCR, VGDP-MS is able to see it and cancel the VGDP instance.  
+**Wait DUCR/DDCR InProgress**: As mentioned above, VGDP-MS won't start the VGDP instance until DUCR/DDCR turns to ```InProgress```, by which time VGDP Watcher has been started.  
+**Record VGDP Starts**: This generates the VGDP Instance Starts event.  
+**VGDP Callbacks**: When VGDP comes to one of the terminal states (i.e., completed, failed, cancelled), the corresponding callback is called.  
+**Record VGDP Ends**: This generates the VGDP Instance Completes/Fails/Cancelled event, and also generates backupPod/restorePod termination message.    
+**Record VGDP Progress**: This periodically generates/updates the Progress event with totalBytes/bytesDone to indicate the progress of the data movement.  
+**Set VGDP Output**: This writes the termination message to the backupPod/restorePod's termination log (by default, it is written to ```/dev/termination-log```).  
+
+If VGDP completes, VGDP Instance Completes event and backupPod/restorePod termination shares the same message as below:  
+```
+type BackupResult struct {
+	SnapshotID    string              `json:"snapshotID"`
+	EmptySnapshot bool                `json:"emptySnapshot"`
+	Source        exposer.AccessPoint `json:"source,omitempty"`
+}
+```
+```
+type RestoreResult struct {
+	Target exposer.AccessPoint `json:"target,omitempty"`
+}
+```  
+``` 
+type AccessPoint struct {
+	ByPath  string                        `json:"byPath"`
+	VolMode uploader.PersistentVolumeMode `json:"volumeMode"`
+}
+``` 
+
+The existing VGDP result structures are actually being reused, we just add the json markers so that they can be marshalled.  
+
+As mentioned above, once VGDP-MS ends in any way, the backupPod/restorePod terminates and never restarts, so the end of VGDP-MS means the end of DU/DD.  
+
+For Progress update, the existing Progress structure is being reused:  
+``` 
+type Progress struct {
+	TotalBytes int64 `json:"totalBytes,omitempty"`
+	BytesDone  int64 `json:"doneBytes,omitempty"`
+}
+``` 
+
+### Log Collection
+During the running of VGDP instance, some logs are generated which are important for troubleshooting. This includes all the logs generated by the uploader and repository. Therefore, it is important to collect these logs.  
+On the other hand, the logs are now generated in the backupPod/restorePod, while the backupPod/restorePod is deleted immediately after the data movement completes. Therefore, by default, ```velero debug``` is not able to collect these logs.  
+
+As a solution, we use logrus's hook mechanism to redirect the backupPod/restorePod's logs into node-agent's log, so that ```velero debug``` could collect VGDP logs as is without any changes.  
+
+Below diagram shows how VGDP logs are redirected:  
+![vgdp-ms-3.png](vgdp-ms-3.png) 
+
+This log redirecting mechanism is thread safe since the hook acquires the write lock before writing the log buffer, so it guarantees that in the node-agent log there is no corruptions after redirecting the log, and the redirected logs and the original node-agent logs are not projected into each other.     
+
+## node-agent
+node-agent is still required. Even though VGDP is now not running inside node-agent, node-agent still hosts the data mover controller which reconciles DUCR/DDCR and operates DUCR/DDCR in other steps before the VGDP instance is started, i.e., Accept, Expose, etc.  
+Privileged mode and root user are not required for node-agent anymore by Volume Snapshot Data Movement, however, they are still required by PVB(PodVolumeBackup) and PVR(PodVolumeRestore). Therefore, we will keep the node-agent deamonset as is, for any users who don't use PVB/PVR and have concern about the privileged mode/root user, they need to manually modify the deamonset spec to remove the dependencies.  
+
+## CRD Changes
+There is no changes to any CRD.  
+
+## Installation Changes
+No changes to installation, the backupPod/restorePod's configurations are all inherited from node-agent.  
+
+## Upgrade
+Upgrade is not impacted.  
+
+## CLI
+CLI is not changed.
+
+
+
+[1]: ../Implemented/unified-repo-and-kopia-integration/unified-repo-and-kopia-integration.md
+[2]: ../volume-snapshot-data-movement/volume-snapshot-data-movement.md
+[3]: https://kubernetes.io/blog/2022/09/02/cosi-kubernetes-object-storage-management/
+[4]: ../Implemented/node-agent-concurrency.md
+
@@ -37,7 +37,7 @@ Moreover, we would like to create a general workflow to variations during the da

 ## Architecture of Volume Snapshot Data Movement

-## Workflows  
+### Workflows  

 Here are the diagrams that illustrate components and workflows for backup and restore respectively.  
 For backup, we intend to create an extensive architecture for various snapshot types, snapshot accesses and various data accesses. For example, the snapshot specific operations are isolated in Data Mover Plugin and Exposer. In this way, we only need to change the two modules for variations. Likely, the data access details are isolated into uploaders, so different uploaders could be plugged into the workflow seamlessly.  
@@ -52,7 +52,7 @@ Below is the backup workflow:
 Below is the restore workflow:  
 ![restore-workflow.png](restore-workflow.png)  

-## Components
+### Components
 Below are the generic components in the data movement workflow:  

 **Velero**: Velero controls the backup/restore workflow, it calls BIA/RIA V2 to backup/restore an object that involves data movement, specifically, a PVC or a PV.  
@@ -69,13 +69,13 @@ DMs take the responsibility to handle DUCR/DDCRs, Velero provides a built-in DM
 **Velero Generic Data Path (VGDP)**: VGDP is the collective of modules that is introduced in [Unified Repository design][1]. Velero uses these modules to finish data transmission for various purposes. In includes uploaders and the backup repository.  
 **Uploader**: Uploader is the module in VGDP that reads data from the source and writes to backup repository for backup; while read data from backup repository and write to the restore target for restore. At present, only file system uploader is supported. In future, the block level uploader will be added. For file system and basic block uploader, only Kopia uploader will be used, Restic will not be integrated with VBDM.   

-## Replacement
+### Replacement
 3rd parties could integrate their own data movement into Velero by replacing VBDM with their own DMs. The DMs should process DUCR/DDCRs appropriately and finally put them into one of the terminal states as shown in the DataUpload CRD and DataDownload CRD sections.  
 Theoretically, replacing the DMP is also allowed. In this way, the entire workflow is customized, so this is out of the scope of this design.  

-# Detailed Design
+## Detailed Design

-## Backup Sequence
+### Backup Sequence
 Below are the data movement actions and sequences during backup:  
 ![backup-sequence.png](backup-sequence.png) 

@@ -150,7 +150,7 @@ We keep VGDP reused for VBDM, so everything inside VGDP are kept as is. For deta
 When VGDP completes backup, it returns an ID that represent the root object saved into the backup repository for this backup, through the root object, we will be able to enumerate the entire backup data.  
 This Repo Snapshot ID will be saved along with the DUCR.  

-## DataUpload CRD
+### DataUpload CRD
 Below are the essential fields of DataUpload CRD. The CRD covers below information:
 - The information to manipulate the specified snapshot
 - The information to manipulate the specified data mover
@@ -351,7 +351,7 @@ spec:

 ```

-## Restore Sequence
+### Restore Sequence

 Below are the data movement actions sequences during restore:  
 ![restore-sequence.png](restore-sequence.png) 
@@ -387,7 +387,7 @@ This also means that Velero should not restore the PV if a data movement restore

 For restore, VBDM doesn’t need to persist anything.  

-## DataDownload CRD
+### DataDownload CRD
 Below are the essential fields of DataDownload CRD. The CRD covers below information:
 - The information to manipulate the target volume
 - The information to manipulate the specified data mover
@@ -5,7 +5,7 @@ go 1.22.0
 require (
 	cloud.google.com/go/storage v1.40.0
 	github.com/Azure/azure-sdk-for-go/sdk/azcore v1.11.1
-	github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.2
+	github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.6.0
 	github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v5 v5.6.0
 	github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.5.0
 	github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.3.2
@@ -42,9 +42,9 @@ require (
 	go.uber.org/zap v1.27.0
 	golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1
 	golang.org/x/mod v0.17.0
-	golang.org/x/net v0.24.0
+	golang.org/x/net v0.26.0
 	golang.org/x/oauth2 v0.19.0
-	golang.org/x/text v0.14.0
+	golang.org/x/text v0.16.0
 	google.golang.org/api v0.172.0
 	google.golang.org/grpc v1.63.2
 	google.golang.org/protobuf v1.33.0
@@ -68,7 +68,7 @@ require (
 	cloud.google.com/go/compute v1.24.0 // indirect
 	cloud.google.com/go/compute/metadata v0.2.3 // indirect
 	cloud.google.com/go/iam v1.1.7 // indirect
-	github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.2 // indirect
+	github.com/Azure/azure-sdk-for-go/sdk/internal v1.8.0 // indirect
 	github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 // indirect
 	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.5.4 // indirect
 	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.14.11 // indirect
@@ -157,10 +157,10 @@ require (
 	go.opentelemetry.io/otel/trace v1.25.0 // indirect
 	go.starlark.net v0.0.0-20201006213952-227f4aabceb5 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
-	golang.org/x/crypto v0.22.0 // indirect
+	golang.org/x/crypto v0.24.0 // indirect
 	golang.org/x/sync v0.7.0 // indirect
-	golang.org/x/sys v0.19.0 // indirect
-	golang.org/x/term v0.19.0 // indirect
+	golang.org/x/sys v0.21.0 // indirect
+	golang.org/x/term v0.21.0 // indirect
 	golang.org/x/time v0.5.0 // indirect
 	gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
 	google.golang.org/genproto v0.0.0-20240227224415-6ceb2ff114de // indirect
@@ -50,10 +50,10 @@ cloud.google.com/go/storage v1.40.0/go.mod h1:Rrj7/hKlG87BLqDJYtwR0fbPld8uJPbQ2u
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
 github.com/Azure/azure-sdk-for-go/sdk/azcore v1.11.1 h1:E+OJmp2tPvt1W+amx48v1eqbjDYsgN+RzP4q16yV5eM=
 github.com/Azure/azure-sdk-for-go/sdk/azcore v1.11.1/go.mod h1:a6xsAQUZg+VsS3TJ05SRp524Hs4pZ/AeFSr5ENf0Yjo=
-github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.2 h1:FDif4R1+UUR+00q6wquyX90K7A8dN+R5E8GEadoP7sU=
-github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.2/go.mod h1:aiYBYui4BJ/BJCAIKs92XiPyQfTaBWqvHujDwKb6CBU=
-github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.2 h1:LqbJ/WzJUwBf8UiaSzgX7aMclParm9/5Vgp+TY51uBQ=
-github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.2/go.mod h1:yInRyqWXAuaPrgI7p70+lDDgh3mlBohis29jGMISnmc=
+github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.6.0 h1:U2rTu3Ef+7w9FHKIAXM6ZyqF3UOWJZ12zIm8zECAFfg=
+github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.6.0/go.mod h1:9kIvujWAA58nmPmWB1m23fyWic1kYZMxD9CxaWn4Qpg=
+github.com/Azure/azure-sdk-for-go/sdk/internal v1.8.0 h1:jBQA3cKT4L2rWMpgE7Yt3Hwh2aUj8KXjIGLxjHeYNNo=
+github.com/Azure/azure-sdk-for-go/sdk/internal v1.8.0/go.mod h1:4OG6tQ9EOP/MT0NMjDlRzWoVFxfu9rN9B2X+tlSVktg=
 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v5 v5.6.0 h1:ui3YNbxfW7J3tTFIZMH6LIGRjCngp+J+nIFlnizfNTE=
 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v5 v5.6.0/go.mod h1:gZmgV+qBqygoznvqo2J9oKZAFziqhLZ2xE/WVUmzkHA=
 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal/v2 v2.0.0 h1:PTFGRSlMKCQelWwxUyYVEUqseBJVemLyqWJjvMyt0do=
@@ -199,8 +199,6 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
 github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
-github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI=
-github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ=
 github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM=
 github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
 github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
@@ -649,8 +647,8 @@ github.com/robfig/cron v1.1.0/go.mod h1:JGuDeoQd7Z6yL4zQhZ3OPEVHB7fL6Ka6skscFHfm
 github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
 github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
 github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
-github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
-github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
+github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8=
+github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4=
 github.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc=
 github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
 github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
@@ -795,8 +793,8 @@ golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPh
 golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
 golang.org/x/crypto v0.0.0-20220214200702-86341886e292/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
-golang.org/x/crypto v0.22.0 h1:g1v0xeRhjcugydODzvb3mEM9SQ0HGp9s/nh3COQ/C30=
-golang.org/x/crypto v0.22.0/go.mod h1:vr6Su+7cTlO45qkww3VDJlzDn0ctJvRgYbC2NvXHt+M=
+golang.org/x/crypto v0.24.0 h1:mnl8DM0o513X8fdIkmyFE/5hTYxbwYOjDS/+rK6qpRI=
+golang.org/x/crypto v0.24.0/go.mod h1:Z1PMYSOR5nyMcyAVAIQSKCDwalqy85Aqn1x3Ws4L5DM=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -883,8 +881,8 @@ golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96b
 golang.org/x/net v0.0.0-20210520170846-37e1c6afe023/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
-golang.org/x/net v0.24.0 h1:1PcaxkF854Fu3+lvBIx5SYn9wRlBzzcnHZSiaFFAb0w=
-golang.org/x/net v0.24.0/go.mod h1:2Q7sJY5mzlzWjKtYUEXSlBWCdyaioyXzRB2RtU8KVE8=
+golang.org/x/net v0.26.0 h1:soB7SVo0PWrY4vPW/+ay0jKDNScG2X9wFeYlXIvJsOQ=
+golang.org/x/net v0.26.0/go.mod h1:5YKkiSynbBIh3p6iOc/vibscux0x38BZDkn8sCUPxHE=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -986,14 +984,14 @@ golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.19.0 h1:q5f1RH2jigJ1MoAWp2KTp3gm5zAGFUTarQZ5U386+4o=
-golang.org/x/sys v0.19.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/sys v0.21.0 h1:rF+pYz3DAGSQAxAu1CbC7catZg4ebC4UIeIhKxBZvws=
+golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
 golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
-golang.org/x/term v0.19.0 h1:+ThwsDv+tYfnJFhF4L8jITxu1tdTWRTZpdsWgEgjL6Q=
-golang.org/x/term v0.19.0/go.mod h1:2CuTdWZ7KHSQwUzKva0cbMg6q2DMI3Mmxp+gKJbskEk=
+golang.org/x/term v0.21.0 h1:WVXCp+/EBEHOj53Rvu+7KiT/iElMrO8ACK16SMZ3jaA=
+golang.org/x/term v0.21.0/go.mod h1:ooXLefLobQVslOqselCNF4SxFAaoS6KujMbsGzSDmX0=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -1003,8 +1001,8 @@ golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
-golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
-golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
+golang.org/x/text v0.16.0 h1:a94ExnEXNtEwYLGJSIUxnWoxoRz/ZcCsV63ROupILh4=
+golang.org/x/text v0.16.0/go.mod h1:GhwF1Be+LQoKShO3cGOHzqOgRrGaYc9AvblQOmPVHnI=
 golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
@@ -1071,8 +1069,8 @@ golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4f
 golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
 golang.org/x/tools v0.1.2/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
 golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
-golang.org/x/tools v0.16.1 h1:TLyB3WofjdOEepBHAU20JdNC1Zbg87elYofWYAY5oZA=
-golang.org/x/tools v0.16.1/go.mod h1:kYVVN6I1mBNoB1OX+noeBjbRk4IUEPa7JJ+TJMEooJ0=
+golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d h1:vU5i/LfpvrRCpgM/VPfJLg5KjxD3E+hfT1SH+d9zLwg=
+golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -181,7 +181,7 @@ func matchConditions(u *unstructured.Unstructured, rules []MatchRule, _ logrus.F
 	p := &JSONPatcher{patches: fixed}
 	_, err := p.applyPatch(u)
 	if err != nil {
-		if errors.Is(err, jsonpatch.ErrTestFailed) {
+		if errors.Is(err, jsonpatch.ErrTestFailed) || errors.Is(err, jsonpatch.ErrMissing) {
 			return false, nil
 		}
 		return false, err
@@ -1767,6 +1767,35 @@ func TestResourceModifiers_conditional_patches(t *testing.T) {
 			wantErr:       false,
 			wantObj:       cmWithLabelAToB.DeepCopy(),
 		},
+		{
+			name: "missing condition path and skip patches",
+			rm: &ResourceModifiers{
+				Version: "v1",
+				ResourceModifierRules: []ResourceModifierRule{
+					{
+						Conditions: Conditions{
+							GroupResource: "*",
+							Namespaces:    []string{"fake"},
+							Matches: []MatchRule{
+								{
+									Path:  "/metadata/labels/a/b",
+									Value: "c",
+								},
+							},
+						},
+						MergePatches: []JSONMergePatch{
+							{
+								PatchData: `{"metadata":{"labels":{"a":"c"}}}`,
+							},
+						},
+					},
+				},
+			},
+			obj:           cmWithLabelAToB.DeepCopy(),
+			groupResource: "configmaps",
+			wantErr:       false,
+			wantObj:       cmWithLabelAToB.DeepCopy(),
+		},
 	}

 	for _, tt := range tests {
@@ -16,11 +16,16 @@ limitations under the License.
 package resourcepolicies

 import (
+	"context"
 	"fmt"
 	"strings"

 	"github.com/pkg/errors"
+	"github.com/sirupsen/logrus"
 	v1 "k8s.io/api/core/v1"
+	crclient "sigs.k8s.io/controller-runtime/pkg/client"
+
+	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
 )

 type VolumeActionType string
@@ -148,7 +153,43 @@ func (p *Policies) Validate() error {
 	return nil
 }

-func GetResourcePoliciesFromConfig(cm *v1.ConfigMap) (*Policies, error) {
+func GetResourcePoliciesFromBackup(
+	backup velerov1api.Backup,
+	client crclient.Client,
+	logger logrus.FieldLogger,
+) (resourcePolicies *Policies, err error) {
+	if backup.Spec.ResourcePolicy != nil &&
+		strings.EqualFold(backup.Spec.ResourcePolicy.Kind, ConfigmapRefType) {
+		policiesConfigMap := &v1.ConfigMap{}
+		err = client.Get(
+			context.Background(),
+			crclient.ObjectKey{Namespace: backup.Namespace, Name: backup.Spec.ResourcePolicy.Name},
+			policiesConfigMap,
+		)
+		if err != nil {
+			logger.Errorf("Fail to get ResourcePolicies %s ConfigMap with error %s.",
+				backup.Namespace+"/"+backup.Spec.ResourcePolicy.Name, err.Error())
+			return nil, fmt.Errorf("fail to get ResourcePolicies %s ConfigMap with error %s",
+				backup.Namespace+"/"+backup.Spec.ResourcePolicy.Name, err.Error())
+		}
+		resourcePolicies, err = getResourcePoliciesFromConfig(policiesConfigMap)
+		if err != nil {
+			logger.Errorf("Fail to read ResourcePolicies from ConfigMap %s with error %s.",
+				backup.Namespace+"/"+backup.Name, err.Error())
+			return nil, fmt.Errorf("fail to read the ResourcePolicies from ConfigMap %s with error %s",
+				backup.Namespace+"/"+backup.Name, err.Error())
+		} else if err = resourcePolicies.Validate(); err != nil {
+			logger.Errorf("Fail to validate ResourcePolicies in ConfigMap %s with error %s.",
+				backup.Namespace+"/"+backup.Name, err.Error())
+			return nil, fmt.Errorf("fail to validate ResourcePolicies in ConfigMap %s with error %s",
+				backup.Namespace+"/"+backup.Name, err.Error())
+		}
+	}
+
+	return resourcePolicies, nil
+}
+
+func getResourcePoliciesFromConfig(cm *v1.ConfigMap) (*Policies, error) {
 	if cm == nil {
 		return nil, fmt.Errorf("could not parse config from nil configmap")
 	}
@@ -231,7 +231,7 @@ func TestGetResourcePoliciesFromConfig(t *testing.T) {
 	}

 	// Call the function and check for errors
-	resPolicies, err := GetResourcePoliciesFromConfig(cm)
+	resPolicies, err := getResourcePoliciesFromConfig(cm)
 	assert.Nil(t, err)

 	// Check that the returned resourcePolicies object contains the expected data
@@ -92,6 +92,9 @@ type BackupVolumeInfo struct {
 	// Snapshot completes timestamp.
 	CompletionTimestamp *metav1.Time `json:"completionTimestamp,omitempty"`

+	// Whether the volume data is backed up successfully.
+	Result VolumeResult `json:"result,omitempty"`
+
 	CSISnapshotInfo          *CSISnapshotInfo          `json:"csiSnapshotInfo,omitempty"`
 	SnapshotDataMovementInfo *SnapshotDataMovementInfo `json:"snapshotDataMovementInfo,omitempty"`
 	NativeSnapshotInfo       *NativeSnapshotInfo       `json:"nativeSnapshotInfo,omitempty"`
@@ -99,6 +102,14 @@ type BackupVolumeInfo struct {
 	PVInfo                   *PVInfo                   `json:"pvInfo,omitempty"`
 }

+type VolumeResult string
+
+const (
+	VolumeResultSucceeded VolumeResult = "succeeded"
+	VolumeResultFailed    VolumeResult = "failed"
+	//VolumeResultCanceled  VolumeResult = "canceled"
+)
+
 type RestoreVolumeInfo struct {
 	// The name of the restored PVC
 	PVCName string `json:"pvcName,omitempty"`
@@ -139,6 +150,9 @@ type CSISnapshotInfo struct {

 	// The Async Operation's ID.
 	OperationID string `json:"operationID,omitempty"`
+
+	// The VolumeSnapshot's Status.ReadyToUse value
+	ReadyToUse *bool
 }

 // SnapshotDataMovementInfo is used for displaying the snapshot data mover status.
@@ -162,6 +176,9 @@ type SnapshotDataMovementInfo struct {

 	// Moved snapshot data size.
 	Size int64 `json:"size"`
+
+	// The DataUpload's Status.Phase value
+	Phase velerov2alpha1.DataUploadPhase
 }

 // NativeSnapshotInfo is used for displaying the Velero native snapshot status.
@@ -180,6 +197,9 @@ type NativeSnapshotInfo struct {

 	// The cloud provider snapshot volume's IOPS.
 	IOPS string `json:"iops"`
+
+	// The NativeSnapshot's Status.Phase value
+	Phase SnapshotPhase
 }

 func newNativeSnapshotInfo(s *Snapshot) *NativeSnapshotInfo {
@@ -192,6 +212,7 @@ func newNativeSnapshotInfo(s *Snapshot) *NativeSnapshotInfo {
 		VolumeType:     s.Spec.VolumeType,
 		VolumeAZ:       s.Spec.VolumeAZ,
 		IOPS:           strconv.FormatInt(iops, 10),
+		Phase:          s.Status.Phase,
 	}
 }

@@ -219,6 +240,9 @@ type PodVolumeInfo struct {
 	// The PVB-taken k8s node's name.
 	// This field will be empty when the struct is used to represent a podvolumerestore.
 	NodeName string `json:"nodeName,omitempty"`
+
+	// The PVB's Status.Phase value
+	Phase velerov1api.PodVolumeBackupPhase
 }

 func newPodVolumeInfoFromPVB(pvb *velerov1api.PodVolumeBackup) *PodVolumeInfo {
@@ -230,6 +254,7 @@ func newPodVolumeInfoFromPVB(pvb *velerov1api.PodVolumeBackup) *PodVolumeInfo {
 		PodName:        pvb.Spec.Pod.Name,
 		PodNamespace:   pvb.Spec.Pod.Namespace,
 		NodeName:       pvb.Spec.Node,
+		Phase:          pvb.Status.Phase,
 	}
 }

@@ -349,13 +374,20 @@ func (v *BackupVolumesInformation) generateVolumeInfoForVeleroNativeSnapshot() {

 	for _, nativeSnapshot := range v.NativeSnapshots {
 		if pvcPVInfo := v.pvMap.retrieve(nativeSnapshot.Spec.PersistentVolumeName, "", ""); pvcPVInfo != nil {
+			volumeResult := VolumeResultFailed
+			if nativeSnapshot.Status.Phase == SnapshotPhaseCompleted {
+				volumeResult = VolumeResultSucceeded
+			}
 			volumeInfo := &BackupVolumeInfo{
-				BackupMethod:       NativeSnapshot,
-				PVCName:            pvcPVInfo.PVCName,
-				PVCNamespace:       pvcPVInfo.PVCNamespace,
-				PVName:             pvcPVInfo.PV.Name,
-				SnapshotDataMoved:  false,
-				Skipped:            false,
+				BackupMethod:      NativeSnapshot,
+				PVCName:           pvcPVInfo.PVCName,
+				PVCNamespace:      pvcPVInfo.PVCNamespace,
+				PVName:            pvcPVInfo.PV.Name,
+				SnapshotDataMoved: false,
+				Skipped:           false,
+				// Only set Succeeded to true when the NativeSnapshot's phase is Completed,
+				// although NativeSnapshot doesn't check whether the snapshot creation result.
+				Result:             volumeResult,
 				NativeSnapshotInfo: newNativeSnapshotInfo(nativeSnapshot),
 				PVInfo: &PVInfo{
 					ReclaimPolicy: string(pvcPVInfo.PV.Spec.PersistentVolumeReclaimPolicy),
@@ -451,6 +483,7 @@ func (v *BackupVolumesInformation) generateVolumeInfoForCSIVolumeSnapshot() {
 					Driver:         volumeSnapshotClass.Driver,
 					SnapshotHandle: snapshotHandle,
 					OperationID:    operation.Spec.OperationID,
+					ReadyToUse:     volumeSnapshot.Status.ReadyToUse,
 				},
 				PVInfo: &PVInfo{
 					ReclaimPolicy: string(pvcPVInfo.PV.Spec.PersistentVolumeReclaimPolicy),
@@ -484,6 +517,14 @@ func (v *BackupVolumesInformation) generateVolumeInfoFromPVB() {
 			CompletionTimestamp: pvb.Status.CompletionTimestamp,
 			PVBInfo:             newPodVolumeInfoFromPVB(pvb),
 		}
+
+		// Only set Succeeded to true when the PVB's phase is Completed.
+		if pvb.Status.Phase == velerov1api.PodVolumeBackupPhaseCompleted {
+			volumeInfo.Result = VolumeResultSucceeded
+		} else {
+			volumeInfo.Result = VolumeResultFailed
+		}
+
 		pvcName, err := pvcByPodvolume(context.TODO(), v.crClient, pvb.Spec.Pod.Name, pvb.Spec.Pod.Namespace, pvb.Spec.Volume)
 		if err != nil {
 			v.logger.WithError(err).Warn("Fail to get PVC from PodVolumeBackup: ", pvb.Name)
@@ -582,6 +623,7 @@ func (v *BackupVolumesInformation) generateVolumeInfoFromDataUpload() {
 						DataMover:    dataMover,
 						UploaderType: kopia,
 						OperationID:  operation.Spec.OperationID,
+						Phase:        dataUpload.Status.Phase,
 					},
 					PVInfo: &PVInfo{
 						ReclaimPolicy: string(pvcPVInfo.PV.Spec.PersistentVolumeReclaimPolicy),
@@ -662,17 +704,17 @@ type RestoreVolumeInfoTracker struct {

 	// map of PV name to the NativeSnapshotInfo from which the PV is restored
 	pvNativeSnapshotMap map[string]*NativeSnapshotInfo
-	// map of PV name to the CSISnapshot object from which the PV is restored
-	pvCSISnapshotMap map[string]snapshotv1api.VolumeSnapshot
-	datadownloadList *velerov2alpha1.DataDownloadList
-	pvrs             []*velerov1api.PodVolumeRestore
+	// map of PVC object to the CSISnapshot object from which the PV is restored
+	// the key is in the form of $pvc-ns/$pvc-name
+	pvcCSISnapshotMap map[string]snapshotv1api.VolumeSnapshot
+	datadownloadList  *velerov2alpha1.DataDownloadList
+	pvrs              []*velerov1api.PodVolumeRestore
 }

 // Populate data objects in the tracker, which will be used to generate the RestoreVolumeInfo array in Result()
 // The input param resourceList should be the final result of the restore.
 func (t *RestoreVolumeInfoTracker) Populate(ctx context.Context, restoredResourceList map[string][]string) {
 	pvcs := RestoredPVCFromRestoredResourceList(restoredResourceList)
-
 	t.Lock()
 	defer t.Unlock()
 	for item := range pvcs {
@@ -684,25 +726,26 @@ func (t *RestoreVolumeInfoTracker) Populate(ctx context.Context, restoredResourc
 			log.WithError(err).Error("Failed to get PVC")
 			continue
 		}
-		if pvc.Status.Phase != corev1api.ClaimBound || pvc.Spec.VolumeName == "" {
-			log.Info("PVC is not bound or has no volume name")
-			continue
-		}
-		pv := &corev1api.PersistentVolume{}
-		if err := t.client.Get(ctx, kbclient.ObjectKey{Name: pvc.Spec.VolumeName}, pv); err != nil {
-			log.WithError(err).Error("Failed to get PV")
-		} else {
-			t.pvPvc.insert(*pv, pvcName, pvcNS)
-		}
 		// Collect the CSI VolumeSnapshot objects referenced by the restored PVCs,
 		if pvc.Spec.DataSource != nil && pvc.Spec.DataSource.Kind == "VolumeSnapshot" {
 			vs := &snapshotv1api.VolumeSnapshot{}
 			if err := t.client.Get(ctx, kbclient.ObjectKey{Namespace: pvcNS, Name: pvc.Spec.DataSource.Name}, vs); err != nil {
 				log.WithError(err).Error("Failed to get VolumeSnapshot")
 			} else {
-				t.pvCSISnapshotMap[pv.Name] = *vs
+				t.pvcCSISnapshotMap[pvc.Namespace+"/"+pvcName] = *vs
 			}
 		}
+		if pvc.Status.Phase == corev1api.ClaimBound && pvc.Spec.VolumeName != "" {
+			pv := &corev1api.PersistentVolume{}
+			if err := t.client.Get(ctx, kbclient.ObjectKey{Name: pvc.Spec.VolumeName}, pv); err != nil {
+				log.WithError(err).Error("Failed to get PV")
+			} else {
+				t.pvPvc.insert(*pv, pvcName, pvcNS)
+			}
+		} else {
+			log.Warn("PVC is not bound or has no volume name")
+			continue
+		}
 	}
 	if err := t.client.List(ctx, t.datadownloadList, &kbclient.ListOptions{
 		Namespace:     t.restore.Namespace,
@@ -761,21 +804,35 @@ func (t *RestoreVolumeInfoTracker) Result() []*RestoreVolumeInfo {
 	}

 	// Generate RestoreVolumeInfo for PVs restored from CSISnapshots
-	for pvName, csiSnapshot := range t.pvCSISnapshotMap {
+	for pvc, csiSnapshot := range t.pvcCSISnapshotMap {
+		n := strings.Split(pvc, "/")
+		if len(n) != 2 {
+			t.log.Warnf("Invalid PVC key '%s' in the pvc-CSISnapshot map, skip populating it to volume info", pvc)
+			continue
+		}
+		pvcNS, pvcName := n[0], n[1]
+		var restoreSize int64 = 0
+		if csiSnapshot.Status != nil && csiSnapshot.Status.RestoreSize != nil {
+			restoreSize = csiSnapshot.Status.RestoreSize.Value()
+		}
+		vscName := ""
+		if csiSnapshot.Spec.Source.VolumeSnapshotContentName != nil {
+			vscName = *csiSnapshot.Spec.Source.VolumeSnapshotContentName
+		}
 		volumeInfo := &RestoreVolumeInfo{
-			PVName:            pvName,
+			PVCNamespace:      pvcNS,
+			PVCName:           pvcName,
 			SnapshotDataMoved: false,
 			RestoreMethod:     CSISnapshot,
 			CSISnapshotInfo: &CSISnapshotInfo{
 				SnapshotHandle: csiSnapshot.Annotations[VolumeSnapshotHandleAnnotation],
-				Size:           csiSnapshot.Status.RestoreSize.Value(),
+				Size:           restoreSize,
 				Driver:         csiSnapshot.Annotations[CSIDriverNameAnnotation],
-				VSCName:        *csiSnapshot.Spec.Source.VolumeSnapshotContentName,
+				VSCName:        vscName,
 			},
 		}
-		if pvcPVInfo := t.pvPvc.retrieve(pvName, "", ""); pvcPVInfo != nil {
-			volumeInfo.PVCName = pvcPVInfo.PVCName
-			volumeInfo.PVCNamespace = pvcPVInfo.PVCNamespace
+		if pvcPVInfo := t.pvPvc.retrieve("", pvcName, pvcNS); pvcPVInfo != nil {
+			volumeInfo.PVName = pvcPVInfo.PV.Name
 		}
 		volumeInfos = append(volumeInfos, volumeInfo)
 	}
@@ -829,7 +886,7 @@ func NewRestoreVolInfoTracker(restore *velerov1api.Restore, logger logrus.FieldL
 			data: make(map[string]pvcPvInfo),
 		},
 		pvNativeSnapshotMap: make(map[string]*NativeSnapshotInfo),
-		pvCSISnapshotMap:    make(map[string]snapshotv1api.VolumeSnapshot),
+		pvcCSISnapshotMap:   make(map[string]snapshotv1api.VolumeSnapshot),
 		datadownloadList:    &velerov2alpha1.DataDownloadList{},
 	}
 }
@@ -200,6 +200,58 @@ func TestGenerateVolumeInfoForVeleroNativeSnapshot(t *testing.T) {
 			},
 			expectedVolumeInfos: []*BackupVolumeInfo{},
 		},
+		{
+			name: "Normal native snapshot with failed phase",
+			pvMap: map[string]pvcPvInfo{
+				"testPV": {
+					PVCName:      "testPVC",
+					PVCNamespace: "velero",
+					PV: corev1api.PersistentVolume{
+						ObjectMeta: metav1.ObjectMeta{
+							Name:   "testPV",
+							Labels: map[string]string{"a": "b"},
+						},
+						Spec: corev1api.PersistentVolumeSpec{
+							PersistentVolumeReclaimPolicy: corev1api.PersistentVolumeReclaimDelete,
+						},
+					},
+				},
+			},
+			nativeSnapshot: Snapshot{
+				Spec: SnapshotSpec{
+					PersistentVolumeName: "testPV",
+					VolumeIOPS:           int64Ptr(100),
+					VolumeType:           "ssd",
+					VolumeAZ:             "us-central1-a",
+				},
+				Status: SnapshotStatus{
+					ProviderSnapshotID: "pvc-b31e3386-4bbb-4937-95d-7934cd62-b0a1-494b-95d7-0687440e8d0c",
+					Phase:              SnapshotPhaseFailed,
+				},
+			},
+			expectedVolumeInfos: []*BackupVolumeInfo{
+				{
+					PVCName:      "testPVC",
+					PVCNamespace: "velero",
+					PVName:       "testPV",
+					BackupMethod: NativeSnapshot,
+					Result:       VolumeResultFailed,
+					PVInfo: &PVInfo{
+						ReclaimPolicy: "Delete",
+						Labels: map[string]string{
+							"a": "b",
+						},
+					},
+					NativeSnapshotInfo: &NativeSnapshotInfo{
+						SnapshotHandle: "pvc-b31e3386-4bbb-4937-95d-7934cd62-b0a1-494b-95d7-0687440e8d0c",
+						VolumeType:     "ssd",
+						VolumeAZ:       "us-central1-a",
+						IOPS:           "100",
+						Phase:          SnapshotPhaseFailed,
+					},
+				},
+			},
+		},
 		{
 			name: "Normal native snapshot",
 			pvMap: map[string]pvcPvInfo{
@@ -226,6 +278,7 @@ func TestGenerateVolumeInfoForVeleroNativeSnapshot(t *testing.T) {
 				},
 				Status: SnapshotStatus{
 					ProviderSnapshotID: "pvc-b31e3386-4bbb-4937-95d-7934cd62-b0a1-494b-95d7-0687440e8d0c",
+					Phase:              SnapshotPhaseCompleted,
 				},
 			},
 			expectedVolumeInfos: []*BackupVolumeInfo{
@@ -234,6 +287,7 @@ func TestGenerateVolumeInfoForVeleroNativeSnapshot(t *testing.T) {
 					PVCNamespace: "velero",
 					PVName:       "testPV",
 					BackupMethod: NativeSnapshot,
+					Result:       VolumeResultSucceeded,
 					PVInfo: &PVInfo{
 						ReclaimPolicy: "Delete",
 						Labels: map[string]string{
@@ -245,6 +299,7 @@ func TestGenerateVolumeInfoForVeleroNativeSnapshot(t *testing.T) {
 						VolumeType:     "ssd",
 						VolumeAZ:       "us-central1-a",
 						IOPS:           "100",
+						Phase:          SnapshotPhaseCompleted,
 					},
 				},
 			},
@@ -274,6 +329,7 @@ func TestGenerateVolumeInfoForVeleroNativeSnapshot(t *testing.T) {
 func TestGenerateVolumeInfoForCSIVolumeSnapshot(t *testing.T) {
 	resourceQuantity := resource.MustParse("100Gi")
 	now := metav1.Now()
+	readyToUse := true
 	tests := []struct {
 		name                  string
 		volumeSnapshot        snapshotv1api.VolumeSnapshot
@@ -381,6 +437,7 @@ func TestGenerateVolumeInfoForCSIVolumeSnapshot(t *testing.T) {
 					BoundVolumeSnapshotContentName: stringPtr("testContent"),
 					CreationTime:                   &now,
 					RestoreSize:                    &resourceQuantity,
+					ReadyToUse:                     &readyToUse,
 				},
 			},
 			volumeSnapshotClass:   *builder.ForVolumeSnapshotClass("testClass").Driver("pd.csi.storage.gke.io").Result(),
@@ -427,6 +484,7 @@ func TestGenerateVolumeInfoForCSIVolumeSnapshot(t *testing.T) {
 						Size:           107374182400,
 						VSCName:        "testContent",
 						OperationID:    "testID",
+						ReadyToUse:     &readyToUse,
 					},
 					PVInfo: &PVInfo{
 						ReclaimPolicy: "Delete",
@@ -506,6 +564,7 @@ func TestGenerateVolumeInfoFromPVB(t *testing.T) {
 					PVCNamespace: "",
 					PVName:       "",
 					BackupMethod: PodVolumeBackup,
+					Result:       VolumeResultFailed,
 					PVBInfo: &PodVolumeInfo{
 						PodName:      "testPod",
 						PodNamespace: "velero",
@@ -537,7 +596,7 @@ func TestGenerateVolumeInfoFromPVB(t *testing.T) {
 			expectedVolumeInfos: []*BackupVolumeInfo{},
 		},
 		{
-			name: "PVB's volume has a PVC",
+			name: "PVB's volume has a PVC with failed phase",
 			pvMap: map[string]pvcPvInfo{
 				"testPV": {
 					PVCName:      "testPVC",
@@ -553,7 +612,13 @@ func TestGenerateVolumeInfoFromPVB(t *testing.T) {
 					},
 				},
 			},
-			pvb: builder.ForPodVolumeBackup("velero", "testPVB").PodName("testPod").PodNamespace("velero").StartTimestamp(&now).CompletionTimestamp(&now).Result(),
+			pvb: builder.ForPodVolumeBackup("velero", "testPVB").
+				PodName("testPod").
+				PodNamespace("velero").
+				StartTimestamp(&now).
+				CompletionTimestamp(&now).
+				Phase(velerov1api.PodVolumeBackupPhaseFailed).
+				Result(),
 			pod: builder.ForPod("velero", "testPod").Containers(&corev1api.Container{
 				Name: "test",
 				VolumeMounts: []corev1api.VolumeMount{
@@ -580,9 +645,74 @@ func TestGenerateVolumeInfoFromPVB(t *testing.T) {
 					BackupMethod:        PodVolumeBackup,
 					StartTimestamp:      &now,
 					CompletionTimestamp: &now,
+					Result:              VolumeResultFailed,
 					PVBInfo: &PodVolumeInfo{
 						PodName:      "testPod",
 						PodNamespace: "velero",
+						Phase:        velerov1api.PodVolumeBackupPhaseFailed,
+					},
+					PVInfo: &PVInfo{
+						ReclaimPolicy: string(corev1api.PersistentVolumeReclaimDelete),
+						Labels:        map[string]string{"a": "b"},
+					},
+				},
+			},
+		},
+		{
+			name: "PVB's volume has a PVC",
+			pvMap: map[string]pvcPvInfo{
+				"testPV": {
+					PVCName:      "testPVC",
+					PVCNamespace: "velero",
+					PV: corev1api.PersistentVolume{
+						ObjectMeta: metav1.ObjectMeta{
+							Name:   "testPV",
+							Labels: map[string]string{"a": "b"},
+						},
+						Spec: corev1api.PersistentVolumeSpec{
+							PersistentVolumeReclaimPolicy: corev1api.PersistentVolumeReclaimDelete,
+						},
+					},
+				},
+			},
+			pvb: builder.ForPodVolumeBackup("velero", "testPVB").
+				PodName("testPod").
+				PodNamespace("velero").
+				StartTimestamp(&now).
+				CompletionTimestamp(&now).
+				Phase(velerov1api.PodVolumeBackupPhaseCompleted).
+				Result(),
+			pod: builder.ForPod("velero", "testPod").Containers(&corev1api.Container{
+				Name: "test",
+				VolumeMounts: []corev1api.VolumeMount{
+					{
+						Name:      "testVolume",
+						MountPath: "/data",
+					},
+				},
+			}).Volumes(
+				&corev1api.Volume{
+					Name: "",
+					VolumeSource: corev1api.VolumeSource{
+						PersistentVolumeClaim: &corev1api.PersistentVolumeClaimVolumeSource{
+							ClaimName: "testPVC",
+						},
+					},
+				},
+			).Result(),
+			expectedVolumeInfos: []*BackupVolumeInfo{
+				{
+					PVCName:             "testPVC",
+					PVCNamespace:        "velero",
+					PVName:              "testPV",
+					BackupMethod:        PodVolumeBackup,
+					StartTimestamp:      &now,
+					CompletionTimestamp: &now,
+					Result:              VolumeResultSucceeded,
+					PVBInfo: &PodVolumeInfo{
+						PodName:      "testPod",
+						PodNamespace: "velero",
+						Phase:        velerov1api.PodVolumeBackupPhaseCompleted,
 					},
 					PVInfo: &PVInfo{
 						ReclaimPolicy: string(corev1api.PersistentVolumeReclaimDelete),
@@ -770,10 +900,15 @@ func TestGenerateVolumeInfoFromDataUpload(t *testing.T) {
 		},
 		{
 			name: "Normal DataUpload case",
-			dataUpload: builder.ForDataUpload("velero", "testDU").DataMover("velero").CSISnapshot(&velerov2alpha1.CSISnapshotSpec{
-				VolumeSnapshot: "testVS",
-				SnapshotClass:  "testClass",
-			}).SnapshotID("testSnapshotHandle").StartTimestamp(&now).Result(),
+			dataUpload: builder.ForDataUpload("velero", "testDU").
+				DataMover("velero").
+				CSISnapshot(&velerov2alpha1.CSISnapshotSpec{
+					VolumeSnapshot: "testVS",
+					SnapshotClass:  "testClass",
+				}).SnapshotID("testSnapshotHandle").
+				StartTimestamp(&now).
+				Phase(velerov2alpha1.DataUploadPhaseCompleted).
+				Result(),
 			volumeSnapshotClass: builder.ForVolumeSnapshotClass("testClass").Driver("pd.csi.storage.gke.io").Result(),
 			operation: &itemoperation.BackupOperation{
 				Spec: itemoperation.BackupOperationSpec{
@@ -832,6 +967,7 @@ func TestGenerateVolumeInfoFromDataUpload(t *testing.T) {
 						DataMover:    "velero",
 						UploaderType: "kopia",
 						OperationID:  "testOperation",
+						Phase:        velerov2alpha1.DataUploadPhaseCompleted,
 					},
 					PVInfo: &PVInfo{
 						ReclaimPolicy: string(corev1api.PersistentVolumeReclaimDelete),
@@ -933,7 +1069,7 @@ func TestRestoreVolumeInfoResult(t *testing.T) {
 					data: make(map[string]pvcPvInfo),
 				},
 				pvNativeSnapshotMap: map[string]*NativeSnapshotInfo{},
-				pvCSISnapshotMap:    map[string]snapshotv1api.VolumeSnapshot{},
+				pvcCSISnapshotMap:   map[string]snapshotv1api.VolumeSnapshot{},
 				datadownloadList:    &velerov2alpha1.DataDownloadList{},
 				pvrs:                []*velerov1api.PodVolumeRestore{},
 			},
@@ -968,8 +1104,8 @@ func TestRestoreVolumeInfoResult(t *testing.T) {
 						IOPS:           "10000",
 					},
 				},
-				pvCSISnapshotMap: map[string]snapshotv1api.VolumeSnapshot{},
-				datadownloadList: &velerov2alpha1.DataDownloadList{},
+				pvcCSISnapshotMap: map[string]snapshotv1api.VolumeSnapshot{},
+				datadownloadList:  &velerov2alpha1.DataDownloadList{},
 				pvrs: []*velerov1api.PodVolumeRestore{
 					builder.ForPodVolumeRestore("velero", "testRestore-1234").
 						PodNamespace("testNS").
@@ -1031,8 +1167,8 @@ func TestRestoreVolumeInfoResult(t *testing.T) {
 					},
 				},
 				pvNativeSnapshotMap: map[string]*NativeSnapshotInfo{},
-				pvCSISnapshotMap: map[string]snapshotv1api.VolumeSnapshot{
-					"testPV": *builder.ForVolumeSnapshot("sourceNS", "testCSISnapshot").
+				pvcCSISnapshotMap: map[string]snapshotv1api.VolumeSnapshot{
+					"testNS/testPVC": *builder.ForVolumeSnapshot("sourceNS", "testCSISnapshot").
 						ObjectMeta(
 							builder.WithAnnotations(VolumeSnapshotHandleAnnotation, "csi-snap-001",
 								CSIDriverNameAnnotation, "test-csi-driver"),
@@ -1101,7 +1237,7 @@ func TestRestoreVolumeInfoResult(t *testing.T) {
 					},
 				},
 				pvNativeSnapshotMap: map[string]*NativeSnapshotInfo{},
-				pvCSISnapshotMap:    map[string]snapshotv1api.VolumeSnapshot{},
+				pvcCSISnapshotMap:   map[string]snapshotv1api.VolumeSnapshot{},
 				datadownloadList: &velerov2alpha1.DataDownloadList{
 					Items: []velerov2alpha1.DataDownload{
 						*builder.ForDataDownload("velero", "testDataDownload-1").
@@ -143,20 +143,24 @@ func (v volumeHelperImpl) ShouldPerformFSBackup(volume corev1api.Volume, pod cor
 	}

 	if v.volumePolicy != nil {
-		pvc, err := kubeutil.GetPVCForPodVolume(&volume, &pod, v.client)
-		if err != nil {
-			v.logger.WithError(err).Errorf("fail to get PVC for pod %s", pod.Namespace+"/"+pod.Name)
-			return false, err
-		}
-		pv, err := kubeutil.GetPVForPVC(pvc, v.client)
-		if err != nil {
-			v.logger.WithError(err).Errorf("fail to get PV for PVC %s", pvc.Namespace+"/"+pvc.Name)
-			return false, err
+		var resource interface{}
+		resource = &volume
+		if volume.VolumeSource.PersistentVolumeClaim != nil {
+			pvc, err := kubeutil.GetPVCForPodVolume(&volume, &pod, v.client)
+			if err != nil {
+				v.logger.WithError(err).Errorf("fail to get PVC for pod %s", pod.Namespace+"/"+pod.Name)
+				return false, err
+			}
+			resource, err = kubeutil.GetPVForPVC(pvc, v.client)
+			if err != nil {
+				v.logger.WithError(err).Errorf("fail to get PV for PVC %s", pvc.Namespace+"/"+pvc.Name)
+				return false, err
+			}
 		}

-		action, err := v.volumePolicy.GetMatchAction(pv)
+		action, err := v.volumePolicy.GetMatchAction(resource)
 		if err != nil {
-			v.logger.WithError(err).Errorf("fail to get VolumePolicy match action for PV %s", pv.Name)
+			v.logger.WithError(err).Error("fail to get VolumePolicy match action for volume")
 			return false, err
 		}

@@ -552,6 +552,32 @@ func TestVolumeHelperImpl_ShouldPerformFSBackup(t *testing.T) {
 			shouldFSBackup: true,
 			expectedErr:    false,
 		},
+		{
+			name: "Volume source is emptyDir, VolumePolicy match, return true and no error",
+			pod: builder.ForPod("ns", "pod-1").
+				Volumes(
+					&corev1.Volume{
+						Name: "",
+						VolumeSource: corev1.VolumeSource{
+							EmptyDir: &corev1.EmptyDirVolumeSource{},
+						},
+					}).Result(),
+			resourcePolicies: &resourcepolicies.ResourcePolicies{
+				Version: "v1",
+				VolumePolicies: []resourcepolicies.VolumePolicy{
+					{
+						Conditions: map[string]interface{}{
+							"volumeTypes": []string{"emptyDir"},
+						},
+						Action: resourcepolicies.Action{
+							Type: resourcepolicies.FSBackup,
+						},
+					},
+				},
+			},
+			shouldFSBackup: true,
+			expectedErr:    false,
+		},
 		{
 			name: "VolumePolicy match, action type is not fs-backup, return false and no error",
 			pod: builder.ForPod("ns", "pod-1").
@@ -40,6 +40,7 @@ import (
 	"github.com/vmware-tanzu/velero/pkg/kuberesource"
 	"github.com/vmware-tanzu/velero/pkg/label"
 	plugincommon "github.com/vmware-tanzu/velero/pkg/plugin/framework/common"
+	"github.com/vmware-tanzu/velero/pkg/plugin/utils/volumehelper"
 	"github.com/vmware-tanzu/velero/pkg/plugin/velero"
 	biav2 "github.com/vmware-tanzu/velero/pkg/plugin/velero/backupitemaction/v2"
 	uploaderUtil "github.com/vmware-tanzu/velero/pkg/uploader/util"
@@ -229,6 +230,22 @@ func (p *pvcBackupItemAction) Execute(
 		return item, nil, "", nil, nil
 	}

+	shouldSnapshot, err := volumehelper.ShouldPerformSnapshotWithBackup(
+		item,
+		kuberesource.PersistentVolumeClaims,
+		*backup,
+		p.crClient,
+		p.log,
+	)
+	if err != nil {
+		return nil, nil, "", nil, err
+	}
+	if !shouldSnapshot {
+		p.log.Debugf("CSI plugin skip snapshot for PVC %s according to the VolumeHelper setting.",
+			pvc.Namespace+"/"+pvc.Name)
+		return nil, nil, "", nil, err
+	}
+
 	vs, err := p.createVolumeSnapshot(pvc, backup)
 	if err != nil {
 		return nil, nil, "", nil, err
@@ -61,6 +61,7 @@ func TestExecute(t *testing.T) {
 		expectedBackup     *velerov1api.Backup
 		expectedDataUpload *velerov2alpha1.DataUpload
 		expectedPVC        *corev1.PersistentVolumeClaim
+		resourcePolicy     *corev1.ConfigMap
 	}{
 		{
 			name:        "Skip PVC BIA when backup is in finalizing phase",
@@ -127,6 +128,16 @@ func TestExecute(t *testing.T) {
 					builder.WithLabels(velerov1api.BackupNameLabel, "test", velerov1api.VolumeSnapshotLabel, "")).
 				VolumeName("testPV").StorageClass("testSC").Phase(corev1.ClaimBound).Result(),
 		},
+		{
+			name:           "Test ResourcePolicy",
+			backup:         builder.ForBackup("velero", "test").ResourcePolicies("resourcePolicy").SnapshotVolumes(false).Result(),
+			resourcePolicy: builder.ForConfigMap("velero", "resourcePolicy").Data("policy", "{\"version\":\"v1\", \"volumePolicies\":[{\"conditions\":{\"csi\": {}},\"action\":{\"type\":\"snapshot\"}}]}").Result(),
+			pvc:            builder.ForPersistentVolumeClaim("velero", "testPVC").VolumeName("testPV").StorageClass("testSC").Phase(corev1.ClaimBound).Result(),
+			pv:             builder.ForPersistentVolume("testPV").CSI("hostpath", "testVolume").Result(),
+			sc:             builder.ForStorageClass("testSC").Provisioner("hostpath").Result(),
+			vsClass:        builder.ForVolumeSnapshotClass("tescVSClass").Driver("hostpath").ObjectMeta(builder.WithLabels(velerov1api.VolumeSnapshotClassSelectorLabel, "")).Result(),
+			expectedErr:    nil,
+		},
 	}

 	for _, tc := range tests {
@@ -147,6 +158,9 @@ func TestExecute(t *testing.T) {
 			if tc.vsClass != nil {
 				require.NoError(t, crClient.Create(context.Background(), tc.vsClass))
 			}
+			if tc.resourcePolicy != nil {
+				require.NoError(t, crClient.Create(context.Background(), tc.resourcePolicy))
+			}

 			pvcBIA := pvcBackupItemAction{
 				log:      logger,
@@ -190,6 +204,8 @@ func TestExecute(t *testing.T) {
 			resultUnstructed, _, _, _, err := pvcBIA.Execute(&unstructured.Unstructured{Object: pvcMap}, tc.backup)
 			if tc.expectedErr != nil {
 				require.Equal(t, err, tc.expectedErr)
+			} else {
+				require.NoError(t, err)
 			}

 			if tc.expectedDataUpload != nil {
@@ -95,6 +95,7 @@ type Backupper interface {
 		outBackupFile io.Writer,
 		backupItemActionResolver framework.BackupItemActionResolverV2,
 		asyncBIAOperations []*itemoperation.BackupOperation,
+		backupStore persistence.BackupStore,
 	) error
 }

@@ -610,6 +611,7 @@ func (kb *kubernetesBackupper) FinalizeBackup(
 	outBackupFile io.Writer,
 	backupItemActionResolver framework.BackupItemActionResolverV2,
 	asyncBIAOperations []*itemoperation.BackupOperation,
+	backupStore persistence.BackupStore,
 ) error {
 	gzw := gzip.NewWriter(outBackupFile)
 	defer gzw.Close()
@@ -726,7 +728,7 @@ func (kb *kubernetesBackupper) FinalizeBackup(
 		}).Infof("Updated %d items out of an estimated total of %d (estimate will change throughout the backup finalizer)", len(backupRequest.BackedUpItems), totalItems)
 	}

-	backupStore, volumeInfos, err := kb.getVolumeInfos(*backupRequest.Backup, log)
+	volumeInfos, err := backupStore.GetBackupVolumeInfos(backupRequest.Backup.Name)
 	if err != nil {
 		log.WithError(err).Errorf("fail to get the backup VolumeInfos for backup %s", backupRequest.Name)
 		return err
@@ -810,34 +812,6 @@ type tarWriter interface {
 	WriteHeader(*tar.Header) error
 }

-func (kb *kubernetesBackupper) getVolumeInfos(
-	backup velerov1api.Backup,
-	log logrus.FieldLogger,
-) (persistence.BackupStore, []*volume.BackupVolumeInfo, error) {
-	location := &velerov1api.BackupStorageLocation{}
-	if err := kb.kbClient.Get(context.Background(), kbclient.ObjectKey{
-		Namespace: backup.Namespace,
-		Name:      backup.Spec.StorageLocation,
-	}, location); err != nil {
-		return nil, nil, errors.WithStack(err)
-	}
-
-	pluginManager := kb.pluginManager(log)
-	defer pluginManager.CleanupClients()
-
-	backupStore, storeErr := kb.backupStoreGetter.Get(location, pluginManager, log)
-	if storeErr != nil {
-		return nil, nil, storeErr
-	}
-
-	volumeInfos, err := backupStore.GetBackupVolumeInfos(backup.Name)
-	if err != nil {
-		return nil, nil, err
-	}
-
-	return backupStore, volumeInfos, nil
-}
-
 // updateVolumeInfos update the VolumeInfos according to the AsyncOperations
 func updateVolumeInfos(
 	volumeInfos []*volume.BackupVolumeInfo,
@@ -863,6 +837,13 @@ func updateVolumeInfos(
 				volumeInfos[index].SnapshotDataMovementInfo.SnapshotHandle = dataUpload.Status.SnapshotID
 				volumeInfos[index].SnapshotDataMovementInfo.RetainedSnapshot = dataUpload.Spec.CSISnapshot.VolumeSnapshot
 				volumeInfos[index].SnapshotDataMovementInfo.Size = dataUpload.Status.Progress.TotalBytes
+				volumeInfos[index].SnapshotDataMovementInfo.Phase = dataUpload.Status.Phase
+
+				if dataUpload.Status.Phase == velerov2alpha1.DataUploadPhaseCompleted {
+					volumeInfos[index].Result = volume.VolumeResultSucceeded
+				} else {
+					volumeInfos[index].Result = volume.VolumeResultFailed
+				}
 			}
 		}
 	}
@@ -879,6 +860,13 @@ func updateVolumeInfos(
 					// VSC and VS status. When the controller finds they reach to the ReadyToUse state,
 					// The operation.Status.Updated is set as the found time.
 					volumeInfos[volumeIndex].CompletionTimestamp = operations[opIndex].Status.Updated
+
+					// Set Succeeded to true when the operation has no error.
+					if operations[opIndex].Status.Error == "" {
+						volumeInfos[volumeIndex].Result = volume.VolumeResultSucceeded
+					} else {
+						volumeInfos[volumeIndex].Result = volume.VolumeResultFailed
+					}
 				}
 			}
 		}
@@ -54,8 +54,6 @@ import (
 	"github.com/vmware-tanzu/velero/pkg/kuberesource"
 	"github.com/vmware-tanzu/velero/pkg/persistence"
 	persistencemocks "github.com/vmware-tanzu/velero/pkg/persistence/mocks"
-	"github.com/vmware-tanzu/velero/pkg/plugin/clientmgmt"
-	pluginmocks "github.com/vmware-tanzu/velero/pkg/plugin/mocks"
 	"github.com/vmware-tanzu/velero/pkg/plugin/velero"
 	biav2 "github.com/vmware-tanzu/velero/pkg/plugin/velero/backupitemaction/v2"
 	vsv1 "github.com/vmware-tanzu/velero/pkg/plugin/velero/volumesnapshotter/v1"
@@ -4519,23 +4517,6 @@ func TestBackupNamespaces(t *testing.T) {
 	}
 }

-func TestGetVolumeInfos(t *testing.T) {
-	h := newHarness(t)
-	pluginManager := new(pluginmocks.Manager)
-	backupStore := new(persistencemocks.BackupStore)
-	h.backupper.pluginManager = func(logrus.FieldLogger) clientmgmt.Manager { return pluginManager }
-	h.backupper.backupStoreGetter = NewFakeSingleObjectBackupStoreGetter(backupStore)
-	backupStore.On("GetBackupVolumeInfos", "backup-01").Return([]*volume.BackupVolumeInfo{}, nil)
-	pluginManager.On("CleanupClients").Return()
-
-	backup := builder.ForBackup("velero", "backup-01").StorageLocation("default").Result()
-	bsl := builder.ForBackupStorageLocation("velero", "default").Result()
-	require.NoError(t, h.backupper.kbClient.Create(context.Background(), bsl))
-
-	_, _, err := h.backupper.getVolumeInfos(*backup, h.log)
-	require.NoError(t, err)
-}
-
 func TestUpdateVolumeInfos(t *testing.T) {
 	timeExample := time.Date(2014, 6, 5, 11, 56, 45, 0, time.Local)
 	now := metav1.NewTime(timeExample)
@@ -4548,6 +4529,39 @@ func TestUpdateVolumeInfos(t *testing.T) {
 		volumeInfos         []*volume.BackupVolumeInfo
 		expectedVolumeInfos []*volume.BackupVolumeInfo
 	}{
+		{
+			name: "CSISnapshot VolumeInfo update with Operation fails",
+			operations: []*itemoperation.BackupOperation{
+				{
+					Spec: itemoperation.BackupOperationSpec{
+						OperationID: "test-operation",
+					},
+					Status: itemoperation.OperationStatus{
+						Error:   "failed",
+						Updated: &now,
+					},
+				},
+			},
+			volumeInfos: []*volume.BackupVolumeInfo{
+				{
+					BackupMethod:        volume.CSISnapshot,
+					CompletionTimestamp: &metav1.Time{},
+					CSISnapshotInfo: &volume.CSISnapshotInfo{
+						OperationID: "test-operation",
+					},
+				},
+			},
+			expectedVolumeInfos: []*volume.BackupVolumeInfo{
+				{
+					BackupMethod:        volume.CSISnapshot,
+					CompletionTimestamp: &now,
+					Result:              volume.VolumeResultFailed,
+					CSISnapshotInfo: &volume.CSISnapshotInfo{
+						OperationID: "test-operation",
+					},
+				},
+			},
+		},
 		{
 			name: "CSISnapshot VolumeInfo update",
 			operations: []*itemoperation.BackupOperation{
@@ -4573,6 +4587,7 @@ func TestUpdateVolumeInfos(t *testing.T) {
 				{
 					BackupMethod:        volume.CSISnapshot,
 					CompletionTimestamp: &now,
+					Result:              volume.VolumeResultSucceeded,
 					CSISnapshotInfo: &volume.CSISnapshotInfo{
 						OperationID: "test-operation",
 					},
@@ -4580,13 +4595,14 @@ func TestUpdateVolumeInfos(t *testing.T) {
 			},
 		},
 		{
-			name:       "DataUpload VolumeInfo update",
+			name:       "DataUpload VolumeInfo update with fail phase",
 			operations: []*itemoperation.BackupOperation{},
 			dataUpload: builder.ForDataUpload("velero", "du-1").
 				CompletionTimestamp(&now).
 				CSISnapshot(&velerov2alpha1.CSISnapshotSpec{VolumeSnapshot: "vs-1"}).
 				SnapshotID("snapshot-id").
 				Progress(shared.DataMoveOperationProgress{TotalBytes: 1000}).
+				Phase(velerov2alpha1.DataUploadPhaseFailed).
 				SourceNamespace("ns-1").
 				SourcePVC("pvc-1").
 				Result(),
@@ -4605,11 +4621,51 @@ func TestUpdateVolumeInfos(t *testing.T) {
 					PVCName:             "pvc-1",
 					PVCNamespace:        "ns-1",
 					CompletionTimestamp: &now,
+					Result:              volume.VolumeResultFailed,
 					SnapshotDataMovementInfo: &volume.SnapshotDataMovementInfo{
 						DataMover:        "velero",
 						RetainedSnapshot: "vs-1",
 						SnapshotHandle:   "snapshot-id",
 						Size:             1000,
+						Phase:            velerov2alpha1.DataUploadPhaseFailed,
+					},
+				},
+			},
+		},
+		{
+			name:       "DataUpload VolumeInfo update",
+			operations: []*itemoperation.BackupOperation{},
+			dataUpload: builder.ForDataUpload("velero", "du-1").
+				CompletionTimestamp(&now).
+				CSISnapshot(&velerov2alpha1.CSISnapshotSpec{VolumeSnapshot: "vs-1"}).
+				SnapshotID("snapshot-id").
+				Progress(shared.DataMoveOperationProgress{TotalBytes: 1000}).
+				Phase(velerov2alpha1.DataUploadPhaseCompleted).
+				SourceNamespace("ns-1").
+				SourcePVC("pvc-1").
+				Result(),
+			volumeInfos: []*volume.BackupVolumeInfo{
+				{
+					PVCName:             "pvc-1",
+					PVCNamespace:        "ns-1",
+					CompletionTimestamp: &metav1.Time{},
+					SnapshotDataMovementInfo: &volume.SnapshotDataMovementInfo{
+						DataMover: "velero",
+					},
+				},
+			},
+			expectedVolumeInfos: []*volume.BackupVolumeInfo{
+				{
+					PVCName:             "pvc-1",
+					PVCNamespace:        "ns-1",
+					CompletionTimestamp: &now,
+					Result:              volume.VolumeResultSucceeded,
+					SnapshotDataMovementInfo: &volume.SnapshotDataMovementInfo{
+						DataMover:        "velero",
+						RetainedSnapshot: "vs-1",
+						SnapshotHandle:   "snapshot-id",
+						Size:             1000,
+						Phase:            velerov2alpha1.DataUploadPhaseCompleted,
 					},
 				},
 			},
@@ -24,8 +24,6 @@ import (
 	"strings"
 	"time"

-	"github.com/vmware-tanzu/velero/internal/volumehelper"
-
 	"github.com/pkg/errors"
 	"github.com/sirupsen/logrus"
 	corev1api "k8s.io/api/core/v1"
@@ -42,6 +40,7 @@ import (
 	"github.com/vmware-tanzu/velero/internal/hook"
 	"github.com/vmware-tanzu/velero/internal/resourcepolicies"
 	"github.com/vmware-tanzu/velero/internal/volume"
+	"github.com/vmware-tanzu/velero/internal/volumehelper"
 	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
 	"github.com/vmware-tanzu/velero/pkg/archive"
 	"github.com/vmware-tanzu/velero/pkg/client"
@@ -741,6 +741,23 @@ func (r *itemCollector) collectNamespaces(
 		return nil, errors.WithStack(err)
 	}

+	for _, includedNSName := range r.backupRequest.Backup.Spec.IncludedNamespaces {
+		nsExists := false
+		// Skip checking the namespace existing when it's "*".
+		if includedNSName == "*" {
+			continue
+		}
+		for _, unstructuredNS := range unstructuredList.Items {
+			if unstructuredNS.GetName() == includedNSName {
+				nsExists = true
+			}
+		}
+
+		if !nsExists {
+			log.Errorf("fail to get the namespace %s specified in backup.Spec.IncludedNamespaces", includedNSName)
+		}
+	}
+
 	var singleSelector labels.Selector
 	var orSelectors []labels.Selector

@@ -224,6 +224,17 @@ func TestItemCollectorBackupNamespaces(t *testing.T) {
 			},
 			expectedTrackedNS: []string{"ns1", "ns2"},
 		},
+		{
+			name:   "ns specified by the IncludeNamespaces cannot be found",
+			backup: builder.ForBackup("velero", "backup").IncludedNamespaces("ns1", "invalid", "*").Result(),
+			ie:     collections.NewIncludesExcludes().Includes("ns1", "invalid", "*"),
+			namespaces: []*corev1.Namespace{
+				builder.ForNamespace("ns1").ObjectMeta(builder.WithLabels("name", "ns1")).Result(),
+				builder.ForNamespace("ns2").Result(),
+				builder.ForNamespace("ns3").Result(),
+			},
+			expectedTrackedNS: []string{"ns1"},
+		},
 	}

 	for _, tc := range tests {
@@ -139,7 +139,7 @@ func (o *Options) runRepoPrune(f velerocli.Factory, namespace string, logger log
 			credentials.CredentialGetter{
 				FromFile:   credentialFileStore,
 				FromSecret: credentialSecretStore,
-			}, o.RepoType, cli, logger)
+			}, o.RepoType, logger)
 	}

 	// backupRepository
@@ -619,6 +619,7 @@ func describNativeSnapshot(d *Describer, details bool, info *volume.BackupVolume
 		d.Printf("\t\t\tType:\t%s\n", info.NativeSnapshotInfo.VolumeType)
 		d.Printf("\t\t\tAvailability Zone:\t%s\n", info.NativeSnapshotInfo.VolumeAZ)
 		d.Printf("\t\t\tIOPS:\t%s\n", info.NativeSnapshotInfo.IOPS)
+		d.Printf("\t\t\tResult:\t%s\n", info.Result)
 	} else {
 		d.Printf("\t\t%s: specify --details for more information\n", info.PVName)
 	}
@@ -662,6 +663,7 @@ func describeLocalSnapshot(d *Describer, details bool, info *volume.BackupVolume
 		d.Printf("\t\t\t\tStorage Snapshot ID: %s\n", info.CSISnapshotInfo.SnapshotHandle)
 		d.Printf("\t\t\t\tSnapshot Size (bytes): %d\n", info.CSISnapshotInfo.Size)
 		d.Printf("\t\t\t\tCSI Driver: %s\n", info.CSISnapshotInfo.Driver)
+		d.Printf("\t\t\t\tResult: %s\n", info.Result)
 	} else {
 		d.Printf("\t\t\tSnapshot: %s\n", "included, specify --details for more information")
 	}
@@ -683,6 +685,7 @@ func describeDataMovement(d *Describer, details bool, info *volume.BackupVolumeI
 		d.Printf("\t\t\t\tData Mover: %s\n", dataMover)
 		d.Printf("\t\t\t\tUploader Type: %s\n", info.SnapshotDataMovementInfo.UploaderType)
 		d.Printf("\t\t\t\tMoved data Size (bytes): %d\n", info.SnapshotDataMovementInfo.Size)
+		d.Printf("\t\t\t\tResult: %s\n", info.Result)
 	} else {
 		d.Printf("\t\t\tData Movement: %s\n", "included, specify --details for more information")
 	}
@@ -22,17 +22,15 @@ import (
 	"text/tabwriter"
 	"time"

-	"github.com/vmware-tanzu/velero/internal/volume"
-	"github.com/vmware-tanzu/velero/pkg/itemoperation"
-
-	"github.com/stretchr/testify/require"
-
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 	v1 "k8s.io/api/core/v1"

-	"github.com/vmware-tanzu/velero/pkg/builder"
-
+	"github.com/vmware-tanzu/velero/internal/volume"
 	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
+	velerov2alpha1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1"
+	"github.com/vmware-tanzu/velero/pkg/builder"
+	"github.com/vmware-tanzu/velero/pkg/itemoperation"
 )

 func TestDescribeUploaderConfig(t *testing.T) {
@@ -353,6 +351,7 @@ func TestDescribeNativeSnapshots(t *testing.T) {
 				{
 					BackupMethod: volume.NativeSnapshot,
 					PVName:       "pv-1",
+					Result:       volume.VolumeResultSucceeded,
 					NativeSnapshotInfo: &volume.NativeSnapshotInfo{
 						SnapshotHandle: "snapshot-1",
 						VolumeType:     "ebs",
@@ -368,6 +367,7 @@ func TestDescribeNativeSnapshots(t *testing.T) {
      Type:               ebs
      Availability Zone:  us-east-2
      IOPS:               1000 mbps
+      Result:             succeeded
 `,
 		},
 	}
@@ -438,6 +438,7 @@ func TestCSISnapshots(t *testing.T) {
 					PVCNamespace:          "pvc-ns-2",
 					PVCName:               "pvc-2",
 					PreserveLocalSnapshot: true,
+					Result:                volume.VolumeResultSucceeded,
 					CSISnapshotInfo: &volume.CSISnapshotInfo{
 						SnapshotHandle: "snapshot-2",
 						Size:           1024,
@@ -456,6 +457,7 @@ func TestCSISnapshots(t *testing.T) {
        Storage Snapshot ID: snapshot-2
        Snapshot Size (bytes): 1024
        CSI Driver: fake-driver
+        Result: succeeded
 `,
 		},
 		{
@@ -487,6 +489,7 @@ func TestCSISnapshots(t *testing.T) {
 					PVCNamespace:      "pvc-ns-4",
 					PVCName:           "pvc-4",
 					SnapshotDataMoved: true,
+					Result:            volume.VolumeResultSucceeded,
 					SnapshotDataMovementInfo: &volume.SnapshotDataMovementInfo{
 						DataMover:      "velero",
 						UploaderType:   "fake-uploader",
@@ -503,6 +506,7 @@ func TestCSISnapshots(t *testing.T) {
        Data Mover: velero
        Uploader Type: fake-uploader
        Moved data Size (bytes): 0
+        Result: succeeded
 `,
 		},
 		{
@@ -512,12 +516,14 @@ func TestCSISnapshots(t *testing.T) {
 					BackupMethod:      volume.CSISnapshot,
 					PVCNamespace:      "pvc-ns-5",
 					PVCName:           "pvc-5",
+					Result:            volume.VolumeResultFailed,
 					SnapshotDataMoved: true,
 					SnapshotDataMovementInfo: &volume.SnapshotDataMovementInfo{
 						UploaderType:   "fake-uploader",
 						SnapshotHandle: "fake-repo-id-5",
 						OperationID:    "fake-operation-5",
 						Size:           100,
+						Phase:          velerov2alpha1.DataUploadPhaseFailed,
 					},
 				},
 			},
@@ -529,6 +535,7 @@ func TestCSISnapshots(t *testing.T) {
        Data Mover: velero
        Uploader Type: fake-uploader
        Moved data Size (bytes): 100
+        Result: failed
 `,
 		},
 	}
@@ -371,6 +371,7 @@ func describNativeSnapshotInSF(details bool, info *volume.BackupVolumeInfo, snap
 		snapshotInfo["type"] = info.NativeSnapshotInfo.VolumeType
 		snapshotInfo["availabilityZone"] = info.NativeSnapshotInfo.VolumeAZ
 		snapshotInfo["IOPS"] = info.NativeSnapshotInfo.IOPS
+		snapshotInfo["result"] = string(info.Result)

 		snapshotDetails[info.PVName] = snapshotInfo
 	} else {
@@ -421,6 +422,7 @@ func describeLocalSnapshotInSF(details bool, info *volume.BackupVolumeInfo, snap
 		localSnapshot["storageSnapshotID"] = info.CSISnapshotInfo.SnapshotHandle
 		localSnapshot["snapshotSize(bytes)"] = info.CSISnapshotInfo.Size
 		localSnapshot["csiDriver"] = info.CSISnapshotInfo.Driver
+		localSnapshot["result"] = string(info.Result)

 		snapshotDetail["snapshot"] = localSnapshot
 	} else {
@@ -444,6 +446,7 @@ func describeDataMovementInSF(details bool, info *volume.BackupVolumeInfo, snaps
 		dataMovement["dataMover"] = dataMover

 		dataMovement["uploaderType"] = info.SnapshotDataMovementInfo.UploaderType
+		dataMovement["result"] = string(info.Result)

 		snapshotDetail["dataMovement"] = dataMovement
 	} else {
@@ -21,17 +21,14 @@ import (
 	"testing"
 	"time"

+	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
-
 	v1 "k8s.io/api/core/v1"

 	"github.com/vmware-tanzu/velero/internal/volume"
-	"github.com/vmware-tanzu/velero/pkg/util/results"
-
-	"github.com/stretchr/testify/assert"
-
 	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
 	"github.com/vmware-tanzu/velero/pkg/builder"
+	"github.com/vmware-tanzu/velero/pkg/util/results"
 )

 func TestDescribeBackupInSF(t *testing.T) {
@@ -314,6 +311,7 @@ func TestDescribeNativeSnapshotsInSF(t *testing.T) {
 				{
 					BackupMethod: volume.NativeSnapshot,
 					PVName:       "pv-1",
+					Result:       volume.VolumeResultSucceeded,
 					NativeSnapshotInfo: &volume.NativeSnapshotInfo{
 						SnapshotHandle: "snapshot-1",
 						VolumeType:     "ebs",
@@ -330,6 +328,7 @@ func TestDescribeNativeSnapshotsInSF(t *testing.T) {
 						"type":             "ebs",
 						"availabilityZone": "us-east-2",
 						"IOPS":             "1000 mbps",
+						"result":           "succeeded",
 					},
 				},
 			},
@@ -401,6 +400,7 @@ func TestDescribeCSISnapshotsInSF(t *testing.T) {
 					PVCNamespace:          "pvc-ns-2",
 					PVCName:               "pvc-2",
 					PreserveLocalSnapshot: true,
+					Result:                volume.VolumeResultSucceeded,
 					CSISnapshotInfo: &volume.CSISnapshotInfo{
 						SnapshotHandle: "snapshot-2",
 						Size:           1024,
@@ -420,6 +420,7 @@ func TestDescribeCSISnapshotsInSF(t *testing.T) {
 							"storageSnapshotID":   "snapshot-2",
 							"snapshotSize(bytes)": int64(1024),
 							"csiDriver":           "fake-driver",
+							"result":              "succeeded",
 						},
 					},
 				},
@@ -457,6 +458,7 @@ func TestDescribeCSISnapshotsInSF(t *testing.T) {
 					PVCNamespace:      "pvc-ns-4",
 					PVCName:           "pvc-4",
 					SnapshotDataMoved: true,
+					Result:            volume.VolumeResultSucceeded,
 					SnapshotDataMovementInfo: &volume.SnapshotDataMovementInfo{
 						DataMover:      "velero",
 						UploaderType:   "fake-uploader",
@@ -473,6 +475,7 @@ func TestDescribeCSISnapshotsInSF(t *testing.T) {
 							"operationID":  "fake-operation-4",
 							"dataMover":    "velero",
 							"uploaderType": "fake-uploader",
+							"result":       "succeeded",
 						},
 					},
 				},
@@ -484,6 +487,7 @@ func TestDescribeCSISnapshotsInSF(t *testing.T) {
 				{
 					BackupMethod:      volume.CSISnapshot,
 					PVCNamespace:      "pvc-ns-4",
+					Result:            volume.VolumeResultFailed,
 					PVCName:           "pvc-4",
 					SnapshotDataMoved: true,
 					SnapshotDataMovementInfo: &volume.SnapshotDataMovementInfo{
@@ -501,6 +505,7 @@ func TestDescribeCSISnapshotsInSF(t *testing.T) {
 							"operationID":  "fake-operation-4",
 							"dataMover":    "velero",
 							"uploaderType": "fake-uploader",
+							"result":       "failed",
 						},
 					},
 				},
@@ -21,7 +21,6 @@ import (
 	"context"
 	"fmt"
 	"os"
-	"strings"
 	"time"

 	snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1"
@@ -464,7 +463,7 @@ func (b *backupReconciler) prepareBackupRequest(backup *velerov1api.Backup, logg
 	}

 	// validate the included/excluded namespaces
-	for _, err := range b.validateNamespaceIncludesExcludes(request.Spec.IncludedNamespaces, request.Spec.ExcludedNamespaces) {
+	for _, err := range collections.ValidateNamespaceIncludesExcludes(request.Spec.IncludedNamespaces, request.Spec.ExcludedNamespaces) {
 		request.Status.ValidationErrors = append(request.Status.ValidationErrors, fmt.Sprintf("Invalid included/excluded namespace lists: %v", err))
 	}

@@ -473,20 +472,11 @@ func (b *backupReconciler) prepareBackupRequest(backup *velerov1api.Backup, logg
 		request.Status.ValidationErrors = append(request.Status.ValidationErrors, "encountered labelSelector as well as orLabelSelectors in backup spec, only one can be specified")
 	}

-	if request.Spec.ResourcePolicy != nil && strings.EqualFold(request.Spec.ResourcePolicy.Kind, resourcepolicies.ConfigmapRefType) {
-		policiesConfigmap := &corev1api.ConfigMap{}
-		err := b.kbClient.Get(context.Background(), kbclient.ObjectKey{Namespace: request.Namespace, Name: request.Spec.ResourcePolicy.Name}, policiesConfigmap)
-		if err != nil {
-			request.Status.ValidationErrors = append(request.Status.ValidationErrors, fmt.Sprintf("failed to get resource policies %s/%s configmap with err %v", request.Namespace, request.Spec.ResourcePolicy.Name, err))
-		}
-		res, err := resourcepolicies.GetResourcePoliciesFromConfig(policiesConfigmap)
-		if err != nil {
-			request.Status.ValidationErrors = append(request.Status.ValidationErrors, errors.Wrapf(err, fmt.Sprintf("resource policies %s/%s", request.Namespace, request.Spec.ResourcePolicy.Name)).Error())
-		} else if err = res.Validate(); err != nil {
-			request.Status.ValidationErrors = append(request.Status.ValidationErrors, errors.Wrapf(err, fmt.Sprintf("resource policies %s/%s", request.Namespace, request.Spec.ResourcePolicy.Name)).Error())
-		}
-		request.ResPolicies = res
+	resourcePolicies, err := resourcepolicies.GetResourcePoliciesFromBackup(*request.Backup, b.kbClient, logger)
+	if err != nil {
+		request.Status.ValidationErrors = append(request.Status.ValidationErrors, err.Error())
 	}
+	request.ResPolicies = resourcePolicies

 	return request
 }
@@ -596,24 +586,6 @@ func (b *backupReconciler) validateAndGetSnapshotLocations(backup *velerov1api.B
 	return providerLocations, nil
 }

-func (b *backupReconciler) validateNamespaceIncludesExcludes(includedNamespaces, excludedNamespaces []string) []error {
-	var errs []error
-	if errs = collections.ValidateNamespaceIncludesExcludes(includedNamespaces, excludedNamespaces); len(errs) > 0 {
-		return errs
-	}
-
-	namespace := &corev1api.Namespace{}
-	for _, name := range collections.NewIncludesExcludes().Includes(includedNamespaces...).GetIncludes() {
-		if name == "" || name == "*" {
-			continue
-		}
-		if err := b.kbClient.Get(context.Background(), kbclient.ObjectKey{Name: name}, namespace); err != nil {
-			errs = append(errs, err)
-		}
-	}
-	return errs
-}
-
 // runBackup runs and uploads a validated backup. Any error returned from this function
 // causes the backup to be Failed; if no error is returned, the backup's status's Errors
 // field is checked to see if the backup was a partial failure.
@@ -85,6 +85,7 @@ func (b *fakeBackupper) FinalizeBackup(
 	outBackupFile io.Writer,
 	backupItemActionResolver framework.BackupItemActionResolverV2,
 	asyncBIAOperations []*itemoperation.BackupOperation,
+	backupStore persistence.BackupStore,
 ) error {
 	args := b.Called(logger, backup, inBackupFile, outBackupFile, backupItemActionResolver, asyncBIAOperations)
 	return args.Error(0)
@@ -190,16 +191,10 @@ func TestProcessBackupValidationFailures(t *testing.T) {
 		},
 		{
 			name:           "use old filter parameters and new filter parameters together",
-			backup:         defaultBackup().IncludeClusterResources(true).IncludedNamespaceScopedResources("Deployment").IncludedNamespaces("foo").Result(),
+			backup:         defaultBackup().IncludeClusterResources(true).IncludedNamespaceScopedResources("Deployment").IncludedNamespaces("default").Result(),
 			backupLocation: defaultBackupLocation,
 			expectedErrs:   []string{"include-resources, exclude-resources and include-cluster-resources are old filter parameters.\ninclude-cluster-scoped-resources, exclude-cluster-scoped-resources, include-namespace-scoped-resources and exclude-namespace-scoped-resources are new filter parameters.\nThey cannot be used together"},
 		},
-		{
-			name:           "nonexisting namespace",
-			backup:         defaultBackup().IncludedNamespaces("non-existing").Result(),
-			backupLocation: defaultBackupLocation,
-			expectedErrs:   []string{"Invalid included/excluded namespace lists: namespaces \"non-existing\" not found"},
-		},
 	}

 	for _, test := range tests {
@@ -214,11 +209,10 @@ func TestProcessBackupValidationFailures(t *testing.T) {
 			require.NoError(t, err)

 			var fakeClient kbclient.Client
-			namespace := builder.ForNamespace("foo").Result()
 			if test.backupLocation != nil {
-				fakeClient = velerotest.NewFakeControllerRuntimeClient(t, test.backupLocation, namespace)
+				fakeClient = velerotest.NewFakeControllerRuntimeClient(t, test.backupLocation)
 			} else {
-				fakeClient = velerotest.NewFakeControllerRuntimeClient(t, namespace)
+				fakeClient = velerotest.NewFakeControllerRuntimeClient(t)
 			}

 			c := &backupReconciler{
@@ -1574,43 +1568,6 @@ func TestValidateAndGetSnapshotLocations(t *testing.T) {
 	}
 }

-func TestValidateNamespaceIncludesExcludes(t *testing.T) {
-	namespace := builder.ForNamespace("default").Result()
-	reconciler := &backupReconciler{
-		kbClient: velerotest.NewFakeControllerRuntimeClient(t, namespace),
-	}
-
-	// empty string as includedNamespaces
-	includedNamespaces := []string{""}
-	excludedNamespaces := []string{"test"}
-	errs := reconciler.validateNamespaceIncludesExcludes(includedNamespaces, excludedNamespaces)
-	assert.Empty(t, errs)
-
-	// "*" as includedNamespaces
-	includedNamespaces = []string{"*"}
-	excludedNamespaces = []string{"test"}
-	errs = reconciler.validateNamespaceIncludesExcludes(includedNamespaces, excludedNamespaces)
-	assert.Empty(t, errs)
-
-	// invalid namespaces
-	includedNamespaces = []string{"1@#"}
-	excludedNamespaces = []string{"2@#"}
-	errs = reconciler.validateNamespaceIncludesExcludes(includedNamespaces, excludedNamespaces)
-	assert.Len(t, errs, 2)
-
-	// not exist namespaces
-	includedNamespaces = []string{"non-existing-namespace"}
-	excludedNamespaces = []string{}
-	errs = reconciler.validateNamespaceIncludesExcludes(includedNamespaces, excludedNamespaces)
-	assert.Len(t, errs, 1)
-
-	// valid namespaces
-	includedNamespaces = []string{"default"}
-	excludedNamespaces = []string{}
-	errs = reconciler.validateNamespaceIncludesExcludes(includedNamespaces, excludedNamespaces)
-	assert.Empty(t, errs)
-}
-
 // Test_getLastSuccessBySchedule verifies that the getLastSuccessBySchedule helper function correctly returns
 // the completion timestamp of the most recent completed backup for each schedule, including an entry for ad-hoc
 // or non-scheduled backups.
@@ -184,6 +184,7 @@ func (r *backupFinalizerReconciler) Reconcile(ctx context.Context, req ctrl.Requ
 			outBackupFile,
 			backupItemActionsResolver,
 			operations,
+			backupStore,
 		)
 		if err != nil {
 			log.WithError(err).Error("error finalizing Backup")
@@ -225,7 +225,7 @@ func TestBackupFinalizerReconcile(t *testing.T) {
 			backupStore.On("GetBackupVolumeInfos", mock.Anything).Return(nil, nil)
 			backupStore.On("PutBackupVolumeInfos", mock.Anything, mock.Anything).Return(nil)
 			pluginManager.On("GetBackupItemActionsV2").Return(nil, nil)
-			backupper.On("FinalizeBackup", mock.Anything, mock.Anything, mock.Anything, mock.Anything, framework.BackupItemActionResolverV2{}, mock.Anything).Return(nil)
+			backupper.On("FinalizeBackup", mock.Anything, mock.Anything, mock.Anything, mock.Anything, framework.BackupItemActionResolverV2{}, mock.Anything, mock.Anything).Return(nil)
 			_, err := reconciler.Reconcile(context.TODO(), ctrl.Request{NamespacedName: types.NamespacedName{Namespace: test.backup.Namespace, Name: test.backup.Name}})
 			gotErr := err != nil
 			assert.Equal(t, test.expectError, gotErr)
@@ -40,6 +40,7 @@ import (
 	"github.com/vmware-tanzu/velero/pkg/plugin/clientmgmt"
 	"github.com/vmware-tanzu/velero/pkg/util/kube"

+	corev1api "k8s.io/api/core/v1"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 )
@@ -272,6 +273,20 @@ func (b *backupSyncReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 			for _, snapCont := range snapConts {
 				// TODO: Reset ResourceVersion prior to persisting VolumeSnapshotContents
 				snapCont.ResourceVersion = ""
+				// Make the VolumeSnapshotContent static
+				snapCont.Spec.Source = snapshotv1api.VolumeSnapshotContentSource{
+					SnapshotHandle: snapCont.Status.SnapshotHandle,
+				}
+				// Set VolumeSnapshotRef to none exist one, because VolumeSnapshotContent
+				// validation webhook will check whether name and namespace are nil.
+				// external-snapshotter needs Source pointing to snapshot and VolumeSnapshot
+				// reference's UID to nil to determine the VolumeSnapshotContent is deletable.
+				snapCont.Spec.VolumeSnapshotRef = corev1api.ObjectReference{
+					APIVersion: snapshotv1api.SchemeGroupVersion.String(),
+					Kind:       "VolumeSnapshot",
+					Namespace:  "ns-" + string(snapCont.UID),
+					Name:       "name-" + string(snapCont.UID),
+				}
 				err := b.client.Create(ctx, snapCont, &client.CreateOptions{})
 				switch {
 				case err != nil && apierrors.IsAlreadyExists(err):
@@ -334,10 +334,19 @@ func (r *DataDownloadReconciler) runCancelableDataPath(ctx context.Context, fsRe
 	}

 	log.WithField("path", path.ByPath).Debug("Found host path")
-	if err := fsRestore.Init(ctx, dd.Spec.BackupStorageLocation, dd.Spec.SourceNamespace, datamover.GetUploaderType(dd.Spec.DataMover),
-		velerov1api.BackupRepositoryTypeKopia, "", r.repositoryEnsurer, r.credentialGetter); err != nil {
+
+	if err := fsRestore.Init(ctx, &datapath.FSBRInitParam{
+		BSLName:           dd.Spec.BackupStorageLocation,
+		SourceNamespace:   dd.Spec.SourceNamespace,
+		UploaderType:      datamover.GetUploaderType(dd.Spec.DataMover),
+		RepositoryType:    velerov1api.BackupRepositoryTypeKopia,
+		RepoIdentifier:    "",
+		RepositoryEnsurer: r.repositoryEnsurer,
+		CredentialGetter:  r.credentialGetter,
+	}); err != nil {
 		return r.errorOut(ctx, dd, err, "error to initialize data path", log)
 	}
+
 	log.WithField("path", path.ByPath).Info("fs init")

 	if err := fsRestore.StartRestore(dd.Spec.SnapshotID, path, dd.Spec.DataMoverConfig); err != nil {
@@ -338,23 +338,38 @@ func (r *DataUploadReconciler) Reconcile(ctx context.Context, req ctrl.Request)

 func (r *DataUploadReconciler) runCancelableDataUpload(ctx context.Context, fsBackup datapath.AsyncBR, du *velerov2alpha1api.DataUpload, res *exposer.ExposeResult, log logrus.FieldLogger) (reconcile.Result, error) {
 	log.Info("Run cancelable dataUpload")
+
 	path, err := exposer.GetPodVolumeHostPath(ctx, res.ByPod.HostingPod, res.ByPod.VolumeName, r.client, r.fileSystem, log)
 	if err != nil {
 		return r.errorOut(ctx, du, err, "error exposing host path for pod volume", log)
 	}

 	log.WithField("path", path.ByPath).Debug("Found host path")
-	if err := fsBackup.Init(ctx, du.Spec.BackupStorageLocation, du.Spec.SourceNamespace, datamover.GetUploaderType(du.Spec.DataMover),
-		velerov1api.BackupRepositoryTypeKopia, "", r.repoEnsurer, r.credentialGetter); err != nil {
+
+	if err := fsBackup.Init(ctx, &datapath.FSBRInitParam{
+		BSLName:           du.Spec.BackupStorageLocation,
+		SourceNamespace:   du.Spec.SourceNamespace,
+		UploaderType:      datamover.GetUploaderType(du.Spec.DataMover),
+		RepositoryType:    velerov1api.BackupRepositoryTypeKopia,
+		RepoIdentifier:    "",
+		RepositoryEnsurer: r.repoEnsurer,
+		CredentialGetter:  r.credentialGetter,
+	}); err != nil {
 		return r.errorOut(ctx, du, err, "error to initialize data path", log)
 	}
+
 	log.WithField("path", path.ByPath).Info("fs init")

 	tags := map[string]string{
 		velerov1api.AsyncOperationIDLabel: du.Labels[velerov1api.AsyncOperationIDLabel],
 	}

-	if err := fsBackup.StartBackup(path, datamover.GetRealSource(du.Spec.SourceNamespace, du.Spec.SourcePVC), "", false, tags, du.Spec.DataMoverConfig); err != nil {
+	if err := fsBackup.StartBackup(path, du.Spec.DataMoverConfig, &datapath.FSBRStartParam{
+		RealSource:     datamover.GetRealSource(du.Spec.SourceNamespace, du.Spec.SourcePVC),
+		ParentSnapshot: "",
+		ForceFull:      false,
+		Tags:           tags,
+	}); err != nil {
 		return r.errorOut(ctx, du, err, "error starting data path backup", log)
 	}

@@ -51,7 +51,6 @@ import (
 	"github.com/vmware-tanzu/velero/pkg/datapath"
 	"github.com/vmware-tanzu/velero/pkg/exposer"
 	"github.com/vmware-tanzu/velero/pkg/metrics"
-	"github.com/vmware-tanzu/velero/pkg/repository"
 	velerotest "github.com/vmware-tanzu/velero/pkg/test"
 	"github.com/vmware-tanzu/velero/pkg/uploader"
 	"github.com/vmware-tanzu/velero/pkg/util/boolptr"
@@ -307,11 +306,11 @@ type fakeDataUploadFSBR struct {
 	clock      clock.WithTickerAndDelayedExecution
 }

-func (f *fakeDataUploadFSBR) Init(ctx context.Context, bslName string, sourceNamespace string, uploaderType string, repositoryType string, repoIdentifier string, repositoryEnsurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter) error {
+func (f *fakeDataUploadFSBR) Init(ctx context.Context, param interface{}) error {
 	return nil
 }

-func (f *fakeDataUploadFSBR) StartBackup(source datapath.AccessPoint, realSource string, parentSnapshot string, forceFull bool, tags map[string]string, uploaderConfigs map[string]string) error {
+func (f *fakeDataUploadFSBR) StartBackup(source datapath.AccessPoint, uploaderConfigs map[string]string, param interface{}) error {
 	du := f.du
 	original := f.du.DeepCopy()
 	du.Status.Phase = velerov2alpha1api.DataUploadPhaseCompleted
@@ -159,8 +159,15 @@ func (r *PodVolumeBackupReconciler) Reconcile(ctx context.Context, req ctrl.Requ

 	log.WithField("path", path.ByPath).Debugf("Found host path")

-	if err := fsBackup.Init(ctx, pvb.Spec.BackupStorageLocation, pvb.Spec.Pod.Namespace, pvb.Spec.UploaderType,
-		podvolume.GetPvbRepositoryType(&pvb), pvb.Spec.RepoIdentifier, r.repositoryEnsurer, r.credentialGetter); err != nil {
+	if err := fsBackup.Init(ctx, &datapath.FSBRInitParam{
+		BSLName:           pvb.Spec.BackupStorageLocation,
+		SourceNamespace:   pvb.Spec.Pod.Namespace,
+		UploaderType:      pvb.Spec.UploaderType,
+		RepositoryType:    podvolume.GetPvbRepositoryType(&pvb),
+		RepoIdentifier:    pvb.Spec.RepoIdentifier,
+		RepositoryEnsurer: r.repositoryEnsurer,
+		CredentialGetter:  r.credentialGetter,
+	}); err != nil {
 		return r.errorOut(ctx, &pvb, err, "error to initialize data path", log)
 	}

@@ -179,7 +186,12 @@ func (r *PodVolumeBackupReconciler) Reconcile(ctx context.Context, req ctrl.Requ
 		}
 	}

-	if err := fsBackup.StartBackup(path, "", parentSnapshotID, false, pvb.Spec.Tags, pvb.Spec.UploaderSettings); err != nil {
+	if err := fsBackup.StartBackup(path, pvb.Spec.UploaderSettings, &datapath.FSBRStartParam{
+		RealSource:     "",
+		ParentSnapshot: parentSnapshotID,
+		ForceFull:      false,
+		Tags:           pvb.Spec.Tags,
+	}); err != nil {
 		return r.errorOut(ctx, &pvb, err, "error starting data path backup", log)
 	}

@@ -41,7 +41,6 @@ import (
 	"github.com/vmware-tanzu/velero/pkg/builder"
 	"github.com/vmware-tanzu/velero/pkg/datapath"
 	"github.com/vmware-tanzu/velero/pkg/metrics"
-	"github.com/vmware-tanzu/velero/pkg/repository"
 	velerotest "github.com/vmware-tanzu/velero/pkg/test"
 )

@@ -99,11 +98,11 @@ type fakeFSBR struct {
 	clock  clock.WithTickerAndDelayedExecution
 }

-func (b *fakeFSBR) Init(ctx context.Context, bslName string, sourceNamespace string, uploaderType string, repositoryType string, repoIdentifier string, repositoryEnsurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter) error {
+func (b *fakeFSBR) Init(ctx context.Context, param interface{}) error {
 	return nil
 }

-func (b *fakeFSBR) StartBackup(source datapath.AccessPoint, realSource string, parentSnapshot string, forceFull bool, tags map[string]string, uploaderConfigs map[string]string) error {
+func (b *fakeFSBR) StartBackup(source datapath.AccessPoint, uploaderConfigs map[string]string, param interface{}) error {
 	pvb := b.pvb

 	original := b.pvb.DeepCopy()
@@ -141,8 +141,15 @@ func (c *PodVolumeRestoreReconciler) Reconcile(ctx context.Context, req ctrl.Req

 	log.WithField("path", volumePath.ByPath).Debugf("Found host path")

-	if err := fsRestore.Init(ctx, pvr.Spec.BackupStorageLocation, pvr.Spec.SourceNamespace, pvr.Spec.UploaderType,
-		podvolume.GetPvrRepositoryType(pvr), pvr.Spec.RepoIdentifier, c.repositoryEnsurer, c.credentialGetter); err != nil {
+	if err := fsRestore.Init(ctx, &datapath.FSBRInitParam{
+		BSLName:           pvr.Spec.BackupStorageLocation,
+		SourceNamespace:   pvr.Spec.SourceNamespace,
+		UploaderType:      pvr.Spec.UploaderType,
+		RepositoryType:    podvolume.GetPvrRepositoryType(pvr),
+		RepoIdentifier:    pvr.Spec.RepoIdentifier,
+		RepositoryEnsurer: c.repositoryEnsurer,
+		CredentialGetter:  c.credentialGetter,
+	}); err != nil {
 		return c.errorOut(ctx, pvr, err, "error to initialize data path", log)
 	}

@@ -22,6 +22,8 @@ import (
 	"sync"
 	"time"

+	storagev1api "k8s.io/api/storage/v1"
+
 	"github.com/pkg/errors"
 	"github.com/sirupsen/logrus"
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
@@ -304,6 +306,27 @@ func (ctx *finalizerContext) patchDynamicPVWithVolumeInfo() (errs results.Result
 						return false, err
 					}

+					// We are handling a common but specific scenario where a PVC is in a pending state and uses a storage class with
+					// VolumeBindingMode set to WaitForFirstConsumer. In this case, the PV patch step is skipped to avoid
+					// failures due to the PVC not being bound, which could cause a timeout and result in a failed restore.
+					if pvc != nil && pvc.Status.Phase == v1.ClaimPending {
+						// check if storage class used has VolumeBindingMode as WaitForFirstConsumer
+						scName := *pvc.Spec.StorageClassName
+						sc := &storagev1api.StorageClass{}
+						err = ctx.crClient.Get(context.Background(), client.ObjectKey{Name: scName}, sc)
+
+						if err != nil {
+							errs.Add(restoredNamespace, err)
+							return false, err
+						}
+						// skip PV patch step for this scenario
+						// because pvc would not be bound and the PV patch step would fail due to timeout thus failing the restore
+						if *sc.VolumeBindingMode == storagev1api.VolumeBindingWaitForFirstConsumer {
+							log.Warnf("skipping PV patch to restore custom reclaim policy, if any: StorageClass %s used by PVC %s has VolumeBindingMode set to WaitForFirstConsumer, and the PVC is also in a pending state", scName, pvc.Name)
+							return true, nil
+						}
+					}
+
 					if pvc.Status.Phase != v1.ClaimBound || pvc.Spec.VolumeName == "" {
 						log.Debugf("PVC: %s not ready", pvc.Name)
 						return false, nil
@@ -33,6 +33,26 @@ import (
 	"github.com/vmware-tanzu/velero/pkg/util/filesystem"
 )

+// FSBRInitParam define the input param for FSBR init
+type FSBRInitParam struct {
+	BSLName           string
+	SourceNamespace   string
+	UploaderType      string
+	RepositoryType    string
+	RepoIdentifier    string
+	RepositoryEnsurer *repository.Ensurer
+	CredentialGetter  *credentials.CredentialGetter
+	Filesystem        filesystem.Interface
+}
+
+// FSBRStartParam define the input param for FSBR start
+type FSBRStartParam struct {
+	RealSource     string
+	ParentSnapshot string
+	ForceFull      bool
+	Tags           map[string]string
+}
+
 type fileSystemBR struct {
 	ctx            context.Context
 	cancel         context.CancelFunc
@@ -61,8 +81,9 @@ func newFileSystemBR(jobName string, requestorType string, client client.Client,
 	return fs
 }

-func (fs *fileSystemBR) Init(ctx context.Context, bslName string, sourceNamespace string, uploaderType string, repositoryType string,
-	repoIdentifier string, repositoryEnsurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter) error {
+func (fs *fileSystemBR) Init(ctx context.Context, param interface{}) error {
+	initParam := param.(*FSBRInitParam)
+
 	var err error
 	defer func() {
 		if err != nil {
@@ -75,27 +96,27 @@ func (fs *fileSystemBR) Init(ctx context.Context, bslName string, sourceNamespac
 	backupLocation := &velerov1api.BackupStorageLocation{}
 	if err = fs.client.Get(ctx, client.ObjectKey{
 		Namespace: fs.namespace,
-		Name:      bslName,
+		Name:      initParam.BSLName,
 	}, backupLocation); err != nil {
-		return errors.Wrapf(err, "error getting backup storage location %s", bslName)
+		return errors.Wrapf(err, "error getting backup storage location %s", initParam.BSLName)
 	}

 	fs.backupLocation = backupLocation

-	fs.backupRepo, err = repositoryEnsurer.EnsureRepo(ctx, fs.namespace, sourceNamespace, bslName, repositoryType)
+	fs.backupRepo, err = initParam.RepositoryEnsurer.EnsureRepo(ctx, fs.namespace, initParam.SourceNamespace, initParam.BSLName, initParam.RepositoryType)
 	if err != nil {
-		return errors.Wrapf(err, "error to ensure backup repository %s-%s-%s", bslName, sourceNamespace, repositoryType)
+		return errors.Wrapf(err, "error to ensure backup repository %s-%s-%s", initParam.BSLName, initParam.SourceNamespace, initParam.RepositoryType)
 	}

-	err = fs.boostRepoConnect(ctx, repositoryType, credentialGetter)
+	err = fs.boostRepoConnect(ctx, initParam.RepositoryType, initParam.CredentialGetter)
 	if err != nil {
-		return errors.Wrapf(err, "error to boost backup repository connection %s-%s-%s", bslName, sourceNamespace, repositoryType)
+		return errors.Wrapf(err, "error to boost backup repository connection %s-%s-%s", initParam.BSLName, initParam.SourceNamespace, initParam.RepositoryType)
 	}

-	fs.uploaderProv, err = provider.NewUploaderProvider(ctx, fs.client, uploaderType, fs.requestorType, repoIdentifier,
-		fs.backupLocation, fs.backupRepo, credentialGetter, repokey.RepoKeySelector(), fs.log)
+	fs.uploaderProv, err = provider.NewUploaderProvider(ctx, fs.client, initParam.UploaderType, fs.requestorType, initParam.RepoIdentifier,
+		fs.backupLocation, fs.backupRepo, initParam.CredentialGetter, repokey.RepoKeySelector(), fs.log)
 	if err != nil {
-		return errors.Wrapf(err, "error creating uploader %s", uploaderType)
+		return errors.Wrapf(err, "error creating uploader %s", initParam.UploaderType)
 	}

 	fs.initialized = true
@@ -103,10 +124,10 @@ func (fs *fileSystemBR) Init(ctx context.Context, bslName string, sourceNamespac
 	fs.log.WithFields(
 		logrus.Fields{
 			"jobName":          fs.jobName,
-			"bsl":              bslName,
-			"source namespace": sourceNamespace,
-			"uploader":         uploaderType,
-			"repository":       repositoryType,
+			"bsl":              initParam.BSLName,
+			"source namespace": initParam.SourceNamespace,
+			"uploader":         initParam.UploaderType,
+			"repository":       initParam.RepositoryType,
 		}).Info("FileSystemBR is initialized")

 	return nil
@@ -129,14 +150,16 @@ func (fs *fileSystemBR) Close(ctx context.Context) {
 	fs.log.WithField("user", fs.jobName).Info("FileSystemBR is closed")
 }

-func (fs *fileSystemBR) StartBackup(source AccessPoint, realSource string, parentSnapshot string, forceFull bool, tags map[string]string, uploaderConfig map[string]string) error {
+func (fs *fileSystemBR) StartBackup(source AccessPoint, uploaderConfig map[string]string, param interface{}) error {
 	if !fs.initialized {
 		return errors.New("file system data path is not initialized")
 	}

+	backupParam := param.(*FSBRStartParam)
+
 	go func() {
-		snapshotID, emptySnapshot, err := fs.uploaderProv.RunBackup(fs.ctx, source.ByPath, realSource, tags, forceFull,
-			parentSnapshot, source.VolMode, uploaderConfig, fs)
+		snapshotID, emptySnapshot, err := fs.uploaderProv.RunBackup(fs.ctx, source.ByPath, backupParam.RealSource, backupParam.Tags, backupParam.ForceFull,
+			backupParam.ParentSnapshot, source.VolMode, uploaderConfig, fs)

 		if err == provider.ErrorCanceled {
 			fs.callbacks.OnCancelled(context.Background(), fs.namespace, fs.jobName)
@@ -192,7 +215,7 @@ func (fs *fileSystemBR) Cancel() {

 func (fs *fileSystemBR) boostRepoConnect(ctx context.Context, repositoryType string, credentialGetter *credentials.CredentialGetter) error {
 	if repositoryType == velerov1api.BackupRepositoryTypeKopia {
-		if err := repoProvider.NewUnifiedRepoProvider(*credentialGetter, repositoryType, fs.client, fs.log).BoostRepoConnect(ctx, repoProvider.RepoParam{BackupLocation: fs.backupLocation, BackupRepo: fs.backupRepo}); err != nil {
+		if err := repoProvider.NewUnifiedRepoProvider(*credentialGetter, repositoryType, fs.log).BoostRepoConnect(ctx, repoProvider.RepoParam{BackupLocation: fs.backupLocation, BackupRepo: fs.backupRepo}); err != nil {
 			return err
 		}
 	} else {
@@ -100,13 +100,13 @@ func TestAsyncBackup(t *testing.T) {
 			fs.initialized = true
 			fs.callbacks = test.callbacks

-			err := fs.StartBackup(AccessPoint{ByPath: test.path}, "", "", false, nil, map[string]string{})
+			err := fs.StartBackup(AccessPoint{ByPath: test.path}, map[string]string{}, &FSBRStartParam{})
 			require.NoError(t, err)

 			<-finish

-			assert.Equal(t, asyncErr, test.err)
-			assert.Equal(t, asyncResult, test.result)
+			assert.Equal(t, test.err, asyncErr)
+			assert.Equal(t, test.result, asyncResult)
 		})
 	}

@@ -47,7 +47,7 @@ func (m *Manager) CreateFileSystemBR(jobName string, requestorType string, ctx c
 	m.trackerLock.Lock()
 	defer m.trackerLock.Unlock()

-	if len(m.tracker) == m.cocurrentNum {
+	if len(m.tracker) >= m.cocurrentNum {
 		return nil, ConcurrentLimitExceed
 	}

@@ -0,0 +1,93 @@
+// Code generated by mockery v2.39.1. DO NOT EDIT.
+
+package mocks
+
+import (
+	context "context"
+
+	mock "github.com/stretchr/testify/mock"
+	datapath "github.com/vmware-tanzu/velero/pkg/datapath"
+)
+
+// AsyncBR is an autogenerated mock type for the AsyncBR type
+type AsyncBR struct {
+	mock.Mock
+}
+
+// Cancel provides a mock function with given fields:
+func (_m *AsyncBR) Cancel() {
+	_m.Called()
+}
+
+// Close provides a mock function with given fields: ctx
+func (_m *AsyncBR) Close(ctx context.Context) {
+	_m.Called(ctx)
+}
+
+// Init provides a mock function with given fields: ctx, param
+func (_m *AsyncBR) Init(ctx context.Context, param interface{}) error {
+	ret := _m.Called(ctx, param)
+
+	if len(ret) == 0 {
+		panic("no return value specified for Init")
+	}
+
+	var r0 error
+	if rf, ok := ret.Get(0).(func(context.Context, interface{}) error); ok {
+		r0 = rf(ctx, param)
+	} else {
+		r0 = ret.Error(0)
+	}
+
+	return r0
+}
+
+// StartBackup provides a mock function with given fields: source, dataMoverConfig, param
+func (_m *AsyncBR) StartBackup(source datapath.AccessPoint, dataMoverConfig map[string]string, param interface{}) error {
+	ret := _m.Called(source, dataMoverConfig, param)
+
+	if len(ret) == 0 {
+		panic("no return value specified for StartBackup")
+	}
+
+	var r0 error
+	if rf, ok := ret.Get(0).(func(datapath.AccessPoint, map[string]string, interface{}) error); ok {
+		r0 = rf(source, dataMoverConfig, param)
+	} else {
+		r0 = ret.Error(0)
+	}
+
+	return r0
+}
+
+// StartRestore provides a mock function with given fields: snapshotID, target, dataMoverConfig
+func (_m *AsyncBR) StartRestore(snapshotID string, target datapath.AccessPoint, dataMoverConfig map[string]string) error {
+	ret := _m.Called(snapshotID, target, dataMoverConfig)
+
+	if len(ret) == 0 {
+		panic("no return value specified for StartRestore")
+	}
+
+	var r0 error
+	if rf, ok := ret.Get(0).(func(string, datapath.AccessPoint, map[string]string) error); ok {
+		r0 = rf(snapshotID, target, dataMoverConfig)
+	} else {
+		r0 = ret.Error(0)
+	}
+
+	return r0
+}
+
+// NewAsyncBR creates a new instance of AsyncBR. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
+// The first argument is typically a *testing.T value.
+func NewAsyncBR(t interface {
+	mock.TestingT
+	Cleanup(func())
+}) *AsyncBR {
+	mock := &AsyncBR{}
+	mock.Mock.Test(t)
+
+	t.Cleanup(func() { mock.AssertExpectations(t) })
+
+	return mock
+}
@@ -1,86 +0,0 @@
-// Code generated by mockery v2.20.0. DO NOT EDIT.
-
-package mocks
-
-import (
-	context "context"
-
-	credentials "github.com/vmware-tanzu/velero/internal/credentials"
-	datapath "github.com/vmware-tanzu/velero/pkg/datapath"
-
-	mock "github.com/stretchr/testify/mock"
-
-	repository "github.com/vmware-tanzu/velero/pkg/repository"
-)
-
-// AsyncBR is an autogenerated mock type for the AsyncBR type
-type AsyncBR struct {
-	mock.Mock
-}
-
-// Cancel provides a mock function with given fields:
-func (_m *AsyncBR) Cancel() {
-	_m.Called()
-}
-
-// Close provides a mock function with given fields: ctx
-func (_m *AsyncBR) Close(ctx context.Context) {
-	_m.Called(ctx)
-}
-
-// Init provides a mock function with given fields: ctx, bslName, sourceNamespace, uploaderType, repositoryType, repoIdentifier, repositoryEnsurer, credentialGetter
-func (_m *AsyncBR) Init(ctx context.Context, bslName string, sourceNamespace string, uploaderType string, repositoryType string, repoIdentifier string, repositoryEnsurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter) error {
-	ret := _m.Called(ctx, bslName, sourceNamespace, uploaderType, repositoryType, repoIdentifier, repositoryEnsurer, credentialGetter)
-
-	var r0 error
-	if rf, ok := ret.Get(0).(func(context.Context, string, string, string, string, string, *repository.Ensurer, *credentials.CredentialGetter) error); ok {
-		r0 = rf(ctx, bslName, sourceNamespace, uploaderType, repositoryType, repoIdentifier, repositoryEnsurer, credentialGetter)
-	} else {
-		r0 = ret.Error(0)
-	}
-
-	return r0
-}
-
-// StartBackup provides a mock function with given fields: source, realSource, parentSnapshot, forceFull, tags, dataMoverConfig
-func (_m *AsyncBR) StartBackup(source datapath.AccessPoint, realSource string, parentSnapshot string, forceFull bool, tags map[string]string, dataMoverConfig map[string]string) error {
-	ret := _m.Called(source, realSource, parentSnapshot, forceFull, tags, dataMoverConfig)
-
-	var r0 error
-	if rf, ok := ret.Get(0).(func(datapath.AccessPoint, string, string, bool, map[string]string, map[string]string) error); ok {
-		r0 = rf(source, realSource, parentSnapshot, forceFull, tags, dataMoverConfig)
-	} else {
-		r0 = ret.Error(0)
-	}
-
-	return r0
-}
-
-// StartRestore provides a mock function with given fields: snapshotID, target, dataMoverConfig
-func (_m *AsyncBR) StartRestore(snapshotID string, target datapath.AccessPoint, dataMoverConfig map[string]string) error {
-	ret := _m.Called(snapshotID, target, dataMoverConfig)
-
-	var r0 error
-	if rf, ok := ret.Get(0).(func(string, datapath.AccessPoint, map[string]string) error); ok {
-		r0 = rf(snapshotID, target, dataMoverConfig)
-	} else {
-		r0 = ret.Error(0)
-	}
-
-	return r0
-}
-
-type mockConstructorTestingTNewAsyncBR interface {
-	mock.TestingT
-	Cleanup(func())
-}
-
-// NewAsyncBR creates a new instance of AsyncBR. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
-func NewAsyncBR(t mockConstructorTestingTNewAsyncBR) *AsyncBR {
-	mock := &AsyncBR{}
-	mock.Mock.Test(t)
-
-	t.Cleanup(func() { mock.AssertExpectations(t) })
-
-	return mock
-}
@@ -19,8 +19,6 @@ package datapath
 import (
 	"context"

-	"github.com/vmware-tanzu/velero/internal/credentials"
-	"github.com/vmware-tanzu/velero/pkg/repository"
 	"github.com/vmware-tanzu/velero/pkg/uploader"
 )

@@ -32,14 +30,14 @@ type Result struct {

 // BackupResult represents the result of a backup
 type BackupResult struct {
-	SnapshotID    string
-	EmptySnapshot bool
-	Source        AccessPoint
+	SnapshotID    string      `json:"snapshotID"`
+	EmptySnapshot bool        `json:"emptySnapshot"`
+	Source        AccessPoint `json:"source,omitempty"`
 }

 // RestoreResult represents the result of a restore
 type RestoreResult struct {
-	Target AccessPoint
+	Target AccessPoint `json:"target,omitempty"`
 }

 // Callbacks defines the collection of callbacks during backup/restore
@@ -59,10 +57,10 @@ type AccessPoint struct {
 // AsyncBR is the interface for asynchronous data path methods
 type AsyncBR interface {
 	// Init initializes an asynchronous data path instance
-	Init(ctx context.Context, bslName string, sourceNamespace string, uploaderType string, repositoryType string, repoIdentifier string, repositoryEnsurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter) error
+	Init(ctx context.Context, param interface{}) error

 	// StartBackup starts an asynchronous data path instance for backup
-	StartBackup(source AccessPoint, realSource string, parentSnapshot string, forceFull bool, tags map[string]string, dataMoverConfig map[string]string) error
+	StartBackup(source AccessPoint, dataMoverConfig map[string]string, param interface{}) error

 	// StartRestore starts an asynchronous data path instance for restore
 	StartRestore(snapshotID string, target AccessPoint, dataMoverConfig map[string]string) error
@@ -171,7 +171,7 @@ func (e *csiSnapshotExposer) Expose(ctx context.Context, ownerObject corev1.Obje
 	curLog.WithField("pvc name", backupPVC.Name).Info("Backup PVC is created")
 	defer func() {
 		if err != nil {
-			kube.DeletePVAndPVCIfAny(ctx, e.kubeClient.CoreV1(), backupPVC.Name, backupPVC.Namespace, curLog)
+			kube.DeletePVAndPVCIfAny(ctx, e.kubeClient.CoreV1(), backupPVC.Name, backupPVC.Namespace, 0, curLog)
 		}
 	}()

@@ -271,13 +271,15 @@ func (e *csiSnapshotExposer) PeekExposed(ctx context.Context, ownerObject corev1
 	return nil
 }

+const cleanUpTimeout = time.Minute
+
 func (e *csiSnapshotExposer) CleanUp(ctx context.Context, ownerObject corev1.ObjectReference, vsName string, sourceNamespace string) {
 	backupPodName := ownerObject.Name
 	backupPVCName := ownerObject.Name
 	backupVSName := ownerObject.Name

 	kube.DeletePodIfAny(ctx, e.kubeClient.CoreV1(), backupPodName, ownerObject.Namespace, e.log)
-	kube.DeletePVAndPVCIfAny(ctx, e.kubeClient.CoreV1(), backupPVCName, ownerObject.Namespace, e.log)
+	kube.DeletePVAndPVCIfAny(ctx, e.kubeClient.CoreV1(), backupPVCName, ownerObject.Namespace, cleanUpTimeout, e.log)

 	csi.DeleteVolumeSnapshotIfAny(ctx, e.csiSnapshotClient, backupVSName, ownerObject.Namespace, e.log)
 	csi.DeleteVolumeSnapshotIfAny(ctx, e.csiSnapshotClient, vsName, sourceNamespace, e.log)
@@ -652,14 +652,7 @@ func TestPeekExpose(t *testing.T) {
 			Name:      backup.Name,
 		},
 		Status: corev1.PodStatus{
-			Phase: corev1.PodPending,
-			Conditions: []corev1.PodCondition{
-				{
-					Type:    corev1.PodScheduled,
-					Reason:  "Unschedulable",
-					Message: "unrecoverable",
-				},
-			},
+			Phase: corev1.PodFailed,
 		},
 	}

@@ -689,7 +682,7 @@ func TestPeekExpose(t *testing.T) {
 			kubeClientObj: []runtime.Object{
 				backupPodUrecoverable,
 			},
-			err: "Pod is unschedulable: unrecoverable",
+			err: "Pod is in abnormal state Failed",
 		},
 		{
 			name:        "succeed",
@@ -109,7 +109,7 @@ func (e *genericRestoreExposer) Expose(ctx context.Context, ownerObject corev1.O

 	defer func() {
 		if err != nil {
-			kube.DeletePVAndPVCIfAny(ctx, e.kubeClient.CoreV1(), restorePVC.Name, restorePVC.Namespace, curLog)
+			kube.DeletePVAndPVCIfAny(ctx, e.kubeClient.CoreV1(), restorePVC.Name, restorePVC.Namespace, 0, curLog)
 		}
 	}()

@@ -200,7 +200,7 @@ func (e *genericRestoreExposer) CleanUp(ctx context.Context, ownerObject corev1.
 	restorePVCName := ownerObject.Name

 	kube.DeletePodIfAny(ctx, e.kubeClient.CoreV1(), restorePodName, ownerObject.Namespace, e.log)
-	kube.DeletePVAndPVCIfAny(ctx, e.kubeClient.CoreV1(), restorePVCName, ownerObject.Namespace, e.log)
+	kube.DeletePVAndPVCIfAny(ctx, e.kubeClient.CoreV1(), restorePVCName, ownerObject.Namespace, 0, e.log)
 }

 func (e *genericRestoreExposer) RebindVolume(ctx context.Context, ownerObject corev1.ObjectReference, targetPVCName string, sourceNamespace string, timeout time.Duration) error {
@@ -440,14 +440,7 @@ func TestRestorePeekExpose(t *testing.T) {
 			Name:      restore.Name,
 		},
 		Status: corev1api.PodStatus{
-			Phase: corev1api.PodPending,
-			Conditions: []corev1api.PodCondition{
-				{
-					Type:    corev1api.PodScheduled,
-					Reason:  "Unschedulable",
-					Message: "unrecoverable",
-				},
-			},
+			Phase: corev1api.PodFailed,
 		},
 	}

@@ -474,7 +467,7 @@ func TestRestorePeekExpose(t *testing.T) {
 			kubeClientObj: []runtime.Object{
 				restorePodUrecoverable,
 			},
-			err: "Pod is unschedulable: unrecoverable",
+			err: "Pod is in abnormal state Failed",
 		},
 		{
 			name:         "succeed",
@@ -0,0 +1,62 @@
+/*
+Copyright the Velero contributors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package volumehelper
+
+import (
+	"github.com/sirupsen/logrus"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+	crclient "sigs.k8s.io/controller-runtime/pkg/client"
+
+	"github.com/vmware-tanzu/velero/internal/resourcepolicies"
+	"github.com/vmware-tanzu/velero/internal/volumehelper"
+	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
+	"github.com/vmware-tanzu/velero/pkg/util/boolptr"
+)
+
+// ShouldPerformSnapshotWithBackup is used for third-party plugins.
+// It supports to check whether the PVC or PodVolume should be backed
+// up on demand. On the other hand, the volumeHelperImpl assume there
+// is a VolumeHelper instance initialized before calling the
+// ShouldPerformXXX functions.
+func ShouldPerformSnapshotWithBackup(
+	unstructured runtime.Unstructured,
+	groupResource schema.GroupResource,
+	backup velerov1api.Backup,
+	crClient crclient.Client,
+	logger logrus.FieldLogger,
+) (bool, error) {
+	resourcePolicies, err := resourcepolicies.GetResourcePoliciesFromBackup(
+		backup,
+		crClient,
+		logger,
+	)
+	if err != nil {
+		return false, err
+	}
+
+	volumeHelperImpl := volumehelper.NewVolumeHelperImpl(
+		resourcePolicies,
+		backup.Spec.SnapshotVolumes,
+		logger,
+		crClient,
+		boolptr.IsSetToTrue(backup.Spec.DefaultVolumesToFsBackup),
+		true,
+	)
+
+	return volumeHelperImpl.ShouldPerformSnapshot(unstructured, groupResource)
+}
@@ -128,7 +128,7 @@ func NewManager(
 	mgr.providers[velerov1api.BackupRepositoryTypeKopia] = provider.NewUnifiedRepoProvider(credentials.CredentialGetter{
 		FromFile:   credentialFileStore,
 		FromSecret: credentialSecretStore,
-	}, velerov1api.BackupRepositoryTypeKopia, client, mgr.log)
+	}, velerov1api.BackupRepositoryTypeKopia, mgr.log)

 	return mgr
 }
@@ -29,7 +29,6 @@ import (
 	"github.com/kopia/kopia/repo"
 	"github.com/pkg/errors"
 	"github.com/sirupsen/logrus"
-	"sigs.k8s.io/controller-runtime/pkg/client"

 	"github.com/vmware-tanzu/velero/internal/credentials"
 	velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
@@ -44,7 +43,6 @@ type unifiedRepoProvider struct {
 	workPath         string
 	repoService      udmrepo.BackupRepoService
 	repoBackend      string
-	cli              client.Client
 	log              logrus.FieldLogger
 }

@@ -75,13 +73,11 @@ const (
 func NewUnifiedRepoProvider(
 	credentialGetter credentials.CredentialGetter,
 	repoBackend string,
-	cli client.Client,
 	log logrus.FieldLogger,
 ) Provider {
 	repo := unifiedRepoProvider{
 		credentialGetter: credentialGetter,
 		repoBackend:      repoBackend,
-		cli:              cli,
 		log:              log,
 	}

@@ -121,14 +121,7 @@ func IsPodUnrecoverable(pod *corev1api.Pod, log logrus.FieldLogger) (bool, strin
 		return true, fmt.Sprintf("Pod is in abnormal state %s", pod.Status.Phase)
 	}

-	if pod.Status.Phase == corev1api.PodPending && len(pod.Status.Conditions) > 0 {
-		for _, condition := range pod.Status.Conditions {
-			if condition.Type == corev1api.PodScheduled && condition.Reason == "Unschedulable" {
-				log.Warnf("Pod is unschedulable %s", condition.Message)
-				return true, fmt.Sprintf("Pod is unschedulable: %s", condition.Message)
-			}
-		}
-	}
+	// removed "Unschedulable" check since unschedulable condition isn't always permanent

 	// Check the Status field
 	for _, containerStatus := range pod.Status.ContainerStatuses {
@@ -401,21 +401,6 @@ func TestIsPodUnrecoverable(t *testing.T) {
 			},
 			want: false,
 		},
-		{
-			name: "pod is unschedulable",
-			pod: &corev1api.Pod{
-				Status: corev1api.PodStatus{
-					Phase: corev1api.PodPending,
-					Conditions: []corev1api.PodCondition{
-						{
-							Type:   corev1api.PodScheduled,
-							Reason: "Unschedulable",
-						},
-					},
-				},
-			},
-			want: true,
-		},
 		{
 			name: "pod is normal",
 			pod: &corev1api.Pod{
@@ -41,9 +41,10 @@ const (
 	waitInternal = 2 * time.Second
 )

-// DeletePVAndPVCIfAny deletes PVC and delete the bound PV if it exists and log an error when the deletion fails
-// We first set the reclaim policy of the PV to Delete, then PV will be deleted automatically when PVC is deleted.
-func DeletePVAndPVCIfAny(ctx context.Context, client corev1client.CoreV1Interface, pvcName, pvcNamespace string, log logrus.FieldLogger) {
+// DeletePVAndPVCIfAny deletes PVC and delete the bound PV if it exists and log an error when the deletion fails.
+// It first sets the reclaim policy of the PV to Delete, then PV will be deleted automatically when PVC is deleted.
+// If ensureTimeout is not 0, it waits until the PVC is deleted or timeout.
+func DeletePVAndPVCIfAny(ctx context.Context, client corev1client.CoreV1Interface, pvcName, pvcNamespace string, ensureTimeout time.Duration, log logrus.FieldLogger) {
 	pvcObj, err := client.PersistentVolumeClaims(pvcNamespace).Get(ctx, pvcName, metav1.GetOptions{})
 	if err != nil {
 		if apierrors.IsNotFound(err) {
@@ -69,7 +70,7 @@ func DeletePVAndPVCIfAny(ctx context.Context, client corev1client.CoreV1Interfac
 		}
 	}

-	if err := client.PersistentVolumeClaims(pvcNamespace).Delete(ctx, pvcName, metav1.DeleteOptions{}); err != nil {
+	if err := EnsureDeletePVC(ctx, client, pvcName, pvcNamespace, ensureTimeout); err != nil {
 		log.Warnf("failed to delete pvc %s/%s with err %v", pvcNamespace, pvcName, err)
 	}
 }
@@ -118,12 +119,17 @@ func DeletePVIfAny(ctx context.Context, pvGetter corev1client.CoreV1Interface, p
 }

 // EnsureDeletePVC asserts the existence of a PVC by name, deletes it and waits for its disappearance and returns errors on any failure
+// If timeout is 0, it doesn't wait and return nil
 func EnsureDeletePVC(ctx context.Context, pvcGetter corev1client.CoreV1Interface, pvc string, namespace string, timeout time.Duration) error {
 	err := pvcGetter.PersistentVolumeClaims(namespace).Delete(ctx, pvc, metav1.DeleteOptions{})
 	if err != nil {
 		return errors.Wrapf(err, "error to delete pvc %s", pvc)
 	}

+	if timeout == 0 {
+		return nil
+	}
+
 	err = wait.PollUntilContextTimeout(ctx, waitInternal, timeout, true, func(ctx context.Context) (bool, error) {
 		_, err := pvcGetter.PersistentVolumeClaims(namespace).Get(ctx, pvc, metav1.GetOptions{})
 		if err != nil {
@@ -138,7 +144,7 @@ func EnsureDeletePVC(ctx context.Context, pvcGetter corev1client.CoreV1Interface
 	})

 	if err != nil {
-		return errors.Wrapf(err, "error to retrieve pvc info for %s", pvc)
+		return errors.Wrapf(err, "error to ensure pvc deleted for %s", pvc)
 	}

 	return nil
@@ -333,6 +333,7 @@ func TestDeletePVCIfAny(t *testing.T) {
 		logMessage    string
 		logLevel      string
 		logError      string
+		ensureTimeout time.Duration
 	}{
 		{
 			name:         "pvc not found",
@@ -362,22 +363,6 @@ func TestDeletePVCIfAny(t *testing.T) {
 			pvcName:      "fake-pvc",
 			pvcNamespace: "fake-namespace",
 			pvName:       "fake-pv",
-			kubeReactors: []reactor{
-				{
-					verb:     "get",
-					resource: "persistentvolumeclaims",
-					reactorFunc: func(action clientTesting.Action) (handled bool, ret runtime.Object, err error) {
-						return true, pvcObject, nil
-					},
-				},
-				{
-					verb:     "delete",
-					resource: "persistentvolumeclaims",
-					reactorFunc: func(action clientTesting.Action) (handled bool, ret runtime.Object, err error) {
-						return true, nil, nil
-					},
-				},
-			},
 			kubeClientObj: []runtime.Object{
 				pvcObject,
 				pvObject,
@@ -391,13 +376,6 @@ func TestDeletePVCIfAny(t *testing.T) {
 			pvcNamespace: "fake-namespace",
 			pvName:       "fake-pv",
 			kubeReactors: []reactor{
-				{
-					verb:     "get",
-					resource: "persistentvolumeclaims",
-					reactorFunc: func(action clientTesting.Action) (handled bool, ret runtime.Object, err error) {
-						return true, pvcObject, nil
-					},
-				},
 				{
 					verb:     "delete",
 					resource: "persistentvolumeclaims",
@@ -407,10 +385,10 @@ func TestDeletePVCIfAny(t *testing.T) {
 				},
 			},
 			kubeClientObj: []runtime.Object{
-				pvcObject,
+				pvcWithVolume,
 				pvObject,
 			},
-			logMessage: "failed to delete pvc fake-namespace/fake-pvc with err fake-delete-error",
+			logMessage: "failed to delete pvc fake-namespace/fake-pvc with err error to delete pvc fake-pvc: fake-delete-error",
 			logLevel:   "level=warning",
 		},
 		{
@@ -426,13 +404,6 @@ func TestDeletePVCIfAny(t *testing.T) {
 						return true, nil, errors.New("fake-get-error")
 					},
 				},
-				{
-					verb:     "get",
-					resource: "persistentvolumeclaims",
-					reactorFunc: func(action clientTesting.Action) (handled bool, ret runtime.Object, err error) {
-						return true, pvcWithVolume, nil
-					},
-				},
 			},
 			kubeClientObj: []runtime.Object{
 				pvcWithVolume,
@@ -451,20 +422,6 @@ func TestDeletePVCIfAny(t *testing.T) {
 				pvObject,
 			},
 			kubeReactors: []reactor{
-				{
-					verb:     "get",
-					resource: "persistentvolumeclaims",
-					reactorFunc: func(action clientTesting.Action) (handled bool, ret runtime.Object, err error) {
-						return true, pvcWithVolume, nil
-					},
-				},
-				{
-					verb:     "get",
-					resource: "persistentvolumes",
-					reactorFunc: func(action clientTesting.Action) (handled bool, ret runtime.Object, err error) {
-						return true, pvObject, nil
-					},
-				},
 				{
 					verb:     "patch",
 					resource: "persistentvolumes",
@@ -486,28 +443,39 @@ func TestDeletePVCIfAny(t *testing.T) {
 				pvcWithVolume,
 				pvObject,
 			},
+		},
+		{
+			name:         "delete pv pvc success but wait fail",
+			pvcName:      "fake-pvc",
+			pvcNamespace: "fake-namespace",
+			pvName:       "fake-pv",
+			kubeClientObj: []runtime.Object{
+				pvcWithVolume,
+				pvObject,
+			},
 			kubeReactors: []reactor{
 				{
-					verb:     "get",
+					verb:     "delete",
 					resource: "persistentvolumeclaims",
 					reactorFunc: func(action clientTesting.Action) (handled bool, ret runtime.Object, err error) {
 						return true, pvcWithVolume, nil
 					},
 				},
-				{
-					verb:     "get",
-					resource: "persistentvolumes",
-					reactorFunc: func(action clientTesting.Action) (handled bool, ret runtime.Object, err error) {
-						return true, pvObject, nil
-					},
-				},
-				{
-					verb:     "patch",
-					resource: "persistentvolumes",
-					reactorFunc: func(action clientTesting.Action) (handled bool, ret runtime.Object, err error) {
-						return true, pvObject, nil
-					},
-				},
+			},
+			ensureTimeout: time.Second,
+			logMessage:    "failed to delete pvc fake-namespace/fake-pvc with err error to ensure pvc deleted for fake-pvc: context deadline exceeded",
+			logLevel:      "level=warning",
+		},
+		{
+			name:         "delete pv pvc success, wait won't succeed but ensureTimeout is 0",
+			pvcName:      "fake-pvc",
+			pvcNamespace: "fake-namespace",
+			pvName:       "fake-pv",
+			kubeClientObj: []runtime.Object{
+				pvcWithVolume,
+				pvObject,
+			},
+			kubeReactors: []reactor{
 				{
 					verb:     "delete",
 					resource: "persistentvolumeclaims",
@@ -530,7 +498,7 @@ func TestDeletePVCIfAny(t *testing.T) {
 			var kubeClient kubernetes.Interface = fakeKubeClient

 			logMessage := ""
-			DeletePVAndPVCIfAny(context.Background(), kubeClient.CoreV1(), test.pvcName, test.pvcNamespace, velerotest.NewSingleLogger(&logMessage))
+			DeletePVAndPVCIfAny(context.Background(), kubeClient.CoreV1(), test.pvcName, test.pvcNamespace, test.ensureTimeout, velerotest.NewSingleLogger(&logMessage))

 			if len(test.logMessage) > 0 {
 				assert.Contains(t, logMessage, test.logMessage)
@@ -623,6 +591,7 @@ func TestEnsureDeletePVC(t *testing.T) {
 		pvcName   string
 		namespace string
 		reactors  []reactor
+		timeout   time.Duration
 		err       string
 	}{
 		{
@@ -631,11 +600,27 @@ func TestEnsureDeletePVC(t *testing.T) {
 			namespace: "fake-ns",
 			err:       "error to delete pvc fake-pvc: persistentvolumeclaims \"fake-pvc\" not found",
 		},
+		{
+			name:      "0 timeout",
+			pvcName:   "fake-pvc",
+			namespace: "fake-ns",
+			clientObj: []runtime.Object{pvcObject},
+			reactors: []reactor{
+				{
+					verb:     "delete",
+					resource: "persistentvolumeclaims",
+					reactorFunc: func(action clientTesting.Action) (handled bool, ret runtime.Object, err error) {
+						return true, pvcObject, nil
+					},
+				},
+			},
+		},
 		{
 			name:      "wait fail",
 			pvcName:   "fake-pvc",
 			namespace: "fake-ns",
 			clientObj: []runtime.Object{pvcObject},
+			timeout:   time.Millisecond,
 			reactors: []reactor{
 				{
 					verb:     "get",
@@ -645,7 +630,24 @@ func TestEnsureDeletePVC(t *testing.T) {
 					},
 				},
 			},
-			err: "error to retrieve pvc info for fake-pvc: error to get pvc fake-pvc: fake-get-error",
+			err: "error to ensure pvc deleted for fake-pvc: error to get pvc fake-pvc: fake-get-error",
+		},
+		{
+			name:      "wait timeout",
+			pvcName:   "fake-pvc",
+			namespace: "fake-ns",
+			clientObj: []runtime.Object{pvcObject},
+			timeout:   time.Millisecond,
+			reactors: []reactor{
+				{
+					verb:     "delete",
+					resource: "persistentvolumeclaims",
+					reactorFunc: func(action clientTesting.Action) (handled bool, ret runtime.Object, err error) {
+						return true, pvcObject, nil
+					},
+				},
+			},
+			err: "error to ensure pvc deleted for fake-pvc: context deadline exceeded",
 		},
 	}

@@ -659,7 +661,7 @@ func TestEnsureDeletePVC(t *testing.T) {

 			var kubeClient kubernetes.Interface = fakeKubeClient

-			err := EnsureDeletePVC(context.Background(), kubeClient.CoreV1(), test.pvcName, test.namespace, time.Millisecond)
+			err := EnsureDeletePVC(context.Background(), kubeClient.CoreV1(), test.pvcName, test.namespace, test.timeout)
 			if err != nil {
 				assert.EqualError(t, err, test.err)
 			} else {
@@ -123,8 +123,6 @@ spec:
    ttl: 24h0m0s
    # whether pod volume file system backup should be used for all volumes by default.
    defaultVolumesToFsBackup: true
-    # The labels you want on backup objects, created from this schedule (instead of copying the labels you have on schedule object itself).
-    # When this field is set, the labels from the Schedule resource are not copied to the Backup resource.
    # Whether snapshot data should be moved. If set, data movement is launched after the snapshot is created.
    snapshotMoveData: true
    # The data mover to be used by the backup. If the value is "" or "velero", the built-in data mover will be used.
@@ -133,6 +131,8 @@ spec:
    uploaderConfig:
        # ParallelFilesUpload is the number of files parallel uploads to perform when using the uploader.
        parallelFilesUpload: 10
+    # The labels you want on backup objects, created from this schedule (instead of copying the labels you have on schedule object itself).
+    # When this field is set, the labels from the Schedule resource are not copied to the Backup resource.
    metadata:
      labels:
        labelname: somelabelvalue
@@ -102,3 +102,7 @@ Check that the Cluster 2 is behaving as expected:
    Your data that was backed up from Cluster 1 should now be available on Cluster 2.

 If you encounter issues, make sure that Velero is running in the same namespace in both clusters.
+
+
+## Notice
+If the two clusters couldn't share the snapshots generated by backup, for example migration from EKS to AKS, then please consider using [the file system backup](file-system-backup.md) or [the snapshot data mover](csi-snapshot-data-movement.md).
@@ -241,6 +241,8 @@ data:
  <old-node-name>: <new-node-name>
 ```

+Note: This feature is deprecated as of Velero 1.15, following Velero deprecation policy. This feature is primarily used to remedy some problems in old Kubernetes versions as described [here](https://github.com/vmware-tanzu/velero/pull/2377). It may not work with the new features of Kubernetes and Velero. E.g., it doesn't work for PVCs with ```WaitForFirstConsumer``` as the ```volumeBindingMode```. These kind of PVCs won't be bound until the pod is scheduled and the scheduler will overwrite the selected-node annotation to the node where the pod is scheduled to.  
+
 ## Restoring into a different namespace

 Velero can restore resources into a different namespace than the one they were backed up from. To do this, use the `--namespace-mappings` flag:
@@ -123,12 +123,12 @@ spec:
    ttl: 24h0m0s
    # whether pod volume file system backup should be used for all volumes by default.
    defaultVolumesToFsBackup: true
-    # The labels you want on backup objects, created from this schedule (instead of copying the labels you have on schedule object itself).
-    # When this field is set, the labels from the Schedule resource are not copied to the Backup resource.
    # Whether snapshot data should be moved. If set, data movement is launched after the snapshot is created.
    snapshotMoveData: true
    # The data mover to be used by the backup. If the value is "" or "velero", the built-in data mover will be used.
    datamover: velero
+    # The labels you want on backup objects, created from this schedule (instead of copying the labels you have on schedule object itself).
+    # When this field is set, the labels from the Schedule resource are not copied to the Backup resource.
    metadata:
      labels:
        labelname: somelabelvalue
@@ -123,8 +123,6 @@ spec:
    ttl: 24h0m0s
    # whether pod volume file system backup should be used for all volumes by default.
    defaultVolumesToFsBackup: true
-    # The labels you want on backup objects, created from this schedule (instead of copying the labels you have on schedule object itself).
-    # When this field is set, the labels from the Schedule resource are not copied to the Backup resource.
    # Whether snapshot data should be moved. If set, data movement is launched after the snapshot is created.
    snapshotMoveData: true
    # The data mover to be used by the backup. If the value is "" or "velero", the built-in data mover will be used.
@@ -133,6 +131,8 @@ spec:
    uploaderConfig:
        # ParallelFilesUpload is the number of files parallel uploads to perform when using the uploader.
        parallelFilesUpload: 10
+    # The labels you want on backup objects, created from this schedule (instead of copying the labels you have on schedule object itself).
+    # When this field is set, the labels from the Schedule resource are not copied to the Backup resource.
    metadata:
      labels:
        labelname: somelabelvalue
@@ -123,8 +123,6 @@ spec:
    ttl: 24h0m0s
    # whether pod volume file system backup should be used for all volumes by default.
    defaultVolumesToFsBackup: true
-    # The labels you want on backup objects, created from this schedule (instead of copying the labels you have on schedule object itself).
-    # When this field is set, the labels from the Schedule resource are not copied to the Backup resource.
    # Whether snapshot data should be moved. If set, data movement is launched after the snapshot is created.
    snapshotMoveData: true
    # The data mover to be used by the backup. If the value is "" or "velero", the built-in data mover will be used.
@@ -133,6 +131,8 @@ spec:
    uploaderConfig:
        # ParallelFilesUpload is the number of files parallel uploads to perform when using the uploader.
        parallelFilesUpload: 10
+    # The labels you want on backup objects, created from this schedule (instead of copying the labels you have on schedule object itself).
+    # When this field is set, the labels from the Schedule resource are not copied to the Backup resource.
    metadata:
      labels:
        labelname: somelabelvalue
@@ -19,7 +19,7 @@ package test
 import (
 	"context"
 	"fmt"
-	"math/rand"
+	"math/rand/v2"
 	"strings"
 	"time"

@@ -105,8 +105,7 @@ func (t *TestCase) Init() error {
 }

 func (t *TestCase) GenerateUUID() string {
-	rand.Seed(time.Now().UnixNano())
-	return fmt.Sprintf("%08d", rand.Intn(100000000))
+	return fmt.Sprintf("%08d", rand.IntN(100000000))
 }

 func (t *TestCase) CreateResources() error {
@@ -168,9 +167,16 @@ func (t *TestCase) Verify() error {
 func (t *TestCase) Start() error {
 	t.Ctx, t.CtxCancel = context.WithTimeout(context.Background(), 1*time.Hour)
 	veleroCfg := t.GetTestCase().VeleroCfg
-	if (veleroCfg.CloudProvider == Azure || veleroCfg.CloudProvider == AWS) && strings.Contains(t.GetTestCase().CaseBaseName, "nodeport") {
+
+	if (veleroCfg.CloudProvider == Azure || veleroCfg.CloudProvider == AWS) &&
+		strings.Contains(t.GetTestCase().CaseBaseName, "nodeport") {
 		Skip("Skip due to issue https://github.com/kubernetes/kubernetes/issues/114384 on AKS")
 	}
+
+	if veleroCfg.UploaderType == UploaderTypeRestic &&
+		strings.Contains(t.GetTestCase().CaseBaseName, "ParallelFiles") {
+		Skip("Skip Parallel Files upload and download test cases for environments using Restic as uploader.")
+	}
 	return nil
 }

@@ -178,11 +184,15 @@ func (t *TestCase) Clean() error {
 	veleroCfg := t.GetTestCase().VeleroCfg
 	if !veleroCfg.Debug {
 		By(fmt.Sprintf("Clean namespace with prefix %s after test", t.CaseBaseName), func() {
-			CleanupNamespaces(t.Ctx, t.Client, t.CaseBaseName)
+			if err := CleanupNamespaces(t.Ctx, t.Client, t.CaseBaseName); err != nil {
+				fmt.Println("Fail to cleanup namespaces: ", err)
+			}
 		})
 		By("Clean backups after test", func() {
 			veleroCfg.ClientToInstallVelero = &t.Client
-			DeleteAllBackups(t.Ctx, &veleroCfg)
+			if err := DeleteAllBackups(t.Ctx, &veleroCfg); err != nil {
+				fmt.Println("Fail to clean backups after test: ", err)
+			}
 		})
 	}
 	return nil
@@ -38,6 +38,8 @@ const AWS = "aws"
 const Gcp = "gcp"
 const Vsphere = "vsphere"

+const UploaderTypeRestic = "restic"
+
 var PublicCloudProviders = []string{AWS, Azure, Gcp, Vsphere}
 var LocalCloudProviders = []string{Kind, VanillaZFS}
 var CloudProviders = append(PublicCloudProviders, LocalCloudProviders...)
				`@@ -0,0 +1 @@`
				`Ignore missing path error in conditional match`
				`@@ -0,0 +1 @@`
				`Add the design for data mover micro service`
				`@@ -0,0 +1 @@`
				`Don't consider unschedulable pods unrecoverable`
				`@@ -0,0 +1 @@`
				`Expose the VolumeHelper to third-party plugins.`
				`@@ -0,0 +1 @@`
				`Fix issue #7904, add the deprecation and limitation clarification for change PVC selected-node feature`
				`@@ -0,0 +1 @@`
				`Skip PV patch step in Restoe workflow for WaitForFirstConsumer VolumeBindingMode Pending state PVCs`
				`@@ -0,0 +1 @@`
				`New data path for data mover ms according to design #7574`
				`@@ -0,0 +1 @@`
				`Check whether the namespaces specified in namespace filter exist.`
				`@@ -0,0 +1 @@`
				`Check whether the volume's source is PVC before fetching its PV.`
				`@@ -0,0 +1 @@`
				`Fix issue #7972, sync the backupPVC deletion in expose clean up`